Detecting and managing differences between AI and Humans is becoming more complex daily. We are the first to use scientific method rooted in physics to tackle these complexities.
Technology for AI-generation of text/language, images, and audio has attained the point where it is becoming difficult to distinguish it from human or natural sources. This is important not only for authorship integrity and trustworthiness but also for cleanly identifying data produced by such models versus that which is used for training such models, avoiding a self-reinforcing cycle of inbred models.
We found that Variety, Energy, and their ratio, V/E, can be used to reliably predict whether text is human- or AI-written, and even released a free Chrome Extension to prove it.
Here we show this same technique can also apply to audio and images. While we finish up on a larger, more thorough exposition of V/E in identifying AI content, this short piece acts as a precursor wave to attract interest and comments.
Variety
The Achille’s Heel of autoregressively-generated AI content (which includes GPT-models, Dall-E, Diffusion models, etc.) is, by construction, in their global structure: e.g., any model that iteratively constructs text word-by-word cannot be expected to show an ingenious correlation between distantly separated words. A global metric may bring this out.
For any abstract pattern (text, audio, image, etc.), we focus here on one global metric called Variety, roughly defined by ‘how different any part of the pattern is from any other part.’ Introduced by physicists in the 1990s to address the origins of quantum physics, we adopt a more pedestrian though imminently practical usage in tagging AI-generated content. We craft what seems to be a reasonable definition of Variety (though there is certainly much freedom here) wherein the pattern is a function and we directly measure how different the function is in each point’s neighborhood from all the other possible neighborhoods:
Here x and y denote all pairs of points in the domain, neighborhoods of which we compare by displacements of size z. The reader can easily see that a constant function has Variety=0, and the quantity vaguely increases with the nonlinearity of f, but not uniformly: too much repetition or periodic behavior in f will again lower Variety, so that the function with the most Variety must exhibit enough, but not too much, change in its value in a given interval. Random functions, in particular, have only a middling variety.
Now it is only human to be tempted to ask the “God question”: what is the function of maximal variety? This would be akin to finding “the most interesting function.” Without further constraints, this question is very hard to answer. But for functions restricted to the form f=sin(nx) in the unit interval, for example, Variety is maximized for n=6:
This makes intuitive sense because, recalling our motivating definition, every point on the function that oscillates exactly once on this interval (n=6 is closest to 2pi ~ 6.283..) is as different as it could be from other points in its value and derivatives: of all the curves above, an ant crawling on the n=6 curve would at all times know its position best by local height and curvature.
Without restricting the function class, we may attempt to empirically ‘guess’ some functions and hope to zero in on the ones with the highest Variety. We can, for example, define a simple feed-forward neural net with bounded output value and a loss function equal to (negative) Variety and train with random inputs on a given (discrete) interval. This gives some idea of what high-variety-like functions may look like.
Here we see the output of such a trained net can easily achieve a Variety higher than its random input. Since the net is randomly initialized, however, every training essentially converges to a different solution, which may not be maximal. We thus leave the “god question” for the time being …
Energy
The fact that we had to bound a function f to maximize Variety follows from the fact that Variety itself scales with f. Instead, we can maximize Variety subject to penalizing scale, a ‘laziness’ principle that rewards minimal amplitude changes. As a first attempt, we take this to mean, while maximizing Variety, one must also minimize the ‘Energy,’ defined for simplicity (and some physicality) as the integrated first-derivative-squared of the function over its interval:
Our intuition is that patterns in Nature exist with a certain balance between Variety and Energy, maximizing the ratio V/E, and hypothesize that artificial simulations of Nature, on the other hand, won’t optimize for this.
Language/Text
As a brief summary, we mapped words to corpus frequencies to create a 1-D representation of text, e.g., for the phrase “the quick brown fox jumped over the fence,”
and directly applied the Variety and Energy definitions above to the word amplitudes. We found the ratio of Variety to Energy was a useful feature in distinguishing human from ChatGPT text on a statistical, ensemble basis:
In the meantime, we built a naïve logistic regression classifier using V/E as a feature (alongside vocabulary and sentence length features), allowing us to identify ChatGPT-generated text with 82% accuracy and a trustworthiness metric, putting us at the top of commercial AI text id tools pack next to GPTZero, and leading ZeroGPT, Sapling, CopyLeaks, and others. This allows us, for example, to reliably distinguish real from AI-generated news articles:
Chrome Plugin distinguishing human (left) from AI-written journalism (right)
Please feel free to play with this tool and let us know what you think. Optimizing performance (tuning features, a bag of ML tricks, etc.) is in the works, so more on that in a later post.
Audio/Music
Like text, audio data can also be seen as a 1-D signal (at any given time step you have a list of frequency-amplitude pairs) so the same technique used for text above can apply to say, midi files.
Comparing a classical Beethoven piece (Fur Elise) to an AI-generated composition, for example, we see that the AI piece has less Variety, more Energy, and hence a lower value of the ratio V/E.
Analysis is ongoing for more complex musical pieces and in other formats (e.g., wav), but so far V/E seems to work here too.
Images
The foregoing definitions of Variety and Energy apply equally well to data in any number of dimensions. For 2-dimensional data, for example, the neighborhoods around each point are rings instead of intervals. Training a model to find the maximal variety pattern in a given patch of area is of course harder than in 1-D, but we can again approximate with a simple feed-forward neural net trained on a discrete grid.
For a 32x32 grayscale image, for example, the output of such a net has much higher variety than its random input, and even more than a (famous) human-designed image:
In terms of V/E, however, the human-designed image wins easily. Inspired by this, we investigate whether all artificially created images tend to have smaller V/E.
Trying this hypothesis on images generated with StableDiffusion, with the prompt “photo of a snowy street in Buffalo, NY after a blizzard with 4 feet of snow”, chosen from a recent event to avoid any potential correlation with the pre-2022 model training set, we generated several hundred photos and also extracted actual photos taken of said weather event, randomly selected from Google Images with the same search string.
Noting that the AI-generated images looked like rather similar head-on street views, and not wanting this bias to creep into our analysis, we decided to sample 50x50 sub-patches:
Now measuring V,E, and V/E across these subpatches, we found that statistically, the AI images have more Variety and Energy, but lower V/E. This was also the case for another domain, human face generation: we compared 1000 real faces with 1000 StyleGAN-generated faces, again comparing across random sub patches:
We see again that AI-generated images have lower V/E. Thus, for images, as for text and audio, V/E may be a generically useful feature in a classifier to identify AI-generation.
Conclusion
For the first time, we see there are generalized automated metrics one can apply to detect AI-generated content in multiple formats based on the fact that current AI models optimize locally and tend to misalign global structures. Optimizing the ratio of Variety to Energy, V/E seems to be something humans and Nature are geared to do, while current AI methods lacking global constraints are blind to this. Variety and Energy are definitively correlated with whether content is AI-generated or not, but by themselves may not be sufficient. Combination with other, domain-specific features may allow one to effectively train a classifier, as we have done for text.
Comments