Last night, a bit blurry-eyed from video editing, I cruised the latest AI news to find yet a new trouble-spot with LLMs. When an LLM is recursively trained on its own output, then that LLM’s output becomes progressively degraded.
I went to bed with that thought swimming in my head … and woke up with a thought about salmon fishing.
So, the analogy is this. When we raise fish in fish-farms (aquaculture), then its much like other forms of raising poultry or animals for what they generate (eggs or milk), or to directly consume those animals. This means: crowded conditions, potential for disease, and (in the case of fish-farming) numerous horror stories of them swimming around in endless circles, eating the feces produced by the fish immediately in front of them.
Created 14 July 2028; uploaded 22 July 2018.
Yuchy-thought, right?
We can imagine (with the garbage-in, garbage-out analogy) that this does not give us the highest-quality of “produced food.”
Three Key Parallels: Feeding Salmon; Feeding LLMs
Three immediate parallels jump out:
- Quality of the output is correlated with quality of the input (“Garbage in; garbage out”),
- Environmental protection is important, and
- The real issue is both BIGGER and DIFFERENT FROM the apparent issue.
Parallel 1: LLMs Degrade When They Feed on Themselves
I learned of Shumailov et al.’s recent (2024) publication from an article put out just two days ago, by Elizabeth Gibney. (
We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs).”
Ilia Shumailov, Nature, 2024.
The need to prune content more attentively has led to changes in Google’s core search engine algorithms.
We all know that the quality of search engine returns has decreased remarkably over the last several months, right?
In fact, Google has recently retrenched its search capability (Chopra, 2024, Germain, 2024) – with the intent of focusing on “quality” content – that is, avoiding AI-generated content. This has had HUGELY mixed result, with many legitimate businesses losing massive amounts of traffic. However, we see the intention (of finding and presenting high-quality content) as being useful.
Of course, Google has hastened to do this because of emerging competition from Perplexity.ai, Microsoft’s Bing, and (soon, now in early test phase) OpenAI’s SearchGPT. (See Robison, 2024, for recent word on SearchGPT.)
The point is – we have already put so much AI-generated {*ahem*} “fecal content” into our internet that the quality of “consumable material” has degraded.
Substantially.
As a result, Google (motivated in part by this, and in part by competition) is attempting to “not-consume” AI-generated content.
Who can blame them? (I don’t like reading AI-generated “gunk” either.)
Parallel 2: Environmental Protection
Salmon are big, tough, voracious fish. Sometimes, they “break out” of their fisheries. Now free to scavenge in their surround, they can quickly exhaust an ecosystem. A “farmed fish” – once escaped – is a danger to its environment.
Similarly, a “jailbreaking” LLM is a dangerous thing. (See work at Anthropic; Hubinger et al. 2024, Anil et al., 2024.)
Parallel 3: The Real Issue Is More Complex
The real problem is not that we have and will use LLMs, just as the real problem is not that we have industrial fishing. (Similarly, we have industrial farming of poultry, various animals that we raise for slaughter, and industrial farming of a wide range of plants.)
The real problem is that we are now creating an “AI ecosystem,” on which we will become increasingly dependent, and this ecosystem needs to be managed intelligently so that we have enough “AI content diversity,” much as we need ecodiversity in our food supply, and we need to protect and maintain our various distinct ecosystems.
Reading about LLMs “feeding on themselves” motivated me. I was moved to go find LeCun’s “JEPA” paper again – the one that I’ve been slowly reading (and in parts, re-reading) for the past two months.
And earlier this week, I was gathering my courage and wading into Friston et al.’s “Free Energy Principle Simplified …” paper (2023). (Hint: It starts with the Langevin equation. It is NOT simple!)
And of course, I’ve been reading Hafner et al.’s (2020, rev. 2022) paper on “Action Perception Divergence.”
So … Hafner et al. includes Friston; there is a community who have been evolving active inference into Action Perception Divergence (APD). We can group these two papers (Friston et al. 2023 and Hafner et al., 2020, rev. 2022) into a set of evolving “active inference” works.
LeCun’s JEPA (2020) is a bit more stand-alone.
What unifies these schools of thought (Friston et al., LeCun) is that they BOTH present ways of thinking about AI (AGI) systems that have both “action” and “perception” agents.
The way in which Hafner et al. and Friston et al. think about these agents is very different from LeCun. A key difference is that the Hafner/Friston approach is inherently generative. LeCun, on the other hand, is eager to point out the weakness of generative methods. (He starts by identifying limitations of contrastive divergence, which is used to train energy-based neural networks, specifically Boltzmann machines.)
What to Make of All This
Three things stand out as “lessons to be learned”:
- “Objects in the mirror …” – AGI is closer than we think We have top scientists from two VERY different groups, with VERY different approaches/methods, addressing the same problem. It will not take much longer to create autonomous or semi-autonomous AI systems that go running around, both “sensing” their environments and “acting upon” their environments. We can only speculate as to what will happen as this takes place. (Likely, much more than re-arrangements of search engine results.)
- We have already introduced a massive amount of AI-generated pollution into our AI ecosystem – much as we’ve introduced plastics and microplastics and various toxic chemicals and heavy metals into our physical ecosystem. It’s been easy to casually introduce this garbage (and much of what has been created in the past two years IS “garbage”), and it will be MUCH harder to remove it than it was to let it in.
- We are now about to introduce “agents” into this AI ecosystem – and the agents will make their influence felt. It is up to us to decide whether we’re introducing a series of invasive species or a series of controlled and helpful ones. (There will likely be multiple instances of both.)
This is so much beyond, so hugely BIGGER than anything as small-scaled as even the largest LLM. It’s both hugely bigger and DIFFERENT FROM any form of search engine.
We’re actively creating an AI ecosystem that not only contains resources (knowledge, links to knowledge, and abilities to help us do tasks), but one in which various elements will operate with minimal control.
It is not unreasonable to expect that there will be some “breakouts.”
If there can be breakouts in fish hatcheries, why not in AIs?
This is not to be fear-mongering, but … if AIs can be hacked, what will happen when they have more “motility” – that is, enabled actions?
As an example, right now AIs are still relatively moored to their surroundings. By way of analogy, think of ocean polyps that are attached to existing reefs or the ocean floor.
The essence of what Hafner et al. and Friston et al. are proposing (as one school, with corporate homes with both Google and Verses.ai), and LeCun (with a home at Meta) is that emerging AGIs will be able to “act.” We’re now designing methods to control these actions.
The AIs that we have right now are – in an evolutionary sense – close to corals. They are attached to specific sites. They do not roam freely. (Also, they’re really not very smart.)
Who knows what the future will bring?
Alianna J. Maren, Ph.D.
Founder and Chief Scientist, Themesis, Inc.
Friday, July 26, 2024.
How to Be Informed
We regularly produce not only blogposts but also YouTubes, and offer short courses and host a monthly Salon. To stay informed (and get first word on any flash sales), sign up with us.
Step 1. Go to our “About” page.
Step 2. Scroll down, find the “Opt-In” form, fill it in.
Step 3. Do the “email confirmation” thing – and then move your Themesis emails to your preferred Inbox – and be sure to open them! (Opening your emails is an important step – we DO clean house occasionally!)
References and Resources
LLM References
- Gibney, Elizabeth. 2024. “AI Models Fed AI-Generated Data Quickly Spew Nonsense.” Nature (2024 July 24). (Accessed July 25, 2024; available at Nature.)
- Shumailov, I., Z. Shumaylov, Y. Zhao, et al. 2024. “AI Models Collapse when Trained on Recursively Generated Data.” Nature 631: 755–759. doi:10.1038/s41586-024-07566-y (Accessed July 26, 2024; available at Nature.)
Google Search and Related
- Chopra, Tina. 2024. “Understanding the latest Google algorithm changes and what it means for AI content.” LinkedIn. (March 14, 2024). (Accessed July 26, 2024, available on LinkedIn.)
- Germaine, Thomas. 2024. “Google just updated its algorithm. The Internet will never be the same.” BBC Online (15 May 2024). (Accessed July 26, 2024; available at BBC.)
- Robison, Kyla. 2024. “OpenAI announces SearchGPT, its AI-powered search engine.” The Verge (July 25, 2024). (Accessed July 26, 2024; OpenAI’s SearchGPT.)
Jailbreaking in AIs
- Hubinger, Evan, et al. 2024. “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.” arXiv:2401.05566v3 [cs.CR] 17 Jan 2024. (Accessed Jun 18, 2024; available from arXiv.)
Here’s a more recent Anthropic paper on “many-shot jailbreaking.”
- Anil, C. et al. 2024. “Many-shot Jailbreaking.” Anthropic White Paper. (April 2, 2024). (Accessed Jun 18, 2024; available at Anthropic.)
Important AI Papers (“Action” and “Sensing” Agents)
- Friston, Karl, Lancelot Da Costa, Noor Sajid, Conor Heins, Kai Ueltzhöffer, Grigorios A. Pavliotis, and Thomas Parr. 2023. “The Free Energy Principle Made Simpler but Not Too Simple.” Physics Reports 1024 (19 June 2023):1-29. (Accessed July 27, 2024, available online at Volume 1024 .) (Available also at arXiv:2201.06387v3 [cond-mat.stat-mech]; accessed July 27, 2024. Available online at arXiv.)
- Hafner, Danijar, Pedro A. Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess. “Action and Perception as Divergence Minimization.” arXiv:2009.01791v3 [cs.AI] 13 Feb 2022. (Accessed May 20, 2024. Available online at arXiv.)
- LeCun, Yann. 2022. “A Path Towards Autonomous Machine Intelligence. Version 0.9.2, 2022-06-27” OpenReview (June 27, 2022). (Accessed May 20, 2024. Available online at OpenReview.)
Salmon Fishery Resources
I found this blogpost by Emily De Sousa useful – it is recent enough (2021), it is produced by someone who has expertise in the aquaculture industry, and it seems well-balanced. (A lot of the articles /YouTubes on this subject are designed to elicit emotional responses.)
- De Sousa, Emily. 2021. “Farmed Salmon Myths Debunked by a Fisheries Scientist.” Seaside with Emily (Blogpost series; Dec. 15, 2021). (Accessed July 26, 2024; available at Salmon Fishing.)