One of the most important things that we can do, in creating AGI (artificial general intelligence), is to work through the latent variable issues that are foremost in AI and machine learning (ML) research now. We identified these in our July 10, 2023 blogpost on Latent Variables in Neural Networks and Machine Learning.
We wrote that blogpost without an accompanying YouTube vid, but it had become such an important resource for students Northwestern University’s Master of Science in Data Science (MSDS) course on AI and Deep Learning (MSDS 458) that we devoted a lot of our last Synch session to this topic.
And we were left with the feeling … we needed to go into this latent variables topic more deeply.
That’s why, this Sunday (Dec. 9th, 2023), we’re hosting the inaugural Themesis AI Salon on the topic of Latent Variables and AGI.
How to Join the AI Salon
Salon invitations are given ONLY to people who are members of the Themesis community. This means that in order to join, you need to Opt-In with Themesis, go through all the opting-in protocols (respond to the confirmation email, etc.).
Once you’ve opted-in, then check your email inbox for word about the upcoming Salon. You may not get that email immediately – the invitation emails are sent out periodically as the Salon date approaches.
It will help if you move your Themesis emails to whatever folder is your “preferred” reading folder.
Minimal Preps
Unlike a lecture where you can just show up, this is a Salon – a place for discussion.
I WILL present a short (5-10 min) overview. But the point of the overview will not be to teach; just to ground us in some key points that are relevant to our theme.
Thus, if you’re going to show up, we recommend AT LEAST a minimal prep. This could be anywhere between two-to-eight hours, stretched over the week leading up to the Salon itself.
In the rest of this post, we suggest some reading (and YouTube-watching) that will inform you – in the order of increasing difficulty / immersion.
The Easiest Starting Points
Just to start wrapping our heads around the notion of self-supervised learning, I like this post by Gaudenz Boesch out of Viso on Self-Supervised Learning: Everything that You Need to Know (2024). (Hmph. It’s written in 2023.)
Another very fast, skim-level read is this GeeksforGeeks paper on Self-Supervised Learning (SSL).
Neither of these gives depth, but if you’re trying to get a fast overview before diving deep, they’re good starting points.
The Prior Post (with Link to LeCun YouTube)
I started putting together materials for this Salon back in July, 2023, with this post on Latent Variables (mentioned above). It contained THIS Yann LeCun YouTube:
Even though this is very good – and solid – LeCun supposes a great deal of knowledge on the part of his viewer. In the prior blogpost, I gave a few summary points – and the important material in LeCun’s presentation start about 11 or 12 minutes in – but this is not a good place in which to “learn” the methods that he references. Just a good place to put that knowledge that you already have into context.
Neurophysiology and Latent Variables: Two Recent MIT Studies
As a next step, and as a prelude to going deeper, a recent article published in MIT News discusses two articles published by two different groups, each addressing a different form of self-supervised learning and relating it to how mammalian brains might learn. Both of these articles are scheduled for presentation at the December 2023 NeurIPS conference.
The first of these two articles is by Nayebe et al. (2023). Their key finding is that “neural responses are currently best predicted by models trained to predict the future state of their environment in the
latent space of pretrained foundation models optimized for dynamic scenes in a self-supervised manner.”
The second is by Schaeffer, Khona, and Fiete (2023).
[This section is in progress; check back for updates – AJM, Weds., Dec. 6th, 2023; 3:44PM HI Time.]
A Possible Cause for Latent Variable Problems
It is possible … just possible, not saying this for certain … that one of the reasons that we’re having so much trouble bringing latent variables into the useful state that we desire (i.e., mitigating issues such as collapse) is that these latent elements are just too simple for the task at hand.
Backtracking just a bit … if we’re talking about variational autoencoders, then we’re really talking about (restricted) Boltzmann machines. If this is what we’re dealing with, then our latent variables are a set of – essentially, very simple on/off nodes, with not much internal complexity. The only things that we can really vary during the training are the connection weights between whatever we have as the “observed” world (“visible” nodes) and the states of these “latent” nodes.
We can’t expand the set of latent variables too much, and we can’t make it too small.
And yet, we’re attempting to use this (very constrained) latent variable set to describe (potentially) VERY large-dimensional, complex spaces.
That might just be a bit more than this simple set of latent variables can do.
Not saying that for certain; it’s just a thought nudging around my brain.
And that thought is inspired – very much – from the studies that I did in the neurophysiology that I believe can support AGI, which I put out two weeks ago as the Next-Era AGI: Neurophysiology Basis, and the accompanying YouTube:
Variational Methods
A LONG-TERM READING PLAN might include this excellent paper on variational methods by Kingma and Welling (2019). This one is a great foundation, but not an easy read – nor particularly intuitive. (Although, for a very mathematically-grounded text, Kingma and Welling do their very best to be intuitive and provide interpretations.)
Resources and References
Neurophysiology and Latent Variables
- Nayebi, Aran, Rishi Rajalinham, Mehrdad Jazayeri, and Guangyu Robert Yang. 2023. “Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes.” (To be presented at the) 2023 Conference on Neural Information Processing Systems (NeurIPS). arXiv:2305.11772v2 [cs.AI] 25 Oct 2023. (pdf)
- Schaeffer, Rylan, Mikail Khona, and Ila Rani Fiete. 2023. “No Free Lunch from Deep Learning in Neuroscience: A Case Study through Models of the Entorhinal-Hippocampal Circuit.” (To be presented at the) 2023 Conference on Neural Information Processing Systems (NeurIPS). bioRxiv (Aug. 15, 2023; updated Oct., 2023). doi:10.1101/2022.08.07.503109. (pdf)
Variational Methods
AJM’s Note: This is the Kingma and Welling paper that I mentioned above in Variational Methods.
- Kingma, Diederik P. and Max Welling. 2019, “An Introduction to Variational Autoencoders”, Foundations and Trends (R) in Machine Learning 12(4) (28 Nov. 2019): 307-392. doi:10.1561/2200000056. (NOW listing at publisher’s site). (Available as arXiv:1906.02691v3 [cs.LG] 11 Dec 2019, arXiv listing.)
1 comment
Comments are closed.