Teaching yourself generative AI (“gen-AI”) has to be one of the hardest things in the world.
The really important, classic papers – the ones that you WISH THAT YOU COULD READ – all presuppose that you have a lot of knowledge coming in, about all SORTS of things.
The situation is the same as it was back in 2017, when someone posted on Quora this pained, frustrated plea:
How can I develop a deep/unified view of statistical mechanics, information theory and machine learning? I learned all three fields independently, and many of the same concepts show up in all of them. I feel my understanding lacks a unified view.“
Query post on Quora, several years ago.
What we need is just enough of the core, fundamental concepts – JUST ENOUGH, no more, no less – and then have someone relate these concepts to how they support energy-based neural networks (e.g., Boltzmann machines and all their kin) as well as variational inference (and all of ITS kindred).
There’s a lot of material out there, but too much goes way beyond what we need, or uses more abstract (read “fancy”) mathematics than we absolutely need.
All too often, there are these disparate concepts – but no one is pulling them together in the sense of “what makes energy-based neural networks WORK?”
Book Chapters. Recently Updated, Access PDFs Now
You know (you DO know, don’t you?) that Themesis has launched the early-alpha version of what will ultimately be our flagship, the Top Ten Terms in Statistical Mechanics.
What started as a simple vocabulary lesson has evolved into so, SO MUCH more!
You can enroll in the full three-week course:
https:themesis.thinkific.com
We officially started on Monday, Jan. 15th, but we’re running asynch, and I am VERY involved in talking w/ students and getting real-time feedback, so this is an EXCELLENT TIME to join.
But even if you don’t enroll, we’re making some resources available to everyone, free.
Over this last week, I’ve revised three book chapters from the book that I started working on pre-2000. (And pre-switch to YouTubes, because starting early 2000, no one had the energy or focus to read highly technical materials – YouTubes were the best way in which to connect with people)
These are three chapters from Statistical Mechanics, Neural Networks, and Artificial Intelligence.
https://themesis.com/book/
These chapters give you the BASICs on statistical mechanics – just the terms that you need to know to understand references in some classic papers. Also, you get Chapter 9, which will help you immediately understand the connection between the equations and the architecture (and the capabilities) of the Hopfield neural network and the (restricted) Boltzmann machine.
Three Chapters Plus the Précis
Précis
The Précis for Statistical Mechanics, Neural Networks, and Artificial Intelligence (the book) is a short overview of the book’s contents. It can be read independently of the book, and can be used as an overview and introduction.
Three Easy-to-Read Chapters
These chapters each have just enough in the way of equations to do the job. Simplest possible way of writing the equations. LOTS of figures and diagrams. LOTS of explanatory text.
They are the easiest-possible, gentlest introduction to the truly arcane subject of statistical mechanics, which underlies ALL of energy-based neural networks and variational inference.
- Chapter 9: The Hopfield Network and the Boltzmann Machine – a relatively easy-to-read discussion of the single equation (with two variants) that governs both the Hopfield network and the Boltzmann machine; chapter presents the two variant equations (one from each network), eight figures, lots of explanation-text – the EASIEST possible discussion of how their respective equations connect to their architectures and vice-versa,
- Chapter 10: Introduction to Statistical Mechanics: Microstates and the Partition Function – statistical mechanics relative to neural networks – as reader-friendly as statistical mechanics can possibly be; lots of figures, example of computing microstates (essential to understanding partition functions, entropy, and free energy).
- Chapter 11: Free Energy – free energy relative to neural networks and variational inference – a lot of attention to notation.
Again, these are free, they are as EASY-TO-READ as I can possibly make them, and it would be a good use of your time to read these over the weekend.
The link, once again, is:
https://themesis.com/book/
The Kullback-Leibler Divergence – and the REVERSE Kullback-Leibler Divergence
Another very important element in gen-AI is the notion of the Kullback-Leibler divergence. (Commonly called the “KL divergence.”)
I had made a blooper. YET AGAIN. I misunderstood something very important – about how it was not the actual KL divergence that was being used in energy-based neural networks and variational inference, but the REVERSE KL divergence.
I’ll be working on this over the next several days. Expect to hear more.
Until then – please enjoy the book chapters!
Best wishes from all of us at Themesis to you! – AJM