Understanding Entropy: Essential to Mastering Advanced AI

Have you been stumped when trying to read the classic AI papers? Are notions such as free energy and entropy confusing? This is probably because ALL areas of advanced AI are based, to some extent, on statistical mechanics. That means that you need to understand some “stat mech” rudiments to get through those papers.

One of the key concepts in statistical mechanics is that of free energy. It doesn’t just show up in all major AI areas; it is foundational. Figure 1 shows key areas where stat mech provides the basis.

Figure 1. Free energy is foundational to variational Bayes. The most interesting and promising area of AI is active inference, which uses variational Bayes as a starting point. This is likely to supersede work done in reinforcement learning. Taken from the YouTube video on Statistical Mechanics: Foundational to AI; link given below.

Figure 1 shows the role of free energy in variational Bayes. You can get a good start on understanding how variational Bayes works with the first entropy-focused blogpost: http://www.aliannajmaren.com/2018/02/13/wrapping-our-heads-around-entropy/

You can access that full YouTube vid, from which Figure 1 was taken, here:

Maren, Alianna J. 2021. “Statistical Mechanics: Foundational to Artificial Intelligence.” Themesis YouTube Channel (Aug. 26, 2021). Accessed March 27, 2022.

The YouTube presented above discusses free energy in an overall way, with particular attention to the energy term. (Other YouTube vids on the same Themesis YouTube channel give more information.)

In contrast to the energy (enthalpy) term in free energy, we also need to understand entropy.

Two Useful Blogposts to Get Started

This first blogpost gives you the basic entropy equation, along with essential graphs – so that you get a clear visual of how entropy works.

Maren, Alianna J. 2018. “Wrapping Our Heads around Entropy.” aliannajmaren.com. (Feb. 13, 2018). Accessed Mar. 28, 2022.

http://www.aliannajmaren.com/2018/02/13/wrapping-our-heads-around-entropy/

This second blogpost helps you understand entropy in the overall context of a free energy equation. Coupled with the blogpost above, you’ll get a pretty solid grounding – without spending a year studying statistical mechanics!

Maren, Alianna J. 2018. “What We Really Need to Know about Entropy.” aliannajmaren.com. (Feb. 28, 2018). Accessed Mar. 28, 2022.

http://www.aliannajmaren.com/2018/02/28/what-we-really-need-to-know-about-entropy/

To your health, well-being, and joyous success! – A.J. Maren

Book Resources

Maren, Alianna J. (Draft; in progress.) Statistical Mechanics, Neural Networks, and Artificial Intelligence. Accessed Mar. 28, 2022. http://www.aliannajmaren.com/book/. In particular, see drafts for Chapters 9 (“The Hopfield and Boltzmann Machine Neural Networks”) and 10 (“Statistical Mechanics – the Gentlest Possible Introduction”)

Previous Related Blogs

Maren, Alianna J. 2018. “What We Really Need to Know about Entropy.” aliannajmaren.com. (Feb. 28, 2018). Accessed Mar. 28, 2022. http://www.aliannajmaren.com/2018/02/28/what-we-really-need-to-know-about-entropy/

Maren, Alianna J. 2018. “Wrapping Our Heads around Entropy.” Alianna J. Maren blogpost series. (Feb. 13, 2018). (Accessed Mar. 28, 2022.) http://www.aliannajmaren.com/2018/02/13/wrapping-our-heads-around-entropy/

Previous Related YouTubes

Maren, A.J. 2021. “Statistical Mechanics: Foundational to Artificial Intelligence.” Themesis YouTube Channel (Aug. 26, 2021). Accessed March 27, 2022. https://www.youtube.com/watch?v=y9ajEXdx54Q&t=2s

Good Vibes

If I want to connect in with the time of Hypatia (see below), thanks to the offerings of YouTube, I can listen to some recreations (in spirit if not exactitude) of music of that time: “Ancient Egyptian Music & 4K Egypt Landscapes | Arabian Music Playlist | The Pharaohs’ Golden Parade.”

Famous Salonnières, High Priestesses, and Professors!

Hypatia was a philosopher and astronomer, who taught in Alexandria (in Egypt) about 400 years after the birth of Christ.

She was, in her time, the world’s leading mathematician and astronomer, the only woman for whom such claim can be made. She was also a popular teacher and lecturer on philosophical topics of a less-specialist nature, attracting many loyal students and large audiences.
(Hypatia)

Hypatia was not so much a salonnière, as she was a professor of mathematics, astronomy, and philosophy, in what was the leading university of her day.

Figure 2. Robert Trewick Bone: *Hypatia Teaching at Alexandria*, watercolour and brown ink on paper; in the Yale Center for British Art, New Haven, Connecticut.
Yale Center for British Art, Paul Mellon Collection, B1975.4.1795

Hypatia was brutally murdered by a mob of Christian monks. One reason that they did this was because she was a leading teacher of Neoplatonism, which was a pagan notion, and thus contrary to the message espoused by the emerging Christian power structure.

Hypatia did research, taught, and held true to her beliefs in a time of great political upheaval and unrest. Her teachings polarized people, as some were attracted to the purity of the Neoplatonic school of thinking, whereas others allowed only for a more rigid, Christian-doctrine-based belief system. (Not so different from political parties today.) She paid for being true to her ideals with her life.

Zielinsky, Sarah. 2010. “Hypatia, Ancient Alexandria’s Great Female Scholar.” Smithsonian Magazine. (March 14, 2010.) (Accessed Mar. 27, 2022.) https://www.smithsonianmag.com/history/hypatia-ancient-alexandrias-great-female-scholar-10942888/