To err is human. REALLY screwing things up … makes for interesting stories, and the basis for the first-ever Themesis “coffee reward” offer.

So here’s what’s at stake: there is a blooper, and it involves notation (more on that shortly), and we (that is, Themesis, Inc.) will send an 8-oz package of Kona coffee to each of the FIRST THREE people who identify the bloop (and God, I hope there’s only that one), and send the bloop-identification to us at themesisinc1 (at) gmail (dot) com.

To further incentivize you – as you drink your first cup of morning joe this weekend, and think to yourself – “Damn! This could be 100% Kona coffee” – here’s the link to Menehune Coffee Company, on the Big Island, https://www.menehunecoffee.com/

You can even go to Menehune’s “Shop” page, and decide which 8-oz bag you’d prefer – dark, medium-dark, or medium, whole or ground, and there’s even a decaf option. (What’s the point of that, though?)

Menehune Coffee Co. is where I’ve been getting my coffee for the last six years, and what you’d get (if you win this blooper-finding contest) is what they use to make their own in-house coffee. Good stuff!

So, to reiterate: the FIRST THREE people who email us at the above Themesis email address, and correctly identify the bloop, will each get an 8-oz bag (of their choice) that we’ll ask the dear folks at Menehune Coffee Co. to ship to you. And with your permission, we’ll publish your names and professional affiliations in my next blog post.

This contest is open ONLY until I get my “Big Mama” computer back. “Mama” is currently in my car, wrapped safely in multiple layers of bubble wrap, and the car is loaded (I most sincerely hope) onto a barge that is even now riding the Pacific Ocean waves. “Mama” (and the car) should arrive in port today. She should be available for pickup this coming week. We will see. (It’s been a long wait … the biggest part of the wait was booking a slot on the Honolulu-to-Kauai leg of the ocean journey.)

Now that I’ve whetted your interest (and really, you already planned to spend this weekend reading a physics paper, didn’t you?), here are the details.

The Paper (Where the Big Bloop Is Located)

About six years ago – even before my move to the Big Island – I was immersing myself in variational Bayes, and particularly in “active inference,” a notion espoused by Karl Friston. (References will be at the end of this post.)

Variational Bayes is at the forefront of machine learning / AI.

My personal belief is that active inference, based on varaitional Bayes, is going to be the front-runner method for planning and inference. My gut tells me that this will SIGNIFICANTLY come out ahead of reinforcement learning, which is currently getting all the attention.

That said, active inference (espoused by Friston and colleagues) is not that easy to understand.

It took a long time – and multiple VERY LONG stints of just “holding the equation” (a sort of Zen-ish thing applied to equations) in order to get the essence.

What REALLY helped, though, was that I wrote a tutorial paper for myself.

Here’s the link: https://arxiv.org/abs/1906.08804

The key thing for the blooper: It is notational – and actually is not in the equation, it is in my interpretation of the notation for a specific equation.

The Backstory

The thing that bites us in the soft-and-tenders is NEVER the “big concept.” It’s always some delicate, nuanced little detail – and is MOST OFTEN in the notation. (I’ve been thrown sideways by notation-things more times than I care to remember.)

In this case, the whole reason for writing the “self-tutorial” paper was that there were three papers involved.

I was trying to teach myself active inference, a Friston notion. Active inference is built on variational Bayes.

I really wanted to stay consistent with Friston’s notation, as I planned to refer to him again and again – so the papers that I read to teach myself variational Bayes were those that Friston cited – works by Matthew Beal (his doctoral dissertation) and a very solid paper by Blei et al. (2016; and ALL REFERENCES ARE GIVEN AT THE END).

The problem was – the Beal and the Blei papers used contrari-wise notation. It was as though one of them was using “y = f(x)” and the other was using “x = f(y).” The function, f, was the same in both cases – but the notation was reversed. (It was actually a bit more complex than that.)

The only recourse was to build my own paper, in which I translated the notation from one to the other. This is why I called this “Variational Bayes” paper of mine a “Rosetta stone” – it translated notation betwen Friston, Beal, and Blei et al. into a common reference frame.

This was a truly mind-warping experience – and there were times that I was so wrapped up in the abstract equation-space that I couldn’t speak a coherent sentence in English.

At last, though, it was done. All 62 pages of it. And I had it reviewed – TWICE – by someone who could check that I was interpreting Friston correctly. (That was the real challenge, involving several rewrites.)

That reviewer – excellent though he was, and careful – and painstaking – ALSO missed my “big bloop.”

In fact, I didn’t find the bloop until a few months ago, when I was building my next paper. (Notice that I’m saying “building” – it’s more a construction project than a writing project.) During this – and yes again, dealing with notation – I went back to my own sources. And discovered the bloop.

Well, there was simply no time to deal. DEALING with this bloop would mean going back to the original document, studying it carefully, and cross-checking (once again) with all my source materials.

There were intervening tasks. The (first-ever) corporate tax reports were due, and then there was the big inter-island move, and on top of it, piles and PILES of grading.

Well, now the tax reports have been turned in. The move is accomplished, more-or-less. And everybody’s everything has been graded, and final grades for Spring Quarter have been uploaded to the great registrar-in-the-sky, and sanity prevails once again.

When “Big Mama” (my computer) arrives, I can do ALL SORTS of work – because that’s where all the main programs are installed. LaTeX, with its many “include” files. Python, with my research programs. Camtasia, which I use to format the YouTube vids.

UNTIL SUCH TIME … I can at least write this blog. And invite you to take this week to go on a “blooper-hunt” – that’s about all the time that you’ll have, since I need to fix that blooper pretty fast.

Have fun, darlings!

Alianna J. Maren, Ph.D.

Founder and Chief Scientist, Themesis, Inc.

References

Beal, M.J. 2003. Variational Algorithms for Approximate Bayesian Inference. PhD thesis, University College London, 2003. PDF: http://www.cse.buffalo.edu/faculty/mbeal/papers/beal03.pdf

Blei, D.M., A. Kucukelbir, and J. D. McAuliffe, 2016. “Variational inference: A review for statisticians.” arXiv:1601.00670v4 [stat:CO], 2 Nov 2016.

Friston, K. 2010. “The Free-Energy Principle: A Unified Brain Theory?,” Nat. Rev. Neurosci. 11:127138. doi:10.1038/nrn2787.

Friston, K. 2013. “Life as we know it,” Journal of The Royal Society Interface 10 (86).

Friston, K., M. Levin, B. Sengupta, and G. Pezzulo. 2015. “Knowing Ones Place: A Free-Energy Approach to Pattern Regulation,” J. R. Soc. Interface 12: 20141383. doi:10.1098/rsif.2014.1383; (Accessed 17 June 2002; http://dx.doi.org/10.1098/rsif.2014.1383

Friston, K., M. Levin, B. Sengupta, and G. Pezzulo. 2015. “Knowing Ones Place: A Free-Energy Approach to Pattern Regulation,” J. R. Soc. Interface 12: 20141383. doi:10.1098/rsif.2014.1383; (Accessed 17 June 2002; http://dx.doi.org/10.1098/rsif.2014.1383

Maren, A.J. 2019. “Derivation of the Variational Bayes Equations.” arXiv:1906.08804v4 [cs.NE] 30 Jul 2019. (Accessed on June 17, 2022: arXiv:1906.08804v4 [cs.NE].)

Related YouTubes

This is not, alas, my first public blooper.

In fact, my history is replete with bloops – sometimes very embarrassing!

So, sometimes we just need to capitalize on our weaknesses as well as our strengths.

Here’s a nicely-curated example of some previous bloops!

Maren, A.J. 2020. “Math Bloopers! Dr. AJ’s First Candid Blooper Reveal.” Alianna J. Maren YouTube Channel (5 Nov. 2020). (Accessed 17 June 2022; https://www.youtube.com/watch?v=6rm-QER3zQE&list=PLUf2R_am1DRJrbfmOcdqPv392EMckOyMU&index=1&t=268s )

Related Blogposts (Especially on Friston!)

(Presented in suggested reading order.)

Maren, Alianna J. 2017. “How to Read Karl Friston (In the Original Greek).” Blogpost, aliannajmaren.com (July 27, 2017). (Accessed 17 June 2022: https://www.aliannajmaren.com/2017/07/27/how-to-read-karl-friston-in-the-original-greek/ )

Maren, Alianna J. 2019. “Interpreting Karl Friston: Round Deux.” Blogpost, aliannajmaren.com (July 31, 2019). (Accessed 17 June 2022: (Accessed 17 June 2022; https://www.aliannajmaren.com/2019/07/31/interpreting-karl-friston-round-deux/ )

Maren, Alianna J. 2018. “Wrapping Our Heads around Entropy” Blogpost, aliannajmaren.com (February 13, 2018). (Accessed 17 June 2022: (Accessed 17 June 2022; https://www.aliannajmaren.com/2018/02/13/wrapping-our-heads-around-entropy/)

Maren, Alianna J. 2018. “What We Really Need to Know about Entropy” Blogpost, aliannajmaren.com (February 28, 2018). (Accessed 17 June 2022; https://www.aliannajmaren.com/2019/07/31/interpreting-karl-friston-round-deux/ )

Recommended Summer Read

You read, above, that I was tracking active inference, and that I believe that it will become the fore-runner method, topping out reinforcement learning.

Active inference is something that Karl Friston has evolved, over the past more-than-decade. His works are notoriously difficult to read.

Recently, Noor Sajid was the lead author of “Active Inference: Demystified and Compared” (arXiv, 2022).

I like this paper for several reasons:

• Honest comparative weighting: If you’ve wanted to do a contrast-and-compare between reinforcement learning and active inference, this is the best go-to resource. While three of the authors are advocates of active inference, one (Philip Ball) is an advocate of reinforcement learning, so there is substantial discussion of both methods.
• Substantial review paper: At 69 pages, this is not a trivial read – and yet, it flows easily. There are solid twelve pages of references at the end; this is indeed a review paper, and it cites excellent and useful works.
• Easiest and most accessible introduction to active inference: Sajid has focused on making as clear an exposition as possible. For that reason, I’ve recommended this paper to several students and colleagues, and will settle down and make this my first-priority summer read.

Here’s an extract that summarizes why I think that active inference will be so useful:

… in active inference an agent’s interaction with the environment is determined by action sequences that minimize expected free energy (and not the expected value of a reward signal).

Sajid et al., 2020, p. 5

Free energy minimization is a universal principle. This is why I’ve got such a strong gut-feeling that an approach based on free energy minimization is likely to pan out well over the future.

Reference & Abstract

Sajid, N., Philip J. Ball, Thomas Parr, and Karl J. Friston. 2020. “Active Inference: Demystified and Compared.” arXiv:1909.10863v3 [cs.AI] 30 Oct 2020. (Accessed 17 June 2022; https://arxiv.org/abs/1909.10863 )

Abstract:

Active inference is a first principle account of how autonomous agents operate in dynamic, non-stationary environments. This problem is also considered in reinforcement learning, but limited work exists on comparing the two approaches on the same discrete-state environments. In this paper, we provide: 1) an accessible overview of the discretestate formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in reinforcement learning; 2) an explicit discrete-state comparison between active inference and reinforcement learning on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of reinforcement learning. We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration — and account for uncertainty about their environment — in a Bayes–optimal fashion. Furthermore, we show that the reliance on an explicit reward signal in reinforcement learning is removed in active inference, where reward can simply be treated as another observation we have a preference over; even in the total absence of rewards, agent behaviors are learned through preference learning. We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based reinforcement learning agents; by placing zero prior preferences over rewards and by learning the prior preferences
over the observations corresponding to reward. We conclude by noting that this formalism can be applied to more complex settings; e.g., robotic arm movement, Atari games, etc., if appropriate generative models can be formulated. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation, and demonstrate these behaviors in a OpenAI gym environment, alongside reinforcement learning agents.