KL Divergence and Reverse KL Divergence

The simplest and most straightforward way to start learning the generative AI fundamentals is to understand the Kullback-Leibler divergence (KL divergence), and also the REVERSE KL divergence.

This YouTube will get you started:

The entry point to learning generative AI fundamentals is through the REVERSE Kullback-Leibler divergence.

This short (12-pg) Technical Note will walk you through the notation and the vocabulary for the both the (regular) KL divergence and the reverse KL divergence – and discuss how and when each is used.

(We will upload this Technical Note to arXiv shortly; for now, it is accessible here to get first rounds of feedback.)

Maren, Alianna J. 2024. “Minding Your P’s and Q’s: Notational Variations Expressing the Kullback-Leibler Divergence.” Themesis, Inc. Technical Note THM TN2024-001 (ajm). (PDF last accessed Feb. 02, 2024.)