The science of thermodynamics deals with things like how to make a steam engine more efficient.
Surprisingly, the important concepts in thermodynamics are not only related to the “big ideas” in neural networks (including DL, or deep learning) and machine learning (ML), they are foundational.
This means, you can’t understand how an energy-based neural network works until you have at least a rudimentary grasp of thermodynamics. Similarly, you need basic thermodynamics concepts in order to understand all the different variational methods.
Our phrasebook approach will give you just enough thermodynamics understanding so that you can read one of the classic (or recent) important AI/DL/ML papers, and follow along.
You won’t necessarily have depth of understanding – at least on the first few reading passes. (That’s ok; very few of us do.)
However, you should be able to identify phrases that come from the world of thermodynamics, or its more granular expression as statistical mechanics. You’ll be able to recognize when the authors are transitioning from talking about statistical mechanics to Bayesian logic, and back again. (This kind of transitioning happens a LOT in these papers, so this is a useful skill.)
Thermodynamics – What It Is, How It Evolved
Thermodynamics first evolved as a science in the early 1800’s, just as various heat engines were being developed and used. An example of a heat engine is a steam locomotive or a turbine engine.
In an early steam locomotive, for example, burning coal was used to heat the steam boiler, creating steam. (Heat energy was used to vaporize water, in a phase transition that used a lot of heat.) The steam then propelled pistons, so the heat energy in the steam was transferred into kinetic energy (moving pistons). The moving pistons then powered the train’s movement.
While steam locomotives were invented in the very early 1800’s, it was not until about twenty years later that the emerging science of thermodynamics began to provide theory that could guide improvements in engine efficiency.
Some Basic Thermodynamics Concepts
Thermodynamics as a foundational science uses the notions of heat energy (transfer of thermal energy) and work energy (mechanical transfer of energy).
Thermodynamics also deals with the notions of enthalpy (the heat content of a system) and entropy (a measure of the distribution of the system’s components across all possible states).
Thermodynamics deals with the macroscopic properties of a system. In contrast, when we study statistical mechanics (or statistical thermodynamics – for our purposes, the same thing), we are looking at the microscopic properties – assessing the nature of individual elements in the system.
The beauty of statistical mechanics is that, when we apply statistical mechanics principles to a large (macroscopic) number of particles, we obtain the macroscopic thermodynamic properties!
A Simple Illustration
This pretty little holiday table decoration is actually a good example of a thermodynamic heat engine. It might be much smaller in scale – and not nearly as efficient – as a locomotive, a powerplant, or a car – but the same principles apply throughout.
Let’s keep in mind that the purpose of a thermodynamic heat engine is to do work. This doesn’t mean “work” in the sense of writing a report or washing dishes. In thermodynamics, the notion of “work” is very specific – it means doing something in the physical world, such as expanding a piston (due to heating up the gas inside the piston’s cylinder).
In short, the goal of a thermodynamic heat engine is to convert heat into work; to heat something up (typically, steam or a gas), and use that heated “something” to push against some physical thing (such as a piston or a turbine), and get the desired resultant kinetic motion.
In that sense, this little decoration is a fairly complex heat engine, in that several steps are involved.
- Burn the wax in the candle; this is a thermodynamic reaction that releases heat, and that heat is transferred to not only the surrounding air molecules, but also to the new molecules of carbon dioxide and water that are produced when we burn (break down) the more complex wax molecules.
- These newly-heated molecules move about more than their neighboring molecules. They have more kinetic energy.
- These faster-moving molecules move away from the candle flame. They also bump into their neighboring molecules, and transfer some of their heat energy to their neighbors. Ultimately, some of these faster-moving molecules travel up to the “turbin blades” from which the thin metal angels are suspended.
- The turbine blades are angled, so that the warmer, faster-moving air molecules hitting them transmit more kinetic energy to the undersides of the blades, and the blades spin. Transfer of kinetic energy – from air to the metal!
This is not a very efficient process. In fact, it is horribly inefficient.
Improving Thermodynamic Efficiency
If we were to design for thermodynamic efficiency, we would be concerned with two things:
- Increasing the total heat energy available – which is why we fuel turbine engines with coal instead of wood. Instead of using diesel fuel in a jet engine, we use high-octane jet fuel, which releases more heat when burning.
- Reducing the heat loss to anything OTHER than our desired results.
Reducing heat loss is important. This is why, when we want to heat a room, we get much more heat from a cast-iron wood-burning stove than from a fire in a fireplace.
In both systems, wood burns and heats the surrounding air molecules. In a fireplace, a lot of the hot air molecules are drawn up the chimney. In a wood-burning stove, however, the air molecules more often hit (and heat) the iron sides before they go up through the chimney.
Or – to understand the whole process – before the hot air molecules in a stove go up the chimney, they’ve usually transferred a lot of their kinetic energy to the iron stove’s sides. These iron sides then get hot. (Kinetic energy gets transformed into heat energy in the iron.)
Then, the iron walls heat the air molecules in the outside air around them. These heated air molecules travel around the room, making the room warm.
In short, in a fireplace, a lot of the heat energy produced is lost. This is very much related to the notion of entropy, which we’ll discuss in a future lesson.
The take-away from today’s discussion is that concepts that you may have already studied, such as entropy (e.g., Shannon entropy or marginal entropy; both concepts from information theory) have their roots in classic thermodynamics.
There is one more topic that is very important: the notion of free energy.
In the examples that we’ve used here, free energy is the energy that is available, after heat loss, to do the work – such as heat a room, move a piston, or spin little angels around in a decoration.
Where Thermodynamics Fits in with Neural Networks and Machine Learning
The free energy notion is fundamental in both energy-based neural networks and variational inference.
- Variational methods use the notion of free energy, which is a macroscopic thermodynamic principle.
- Energy-based neural networks (the Little-Hopfield neural network, the Boltzmann machine, and all forms of deep learning as well as GANs – generative adversarial networks) use statistical mechanics principles, and free energy is still important in statistical mechanics!
Thermodynamics – It’s More than Just Steam Engines
After its introduction as a means of understanding heat engines, thermodynamics continued to evolve as a science. It became a mainstay of chemistry, helping scientists to understand chemical reactions. This science helps us understand, for example, whether or not we need to supply heat for a chemical reaction to occur, or whether the chemical reaction produces heat. (However, all chemical reactions need a little “energetic boost” in order to get started. Otherwise, we would never live in a stable world – everything would be constantly reacting!)
For example, every time that we cook or bake something, we are using heat energy to cause a chemical reaction. When we sauté a mirepoix (mixture of carrots, onions, and celery), for example, and get a bit of caramelization, we’ve created a chemical reaction.
Other chemical reactions release heat. Burning coal or any substance that creates heat and warmth is done as a chemical reaction that breaks down the organic substance into simpler components, e.g. carbon dioxide and water. Heat is released as a by-product of this reaction.
In short, we now use thermodynamics to explain a lot of what happens in the world around us – it is one of the most pragmatic and useful sciences that we have!
Exercise: Observe and Reflect
As you go throughout your day, take note of when you do something that involves thermodynamics:
- Driving a car – combustion reactions transfer heat to mechanical motion,
- Cooking dinner – heat causes chemical reactions in your food,
- Heating your home in winter – if you use an oil furnace, then you are burning oil to create heat energy as well as the simple by-products of water and carbon dioxide (both vented to the atmosphere).
When you notice that a thermodynamic process is taking place, consider whether or not we have engineered that process for maximal efficiency. This means that scientists and engineers have used thermodynamics to improve efficiency – in a chemical process or mechanical device.
You can jump ahead to tomorrow’s content on Day 3: Free Energy.
You can also jump back to Day 1: Introduction.