4.3 Beyond Single Layers: Stacking for Deep Learning

The Restricted Boltzmann Machine, as presented in Section 4.2 Restricted Boltzmann Machine (RBM), is a powerful generative model with a single layer of latent variables. However, many real-world datasets, i.e. images, speech, natural language, exhibit hierarchical structure. Low-level features (edges, phonemes, word fragments) combine to form mid-level features (shapes, syllables, phrases), which in turn combine to form high-level concepts (objects, words, semantics). A single-layer latent variable model, no matter how many hidden units it possesses, may struggle to capture such multi-level abstractions efficiently. This limitation motivates the stacking of RBMs to form Deep Belief Networks (DBNs) and Deep Boltzmann Machines (DBMs). Stacking transforms a shallow generative model into a deep, hierarchical one, capable of learning progressively more abstract representations of the data. This section describes the greedy layer-wise training procedure that made deep generative models practical, and explains how stacked RBMs connect to the broader landscape of deep learning.

The Motivation: Hierarchical Feature Learning

Consider the task of modeling a collection of face images. A shallow RBM with a single hidden layer might learn localized Gabor-like edge detectors as its hidden features. These are useful, but they fail to capture the higher-order regularities of faces: the spatial arrangement of eyes, nose, and mouth, or the characteristic symmetries of facial structure.

A deep architecture, by contrast, can learn a hierarchy of representations:

• First hidden layer: Detects low-level features such as oriented edges and corners.

• Second hidden layer: Combines edges to form mid-level features like facial parts (eye, nose, mouth detectors).

• Third hidden layer: Combines facial parts to form high-level, abstract representations of whole faces.

This hierarchical composition mirrors the structure of the visual cortex and enables the model to represent complex data distributions with exponentially fewer parameters than a shallow model of comparable representational power.

Deep Belief Networks: Composition of RBMs

A Deep Belief Network (DBN) is a generative model composed of multiple layers of stochastic, latent variables. The top two layers form an undirected RBM, while the lower layers form a directed sigmoid belief network. Specifically, a DBN with $L$ hidden layers is defined as:

• The top two layers, $\mathbf{h}^{(L-1)}$ and $\mathbf{h}^{(L)}$, form an RBM.

• The remaining layers are connected via directed, top-down generative weights: $\mathbf{h}^{(l)}$ generates $\mathbf{h}^{(l-1)}$ for $l = L-1, \ldots, 1$, with $\mathbf{h}^{(0)} = \mathbf{v}$.

The joint distribution factorizes as:

$$P(\mathbf{v}, \mathbf{h}^{(1)}, \ldots, \mathbf{h}^{(L)}) = P(\mathbf{h}^{(L-1)}, \mathbf{h}^{(L)}) \prod_{l=1}^{L-2} P(\mathbf{h}^{(l)} \mid \mathbf{h}^{(l+1)})$$

where $P(\mathbf{h}^{(L-1)}, \mathbf{h}^{(L)})$ is the RBM distribution at the top, and the conditional distributions $P(\mathbf{h}^{(l)} \mid \mathbf{h}^{(l+1)})$ are those of a sigmoid belief network.

The directed connections enable efficient ancestral sampling: to generate a sample, one first samples from the top RBM to obtain $\mathbf{h}^{(L-1)}$ and $\mathbf{h}^{(L)}$, then propagates downward through the directed layers, sampling each layer in turn.

Greedy Layer-Wise Training

Training a deep generative model with many layers of latent variables is challenging due to the explaining-away effect and the intractability of exact inference. The breakthrough that made DBNs practical was greedy layer-wise pre-training, introduced by Hinton, Osindero, and Teh in 2006.

The core idea is elegantly simple: train one RBM at a time, and stack them.

Algorithm: Greedy Layer-Wise Training of a DBN

1. Train the first RBM:

• Treat the visible data $\mathbf{v}$ as the visible layer and introduce a hidden layer $\mathbf{h}^{(1)}$.

• Train an RBM on the data using contrastive divergence.

• After training, the RBM defines a generative model $P(\mathbf{v}, \mathbf{h}^{(1)})$ and an inference model $P(\mathbf{h}^{(1)} \mid \mathbf{v})$.

2. Generate training data for the next layer:

• Use the trained inference model to compute the posterior probabilities $P(\mathbf{h}^{(1)} \mid \mathbf{v})$ for each training example.

• These posterior probabilities (or samples from them) become the “data” for training the next RBM. They represent the data in the feature space learned by the first hidden layer.

3. Train the second RBM:

• Treat the vectors $\mathbf{h}^{(1)}$ as visible units and introduce a second hidden layer $\mathbf{h}^{(2)}$.

• Train an RBM on these feature vectors.

4. Repeat for as many layers as desired.

After this unsupervised pre-training, the stacked RBMs can be converted into a DBN by discarding the bottom-up recognition weights of all but the first layer, and using the top-down generative weights to define the directed model. Alternatively, weights can be used to initialize a deep neural network for discriminative fine-tuning using backpropagation.

Why Greedy Layer-Wise Training Works

The effectiveness of greedy layer-wise training can be understood from several perspectives:

1. Variational Lower Bound: Adding a new hidden layer and training it as an RBM on the inferred features of the previous layer can be shown to improve a variational lower bound on the log-likelihood of the data under the expanded model. Each additional layer tightens the bond.

2. Feature Abstraction: Each RBM learns to model the distribution of features produced by the layer below. Since these features are already somewhat abstract and decorrelated, the next RBM can focus on capturing higher-order dependencies among them.

3. Avoiding Local Minima: Deep supervised networks trained from random initializations often get stuck in poor local minima due to the complex, non-convex loss landscape. Pre-training initializes the weights in a region of parameter space that corresponds to a sensible generative model of the data, providing a far better starting point for subsequent discriminative fine-tuning.

Deep Boltzmann Machines (DBMs)

A closely related architecture is the Deep Boltzmann Machine (DBM), introduced by Salakhutdinov and Hinton in 2009. Unlike a DBN, where only the top two layers form an undirected RBM, a DBM is a fully undirected model with multiple hidden layers. All connections are symmetric, and there are no directed edges.

The energy function for a DBM with two hidden layers is:

$$E(\mathbf{v}, \mathbf{h}^{(1)}, \mathbf{h}^{(2)}) = -\mathbf{v}^\top \mathbf{W}^{(1)} \mathbf{h}^{(1)} - (\mathbf{h}^{(1)})^\top \mathbf{W}^{(2)} \mathbf{h}^{(2)} - \mathbf{b}^\top \mathbf{v} - (\mathbf{c}^{(1)})^\top \mathbf{h}^{(1)} - (\mathbf{c}^{(2)})^\top \mathbf{h}^{(2)}$$

DBMs retain the full undirected nature of the Boltzmann machine, which makes them more principled probabilistic models than DBNs. However, exact inference is even more challenging, and training typically requires a more sophisticated procedure, often involving mean-field approximations or persistent contrastive divergence with carefully designed initialization.

In practice, DBNs have been more widely adopted due to their simpler training procedure and direct compatibility with feedforward neural network initialization.

From Unsupervised Pre-Training to Discriminative Fine-Tuning

One of the most impactful applications of stacked RBMs was unsupervised pre-training for deep supervised learning. Before the widespread adoption of techniques like batch normalization, ReLU activations, and large labeled datasets (e.g., ImageNet), training deep neural networks with many hidden layers was notoriously difficult. Gradients would vanish or explode, and optimization would stall in poor local minima.

The RBM pre-training pipeline provided a solution:

1. Pre-train a stack of RBMs in an unsupervised manner on unlabeled data (which was often abundant).

2. Unroll the stacked RBMs to form a deep feedforward network with the same weights. This network effectively implements the bottom-up inference pathway.

3. Add a classification layer on top of the highest-level features.

4. Fine-tune the entire network using backpropagation on the labeled data.

This approach achieved state-of-the-art results on MNIST and, more impressively, enabled the first significant reduction in error rates on the TIMIT speech recognition benchmark. It demonstrated that deep architectures could be trained effectively if initialized properly, paving the way for the deep learning revolution.

The End of the Pre-Training Era and the Legacy of Stacked RBMs

By the early 2010s, advances in optimization (momentum, RMSprop, Adam), activation functions (ReLU and variants), regularization (dropout), and the availability of massive labeled datasets made unsupervised pre-training less necessary for many supervised tasks. End-to-end supervised training of deep networks from random initialization became the norm.

However, the conceptual legacy of stacked RBMs endures:

• They established the viability of deep architectures and the importance of hierarchical feature learning.

• They demonstrated the value of unsupervised representation learning as a precursor to supervised tasks.

• They introduced contrastive divergence and greedy layer-wise training, which remain valuable tools in the generative modeling toolkit.

• They inspired subsequent generative models, including Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), which also learn hierarchical latent representations, albeit with different training objectives.

Moreover, in domains where labeled data is scarce but unlabeled data is plentiful, unsupervised pre-training with RBMs or their modern descendants (e.g., deep belief networks, deep Boltzmann machines) remains a relevant and effective strategy.

Stacking Beyond RBMs: The General Principle

The principle of stacking simple modules to build deep, hierarchical models extends far beyond RBMs. Modern deep learning is built on this exact idea:

• Convolutional neural networks stack convolutional layers to learn increasingly complex visual features.

• Transformers stack self-attention layers to capture hierarchical linguistic structure.

• Diffusion models stack denoising steps to gradually refine generated samples.

The RBM stack was the first practical demonstration that greedy layer-wise construction could bootstrap the training of deep generative models. This insight remains a cornerstone of deep learning methodology.

Summary

• Stacking RBMs forms Deep Belief Networks (DBNs) and Deep Boltzmann Machines (DBMs), enabling hierarchical feature learning.

• Greedy layer-wise training trains one RBM at a time, using the inferred features of one layer as data for the next.

• DBNs combine an undirected top RBM with directed lower layers, while DBMs are fully undirected.

• Pre-trained stacks of RBMs provided effective weight initializations for deep supervised networks, fueling early deep learning successes.

• Although largely supplanted by end-to-end supervised training on large datasets, stacked RBMs remain historically significant and practically useful in label-scarce regimes.

• The concept of hierarchical representation learning pioneered by stacked RBMs is foundational to modern deep learning architectures.