DBN Classification: Deep Belief Network#

Building on RBM-based classification, we further construct a Deep Belief Network (DBN) that stacks multiple RBMs to achieve more powerful feature-learning capability.

Objectives#

I. RBM Principles Overview#

A Restricted Boltzmann Machine (RBM) is an energy-based probabilistic graphical model consisting of a Visible Layer and a Hidden Layer with no intra-layer connections and full inter-layer connections. Its core task is to learn the latent feature distribution of data via unsupervised learning.

1. Model Structure#

Visible layer (``v``): explicit representation of input data (e.g. pixel values).
Hidden layer (``h``): extracted latent features.
Weight matrix (``W``): weights connecting visible and hidden layers.
Biases: visible bias (b) and hidden bias (c).

2. Energy Function & Probability Distribution#

The RBM energy function is defined as:

$E(\mathbf{v}, \mathbf{h}) = -\mathbf{v}^T W \mathbf{h} - \mathbf{b}^T \mathbf{v} - \mathbf{c}^T \mathbf{h}$

The joint probability is given by the Boltzmann distribution:

$P(\mathbf{v}, \mathbf{h}) = \frac{ e^{-E(\mathbf{v}, \mathbf{h})} }{Z}$

where $Z$ is the partition function (normalization constant). The marginal distribution over the visible layer is:

$P(\mathbf{v}) = \sum_{\mathbf{h}} P(\mathbf{v}, \mathbf{h})$

3. Conditional Independence#

Because there are no within-layer connections, the hidden units are conditionally independent given the visible layer, and vice versa:

$P(h_j=1|\mathbf{v}) = \sigma\left(\sum_i W_{ij} v_i + c_j\right)$

$P(v_i=1|\mathbf{h}) = \sigma\left(\sum_j W_{ij} h_j + b_i\right)$

where $\sigma(x)=\frac{1}{1+e^{-x}}$ is the sigmoid activation function.

4. Training Objective#

Parameters (W, b, c) are learned by maximizing the likelihood. The objective is the negative log-likelihood:

$\mathcal{L} = -\sum_{\mathbf{v}} \log P(\mathbf{v})$

Contrastive Divergence (CD) is used to approximate gradients. The update rule is:

$\Delta W_{ij} = \epsilon \left(\langle v_i h_j \rangle_{\text{data}} - \langle v_i h_j \rangle_{\text{recon}}\right)$

where $\epsilon$ is the learning rate, and $\langle \cdot \rangle_{\text{data}}$ and $\langle \cdot \rangle_{\text{recon}}$ denote expectations under the data and reconstruction distributions, respectively.

II. Overall Module Architecture & Training Modes#

The code implements a PyTorch-based Deep Belief Network with a layered architecture that supports the complete pipeline from unsupervised pre-training to supervised learning.

Module Architecture:
- ``DBNPretrainer`` stacks multiple RBMs and performs layer-wise unsupervised pre-training, exposing a feature-extraction interface.
- ``AbstractSupervisedDBN`` defines the generic interface for DBN supervised learning, supporting multi-mode training strategies including pre-training, fine-tuning, classifier training, and prediction.
- ``AbstractSupervisedDBNClassifier``, built on AbstractSupervisedDBN, provides PyTorch utility wrappers for feature extraction and classifier integration (logistic regression, SVM, random forest, etc.).
- ``SupervisedDBNClassification`` implements the concrete classification task, fine-tuned network construction, and training.
Training Modes:
1. Unsupervised mode: pre-training only, for feature extraction.
2. Classifier mode: pre-training + downstream classifier training.
3. Fine-tuning mode: pre-training + back-propagation fine-tuning of the entire network.

III. Core Classes & Interface Overview#

1. DBNPretrainer – Unsupervised DBN Training#

Key Parameters:
- hidden_layers_structure: number of hidden units per layer (default two layers [100, 100])
- learning_rate_rbm: RBM learning rate (default 0.1)
- n_epochs_rbm: training epochs per RBM layer (default 10)
- batch_size: mini-batch size (default 100)
- verbose: print training logs (default True)
- shuffle: shuffle data each epoch (default True)
- drop_last: drop last incomplete batch (default False)
- random_state: random seed
Device support: automatically chooses GPU (cuda) or CPU
Core Methods:
- Creating an RBM layer (``create_rbm_layer``) initializes an RBM by using RestrictedBoltzmannMachine to set visible and hidden dimensions.
- Single-batch training step (``_train_batch``)
  1. Positive phase: compute hidden activation probability $P(\mathbf{h}|\mathbf{v})$ .
  2. Negative phase: generate reconstructed samples via the SimulatedAnnealingOptimizer sampler.
  3. Objective: minimize energy plus weight decay (L2 regularization).
  4. Back-propagation: update weights and biases.
- Single-layer RBM training (``_train_rbm_layer``) – initializes an SGD optimizer and uses a DataLoader for mini-batches.
- Pre-train stacked RBMs (``fit``)
- Feature transformation, layer-wise extraction (``transform``)

2. AbstractSupervisedDBN – Abstract Interface Definition#

Key Parameters:
- fine_tuning: mode selection (default False)
- learning_rate: fine-tuning learning rate (default 0.1)
- n_iter_backprop: back-prop iterations (default 100)
- l2_regularization: L2 regularization (default 1e-4)
- activation_function: activation function (default 'sigmoid')
- dropout_p: Dropout probability (default 0.0)

3. AbstractSupervisedDBNClassifier – Classifier-Mode Implementation & Fine-Tuning Utilities#

Key Parameters:
- classifier_type: choice of classifier (default logistic regression)
- clf_C: regularization strength (default 1.0)
- clf_iter: iterations (default 100)

4. SupervisedDBNClassification – Concrete Classification Implementation#

Fine-tuned network construction: initialized with pre-trained weights (default two RBM layers, no Dropout)
- Network: Input → [Linear + Activation + Dropout] × N → Output
- Linear layers: initialized with corresponding RBM weights
- Output layer: randomly initialized and learned during fine-tuning
Training Strategy:
- Initialize with pre-trained weights
- CrossEntropyLoss loss function
- SGD optimizer + L2 regularization
- Dropout supported for over-fitting prevention

5. Additional Topics#

Data loading (``load_data``)
- Dataset: sklearn.datasets.load_digits (8×8 handwritten digit images).
- Augmentation: shifts images up, down, left, right to enlarge the dataset.
Training visualization (``_visualize_training_progress``, set ``plot_img=True``)
- Weights & gradients: real-time monitoring of weight matrices and their gradients.
- Generated samples: real-time display of the model’s ability to generate new samples.
- Reconstructions: visual evolution of reconstruction error.
Result visualization (``RBMVisualizer`` class)
- Post-training RBM weight visualization
- Classification results: confusion-matrix visualization
- Reconstructions: encode-decode of test images after training