DBN Classification: Deep Belief Network#

Building on RBM-based classification, we further construct a Deep Belief Network (DBN) that stacks multiple RBMs to achieve more powerful feature-learning capability.

Objectives#

I. RBM Principles Overview#

A Restricted Boltzmann Machine (RBM) is an energy-based probabilistic graphical model consisting of a Visible Layer and a Hidden Layer with no intra-layer connections and full inter-layer connections. Its core task is to learn the latent feature distribution of data via unsupervised learning.

1. Model Structure#

  • Visible layer (v): explicit representation of input data (e.g. pixel values).

  • Hidden layer (h): extracted latent features.

  • Weight matrix (W): weights connecting visible and hidden layers.

  • Biases: visible bias (b) and hidden bias (c).

2. Energy Function & Probability Distribution#

The RBM energy function is defined as:

E(v,h)=−vTWh−bTv−cTh E(\mathbf{v}, \mathbf{h}) = -\mathbf{v}^T W \mathbf{h} - \mathbf{b}^T \mathbf{v} - \mathbf{c}^T \mathbf{h}

The joint probability is given by the Boltzmann distribution:

P(v,h)=e−E(v,h)Z P(\mathbf{v}, \mathbf{h}) = \frac{ e^{-E(\mathbf{v}, \mathbf{h})} }{Z}

where ZZ is the partition function (normalization constant). The marginal distribution over the visible layer is:

P(v)=∑hP(v,h) P(\mathbf{v}) = \sum_{\mathbf{h}} P(\mathbf{v}, \mathbf{h})

3. Conditional Independence#

Because there are no within-layer connections, the hidden units are conditionally independent given the visible layer, and vice versa:

P(hj=1∣v)=σ(∑iWijvi+cj) P(h_j=1|\mathbf{v}) = \sigma\left(\sum_i W_{ij} v_i + c_j\right)
P(vi=1∣h)=σ(∑jWijhj+bi) P(v_i=1|\mathbf{h}) = \sigma\left(\sum_j W_{ij} h_j + b_i\right)

where σ(x)=11+e−x\sigma(x)=\frac{1}{1+e^{-x}} is the sigmoid activation function.

4. Training Objective#

Parameters (W, b, c) are learned by maximizing the likelihood. The objective is the negative log-likelihood:

L=−∑vlog⁡P(v) \mathcal{L} = -\sum_{\mathbf{v}} \log P(\mathbf{v})

Contrastive Divergence (CD) is used to approximate gradients. The update rule is:

ΔWij=ϵ(⟨vihj⟩data−⟨vihj⟩recon) \Delta W_{ij} = \epsilon \left(\langle v_i h_j \rangle_{\text{data}} - \langle v_i h_j \rangle_{\text{recon}}\right)

where ϵ\epsilon is the learning rate, and ⟨⋅⟩data\langle \cdot \rangle_{\text{data}} and ⟨⋅⟩recon\langle \cdot \rangle_{\text{recon}} denote expectations under the data and reconstruction distributions, respectively.

II. Overall Module Architecture & Training Modes#

The code implements a PyTorch-based Deep Belief Network with a layered architecture that supports the complete pipeline from unsupervised pre-training to supervised learning.

  • Module Architecture:

    • DBNPretrainer stacks multiple RBMs and performs layer-wise unsupervised pre-training, exposing a feature-extraction interface.

    • AbstractSupervisedDBN defines the generic interface for DBN supervised learning, supporting multi-mode training strategies including pre-training, fine-tuning, classifier training, and prediction.

    • AbstractSupervisedDBNClassifier, built on AbstractSupervisedDBN, provides PyTorch utility wrappers for feature extraction and classifier integration (logistic regression, SVM, random forest, etc.).

    • SupervisedDBNClassification implements the concrete classification task, fine-tuned network construction, and training.

  • Training Modes:

    1. Unsupervised mode: pre-training only, for feature extraction.

    2. Classifier mode: pre-training + downstream classifier training.

    3. Fine-tuning mode: pre-training + back-propagation fine-tuning of the entire network.

III. Core Classes & Interface Overview#

1. DBNPretrainer – Unsupervised DBN Training#

  • Key Parameters:

    • hidden_layers_structure: number of hidden units per layer (default two layers [100, 100])

    • learning_rate_rbm: RBM learning rate (default 0.1)

    • n_epochs_rbm: training epochs per RBM layer (default 10)

    • batch_size: mini-batch size (default 100)

    • verbose: print training logs (default True)

    • shuffle: shuffle data each epoch (default True)

    • drop_last: drop last incomplete batch (default False)

    • random_state: random seed

  • Device support: automatically chooses GPU (cuda) or CPU

  • Core Methods:

    • Creating an RBM layer (create_rbm_layer) initializes an RBM by using RestrictedBoltzmannMachine to set visible and hidden dimensions.

    • Single-batch training step (_train_batch)

      1. Positive phase: compute hidden activation probability P(h∣v)P(\mathbf{h}|\mathbf{v}).

      2. Negative phase: generate reconstructed samples via the SimulatedAnnealingOptimizer sampler.

      3. Objective: minimize energy plus weight decay (L2 regularization).

      4. Back-propagation: update weights and biases.

    • Single-layer RBM training (_train_rbm_layer)

      • Initialize the optimizer: use stochastic gradient descent (SGD) to optimize the parameters.

      • DataLoader handles mini-batch data.

    • Pre-train stacked RBMs (fit)

    • Feature transformation, layer-wise extraction (transform)

2. AbstractSupervisedDBN – Abstract Interface Definition#

  • Key Parameters:

    • fine_tuning: mode selection (default False)

    • learning_rate: fine-tuning learning rate (default 0.1)

    • n_iter_backprop: back-prop iterations (default 100)

    • l2_regularization: L2 regularization (default 1e-4)

    • activation_function: activation function (default 'sigmoid')

    • dropout_p: Dropout probability (default 0.0)

3. AbstractSupervisedDBNClassifier – Classifier-Mode Implementation & Fine-Tuning Utilities#

  • Key Parameters:

    • classifier_type: choice of classifier (default logistic regression)

    • clf_C: regularization strength (default 1.0)

    • clf_iter: iterations (default 100)

4. SupervisedDBNClassification – Concrete Classification Implementation#

  • Fine-tuned network construction: initialized with pre-trained weights (default two RBM layers, no Dropout)

    • Network: Input → [Linear + Activation + Dropout] × N → Output

    • Linear layers: initialized with corresponding RBM weights

    • Output layer: randomly initialized and learned during fine-tuning

  • Training Strategy:

    • Initialize with pre-trained weights

    • CrossEntropyLoss loss function

    • SGD optimizer + L2 regularization

    • Dropout supported for over-fitting prevention

5. Additional Topics#

  • Data loading (load_data)

    • Dataset: sklearn.datasets.load_digits (8×8 handwritten digit images).

    • Augmentation: shifts images up, down, left, right to enlarge the dataset.

  • Training visualization (_visualize_training_progress, set plot_img=True)

    • Weights & gradients: real-time monitoring of weight matrices and their gradients.

    • Generated samples: real-time display of the model’s ability to generate new samples.

    • Reconstructions: visual evolution of reconstruction error.

  • Result visualization (RBMVisualizer class)

    • Post-training RBM weight visualization

    • Classification results: confusion-matrix visualization

    • Reconstructions: encode-decode of test images after training

On this page