Neural Networks: Perceptron, Backpropagation & RBF Networks Explained

✅ Unit 3: Perceptron, Backpropagation & Its Variants

⸻

🔹 A. Single Layer Perceptron Network

📌 1. Gradient Optimization Procedures

• Gradient descent is a fundamental technique for optimizing weight updates in neural networks.
• Goal: minimize the error/loss function.
• Update rule:

w_new = w_old − η ∂E/∂w

where:

• η: learning rate
• ∂E/∂w: gradient of the error

📌 2. Hebbian Learning Rule

• Based on the principle: “Neurons that fire together wire together.”
• Weight update rule:

ΔW_ij = η x_i y_j

• x_i: input
• y_j: output
• No error term involved (unsupervised)

📌 3. Perceptron Learning Algorithm

• Used in binary classification (outputs either 0 or 1).
• Algorithm:
  • Initialize weights randomly.
  • For each input:
    • Compute output: y = step(w · x + b)
    • Update weights if output ≠ desired output: w = w + η (d − y) x

📌 4. Convergence Theorem

• Statement: If training data is linearly separable, the perceptron algorithm will converge in a finite number of steps.
• Implication: Guarantees a solution for linearly separable data.

📌 5. Delta Rule

• Also known as the Widrow-Hoff rule (used in ADALINE).
• Uses actual output, not binary output.
• Applies gradient descent to minimize squared error:

Δw = η (d − y) x

📌 6. Limitations of Single Layer Perceptron

• Cannot solve problems like XOR.
• Can only learn linearly separable data.
• No hidden layers → no abstraction.

⸻

🔹 B. Multilayer Perceptron (MLP)

📌 1. Structure

• Has at least one hidden layer.
• Layers: input → hidden → output
• Neurons use nonlinear activation functions like sigmoid, ReLU, tanh.

📌 2. Backpropagation Algorithm

• Based on gradient descent and the chain rule.
• Steps:
  1. Forward pass – compute output.
  2. Backward pass – calculate error at each layer.
  3. Update weights using gradients.

📌 3. Applications

• Image and speech recognition.
• Natural language processing.
• Classification and regression tasks.

⸻

🔹 C. Learning Rate & Backpropagation Variants

📌 1. Learning Rate Effect

• Too high → weights jump too far → no convergence.
• Too low → slow learning.
• An ideal learning rate leads to fast and stable convergence.

📌 2. Variants of Backpropagation

• Momentum: adds a fraction of the previous weight change to the current update.

Δw = η δx + α ΔW_prev

• Adaptive learning rates: adjust the learning rate over time (e.g., AdaGrad, RMSProp, Adam).

⸻

✅ Unit 4: Radial Basis Function Neural Networks

⸻

🔹 A. Components & Structure

📌 1. Components

• Input layer
• Hidden layer: uses RBFs (e.g., Gaussian function).
• Output layer: linear combination of hidden outputs.

📌 2. Information Processing

• Each RBF neuron calculates the distance from the input to its center.
• Applies the radial function:

φ(x) = e^{−‖x − c‖² / (2σ²)}

📌 3. Analytical Considerations

• Select centers: random, k-means, or supervised methods.
• Set spread (σ): controls how “wide” each RBF is.

⸻

🔹 B. Training & Comparison

📌 1. Equation System

• Training is often divided into two steps:
  • Fixing RBF centers and widths.
  • Solving linear weights at the output layer.

📌 2. Gradient Strategies

• Adjust both output weights and possibly centers using gradient descent.

📌 3. Growing RBF Networks

• Starts with few neurons.
• Adds new ones if error > threshold.

📌 4. Comparison: RBF vs. MLP