Neural Networks: Core Concepts and Architectures

1. What is ANN? What is a Neuron?

Artificial Neural Networks (ANNs) are inspired by the biological neural networks of the human brain. They are a set of algorithms designed to recognize patterns. ANN consists of layers: input layer, hidden layers, and output layer. Each layer is made up of nodes called neurons.

A neuron in ANN is a mathematical function modeled after a biological neuron. It receives one or more inputs, applies a weight and bias to them, sums them up, and passes the result through an activation function to produce output.

ANNs are widely used in tasks like image recognition, natural language processing, and more. They learn from data by adjusting weights using optimization algorithms like gradient descent.

2. What is Optimization Technique (Gradient Descent)?

Optimization in machine learning is the process of minimizing or maximizing a function. In the case of neural networks, the goal is to minimize the loss (error) by adjusting weights.

Gradient Descent is the most common optimization algorithm. It works by calculating the gradient (partial derivatives) of the loss function with respect to weights. The weights are then updated in the opposite direction of the gradient to reduce the loss.

There are types of gradient descent:

• Batch Gradient Descent: Uses entire dataset.
• Stochastic Gradient Descent: Uses one sample at a time.
• Mini-Batch Gradient Descent: Uses small batch sizes.

3. Forward and Backward Propagation

Forward Propagation: In this step, the input passes through each layer of the network. The neuron processes inputs using weights, adds bias, and applies an activation function. The final result is the prediction or output.

Backward Propagation: After prediction, the error (loss) is calculated. This error is propagated back through the network. The weights are adjusted using gradient descent based on the contribution of each neuron to the error.

Both processes are repeated for several iterations (epochs) until the model learns well.

4. Supervised vs. Unsupervised Learning

Supervised learning is like teaching with examples and answers, while unsupervised learning finds hidden patterns without answers.

5. Explain Multilayer Perceptron

A Multilayer Perceptron (MLP) is a type of ANN that consists of three or more layers (input, hidden, and output). It is a fully connected network, meaning each neuron in one layer connects to all neurons in the next.

Each neuron processes input, applies a weight, adds a bias, and uses an activation function like ReLU or Sigmoid. MLP uses backpropagation and gradient descent for training.

MLPs are used in speech recognition, image classification, and many other applications.

6. What is CNN? Building Blocks of CNN?

Convolutional Neural Networks (CNNs) are a special type of neural network used mainly for image data. CNNs capture spatial and temporal patterns.

Building Blocks:

• Convolutional Layer: Applies filters to input to extract features.
• ReLU Activation: Adds non-linearity.
• Pooling Layer: Reduces spatial size (e.g., Max Pooling).
• Fully Connected Layer: Combines features to make final prediction.

CNNs are used in facial recognition, self-driving cars, and more.

7. Filters, Convolution, Pooling & Activation

• Filters/Kernels: Small matrices (e.g., 3x3) that slide over the input to detect features like edges, curves.
• Convolution: The process of applying the filter to the input by taking dot products.
• Pooling: Reduces the spatial dimensions. Max pooling takes the maximum value from a window.
• Activation Function: Adds non-linearity. Common ones are ReLU, Sigmoid, Tanh.

8. Overfitting and Underfitting in NN

• Overfitting: Model performs well on training data but poorly on test data. It memorizes instead of learning.
  • Solutions: Regularization, Dropout, Early stopping.
• Underfitting: Model performs poorly on both training and test data. It is too simple.
  • Solutions: Use a deeper model, train longer, remove constraints.

9. Explain LSTM & GRU

LSTM and GRU are advanced RNNs used to remember long-term dependencies.

• LSTM has gates: input, forget, and output. These gates control what to remember, forget, and output.
• GRU combines gates (reset and update), making it simpler than LSTM.

They are used in tasks like machine translation, music generation, and chatbot creation.

10. Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs) have two parts:

• Generator: Creates fake data.
• Discriminator: Identifies if the data is real or fake.

Both compete. The generator tries to fool the discriminator, and the discriminator tries to catch it. Over time, the generator creates realistic data. Used in image generation, deepfake, art, and more.

11. Normalization and Hyperparameter Tuning

• Normalization: Scaling input features to a common range. It helps in faster and better training.
  • Types: Min-Max Scaling, Z-score Standardization.
• Hyperparameter Tuning: Selecting the best parameters like learning rate, batch size, number of layers, etc.
  • Methods: Grid Search, Random Search, Bayesian Optimization.

Both are essential for improving model accuracy and training stability.

12. Transfer Learning in Vision

Transfer learning is a machine learning technique where a model trained on one task is repurposed as the foundation for a second task. This approach is beneficial when the second task is related to the first or when data for the second task is limited.

Using learned features from the initial task, the model can adapt more efficiently to the new task, accelerating learning and improving performance. Transfer learning also reduces the risk of overfitting, as the model already incorporates generalizable features useful for the second task.

Why Transfer Learning in Vision?

Training convolutional neural networks (CNNs) from scratch for image classification or detection often requires:

• Huge datasets (like ImageNet with over 14 million images),
• High computational resources (GPUs/TPUs), and
• A lot of time.

Transfer learning helps overcome these challenges by reusing pre-trained models, reducing the need for large datasets and computing resources.

Advantages of Transfer Learning

• Requires less training data.
• Faster training time.
• Often leads to better accuracy, especially with limited data.
• Leverages generalized feature extraction from large-scale models.

Common Pre-trained Models

These models are trained on large image datasets like ImageNet and are commonly used:

• VGGNet
• ResNet (Residual Network)
• InceptionNet
• EfficientNet
• MobileNet

13. Recurrent Neural Networks (RNN)

A Recurrent Neural Network (RNN) is a type of neural network used to process sequential data — like time series, text, or speech. Unlike traditional neural networks, RNNs have a memory of previous inputs, making them ideal for tasks where the order of data matters.

Why RNN?

In normal neural networks, each input is processed independently. In tasks like language translation, stock price prediction, or sentiment analysis, the current output depends on previous inputs. RNNs solve this by having loops in their structure to retain memory.

Working of RNN

1. Takes first input x₁ and processes it.
2. Stores the hidden state h₁ (memory).
3. Takes next input x₂, and uses h₁ to process it.
4. Repeats this for the whole sequence.

The formula: hₜ = tanh(W * xₜ + U * hₜ₋₁ + b)

Types of RNN

• Vanilla RNN: Processes sequences step-by-step; struggles with long-term memory.
• LSTM: Uses three gates to manage memory; good for long sequences.
• GRU: Simplified LSTM with two gates; faster performance.
• Bidirectional RNN: Reads input forward and backward.
• Deep RNN: Multiple layers for complex patterns.

14. Deep Dive into CNN Architecture

A Convolutional Neural Network (CNN) is a deep learning model mainly used for image classification, object detection, and computer vision tasks.

CNN Layers

• Input Layer: Receives the input image (e.g., 28x28x1).
• Convolutional Layer: Applies filters to extract features like edges.
• Pooling Layer: Reduces size (e.g., Max Pooling) to speed up computation.
• Flattening Layer: Converts 2D maps into a 1D vector.
• Fully Connected Layer: Performs classification using Softmax or Sigmoid.
• Output Layer: Gives final prediction probabilities.

15. What is an Autoencoder?

An autoencoder is a type of artificial neural network used to learn efficient data representations in an unsupervised manner. It is mainly used for dimensionality reduction, feature learning, and data reconstruction.

Structure of Autoencoder

1. Encoder: Compresses the input into a smaller representation.
2. Latent Space (Code): The compressed data.
3. Decoder: Reconstructs the original input from the code.