Understanding Parameters And Notation In A Multi Layer Perceptron

Introduction

As we move deeper into neural networks, one concept that consistently creates confusion is the training process of a Multi-Layer Perceptron (MLP). Most learners eventually encounter the Backpropagation algorithm, and that is usually the point where things start to feel overwhelming.

Interestingly, the difficulty is often not caused by the mathematics itself. In many cases, the real problem begins much earlier. Neural networks contain a large number of weights, biases, activations, layers, and connections. If we do not develop a clear notation system for identifying these components, the equations used during training quickly become difficult to follow.

Before learning how backpropagation updates weights and biases, it is essential to understand how these parameters are organized and named inside a neural network. Once this foundation is clear, the training algorithms become significantly easier to understand.

In this tutorial, we will focus on three important topics:

1. Calculating the number of trainable parameters in a neural network.
2. Understanding how biases are represented.
3. Understanding how activations and weights are notated.

Although these concepts may appear simple at first glance, they form the foundation for everything that follows in neural network training.

A Sample Neural Network Architecture

Consider the following neural network structure:

• Input Layer: 4 neurons
• Hidden Layer 1: 3 neurons
• Hidden Layer 2: 2 neurons
• Output Layer: 1 neuron

We can label the layers as:

• Layer 0 → Input Layer
• Layer 1 → Hidden Layer 1
• Layer 2 → Hidden Layer 2
• Layer 3 → Output Layer

The input data contains four features. For example, a training record may look like this $x_1$, $x_2$, $x_3$, $x_4$. Whenever a training example enters the network, the values of these four features are supplied to the four neurons in the input layer.

Before discussing training, let us answer an important question.

What Are Trainable Parameters?

A trainable parameter is a value whose final value is learned during training. In a neural network, the trainable parameters are:

• Weights
• Biases

During training, the backpropagation algorithm continuously adjusts these values so that the network produces better predictions. Therefore, whenever someone asks:

How many trainable parameters does this neural network contain?

they are essentially asking:

How many weights and biases must be learned during training?

Understanding how to calculate this number is a fundamental skill.

Calculating the Number of Weights

Let us start by counting the weights. A weight exists between every pair of connected neurons belonging to adjacent layers.

Between Input Layer and Hidden Layer 1

The input layer contains 4 neurons. The first hidden layer contains 3 neurons. Every input neuron connects to every hidden neuron. Therefore:

$$4×3=12$$

weights exist between these two layers.

Between Hidden Layer 1 and Hidden Layer 2

The first hidden layer contains 3 neurons. The second hidden layer contains 2 neurons. Again, every neuron connects to every neuron in the next layer.

Therefore:

$$3×2=6$$

weights exist here.

Between Hidden Layer 2 and Output Layer

The second hidden layer contains 2 neurons. The output layer contains 1 neuron.

Therefore:

$$2×1=2$$

weights exist here.

Total Number of Weights

Adding all weights together:

$$12+6+2=20$$

Calculating the Number of Biases

Every neuron except input neurons typically has one bias.

Therefore:

• Hidden Layer 1: 3 neurons 3 biases
• Hidden Layer 2: 2 neurons 2 biases
• Output Layer: 1 neuron 1 bias

Total Number of Biases

$$3+2+1=6$$

Total Trainable Parameters

Now we combine weights and biases.

$$Total Parameters=Weights+Biases \\ 20+6=26$$

During training, backpropagation will determine the optimal values for all 26 parameters.

Bias Notation

Now let us understand how biases are represented mathematically.

A bias is generally denoted as:

$$b_{ij}$$

where:

• i represents the layer number
• j represents the neuron number inside that layer

This notation immediately tells us where the bias belongs.

Concept	Traditional Notation	Recommended Neural Network Notation	Meaning
Weight	w111	w11(1)	Connection from neuron 1 in the previous layer to neuron 1 in Layer 1
Weight	w223	w23(2)	Connection from neuron 2 in Layer 1 to neuron 3 in Layer 2
Bias	b11	b1(1)	Bias of neuron 1 in Layer 1
Bias	b22	b2(2)	Bias of neuron 2 in Layer 2
Activation (Output)	o11	o1(1)	Output of neuron 1 in Layer 1
Activation (Output)	o21	o1(2)	Output of neuron 1 in Layer 2

Activation (Output) Notation

Each neuron produces an output after applying its activation function.

These outputs are often represented using

$$o_{ij}$$

or sometimes

$$a_{ij}$$

​ depending on the textbook or framework.

where:

• i = layer number
• j = neuron number

Weight Notation

Weight notation is slightly more detailed because a weight connects two neurons.

A common notation is:

$$w^k{ij}$$

​

where:

• k = destination layer
• i = source neuron number
• j = destination neuron number

The notation identifies:

• Which layer the connection enters.
• Which neuron it originates from.
• Which neuron it connects to.