What Is An Activation Function and Why Do we Need It?

When people first learn about neural networks, they often focus on layers, neurons, weights, biases, and training algorithms. While all of these concepts are important, there is one component that quietly determines whether a neural network will be capable of learning complex patterns or remain nothing more than a glorified linear model. That component is the activation function.

At first glance, an activation function may appear to be just another mathematical formula placed inside a neuron. In fact, when beginners encounter functions such as Sigmoid, Tanh, or ReLU for the first time, they often wonder why these functions exist at all. If a neuron is already calculating a weighted sum of its inputs, why can't we simply use that value directly? Why introduce an additional mathematical function into the process?

The answer to these questions lies at the heart of neural networks. Without activation functions, modern neural networks would lose most of their power. Image recognition systems would struggle to identify objects, speech recognition systems would fail to capture complex patterns in audio, and large language models would not be capable of learning the sophisticated relationships that make them useful. In many ways, activation functions are what transform a collection of mathematical equations into a system capable of learning complex relationships from data.

To understand why activation functions are so important, we must first examine what happens when they are absent. Surprisingly, this journey begins with a neural network that appears perfectly reasonable but turns out to be far less powerful than it looks.

Building a Neural Network Without Activation Functions

Suppose we decide to build a neural network for a simple classification problem. The network contains multiple layers, several neurons in each layer, and dozens of trainable weights. At first glance, the architecture looks sophisticated. Compared to a single perceptron, it contains significantly more parameters and appears capable of learning far more complex patterns.

Consider a simple neuron that receives two inputs, $x_1$ and $x_2$. The neuron computes a weighted sum:

$$z=w_1​x_1​+w_2​x_2​+b$$

This weighted sum is then passed to the next layer. If every neuron in the network performs the same operation, we effectively create a multi-layer system where each layer repeatedly applies weighted sums to the outputs of previous layers.

From a structural perspective, this seems promising. After all, if one layer can learn something useful, then multiple layers should be able to learn something even more powerful. This assumption feels intuitive because we often associate depth with complexity. A deeper network appears more sophisticated than a shallow one, and it is natural to assume that adding layers automatically increases the model's ability to learn.

However, mathematics tells a very different story.

The Surprising Problem with Stacking Linear Layers

To understand the problem, let us simplify the network and focus on only two layers.

The first layer calculates:

$$z_1​=w_1​x+b_1​$$

The output of this layer becomes the input to the next layer:

$$z_2​=w_2​z_1​+b_2​$$

If we substitute the first equation into the second, we get:

$$z_2​=w_2​(w_1​x+b_1​)+b_2​$$

Expanding the expression gives:

$$z_2​=(w_2​w_1​)x+(w_2​b_1​+b_2​)$$

Notice something interesting. Even though we used two separate layers, the final equation is still a simple linear equation of the form:

$$z_2​=Wx+B$$

where $W$ and $B$ are simply new constants.

In other words, the two-layer network can be rewritten as a single-layer model. This is a surprising result because it means the second layer has not actually increased the mathematical complexity of the network. It may have introduced additional parameters, but the overall relationship between input and output remains linear. The same thing happens if we add three layers, five layers, or even fifty layers.

As long as every layer performs only linear transformations, the entire network collapses into a single linear transformation.

Why This Is a Serious Limitation

At this point, you might wonder why this is a problem. If the network can still make predictions, why should we care whether the relationship is linear?

The answer becomes clear when we revisit one of the biggest limitations of the perceptron. A perceptron can only create a linear decision boundary. This means it performs well when the classes can be separated by a straight line, but it struggles when the underlying pattern is non-linear.

Now imagine that we build a neural network containing ten layers but remove every activation function.

Although the network looks deeper and more sophisticated, it still behaves like a linear model. Consequently, it inherits the same limitation as a single perceptron. No matter how many layers we add, the network remains incapable of learning truly non-linear patterns.

This observation reveals a profound truth about neural networks: Depth alone does not create intelligence.

A network becomes powerful only when its layers can perform transformations that go beyond simple linear relationships. The challenge, therefore, is finding a way to introduce non-linearity into the network.

The Need for Non-Linearity

Real-world data rarely follows perfectly linear relationships. Consider some examples.

A person's salary does not increase linearly with experience throughout their career. A patient's risk of disease may increase dramatically after certain thresholds are crossed. The relationship between pixels in an image and the object represented by that image is enormously complex. Even language, which humans process effortlessly, contains intricate patterns that cannot be represented using simple linear equations.

Because real-world relationships are often non-linear, a useful machine learning model must be capable of learning non-linear functions. This is where activation functions enter the picture.

Instead of passing the weighted sum directly to the next layer, we first transform it using another mathematical function. This transformation changes the shape of the relationship between inputs and outputs, allowing the network to learn patterns that would otherwise be impossible.

In essence, activation functions break the linearity of the network and give it the flexibility required to model real-world data.

Introducing the Activation Function

An activation function is applied immediately after the weighted sum has been calculated.

The neuron first computes:

$$z=w_1x_1+w_2x_2+b$$

Then the activation function transforms that value:

$$a=f(z)$$

where $f$ represents the activation function.

The resulting value a is passed to the next layer.

This may seem like a small modification, but it completely changes the capabilities of the network. Once activation functions are introduced, multiple layers can no longer be collapsed into a single linear equation. Each layer now performs a non-linear transformation, allowing the network to gradually build increasingly complex representations of the data.

A First Look at the Sigmoid Function

One of the earliest and most famous activation functions is the Sigmoid function.

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

The sigmoid function converts any input into a value between 0 and 1.

Large positive inputs produce outputs close to 1, while large negative inputs produce outputs close to 0. Values near zero are mapped to outputs near 0.5.

Because of this behavior, sigmoid outputs can be interpreted as probabilities, making the function particularly useful for binary classification problems.

More importantly, the sigmoid introduces non-linearity. Once it is applied inside a network, the network gains the ability to learn relationships that are impossible for a purely linear model.

Why Activation Functions Make Neural Networks Powerful

The true power of neural networks does not come from their layers alone. It comes from the combination of layers and activation functions working together.

Each layer performs a transformation of the data. The activation function then introduces non-linearity before the transformed data is passed to the next layer. As information flows through the network, increasingly sophisticated patterns can be learned and represented.

This process allows neural networks to recognize faces, understand speech, translate languages, generate images, and perform countless other tasks that would be impossible for purely linear models.

In many ways, activation functions are the bridge between simple mathematical equations and intelligent behavior. Without them, neural networks would remain little more than stacked linear regressions. With them, they become capable of approximating remarkably complex functions and learning intricate patterns hidden within data.