Understanding the Sigmoid Function

Among all the activation functions used in Machine Learning and Neural Networks, the Sigmoid function is probably the most famous. Even people who are new to Deep Learning often encounter the sigmoid function very early in their learning journey because it appears in Logistic Regression, Perceptrons, Binary Classification problems, and the history of neural networks itself.

At first glance, the sigmoid function may look like just another mathematical formula. However, its popularity comes from a very useful property: it takes any real number as input and converts it into a value between 0 and 1. Because of this behavior, the output can be interpreted as a probability, making the sigmoid function particularly useful for binary classification tasks.

While modern neural networks often prefer activation functions such as ReLU and GELU in hidden layers, the sigmoid function remains an important concept because it helps us understand how activation functions work and why they were introduced in the first place.

In this tutorial, we will explore the intuition behind the sigmoid function, understand its mathematical behavior, learn why it became so popular, and discuss both its strengths and limitations.

The Problem Before Sigmoid

To appreciate the sigmoid function, we must first understand the problem it was designed to solve.

Consider a neuron inside a neural network. After receiving its inputs, the neuron calculates a weighted sum:

$$z=w_1​x_1​+w_2​x_2​+b$$

The value of z can be almost anything.

For example:

Input	Output (z)
Example 1	-15
Example 2	2.3
Example 3	78
Example 4	-120

The challenge is that these values are difficult to interpret directly.

Suppose we are building a model to predict whether a student will get placed during campus recruitment. If the neuron produces an output of 78, what does that mean? Does it indicate a high probability of placement? A medium probability? A low probability? The raw output itself does not provide an intuitive answer. Ideally, we would like the model to produce values that are easier to interpret. A number such as 0.95 immediately suggests a 95% probability, while 0.10 suggests a 10% probability.

The sigmoid function was designed to perform exactly this transformation.

What Is the Sigmoid Function?

The sigmoid function is a mathematical function that converts any real number into a value between 0 and 1.

Its formula is:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Although the equation may look intimidating at first, the important thing to remember is not the formula itself but its behavior.

No matter what value of z you provide:

• The output will never be less than 0.
• The output will never be greater than 1.
• Extremely large positive values move toward 1.
• Extremely large negative values move toward 0.

This behavior makes the sigmoid function an excellent candidate for probability estimation. For example:

z	Sigmoid Output
-10	0.000045
-2	0.119
0	0.5
2	0.881
10	0.99995

Why Is It Called a Sigmoid Function?

The term sigmoid comes from the shape of its graph. When plotted on a graph, the curve resembles an elongated "S". Unlike a straight line, which continues infinitely in both directions, the sigmoid curve gradually flattens near its upper and lower limits. This flattening behavior is known as saturation.

As the input becomes extremely positive, the output gets closer and closer to 1.As the input becomes extremely negative, the output gets closer and closer to 0. However, it never actually reaches either value.

This smooth transition is one of the reasons the sigmoid function became popular in early neural networks.

Understanding Sigmoid Through a Placement Prediction Example

Suppose we are building a model that predicts whether a student will get placed. After processing the student's CGPA and IQ score, the neuron computes:

$$z=4.5$$

This value is then passed through the sigmoid function:

$$σ(4.5)$$

The result is approximately:

$$0.989$$

This means the model predicts a 98.9% probability of placement.

Now consider another student:

$$z=−4.5$$

Applying the sigmoid function gives:

$$0.011$$

This corresponds to approximately a 1.1% probability of placement. Notice what the sigmoid function has done. It has transformed arbitrary numerical values into probabilities that humans can easily interpret. This ability made sigmoid extremely useful in classification problems.

Why Sigmoid Became Popular in Neural Networks

The sigmoid function gained popularity for several reasons.

1. It introduces non-linearity into the network. Without activation functions, neural networks would simply behave like linear models regardless of how many layers they contained.
2. Its output range between 0 and 1 makes it easy to interpret as a probability.
3. The sigmoid function is smooth and differentiable. Since neural networks are trained using Gradient Descent and Backpropagation, differentiability is an important requirement.

Because of these properties, sigmoid became one of the most widely used activation functions during the early years of neural network research. For many years, it was considered the default activation function for hidden layers.

The Hidden Problem with Sigmoid

Although sigmoid has several advantages, researchers eventually discovered a significant limitation. To understand the issue, look carefully at the shape of the sigmoid curve.

Near the center of the curve, small changes in the input produce noticeable changes in the output.However, near the extremes, the curve becomes almost flat. When the curve becomes flat, its slope approaches zero. This creates a problem during training because Gradient Descent relies on gradients to update weights. If the gradients become extremely small, weight updates also become extremely small. As a result, learning slows down dramatically.

This phenomenon is known as the Vanishing Gradient Problem.

The vanishing gradient problem became one of the major reasons researchers began searching for alternative activation functions.

Why Sigmoid Is Rarely Used in Hidden Layers Today

Modern neural networks often contain dozens or even hundreds of layers. When sigmoid activations are used repeatedly across many layers, the vanishing gradient problem becomes more severe.

Training deep networks becomes difficult because gradients shrink as they move backward through the network. For this reason, activation functions such as ReLU, Leaky ReLU, GELU, and Swish have largely replaced sigmoid in hidden layers.

These newer activation functions allow gradients to flow more effectively, making deep networks easier to train. However, this does not mean sigmoid has disappeared entirely.

Where Sigmoid Is Still Used Today

Even though sigmoid is rarely used in hidden layers, it remains extremely important in binary classification tasks. Whenever a model needs to answer a question with two possible outcomes, sigmoid is often used in the output layer.

Examples include:

• Spam vs Not Spam
• Fraud vs Not Fraud
• Disease vs No Disease
• Pass vs Fail
• Placement vs No Placement

In these scenarios, the sigmoid output naturally represents the probability of belonging to the positive class. Because of this property, sigmoid remains a fundamental component of Logistic Regression and many neural network architectures.

Summary

The sigmoid function is one of the most important activation functions in the history of Machine Learning and Neural Networks. Its primary purpose is to transform arbitrary numerical values into outputs between 0 and 1, making those outputs easy to interpret as probabilities.

Its smooth S-shaped curve introduces non-linearity into neural networks and enables models to learn relationships that would be impossible for purely linear systems. These properties made sigmoid the dominant activation function in early neural network research.

However, the sigmoid function also suffers from the vanishing gradient problem, which makes training deep neural networks difficult. As a result, modern architectures typically prefer activation functions such as ReLU and GELU in hidden layers.

Despite these limitations, sigmoid remains the standard choice for many binary classification problems and continues to play an important role in Machine Learning.