Welcome back!
our journey of understanding backpropagation in detail.
Let’s briefly recall what we covered in Part 1 so far.
Contents
A Quick Recap
We first started trying to understand neural networks using a simple dataset.
In that process, we first built a small neural network and understood how it makes predictions through the process of forward propagation.
We then observed that the predicted values were nowhere close to the actual values, resulting in a large error.
Now, we wanted to train this neural network to perform better.
By comparing it with simple linear regression, we observed that, in our neural network, the loss depends on seven parameters.
[w_1,w_2,w_3,w_4,b_1,b_2,b_3]
Our complete loss function looked like this:
[
L(w_1,w_2,w_3,w_4,b_1,b_2,b_3)=frac{1}{n}sum_{i=1}^{n}left(y_i-left(w_3mathrm{ReLU}(w_1x_i+b_1)+w_4mathrm{ReLU}(w_2x_i+b_2)+b_3right)right)^2
]
Next, just as we did in simple linear regression, we wanted to differentiate the loss with respect to each parameter.
We started with and used the classical differentiation method. Eventually, we arrived at the following result:
[
frac{partial L}{partial w_1}
=
-frac{2}{n}
sum_{i=1}^{n}
(y_i-hat{y}_i)
w_3
mathrm{ReLU}'(w_1x_i+b_1)
x_i
]
Do We Really Have to Repeat This?
Now it’s time to think.
We differentiated the loss with respect to , which means we were trying to find how the loss changes as changes.
We used the classical differentiation method here.
We now have six more parameters to differentiate the loss with respect to.
What do you think?
Do we need to proceed with the same method?
No.
The Chain Rule to the Rescue
If you remember, we discussed the chain rule in Part 1.
We used a simple example.
We considered
[
y=x^2
]
and
[
z=y^3
]
We noticed that (z) does not depend directly on (x).
Instead, the relationship looks like
[
x rightarrow y rightarrow z.
]
This means that when (x) changes, it first changes (y), and the change in (y) then affects (z).
Since the effect of (x) reaches (z) through an intermediate variable, we cannot differentiate (z) with respect to (x) directly. Instead, we used the chain rule.
[
frac{dz}{dx}
=
frac{dz}{dy}
cdot
frac{dy}{dx}
]
The chain rule tells us that instead of taking one large step from (x) to (z), we can break the problem into smaller steps by following the dependency path.
Following the Dependency Path
Now the question is can we apply this same idea to our neural network.
The answer is yes.
To do that, we first need to understand how a change in , travels through the neural network before it finally affects the loss.
Let’s write down the equations of our neural network.
[
z_1=w_1x+b_1
]
(z_1) depends on the weight (w_1), the input (x), and the bias (b_1).
Next, we applied the ReLU activation function.
[
a_1=mathrm{ReLU}(z_1)
]
Here, the activation (a_1) depends entirely on (z_1).
The prediction of our network is
[
hat{y}=w_3a_1+w_4a_2+b_3.
]
we can see that, the prediction depends on the activation (a_1).
Finally, we computed the loss.
[
L=(y-hat{y})^2.
]
The loss depends on the prediction.
If we connect all these relationships together, we get
[
w_1
rightarrow
z_1
rightarrow
a_1
rightarrow
hat{y}
rightarrow
L.
]
Comparing this with the simple example we considered.
Simple example:
[
x
rightarrow
y
rightarrow
z.
]
Neural network:
[
w_1
rightarrow
z_1
rightarrow
a_1
rightarrow
hat{y}
rightarrow
L
]
As we already said in part 1, the only difference is that the neural network has a longer dependency chain, but the underlying idea remains the same.
Partial Derivatives vs. the Chain Rule
At this point, we may get confused between partial derivatives and the chain rule.
Let’s clarify both ideas before moving forward.
In the simple example,
[
y=x^2
]
(y) is only depended on input variable, (x). So, to find change in (y) w.r.t (x), ordinary derivatives are sufficient, and we write
[
frac{dy}{dx}
]
But, consider the equation
[
z_1=w_1x+b_1
]
Here (z_1) depends on three variables: (w_1), (x), and (b_1).
Now, we want to know how (z_1) changes when only (w_1) changes.
For this, we temporarily keep (x) and (b_1) fixed.
Therefore, we write
[
frac{partial z_1}{partial w_1}
]
The symbol (partial) tells us that we are changing only one variable while keeping the remaining variables as constants during that particular calculation.
This is what we know about partial derivatives.
Now, coming to the chain rule, it is a different concept.
Let’s be careful not to confuse these two different concepts.
A partial derivative answers the question:
Which variable are we changing?
For example,
[
frac{partial z_1}{partial w_1}
]
which means only (w_1) is allowed to change.
The chain rule answers a different question:
How does the effect of one variable travel through several intermediate computations?
In our neural network,
[
w_1
rightarrow
z_1
rightarrow
a_1
rightarrow
hat{y}
rightarrow
L
]
As the effect of changing , reaches the loss through every intermediate parameter, we multiply the derivatives along this path.
Therefore,
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial w_1}
]
We can observe that nothing new has happened.
We are still applying the same chain rule that we learned in Part 1.
The only difference is that the chain is now longer and the functions involve multiple variables, which is why we use partial derivatives instead of ordinary derivatives.
Applying the Chain Rule to Our Neural Network
Step 1:
We now know that
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial w_1}
]
To compute this gradient, we first need to calculate each partial derivative one by one.
To keep the chain rule easy to follow, we’ll derive the gradient for one data point.
In other words, for this derivation we consider the case where
[
n=1.
]
Therefore, the Mean Squared Error (MSE) loss
[
L=frac{1}{n}sum_{i=1}^{n}(y_i-hat{y}_i)^2
]
simplifies to
[
L=(y-hat{y})^2.
]
Once we get an idea of how to compute the gradient for a single data point, extending it to the entire dataset is straightforward.
We simply average the gradients over all data points, just as we did in Part 1.
The first derivative we need to compute is
[
frac{partial L}{partial hat{y}}.
]
But before differentiating, let’s first understand what this derivative tells us.
The actual value
[
y
]
comes from our dataset, so during differentiation it is considered as a constant.
The only quantity that can change here is the prediction
[
hat{y}
]
Therefore,
[
frac{partial L}{partial hat{y}}
]
tells us:
If the prediction changes by a small amount, how much does the loss change?
Now that we know what this derivative represents, let’s compute it.
Since
[
L=(y-hat{y})^2
]
we first apply the power rule.
Differentiate the outer square.
[
frac{partial L}{partial hat{y}}
=
2(y-hat{y})
cdot
frac{partial (y-hat{y})}{partial hat{y}}
]
Now differentiate the expression inside.
The derivative of
[
y
]
with respect to
[
hat{y}
]
is
[
0
]
because the actual value does not change.
The derivative of
[
-hat{y}
]
with respect to
[
hat{y}
]
is
[
-1
]
Therefore,
[
frac{partial (y-hat{y})}{partial hat{y}}
=
-1
]
Substituting this back into the previous equation gives
[
frac{partial L}{partial hat{y}}
=
2(y-hat{y})
(-1)
]
Finally,
[
frac{partial L}{partial hat{y}}
=
-2(y-hat{y})
]
This tells us how the loss changes whenever the prediction changes.
Step 2:
Now the next derivative we need to compute is
[
frac{partial hat{y}}{partial a_1}
]
Let’s first see what this derivative tells us.
Recall that the output neuron computes the prediction using the equation
[
hat{y}=w_3a_1+w_4a_2+b_3
]
As we are differentiating with respect to
[
a_1
]
all the other quantities
[
w_3,;w_4,;a_2,;text{and};b_3
]
are treated as constants.
Therefore,
[
frac{partial hat{y}}{partial a_1}
]
tells us
If the activation [a_1] changes by a small amount, how much does the prediction [hat{y}] change?
Now let’s calculate.
Starting with
[
hat{y}=w_3a_1+w_4a_2+b_3
]
the derivative of
[
w_3a_1
]
with respect to
[
a_1
]
is
[
w_3
]
The derivative of
[
w_4a_2
]
is
[
0
]
because it does not depend on
[
a_1
]
Similarly,
[
b_3
]
is a constant, so its derivative is
[
0
]
Therefore,
[
frac{partial hat{y}}{partial a_1}
=
w_3
]
This tells us that for every one unit of increase in
[
a_1
]
the prediction changes by
[
w_3
]
Step 3:
The next link in our chain is
[
frac{partial a_1}{partial z_1}
]
Earlier, we computed the activation of the first hidden neuron using the ReLU activation function.
[
a_1=mathrm{ReLU}(z_1)
]
Therefore,
[
frac{partial a_1}{partial z_1}
]
tells us
If the input to the ReLU function changes by a small amount, how much does its output change?
Since
[
a_1=mathrm{ReLU}(z_1)
]
its derivative is simply the derivative of the ReLU function.
Therefore,
[
frac{partial a_1}{partial z_1}
=
mathrm{ReLU}'(z_1)
]
We know that the derivative of the ReLU function depends on the value of its input.
[
text{If } z_1 gt 0,quad mathrm{ReLU}'(z_1)=1.
]
[
text{If } z_1 lt 0,quad mathrm{ReLU}'(z_1)=0.
]
This may seem confusing at first, so let’s understand it with a simple example.
We just said that the derivative of the ReLU function depends on the value of its input.
If [z_1 gt 0] then [mathrm{ReLU}'(z_1)=1]
If [z_1 lt 0] then [mathrm{ReLU}'(z_1)=0]
But how?
Let’s understand this with a simple example.
Suppose
[
z_1=3
]
The ReLU output is
[
mathrm{ReLU}(3)=3
]
Now increase the input slightly from 3 to 4.
The new output becomes
[
mathrm{ReLU}(4)=4
]
Observe what happened here.
The input increased by
[
4-3=1
]
and the output also increased by
[
4-3=1
]
Therefore, the derivative is
[
frac{d(mathrm{ReLU})}{dz}
=
frac{1}{1}
=
1
]
Now let’s say the input is
[
z_1=10
]
Then
[
mathrm{ReLU}(10)=10
]
If we increase the input to
[
11
]
the output becomes
[
mathrm{ReLU}(11)=11
]
Again, the input increases by 1 and the output also increases by 1.
Therefore, the derivative is
[
frac{d(mathrm{ReLU})}{dz}
=
frac{1}{1}
=
1
]
So the derivative is still
[
1
]
We can see that the derivative is not equal to the value of the input.
Even though the input changed from 3 to 10, the slope of the ReLU function remained the same.
Now consider a negative input.
Suppose
[
z_1=-3
]
Then
[
mathrm{ReLU}(-3)=0
]
Now, if we increase the input slightly to
[
-2
]
the output is still
[
mathrm{ReLU}(-2)=0
]
The input changed, but the output did not.
Therefore,
[
frac{d(mathrm{ReLU})}{dz}
=
0
]
This is why the derivative of the ReLU function is
[
mathrm{ReLU}'(z_1)=1
]
for positive inputs, and
[
mathrm{ReLU}'(z_1)=0
]
for negative inputs.
In other words, we can say that the gradient passes through without changing whenever the input is positive and becomes zero whenever the input is negative.
Step 4:
We have now calculated all the derivatives except one.
The last derivative we need to calculate is
[
frac{partial z_1}{partial w_1}
]
Recall that
[
z_1=w_1x+b_1
]
Since [x] and [b_1] are constants with respect to [w_1] only the term [w_1x] changes when differentiating with respect to [w_1]
Therefore,
[
frac{partial z_1}{partial w_1}
=
x
]
Putting It All Together
We have now computed all the partial derivatives required by the chain rule.
They are:
[
frac{partial L}{partial hat{y}}
=
-2(y-hat{y})
]
[
frac{partial hat{y}}{partial a_1}
=
w_3
]
[
frac{partial a_1}{partial z_1}
=
mathrm{ReLU}'(z_1)
]
[
frac{partial z_1}{partial w_1}
=
x
]
We have already seen that the chain rule tells us
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial w_1}
]
Now substitute each partial derivative into this expression.
We get
[
frac{partial L}{partial w_1}
=
-2(y-hat{y})
cdot
w_3
cdot
mathrm{ReLU}'(z_1)
cdot
x
]
Since
[
z_1=w_1x+b_1
]
we can also write it as
[
frac{partial L}{partial w_1}
=
-2(y-hat{y})
cdot
w_3
cdot
mathrm{ReLU}'(w_1x+b_1)
cdot
x
]
This is the gradient of the loss with respect to the weight
[
w_1
]
It tells us how much the loss changes when we make a very small change in
[
w_1
]
As we know, during gradient descent, this is used to update the weight in the direction that reduces the loss.
An Interesting Observation
Notice something interesting.
We’ve arrived at exactly the same gradient we derived in Part 1 but this time we reached it using the chain rule instead of classical differentiation.
We just discovered a much more systematic way to compute gradients.
Although this derivation looks longer, observe what happened.
Every step was simple. We never had to differentiate the entire neural network at once, but we only differentiated one small part at a time.
The Real Challenge
Now what’s next?
We have successfully calculated the gradient for one parameter, but there are still six more parameters left.
Should we repeat the same process for each parameter?
Let’s see what happens when we try to repeat the process.
First, let’s have a look at the chain rule equations for all the parameters.
[
frac{partial L}{partial w_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial w_1}
]
[
frac{partial L}{partial b_1}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_1}
cdot
frac{partial a_1}{partial z_1}
cdot
frac{partial z_1}{partial b_1}
]
[
frac{partial L}{partial w_2}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_2}
cdot
frac{partial a_2}{partial z_2}
cdot
frac{partial z_2}{partial w_2}
]
[
frac{partial L}{partial b_2}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial a_2}
cdot
frac{partial a_2}{partial z_2}
cdot
frac{partial z_2}{partial b_2}
]
[
frac{partial L}{partial w_3}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial w_3}
]
[
frac{partial L}{partial w_4}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial w_4}
]
[
frac{partial L}{partial b_3}
=
frac{partial L}{partial hat{y}}
cdot
frac{partial hat{y}}{partial b_3}
]
Now, by looking at the equations, we can observe that many of the partial derivatives appear repeatedly.
For example,
[
frac{partial L}{partial hat{y}}
]
appears in every single equation.
Similarly,
[
frac{partial hat{y}}{partial a_1}
]
and
[
frac{partial a_1}{partial z_1}
]
appear in both the gradients of [w_1] and [b_1]
In the same way,
[
frac{partial hat{y}}{partial a_2}
]
and
[
frac{partial a_2}{partial z_2}
]
appear in both the gradients of [w_2] and [b_2]
This means that if we calculate the gradient for each parameter separately, we will repeatedly compute many of the same partial derivatives again and again.
As the number of parameters in a neural network grows, repeating these calculations quickly becomes computationally expensive.
While this may not look like a problem in a small neural network, modern neural networks often contain thousands or even millions of parameters.
Repeating the same calculations would require significantly more computational resources, and it increases the training time.
There Has to Be a Better Way
So, is there a better way to compute all these gradients without repeating the same calculations?
Yes.
But we don’t need a new mathematical concept. The same chain rule is enough.
What changes is how we apply it.
Instead of repeating the same computations for every parameter, we organize them in a way that it reuses intermediate results.
This idea is the foundation of backpropagation.
But knowing that repeated computations can be reused is only the beginning.
The real question is: how does a neural network know what to compute first, what to compute next, and how are all the gradients obtained in one efficient backward pass?
That’s exactly what we’ll explore in Part 3.
Conclusion
We’ve now built all the intuition we need.
We understand how gradients flow through a neural network, why the chain rule is essential, and why repeated computations become a problem as neural networks grow.
The only question left is that how can the same chain rule compute gradients for millions of parameters without repeating the same calculations?
In the next part, we’ll see how this works and gradually build an understanding for backpropagation.
I hope you found this blog helpful.
If you’re new to this series and would like to read the previous articles, you can find them here.
I’d love to hear your thoughts. If you have any questions or feedback, feel free to leave a comment on LinkedIn.
Learning never exhausts the mind.
— Leonardo da Vinci
Thanks for reading!