Derivatives Just in Time

Goal

By the end of this lesson, you can interpret a derivative as local sensitivity, predict its sign from a simple curve, and compare a numerical finite-difference estimate with an exact derivative.

A derivative answers a small-change question

You do not need to begin with a page of calculus symbols.

Start with this question:

If I nudge the input a tiny amount to the right, does the output rise or fall, and how quickly?

That local rate of change is the derivative.

Read derivative sign from a bowl-shaped function

Consider:

f(w) = (w - 3)²

The function is smallest at w = 3.

At w = 1, moving right toward 3 makes the function smaller. The derivative is negative.

At w = 5, moving right moves farther away from 3 and makes the function larger. The derivative is positive.

At w = 3, a tiny movement in either direction increases the function symmetrically, so the derivative is zero.

The exact derivative is:

f'(w) = 2(w - 3)

Now the formula matches the geometry you already reasoned about.

Estimate the derivative numerically

A centered finite difference uses a small step h:

(f(w+h) - f(w-h)) / (2h)

This compares nearby values on both sides of the point.

If h is reasonably small, the estimate should be close to the exact derivative for this smooth function.

If h is huge, you are averaging behavior across a wide region rather than measuring a truly local slope.

Extremely tiny h can also create floating-point precision problems in real numerical code. Gradient checking therefore uses a sensible small scale rather than “as small as possible.”

Compare exact and numerical slopes in the Lab

1. Click Run and compare the numerical derivative with 2*(w-3).
2. Set the evaluation point to w = 3 and predict the sign/value before running.
3. Try w = 5 and predict the sign again.
4. Increase the finite-difference step substantially and observe whether the approximation changes.
5. Reset the starter afterward.

Loading lab…

How derivative sign connects to gradient descent

Gradient descent subtracts the derivative or gradient.

• negative derivative → subtracting it moves the parameter upward;
• positive derivative → subtracting it moves the parameter downward;
• near-zero derivative → the local update becomes small.

For this bowl-shaped function, those movements point toward the minimum at 3.

Debugging derivative code

When a derivative estimate looks wrong, check:

1. the function you are differentiating;
2. the evaluation point;
3. the step size;
4. subtraction order;
5. whether you are comparing the same scalar objective.

Reversing f(w+h) - f(w-h) flips the estimated sign, which can reverse a training update.

Read a derivative as a local sensitivity

Suppose a small change in w from 2.00 to 2.01 changes loss from 4.00 to about 4.06.

A rough local sensitivity is:

change in loss / change in w
≈ 0.06 / 0.01
≈ 6

That does not mean increasing w by 1 will always increase loss by exactly 6. A derivative describes the slope near the current point.

This local nature is why gradient descent takes repeated steps and recomputes gradients after parameters move.

Sign and magnitude answer different questions

• positive derivative: increasing the variable locally raises the output;
• negative derivative: increasing the variable locally lowers the output;
• large magnitude: the output is locally sensitive to that variable;
• near zero: small local movement has little first-order effect.

Do not interpret a zero derivative as “this parameter can never matter.” Nonlinear networks can have regions where a local slope is zero even though the parameter affects behavior elsewhere.

Quick Check

1. What does derivative sign describe locally?

Whether the function is globally increasing over its entire domainWhether a tiny increase in the input raises or lowers the outputHow far the optimizer will move after applying any learning rate

2. What should the derivative be near the minimum of `(w-3)^2` at `w=3`?

Undefined, because a minimum has no directionNear zero, because the bowl is flat at its lowest pointPositive, because the loss itself is nonnegative

3. Why use a small finite-difference step?

To measure average sensitivity over a wide neighborhood instead of local behaviorTo guarantee the finite-difference estimate is exactly equal to the analytic derivativeTo measure local sensitivity

0 of 3 questions answered.

Key Takeaways

• A derivative measures local sensitivity.
• Its sign tells whether the function locally rises or falls as the input increases.
• Finite differences provide an independent numerical gradient check.
• Derivative signs explain gradient-descent update direction.

Next Lesson

Next, you will compute the local derivative pieces in code, combine them, and verify that one parameter update moves loss in the predicted direction.

References

• PyTorch, Autograd.