Math With AmarA C A D E M Y
All math animations

University · Optimization

Take gradient descent steps

Choose a learning rate and see why convergence can be steady or oscillating.

Predict what will happen, press Play, then pause and explain what changed. Every control also works without playback.

Watch the relationship

Paused
Take gradient descent steps. Discrete update: 0 of 12. Position x: 3.5. Loss x²/2: 6.125. Update factor 1 − η: 0.6f(x) = x²/2 · update 0 of 12-4-2024048xDiscrete steps; lines only guide the eye.
Only the one-dimensional quadratic f(x) = x²/2 is modeled. All permitted rates satisfy 0 < η < 2; this bound is not universal for other objectives. The fixed plot spans x = −4.5 to 4.5 and y = 0 to 9.5. Connecting segments guide the eye and are not extra iterates.

Starts paused. Play once, pause anywhere, or use Step to inspect the mathematics. Playback stops when this panel leaves the screen.

Make it your experiment

Change one value. Notice what follows.

The controls adjust the model. Numbers below describe the current frame. Decimals are rounded.

Discrete update
0 of 12
Position x
3.5
Loss x²/2
6.125
Update factor 1 − η
0.6

HD animation studio

From experiment to screen.

Present a crisp Canvas scene, save a full-HD image, or capture your model as a silent video.

Understand what you are seeing

The idea behind the motion.

Gradient descent moves against a function's derivative to reduce its value. For f(x) = x²/2, the update is especially transparent: x becomes (1 − η)x. The magnitude shrinks when 0 < η < 2. Rates above one alternate across the minimum while still converging for this particular quadratic.

A relationship to keep

xₙ₊₁ = xₙ − ηf′(xₙ) = (1 − η)xₙ

Read the symbols alongside the explanation. A diagram shows the relationship; the assumptions tell you when it applies.

  1. STEP 1

    Find the slope

    On this curve the derivative at x is exactly x. The update subtracts η times that derivative. The teal curve is the objective and the amber points are discrete algorithm iterates.

  2. STEP 2

    Compare two kinds of convergence

    With 0 < η < 1, the next point stays on the same side of zero. With 1 < η < 2, the factor 1 − η is negative, so successive points cross the minimum with decreasing distance.

  3. STEP 3

    Check the function value

    Playback shows twelve discrete updates. Each update multiplies f(x) by (1 − η)², which is less than one throughout the permitted range. At η = 1, one step reaches the minimum exactly.

Your turn to explain

Make a prediction. Test your reasoning.

Starting at x₀ = 4 with η = 1.5, what are x₁ and x₂? Does the loss decrease?

Use the values specified in the question. Reset restores the initial values for this investigation.

Compare your explanation

The update factor is −0.5, giving x₁ = −2 and x₂ = 1. The loss x²/2 decreases from 8 to 2 to 0.5 despite alternating sides.

Work through a full lesson

Connect the animation to a worked example and practice questions.

Undergraduate · Calculus

Find the direction of fastest local increase

Maximize a directional derivative under a unit-length constraint.

Undergraduate · Calculus

Check whether a numerical decay method actually decays

Derive the step-size stability condition for forward Euler.