ℓ = 1 − exp(−‖r‖² / 2σ²), with r = ŷ − y, for every model. Like ‖r‖²/2σ² near r = 0, never more than 1, so a far point stops pulling.

Adds λ(θ1² + θ2²) to every point's cost, so it changes J.

The cost. Every surface is one recipe: a model ŷ with two parameters θ1 and θ2, a loss on how far ŷ is from the data point, and a penalty λ(θ1² + θ2²). The loss is squared, ‖r‖², or levels off, 1 − exp(−‖r‖² / 2σ²). So each data point has a cost of its own, ℓ(θ1, θ2), a surface over parameter space, and J is the mean of those surfaces. Hover a point to see its own surface as contours on the map.

The models. Every loss is on r = ŷ − y, how far the prediction is from the target. Point: a data point is a target y = (y1, y2) with no input, and ŷ = (θ1, θ2), so ‖r‖ is the distance from ŷ to the point. Line: a data point is (x, y), and ŷ = θ1·x + θ2. Each point's surface is a trough along the line θ2 = y − θ1·x. With the squared loss J is one bowl; with the loss that levels off, J has a hollow for each line in the data, deepest for the line with the most points. Tanh neuron and wave: the same, for ŷ = θ2·tanh(θ1·x) and ŷ = sin(θ1·x + θ2). For the point model, With the squared loss the minimum is the mean of the points; with the loss that levels off there is a hollow at each clump. Type J: any formula in t1 and t2 (or θ1, θ2), with ^, exp, sin, cos, tanh, sqrt, abs, log; no data, so every step is full-batch.

Data space. click a point to select it, and again to unselect; the selected points are filled dark. Drag a point to move it, and its surface moves with it. double-click empty space adds a point; double-click a point, or hover it and press delete, removes it. Moving a point changes J, not the walks so far.

Step. Every optimizer steps on the same batch; tick one in the Run panel to show it. SGD: θ ← θ − η·g. Momentum: v ← 0.9·v + g, θ ← θ − η·v. Nesterov: the same, with g taken at the look-ahead point θ − 0.9·η·v. AdaGrad: G ← G + g², θ ← θ − η·g / √G. RMSProp: r ← 0.9·r + 0.1·g², θ ← θ − η·g / √r. Adam: Kingma and Ba, β1 = 0.9, β2 = 0.999. Entry by entry throughout. Here g is the gradient of J on the batch, by centred differences. Each has its own η. A run that leaves the plane by a huge margin stops and says diverged.

The surface is J; drag to orbit, scroll to zoom, double-click for the default view. The orange mesh is J on the next batch only, which is what the step actually goes downhill on. From above: the same J as a map. click it to start every optimizer there. The switch in the header picks what both show: J on all points, on the next batch, or on the selected points alone, each on its own colour scale. B = sel makes every step use the selected points as its batch.

space steps, r runs or pauses, ← → look back and forward.

The model. s = w1·x1 + w2·x2 + b, p = σ(s). J = mean of −log p(correct class) over the points, + λ·(w1² + w2²). The bias is not penalized.

Data space. Drag a point to move it. double-click empty space adds a red point, shift-double-click a blue one; double-click a point, or hover it and press delete, removes it. Table lists every point to type exact values. Moving a point changes J, not the walk so far; adding or removing one also restarts the batch schedule.

Step takes one step, θ ← θ − η·∇J_batch. With B = all the batch is every point. With smaller B the points are shuffled once per epoch and taken B at a time; the ringed points are the batch for the next step.

The walk is the real path through (w1, w2, b), coloured red to purple by step. Drag to orbit, scroll to zoom, double-click for the default view. The coloured plane is the slice. Path points behind it are drawn under it.

The slice holds one parameter fixed (w1, w2 or b, in the header) and colours J over the other two, on one colour scale for every slice. Path points fade with their distance from the plane. Arrows on the slice are only the in-plane part of each step.

η and B change how the walk goes. λ, and the dataset, change J itself: the colours and the minimum move, the walk does not.

space steps, r runs or pauses, ← → look back and forward.