Logistic regression

Bend a line into a probability. Move the boundary and watch the log loss react.

Class 0
Class 1
×Misclassified
Ring size = point’s loss
p = 0.1 and 0.9

Tap empty space to add a Class 1 point; drag points to move them. Drag the square to slide the boundary and the circle to turn it.

Sigmoid: score z → probability p

Log loss while training1.0205

Logistic regression is a classifier. It first computes a score z = w₁x₁ + w₂x₂ + b, exactly like a line. Points where z = 0 form the decision boundary; the arrow shows the side where z is positive.

The sigmoid σ squashes any score into a probability between 0 and 1: z = 0 gives 0.5, big positive scores get close to 1, big negative ones close to 0. Making all the weights bigger keeps the same boundary but makes the model more confident: the dashed p = 0.1 and 0.9 lines squeeze in.

Log loss charges −ln(p) for the probability given to the true class. A point the model is unsure about (p = 0.5) costs 0.69; a confident mistake (p = 0.01) costs 4.6. Training is gradient descent on this loss, and the gradient is beautifully simple: each point pushes the weights by its error (p − y) times its features.

z = w₁·x₁ + w₂·x₂ + b p = σ(z) = 1 / (1 + e^(−z)) loss = −[ y·ln p + (1 − y)·ln(1 − p) ] (averaged over points) ∂loss/∂w₁ = mean((p − y)·x₁) ∂loss/∂b = mean(p − y) w ← w − η · gradient

Data

Tapping the plot adds

Weight w₁0.40

Weight w₂−0.60

Bias b0.50

Accuracy (p ≥ 0.5): 8/16
Log loss: 1.021
Steepness |w|: 0.72
Training steps: 0

Point inspector

—

Tap a point (or use the arrows) to see its score, probability and loss.

Learning rate η0.5

Training speed

Label points with

Things to try

Choose “One outlier”, tap the lone square in the top-left and watch its loss in the inspector. Press “Sharper ×2” a few times: the boundary does not move, yet that one point’s loss explodes while the others shrink towards 0.
Choose “Separable” and press Train on Fast. Accuracy hits 100% quickly, but |w| keeps growing and the loss keeps creeping towards 0 without ever reaching it. Why can the model always lower the loss by sharpening?
On “Overlapping”, press Train until the loss settles and Pin that boundary. Now drag the boundary somewhere else that still scores the same accuracy. Same accuracy, but compare the log losses: what does log loss notice that accuracy ignores?
Choose “One outlier”, set η = 5 and press Train. Then Reset weights, set η = 0.01 and train again. Watch the loss chart: one zig-zags, the other crawls.
Set w₁ = w₂ = 0. What probability does every point get? What is the loss if b = 0?

Test yourself: 5-minute drill Next: A tiny neural network