Loss landscape · Ravine
SGD oscillates; Adam glides straight to the bottom
SGD
Vanilla
θ ← θ − α·∇L
Loss
—
Steps
—
Momentum
β = 0.9
v ← β·v + ∇L
θ ← θ − α·v
θ ← θ − α·v
Loss
—
Steps
—
RMSProp
β = 0.9
s ← β·s + (1−β)·∇L²
θ ← θ − α·∇L / √s
θ ← θ − α·∇L / √s
Loss
—
Steps
—
Adam
β₁=0.9 β₂=0.999
m ← β₁·m + (1−β₁)·∇L
v ← β₂·v + (1−β₂)·∇L²
θ ← θ − α·m̂/√v̂
v ← β₂·v + (1−β₂)·∇L²
θ ← θ − α·m̂/√v̂
Loss
—
Steps
—