A from-scratch test of modern optimizers. The signal is the moment estimates, bias correction, and the one change that matters: AdamW decouples weight decay from the gradient instead of folding it into the loss. Here is the implementation and why it wins.
Updated Aug 2026 · Grounded in real Applied AI Engineer interview loops and written to a senior-engineer editorial bar.
A from-scratch test of modern optimizers. The signal is the moment estimates, bias correction, and the one change that matters: AdamW decouples weight decay from the gradient instead of folding it into the loss. Here is the implementation and why it wins.
No lesson covers this question directly yet. These teach the surrounding topic from the beginning.
No comments yet — be the first to share your approach.
