Why do Learning Rates Transfer? Reconciling Optimization and Scaling Limits for Deep Learning
Paper
•
2402.17457
•
Published
math
-
-
Curvature-Informed SGD via General Purpose Lie-Group Preconditioners
Paper • 2402.04553 • Published -
TextGrad: Automatic "Differentiation" via Text
Paper • 2406.07496 • Published • 26 -
Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
Paper • 2405.14578 • Published -
Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates
Paper • 2206.00832 • Published -
Large Language Models as Markov Chains
Paper • 2410.02724 • Published • 31 -
Old Optimizer, New Norm: An Anthology
Paper • 2409.20325 • Published • 3 -
Scaling Law with Learning Rate Annealing
Paper • 2408.11029 • Published • 3