Optimization & Training

Optimizers, normalization, regularization, and training dynamics/techniques.

A topic in the Science corpus, drawn from 2,103 papers.

Trusted voices in Optimization & Training

Ordered by the trust Optimization & Training's own network concentrates on each researcher, most first.

  1. Jimmy Ba — trusted by 10,592 researchers here
  2. Diederik P. Kingma — trusted by 10,281 researchers here
  3. Kaiming He — trusted by 6,781 researchers here
  4. Shaoqing Ren — trusted by 6,656 researchers here
  5. X. Zhang — trusted by 6,606 researchers here
  6. Yoshua Bengio — trusted by 3,728 researchers here
  7. Sergey Ioffe — trusted by 3,763 researchers here
  8. Christian Szegedy — trusted by 3,749 researchers here
  9. Robert Tibshirani — trusted by 2,502 researchers here
  10. Geoffrey E. Hinton — trusted by 3,256 researchers here
  11. Yurii E. Nesterov — trusted by 1,820 researchers here
  12. Frank Hutter — trusted by 2,107 researchers here

Most-cited papers in Optimization & Training

Ordered by the recorded citation count from the source corpus; a citation is not an endorsement.

  1. Deep Residual Learning for Image Recognition (2015, Computer Vision and Pattern Recognition) — 234,194 citations
  2. Adam: A Method for Stochastic Optimization (2014, International Conference on Learning Representations) — 168,773 citations
  3. XGBoost: A Scalable Tree Boosting System (2016) — 57,634 citations
  4. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, International Conference on Machine Learning) — 47,277 citations
  5. Dropout: a simple way to prevent neural networks from overfitting (2014) — 43,754 citations
  6. Convex Optimization (2010) — 40,693 citations
  7. Decoupled Weight Decay Regularization (2017) — 36,546 citations
  8. SMOTE: Synthetic Minority Over-sampling Technique (2002) — 31,281 citations
  9. Proximal Policy Optimization Algorithms (2017, arXiv.org) — 29,461 citations
  10. Understanding the difficulty of training deep feedforward neural networks (2010) — 19,426 citations
  11. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (2017) — 18,997 citations
  12. Numerical optimization (2004) — 17,888 citations
  13. LightGBM: A Highly Efficient Gradient Boosting Decision Tree (2017) — 15,550 citations
  14. A tutorial on support vector regression (2004) — 12,903 citations
  15. Layer Normalization (2016) — 12,877 citations
  16. Ridge Regression: Biased Estimation for Nonorthogonal Problems (2000) — 12,604 citations
  17. mixup: Beyond Empirical Risk Minimization (2017) — 12,076 citations
  18. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016) — 11,798 citations
  19. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization (2011) — 11,425 citations
  20. Identity Mappings in Deep Residual Networks (2016) — 11,282 citations
  21. Random Search for Hyper-Parameter Optimization (2012) — 11,272 citations
  22. Improved Training of Wasserstein GANs (2017) — 11,031 citations
  23. SGDR: Stochastic Gradient Descent with Warm Restarts (2016) — 10,851 citations
  24. Addendum: Regularization and variable selection via the elastic net (2005) — 10,689 citations
  25. Improved Techniques for Training GANs (2016) — 10,497 citations
  26. Overcoming catastrophic forgetting in neural networks (2016) — 10,384 citations
  27. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (2003) — 9,804 citations
  28. Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties (2001) — 9,697 citations
  29. Practical Bayesian Optimization of Machine Learning Algorithms (2012) — 9,594 citations
  30. Algorithms for Non-negative Matrix Factorization (2000) — 9,381 citations
  31. Federated Optimization in Heterogeneous Networks (2018, Conference on Machine Learning and Systems) — 8,339 citations
  32. Trust Region Policy Optimization (2015, International Conference on Machine Learning) — 8,139 citations
  33. Improving neural networks by preventing co-adaptation of feature detectors (2012) — 8,038 citations
  34. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers (2011) — 8,037 citations
  35. Stable signal recovery from incomplete and inaccurate measurements (2005) — 7,765 citations
  36. Optimal Transport: Old and New (2008) — 7,666 citations
  37. Curriculum learning (2009) — 7,325 citations
  38. Addressing Function Approximation Error in Actor-Critic Methods (2018) — 7,119 citations
  39. A Practical Guide to Support Vector Classication (2008) — 7,108 citations
  40. An overview of gradient descent optimization algorithms (2016) — 6,991 citations
  41. ADADELTA: An Adaptive Learning Rate Method (2012) — 6,871 citations
  42. © Institute of Mathematical Statistics, 2004 LEAST ANGLE REGRESSION — 6,720 citations
  43. Stochastic gradient boosting (2002) — 6,637 citations
  44. Introductory Lectures on Convex Optimization - A Basic Course (2014) — 6,525 citations
  45. Special Invited Paper-Additive logistic regression: A statistical view of boosting (2000) — 6,512 citations
  46. Large-Scale Machine Learning with Stochastic Gradient Descent (2010) — 6,396 citations
  47. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015) — 6,140 citations
  48. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019) — 6,079 citations
  49. Neural Architecture Search with Reinforcement Learning (2016) — 5,967 citations
  50. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories (2004) — 5,814 citations

Most active researchers

By papers in this topic: a volume measure, not trust.