A Survey of Optimization Methods From a Machine Learning Perspective

By Shiliang Sun, Zehui Cao, Han Zhu, Jing Zhao. IEEE Transactions on Cybernetics, 2019

672 citations recorded by the source corpus.

Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this article, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Finally, we explore and give some challenges and open problems for the optimization in machine learning.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/3119ea9c7ad7a5e044dc7c267329a4bbf00d0158

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, positive): “Despite the longstanding presence of IL in the realm of artiﬁcial intelligence (AI), recent advancements in computing and sensing technologies coupled with escalating demands for AI applications have propelled IL to the forefront of research [8], [9].”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, methodology): “Three flow examples in hemodynamics applications are presented in Sun et al (2020a), for addressing either stenotic flow and aneurysmal flow, with standardized vessel geometries and varying viscosity.”
A modified Adam algorithm for deep neural network optimization (2023, methodology): “In order to include the NAG algorithm in the Adam algorithm, the momentum expression is modified using the following update rule [6, 10].”
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, methodology): “There are many traditional methods that can be used to solve optimization problems, including gradient descent, Newtons method, conjugate gradient, and heuristic optimization methods [20].”