Computational Science and Engineering

By O. von Stryk. Applied Mechanics and Materials, 2019

419 citations recorded by the source corpus.

With the advent of Deep Learning, training large-scale neural networks has become a crucial task across a wide range of applications. Typically, the training is performed using gradient-based optimization algorithms, with the most popular being Stochastic Gradient Descent and its variants. Random feature models offer a different method for training neural networks, where the parameters of the hidden layers are sampled from a data-agnostic distribution, such as a normal distribution. Building on this concept, a novel approach, known as the Sampling Where It Matters (SWIM) algorithm, was recently proposed. In SWIM, the network parameters of the hidden layers are constructed using a data-driven sampling scheme, followed by solving a linear least squares problem for the output layer. This linear problem is dense and highly ill-conditioned, posing significant challenges for traditional numerical solvers. In this work, we investigate numerical methods for solving the least-squares problems that arise in the context of the SWIM algorithm, with a focus on scalability and efficiency. We propose using a recently developed iterative solver called LSRN, which leverages randomized preconditioning to significantly improve the convergence rate. In addition, we introduce an alternative approach that divides the problem into smaller subproblems, which are solved sequentially. We demonstrate the effectiveness of these methods through a series of numerical experiments, showcasing substantial improvements in training speed and scalability.

Subfields: Molecular Biology

View Paper: https://www.semanticscholar.org/paper/7a649ce3304aa787b91d2c59a09c5dc8ad1a3e96

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

1. Shanhui Fan
2. Shilpa Khatri
3. Anna-Karin Tornberg
4. Tyler W. Hughes
5. Momchil Minkov
6. Ian A. D. Williamson

Citations in the corpus

• A numerical method for two phase flows with insoluble surfactants (2011, methodology): “The numerical method in [32] is based on a scheme presented in [33].”
• Adjoint Method and Inverse Design for Nonlinear Nanophotonic Devices (2018, methodology): “[28] Gilbert Strang, Computational Science and Engineering (Wellesley-Cambridge Press, 2007) chapter 8.”