We need to talk about random seeds
By Steven Bethard. 2022
22 citations recorded by the source corpus.
Modern neural network libraries all take as a hyperparameter a random seed, typically used to determine the initial state of the model parameters. This opinion piece argues that there are some safe uses for random seeds: as part of the hyperparameter search to select a good model, creating an ensemble of several models, or measuring the sensitivity of the training algorithm to the random seed hyperparameter. It argues that some uses for random seeds are risky: using a fixed random seed for"replicability"and varying only the random seed to create score distributions for performance comparison. An analysis of 85 recent publications from the ACL Anthology finds that more than 50% contain risky uses of random seeds.
Subfields: Artificial Intelligence, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Martin Strobel
- Hong Chang
- Reza Shokri
- Prakhar Ganesh
- Prateek Mittal
- Liwei Song
- Lingjuan Lyu
- Yang Zhang
In Evaluation & Benchmarks
Citations in the corpus
- On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “Thus, randomness during training is integral to the success of neural networks, but its impact on model behavior needs to be carefully examined [7, 10].”