SHEPHERD: Serving DNNs in the Wild

By Hong Zhang, Yupeng Tang, Anurag Khandelwal, Ion Stoica. 2023

141 citations recorded by the source corpus.

Subfields: ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

1. Joseph E. Gonzalez
2. Ion Stoica
3. Xin Jin
4. Lianmin Zheng
5. Yibo Zhu

Citations in the corpus

• Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, background): “REEF [21] and Shepherd [61] propose preemption for serving.”
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, background): “Although short-term interval is impossible to predict, the workload pattern over longer timescales (e.g., hours or days) is often predictable [33, 55].”