This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences

Citations in the corpus

• Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, background): “REEF [21] and Shep-herd [61] propose preemption for serving.”
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, background): “DistServe does not implement advanced runtime policies like preemption [26] and fault tolerance [58], which are complementary to disaggregation.”
• SHEPHERD: Serving DNNs in the Wild (2023, methodology): “While recent hardware-based preemptions on newer GPUs [31] may enable better performance, we opt for software-based preemptions adapted from Pipeswitch [21] in SHEPHERD due to its general applicability to commodity GPUs.”