Packet-Level Telemetry in Large Datacenter Networks

By Yibo Zhu, Nanxi Kang, Jiaxin Cao, Albert G. Greenberg, Guohan Lu, Ratul Mahajan, David A. Maltz, Lihua Yuan, Ming Zhang, Ben Y. Zhao and 1 more. Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, 2015

397 citations recorded by the source corpus.

Debugging faults in complex networks often requires capturing and analyzing traffic at the packet level. In this task, datacenter networks (DCNs) present unique challenges with their scale, traffic volume, and diversity of faults. To troubleshoot faults in a timely manner, DCN administrators must a) identify affected packets inside large volume of traffic; b) track them across multiple network components; c) analyze traffic traces for fault patterns; and d) test or confirm potential causes. To our knowledge, no tool today can achieve both the specificity and scale required for this task. We present Everflow, a packet-level network telemetry system for large DCNs. Everflow traces specific packets by implementing a powerful packet filter on top of "match and mirror" functionality of commodity switches. It shuffles captured packets to multiple analysis servers using load balancers built on switch ASICs, and it sends "guided probes" to test or confirm potential faults. We present experiments that demonstrate Everflow's scalability, and share experiences of troubleshooting network faults gathered from running it for over 6 months in Microsoft's DCNs.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/4f0344780b323013f72aec039afc58f7b953bcee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Sonata: query-driven streaming network telemetry (2018, methodology): “All-SP Mirror all incoming packets to the stream processor Gigascope[10],OpenSOC[40], NetQRE[58] Filter-DP Apply only filter operations on the switch EverFlow[60] Max-DP Execute asmanydataflowoperationsas possible on the switch Univmon[26],OpenSketch[56] Fix-REF Iteratively zoom-in one refinenement ”
MegaScale: Scaling Large Language Model Training to More Than 10, 000 GPUs (2024, background): “We believe that our work not only offers practical insights for those working on LLM training, but also paves the way for future research in this rapidly evolving field.”
Language-Directed Hardware Design for Network Performance Monitoring (2017, background): “The same concern limits other strategies that mirror traffic or collect packet digests from switches [8, 42, 65].”
SketchVisor: Robust Network Measurement for Software Packet Processing (2017, methodology): “Some systems [21, 29, 38, 62] attempt to filter traffic by predefined rules, so as to reduce memory usage.”
FlowRadar: A Better NetFlow for Data Centers (2016, methodology): “We then describe three alternative monitoring solutions (Table 1): NetFlow in high-end routers with custom silicon, NetFlow in cheap switches with merchant silicon, and selective mirroring.”