Synetgy: Algorithm-hardware Co-design for ConvNet Accelerators on Embedded FPGAs

By Yifan Yang, Qijing Huang, Bichen Wu, Tianjun Zhang, Liang Ma, Giulio Gambardella, Michaela Blott, Luciano Lavagno, Kees A. Vissers, John Wawrzynek and 1 more. 2018

122 citations recorded by the source corpus.

Using FPGAs to accelerate ConvNets has attracted significant attention in recent years. However, FPGA accelerator design has not leveraged the latest progress of ConvNets. As a result, the key application characteristics such as frames-per-second (FPS) are ignored in favor of simply counting GOPs, and results on accuracy, which is critical to application success, are often not even reported. In this work, we adopt an algorithm-hardware co-design approach to develop a ConvNet accelerator called Synetgy and a novel ConvNet model called DiracDeltaNet. Both the accelerator and ConvNet are tailored to FPGA requirements. DiracDeltaNet, as the name suggests, is a ConvNet with only $1\times 1$ convolutions while spatial convolutions are replaced by more efficient shift operations. DiracDeltaNet achieves competitive accuracy on ImageNet (89.0% top-5), but with 48× fewer parameters and 65× fewer OPs than VGG16. We further quantize DiracDeltaNet's weights to 1-bit and activations to 4-bits, with less than 1% accuracy loss. These quantizations exploit well the nature of FPGA hardware. In short, DiracDeltaNet's small model size, low computational OP count, ultra-low precision and simplified operators allow us to co-design a highly customized computing unit for an FPGA. We implement the computing units for DiracDeltaNet on an Ultra96 SoC system through high-level synthesis. Our accelerator's final top-5 accuracy of 88.2% on ImageNet, is higher than all the previously reported embedded FPGA accelerators. In addition, the accelerator reaches an inference speed of 96.5 FPS on the ImageNet classification task, surpassing prior works with similar accuracy by at least 16.9×.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “A recent line of work focuses on the codesign of algorithm and hardware for accelerating deep models on platforms with more flexibility such as FPGA [274].”
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search (2018, methodology): “Synetgy [28] combines ideas from ShuffleNetV2 and ShiftNet to co-design hardware friendly ConvNets and FPGA accelerators.”
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, background): “Such non-linear structures create dependencies that may be very difficult to optimize for fast inference [37].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “For FPGA implementations, spatial convolutions are replaced by shift operations and depth convolutions followed by 1 × 1 convolutions in DiracDeltaNet [164].”
SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis (2020, background): “Previous work (Iandola et al., 2016; Wu et al., 2016; 2017; 2018a;b; Yang et al., 2018; Gholami et al., 2018; Wu, 2019; Wu et al., 2019; Dai et al., 2019) have shown that optimizing the neural network architecture can lead to significant efficiency improvement in many applications.”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, background): “Many neural nets target to achieve efficiency, including SqueezeNet [16,10,54], MobileNets [12,39,11], ShiftNet [55,61], ShuffleNet [63,28], FBNet [53,57], ChamNet [7], MnasNet [44], and EfficientNet [45].”