Learning Lightweight Lane Detection CNNs by Self Attention Distillation

By Yuenan Hou, Zheng Ma, Chunxiao Liu, Chen Change Loy. 2019

693 citations recorded by the source corpus.

Training deep models for lane detection is challenging due to the very subtle and sparse supervisory signals inherent in lane annotations. Without learning from much richer context, these models often fail in challenging scenarios, e.g., severe occlusion, ambiguous lanes, and poor lighting conditions. In this paper, we present a novel knowledge distillation approach, i.e., Self Attention Distillation (SAD), which allows a model to learn from itself and gains substantial improvement without any additional supervision or labels. Specifically, we observe that attention maps extracted from a model trained to a reasonable level would encode rich contextual information. The valuable contextual information can be used as a form of ‘free’ supervision for further representation learning through performing top- down and layer-wise attention distillation within the net- work itself. SAD can be easily incorporated in any feed- forward convolutional neural networks (CNN) and does not increase the inference time. We validate SAD on three popular lane detection benchmarks (TuSimple, CULane and BDD100K) using lightweight models such as ENet, ResNet- 18 and ResNet-34. The lightest model, ENet-SAD, performs comparatively or even surpasses existing algorithms. Notably, ENet-SAD has 20 × fewer parameters and runs 10 × faster compared to the state-of-the-art SCNN, while still achieving compelling performance in all benchmarks.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…Shmelkov et al. 2017; Cun and Pun 2020; Wang et al. 2019d; Huang et al. 2020; Wei et al. 2018; Hong and Yu 2019; Chawla et al. 2021), lane detection (Hou et al. 2019), person re-identiﬁcation (Wu et al. 2019a), pedestrian detection (Shen et al. 2016), Existing KD-based face recognition methods…”
Interactive Knowledge Distillation (2020, background): “To reduce the resource consumption while maintaining good performance, researchers leverage knowledge distillation techniques [30,6,9,10] to transfer informative knowledge from a cumbersome but well-trained teacher network into a light-weight but unskilled student network.”
Inter-Region Affinity Distillation for Road Marking Segmentation (2020, neutral): “72 ResNet-18-SAD [8] 70.”
RESA: Recurrent Feature-Shift Aggregator for Lane Detection (2020, methodology): “Self-attention distillation (SAD) is proposed to allow a model to learn from itself and gains substantial improvement without any additional supervision or labels (Hou et al. 2019).”
Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection (2020, background): “As noted in [23], many of the cited works do not publish the code to reproduce the results reported [13, 18, 27], or, in some cases, the code is only partially public [11, 10].”
CurveLane-NAS: Unifying Lane-Sensitive Architecture Search and Adaptive Point Blending (2020, methodology): “4 Enet-SAD [11] 50.”
Computing Systems for Autonomous Driving: State of the Art and Challenges (2020, background): “For example, self-attention distillation (SAD) [99] adopts an attention distillation mechanism.”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “Self-Attention Distillation (SAD) (Hou et al. 2019) designs a model to learn from itself and thus gains improvement.”
Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation (2021, background): “• SAD [10] focuses on a lane detection by a layer-wise attention distillation in network itself.”
End-to-end Lane Shape Prediction with Transformers (2020, methodology): “Existing methods [12, 14, 6, 15, 7] take advantage of the powerful representation capabilities of convolution neural networks (CNNs) to improve the performance of lane detection task by a large margin over traditional methods [11, 13, 20] which are based on hand-crafted features and Hough Transform.”
CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution (2021, methodology): “Segmentation-based methods [28, 12, 27, 19, 21, 6] are most common and have achieved impressive performance.”
Comprehensive Attention Self-Distillation for Weakly-Supervised Object Detection (2020, background): “[30, 31] propose cross-layer consistency losses over attention maps for image classification and lane detection.”
MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, background): “Beyond the logit level, SAD [16] performs top-down and layer-wise attention distillation within the network itself.”
Ultra Fast Structure-aware Deep Lane Detection (2020, methodology): “Category Res50-Seg[3] SCNN[22] FD-50[24] Res34-SAD SAD[9] Res18-Ours Res34-Ours”
Intra-class Feature Variation Distillation for Semantic Segmentation (2020, background): “The self-attention distillation (SAD) is introduced in [20] to explore attention maps derived from high-level features as the distillation target for shallow layers.”