University of Leicester
Browse

DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-Level and Feature-Level Fusion

Download (400.34 kB)
conference contribution
posted on 2024-10-29, 11:18 authored by Zixuan Yin, Han Sun, Ningzhong Liu, Huiyu Zhou, Jiaquan Shen

Lidars and cameras play essential roles in autonomous driving, offering complementary information for 3D detection. The state-of-the-art fusion methods integrate them at the feature level, but they mostly rely on the learned soft association between point clouds and images, which lacks interpretability and neglects the hard association between them. In this paper, we combine feature-level fusion with point-level fusion, using hard association established by the calibration matrices to guide the generation of object queries. Specifically, in the early fusion stage, we use the 2D CNN features of images to decorate the point cloud data, and employ two independent sparse convolutions to extract the decorated point cloud features. In the mid-level fusion stage, we initialize the queries with a center heatmap and embed the predicted class labels as auxiliary information into the queries, making the initial positions closer to the actual centers of the targets. Extensive experiments conducted on two popular datasets, i.e. KITTI, Waymo, demonstrate the superiority of DecoratingFusion.

History

Author affiliation

College of Science & Engineering Comp' & Math' Sciences

Source

Artificial Neural Networks and Machine Learning – ICANN 2024 33rd International Conference on Artificial Neural Networks, Lugano, Switzerland, September 17–20, 2024, Proceedings, Part II

Version

  • AM (Accepted Manuscript)

Published in

Artificial Neural Networks and Machine Learning – ICANN 2024

Volume

15017 LNCS

Pagination

108 - 119

Publisher

Springer Nature Switzerland

issn

0302-9743

eissn

1611-3349

isbn

9783031723346

Copyright date

2024

Available date

2024-10-29

Book series

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Temporal coverage: start date

2024-09-17

Temporal coverage: end date

2024-09-20

Language

en

Deposited by

Professor Huiyu Zhou

Deposit date

2024-10-24

Usage metrics

    University of Leicester Publications

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC