Abstract:In response to the performance limitations of single sensors in environmental perception tasks, this paper proposes a transformer-based multimodal 3D object detection method that integrates millimeter-wave radar, LiDAR, and camera data. The proposed framework consists of three key modules: (1) Camera module, which combines image features with initial 3D object predictions to enhance visual representation; (2) Radar module, which addresses the sparsity issue of millimeter-wave radar point clouds by introducing a temporal multi-frame fusion algorithm that aggregates data from five consecutive frames. Additionally, a point cloud weighted fusion algorithm is proposed to construct an enhanced radar point cloud by leveraging the complementary characteristics of millimeter-wave radar and LiDAR; (3) Fusion module, which employs a Transformer decoder to effectively integrate radar point clouds and camera features, thereby improving 3D object detection performance. Experiments are conducted on a custom urban road dataset, and the proposed method is evaluated based on metrics from the nuScenes dataset. The results demonstrate that, compared to existing methods, the proposed approach achieves a 6.38% improvement in mAP and a 5.93% increase in NDS.