ImmFusion for Weather-Robust 3D Human Body Reconstruction

3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. To address this, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for weather-robust 3D human reconstruction is challenging due to the sparse nature of mmWave and the vulnerability of RGB images. To overcome these challenges, a large-scale mmWave dataset was created, consisting of synchronized and calibrated mmWave radar point clouds and RGB(D) images under different weather conditions. Based on this dataset, ImmFusion was designed as the first mmWave-RGB fusion solution to robustly reconstruct 3D human bodies in various weather conditions. ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve robust 3D human body reconstruction in various weather environments.

Category: Artificial Intelligence
Subcategory: Computer Vision
Tags: 3D human reconstructionmmWave radarRGB imagesweather-robust
AI Type: Deep Learning
Programming Languages: Not specified
Frameworks/Libraries: Not specified
Application Areas: 3D human body reconstruction
Manufacturer Company: Not specified
Country: Not specified
Algorithms Used

Transformer

Model Architecture

Image and point backbones, Fusion Transformer Module

Datasets Used

Large-scale mmWave dataset

Performance Metrics

Robustness in various weather conditions

Deployment Options

Not specified

Cloud Based

No

On Premises

No

Features

Weather-robust, mmWave-RGB fusion, Transformer-based

Enterprise

No

Hardware Requirements

Not specified

Supported Platforms

Not specified

Interoperability

Not specified

Security Features

Not specified

Compliance Standards

Not specified

Certifications

Not specified

Open Source

No

Source Code URL

http://Not specified

Documentation URL

http://Not specified

Community Support

Not specified

Contributors

Not specified

Training Data Size

Not specified

Inference Latency

Not specified

Energy Efficiency

Not specified

Explainability Features

Not specified

Ethical Considerations

Not specified

Known Limitations

Not specified

Industry Verticals

Not specified

Use Cases

3D human body reconstruction

Customer Base

Not specified

Integration Options

Not specified

Scalability

Not specified

Support Options

Not specified

SLA

Not specified

User Interface

Not specified

Multi-Language Support

No

Localization

Not specified

Pricing Model

Not specified

Trial Availability

No

Partner Ecosystem

Not specified

Patent Information

Not specified

Regulatory Compliance

Not specified

Version

Not specified

Website URL

http://Not specified

Service Type

Not specified

Has API

No

API Details

Not specified

Business Model

Not specified

Price

0.00

Currency

Not specified

License Type

Not specified

Release Date

01/01/1970

Last Update Date

01/01/1970

Contact Email

Not specified

Contact Phone

Not specified

Social Media Links

http://Not specified

Other Features

Not specified

Published

Yes