Introduction：

This repository holds the codebase, dataset and models for the paper:

Mission: mmWave Radar Person Identification with RGB Cameras.

Requirements:

• Python3 (>3.5)
• PyTorch
• Other Python libraries can be installed by pip install -r requirements.txt

Tips: If you cannot set up the environment locally, you can use the Google's Colab service to run this notebook.

Installation:

Install project

git clone https://github.com/EverRaynor/Mission

Download extra files and extract them to the directory.

straight_road_npy.zip
https://drive.google.com/file/d/1Z5B_TSyzjd6Kav6mklc4KHjDjJQ3aSta/view?usp=sharing
rgb_mmwave_data.zip
https://drive.google.com/file/d/1DtZ1epsKTfqbcmWJaMZqgq5tbhQbyRGq/view?usp=sharing

Codes:

train_midmodal_tri_nln_loc.py -a training script that also includes testing and visualization features.

network_tcmr.py - the network design file.

dataset_me_rgb.py - the dataset loading script.

draw_pose.py - Pose Visualization Code

losses.py - Loss Function Definitions

lime_test.py - Network Interpretability Testing Based on LIME

shap_test.py - Network Interpretability Testing Based on SHAP

smpl_utils_extend.py - Extended SMPL Source Code

Resnet.py - ResNet Source Code

gaitpart.py - GaitPart Source Code

Remaining .py files are configuration and open-source project files, which generally do not need to be changed.

Folders:

loss - Directory for saving training process data

net - Stores some network code

lib - Library directory, includes pre-trained models for HMR and TCMR

log - Directory for saving trained models, ready for direct use

res - Experimental results and visualization display

Training&Evaluation:

Modify the dataset location on line 8294 of dataset_me_rgb.py.

run Mission/train_midmodal_tri_nln_loc.py

Current allfusion_woh notes

The actively used training path in this workspace is:

• train_midmodal_allfusion_woh.py
• model_midmodal_allfusion_woh.py
• model_allfusion_woh_core.py
• mmwave_allfusion_core_modules.py
• dataset_straight_road_zhengdui.py

Important alignment notes for the Mission paper:

• RGB branch now has a TCMR temporal encoder hook.
• The paper-faithful default mmWave point encoder is now pointnet.
• The paper-faithful default train/test split is now 75% / 25% within each identity.
• The paper-faithful default batch size is now 16 and can be overridden with MISSION_BATCH_SIZE / MISSION_EVAL_BATCH_SIZE.
• The paper-faithful default TCMR behavior is now MISSION_TCMR_FREEZE=1, which keeps the pre-trained temporal encoder frozen as a feature extractor.
• The code will automatically try to load TCMR weights from:
  • lib/models/pretrained/base_data/tcmr_demo_model.pth.tar
  • or lib/models/pretrained/base_data/tcmr_demo_model.pth
  • or lib/models/pretrained/base_data/TCMR_pretrained_model_RELEASE/tcmr_table4_3dpw_test.pth.tar
• You can also override the path with MISSION_TCMR_PRETRAINED=/abs/path/to/file.
• If the TCMR file does not exist, the temporal encoder will run with random initialization and the run is no longer paper-faithful on the RGB side.
• mmWave input in straight_road_npy/list_all_ti.npy is 5D (x, y, z, i, v) and the current allfusion path now keeps all 5 dimensions instead of truncating to 3.
• Evaluation now prints eval_num_gallery, eval_num_query, and eval_num_ids so the retrieval protocol is visible in logs.

Important metric note:

• eval_top1, eval_top5, eval_mAP are the main retrieval metrics.
• eval_top1_l is an auxiliary middle-branch retrieval metric, not the primary final metric.