TOMAR: Text-Oriented Multimodal Ambivalence Recognition Approach

---

Elena Ryumina, Maxim Markitantov, Alexandr Axyonov, Fedor Shchetinin, Timur Abdulkadirov, Dmitry Ryumin, Alexey Karpov

---

Abstract

Automatic recognition of human's ambivalence and hesitations is a challenging task because such states may be expressed through inconsistent linguistic, acoustic, facial, and contextual patterns, while top-performing systems often rely on computationally expensive ensembles. We present a Text-Oriented Multimodal Ambivalence Recognition (TOMAR) approach, developed for the 11th Affective & Behavior Analysis in-the-Wild (ABAW) Challenge. TOMAR combines linguistic, acoustic, facial, and scene features using Text Residual Fusion model. This model treats text as an anchor modality and applies gated residual adjustments based on other modalities. Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that the textual modality is the strongest one. The Text Residual Fusion model achieves an average Macro F1-score of 75.14% across the Development and Public Test subsets. On the Private Test subset, TOMAR achieves a Macro F1-score of 78.24%, ranking second (RAS team) in the 11th ABAW Challenge. These results demonstrate that complementary multimodal information can improve the recognition performance without requiring a large model ensemble.

Repository description

This repository contains the unimodal and multimodal pipelines used for binary ambivalence recognition from video. Each modality is implemented as a separate pipeline and can be trained or evaluated independently. Their exported features are combined by the multimodal fusion model.

                         +------------------+
                         |  Input videos    |
                         +---------+--------+
                                   |
             +----------+----------+----------+----------+
             |          |          |          |          |
           audio       depth      face       scene       text
             |          |          |          |          |
             +----------+----------+----------+----------+
                                   |
                         exported features
                                   |
                         +---------v---------+
                         | Multimodal fusion |
                         +---------+---------+
                                   |
                         ambivalence prediction

Repository Structure

Directory	Description
audio/	Audio preprocessing, Mamba-based audio models, feature export, and the original audio-oriented fusion experiments.
depth/	Depth Anything 3 preprocessing, VideoMAE feature extraction, and temporal depth classification.
face/	Face detection, affective feature extraction, face-only training, and multimodal feature export.
scene/	VideoMAE-v2 scene feature extraction and Mamba/Transformer temporal classifiers.
text/	Transcript preparation and Transformer-based text classification with optional custom attention.
multimodal_fusion/	Fusion of audio, text, face, and scene representations into the final prediction.

The depth/vendor/ directory contains the vendored Depth Anything 3 dependency used by the depth extraction pipeline.

Workflow

1. Prepare the dataset and annotations required by the selected modality.
2. Follow the modality-specific README to install its dependencies and run preprocessing or feature extraction.
3. Train and evaluate the unimodal model.
4. Export the learned representations for multimodal fusion.
5. Place the exported representations in the feature directories expected by multimodal_fusion/configs/best_config.yaml.
6. Train or evaluate the fusion model using the instructions in the multimodal_fusion/ directory.

The current fusion configuration combines four modalities:

audio + text + face + scene -> multimodal prediction

The depth pipeline is included as an independently reproducible modality but is not enabled in the current best fusion configuration.

Setup

There is no single shared environment for the whole repository. The pipelines have different model and system dependencies, so install them from the relevant subdirectory and follow its README. Configuration files are stored alongside each pipeline under its configs/ directory.

Examples:

# Scene pipeline
cd scene
pip install -r requirements.txt

# Depth pipeline
cd ../depth
pip install -r requirements.txt

# Text pipeline
cd ../text
pip install -r requirements.txt

Audio uses its own pyproject.toml. Face and multimodal fusion keep their runtime configuration and entry points inside their respective directories; see their linked documentation for exact commands.

Data and Generated Artifacts

Datasets, pretrained weights, extracted features, checkpoints, experiment logs, and submission files are not stored in this repository. Paths to these artifacts are configured independently for each pipeline. Before running an experiment, check the corresponding configuration for local paths and expected split names.

Typical generated artifacts include:

features/       # extracted unimodal or temporal representations
runs/           # checkpoints and training reports
outputs/        # predictions and exported model features
submits/        # formatted submission files

Exact layouts and commands are documented in the README for each modality.