An Interpretable Text-Video Multimodal Depression Recognition System for Mental Health Monitoring

Hyun-eun Seo, Sun-Hee Kim · Seokyeong University Published at KIICE (Korea Institute of Information and Communication Engineering) 2026 Spring Conference, pp. 275–278.

🏆 This paper received the Best Paper Award at the KIICE 2026 Spring Conference, awarded by the Korea Institute of Information and Communication Engineering on May 22, 2026 (Award No. 26-27).

With the acceleration of AI transformation (AX), automatic recognition of users' emotional and mental states in mental health monitoring and conversational AI services is gaining significant importance. However, most existing depression recognition approaches rely on a single modality or general-purpose encoders without domain specialization, limiting recognition performance. This study proposes a multimodal depression recognition system combining a domain-specific text encoder (MentalRoBERTa) with a 3D facial landmark-based video analysis model (ST-GCN), validated on the DAIC-WOZ dataset. The proposed system achieves an MAE of 3.76 on the PHQ-8 regression task, a 30.6% improvement over the Early Fusion baseline (MAE 5.42). Results show that the text-video bimodal configuration outperforms the trimodal setting, and domain-specific encoders outperform general-purpose models (Gemma-2-9b). Additionally, Cross-modal Attention visualization reveals distinct attention patterns between depressive-positive and negative samples, demonstrating model interpretability.

Keywords: Multimodal Emotion Recognition, Depression Detection, MentalRoBERTa, ST-GCN, Cross-modal Attention

The system consists of two encoder branches fused by a two-stage attention module:

• Text branch — each utterance is encoded with MentalRoBERTa (125M params, 768-d), a language model pretrained on mental-health text, then linearly projected to 64-d.
• Video branch — 68 3D facial landmarks (X/Y/Z) per frame are modeled as a graph with anatomically defined edges (jawline, eyebrows, eyes, nose, mouth) and encoded with a 3-stage ST-GCN (spatial graph conv + temporal conv), then global-average-pooled to 64-d.
• Attention Fusion — stage 1 applies intra-modal self-attention within each branch to highlight utterances relevant to depressive symptoms; stage 2 applies cross-modal attention between text and video to fuse complementary information.
• The fused representation is regressed by an MLP to a PHQ-8 depression severity score (0–24).

Treating landmarks as a flat coordinate vector [x1, y1, x2, y2, ..., x68, y68] discards facial topology — the model can tell which coordinate is "the eye" but not the eye–nose–mouth relationship. The proposed model instead builds a facial topology graph (nodes + anatomical edges), extends it along the time axis, and applies ST-GCN so spatial structure and temporal dynamics are learned jointly.

KIICE/
├── models.py                        # EarlyFusionBaseline, MulT, MFMAtt (+ ST-GCN), MFMAtt_Explainable, MFMAtt_Coral
├── data_loader.py                   # DAIC-WOZ alignment, Z-score normalization, ST-GCN landmark loading
├── train_multiseed.py               # multi-seed training (v1)
├── train_multiseed_v2.py            # multi-seed training (v2, reproduces reported hyperparameters)
├── train_baseline_multiseed.py      # Early Fusion / MFM-Att baseline training
├── train_cached.py                  # training from precomputed feature cache
├── eval_checkpoint.py               # evaluate a saved checkpoint (no retraining)
├── measure_inference_time.py        # per-sample inference latency benchmark
├── visualize_attention.py           # cross-modal attention heatmap (generic)
├── visualize_attention_paper.py     # Figure 2 reproduction (paper version)
├── visualize_attention_paper_v3.py  # Figure 2 reproduction (final/v3)
├── run_after_cache.sh               # waits for feature cache, then runs latency benchmark
├── logs/                            # training logs for the runs reported in the paper
├── assets/                          # figures used in this README
└── docs/                            # paper PDF, award certificate, presentation slides

Model checkpoints (*.pt), feature caches (*.pkl) and raw DAIC-WOZ data are not included in this repository — see .gitignore. They are regenerable from the DAIC-WOZ dataset with the scripts above.

pip install -r requirements.txt

Data. Experiments use the DAIC-WOZ dataset (189 participants, PHQ-8 labels; access requires a data use agreement with USC ICT). data_loader.py expects the extracted DAIC-WOZ directory (*_TRANSCRIPT.csv, *_CLNF_features3D.txt, *_COVAREP.csv, *_FORMANT.csv, precomputed *_text_*.npy embeddings) — update DATA_DIR at the top of data_loader.py to point to your local copy.

# Train the proposed model (MentalRoBERTa + ST-GCN, cross-modal attention), 3 seeds
python train_multiseed_v2.py

# Train the Early Fusion baseline, 3 seeds
python train_baseline_multiseed.py

# Evaluate a saved checkpoint only
python eval_checkpoint.py --checkpoint best_proposed_seed42.pt --model mfm_att \
    --modality text_facial --use_stgcn

# Benchmark inference latency
python measure_inference_time.py

# Reproduce the cross-modal attention figure
python visualize_attention_paper_v3.py

DAIC-WOZ official split (train 107 / dev 34 / test 47), PHQ-8 regression, evaluated with MAE and RMSE.

Table 1. Fusion method comparison

Table 2. Modality combination comparison (MAE)

Key findings

• The proposed model improves MAE by 30.6% over the Early Fusion baseline while matching its inference latency (41.6 ms vs. 41.7 ms) — the ST-GCN branch adds no measurable speed cost.
• Text + facial (bimodal) beats text + audio + facial (trimodal) by 0.47 MAE — modality quality matters more than modality count; adding audio hurt performance.
• Swapping the domain-specific text encoder (MentalRoBERTa, 125M) for a general-purpose LLM (Gemma-2-9b, 9B — 72× larger) raised MAE from 3.76 to 6.02, showing that on a small dataset (189 subjects), domain-specific pretraining matters more than parameter count.

Comparing a depression-negative sample (PHQ-8 = 2) against a depression-positive sample (PHQ-8 = 22): for the negative sample, high attention weights concentrate in the early utterances (key index 0–30); for the positive sample, high weights spread across the entire interview (key index 0–140). This is interpreted as depressive cues (negative vocabulary, facial expression changes) appearing persistently throughout the interview for positive samples, versus being concentrated in emotion-related questions early on for negative samples — evidence the model is picking up on a sustained pattern of depressive symptoms rather than isolated cues.

Extended view showing both attention directions (Text → Facial and Facial → Text) for the same negative/positive pair.

Conference presentation slides: docs/KIICE_Presentation_Slides.pdf

@inproceedings{seo2026kiice,
  title     = {An Interpretable Text-Video Multimodal Depression Recognition System for Mental Health Monitoring},
  author    = {Seo, Hyun-eun and Kim, Sun-Hee},
  booktitle = {Proceedings of the KIICE 2026 Spring Conference},
  pages     = {275--278},
  year      = {2026},
  address   = {Republic of Korea},
  note      = {Best Paper Award}
}

Full text: docs/KIICE_Paper.pdf

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Ministry of Science and ICT (RS-2025-24533096).

Validated on a single English-language dataset (DAIC-WOZ, 189 participants), which limits scale and linguistic diversity. Future work: build a Korean depression-interview dataset to test cross-lingual transfer, and improve audio modality quality to make trimodal fusion viable.

1. W. Ren, X. Xue, L. Liu, and J. Huang, "AI applications in depression detection and diagnosis: Bibliometric and visual analysis of trends and future directions," JMIR Mental Health, vol. 12, e79293, 2025.
2. Y. Fang, H. Zhao, H. Zha, and Y. Zhang, "A multimodal fusion model with multi-level attention mechanism for depression detection," Biomedical Signal Processing and Control, vol. 82, 104561, 2023.
3. J. Zhou, J. Ge, Z. Chen, J. Tan, and Y. Li, "MDD-MARF: A multimodal depression detection model based on multi-level attention mechanism and residual fusion," Journal of Biomedical Informatics, vol. 173, 104965, 2026.
4. S. Ji, T. Zhang, L. Ansari, J. Fu, P. Tiwari, and E. Cambria, "MentalBERT: Publicly available pretrained language models for mental health," LREC, pp. 7184–7190, 2022.
5. S. Yan, Y. Xiong, and D. Lin, "Spatial temporal graph convolutional networks for skeleton-based action recognition," AAAI, pp. 7444–7452, 2018.
6. J. Gratch et al., "The Distress Analysis Interview Corpus of human and computer interviews," LREC, pp. 3123–3128, 2014.

_{Code and documentation prepared for public release. Model checkpoints, feature caches, and raw DAIC-WOZ data are excluded — see Setup above to regenerate them.}