Explainable Multimodal Depression Detection Using Dynamic Gate Fusion and Modality-Level SHAP

Abstract

Purpose: Depression often goes undetected due to masked social responses. This study aims to develop an explainable multimodal depression detection framework using an adaptive Dynamic Gate Fusion mechanism with L2 normalization, provide modality-level quantitative explanations via SHAP, and generate clinical decision support reports.

Methods: Utilizing the DAIC-WOZ dataset with PHQ-8 labels, the framework extracts text features via RoBERTa-base (CLS embedding) and audio features via OpenSMILE ComParE with 40-coefficient MFCC. The core architecture integrates a BiLSTM TextEncoder with Self-Attention, an MLP AudioEncoder, and a Dynamic Gate Fusion mechanism with entropy regularization.

Result: Across 25 fold evaluations (Stratified 5-Fold CV × 5 random seeds), the model achieved a Global Mean Macro F1 of 0.6749 ± 0.0921. Three AUC metrics are reported for transparency: (i) per-fold mean AUC of 0.6123 ± 0.1587; (ii) calibrated aggregate ROC-AUC of 0.6723 on pooled validation predictions; and (iii) minimum per-fold AUC of 0.5417 (RS=7), with 6 of 25 folds (24%) below 0.5 due to hard-case clustering rather than systematic failure. The most stable configuration (RS=7) yielded Macro F1 = 0.6928 ± 0.0419. SHAP demonstrated balanced modality utilization (text 44.5%, audio 55.5%). These results are a proof of concept for explainable multimodal fusion, not a ready-to-use tool., with the most stable configuration yielding a Macro F1 of 0.6928 ± 0.0419. SHAP analysis demonstrated balanced modality utilization, with audio contributing 55.5% and text 44.5%. Hard-case analysis successfully identified three clinical patterns: masked depression, psychomotor retardation, and threshold-boundary label ambiguity. These results constitute a proof of concept for explainable multimodal fusion on a small clinical dataset (N=107); generalization to larger, more diverse populations requires further validation.

Novelty: This study advances the field through two primary contributions: (1) adaptive per-sample Dynamic Gate Fusion with L2 normalization and entropy regularization, proven to prevent gate collapse and quantified by ablation (text gate: 0.1%→32%); and (2) modality-level quantitative explainability using SHAP, demonstrating genuine balanced modality utilization (text 44.5%, audio 55.5%). This system is presented as a research proof of concept for explainability rather than a ready-to-use clinical screening tool.