Martin Lund Trinhammer

Advances in multimodal predictive systems are ushering in a new era in human interaction with technology. One particular area receiving increasing attention is engagement estimation, a key building block in achieving safe and responsive systems. Engagement, however, is a thoroughly context-dependent phenomenon, varying considerably across cultures, age groups, and specific contexts. In the present contribution, we address this transferability issue in the literature, taking as the objective frame-level engagement estimation, evaluated across five datasets spanning adult dyadic conversation (NoXi, NoXi-Add, NoXi-J, MPIIGI) and child interaction in free-play settings (PInSoRo). To achieve generalizability across these diverse domains, we propose a configurable multimodal architecture comprising four main modules: multimodal fusion, temporal encoding, interaction modeling, and prediction heads. We report a global score of 0.59 across the evaluations, including a new best score of 0.66 CCC for NoXi-J. To advance both the technical frontier and the psychological understanding of engagement, we further report extensive ablations of feature contributions and the marginal gain of dyadic vs. single-person modeling. Crucially, our final contribution is a sensitivity analysis showing that PInSoRo’s performance is inflated by easy consensus frames and drops sharply once annotator-disagreement frames are included.

The meaning of a smile during conversation depends on what is being said, yet psychotherapy research has historically analyzed facial movements independently of context. We introduce a methodology that links both partners’ smiling in a dyadic setting to the sentiment of a speech turn (positive, neutral, negative). Using open-weight models on local hardware only, we transcribed and diarized attachment interviews, annotated the sentiment of the participant’s speech turns with a large language model (LLM), and analyzed individual smiling and smile synchrony as a function of that sentiment. We applied linear mixed models to test for differences in smiling according to sentiment. Across 70 interviews and 1,294 speech turns, only the interviewer’s smiling differed significantly according to the sentiment of what the participant was saying (interviewer d = .51, q < .001; participant d = .05, q = .165); the pair smiled simultaneously more often during positive than negative turns (d = .21, p < .001). By conditioning smiling on what is being said — sentiment annotated from the conversation as it unfolds rather than created by experimental instruction — our method enables the kind of context-sensitive analysis contemporary theories call for. The pipeline is open and reproducible, runs entirely on local hardware so that no data leave the site, and can be easily modified for any annotation scheme or clinical setting.

Machine-learning systems for mental health increasingly fuse several behavioral modalities, yet no synthesized evidence base exists from which to judge the relative gain of any individual modality. This question is acute for facial movements, as they are produced largely involuntarily, are regulated as biometric data, and therefore carry a privacy cost that evidence of added predictive value should justify. Following OSF preregistration, we systematically searched three databases with forward and backward snowballing. Eligible studies predicted any mental health outcome and reported performance both with and without the facial modality added to a unimodal or bimodal baseline (typically audio, text, or both). Across 73 eligible studies, the median relative gain attributable to the facial modality was 11.7%. Subgroup analyses indicated diminishing returns, with larger gains when adding the facial modality to a unimodal system than to a bimodal one. All studies rated as having a low overall risk of bias under PROBAST+AI (5/78) relied on classical machine learning rather than deep or hybrid architectures, suggesting that accuracy-maximizing incentives diverge from clinical requirements.

Infant affect is often displayed through facial expressions, leaving this modality a key source of insight into the well-being and social functioning of the child. Computational inference of infant affect could critically assist both researchers and clinicians working with infant development and mitigate the need for manual coding. While many studies have explored open-source solutions in the adult domain, only the commercial Baby FaceReader 9 exists for the infant domain. To address this gap, we utilize the recently proposed, open-source infant-native action unit (AU) detection library PyAFAR (Python-based Automated Facial Action Recognition) on a sample of 71 four-month old infants, whose facial expressions were annotated manually frame-by-frame for three minutes according to the Infant Facial Affect (IFA) coding scheme. Using these AUs as features, we classify negative, neutral and positive facial affect using XGBoost and Bayesian filtering, both in a multiclass and a binary setup. Our results show that the AUs estimates from PyAFAR combined with an XGBoost classification model can distinguish positive from neutral and positive from negative affect with AUC scores of 78\% and 76\%. This performance is largely on par with that reported in evaluation studies of the Baby FaceReader 9, when accounting for differences in study setup. Our work indicates that the area of infant facial affect is appropriate for imitation learning, given the availability of two different, commensurable theories underpinning the same phenomenon. Finally, we discuss how next iterations to PyAFAR may benefit from including AUs capturing more variability around infant forehead and mouth engagement.

Numerous studies have explored the possibility of developing automatic detection pipelines that can seamlessly diagnose patients with bipolar disorder (BD) and other mental illnesses. Such novel diagnostic tools increasingly rely on data sources, such as facial movements, whose relationship to BD has yet to be fully outlined. As such, these detection pipelines offer limited clinical value, despite promising performance estimates. A vital next step in achieving clinically reliable models is conducting granular interpretability analyses of which subsets of facial movements are responsible for determining patient or control class membership. In this work, we rely on facial movements encoded as action units (AUs) of 32 participants recorded while watching emotional film clips. Our objective is to delineate the specific facial micro-movements responsible for the differences between patients with BD and controls. We report how the movement of brow lowering (AU4) differentiates patients from controls with AUROC scores of up to 70% by applying the interpretable Fisher's Linear Discriminant Analysis in a binary, supervised classification design. Finally, we critically discuss the implications of this finding concerning current research designs in the context of facial movement analyses for mood disorders.

The delivery of mental healthcare through psychotherapy stands to benefit immensely from developments within Natural Language Processing (NLP), in particular through the automatic identification of patient specific qualities, such as attachment style. Currently, the assessment of attachment style is performed manually using the Patient Attachment Coding System (PACS; Talia et al., 2017), which is complex, resource-consuming and requires extensive training. To enable wide and scalable adoption of attachment informed treatment and research, we propose the first exploratory analysis into automatically assessing patient attachment style from psychotherapy transcripts using NLP classification models. We further analyze the results and discuss the implications of using automated tools for this purpose -- e.g., confusing `preoccupied' patients with `avoidant' likely has a more negative impact on therapy outcomes with respect to other mislabeling. Our work opens an avenue of research enabling more personalized psychotherapy and more targeted research into the mechanisms of psychotherapy through advancements in NLP

Structural changes in psychiatric systems have altered treatment opportunities for patients in need of mental healthcare. These changes are possibly associated with an increase in post-discharge crime, reported in the increase of forensic psychiatric populations. As current risk-assessment tools are time-consuming to administer and offer limited accuracy, this study aims to develop a predictive model designed to identify psychiatric patients at risk of committing crime leading to a future forensic psychiatric treatment course. We utilized the longitudinal quality of the Danish patient registries, identifying the 45.720 adult patients who had contact with the psychiatric system in 2014, of which 474 committed crime leading to a forensic psychiatric treatment course after discharge. Four machine learning models (Logistic Regression, Random Forest, XGBoost and LightGBM) were applied over a range of sociodemographic, judicial, and psychiatric variables. This study achieves a F1-macro score of 76%, with precision = 57% and recall = 47% reported by the LightGBM algorithm. Our model was therefore able to identify 47% of future forensic psychiatric patients, while making correct predictions in 57% of cases. The study demonstrates how a clinically useful initial risk-assessment can be achieved using machine learning on data from patient registries. The proposed approach offers the opportunity to flag potential future forensic psychiatric patients, while in contact with the general psychiatric system, hereby allowing early-intervention initiatives to be activated.