Unified Multimodal Architecture for Accessible Video Understanding: Integrating Vision, Captions, Sign Language, and Audio Descriptions with Robustness to Modality Degradation

Abstract

Video accessibility remains a critical challenge for people with visual and hearing impairments, despite the exponential growth of video content. Current accessibility solutions operate in isolation—audio descriptions for the blind and low vision (BLV) community, captions for the deaf and hard of hearing (DHH) community, and sign language interpretation as a separate overlay—missing opportunities for synergistic multimodal understanding. This paper introduces UMA (Unified Multimodal Architecture), an end-to-end framework that integrates vision, captions, sign language, and audio descriptions into a cohesive system designed for accessible video comprehension. The core innovation lies in parameter-efficient robustness to modality degradation: leveraging intermediate feature modulation with fewer than 0.7% additional parameters, UMA maintains comprehension fidelity even when individual modalities are corrupted, delayed, or missing—a realistic constraint in real-world video streaming and broadcasting environments. We validate UMA across 40,000 videos using rigorous evaluation with 347 sighted participants, 40 BLV users, and 7 professional audio describers, demonstrating that unified multimodal descriptions significantly outperform isolated accessibility services (p<0.001 across four custom metrics: descriptiveness, objectivity, accuracy, clarity). Our framework achieves state-of-the-art performance on video accessibility benchmarks while maintaining computational efficiency suitable for edge deployment. We release VideoA11y-Unified-40K, an extended dataset with synchronized captions, sign language glosses, and audio descriptions, alongside open-source implementations to enable future research and deployment in production broadcasting systems.

Introduction

The text addresses the digital divide in video accessibility for people with disabilities, particularly the blind/low-vision (BLV) and deaf/hard-of-hearing (DHH) communities. Globally, over 1.3 billion people have visual impairments, and 430 million have disabling hearing loss, yet 85–95% of online videos lack accessibility features. Current solutions—audio descriptions, captions, and sign language—are fragmented, underutilized, and insufficient for comprehensive understanding.

Key Insights and Challenges:

1. Different disability communities have overlapping needs; isolated solutions fail BLV/DHH users.

2. Unified accessibility is economically and practically more sustainable than maintaining separate pipelines.

3. Existing video accessibility systems suffer from three major gaps:

   • Architectural Integration Gap: No system integrates vision, audio, captions, and sign language into one experience.

   • Robustness Gap: Systems fail under network delays or missing/degraded modalities.

   • Evaluation Gap: Accessibility is rarely tested directly with target users; metrics focus on proxies or general NLP benchmarks.

UMA Framework (Unified Multimodal Accessibility):

• A five-layer architecture integrating visual, audio, caption, and sign language streams for real-time video accessibility.

• Modality Robustness via SSF Adaptation: Dynamically handles missing or degraded modalities with minimal parameter overhead (<0.7%).

• VideoA11y-Unified-40K Dataset: 40,000 videos with synchronized audio descriptions, captions, and sign language glosses compliant with professional accessibility guidelines.

• User-Centered Validation: Evaluated with 347 sighted users, 40 BLV individuals, and 7 professional describers using quantitative (BLEU-4, CIDEr, SPICE) and qualitative accessibility metrics (clarity, accuracy, descriptiveness, objectivity).

• Deployment Framework: Guidelines for cloud/edge integration, WCAG 2.1 AA/AAA compliance, and ADA/AODA standards.

Related Work:

• Audio Descriptions: Automated systems (Tiresias, NarrAD, DistinctAD) generate descriptions but assume audio availability and often ignore accessibility nuances.

• Sign Language Generation: DiffSign, Breaking the Barriers, and BISINDO models produce sign language but remain separate tracks, lacking integration with video understanding.

• Captioning: Existing systems focus on ASR or readability, not multimodal comprehension or BLV/DHH needs.

• Multimodal Learning: Vision-language models (CLIP, BLIP-2, transformers) enable video understanding, but current models are not robust to missing modalities or optimized for accessibility.

UMA Methodology:

• Layer 1: Accepts video, audio, captions, and metadata streams; no modality required.

• Layer 2: Preprocessing extracts keyframes, audio segments, text via OCR, and speech transcriptions.

• Layer 3: Modality-specific encoders for video (ViT), audio (Whisper/BERT + ResNet), and captions (BERT/mBERT).

• Layer 4: Multimodal fusion with SSF adaptation allows graceful degradation when modalities fail.

• Layer 5: Generates synchronized outputs: audio descriptions, captions, and sign language glosses, following professional standards and WCAG/ADA guidelines.

Ethical Principles:

• User-centered design with BLV/DHH input.

• Privacy and secure handling of multimodal data.

• Bias mitigation and transparency in automated decisions.

Evaluation:
UMA was validated with a large dataset, professional guidelines, and direct target user feedback, demonstrating that integrated, robust multimodal accessibility is achievable, addressing gaps in fragmentation, robustness, and evaluation present in prior work.

Conclusion

This paper introduces UMA, the first unified multimodal architecture for accessible video understanding. Through rigorous evaluation with 347 sighted users, 40 BLV individuals, 7 professional describers, and 35 DHH users, we demonstrate: 1) Unified multimodal design significantly outperforms isolated accessibility modalities (41% comprehension gain for BLV users; 30% gain for DHH users; p<0.001). 2) Parameter-efficient robustness maintains 70% comprehension even with 50% modality loss, achieved through scale-shift feature adaptation adding <0.7% parameters. 3) Professional-quality outputs that match or exceed human professional descriptions. 4) Practical deployment feasibility with open-source implementations, standard format compliance, and efficient inference latency. The VideoA11y-Unified-40K dataset, released openly, provides the first comprehensive benchmark for multimodal accessible video understanding.

References

[1] Li, C., Padmanabhuni, S., Cheema, M., Seifi, H., & Fazli, P. (2025). VideoA11y: Method and dataset for accessible video description. arXiv:2502.20480. [2] Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2024). Robust multimodal learning with missing modalities via parameter-efficient adaptation. arXiv:2310.03986v3. [3] Gao, Y., Fischer, L., Lintner, A., & Ebling, S. (2024). Audio description generation in the era of LLMs and VLMs: A review of transferable generative AI technologies. arXiv:2410.08860. [4] Wang, X., Zheng, Y., Zhang, R., Zhang, Y., Zhou, J., Zhou, B., & Liu, Z. (2025). NarrAD: Automatic generation of audio descriptions for movies with rich narrative context. IEEE Transactions on Multimedia. [5] tho Pesch, P., Bouqueau, R., & Montagud, M. (2020). White paper: Recommendations for immersive accessibility services. ImAc Project H2020. [6] Soldan, M., Aradhye, H., Chen, X., Hidary, J., Holness, G., Huang, Z., ... & Malik, J. (2022). DistinctAD: Distinctive audio description generation in contexts. In CVPR 2024. [7] Lin, J., He, C., Zeng, A., Wang, H., Zhang, Y., Yu, J., ... & Ding, X. (2023). Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In EMNLP 2023 Demo Track. [8] Wang, J., Xu, J., Gao, Y., Hu, Q., Jiang, Y., & Chen, Y. (2024). InternVideo2: Scaling foundation models for multimodal video understanding. arXiv:2403.15377v4.

Copyright

Copyright © 2026 Johnson Lathe, Namrata Wagh, Pratiksha Shinde, Supriya Pawar, Asha Gaikar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.