Spectra

Overview

Multimodal sentiment and harmful-speech analysis for spoken video in Urdu, Punjabi, Roman Urdu, Hindi and English (Python, FastAPI, 10 ML models).

Problem

Fact-checkers, newsrooms and brand-safety teams need to know what is actually said in a video — and how — before they publish, quote or advertise next to it. Watching hours of footage is slow, and most moderation tools read English text only, missing Urdu and Roman Urdu speech, tone of voice, and what is on screen.

Architecture

A video goes through ffmpeg into audio and frames. Whisper produces a timed transcript and an English translation; ECAPA speaker embeddings and clustering split it at every change of speaker. Each sentence is scored by text models (sentiment, emotions, toxicity, hate speech, sarcasm and a DeBERTa zero-shot model for 50+ categories), while wav2vec2 and prosody features read the voice, a ViT model reads facial expression, and CLIP checks a frame every second for disturbing imagery. A fusion step combines them into per-speaker and overall risk scores, and the FastAPI backend runs it all as a background job behind a dashboard.

Technical Decisions

• Two licence profiles: a research profile with every model, and a commercial profile that only loads commercially licensed models, so client work never depends on a non-commercial licence.
• An Urdu, Punjabi and Roman Urdu abuse lexicon on top of the translated text, because translation softens or loses many local slurs.
• Customer reports are shared through a separate Cloudflare Worker that serves a read-only view from private R2 storage, with expiring unguessable links, an agree-to-terms gate, an access log and a recipient-name watermark, so the analysis app itself is never exposed.
• Clear frames of disturbing content are never stored: reports keep only heavily blurred thumbnails.

Challenges

The hard part was avoiding false flags. A sentence that quotes or condemns hate is not hate, so Spectra detects counter-speech and quoted speech and lets a reviewer correct the attribution. Hate categories must be backed by a general hate signal and mention a target group, or their score is reduced. Visual flags must hold across two neighbouring frames, because single-frame spikes are usually look-alikes, such as a microphone read as a gun. Criticism of institutions is shown as informational rather than harmful.

What I Learned

Machine-learning scores are evidence, not verdicts. Most of the value came from making every flag explainable and checkable — linked to the exact second, with the words, voice and face side by side — and from the guard rails that stop a confident-looking number from being wrong. 114 automated tests with stand-in models made it possible to change the pipeline without downloading gigabytes of models every time.

Links

• Send a video for analysis →

Need something similar?