SenseVoice

Open source alternative to Deepgram, AssemblyAI and Hume AI

9.6K stars
MIT
Active this month
Last commit 30 days ago

Last verified Oct 31, 2026

Transform audio into transcribed text with emotion tags across five languages; SenseVoiceSmall runs 15x faster than Whisper-Large and is MIT licensed.

Visit website GitHub9.6K

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who SenseVoice is for#

Teams transcribing Chinese or Cantonese media at scale

SenseVoice-Small outperforms Whisper-Large on AISHELL-1, AISHELL-2, and Wenetspeech, making it the better starting point for Mandarin or Cantonese transcription pipelines. Self-hosting avoids per-minute fees that compound quickly on hours of recorded content.

Skip if:

Your audio is primarily English or a Western European language where Whisper-Large or a managed API already meets your accuracy bar. SenseVoice's Chinese and Cantonese advantage matters most in those specific languages.

AI developers building speech datasets with emotion labels

The combined transcription and emotion tag output annotates a corpus in one pass instead of requiring a second API call. At 15x the throughput of Whisper-Large in benchmarks, it processes large datasets without proportional GPU time costs.

Skip if:

Your dataset requires fine-grained emotion granularity beyond the tag set SenseVoice outputs, or you need speaker-attributed emotion per diarized segment. The emotion model is utterance-level, not sentence-level or speaker-level.

Developers replacing a managed ASR API with self-hosted inference

The FastAPI server wraps SenseVoice as an OpenAI-compatible endpoint, making migration from managed API clients straightforward. MIT license allows commercial use, and inference cost is fixed at server hardware rather than per audio minute.

Skip if:

You need managed scalability with zero infrastructure overhead. SenseVoice self-hosting requires a GPU server for full throughput; the llama.cpp CPU path is slower than GPU-accelerated managed APIs at comparable quality.

Edge and IoT engineers needing CPU-only speech recognition

The llama.cpp/GGUF binary runs SenseVoice with built-in VAD as a single executable on CPU hardware, with no Python or CUDA requirement. Prebuilt binaries are available from the repository's Releases page for deployment without a build environment.

Skip if:

Your edge device lacks the memory or CPU headroom for the GGUF model. The binary targets modern laptop-class CPUs and is not optimized for microcontroller-class hardware.

the problem

The problem it solves#

Managed speech-to-text APIs charge per audio minute, which compounds quickly at production scale. They also return only transcription text; getting emotion labels or audio event tags requires a separate API call, separate billing, and stitching results from two different services. For teams working with Chinese, Cantonese, or mixed-language audio, accuracy across commercial providers varies significantly, and none expose model weights for finetuning on domain vocabulary such as medical terms, company names, or technical jargon.

Self-hosting an open model has historically meant accepting Whisper's latency tradeoff: Whisper-Large is accurate but slow, and Whisper-Small is faster but loses accuracy on Chinese and Cantonese. Building a production pipeline around either requires composing a separate voice activity detection model and a separate emotion classifier and stitching their outputs together before any result is usable.

how SenseVoice solves it

How it solves it#

Five-language ASR with Chinese and Cantonese accuracy

SenseVoiceSmall transcribes Mandarin, Cantonese, English, Japanese, and Korean in a single checkpoint. On AISHELL-1, AISHELL-2, and Wenetspeech benchmarks, the Small model outperforms Whisper-Large on Chinese recognition. Language is detected automatically unless you specify a language code, so mixed-language files do not require preprocessing.

Speech emotion recognition without finetuning

Outputs an emotion tag per utterance as part of the transcription pass, covering neutral, happy, sad, angry, and related classes. In benchmark comparisons reported in the README, SenseVoice-Small surpassed other open source emotion recognition models on the majority of test sets without any target-domain finetuning. SenseVoice-Large reached the top result on nearly all evaluated datasets.

Audio event detection for common sounds

Tags recognized sound events including background music, applause, laughter, crying, coughing, and sneezing alongside the transcription output. Trained primarily on speech data rather than environmental audio, so event classification accuracy has gaps compared to dedicated audio event detection models like BEATS and PANN on the ESC-50 benchmark.

Non-autoregressive architecture for low-latency inference

SenseVoiceSmall uses an end-to-end non-autoregressive encoder that benchmarks at more than 15 times the throughput of Whisper-Large and more than 5 times that of Whisper-Small at similar parameter counts. Long recordings pass through FSMN-VAD segmentation first; the bounded-window script long_audio_no_vad.py handles hour-scale files on fixed GPU memory.

CPU and edge deployment via llama.cpp and GGUF

A prebuilt binary derived from llama.cpp runs SenseVoice with built-in FSMN-VAD as a single self-contained executable, with no Python or CUDA dependency. Download the GGUF model from Hugging Face and point the binary at any WAV file. Prebuilt binaries are available from the repository's Releases page.

Speaker diarization via FunASR pipeline composition

Composing SenseVoiceSmall with CAM++, FSMN-VAD, and a punctuation model through FunASR produces per-sentence output with anonymous speaker labels and timestamps. The diarization output comes from the composed pipeline, not the SenseVoiceSmall checkpoint itself; it requires FunASR 1.4.15 or newer and the local model.py file from this repository.

strengths · trade-offs

Strengths and trade-offs#

Strengths

MIT license with full weight access for finetuningThe MIT license applies to the model weights, not just the code. You can run SenseVoice commercially on your own infrastructure without licensing fees and finetune the checkpoint on proprietary data. Finetuning scripts are included in the repository for adapting to domain-specific vocabulary or improving accuracy on long-tail samples.
Chinese and Cantonese accuracy advantage over WhisperOn the AISHELL-1, AISHELL-2, and Wenetspeech benchmarks, SenseVoice-Small outperforms Whisper-Large on Chinese recognition. This is the primary accuracy case for choosing SenseVoice over Whisper: the Chinese and Cantonese gap is measurable at the Small model tier, while English accuracy is competitive.
Single model pass for transcription, emotion, and audio eventsOne model.generate() call returns the transcript, language tag, emotion tag, and audio event tags together. Managed ASR APIs charge separately for emotion analysis or do not offer it at all. The unified output reduces pipeline complexity and eliminates the latency of chained API calls.
FastAPI server with OpenAI-compatible endpointThe FastAPI service wraps SenseVoice on port 50000 with multi-concurrent request support and client examples in Python, C++, HTML, Java, and C#. This makes it a drop-in self-hosted replacement for managed ASR API endpoints without rewriting client code.

Trade-offs

-Audio event detection lags specialized AED modelsThe README explicitly states that due to limitations in training data and methodology, SenseVoice's event classification performance has gaps compared to BEATS and PANN on the ESC-50 benchmark. If your primary use case is environmental audio tagging rather than speech, a dedicated audio event detection model will outperform SenseVoice on that task.
-GPU memory scales with audio length without VADPassing long recordings directly to model.generate() causes encoder memory to grow with file duration, leading to out-of-memory errors on typical GPU sizes. The recommended workaround is FSMN-VAD segmentation before encoding, or the bounded-window long_audio_no_vad.py script for hour-scale files. This adds pipeline complexity that managed APIs handle transparently.
-Docker prebuilt image is currently privateThe GitHub Container Registry package at ghcr.io/qwenaudio/sensevoice is currently private; anonymous pulls return HTTP 401. Self-hosters must build the Docker image locally from the repository source. The llama.cpp binary path avoids this constraint for CPU deployments.
-Diarization composition is not validated for accuracyThe speaker diarization pipeline composed with CAM++ is documented as tested on a fixed public sample, not validated for diarization accuracy on real recordings. The README explicitly notes this. Teams that need reliable speaker attribution should treat this path as experimental and evaluate it against their own data.

versus alternatives

SenseVoice vs alternatives#

SenseVoice vs AssemblyAI

AssemblyAI is a managed speech-to-text API used in production across transcription, meeting intelligence, and voice analytics products. Both handle multilingual transcription, but they differ on hosting model, pricing structure, and what a single API call returns.

Feature	SenseVoice	AssemblyAI
License	MIT	Proprietary
Self-hosting	Yes (GPU or CPU)	No
Emotion detection	Built-in, no add-on	Paid tier add-on
Audio event tags	Built-in	Not available
Pricing	Free (self-hosted)	Per audio minute
Language support (released weights)	5 languages	100+ languages
Model finetuning	Yes (MIT weights)	No

SenseVoice is the better choice when you need self-hosted inference for data privacy, want emotion and audio event tags in a single call, or process enough audio that per-minute billing is a material cost. Finetuning on domain vocabulary is only possible with SenseVoice, since AssemblyAI does not expose model weights.

AssemblyAI is the stronger fit when you need broader language coverage beyond the five in SenseVoiceSmall, require an enterprise SLA with managed reliability, or need speaker diarization that has been validated at production scale. AssemblyAI's conversational AI features on top of transcripts are also outside SenseVoice's scope.

SenseVoice vs Deepgram

Deeogram is a managed ASR API offering low-latency transcription with its Nova model line. Compared to Deepgram, SenseVoice's primary differentiator is the built-in emotion recognition and audio event detection, which Deepgram does not include in its transcription response. Deepgram's managed service covers far more languages than the five in SenseVoiceSmall, and its Nova-2 model is competitive on English accuracy with no infrastructure overhead.

SenseVoice is the better pick for East Asian language accuracy, self-hosting, and unified emotion output in a single pass. Deepgram remains the stronger option for teams that need managed reliability across many languages, or for English-primary workloads where accuracy and latency are the only decision criteria.

install · quick start

Quick start#

bash

Install the FunASR library via pip to run SenseVoiceSmall in Python.
```bash
pip install "funasr==1.4.14"
```

For CPU or edge deployment without Python, download the prebuilt llama.cpp binary and GGUF model:
```bash
bash runtime/llama.cpp/download-funasr-model.sh sensevoice ./gguf
llama-funasr-sensevoice -m ./gguf/sensevoice-small-f16.gguf --vad ./gguf/fsmn-vad.gguf -a audio.wav
```

tech stack · detected from GitHub

What it's built on#

Languages: CC++Python
Frameworks: FastAPIPyTorch

frequently asked

FAQ#

Does SenseVoice support languages beyond the five in SenseVoiceSmall?

The released SenseVoiceSmall checkpoint supports Mandarin, Cantonese, English, Japanese, and Korean. The broader SenseVoice research paper reports training on more than 50 languages and 400,000 hours of data, but those weights are not in the public release. The checkpoint outputs a nospeech tag for audio that contains no recognizable speech in the supported languages.

Can I finetune SenseVoice on my own vocabulary or domain data?

Yes. The MIT license covers the model weights, and finetuning scripts are included in the repository. Custom finetuning requires cloning the FunASR repository and installing it in development mode. The README points to FunASR's finetuning documentation for the full procedure, which covers data preparation, training configuration, and long-tail sample strategies.

How does SenseVoice compare to Whisper on English transcription accuracy?

The benchmarks in the README focus on Chinese and Cantonese datasets where SenseVoice has a measurable advantage over Whisper-Large. On English datasets such as LibriSpeech and Common Voice, SenseVoice is competitive with Whisper at similar parameter counts. For English-primary workloads, the main differentiator is speed and the built-in emotion and audio event tags rather than an accuracy gap.

Is there a prebuilt Docker image available without building from source?

The ghcr.io/qwenaudio/sensevoice container package is currently private and returns HTTP 401 on anonymous pulls. You must build the image locally using the Dockerfile in the repository root with docker build -t sensevoice. The llama.cpp binary from the Releases page is the easiest zero-build deployment path for CPU use.

What is the difference between SenseVoiceSmall and SenseVoice-Large?

SenseVoiceSmall is the publicly released checkpoint, optimized for low-latency inference across the five supported languages. SenseVoice-Large appears in the emotion recognition benchmarks in the README, where it outperforms SenseVoice-Small and achieves the best result on nearly all evaluated datasets. The Large checkpoint weights are not available in the public model zoo at this writing.

also worth a look

Similar open-source tools#

Pyannote Audio

Alternative toAssemblyAI

Speaker diarization library for Python, pretrained PyTorch pipelines.

10,731 GitHub stars
Licence MIT

RealtimeSTT

Alternative toDeepgram

Real-time speech-to-text library with VAD and wake words

10,212 GitHub stars
Licence MIT

whisper-asr-webservice

Alternative toOpenAI Whisper API

Self-hosted speech recognition API built on OpenAI Whisper

3,367 GitHub stars
Licence MIT

REA (Reverse Engineer Anything)

Featured

Alternative toRevEng.AI

AI-powered reverse engineering for binaries, apps, and assemblies.

407,266 GitHub stars
Licence MIT

Omnara

Featured

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

3,325 GitHub stars
Licence Apache-2.0

God's Eye View

Featured

Alternative toFlightradar24

Real-time global intelligence on a 3D globe

61,172 GitHub stars
Licence MIT with third-party data restrictions

by category

Open source tools in these categories#

436

AI & Machine Learning

freeCodeCamp, REA (Reverse Engineer Anything), OpenClaw

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

12,855 GitHub stars
Licence AGPL-3.0
#Email

Omnara

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

3,325 GitHub stars
Licence Apache-2.0
#AI Agents

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

6,242 GitHub stars
Licence Apache-2.0
#AI Agents

Advertise your product

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

SenseVoice vs AssemblyAI

Feature	SenseVoice	AssemblyAI
License	MIT	Proprietary
Self-hosting	Yes (GPU or CPU)	No
Emotion detection	Built-in, no add-on	Paid tier add-on
Audio event tags	Built-in	Not available
Pricing	Free (self-hosted)	Per audio minute
Language support (released weights)	5 languages	100+ languages
Model finetuning	Yes (MIT weights)	No