Omnilingual Asr: Advancing Automatic Speech Recognition for 1600 Languages

HF Demo: https://huggingface.co/spaces/facebook/omniasr-transcription...

GitHub: https://github.com/facebookresearch/omnilingual-asr

How hard is it to make TTS out of this? A few independent journalists from Belarus asked for TTS in their language, but I am no expert, was thinking about re-using Mozilla's work. What's the easiest way to get working TTS for a language?

This seems like a massive improvement for openly available local ASR. Even the 300M model outperforms whisper-large-v3 according to the paper's benchmarks.

Only a few gb of weights will recognize speech in 1600+ languages.

Freely downloadable and usable by anyone for almost anything.

We truly live in the future.

Just killed my startup. https://6k.ai

Half joking - hopefully, we can still contribute something to this to this field. Looking forward to doing some tests with this.

First, let me say that this is impressive. And then let me pose some questions:

As a linguist, I would like to know more about the kinds of languages this works well with, or does not work well with. For example, half the world's languages are tone languages, and the way tones work varies greatly among these. Some just have high and low tones, while others are considerably more complicated; Thai has high, mid, low, rising and falling. Also, tone is relative, e.g. a man's high tone might be a woman's low tone. And some African languages have tones whose absolute frequencies vary across an utterance. So transcribing tone is a quite different problem from transcribing phonemes--and yet for many tone languages, the tone is crucial.

There are also rare(r) phonemes, like the clicks in many languages of southern Africa. Of course maybe they've already trained on some of these languages.

The HuggingFace demo says "Supported Languages[:] For this public demo, we've restricted transcription to low-resource languages with error rates below 10%." That's unclear: 10% word error rate, or character/ phoneme error rate? The meta.com page refers to character error rate (CER); a 10% character error rate can imply a much higher word error rate (WER), since most words contain several characters/ phonemes. That said, there are ways to get around that, like using a dictionary to select among different paths through possible character sequences so you only get known words, and adding to that a morphological parser for languages that have lots of affixes (meaning not all the word forms will be in the dictionary--think walk, walks, walked, walking--only the first will be in most dictionaries.)

Enquiring minds want to know!

What I really want to know is how well these could work for non-human languages. No, not aliens, but chimpanzees, dolphins, bonobos. We have hundreds or thousands of hours of recordings.

What would it take to start working on them?

I'm going to test this with Voice AI to see how it works compared to Whisper and Parakeet

https://voice-ai.knowii.net

Swedish

Status: Endangered

"The child-bearing generation can use the language among themselves, but it is seldom being transmitted to children."

What!? A lot must have changed in one generation..

I agree that this is a very exciting and really crucial research and I'm glad there is funding for this. But it's very strange that Hungarian is marked as "highly endangered" at https://aidemos.atmeta.com/omnilingualasr/language-globe Highly endangered is supposed to mean "The language is used by grandparents and older generations; while the parent generation may still understand the language, they typically do not speak it to children or among themselves." Then why is Hungarian marked as such? Obviously not true with 14 million active speakers and being the 20th in terms of the most language resources published on the Internet. Additionally, the feedback mechanism seems also broken ("There was an error submitting your feedback. Please try again.")

the global language explorer is fascinating -great work guys

https://aidemos.atmeta.com/omnilingualasr/language-globe

- we are getting closer to BabelFish.. at least for the Earth!

Does anyone else feel like they buried the lead?

> Omnilingual ASR was designed as a community-driven framework. People around the world can extend Omnilingual ASR to new languages by using just a few of their own samples.

The world just got smaller

> Bring Your Own Language

Few-shot new languages is going to be a game changer for linguists

Unfortunately I don't read anything in the paper about improvements to timing/timestamping. In particular unclean word boundaries are hard with wav2vev2.

And their use of LLMs as part of the transcription process makes it likely that they trained the model to correct mispronounciations by the speaker. This lowers CER because the human transcription often corrects for mispronounciations as well, but reduces the ability of the model to actually transcribe what was said.

any insights on latency?