We turned EmbeddingGemma 2 into a speech-to-text model

martianvoid

Pretty cool application of an embedding model

chronogram

What about the quality of the transcription?

mikemzz1

interesting approach. But training ctc head on top of gemma's audio encoder will most likely produce better results and faster inference

alberth

As someone who knows nothing on how this works, does this mean they have to pay the audio at 200x playback speed to transcribe this fast. Or is there another approach?

Imustaskforhelp

This has come at just the right time. Does this work for Hindi or other non English languages though? I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase. I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it out! I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible. [0]: https://huggingface.co/parismitaglobalsolutions/indicconform...

fdb

Is it open source? They mention something about the Apache License but I can't seem to find a link to GitHub or something.

jdw64

It would be nice if Korean also worked. I’d like to experiment with a Korean model too.

llimllib

Apple's on-device transcription model is about 3x slower than their reported results on my more powerful computer, with a similar error rate in my test (which I consider pretty good results) I'll test theirs too if I get around to it Here's a sample CLI wrapping the apple model and providing a simple benchmark: https://github.com/llimllib/transcribe update: doesn't seem like they provide their model and I don't feel like trying to dig it out of their app bundle, so I'm not going to test this one

vladimirralev

Any info on the param size and quantization of each part and in total?

lukko

Interesting... is it overkill to use a multimodal model like this on just audio? Could the model size be decreased dramatically, or does the multimodality help the accuracy and speed in some way?

spiderfarmer

So Marco Arment can sell some more of his Mac Minis.

amelius

Sounds great if you could use it as a server. Otherwise, why would you want this?

bawana

I wonder if this approach can be used to transcribe the soounds made by dolphins and whales

bigtex

I tested a 33 minute local audio only file on a M4 Mac Mini with 24GB of Ram and it took 6.75 seconds to transcribe it using this software. Using MacWhisper/Parakeet3 it was about 7.5 secs using a stopwatch.

yaanfpv

I think this seems to target English only I believe. Pretty good but thinking os using some of these techniques on our multi-lingual ASR models !

Shershah1024

Hi all. I am Shahir. This was my experiment. First of all - Thanks to whoever posted this on HN. It came as a surprise. Just to clarify - This started as just an experiment to see if we can use the fact that text and audio share a vector space in Embedding Gemma 2 to convert audio to text. Had to hack around a few things, but, it worked. Speed was just a side benefit. Was not aiming for it. The accuracy is good, but not in the range of cloud models, but decent enough for everyday use, and this is just a couple of days work. The WER can improve a lot with more training on varied data. Will try that next. I will opensource the whole thing soon. Might need a few days though. Also, releasing the TTS version for this soon. Happy to answer any questions you have.

samuelbalogh

Fantastic! Any plans to support other languages?

saaaaaam

I recently swapped to using Parakeet v3 on my old M1 Macbook Air, using MacWhisper. It was so fast I thought it had failed. As in, by the time I clicked "transcribe" and looked at my screen again a few seconds later the transcript had finished. Absolutely insanely fast.

sbdaman

Maybe it's just a COVID time blending together effect but it still blows my mind how quickly we went from "automatic audio transcriptions are not even worth having on" to "an hour of audio transcribed in 13s locally." Pre-pandemic it seemed like even the very best in class auto-transcription features were just unusably bad. I guess that was like 7 years years ago though.

whstl

For people interested in super quick models: "One cool trick" I like for running those kinds of models very quickly on macOS is converting them to CoreML so they run in the Neural Engine. I had amazing success with Spleeter, Demucs, MVSEP-MDX23-music-separation, DeepFilterNet, Whisper, Hunyuan3D for 3D mesh generation. Plus a lot of others for both work and leisure. I also convert the code to Swift without dependencies, but that's a personal preference. The runtime on some of them went sometimes from 60s to 10s. And since it's just Swift, I can just give them to non-technical co-workers and they don't have to setup a whole Python environment. If you don't wanna do this by hand, this is something that existing LLM models are good at. Even Luna can do it without much fuss.