Desert Ant Labs: local, fast models that run on device

sipjca

at first i got very excited about a new fast transcription model (voz) but turns out its just parakeet v3 with some new inference code which is macOS/iOS specific

nullbio

This is a cool idea. The most useful one for me would be something that can process pdf files into a json schema. Title and tag generation from a post would also be useful. I'm interested in web app though.

ashenke

A lot of the models would be useful in a web context, to improve on the CMS we're making for clients. But they look like most of them are iOS only, few have a node package or something other, and all the benchmark are running it on modern iPhones so I doubt it would be that fast on a 20$ VPS.

library8848

Shiny layer of marketing and proprietary code on top of open models? Voz is Parakeet 0.6B v3 Clear is DeepFilterNet 3 Ear is the language predictor from whisper-tiny ...

illright

I wonder why they only support Apple platforms, citing CoreML. Doesn't Android have a similar framework, ML Kit?

1dom

This is a cool way of approaching AI models. I'm a big fan of local LLMs, local specific models like this should be even more powerful. > Every model is free up to 100k monthly active devices. No tokens, no logins. I dunno about the business model though. Cloud LLM billing makes sense: you're getting another computer to do work with each request, and using their compute via their gateway that they bill you. These local models are like old school software. They're producing the weights, and then giving them to people. If I'm happy with the weights you've given me, and I'm not using your compute for inference and not wanting or needing any updates off you, why should you continue getting money off me and my customers? The whole "but we need to keep it updated for your security" doesn't really work as well for software designed to run fully offline like these local models are. I'm not saying they shouldn't get paid, but I guess I feel a personal sadness that it's less obvious how to successfully monetise such a sincerely useful and beneficial approach towards AI models.

mtlynch

I love this idea and hope to see more on-device models. How do they make money, though? I tried out their demo for Clear, the audio quality improvement model.[0] I'm not sure if it's just I don't have refined enough an ear or their demo is broken, but the "raw" and "enhanced" versions sounded exactly the same to me. [0] https://desertant.com/models/clear/

bronlund

The website looks amazing.

markdog12

> opinionated on-device intelligence > Hate speech triage. On-device moderation that flags hateful, abusive and threatening text What could go wrong here?

nater5000

I definitely think there's a lot to be done with small models dedicated to specific tasks. I've always thought the REAL value is in having large models be able to easily build small models for custom tasks (which I know is kind of a thing), but perhaps just providing the small models directly is the more accessible approach. >accessible via one SDK for Swift, Kotlin, and JavaScript Lol well let me know when there's a Python SDK and I'll give it a try then. Obviously this isn't a deal breaker if you have a real case, but as someone who is willing to spin something up and try it out if there's a quick "pip install" command, this is getting put back on the shelf for now.

lukevp

I would love to use Voz and Ear, but I’d need a version that is competitive with other audio transcription LLMs for platform availability - meaning macOS, Windows and Linux, and supporting GPUs if available.

faangguyindia

Cool! is there a local model for LLM command approval?

init0

Awaiting web version...

viccis

I wish we could pop a tiny one into my phone so that when I type "Will see you" and swipe the word "later" it chooses that instead of "lasso"

agcat

Honestly this is a really cool idea and i am glad that there are companies being built in this space. This is closest to the vision of what i want to do next.

ricardobeat

> Ranks a transcript's best non-overlapping moments: each clip gets scored and ranked. Build strong selections or unique editing features to pick the best sentences in video or audio recordings. This is very impressive for a 248MB model. I wonder how good the results are, as an LLM 10x the size is still quite bad at that.

Dwedit

Testing out Tongue, it detected "馬鹿外人" as Chinese.

anigbrowl

I like the concept and the development choices seem sensible even if they're not my favorites (although I think missing Python is a mistake). The text feels very LLM generated though, and I reflexively discount the value of anything presented with this writing style.

momojo

> The world ships more than a billion capable phones, tablets, and laptops a year, most with a chip built for exactly this work, paid for and idle most of the day. Run the model there and the economics flip: no per-call cost, no round-trip, and nothing leaves the device. This. I run small models (>50MB) for bio-imaging/biotech applications, it feels like every README implies that you need a discrete GPU to get started. While some do, many, especially the most useful ones, do not. Sure it matters if you're also going to do fine-tuning, but I believe your typical user just wants to detect some nuclei and get some cell-body ratios. The laptop on your desk won't be running Meta's SAM, but it has more than enough compute to crunch 100's of your H&E slides overnight.

shelled

I recently had to use dictation for a few weeks and I was pleasantly surprised that many of the apps (in use/vogue) did support models on my 2021 16GB M1 Pro mac (many of those even supported connecting to a remote or local model endpoint) and at the same time for any worthwhile STT enhancement the model size was hitting higher I would have comfortably wanted. Even though I don't necessarily need dictation any more I intend to keep a custom fully offline setup and try these models (not sure they support live/streaming STT). If any of you are interested there are apps like https://github.com/altic-dev/FluidVoice (this one's a great app) and this https://sam-pop.github.io/WhisperDictation . The latter, even though it has just 7 stars right now, seems to be more "intuitive". I just hope they expose a way to "connect" to available models on the machine or remotely)