Dessa: Conversation Model trained only on speech

amitport

not at all creepy

weego

Sorry I'm lost. The sample on the homepage sounds absolutely awful. Not one bit of the cadence individually and together, the tonal ranges in responses or the overall structure and feel is comfortable to listen to, it's so far off passing as 'real' that I feel like I must be missing something.

IshKebab

I'm surprised it took this long! Clearly the next step is joint training on text and audio. The speech realism is very good, but the intelligence is pretty clearly 2024-level.

opem

crazy, if true!

paxy

These bidi/duplex demos always come off as overly artificial to me. It's cool that the model can speak at the same time as you instead of taking turns, but training it to interrupt as much as possible isn't the way to do it. In the very first example, did anyone here find the laugh after the "winter" answer to be convincing? What was the need for it? Why did the human speaker not get to finish a single sentence or finish a thought before the model took over with a monologue?

moonu

Can you instruction-train a duplex model like this? How would that look? I also wonder, if a model trained only on speech can develop world knowledge like this, could you train an image model only on images of text to essentially act as an LLM?

actualwitch

The speech is great, but you can see some cracks even in the first sound bit. near the end he asks the model to speak louder and initially the voice is louder but then goes back to the same level, also losing what she was saying just seconds ago and he has to lead back to the topic.

ttul

These full-duplex models are getting very "Her"-ish lately. gpt-live-1 is fantastic. Dessa, though, seems to be aiming at full human parity. There's a huge market for AI girlfriends that they should be looking into in earnest...

63

Both speakers in the examples sound unnatural. The human doesn't know what to talk about and seems to have little confidence. The robot says things that only mostly make sense and draws on, saying ultimately very little. "How's the winter stuff?" e.g. is not a sentence a person would say. I might say "How are you dealing with the snow?" or "Are you looking forward to winter?," but not "How's the winter stuff?" And there's just so many examples like this. Also, I'm not convinced that this "human-like" conversation with filler words and interruptions and pauses is actually what anyone wants, especially in absence of a face to interpret or a real human being to understand behind it. Even if it worked perfectly, it would feel uncanny and artificial because I know the person isn't real.

ricardobeat

> we figured out how to train Dessa on only one NVIDIA GB200 NVL72. This is the crazy part. It seems pretty smart, and LLMs require 1000x more compute to train. Can this kind of efficiency somehow be achieved with text too?

bityard

Not bad, but I guess I lack the imagination to envision the use cases for such casual conversation. Other than... I suppose a certain demographic would be lining up to use this as the voice for their AI girlfriend. Also I seem to be missing where/how they are selling this.

haarts

Ok. Sounds cool. Now what? Open source weights? An API somewhere? Available on Openrouter? Are they just testing? Where is this going?

ByteAtATime

> LLMs need an insane amount of text. Dessa learned language from hundreds of times fewer words (in fairness, LLMs also have to learn C++ and whatever the hell Angular is). Now that a larger proportion of text is AI, I'm appreciating human-written text more and more; this got a good chuckle out of me.

delegate

I actually thought it sounds really good and natural and it freaked me out in a good way. You can tell that the robots have no idea of what they're talking about, but the voicing, the emotions, etc, really good.. Like talking to a friend on the phone. Also creepy, because of the potential for abuse of this technology. But the interesting part to me is that training on voice only is what allows models to learn emotions - voice transmits a lot of emotional information which text doesn't capture. Language has words for a very limited number of emotions - maybe only the peaks - but there's so many more which we can't describe or transmit via text. You could determine the phonetic tokens which express different kinds of emotions or 'render' text+emotion using a DSL, etc. Exciting stuff.

sgarrity

Now that a robot can speak over people on Zoom calls for me, I’m finally free to write poems all day!

hypfer

Interesting tech, but if these are the hand-picked examples that best represent it, I suppose we're not quite there yet. The intelligence sounds lacking. At this point, anyway. But the idea of learning through voice instead of written text might prove fruitful. After all, text isn't strictly native to humans. It's just a tool we've invented.

ajam1507

This feels like a parlor trick where the AI was trained to mimic informal telephone mannerisms. The actual conversations were pretty bad. Lots of style, not much substance basically.

bushido

I was expecting to run into some uncanny valley-esque. It was surprisingly good. It is kinda scary that we're here already, I can see this being pretty problematic for seniors/public prone to getting scammed.

lrvick

Open weights or it didn't happen.

SwellJoe

So, I tolerate the "ums" and "ahs" and "uhs" in conversation from humans because they're human. If an AI is going to speak to me (gods forbid), it better be concise and clear and focused, or I'm throwing it out the fucking window. It is interesting that they've done this reasonably successfully, impressive technology. Novel ideas can have inherent value, absent direct uses they may have, as they can be a stepping stone toward other inventions. But, I'm having a hard time coming up with a use case that isn't predatory or intentionally misleading. Company support agents pretending to be human, for instance. Or, taking the next step on AI girlfriend/boyfriend type interactions. Or, outright scams. Models already do an excellent job with voice cloning, at least for intonation and timbre, adding realistic conversational tics makes it even more likely grandma is going to fall for it when her "nephew" calls needing help. But, that seems to be extremely common with new tech these days. There's always an angle that is unsavory. Marketing, scams, propaganda, surveillance, killing, that's where the money is, so everything bends toward them.