Automating Immersive Reading

smoores

I took a week of from work recently to reimplement Storyteller's forced alignment algorithm. Storyteller[1] is an open source, self hosted platform for creating, managing, and reading/listening to "readaloud" books — books that have audiobook narration built in and can highlight each sentence (and/or word, with this new algorithm!) as it's read aloud. Forced alignment is the process of determining where each piece of text starts and ends in the audiobook. Anyway, I am really pleased with how the new algorithm turned out! Hopefully someone else finds it interesting as well. [1] https://storyteller-platform.dev

mobeets

This is really impressive! One thing I’ve often wanted is a way to listen to an ebook (say, while cooking), and then later be able to sit down and read from where I left off. This essentially requires the same functionality that Storyteller offers (syncing progress through an ebook and audiobook), though arguably overkill for this use case (since I could handle a more coarse alignment). In any case, is this something Storyteller would be good for? Or are there other tools for the job?

usern20260720

I generated an immersive reading for Athenaze in a few afternoons with Claude

cainxinth

This is how I proofread. I play a screen reader while I read my work. The brain does some autocorrecting that you don’t even notice and occasionally skips over a typo when you read it without the audio.

thelisu

This highlights the core reason why I don't like audiobooks, podcasts, or even movies, compared to reading: it's too slow. Looking at the example, I read two or three times as fast as the audio goes. What I would need is the opposite: some eye-tracking software that plays the audio at the point I am reading. I don't know if it would be any useful though, as it would be very sped-up, but maybe it would increase "immersion" nevertheless.

lstodd

If that's about reading why do I need to zoom to 140 in FF to just begin reading?

nottorp

I thought reading becomes immersive if you like the book...

orphereus

What is the purpose of this outside people with disabilities? Why do you need word highlighting, it's like those TikToks and IG reels where they show one word at a time to keep you hooked. Surely, if you need this to read a book (again, besides any disabilities), you should stop your current habits.

SamBam

I'm curious whether individual-word highlighting is best for people with reading disabilities, or a broad, whole-sentence highlighting might actually be preferable. When I'm reading my eye is jumping back and forth across the sentence. The demo made this abundantly clear -- it was actually very hard to just track the highlighted word. Just using the fact that I consider myself a very good reader as an example, I wonder whether there are advantages to the eye being free to skip back and forth across a sentence. I assume it might be as simple as a user-preference toggle, since you're already highlighting the sentence. You could also try UIs where the highlighting was more "spread out," focused on the word being said but bleeding into the other words on either side (maybe up until the hard stop of the sentence boundary) to lead the eye toward it. I'm imagining something like the red-blue text color of the BeeLine editor.

justinhunt

This is very interesting. I do something related: matching student oral reading transcripts back against the text they were assigned to read.Students will misread then re-read a sentence, skip sentences, skip paragraphs etc. I think its similar because we both seek to assign audio time stamps to sentences and words. But I wonder why your forced alignment algorithm is so heavy duty. (My head started to spin at CTC emissions. ) Probably yours is just way more thorough than mine, My simplistic approach would have been to transcribe the audio. And then run a differencing script chapter by chapter matching the book text with the audio transcript. And then do something similar intra chapter to get sentence and word level time stamps.