How I Used On-Device AI to Build a Better Audiobook App

A long time ago, almost 20 years ago, my wife and I were doing a coast-to-coast road trip across the US. Lots of driving.

At some point we stopped at a Barnes & Noble and bought a 6-CD audiobook set. Yes, CDs. Like animals.

I think it was a Dan Brown book, but I’m not 100% sure. What I do remember very clearly is that we both fell in love with the medium.

There was something magical about it. We were driving for hours, watching the country move around us, and instead of just passing the time, we were inside a story together.

Since then, audiobooks became a big part of my life.

The medium grew too. Audiobooks became more popular, podcasts exploded, Blinkist became a thing, and the production quality around spoken-word content kept getting better. Today you have things like full-cast recordings of Harry Potter, and companies like Soundbooth Theater producing audiobook experiences that feel much closer to audio movies than “someone reading into a microphone.”

My favorite recent example is Dungeon Crawler Carl — an incredible production from Soundbooth, I highly encourage you all to check them out.

The medium got better.

But the apps?

Not as much.

Audiobook apps still mostly press play

Over the years, I mostly used publisher apps.

Audible for English books. IVRIT for books in Hebrew, my native language.

They work. I don’t want to pretend they are broken or useless. They play the book. They remember where you were. They have chapters. They let you change speed.

But after years of listening, I started feeling like the app was not really helping me listen.

It was just playing audio.

And for me, that was not enough.

The real problem: my brain

My main problem with audiobooks is not the audio quality, or the speed controls, or the import flow.

My main problem is that I get distracted.

Sometimes I’m walking. Sometimes I’m driving. Sometimes I’m doing dishes. Sometimes I’m listening before bed and slowly disappearing into that weird half-awake state where I’m technically still listening, but my brain has already left the building.

And then suddenly I realize:

Wait, who is this person? Why are they angry? Did someone die? Am I supposed to know what’s going on here?

This happens in regular books, but it gets much worse in complicated audiobooks.

When I listened to A Song of Ice and Fire, especially the later books, it was close to impossible to keep track of everything and everyone.

More recently, I’ve been listening to the Dungeon Crawler Carl series, which is amazing, but also has a lot of characters, places, running jokes, factions, and context to keep in your head.

I love audiobooks, but I don’t love the feeling that I’m getting a lower-resolution version of the book just because I’m listening instead of reading.

That became the core idea:

You should not have to trade off your understanding of the book just because you are listening to it.

The first idea was probably too weird

My first thought was not actually “let’s build recaps.”

My first thought was games.

I noticed that when I listen, I often need something else to occupy the remaining 20% of my attention. Not something complicated — if the thing takes too much attention, I lose the book. But something simple enough to keep my eyes and hands busy, while the story takes the main part of my brain.

So I thought: what if the game was relevant to the book?

Imagine you are listening to Harry Potter, and the app creates a simple Harry Potter game for you to play while listening. Not a full game. Not Hogwarts Legacy. Just something small, simple, and thematically connected, so your “distraction” keeps you closer to the story instead of pulling you away from it.

I still think this is interesting.

Unfortunately, it turned out to be much harder than I originally thought. Maybe in a future version 🤞.

The practical first solution: “What did I miss?”

Once I admitted that the book-aware mini-game idea was a little too ambitious for a first version, I asked myself a simpler question:

What is the actual pain I’m trying to reduce?

The answer was obvious. I lose track. Constantly and consistently, I find myself confused about something because I was distracted by something that happened in the real world.

So I built a feature called “What did I miss?”

It’s a quick, on-the-fly recap for when you lose track, or when you are jumping back into a book and need context.

Instead of randomly scrubbing backward and hoping you land in the right place, Bloox gives you the context you might be missing so that you have everything you need to enjoy the listening experience.

And because this is an audiobook player, I didn’t want this to be a big wall of text. If you are listening, you probably don’t want to stop and read.

So Bloox speaks the recap to you.

That sounds like a small difference, but it matters. The interface should match the medium.

So I built Bloox

Bloox is my smart, AI audiobook player.

I’ll admit I feel slightly allergic to anything that sounds too much like “AI-powered revolutionary paradigm shift,” so let’s keep it grounded.

The goal is simple: help people listen to audiobooks without constantly losing context.

Today, Bloox does a few things I always wanted from an audiobook app:

It gives you spoken recaps

There are recaps for past listening sessions, and there are on-the-fly recaps for when you get lost. This is the feature I originally built around, and it still feels like the clearest expression of the app.

You are listening. You lose the thread. You ask what you missed. The app helps you get back in.

It has a transcription view I’m weirdly proud of

The transcription view is probably my personal pride and joy. I spent a lot of time making it useful and pretty. It is one of those things that is hard to explain until you use it, but it really changes how the app feels.

You can follow along with the words while listening, and the app can highlight where you are. It makes listening feel more anchored.

It also helps with that “am I missing out by not reading?” feeling, because suddenly listening is not a lesser version of reading. It is its own thing, with its own interface.

It helps with characters

Bloox extracts characters and can highlight them in context while you listen. It can also show you who appears in a chapter.

This is especially helpful for complicated books. The kind where a character shows up after 11 hours of audio and you think:

I know this name. I think I care about this name. I have absolutely no idea why.

I wanted the app to help with that.

Amazon Prime Video has had things like X-Ray for years. Audiobooks deserve that kind of help too.

It handles interruptions better

The world is very inconsiderate toward audiobook listeners.

People call you. Notifications interrupt you. Your kid asks you something. Your dog does something suspiciously quiet in the other room.

Then the book resumes, technically in the right place, but your brain is not in the right place.

Bloox tries to compensate for that with smart rewind, so when the world interrupts you, the app helps you recover context instead of pretending nothing happened.

It has a sleep timer I actually like

Listening before bed is one of my major use cases, so I spent a lot (too much?) of time on the sleep timer.

A sleep timer sounds like table stakes, and maybe it is, but it is also one of those features where the details matter because you use it when you are tired and trying not to think.

I wanted it to feel good, and I can honestly say, hopefully without sounding too braggy, that it is the best sleep timer out there.

It also does the boring important stuff

Of course, it also needs to do the normal audiobook-player things:

• Supports M4B, MP3, multi-file books, and CUE chapter markers
• Imports from Google Drive, iCloud, and the iOS Files app
• First-class support for Libation (your Audible library, exported) and Audiobookshelf
• Playback speed from 0.5x to 2.0x with per-session memory
• Smart chapter handling: embedded, CUE, multi-file, and auto-generated chapters for books that ship without them
• Siri shortcuts and a home-screen widget
• Listening history with durations, progress changes, and chapter transitions

That stuff is not the headline, but if the boring stuff does not work, the magical stuff does not matter.

Why on-device AI matters

The obvious way to build this app would have been to send the transcript to some server, ask a model to summarize it, and send the result back.

That would have been easier in some ways. It also would have created two problems I didn’t want.

First, cost. If every recap requires going to an online model, the app becomes expensive to run. If the app becomes expensive to run, it probably needs to become a subscription. I didn’t want that for myself, and I didn’t want that for users. I am tired of every small useful thing becoming another monthly payment.

Second, privacy. I’m a fairly private person, and I think paying for apps with your privacy is not OK. Your reading habits are personal. Advertisers don’t need to know what you read, what you listen to, what you are curious about, what you are embarrassed about, what you fall asleep to, or what you are trying to learn. All of that is part of your private life.

I didn’t want Bloox to be another app casually sending that context to servers.

So Bloox uses on-device AI models to transcribe and summarize. That means it is free to use, and also that it is much more private and secure than apps that share your context with online LLMs.

• No subscription
• No monthly payment
• No purchase price
• No in-app purchases

Just the app.

The challenges of working with on-device LLMs (Edge AIs)

Building this was not as straightforward as I expected.

The most tricky part was battery. You can make almost anything work if you don’t care about battery life, but this is an audiobook app. People listen for a long time. Often in the background. Often away from a charger.

So I spent a lot of time making sure this was efficient. Today, the benchmarks show that the battery usage is negligible. Which is maybe boring for most users, but a very important point for me as the developer.

It got hairy quickly

There is a version of this article where I pretend everything was clean and obvious. That would be a lie.

I have been experimenting with third-party on-device models like Gemma, Llama, Qwen, and Phi-4. I want this because it could eventually help support devices that don’t have Apple’s Neural Engine, and it could also help with bringing something like Bloox to Android later.

So far, none of them have satisfied me in terms of speed and battery use. They are impressive. They are also not where I need them to be yet.

TTS sucks… apparently

Another surprising challenge was Apple’s text-to-speech engines.

You would think that for an evolved OS in 2026, text-to-speech would be one of the easier parts. It was not. They are simply not great.

AI code agents hate Swift (or maybe they just hate me)

And then there is the “joy” of building iOS apps with AI coding tools.

I use Cursor and Claude Code heavily, and they are incredibly useful, but SwiftUI and modern iOS development are not always straightforward with agents. I had to find and use a variety of skills and rules files to make sure the agents don’t build based on their knowledge cutoff and accidentally design for older versions of Swift and iOS.

One funny example is that every time I asked an agent to review a piece of code, it complained that I was building toward a non-existent iOS version, because I was targeting iOS 26 and the model thought the last version was iOS 18. It kept trying to “fix” my code by reverting the future.

That sucked. Apparently I was not only building an app, I was also arguing with an AI that the future had already happened.

So why I actually built this

Bloox is for people like me:

• Slightly ADHD
• Easily distracted
• Very into audiobooks
• Concerned about privacy
• Not excited to pay yet another subscription

It is also for people who listen to complicated books — fantasy, sci-fi, long series. Books with too many characters, complicated story lines, too many invented words, and one character you are sure was important 17 hours ago but you cannot remember why.

I want to solve for all of these with the app.

If it does not solve everything yet, that’s OK — it’s a work in progress after all.

One important note

Because Bloox supports importing audiobook files, I want to be very clear: this is for books you legally own.

I care about this, and you should too — authors, voice actors, and studios cannot survive piracy.

On my support site, I show how to use Libation to download and manage your own Audible books so you can listen to them in Bloox. That is the use case I care about.

So, what now?

I’m still playing with this. That’s part of the fun.

I want to keep innovating and adding more, especially around the idea that listening can become more interactive, more contextual, and more helpful without becoming distracting or creepy.

And one of the cool things about building on on-device AI is that even if I do very little, the app should get better as the models improve.

But I don’t want to build this in a vacuum. I built Bloox as a side project for myself, but also for the audiobook community.

It is a passion project. All I ask in return is feedback.

If this sounds useful, please download Bloox from the App Store, try it with a book you legally own, and send me feedback or ideas through the app. There are a bunch of places inside the app where you can do that.

And if you know someone who constantly loses track while listening, or someone who loves giant complicated audiobooks and needs a little help keeping up, please send it to them: eladkatz.us/bloox.

Feedback is a gift. Especially when I’m building the thing for people like us.

If you’re trying to pick between audiobook apps, you might also like my hands-on comparison of seven iOS audiobook players for people who own their files.