Shipping an AI Chat With No API Key

There's an AI mode on this site. You can ask it about my work and it answers, with context about whichever page you're standing on. The first version of it did something slightly unusual: it never called an API. No key in an env file, no proxy, no bill that scaled with a Hacker News spike. The whole model ran in your browser, on your machine, on your GPU.

That started as a constraint, became a philosophy, and then — after some months of real traffic on real devices — became a fallback. This is the honest version of that arc, including the part where I changed my mind.

The constraint that started it

A portfolio has an ugly failure mode. It goes quietly viral for a day, every visitor pokes the chat, and you either eat a surprise invoice or watch your one interesting feature rate-limit itself into uselessness on the single day it mattered. A key sitting in client code is worse: it gets scraped and drained within hours, and now you're funding someone else's side project.

The usual answer is a server-side proxy with auth, rate limits and a budget alarm. That's three pieces of infrastructure and an on-call obligation for a page whose purpose is to show people some case studies. The asymmetry bothered me enough to try the other extreme.

Why on-device was worth trying

• No key to leak. There is nothing to steal, because there is no secret.
• No cost curve. A thousand visitors cost exactly what one visitor costs: nothing.
• Actually private. Questions never leave the tab. For a page that is fundamentally about me, letting strangers interrogate it privately felt like the right manners.
• Works offline once the weights are cached — which is a genuinely delightful party trick in a conference wifi dead zone.

There's a fifth reason I'd have been embarrassed to write on a slide: it's a better story. A chat that runs on your machine is a claim about how I think about software, and this site is the argument. That's not nothing, but it also isn't a reason to ship something worse, which is the trap I nearly walked into.

How it actually works

The model is compiled to run against the browser's own GPU. On first use the weights download once and sit in cache; every session after that starts instantly. The context — who I am, what I've built, the projects, the roles, and a short brief for whichever page you're currently reading — is assembled on the client and handed over as a system prompt. No round trip. The enter key does the whole thing.

The interesting engineering wasn't the inference, which is mostly someone else's excellent library. It was the page-aware context: a small index that maps every route to a label, a brief, and the kinds of questions that page invites. Ask about pricing on the work page and you get a different framing than you would on the about page. That index turned out to be the most valuable part of the whole feature, and — as I'd find out — the only part that survived the rewrite intact.

Where it falls apart

On-device is a genuinely worse engineering decision in at least three ways, and pretending otherwise is how you end up shipping something embarrassing:

1. The first load is heavy. Weights are measured in hundreds of megabytes. I gated the download behind an explicit tap and never let it block the page — but a first-time visitor on a phone plan is still being asked for a real favour before they get a single sentence back.
2. Small models are small. A browser-sized model does not reason like a frontier one. Keep its job narrow — answer questions about this portfolio — and it's fine. Let it drift one step wider and it confabulates a job I never had, at a company that doesn't exist, in a confident voice.
3. Device variance is brutal. A new laptop flies. A three-year-old phone crawls, heats up, and occasionally has the tab killed by the OS mid-sentence. The median visitor to a design portfolio is on a phone, in a browser I don't control, doing this once.

The third one is the killer. I built the thing on a machine that made it look great, and the analytics quietly told me most people were meeting a different product than the one I'd designed.

The honest update: what runs today

The chat is now cloud-first, with the on-device model as the fallback. A hosted Gemini model answers when it can; if the call fails, the key is absent, or the network is gone, the browser model picks up where it left off with the same context and the same personality. Same interface, same page-awareness, different engine underneath.

It's less pure and much better. First response arrives in a second or two instead of after a download. Quality is good enough that the answers about my work are worth reading rather than merely impressive for a browser. And every original worry turned out to be manageable with boring engineering: the key lives on the server, the route is rate-limited, and there's a hard monthly ceiling I would notice long before my bank did.

The one thing I refused to give up was graceful degradation. There is no state in which the chat is simply broken. That's the whole reason the on-device path stayed in the codebase instead of getting deleted in the same commit that added the API call.

Designing the fallback ladder

What I actually designed, in the end, wasn't a chat. It was a ladder — four rungs, each one degrading to something that still works:

1. Cloud model. Fast, good, the default path for almost everyone.
2. On-device model. Slower and smaller, invoked when the network or the API isn't there. Announced plainly, never disguised as the same thing.
3. Curated answers. A hand-written set of responses to the dozen questions people actually ask. No model involved.
4. A link to email me. The oldest fallback in the business, and still the one with the best response rate.

Every rung is a complete experience rather than an error state, and each one is honest about which rung you're on. Users forgive a smaller model. What they don't forgive is a spinner that never resolves, or a confident answer from a system that has quietly lost its connection to reality.

What it actually costs to run

The fear that started this whole detour was the bill, so it's only fair to publish the numbers. A conversation with this thing is short — people ask two or three questions, not twenty. The system prompt is the expensive part, and it's the same on almost every request.

Three cheap decisions took the cost from unknown and therefore frightening to a rounding error I can forecast:

• Cache the stable prefix. The persona, the project index and the page brief don't change between requests. Caching that prefix cuts the per-message cost of the largest part of the prompt to almost nothing.
• Cap the output. Answers about a portfolio don't need to be essays. A tight token ceiling improves the writing and halves the spend at the same time — the only optimisation I've ever made that had no tradeoff.
• Rate-limit per session, hard. A handful of messages per visitor per hour. Nobody legitimate has ever hit it; anyone who does is not reading my case studies.

There's a monthly ceiling behind all of it, and when it's reached the ladder simply drops to the next rung instead of failing. That's the part I'd argue is the actual engineering: not the model call, but deciding in advance what happens when the money runs out.

The design-engineering lesson

The interesting decision was never technical. It was where to draw the boundary of the feature. Once I accepted the model might be small, slow or absent, the design fell out of it: constrain the scope, make heavy work a deliberate act, and always keep a non-AI path that's good enough to stand alone.

The second lesson is about ego. I shipped the pure version, wrote a confident post about it, and then the data said the pure version was serving most people worse. Updating the product was easy. Updating the post — this one — took considerably longer, which tells you something about which of the two I was more attached to.

Build the layer that knows what your product means. Everything above it is a vendor you'll swap out twice.