Ask a question. A photorealistic face answers it out loud, lip-synced, in about two seconds. Point it at your own documents and give it your own face.
One GPU container does all three jobs — it answers with an LLM, speaks with Kokoro, and renders the face with MuseTalk. There is no database, no vector store, no second service, and nothing else to deploy.
browser ──question──▶ ┌─────────── ONE Modal container, one GPU ───────────┐
│ LLM (your documents, in context) │
│ ↓ │
│ Kokoro (speech, Apache-2.0, on the same GPU) │
│ ↓ │
│ MuseTalk (the face) ──▶ video, STREAMED │
└────────────────────────────────────────────────────┘
It scales to zero, so an idle deployment costs nothing.
You need a Modal account (free tier is enough to try this) and an API key for any OpenAI-compatible LLM.
pip install modal
modal setup
# 1. Your LLM key. Works with OpenAI, Groq, Together, OpenRouter, DeepSeek,
# a vLLM you host, or Ollama on your own machine — set LLM_BASE_URL to point elsewhere.
modal secret create talking-head-llm LLM_API_KEY=sk-...
# 2. Your documents. PDF, Word, HTML, Markdown, text.
cp ~/my-handbook.pdf ~/my-faq.docx corpus/
# 3. Your face. A ~10 second clip — CHECK IT FIRST, this is the step people skip.
python tools/validate-avatar-clip.py my-clip.mp4
cp my-clip.mp4 assets/source.mp4
# 4. Go. (~15 min the first time: it builds the image and bakes in your avatar.)
modal deploy app.pymodal deploy prints a URL. Open it and ask a question.
This repo deliberately ships with no face. A clip of a person is their likeness, and you need the right to use the one you choose — so the build fails, loudly and with instructions, until you supply one. See NOTICES.md.
Your clip must meet four requirements. The first two fail silently — you get a working avatar that is subtly, permanently unsettling, and you will not know why:
| 1. She must BLINK | The eyes come from your footage — MuseTalk only repaints the mouth. No blink in the clip means no blink, ever, on the page. A photoreal face with dead unblinking eyes is the most uncanny thing you can put on a screen. |
| 2. She must have a RESTING BEAT | ~1 second, still, mouth closed. The idle loop — what she does between questions — is built only from her resting frames. Without them she sits there silently mouthing words forever. |
| 3. Locked-off and evenly lit | Head square to camera, no camera movement, flat light, sharp. |
| 4. 25 fps | Not a preference. MuseTalk derives frame indices from a hardcoded 50 Hz audio clock. |
pip install -r tools/requirements.txt
python tools/validate-avatar-clip.py my-clip.mp4Runs on your laptop, no GPU. It performs the same measurements the real pipeline performs and tells you which take to keep:
my-clip.mp4
1280x720, 25.00 fps, 250 frames (10.0s)
blink : frame 10 (15 closed frames, deepest 6% of open)
rest window : frames 221-239 (mouth openness 0.002 vs median 0.208)
✅ USABLE.
docs/YOUR-AVATAR.md has the full recipe, including a copy-paste prompt if you are generating the clip with Veo rather than filming one.
Every file in corpus/ is read at build time, extracted to text, and placed directly
in the model's context window on every question.
The reflex when you have documents is embeddings, a vector store, a retrieval step. Do the arithmetic first: a 150-page manual is about 75k tokens; modern context windows are 128k–1M. If your corpus fits in the context window, put it in the context window. Nothing to embed, nothing to index, nothing to keep in sync, and no retrieval step that can quietly hand the model the wrong three paragraphs.
You would be adding a database in order to make the answers worse.
When to outgrow this: when the corpus stops fitting (the build warns you), or when documents must change without a rebuild. That second one is the real trigger, and it is about the update path, not the token count. Then you want retrieval, and that is a different program.
| Idle | $0. The container scales to zero. |
| Running | The GPU's hourly rate — about $1/hr for an A10. |
| Per question | Whatever your LLM charges. Typically a fraction of a cent. Speech and rendering are free — they run on the GPU you are already paying for. |
Measured warm, on an A10 — the two legs that run on the GPU:
| Speech (Kokoro) | 0.17s for 7.9s of audio — ~46× realtime |
| First video frame | 0.23s |
| Full render | 7.9s of video in 5.7s — 1.4× realtime, so playback never catches the renderer |
Time-to-first-word is therefore whatever your LLM takes, plus about 0.4 seconds — and it does not grow with the length of the answer, because the video streams.
Cold start is the one thing to plan for. After an idle period the first question takes ~35 seconds: the container boots and loads 2 GB of weights. The page pre-warms the GPU as soon as it loads, which usually hides this — but do not let a stakeholder's first click be the cold one. Before a demo:
KEEP_WARM=1 modal deploy app.py # pins a container warm. Bills ~$1/hr, around the clock.
modal deploy app.py # ...and back to scale-to-zero afterwards.Everything expensive happens once, at build time, on Modal's GPUs. Image.run_function(gpu=...)
runs a function on a GPU during the build, and its filesystem writes persist into the
image layer. So face detection, the VAE latents, the blend masks and the idle loop are all
baked in. A cold container never redoes them.
That is the whole reason for choosing Modal. Building this on a rented GPU box means
dragging a 3 GB python environment, 4.4 GB of weights and a GPU prep step down whatever
network that host happens to have — and hosts vary by three orders of magnitude. (One
wrote at 124 KB/s. Another managed 16 MB of git clone in 25 minutes.) Modal builds on its
own infrastructure and caches the layer, which deletes the failure mode rather than
mitigating it. You also never need Docker locally.
The video streams. The obvious design renders the whole clip, then plays it — which means the wait grows with the length of the answer, punishing a long answer twice. Instead, frames are muxed into a fragmented MP4 and pushed down a chunked HTTP body as they are produced, so the first fragment plays while the rest is still on the GPU. Time-to-first-word is flat no matter how much she has to say.
The renderer serves the web page itself. Not laziness: an https page cannot POST audio
to a plain-http GPU box — browsers block mixed content, silently. Same-origin sidesteps
CORS, TLS and tunnels in one move.
She never freezes. One render loop runs continuously, whatever else is happening. The idle loop — a seamless at-rest cycle with a blink, built from your clip at build time — plays between questions and crossfades into her answer. A frozen face reads as a broken page, and that is the one impression this cannot afford.
All optional, via environment variables on the Modal secret or the image:
- One GPU renders one clip at a time. Concurrent users queue. Scale by raising Modal's container limit; each one is a full GPU.
- The answer is capped at ~160 tokens, because every extra sentence is another second of video to render. This is a latency budget, not a style preference.
- Speech is English by default. Kokoro supports other languages; set
TTS_VOICEandTTS_LANGtogether, or you will get nonsense. - Scanned PDFs yield nothing. An image of a page has no text layer. The build reports them by name rather than pretending. OCR first.
- No authentication. The deployed URL is public: anyone who has it can ask questions and spend your LLM budget. Fine for a demo. Put auth in front of it before it holds anything you would not publish.
Code: MIT. The model weights are not all MIT, and if you intend to use this commercially you need to read NOTICES.md — it is short, and it is the difference between a product you may ship and one you may not.