Concepts / Real-time avatars
Real-time avatars
A Zeli avatar is a live, lip-synced talking head rendered on a GPU server and streamed to your process in real time. The SDK is a thin, fully-typed client — it does not run the models itself.
Two planes
A session runs over two coordinated channels:
- Media plane — a WebRTC connection delivers synchronized audio + video
frames to your process in real time. You consume them as PyAV
VideoFrame/AudioFrameobjects. - Control plane — a control channel (a WebSocket) carries the text you send
(
send_message,talk, talk streams) and the events you receive (transcripts, speech start/stop, emotions, warnings, errors).
You need a running Zeli avatar server and its base URL. The SDK negotiates the connection and hands you frames and events; the rendering, TTS, and (optional) conversational model all run on the server's GPU.
What happens on connect
async with client.connect() as session:
...- The SDK negotiates a WebRTC connection via the server's
POST /connectendpoint (audio + video, receive-only), then emitsCONNECTION_ESTABLISHED. - It opens the control channel and, once the handshake completes, emits
SESSION_READYwith aSessionInfo. - Frames begin flowing on
session.video_frames()andsession.audio_frames(); you drive the avatar with text.
The control gateway
Some capabilities require the server's control gateway:
| Capability | Needs gateway? |
|---|---|
send_message (conversational) | Falls back to HTTP |
talk (direct TTS) | ✅ |
create_talk_stream (incremental) | ✅ |
| Streaming events (speech start/stop, emotions) | ✅ |
Check session.has_control_channel to detect a server that predates the gateway.
On such a server, talk / talk streams raise
SessionError and send_message uses a plain-HTTP fallback.
Deployment shape
The same server codebase serves two topologies from one engine:
- Embedded / loopback — an on-instance avatar for the AI pipeline; auth stays open because nothing leaves the box.
- Standalone — a public box that must set an API key (see Authentication).
Models are self-hostable with Australian data residency (Sydney), and the underlying model is MIT-licensed.