Concepts / Streaming media

Streaming media#

The avatar's audio and video arrive as real-time streams of PyAV frames over WebRTC. Consuming them is a matter of iterating two async iterators.

Frame iterators#

The session exposes inbound media as async iterators:

async with client.connect() as session:
    async for frame in session.video_frames():   # PyAV VideoFrame
        img = frame.to_ndarray(format="rgb24")   # (H, W, 3) uint8
 
    async for frame in session.audio_frames():    # PyAV AudioFrame
        samples = frame.to_ndarray()              # 48 kHz stereo int16

• video_frames() yields PyAV VideoFrame objects — call frame.to_ndarray("rgb24") (or "bgr24" for OpenCV).
• audio_frames() yields PyAV AudioFrame objects carrying 48 kHz stereo int16 PCM.

Both iterators end when the session closes, so async for loops exit cleanly on teardown.

Keeping latency low#

Frames arrive in real time. When a consumer falls behind, the SDK drops the oldest frame rather than growing latency — so a slow renderer degrades to skipped frames, not an ever-increasing delay.

Streaming text in#

Streaming isn't only inbound. When your text arrives incrementally (tokens from your own model), push it with a talk stream so the avatar starts speaking before the sentence is complete:

async with session.create_talk_stream() as talk:
    await talk.send("Streaming ")
    await talk.send("this out loud.", end_of_speech=True)

See Talk streams and Driving the avatar.