Bidirectional Realtime Voice - StepFun Documentation

Enables real-time voice conversation. It accepts audio and text input and produces streaming audio (with a matching text transcript) as output, over a single WebSocket connection.

For model capabilities, supported voices, audio format, and a runnable quick-start, see StepAudio 2.5 Realtime. An open-source reference client is available at Step-Realtime-Console.

Protocol

WebSocket.

Endpoint

wss://api.stepfun.ai/v1/realtime

The model is passed as a query parameter, e.g. wss://api.stepfun.ai/v1/realtime?model=stepaudio-2.5-realtime.

Authentication

Authorization string required
Authentication token in the format Bearer $STEPFUN_API_KEY.

Request Parameters

model string required
The model to use. Currently supports stepaudio-2.5-realtime.

Voices

Set the voice with session.update before the model produces audio. The following system voices are available:

Voice ID	Voice
`soft-spoken-gentleman`	Calm, gentle male voice
`magnetic-voiced-male`	Deep, magnetic male voice
`vibrant-youth`	Youthful, energetic voice
`lively-girl`	Bright, lively female voice
`livelybreezy-female`	Light, breezy female voice
`elegantgentle-female`	Elegant, gentle female voice
`zixinnansheng`	Confident male voice

You can also use a cloned voice: clone it via the Voice cloning API with the stepaudio-2.5-tts model, then pass the returned voice ID as voice.

Only the voice IDs above (or a valid cloned voice ID) are accepted; other values are rejected with a 400 error. A session’s voice cannot be changed once the model has produced audio.

How it works

After the WebSocket connection is established, the client sends Client Events and receives Server Events to drive the interaction.

Common fields

The following fields are common to Client and Server Events:

Field	Type	Description
`event_id`	string	Event ID
`type`	string	Event type (see the lists below)

Client Events

Create / update session (session.update)

Send this event to create or update the session’s default configuration. The client may send it at any time to update the session; any field may be updated at any time except voice. The server responds with a session.updated event.

modalities array<string>
The set of modalities the model may use. Fixed to ["text", "audio"].
instructions string
Default system instructions (i.e. the system message) prepended before the model call. Use this to guide the model’s content and format (e.g. “be very concise”, “be friendly”) and its audio behavior (e.g. “speak quickly”, “inject emotion into your voice”). The model is not guaranteed to follow the instructions, but they provide guidance on the desired behavior.
voice string
The voice used for generation. See Voices for the available system voices and how to use a cloned voice. The voice cannot be changed once the model has produced audio.
turn_detection object optional
Server VAD configuration. Disabled by default.
Show child attributes
- type string required
  Currently supports server_vad; enables server-side VAD when set.
- prefix_padding_ms integer optional
  Only valid when type=server_vad. Amount of audio (in ms) prepended when the start of speech is detected. Default 500.
- silence_duration_ms integer optional
  Only valid when type=server_vad. Silence duration (in ms) after which the user’s speech is considered finished. Default 100.
- energy_awakeness_threshold integer optional
  Only valid when type=server_vad. Energy wake-up threshold, range 0–5000, default 2500. When audio energy exceeds this threshold the user is considered to have started speaking.
input_audio_format string
Input audio format. Currently supports pcm16.
output_audio_format string
Output audio format. Currently supports pcm16.
tools object array optional
The list of functions available for tool calling.
Show child attributes
- type string
  Tool type, function.
- function object
  Function description.
  Show child attributes
  
  name string
  Function name. Letters, digits, and _/- only; keep it under 64 characters.
  
  description string
  Function description. Tells the model what the function does so it can decide when to call it.
  
  parameters object
  The function parameters.
  Show child attributes
  
  type object
  Parameter schema, usually object.
  
  properties object
  Parameter definitions, keyed by parameter name, each described by type and description. See json-schema.

Sample

{
    "event_id": "event_abc",
    "type": "session.update",
    "session": {
        "modalities": ["text", "audio"],
        "instructions": "You are a helpful AI assistant. Answer briefly and warmly.",
        "voice": "soft-spoken-gentleman",
        "input_audio_format": "pcm16",
        "output_audio_format": "pcm16",
        "turn_detection": {
            "type": "server_vad",
            "prefix_padding_ms": 500
        }
    }
}

Append audio (input_audio_buffer.append)

Append audio bytes to the input audio buffer. The server does not acknowledge this event. In Server VAD mode it triggers model inference.

audio string
Base64-encoded audio bytes, in the format specified by input_audio_format in the session configuration.

Sample

{
    "event_id": "event_abc",
    "type": "input_audio_buffer.append",
    "audio": "Base64EncodedAudioData"
}

Commit audio (input_audio_buffer.commit)

Commit the user’s input audio buffer for inference, creating a new user message item in the conversation. The server responds with an input_audio_buffer.committed event. If the input audio buffer is empty, this event produces an error. Sample

{
    "event_id": "event_abc",
    "type": "input_audio_buffer.commit"
}

Clear audio (input_audio_buffer.clear)

Clear the user’s input audio buffer. The server responds with an input_audio_buffer.cleared event. Sample

{
    "event_id": "event_abc",
    "type": "input_audio_buffer.clear"
}

Create conversation item (conversation.item.create)

Add a new item to the conversation context, including messages, function calls, and function-call responses. Use it to populate conversation history or to add new items mid-session, but it cannot populate Assistant audio messages. On success the server responds with conversation.item.created; otherwise it sends an error event.

previous_item_id string
The ID of the previous item.
item object
The message item; see the message parameters.

Sample

{
    "event_id": "event_abc",
    "type": "conversation.item.create",
    "item": {
        "id": "msg_001",
        "type": "message",
        "role": "user",
        "content": [
            {
                "type": "input_text",
                "text": "Hello"
            }
        ]
    }
}

Delete conversation item (conversation.item.delete)

Send this event to remove an item from the conversation history. The server responds with conversation.item.deleted. If the item does not exist, the server responds with an error.

item_id string
The ID of the item to delete.

Sample

{
    "event_id": "event_abc",
    "type": "conversation.item.delete",
    "item_id": "msg_003"
}

Create response (response.create)

Instructs the server to create a Response, i.e. trigger model inference. Sample

{
    "event_id": "event_abc",
    "type": "response.create"
}

Cancel response (response.cancel)

Cancel an in-progress response. The server returns a response.cancelled event, or an error if there is no response to cancel.

{
    "event_id": "event_abc",
    "type": "response.cancel"
}

Server Events

Error (error)

Returned when an error occurs during server processing. This may be a client or server problem; the session continues.

type string
Error type (e.g. invalid_request_error, server_error).
code string
Error code, if any.
message string
Human-readable error message.
event_id string
The event_id of the client event that caused the error, if applicable.

Sample

{
    "event_id": "event_bcd",
    "type": "error",
    "error": {
        "type": "invalid_request_error",
        "code": "invalid_param",
        "message": "Incomplete audio content",
        "event_id": "event_567"
    }
}

Session created (session.created)

Returned when the session is created, automatically emitted as the first server event when a new connection is established. Contains the default session configuration.

modalities array<string>
The set of modalities the model may use. Fixed to ["text", "audio"].
instructions string
Default system instructions prepended before the model call.
voice string
The voice used for generation.
input_audio_format string
Input audio format. Currently supports pcm16.
output_audio_format string
Output audio format. Currently supports pcm16.
volume_ratio number
Output volume ratio. Default 1.

Sample

{
    "event_id": "event_def",
    "type": "session.created",
    "session": {
        "id": "sess_001",
        "object": "realtime.session",
        "model": "stepaudio-2.5-realtime",
        "modalities": ["text", "audio"],
        "voice": "soft-spoken-gentleman",
        "input_audio_format": "pcm16",
        "output_audio_format": "pcm16",
        "volume_ratio": 1
    }
}

Session updated (session.updated)

Returned when the session is updated. Sample

{
    "event_id": "event_def",
    "type": "session.updated",
    "session": {
        "modalities": ["text", "audio"],
        "instructions": "You are a helpful AI assistant.",
        "voice": "soft-spoken-gentleman",
        "input_audio_format": "pcm16",
        "output_audio_format": "pcm16"
    }
}

Speech started (input_audio_buffer.speech_started)

Notifies that valid speech input has started; typically used for interruption scenarios. Emitted only when Server VAD is enabled.

audio_start_ms integer
The start time of the speech within the audio.
item_id string
The item ID.

Sample

{
    "event_id": "event_bcd",
    "type": "input_audio_buffer.speech_started",
    "audio_start_ms": 1000,
    "item_id": "msg_003"
}

Speech stopped (input_audio_buffer.speech_stopped)

Notifies that valid speech input has ended. Emitted only when Server VAD is enabled.

audio_end_ms integer
The end time of the speech within the audio.
item_id string
The item ID.

Sample

{
    "event_id": "event_1718",
    "type": "input_audio_buffer.speech_stopped",
    "audio_end_ms": 2000,
    "item_id": "msg_003"
}

Response audio delta (response.audio.delta)

Returned as the model generates audio.

response_id string
The response ID.
item_id string
The item ID.
output_index int
The index of the output item in the response.
delta string
A Base64-encoded audio delta, in the output_audio_format set when the session was created.

Sample

{
    "event_id": "event_bcd",
    "type": "response.audio.delta",
    "item_id": "msg_008",
    "delta": "Base64EncodedAudioDelta"
}

Response audio done (response.audio.done)

Returned when the model’s audio is complete. Also emitted when a response is interrupted, incomplete, or cancelled.

response_id string
The response ID.
item_id string
The item ID.

{
    "event_id": "event_bcd",
    "type": "response.audio.done",
    "response_id": "resp_001",
    "item_id": "msg_008"
}

Audio transcript delta (response.audio_transcript.delta)

Streams the text transcript of the model’s audio output.

response_id string
The response ID.
item_id string
The item ID.
output_index int
The index of the output item in the response.
delta string
The transcript delta.

{
    "event_id": "event_bcd",
    "type": "response.audio_transcript.delta",
    "item_id": "msg_002",
    "output_index": 0,
    "delta": "Hello, how can I a"
}

Audio transcript done (response.audio_transcript.done)

Returned when the transcript of the model’s audio output finishes streaming. Also emitted when a response is interrupted, incomplete, or cancelled.

response_id string
The response ID.
item_id string
The item ID.
output_index int
The index of the output item in the response.
transcript string
The complete transcript of the audio.

{
    "event_id": "event_4748",
    "type": "response.audio_transcript.done",
    "response_id": "resp_001",
    "item_id": "msg_008",
    "content_index": 0,
    "transcript": "Hello, how can I assist you today?"
}

Thinking delta (response.thinking.delta)

Streams the model’s reasoning process as it is generated.

response_id string
The response ID.
item_id string
The item ID.
output_index int
The index of the output item in the response.
content_index int
The index of the content part in the item’s content array.
delta string
The reasoning delta.

{
    "event_id": "event_thinking_delta",
    "type": "response.thinking.delta",
    "response_id": "resp_001",
    "item_id": "msg_008",
    "output_index": 0,
    "content_index": 0,
    "delta": "Analyzing the request..."
}

Thinking done (response.thinking.done)

Returned when the model’s reasoning process finishes.

response_id string
The response ID.
item_id string
The item ID.
output_index int
The index of the output item in the response.
content_index int
The index of the content part in the item’s content array.
thinking string
The complete reasoning text.

{
    "event_id": "event_thinking_done",
    "type": "response.thinking.done",
    "response_id": "resp_001",
    "item_id": "msg_008",
    "output_index": 0,
    "content_index": 0,
    "thinking": "Complete reasoning process..."
}

Response text delta (response.text.delta)

Returned as the model generates text output.

response_id string
The response ID.
item_id string
The response item ID.
output_index int
The index of the output item in the response.
content_index int
The index of the content part containing the text.
delta string
The generated text delta.

{
    "event_id": "event_4950",
    "type": "response.text.delta",
    "response_id": "resp_001",
    "item_id": "msg_009",
    "output_index": 0,
    "content_index": 0,
    "delta": "Sure, "
}

Response text done (response.text.done)

Returned when the model’s text output is complete. Also emitted when a response is interrupted, incomplete, or cancelled.

response_id string
The response ID.
item_id string
The response item ID.
output_index int
The index of the output item in the response.
content_index int
The index of the content part containing the text.
text string
The complete generated text.

{
    "event_id": "event_5152",
    "type": "response.text.done",
    "response_id": "resp_001",
    "item_id": "msg_009",
    "output_index": 0,
    "content_index": 0,
    "text": "Sure, I can help with that."
}

Conversation item created (conversation.item.created)

Returned when a conversation item is created.

id string
The unique ID of the message. Optional; the server generates one if the client does not provide it.
type string
The item type, usually message.
role string
The role of the sender (user, assistant, system); applies to message items only.
status string
The item status (completed, incomplete).
content array
The message content; applies to message items.

{
    "event_id": "event_bcd",
    "type": "conversation.item.created",
    "previous_item_id": "msg_001",
    "item": {
        "id": "msg_002",
        "object": "realtime.item",
        "type": "message",
        "status": "completed",
        "role": "user",
        "content": [{
            "type": "input_text",
            "transcript": "Hello"
        }]
    }
}

Conversation item deleted (conversation.item.deleted)

Returned when the client deletes a conversation item with conversation.item.delete. Used to keep the server’s conversation history in sync with the client.

item_id string
The conversation item ID.

{
    "event_id": "event_bcd",
    "type": "conversation.item.deleted",
    "item_id": "msg_001"
}

Input audio transcription completed (conversation.item.input_audio_transcription.completed)

The output of the automatic speech recognition (ASR) of the user’s input audio. Transcription starts when the client commits the buffer, or when the buffer is committed in Server VAD mode. It runs asynchronously with response creation, so this event may arrive before or after the response events.

item_id string
The conversation item ID.
content_index int
The index of the audio content part.
transcript string
The complete transcript of the audio.

{
    "event_id": "event_2122",
    "type": "conversation.item.input_audio_transcription.completed",
    "item_id": "msg_003",
    "content_index": 0,
    "transcript": "Hello"
}

Audio buffer committed (input_audio_buffer.committed)

Returned when the client commits the input audio buffer.

previous_item_id string
The previous item ID in the conversation.
item_id string
The conversation item ID.

{
    "event_id": "event_bcd",
    "type": "input_audio_buffer.committed",
    "previous_item_id": "msg_001",
    "item_id": "msg_002"
}

Audio buffer cleared (input_audio_buffer.cleared)

Returned when the client clears the input audio buffer with input_audio_buffer.clear.

{
    "event_id": "event_1314",
    "type": "input_audio_buffer.cleared"
}

Response output item added (response.output_item.added)

Returned when a new item is created during response generation.

output_index int
The index of the output item in the response.
item object
The output item.
Show child attributes
- id string
  The item ID.
- object string
  Always realtime.item.
- type string
  The item type; currently only message.
- status string
  The item status: completed, incomplete, in_progress.
- role string
  The role of the item (message items only): user, assistant, system.
- content array
  The message content. A system item supports input_text; a user item supports input_text and input_audio; an assistant item supports text content.

{
    "event_id": "event_3334",
    "type": "response.output_item.added",
    "response_id": "resp_001",
    "output_index": 0,
    "item": {
        "id": "msg_007",
        "object": "realtime.item",
        "type": "message",
        "status": "in_progress",
        "role": "assistant",
        "content": []
    }
}

Response output item done (response.output_item.done)

Returned when an item is completed. Also emitted when the response is interrupted, incomplete, or cancelled.

output_index int
The index of the output item in the response.
item object
The output item (same structure as in response.output_item.added).

{
    "event_id": "event_3536",
    "type": "response.output_item.done",
    "response_id": "resp_001",
    "output_index": 0,
    "item": {
        "id": "msg_007",
        "object": "realtime.item",
        "type": "message",
        "status": "completed",
        "role": "assistant",
        "content": [
            {
                "type": "text",
                "text": "Sure, I can help with that."
            }
        ]
    }
}

Response content part added (response.content_part.added)

Returned when a new content part is added to an assistant message item during response generation.

response_id string
The response ID.
item_id string
The corresponding item ID.
content_index int
The index of the content part in the item’s content array.
output_index int
The index of the output item in the response.
part object
Show child attributes
- type string
  The content type: text, audio, or thinking.
- audio string
  Base64-encoded audio data (present when type=audio).
- text string
  The generated text (present when type=text).
- thinking string
  The generated reasoning content (present when type=thinking).
- transcript string
  The transcript of the audio (present when type=audio).

{
    "event_id": "event_3738",
    "type": "response.content_part.added",
    "response_id": "resp_001",
    "item_id": "msg_007",
    "output_index": 0,
    "content_index": 0,
    "part": {
        "type": "text",
        "text": ""
    }
}

Response content part done (response.content_part.done)

Returned when a content part is completed. Also emitted when the corresponding response is interrupted, incomplete, or cancelled.

response_id string
The response ID.
item_id string
The corresponding item ID.
content_index int
The index of the content part in the item’s content array.
output_index int
The index of the output item in the response.
part object
Same structure as in response.content_part.added.

{
    "event_id": "event_3940",
    "type": "response.content_part.done",
    "response_id": "resp_001",
    "item_id": "msg_007",
    "output_index": 0,
    "content_index": 0,
    "part": {
        "type": "text",
        "text": "Sure, I can help with that."
    }
}

Response created (response.created)

Returned when a new Response is created. This is the first event of response creation, with an initial status of in_progress.

id string
The unique ID of the response.
object string
The object type, realtime.response.
status string
The response status (in_progress, completed, cancelled, failed, incomplete).
output list
The list of output items generated by the response.

{
    "event_id": "event_3132",
    "type": "response.created",
    "response": {
        "id": "resp_001",
        "object": "realtime.response",
        "status": "in_progress",
        "status_details": null,
        "output": []
    }
}

Response done (response.done)

Returned when the Response finishes streaming. Always emitted, regardless of the final status. The Response object includes all output items but omits the raw audio data.

id string
The unique ID of the response.
object string
The object type, realtime.response.
status string
The final status (completed, cancelled, failed, incomplete).
output list
The list of output items generated by the response.

{
    "event_id": "event_bcd",
    "type": "response.done",
    "response": {
        "id": "resp_001",
        "object": "realtime.response",
        "status": "completed",
        "status_details": null,
        "output": [
            {
                "id": "msg_006",
                "object": "realtime.item",
                "type": "message",
                "status": "completed",
                "role": "assistant",
                "content": [
                    {
                        "type": "text",
                        "text": "Sure, how can I assist you today?"
                    }
                ]
            }
        ]
    }
}

​Protocol

​Endpoint

​Authentication

​Request Parameters

​Voices

​How it works

​Common fields

​Client Events

​Create / update session (session.update)

​Append audio (input_audio_buffer.append)

​Commit audio (input_audio_buffer.commit)

​Clear audio (input_audio_buffer.clear)

​Create conversation item (conversation.item.create)

​Delete conversation item (conversation.item.delete)

​Create response (response.create)

​Cancel response (response.cancel)

​Server Events

​Error (error)

​Session created (session.created)

​Session updated (session.updated)

​Speech started (input_audio_buffer.speech_started)

​Speech stopped (input_audio_buffer.speech_stopped)

​Response audio delta (response.audio.delta)

​Response audio done (response.audio.done)

​Audio transcript delta (response.audio_transcript.delta)

​Audio transcript done (response.audio_transcript.done)

​Thinking delta (response.thinking.delta)

​Thinking done (response.thinking.done)

​Response text delta (response.text.delta)

​Response text done (response.text.done)

​Conversation item created (conversation.item.created)

​Conversation item deleted (conversation.item.deleted)

​Input audio transcription completed (conversation.item.input_audio_transcription.completed)

​Audio buffer committed (input_audio_buffer.committed)

​Audio buffer cleared (input_audio_buffer.cleared)

​Response output item added (response.output_item.added)

​Response output item done (response.output_item.done)

​Response content part added (response.content_part.added)

​Response content part done (response.content_part.done)

​Response created (response.created)

​Response done (response.done)

Protocol

Endpoint

Authentication

Request Parameters

Voices

How it works

Common fields

Client Events

Create / update session (session.update)

Append audio (input_audio_buffer.append)

Commit audio (input_audio_buffer.commit)

Clear audio (input_audio_buffer.clear)

Create conversation item (conversation.item.create)

Delete conversation item (conversation.item.delete)

Create response (response.create)

Cancel response (response.cancel)

Server Events

Error (error)

Session created (session.created)

Session updated (session.updated)

Speech started (input_audio_buffer.speech_started)

Speech stopped (input_audio_buffer.speech_stopped)

Response audio delta (response.audio.delta)

Response audio done (response.audio.done)

Audio transcript delta (response.audio_transcript.delta)

Audio transcript done (response.audio_transcript.done)

Thinking delta (response.thinking.delta)

Thinking done (response.thinking.done)

Response text delta (response.text.delta)

Response text done (response.text.done)

Conversation item created (conversation.item.created)

Conversation item deleted (conversation.item.deleted)

Input audio transcription completed (conversation.item.input_audio_transcription.completed)

Audio buffer committed (input_audio_buffer.committed)

Audio buffer cleared (input_audio_buffer.cleared)

Response output item added (response.output_item.added)

Response output item done (response.output_item.done)

Response content part added (response.content_part.added)

Response content part done (response.content_part.done)

Response created (response.created)

Response done (response.done)