time multimodal-Real-time multimodal interaction flow

The interaction flow between the real-time multimodal server and client.

VAD mode

To enable VAD mode, set the session.turn_detection parameter of the client event to "server_vad". In VAD mode, the server performs Voice Activity Detection (VAD) on incoming audio and responds when it detects speech. This mode suits scenarios where the client continuously sends audio to the server. It is also the default mode.

• The server sends the input_audio_buffer.speech_started event when it detects the start of speech.

• The client can append audio to the buffer at any time by sending the input_audio_buffer.append event.

• The server sends the input_audio_buffer.speech_stopped event when it detects the end of speech.

• The server commits the input audio buffer by sending the input_audio_buffer.committed event.

• The server sends the conversation.item.created event, which contains the user message item created from the audio buffer.

Tool calling flow

When a response requires a tool call in VAD mode:

• After the server detects the end of speech and generates a response, it identifies the need for a tool call.

• The server sends the response.function_call_arguments.delta event with incremental tool call argument data.

• The server sends the response.function_call_arguments.done event to indicate that the arguments are complete.

• The client executes the tool call and retrieves the result.

• The client sends the tool call result using the conversation.item.create event.

• The server automatically generates a response based on the tool call result.

Manual mode

To enable Manual mode, set the session.turn_detection parameter of the client event to null. In this mode, the client requests a response by sending the input_audio_buffer.commit and response.create events. This mode suits push-to-talk scenarios, such as sending voice messages in chat applications.

• The client can append audio to the buffer by sending the input_audio_buffer.append event.

• The client commits the input audio buffer by sending the input_audio_buffer.commit event. This commit creates a new user message item in the conversation.

• The server sends the input_audio_buffer.committed event.

• The client sends the response.create event to request a response from the model.

• The server sends the conversation.item.created event.

Tool calling flow

When a response requires a tool call in Manual mode:

• After the client sends the response.create event, the server generates a response and identifies the need for a tool call.

• The server sends the response.function_call_arguments.delta event with incremental tool call argument data.

• The server sends the response.function_call_arguments.done event to indicate that the arguments are complete.

• The client executes the tool call and retrieves the result.

• The client sends the tool call result using the conversation.item.create event.

• The client sends the response.create event to request a response from the model.

• The server generates a response based on the tool call result and sends it to the client through the response.audio.delta or response.text.delta events.