Multi-turn Conversations

Goals

Learn how to serve a task’s prompt as a role-tagged conversation instead of a single block of text
Understand how harnesses consume conversation-shaped prompts, and how the feature degrades gracefully everywhere else

Prerequisites

openreward>=0.1.157
Completing the Your First Environment tutorial

Introduction

Most environments serve their prompt with get_prompt(): a list of content blocks that every harness wraps in a single {"role": "user"} message. That remains the default — and for most tasks, the right — way to serve a prompt. Some tasks, though, are conversations. A dialogue dataset where the model must write the next reply. A preference task built on chat transcripts. A support conversation the policy must continue. For those, the task carries the conversation so far — user turns, assistant turns, maybe a system prompt — and squeezing it into one user message loses the structure the model was trained on. get_messages() is an additional, opt-in prompt surface for exactly these tasks. The environment returns role-tagged messages, the harness feeds them to the model as a real conversation, and the model writes the next assistant turn. Everything else about the episode — the tool loop, the terminal answer, rewards — proceeds exactly as it always has.

This is not user simulation. The conversation is task data, served once at the start of the episode. The environment does not generate new user turns mid-episode. (If you need a simulated user, see the pattern in Chat Backends — a simulated user is a model role the environment owns, called from inside a tool.)

Serving a conversation

Implement get_messages() instead of get_prompt() — one of the two is required, and each defaults to delegating to the other:

from openreward.environments import Environment, JSONObject, Message, TextBlock

class DialogueEnv(Environment):
    def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
        super().__init__(task_spec)
        self.conversation = task_spec["conversation"]

    def get_messages(self) -> list[Message]:
        return [
            Message(role=turn["role"], blocks=[TextBlock(text=turn["text"])])
            for turn in self.conversation
        ]

    @classmethod
    def list_tasks(cls, split: str) -> list[JSONObject]:
        return [
            {
                "id": "support-1",
                "conversation": [
                    {"role": "system", "text": "You are a support agent for Acme."},
                    {"role": "user", "text": "My order hasn't arrived."},
                    {"role": "assistant", "text": "Sorry to hear that — what's the order number?"},
                    {"role": "user", "text": "It's #4411, ordered two weeks ago."},
                ],
            }
        ]

A Message has a role ("system", "user", or "assistant") and content blocks — TextBlock / ImageBlock, plus the rich block types below, so multimodal and agentic conversations work unchanged. Note that the {"role": ..., "text": ...} dicts in this task spec are just this example’s own convention — task specs are plain JSON, and the SDK doesn’t prescribe how a conversation is stored in them. get_messages() is the translation point where your task data becomes Message objects; store whatever shape suits your data and build the blocks there. Rewards are unaffected: grade the model’s next turn however you would grade any response — a submit tool, a terminal tool, an LLM grader or rubrics.

Consuming a conversation

On the client, session.get_messages() returns the conversation, with optional provider formatting:

with environment.session(task=task) as session:
    # Provider-shaped, ready to send:
    input_messages = session.get_messages(format="openai")
    response = oai_client.chat.completions.create(
        model="gpt-5.2", messages=input_messages, tools=tools,
    )

The format argument mirrors list_tools(format=...):

Format	Returns
`None`	`list[Message]` — the raw conversation
`"openai"` / `"openrouter"`	a chat-completions message list
`"anthropic"`	`{"system": Optional[str], "messages": [...]}` — leading system turns hoisted into the separate system parameter
`"google"`	`{"system_instruction": Optional[str], "contents": [...]}` — the assistant role mapped to `"model"`

Rich conversations: reasoning and tool history

Requires openreward>=0.1.157.

A conversation the policy must continue often contains more than text: past reasoning (interleaved-thinking models carry reasoning across turns, and OpenReward reasoning is plain text meant to be passed back in) and tool interactions (an agent transcript is a conversation). Three further block types cover this, following Anthropic message semantics — everything is a content block, and no role: "tool" exists:

Block	Lives in	Fields
`ReasoningBlock`	`assistant` turns	`text`, optional `signature` (passthrough for provider-signed thinking)
`ToolCallBlock`	`assistant` turns	`id`, `name`, `input`
`ToolResultBlock`	`user` turns	`tool_call_id`, `blocks` (ordinary text/image content — images welcome), `is_error`

Placement is validated on the Message model itself — a tool_result in an assistant turn fails at construction with a clear error, not at the provider.

from openreward.environments import (Message, ReasoningBlock, TextBlock,
                                     ToolCallBlock, ToolResultBlock)

def get_messages(self) -> list[Message]:
    return [
        Message(role="user", blocks=[TextBlock(text="My order #4411 hasn't arrived.")]),
        Message(role="assistant", blocks=[
            ReasoningBlock(text="Two weeks late — check the carrier first."),
            ToolCallBlock(id="tc_1", name="lookup_order", input={"id": "4411"}),
        ]),
        Message(role="user", blocks=[
            ToolResultBlock(tool_call_id="tc_1",
                            blocks=[TextBlock(text='{"status": "stalled_at_hub"}')]),
        ]),
    ]

The example above builds blocks by hand; if you’d rather keep conversations in your task data, store them in the canonical Message wire shape and let pydantic parse them — all block types, placement validation included:

def get_messages(self) -> list[Message]:
    return [Message.model_validate(m) for m in self.task_spec["conversation"]]

{
  "conversation": [
    {"role": "user", "blocks": [{"type": "text", "text": "My order #4411 hasn't arrived."}]},
    {"role": "assistant", "blocks": [
      {"type": "reasoning", "text": "Two weeks late — check the carrier first."},
      {"type": "tool_call", "id": "tc_1", "name": "lookup_order", "input": {"id": "4411"}}
    ]},
    {"role": "user", "blocks": [
      {"type": "tool_result", "tool_call_id": "tc_1",
       "blocks": [{"type": "text", "text": "{\"status\": \"stalled_at_hub\"}"}]}
    ]}
  ]
}

format conversion handles each provider’s shape wherever its schema can represent the block:

Format	Reasoning	Tool calls / results
`"anthropic"`	native `thinking` blocks (`signature` included only when set)	native `tool_use` / `tool_result`
`"openai"` / `"openrouter"`	`reasoning_content` field on the assistant message (the DeepSeek/vLLM/SGLang convention)	native `tool_calls`; each tool result becomes its own `role: "tool"` message. Images inside tool results raise — tool messages are text-only there
`"google"`	raises — no input representation for unencrypted reasoning	native `function_call` / `function_response` parts

Where a schema can’t represent a block, conversion raises a clear ValueError rather than silently dropping content; format=None always returns the full canonical messages.

Compatibility

Both prompt surfaces coexist, and /prompt remains the primary one:

Existing environments and harnesses change nothing. A prompt-based environment behaves byte-identically to before.
get_messages() works against every environment. Against a prompt-based environment it returns the prompt wrapped as a single user message; against a server older than 0.1.157 (which has no /messages route) the client falls back the same way. A harness can adopt get_messages() unconditionally.
Old harnesses still work against conversation environments. /prompt is served for every environment; a conversation flattens to [role]-tagged text blocks (rich blocks render to text: reasoning as <think>…</think>, tool calls as [tool_call name] {args} lines). A conversation consisting of exactly one plain user message flattens to its blocks verbatim.
Rich blocks are versioned. A server’s supported block set is advertised as message_block_types in the tool list (absent on servers before 0.1.157, meaning text/image only). Clients skip block types they don’t know when reading /messages (lossy but safe), so future block types degrade the same way.

Next Steps

Preference Rewards

Conversation-shaped tasks pair naturally with preference-based (A ≻ B) rewards

Get started

Core Concepts

Making Environments

Advanced Environment Topics

Training

Evaluation

Harnesses

Sandbox Providers

Standalone APIs

Integrations

Rollouts

Deployment

Storage & Data

Multi-turn Conversations

Goals

Prerequisites

Introduction

Serving a conversation

Consuming a conversation

Rich conversations: reasoning and tool history

Compatibility

Next Steps

Preference Rewards

​Goals

​Prerequisites

​Introduction

​Serving a conversation

​Consuming a conversation

​Rich conversations: reasoning and tool history

​Compatibility

​Next Steps

Preference Rewards

Goals

Prerequisites

Introduction

Serving a conversation

Consuming a conversation

Rich conversations: reasoning and tool history

Compatibility

Next Steps