Your first environment

Goals

Make a mathematics environment using the OpenReward library.
Deploy the environment to OpenReward.
Sample from the environment using a model of your choice.

Prerequisites

An OpenReward account
An OpenReward API key
An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)

Setup

Environments in OpenReward are written using ORS. ORS is implemented in the OpenReward Python library, and we will use it for this tutorial. You can install the library using pip or uv:

pip install openreward

Background: GSM8K

GSM8K is a classic language model dataset for math word problems released by OpenAI in 2021. Problems are at the grade school level and answers are integers. An example problem and answer from this dataset is shown below:

Dean’s mother gave him $28 to go to the toy store. Dean bought 6 toy cars and 5 teddy bears. Each toy car cost $2 and each teddy bear cost $1. His mother then feels generous and decides to give him an extra $10. How much money does Dean have left?

Answer: 21

We will learn how to build an ORS environment server for GSM8K in this tutorial.

Understanding ORS servers

To begin we’ll initialise our GSM8K project with a basic template and investigate how ORS servers work. Initialise a project using the OpenReward cli:

orwd init gsm8k --template basic
cd gsm8k && ls gsm8k

The template contains server.py, Dockerfile and requirements.txt. If you look inside server.py, you can see a BasicEnvironment is defined. To run this ORS server, install the requirements and run server.py:

pip install -r requirements.txt
python server.py

INFO:     Started server process [92466]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

This environment is now running on port 8080. We will leave this running. Now let’s see how we can interact with the BasicEnvironment. In a different terminal, write the following file to test_environment.py:

from openreward import OpenReward

or_client = OpenReward()
environment = or_client.environments.get(name="basicenvironment")

Because the environment class is named BasicEnvironment, we will pass in the name basicenvironment. This is the same API that we use to get production environments on OpenReward. But in this case we have not passed in a namespace (e.g. OpenAI/SimpleQA), so it defaults to using localhost instead. If your server is running in a different location, you can specify a base_url if you want more control. For example:

environment = or_client.environments.get(
    name="basicenvironment", 
    base_url="https://mydomain.com"
)

Now let’s test interacting with this ORS server. Environments have splits, which are lists of tasks for different purposes such as training and evaluation. To see the available splits on the test_environment.py, add the following to test_environment.py and run:

print(environment.list_splits())

python test_environments.py

['train', 'test']

So there are two splits available in this environment. Next, let’s view the tasks that are available for the train split.

print(environment.list_tasks("train"))

python test_environments.py

[Task(server_name='basicenvironment', environment_name='basicenvironment', task_spec={'id': 'train-0', 'problem': 'What is 2+2?', 'solution': 4}, namespace=None)]

We can see that there is a single task. Here the Task object specifies the task specification, which is one of the primitives of ORS. Next, let us see the available tools in the environment. In ORS, actions are tools, and executing tools is the only way to interact with the environment.

print(environment.list_tools())

python test_environments.py

[ToolSpec(name='answer', description='The answer tool can be used to submit your final answer. Note that this finishes the episode.', input_schema={'description': 'Each tool takes in arguments, and these are specified using types.\n\nExamples:\n- An answer tool might have an answer argument\n- A bash tool might have a command argument ', 'properties': {'answer': {'title': 'Answer', 'type': 'string'}}, 'required': ['answer'], 'title': 'AnswerParams', 'type': 'object'})]

There is a single tool called answer. Note this tool specification is the same as a tool specification in MCP, allowing compatibility with existing model function calling capabilities. Now let’s test calling the answer tool. Write the following script test_tool.py:

environment = or_client.environments.get(name="basicenvironment")
tasks = environment.list_tasks(split="train")
example_task = tasks[0]

with environment.session(task=example_task) as session:
    prompt = session.get_prompt()
    tool_result = session.call_tool("answer", {"answer": "4"})
    print(prompt)
    print(tool_result)

[TextBlock(text='What is 2+2?', detail=None, type='text')]
ToolOutput(blocks=[TextBlock(text='Correct!', detail=None, type='text')], metadata=None, reward=1.0, finished=True)

As we can see, prompt contains a TextBlock with the prompt text for this task. After calling the tool on the session with call_tool - with the tool name answer and tool arguments {"answer": "4"} - we obtain a ToolOutput. The ToolOutput also contains a list of blocks, in this case a TextBlock showing us some text for the agent (Correct!). We also obtain a reward of 1.0, as well as an finished state of True, denoting that the episode has finished. This shows us the basics of how we can interface with an ORS environment server. In the next section we will build a GSM8K environment.

Building the GSM8K environment

First we’ll download the two parquet files from the GSM8K HuggingFace repository and put them in the root of our project:

test-00000-of-00001.parquet
train-00000-of-00001.parquet

A single row of data from the train set looks as follows:

{'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72', 'id': '0'}

Next we’ll write a new server file. This will involve:

Loading the tasks from the parquet files
Verifying the answer is correct - we’ll use the MathVerify library for this.

from math_verify import parse, verify
import pandas as pd
from pydantic import BaseModel

from openreward.environments import Environment, JSONObject, Server, TextBlock, ToolOutput, tool

class GSM8KTaskSpec(BaseModel):
    id: str
    question: str
    answer: str


class AnswerParams(BaseModel):
    answer: str


train_tasks = pd.read_parquet("train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("test-00000-of-00001.parquet").to_dict(orient="records")

for i, task in enumerate(train_tasks):
    task['id'] = str(i)
for i, task in enumerate(test_tasks):
    task['id'] = str(i)


class GSM8K(Environment):
    """
    A GSM8K environment
    """
    def __init__(self, task_spec: JSONObject = {}):
        super().__init__(task_spec)
        self.config = GSM8KTaskSpec.model_validate(task_spec)

    @classmethod
    def list_tasks(cls, split: str) -> list[JSONObject]:
        if split == "train":
            return train_tasks
        elif split == "test":
            return test_tasks
        raise ValueError(f"Unknown split: {split}")

    @classmethod
    def list_splits(cls) -> list[str]:
        return ["train", "test"]

    def get_prompt(self) -> str:
        return [TextBlock(type="text", text=self.config.question)]

    @tool
    def answer(self, params: AnswerParams) -> ToolOutput:
        """
        The answer tool can be used to submit your final answer. Note that this finishes the episode.
        """
        gold = parse(self.config.answer)
        answer = parse(params.answer)
        is_correct = verify(gold, answer)

        if is_correct:
            agent_message = "Correct!"
            reward = 1.0
        else:
            agent_message = "Wrong!"
            reward = 0.0

        return ToolOutput(
            blocks=[TextBlock(type="text", text=agent_message)],
            reward=reward,
            finished=True
        )

if __name__ == "__main__":
    Server([GSM8K]).run()

Now we can test this environment. First run the server as before:

python server.py

Now choose a model provider of your choice and sample from the environment:

OpenAI
Anthropic
Google
OpenRouter

Set your API key

Make sure you have an API key for OpenAI, and set the environment variable:

export OPENAI_API_KEY='your-openai-api-key-here'

Create your code

Save this as sample_agent.py:

from openai import OpenAI
from openreward import OpenReward
import json

or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5"

environment = or_client.environments.get(name="gsm8k")
tasks = environment.get_tasks(split="train")
tools = environment.get_tools(format="openai")

example_task = tasks[0]

with environment.session(task=example_task) as session:
    prompt = session.get_prompt()
    input_list = [{"role": "user", "content": prompt[0].text}]
    finished = False
    print(input_list)

    while not finished:
        response = oai_client.responses.create(
            model=MODEL_NAME,
            tools=tools,
            input=input_list
        )
        print(response.output)

        input_list += response.output

        for item in response.output:
            if item.type == "function_call":
                tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))

                reward = tool_result.reward
                finished = tool_result.finished

                input_list.append({
                    "type": "function_call_output",
                    "call_id": item.call_id,
                    "output": json.dumps({
                        "result": tool_result.blocks[0].text
                    })
                })

                print(input_list[-1])

                if tool_result.finished:
                    finished = True
                    break

Run your code

  python sample_agent.py

Example output:

[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
[ResponseReasoningItem(id='rs_000dde94c785f76800693413cbb66c81928b57cca2b5488c2d', summary=[], type='reasoning', content=None, encrypted_content=None, status=None), ResponseOutputMessage(id='msg_000dde94c785f76800693413ce541881928b37ef5a6ec49cf9', content=[ResponseOutputText(annotations=[], text='72', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"72"}', call_id='call_Eaw188yiAYpNCgQXduR1x0lR', name='answer', type='function_call', id='fc_000dde94c785f76800693413cf1be08192995f531548dffa4d', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_Eaw188yiAYpNCgQXduR1x0lR', 'output': '{"result": "Correct!"}'}

Set your API key

Make sure you have API key for Anthropic, and set the environment variable:

export ANTHROPIC_API_KEY='your-anthropic-api-key-here'

Create your code

Save this as sample_agent.py:

import anthropic
from openreward import OpenReward
import json

or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-5"

environment = or_client.environments.get(name="gsm8k")
tasks = environment.get_tasks(split="train")
tools = environment.get_tools(format="anthropic")

example_task = tasks[0]

with environment.session(task=example_task) as session:
    prompt = session.get_prompt()
    messages = [{"role": "user", "content": prompt[0].text}]
    finished = False
    print(messages)

    while not finished:
        response = ant_client.messages.create(
            model=MODEL_NAME,
            max_tokens=4096,
            tools=tools,
            messages=messages
        )

        print(message)
        messages.append(message)

        if message.stop_reason == "tool_use":
            tool_use = next(block for block in message.content if block.type == "tool_use")
            tool_name = tool_use.name
            tool_input = tool_use.input
            
            tool_result = session.call_tool(tool_name, tool_input)
            reward = tool_result.reward
            finished = tool_result.finished

            messages.append({
                "role": "user",
                "content": [
                    {
                        "type": "tool_result",
                        "tool_use_id": tool_use.id,
                        "content": tool_result.blocks[0].text
                    }
                ]
            })

            print(messages[-1])
            
            if tool_result.finished:
                finished = True
                break

Run your code

  python sample_agent.py

Example output:

[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
Message(id='msg_01DJorrT3UPhnPhdN19ugsyT', content=[TextBlock(citations=None, text='I need to find the total number of clips Natalia sold in April and May.\n\nGiven information:\n- In April, Natalia sold clips to 48 friends\n- In May, she sold half as many clips as in April\n\nLet me calculate:\n- April: 48 clips\n- May: 48 ÷ 2 = 24 clips\n- Total: 48 + 24 = 72 clips', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': '72'}, name='answer', type='tool_use')], model='claude-sonnet-4-5-20250929', role='assistant', stop_reason='tool_use', stop_sequence=None, type='message', usage=Usage(cache_creation=CacheCreation(ephemeral_1h_input_tokens=0, ephemeral_5m_input_tokens=0), cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=608, output_tokens=151, server_tool_use=None, service_tier='standard'))
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}

Set your API key

Make sure you have an API key for Gemini and set it as an environment variable:

export GEMINI_API_KEY='your-gemini-api-key-here'

Create your code

Save this as sample_agent.py:

from google import genai
from google.genai import types
from openreward import OpenReward
import json

or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"

environment = or_client.environments.get(name="gsm8k")
tasks = environment.get_tasks(split="train")
tools = environment.get_tools(format="google")

genai_tools = [types.Tool(function_declarations=[f]) for f in tools]
genai_config = types.GenerateContentConfig(tools=genai_tools)

example_task = tasks[0]

with environment.session(task=example_task) as session:
    prompt = session.get_prompt()
    contents = [
        types.Content(
            role="user", parts=[types.Part(text=prompt[0].text)]
        )
    ]
    finished = False
    print(contents)

    while not finished:

        response = gem_client.models.generate_content(
            model=MODEL_NAME,
            config=genai_config,
            contents=contents
        )

        print(response.candidates[0].content)

        contents.append(response.candidates[0].content) # Append the content from the model's response.

        for part in response.candidates[0].content.parts:
            if part.function_call:
                tool_call = part.function_call
                tool_result = session.call_tool(tool_call.name, tool_call.args)

                reward = tool_result.reward
                finished = tool_result.finished

                function_response_part = types.Part.from_function_response(
                    name=tool_call.name,
                    response={"result": json.dumps({
                        "result": tool_result.blocks[0].text
                    })},
                )

                contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response

                print(contents[-1])

                if tool_result.finished:
                    finished = True
                    break

Run your code

  python sample_agent.py

Example output:

[Content(
parts=[
    Part(
    text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'
    ),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
    args={
    'answer': 'Natalia sold 72 clips altogether in April and May.'
    },
    name='answer'
),
thought_signature=b'\n\xe3\x02\x01r\xc8\xda|#w\x85/\x9e\xd1\xe7mK\xc1\xdd\xf6X\xeayY\x12<\x0fQ\x1608\xedKg\x07\xe3tV\xd0\xe3\xaf\xc6=\xc0.x\x15\x85\xeb\xec+\x84\xad\xf0\x02\xb6\xf8\x88iX\x94\xf8\xcb8g\xf9m\x02\xcd`\xb1\x8fm\x1e\x10\x1a\x14]2\x81H\x7f-\xb6E6\xd8\xf3\xa6kUw\xb7\xc3kr\x14...'
)] role='model'
parts=[Part(
function_response=FunctionResponse(
    name='answer',
    response={
    'result': '{"result": "Correct!"}'
    }
)
)] role='user'

Set your API key

Make sure you have an API key for OpenRouter and set it as an environment variables:

export OPENREWARD_API_KEY='your-openreward-api-key-here'

Create your code

Save this as sample_agent.py:

from openai import OpenAI
from openreward import OpenReward
import json
import os

or_client = OpenReward()
oai_client = OpenAI(
  base_url="https://openrouter.ai/api/v1",
  api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"

environment = or_client.environments.get(name="gsm8k")
tasks = environment.get_tasks(split="train")
tools = environment.get_tools(format="openrouter")

example_task = tasks[0]

with environment.session(task=example_task) as session:
    prompt = await session.get_prompt()
    input_list = [{"role": "user", "content": prompt[0].text}]
    finished = False
    print(input_list)

    while not finished:
        response = oai_client.chat.completions.create(
            model=MODEL_NAME,
            tools=tools,
            messages=input_list
        )
        print(response)

        input_list.append(response)

        tool_calls = response.choices[0].message.tool_calls
        if not tool_calls:
            return
        
        for tool_call in response.choices[0].message.tool_calls:
            tool_name = tool_call.function.name
            tool_args = json.loads(tool_call.function.arguments)
            tool_result = await session.call_tool(tool_name, tool_args)

            reward = tool_result.reward
            finished = tool_result.finished

            input_list.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": json.dumps({
                    "result": tool_result.blocks[0].text
                })
            })

            print(input_list[-1])

            if tool_result.finished:
                finished = True
                break

Run your code

python sample_agent.py

Example output:

[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
ChatCompletion(id='gen-1765024924-6BK4o9Oiefjb7Fe7J7eW', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content="Let's break this down step by step.\n\n1. In April: Natalia sold clips to **48 friends**. This means she sold **48 clips** (assuming one clip per friend).\n2. In May: She sold **half as many clips** as in April.\n   - Half of 48 is \\( 48 \\div 2 = 24 \\).\n3. Altogether in April and May:\n   \\[\n   48 + 24 = 72\n   \\]\n\nSo Natalia sold **72 clips** altogether in April and May.\n\n", refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=[ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "72"}', name='answer'), type='function', index=0)], reasoning=None), native_finish_reason='tool_calls')], created=1765024924, model='deepseek/deepseek-v3.2', object='chat.completion', service_tier=None, system_fingerprint='', usage=CompletionUsage(completion_tokens=153, prompt_tokens=333, total_tokens=486, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, image_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, video_tokens=0), cost=0.00015417, is_byok=False, cost_details={'upstream_inference_cost': None, 'upstream_inference_prompt_cost': 8.991e-05, 'upstream_inference_completions_cost': 6.426e-05}), provider='SiliconFlow')
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': '{"result": "Correct!"}'}

Nice one! We have a working ORS environment. Now we’ll see how how we can host the environment on OpenReward. This is not necessary - you can host your environment wherever you like. However, the benefits of using OpenReward are:

Infrastructure sorted: you do not have to set up infrastructure and compute to host the environment yourself. We take care of this for you and you are only charged based on your actual usage of the environment.
Discovery supercharged: your environment can be discovered and used by other users of the platform, helping drive adoption and attention to your work.

We’ll see how to do this next.

Host on OpenReward

Log into OpenReward, press the plus icon in the navbar and press New Environment:

Next, fill in information about the environment and press Create Environment:

You will be redirected to your new environment and will see setup instructions:

Upload environment assets

We will need a way to use the train and test parquet files in our environment. We’ll upload these to the environment assets: Click on the Assets tab and press Upload Environment Assets: [IMAGE HERE] You can then upload these two files: Which will give you two URLs you can use for developing your environment locally:

https://openreward.ai/your-username/GSM8K/assets/test-00000-of-00001.parquet
https://openreward.ai/your-username/GSM8K/assets/train-00000-of-00001.parquet

We won’t need to upload any other files, so we can now focus on writing the code for our Environment.

Write the Dockerfile and requirements

We now need to amend the Dockerfile to download the parquet files:

FROM python:3.11-slim

WORKDIR /app

# Copy requirements and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY server.py .

# Expose port
EXPOSE 8000

CMD curl -o test-00000-of-00001.parquet https://openreward.ai/your-username/GSM8K/assets/test-00000-of-00001.parquet
CMD curl -o train-00000-of-00001.parquet https://openreward.ai/your-username/GSM8K/assets/train-00000-of-00001.parquet

# Start the server
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000"]

We’ll also update the requirements.txt:

fastapi>=0.115.12
openreward
pandas
uvicorn>=0.34.3
math-verify[antlr4_13_2]

Push to GitHub and connect

Next, push your environment to a GitHub repository. Once ready, go to your OpenReward environment and connect the repository: [IMAGE HERE] You will be given a choice for how much compute you would like to allocate for it. We’ll use a low compute configuration since this is a simple environment. Press build and your environment will be hosted!

Sample from your environment

Now your environment is hosted on OpenReward, we can sample from it:

OpenAI
Anthropic
Google
OpenRouter
Other Models

Set your API keys

Make sure you have API keys for OpenReward and OpenAI, and set these as environment variables:

export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'

Create your code

Save this as quickstart.py:

  from openai import OpenAI
  from openreward import OpenReward
  import json

  or_client = OpenReward()
  oai_client = OpenAI()
  MODEL_NAME = "gpt-5"

  environment = or_client.environments.get(name="yourusername/gsm8k")
  tasks = environment.get_tasks(split="train")
  tools = environment.get_tools(format="openai")

  example_task = tasks[0]

  with environment.session(task=example_task) as session:
      prompt = session.get_prompt()
      input_list = [{"role": "user", "content": prompt[0].text}]
      finished = False
      print(input_list)

      while not finished:
          response = oai_client.responses.create(
              model=MODEL_NAME,
              tools=tools,
              input=input_list
          )
          print(response.output)

          input_list += response.output

          for item in response.output:
              if item.type == "function_call":
                  tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))

                  reward = tool_result.reward
                  finished = tool_result.finished

                  input_list.append({
                      "type": "function_call_output",
                      "call_id": item.call_id,
                      "output": json.dumps({
                          "result": tool_result.blocks[0].text
                      })
                  })

                  print(input_list[-1])

                  if tool_result.finished:
                      finished = True
                      break

Run your code

  python quickstart.py

Example output:

[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
[ResponseReasoningItem(id='rs_000dde94c785f76800693413cbb66c81928b57cca2b5488c2d', summary=[], type='reasoning', content=None, encrypted_content=None, status=None), ResponseOutputMessage(id='msg_000dde94c785f76800693413ce541881928b37ef5a6ec49cf9', content=[ResponseOutputText(annotations=[], text='72', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"72"}', call_id='call_Eaw188yiAYpNCgQXduR1x0lR', name='answer', type='function_call', id='fc_000dde94c785f76800693413cf1be08192995f531548dffa4d', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_Eaw188yiAYpNCgQXduR1x0lR', 'output': '{"result": "Correct!"}'}

Set your API keys

Make sure you have API keys for OpenReward and Anthropic, and set these as environment variables:

export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'

Create your code

Save this as quickstart.py:

  import anthropic
  from openreward import OpenReward
  import json

  or_client = OpenReward()
  ant_client = anthropic.Anthropic()
  MODEL_NAME = "claude-sonnet-4-5"

  environment = or_client.environments.get(name="yourusername/gsm8k")
  tasks = environment.get_tasks(split="train")
  tools = environment.get_tools(format="anthropic")

  example_task = tasks[0]

  with environment.session(task=example_task) as session:
      prompt = session.get_prompt()
      messages = [{"role": "user", "content": prompt[0].text}]
      finished = False
      print(messages)

      while not finished:
          response = ant_client.messages.create(
              model=MODEL_NAME,
              max_tokens=4096,
              tools=tools,
              messages=messages
          )

          print(message)
          messages.append(message)

          if message.stop_reason == "tool_use":
              tool_use = next(block for block in message.content if block.type == "tool_use")
              tool_name = tool_use.name
              tool_input = tool_use.input
              
              tool_result = session.call_tool(tool_name, tool_input)
              reward = tool_result.reward
              finished = tool_result.finished

              messages.append({
                  "role": "user",
                  "content": [
                      {
                          "type": "tool_result",
                          "tool_use_id": tool_use.id,
                          "content": tool_result.blocks[0].text
                      }
                  ]
              })

              print(messages[-1])
              
              if tool_result.finished:
                  finished = True
                  break

Run your code

  python quickstart.py

Example output:

[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
Message(id='msg_01DJorrT3UPhnPhdN19ugsyT', content=[TextBlock(citations=None, text='I need to find the total number of clips Natalia sold in April and May.\n\nGiven information:\n- In April, Natalia sold clips to 48 friends\n- In May, she sold half as many clips as in April\n\nLet me calculate:\n- April: 48 clips\n- May: 48 ÷ 2 = 24 clips\n- Total: 48 + 24 = 72 clips', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': '72'}, name='answer', type='tool_use')], model='claude-sonnet-4-5-20250929', role='assistant', stop_reason='tool_use', stop_sequence=None, type='message', usage=Usage(cache_creation=CacheCreation(ephemeral_1h_input_tokens=0, ephemeral_5m_input_tokens=0), cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=608, output_tokens=151, server_tool_use=None, service_tier='standard'))
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}

Set your API keys

Make sure you have API keys for OpenReward and Gemini, and set these as environment variables:

export GEMINI_API_KEY='your-gemini-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'

Create your code

Save this as quickstart.py:

  from google import genai
  from google.genai import types
  from openreward import OpenReward
  import json

  or_client = OpenReward()
  gem_client = genai.Client()
  MODEL_NAME = "gemini-2.5-flash"

  environment = or_client.environments.get(name="yourusername/gsm8k")
  tasks = environment.get_tasks(split="train")
  tools = environment.get_tools(format="google")

  genai_tools = [types.Tool(function_declarations=[f]) for f in tools]
  genai_config = types.GenerateContentConfig(tools=genai_tools)

  example_task = tasks[0]

  with environment.session(task=example_task) as session:
      prompt = session.get_prompt()
      contents = [
          types.Content(
              role="user", parts=[types.Part(text=prompt[0].text)]
          )
      ]
      finished = False
      print(contents)

      while not finished:

          response = gem_client.models.generate_content(
              model=MODEL_NAME,
              config=genai_config,
              contents=contents
          )

          print(response.candidates[0].content)

          contents.append(response.candidates[0].content) # Append the content from the model's response.

          for part in response.candidates[0].content.parts:
              if part.function_call:
                  tool_call = part.function_call
                  tool_result = session.call_tool(tool_call.name, tool_call.args)

                  reward = tool_result.reward
                  finished = tool_result.finished

                  function_response_part = types.Part.from_function_response(
                      name=tool_call.name,
                      response={"result": json.dumps({
                          "result": tool_result.blocks[0].text
                      })},
                  )

                  contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response

                  print(contents[-1])

                  if tool_result.finished:
                      finished = True
                      break

Run your code

  python quickstart.py

Example output:

[Content(
parts=[
    Part(
    text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'
    ),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
    args={
    'answer': 'Natalia sold 72 clips altogether in April and May.'
    },
    name='answer'
),
thought_signature=b'\n\xe3\x02\x01r\xc8\xda|#w\x85/\x9e\xd1\xe7mK\xc1\xdd\xf6X\xeayY\x12<\x0fQ\x1608\xedKg\x07\xe3tV\xd0\xe3\xaf\xc6=\xc0.x\x15\x85\xeb\xec+\x84\xad\xf0\x02\xb6\xf8\x88iX\x94\xf8\xcb8g\xf9m\x02\xcd`\xb1\x8fm\x1e\x10\x1a\x14]2\x81H\x7f-\xb6E6\xd8\xf3\xa6kUw\xb7\xc3kr\x14...'
)] role='model'
parts=[Part(
function_response=FunctionResponse(
    name='answer',
    response={
    'result': '{"result": "Correct!"}'
    }
)
)] role='user'

Set your API keys

Make sure you have API keys for OpenReward and OpenRouter, and set these as environment variables:

export OPENREWARD_API_KEY='your-openreward-api-key-here'

Create your code

Save this as quickstart.py:

from openai import OpenAI
from openreward import OpenReward
import json
import os

or_client = OpenReward()
oai_client = OpenAI(
  base_url="https://openrouter.ai/api/v1",
  api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"

environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.get_tasks(split="train")
tools = environment.get_tools(format="openrouter")

example_task = tasks[0]

with environment.session(task=example_task) as session:
    prompt = await session.get_prompt()
    input_list = [{"role": "user", "content": prompt[0].text}]
    finished = False
    print(input_list)

    while not finished:
        response = oai_client.chat.completions.create(
            model=MODEL_NAME,
            tools=tools,
            messages=input_list
        )
        print(response)

        input_list.append(response)

        tool_calls = response.choices[0].message.tool_calls
        if not tool_calls:
            return
        
        for tool_call in response.choices[0].message.tool_calls:
            tool_name = tool_call.function.name
            tool_args = json.loads(tool_call.function.arguments)
            tool_result = await session.call_tool(tool_name, tool_args)

            reward = tool_result.reward
            finished = tool_result.finished

            input_list.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": json.dumps({
                    "result": tool_result.blocks[0].text
                })
            })

            print(input_list[-1])

            if tool_result.finished:
                finished = True
                break

Run your code

  python quickstart.py

Example output:

    [{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
    ChatCompletion(id='gen-1765024924-6BK4o9Oiefjb7Fe7J7eW', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content="Let's break this down step by step.\n\n1. In April: Natalia sold clips to **48 friends**. This means she sold **48 clips** (assuming one clip per friend).\n2. In May: She sold **half as many clips** as in April.\n   - Half of 48 is \\( 48 \\div 2 = 24 \\).\n3. Altogether in April and May:\n   \\[\n   48 + 24 = 72\n   \\]\n\nSo Natalia sold **72 clips** altogether in April and May.\n\n", refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=[ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "72"}', name='answer'), type='function', index=0)], reasoning=None), native_finish_reason='tool_calls')], created=1765024924, model='deepseek/deepseek-v3.2', object='chat.completion', service_tier=None, system_fingerprint='', usage=CompletionUsage(completion_tokens=153, prompt_tokens=333, total_tokens=486, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, image_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, video_tokens=0), cost=0.00015417, is_byok=False, cost_details={'upstream_inference_cost': None, 'upstream_inference_prompt_cost': 8.991e-05, 'upstream_inference_completions_cost': 6.426e-05}), provider='SiliconFlow')
    {'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': '{"result": "Correct!"}'}

If you are running with another provider, or using custom models, then here are the main principles to keep in mind.

Set your API key

Make sure you have API keys for OpenReward and OpenRouter, and set these as environment variables:

export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'

Get the environment, tools, and tasks

You’ll need to use the OpenReward client to access the environment and its main information

from openreward import OpenReward
or_client = OpenReward()

environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.get_tasks(split="train")
tools = environment.get_tools(format="openai")
example_task = tasks[0]

Start an environment session

async with environment.session(task=example_task) as session:

Using a context manager, we can start a session with the environment. This defines a scope in which you can use the agent to call tools and get tool results.Above we have selected the first task to sample from, which is a particular problem in the MLEBench-Train environment.

Pass the prompt and tools list into your model

You can get the prompt for the task as follows:

prompt = await session.get_prompt()

You will also need to pass in the tools into the context window of your model, usually somewhere in the system prompt.

Define the core agent loop

An agent will usually keep interacting with an environment until it hits a termination state associated with the environment, or some other imposed limit (e.g. maximum number of terms).In a simple sequential agent model, this could be a while loop like:

while not finished:

where finished is a boolean.

Parse and execute tool calls

In agentic environments, actions are treated as tool calls. That means you need to have a way to parse tool calls from your model’s generations.The key thing to note is that for OpenReward environment, a tool call requires specifying a name (str) and some arguments (dict).To call a tool you will call:

   tool_result = await session.call_tool(tool_name, tool_arguments)

This means that you will need to parse out the tool_name and tool_arguments from your model’s generation and then parse this information into the call_tool method.

Parse and execute tool results

If you have executed an available tool correctly, you will receive a ToolOutput output. This contains attributes for:

reward : an (optional) float denoting reward. For example, submitting a correct math solution through a submit_solution tool might give a reward of 1.0.
finished : a bool specifying whether the episode is finished or not. For example, some tools may end the episode (if for example an agent submits a final answer through submit_solution in a math task).
data : a dict with output of the executing of the tool. For example, the stdout of executing a bash tool. This information should be passed to the agent as feedback.

For our agent loop, some example control flow we might want after receiving a ToolOutput is:

Recording reward for use in a policy gradient algorithm such as GRPO or PPO
Breaking out of the core agent loop if finished=True
Adding data to the context window as feedback and continuing with the next model generation

Next Steps

Build a Computer environment

Build a more advanced environment that uses Computers

Evaluate with OpenReward

Learn how to run evals with OpenReward.

Train with OpenReward

Learn how to use OpenReward for reinforcement learning

Get started

Using Environments

Making Environments

Your first environment

Goals

Prerequisites

Setup

Background: GSM8K

Understanding ORS servers

Building the GSM8K environment

Host on OpenReward

Upload environment assets

Write the Dockerfile and requirements

Push to GitHub and connect

Sample from your environment

Next Steps

Build a Computer environment

Evaluate with OpenReward

Train with OpenReward

Get started

Using Environments

Making Environments

​Goals

​Prerequisites

​Setup

​Background: GSM8K

​Understanding ORS servers

​Building the GSM8K environment

​Host on OpenReward

​Upload environment assets

​Write the Dockerfile and requirements

​Push to GitHub and connect

​Sample from your environment

​Next Steps

Build a Computer environment

Evaluate with OpenReward

Train with OpenReward

Goals

Prerequisites

Setup

Background: GSM8K

Understanding ORS servers

Building the GSM8K environment

Host on OpenReward

Upload environment assets

Write the Dockerfile and requirements

Push to GitHub and connect

Sample from your environment

Next Steps