# Backdated Search & Fetch API
Source: https://docs.openreward.ai/api-reference/backdated-search
A point-in-time web search and page-fetch API over a frozen news archive
## Overview
The Backdated Search & Fetch API answers one question no live search API can:
**"what did the web know on date X?"**
Every request carries an `as_of` date. Search returns only documents crawled on
or before that date, and fetch returns the article's bytes **as they were
archived at that time**. The backing corpus is
a frozen archive, so the same `as_of` query returns the same results forever.
This is the same engine behind the [`BackSearchToolset`](/environments/backdated-web-tools)
you can declare in an environment, available here as a standalone API.
This API never touches the live web: it searches an immutable crawl, so results are **reproducible** and carry
**no post-cutoff lookahead**. That makes it suitable for leakage-free evals,
forecasting benchmarks, and defensible point-in-time backtests.
## Base URL & authentication
```
https://search.openreward.ai
```
Authenticate with your OpenReward API key in the `x-api-key` header (`or_...` key).
```bash theme={null}
export OPENREWARD_API_KEY="or_xxxxxxxx"
```
Search runs over a fixed historical corpus. Choose `as_of` dates **inside the
covered window**. Current coverage: **December 2025 → June 2026**
## Pricing
Pay-as-you-go against your prepaid OpenReward balance.
| Endpoint | Price |
| -------------- | --------------------------------------- |
| `POST /search` | \*\*$10 / 1,000 requests** ($0.01 each) |
| `POST /fetch` | \*\*$2 / 1,000 requests** ($0.002 each) |
Only successful requests are billed. A search that errors, or a fetch that finds
no capture on or before `as_of` (a `404`), is **not** charged. When your balance
is exhausted the API returns [`402 Payment Required`](#budget--errors). Top up
to continue.
***
## POST /search
Full-text + semantic search over the archive, cut off at `as_of`.
### Request body
| Field | Type | Required | Description |
| ----------------- | --------- | -------- | --------------------------------------------------------------------------------------------------------------------------- |
| `query` | string | **yes** | The search query. |
| `as_of` | string | **yes** | Point-in-time cutoff, ISO `YYYY-MM-DD`. Only documents crawled on or before this date (end-of-day, inclusive) are returned. |
| `k` | integer | no | Number of hits to return. Default `10`. |
| `mode` | string | no | `hybrid` (BM25 + E5 vector fusion, best quality — default), `bm25` (fast lexical), or `dense` (pure vector). |
| `lang` | string | no | ISO-639-1 language filter, e.g. `en`. |
| `site` | string | no | Restrict to a single host, e.g. `reuters.com`. |
| `allowed_domains` | string\[] | no | Only return results from these hosts. Mutually exclusive with `blocked_domains`. |
| `blocked_domains` | string\[] | no | Exclude these hosts. |
| `sort` | string | no | `relevance` (default) or `newest`. |
### Response
```json theme={null}
{
"mode": "hybrid",
"candidates": 25,
"hits": [
{
"url": "https://example.com/article",
"title": "Article title",
"snippet": "first ~200 chars of article text",
"crawl_date": "2025-12-01T17:33:40Z",
"publish_date": "2025-12-01T00:00:00Z",
"host": "example.com",
"score": 0.976
}
],
"timing": { "total_ms": 587.3 }
}
```
| Field | Description |
| --------------------- | --------------------------------------------------------------------------------------------- |
| `hits[].crawl_date` | When the page was archived. **This is what `as_of` gates on** — never later than your cutoff. |
| `hits[].publish_date` | The article's own stated publish date (best-effort; may be null). |
| `hits[].score` | Rank pseudo-score in `[0,1]` (`1.0` = top hit). In `hybrid` it's the fused score. |
| `candidates` | Unique-URL candidates considered before rerank (meaningful in `hybrid`). |
### Example
```bash theme={null}
curl -X POST https://search.openreward.ai/search \
-H "x-api-key: $OPENREWARD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "central bank statement on interest rates",
"as_of": "2026-01-15",
"k": 5,
"mode": "hybrid"
}'
```
```python theme={null}
import os, requests
resp = requests.post(
"https://search.openreward.ai/search",
headers={"x-api-key": os.environ["OPENREWARD_API_KEY"]},
json={
"query": "central bank statement on interest rates",
"as_of": "2026-01-15",
"k": 5,
"mode": "hybrid",
},
)
for hit in resp.json()["hits"]:
print(hit["crawl_date"], hit["title"], hit["url"])
```
`hybrid` handles paraphrase that lexical search misses. A query for
*"Eastern European territorial dispute escalation"* surfaces Russia/Ukraine
coverage even when those keywords aren't present.
***
## POST /fetch
Retrieve a page's content **as it was archived on or before `as_of`**, the
point-in-time counterpart to search. Returns the extracted article text.
### Request body
| Field | Type | Required | Description |
| -------------- | ------- | -------- | ----------------------------------------------------------------------- |
| `url` | string | **yes** | The page to fetch. |
| `as_of` | string | **yes** | Cutoff `YYYY-MM-DD`. Returns the latest capture on or before this date. |
| `summarize` | boolean | no | If `true`, also return a summary of the page. |
| `prompt` | string | no | Guiding prompt for the summary (with `summarize: true`). |
| `include_html` | boolean | no | Include the raw archived HTML alongside extracted text. |
### Response
```json theme={null}
{
"url": "https://example.com/article",
"crawl_date": "2026-01-14T09:12:03Z",
"publish_date": "2026-01-14T00:00:00Z",
"title": "Article title",
"text": "full extracted article text as archived…",
"summary": null,
"host": "example.com"
}
```
If there is no capture of the URL on or before `as_of`, the API returns `404`
(and does not bill the request).
### Example
```bash theme={null}
curl -X POST https://search.openreward.ai/fetch \
-H "x-api-key: $OPENREWARD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/article",
"as_of": "2026-01-15"
}'
```
```python theme={null}
import os, requests
resp = requests.post(
"https://search.openreward.ai/fetch",
headers={"x-api-key": os.environ["OPENREWARD_API_KEY"]},
json={"url": "https://example.com/article", "as_of": "2026-01-15"},
)
page = resp.json()
print(page["crawl_date"], page["title"])
print(page["text"][:500])
```
`as_of` gates on **crawl date**, not the article's stated publish date. A page
that was first archived after your cutoff will not be returned even if it claims
an earlier publish date.
***
## Budget & errors
| Status | Meaning |
| ------ | --------------------------------------------------------------------------------------------------- |
| `200` | Success (billed). |
| `400` | Bad request — e.g. both `allowed_domains` and `blocked_domains` set, or an invalid URL. Not billed. |
| `401` | Missing `x-api-key`. |
| `403` | Invalid or disabled API key. |
| `402` | **Balance exhausted.** Top up to continue. Not billed. |
| `404` | (fetch) No capture of the URL on or before `as_of`. Not billed. |
## Checking your usage
`GET https://api.openreward.ai/v1/billing/api-usage` returns metered search and
fetch usage for the account behind your API key — requests, billable cost.
### Query parameters
| Field | Type | Description |
| ------------- | ------- | ----------------------------------------------------- |
| `days` | integer | Window to report, 1–90. Default `30`. |
| `service` | string | Filter to `search` or `fetch`. Omit for both. |
| `granularity` | string | `total` (default) or `daily` for a per-day breakdown. |
### Response
```json theme={null}
{
"since": "2026-06-21T00:00:00.000Z",
"until": "2026-07-21T08:11:29.856Z",
"byService": [
{ "service": "search", "requests": 1200, "units": "1200.000", "cost": "12.000000", "pendingRequests": 0 },
{ "service": "fetch", "requests": 300, "units": "300.000", "cost": "0.600000", "pendingRequests": 0 }
],
"totals": { "requests": 1500, "cost": "12.600000", "pendingRequests": 0 }
}
```
```bash theme={null}
curl "https://api.openreward.ai/v1/billing/api-usage?days=30&granularity=daily" \
-H "x-api-key: $OPENREWARD_API_KEY"
```
```bash theme={null}
orwd usage # summary for the last 30 days
orwd usage --daily # broken down by day
orwd usage --service search
```
## Adding credits
**To top up** open your organization's
billing page at `openreward.ai//billing` (for example,
[openreward.ai/GeneralReasoning/billing](https://openreward.ai/GeneralReasoning/billing)).
## See also
* [Backdated Web Tools](/environments/backdated-web-tools) — the same engine as
an environment toolset (`BackSearchToolset`) or SDK classes (`Backsearch` /
`Backfetch`).
# Architecture Overview
Source: https://docs.openreward.ai/concepts/architecture
Understanding OpenReward environments and sandboxes
OpenReward provides infrastructure for hosting and running AI agent environments. This infrastructure has two main components:
* **Environments** specify tasks, tools, rewards and basic state.
* **Sandboxes** (optional) give the agent access to a computer in the environment.
Note that not all environments require sandboxes, but this is becoming increasingly common as we train agents on computer-based tasks. This is often called **agentic reinforcement learning**.
## What are environments?
An environment is a simulated space where an agent can perform a task using tools and resources, and receive rewards for good or bad actions.
In the language of reinforcement learning, the underlying abstraction is a [POMDP](https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process), which specifies a system that an agent interacts with, the actions that allow the agents to interact with it, and rewards for the agent.
On OpenReward, we treat environments as passive FastAPI-style servers that an agent can interact with. This maintains strict separation of concerns between the agent and the environment, allowing for easier development and more robustness to changes in agent harnesses.
The standard for how agents talk to environments is called **ORS**, which [you can read about in full detail here](https://openrewardstandard.io).
Briefly speaking, an ORS server provides:
* **Tasks** - tasks are the core problems to be solved, including the initial prompts
* **Tools** - tools are the actions an agent can take in the environment
* **Splits** - splits organise tasks into groups, e.g. for training and evaluation
* **Statefulness** - agent actions in a session can affect state
* **Tool Results** - including tool feedback, rewards and termination signals
But at its heart, an ORS service is a web service that you can call. You can run these servers locally, or in the cloud. OpenReward is a managed service for hosting ORS environments on the internet.
## What are sandboxes?
A **sandbox** is an isolated container for running code. They are often used in conjunction with environments when an agent needs access to a computer. For example, a software engineering task would require interacting with a filesystem, writing and executing files, and more. We use sandboxes, rather than the same compute as the environment, to isolate the agent's actions - and prevent them from interfering with the running of the environment!
A sandbox provides:
* **Isolated execution** environment
* **Configurable resources** (CPU, memory)
* **Network isolation** options
* **Automatic cleanup** after use
In the context of ORS, a sandbox is usually initialised when a new session is created, and torn down when the session ends. But the exact use can vary; for example, some environments may only use a sandbox for a particular tool execution instead of the entire session.
ORS environments can be run with any sandbox provider - it is fully interoperable. OpenReward provides our own sandbox solution, but we encourage you to use whatever works best for you - and these docs contain detailed guides on how to use environments with popular providers such as Daytona, E2B and Modal.
## How environments and sandboxes work together
We have briefly touched upon how these two pillars of infrastructure interact, but we consider a few popular patterns below.
**Pattern 1: Environment-Only**
For environments that don't need code execution:
```text theme={null}
Agent → ORS Environment
```
Examples include mathematics or multiple-choice question answering benchmarks where the agent is not given access to a computer.
**Pattern 2: Environment + Sandboxes**
For environments requiring code execution:
```text theme={null}
Agent → ORS Environment → Sandbox
```
Examples include software engineering and knowledge-work benchmarks where the agent reads and writes to a filesystem. The agent connects to an ORS environment, which then initialises a sandbox and utilises it for certain tools in the environment (e.g. bash, write, read, grep).
## Where is data stored?
Data on OpenReward can come from external sources, for example HuggingFace datasets or public buckets, or you can upload assets to the environment and use those.
There are two main uses for data in ORS environments:
* **Environment data**. This is data that powers the underlying environment; for example, tasks that are contained in a parquet or .jsonl, or it could be information the environment but not the agent has access to (e.g. ground truth labels).
* **Sandbox data**. This is data that the agent has access to for solving the task. For example, in a Kaggle competition environment, the agent might have access to a train and validation dataset on their machine to build models with.
We have a [full set of docs on where data lives here](/environments/where-environment-data-lives). The key things to note about the use of OpenReward storage is that:
* The hosted environment mounts storage at the location: `/orwd_data/`
* Sandboxes can choose the path to mount the storage at
## Deployment Flow
OpenReward deploys ORS servers, and we do this via the following flow:
```text theme={null}
1. The user writes environment code along with a Dockerimage
↓
2. The user pushes their code to a GitHub repository
↓
3. The user connects an OpenReward environment to their GitHub
↓
4. OpenReward builds an image and deploys an environment server
↓
5. The server automatically scales based on connections
```
OpenReward does not host code; that stays on GitHub. In essence it does one job which is to take the code (along with any data) and host an API endpoint for it. That endpoint can then be used for training, distillation or evaluation.
## Getting Started
Learn about environment servers
Explore ephemeral execution containers
Configure cloud storage access
Deploy your first environment
# Environments
Source: https://docs.openreward.ai/concepts/environments
Long-running ORS servers with automatic scaling
An environment on OpenReward is a long-running server that implements the **[Open Reward Standard (ORS)](https://openrewardstandard.io)** for agent environments.
When an environment is created on OpenReward, a workspace is provisioned and an API endpoint provided.
Environments provide:
* **Tasks** - tasks are the core problems to be solved, including the initial prompts
* **Tools** - tools are the actions an agent can take in the environment
* **Splits** - splits organise tasks into groups, e.g. for training and evaluation
* **Statefulness** - agent actions in a session can affect state
* **Tool Results** - including tool feedback, rewards and termination signals
Agents connect to environments via sessions, which are stateful connections.
ORS environments can be run anywhere. OpenReward hosts them and provides managed infrastructure, for example autoscaling based on demand.
## Environment Lifecycle
### 1. Creation
Create environments through the [OpenReward website](https://openreward.ai):
```text theme={null}
1. In header of the website, click the + button
↓
2. Click "New environment"
↓
3. Enter name and description
↓
4. Set visibility (public/private)
↓
5. Environment created with workspace
```
### 2. Deployment
Once you have created your environment, you can deploy your [ORS](https://openrewardstandard.io) server code to it.
First, you will need to develop your environment locally. You can read the [local development guide](/deployment/local-development) for more details on how to develop your environment server locally.
Once you are ready to deploy, you should undertake the following steps:
```text theme={null}
1. Upload your code to a GitHub repository, along with a Dockerimage
↓
2. Connect the GitHub repository to your environment
↓
3. OpenReward automatically builds and deploys whenever you push to main
↓
4. Environment server starts accepting sessions
```
See [the GitHub integration guide](/deployment/github-integration) for more details.
### 3. Running
Once deployed, agents can connect to your environment:
**Agent interaction:**
```python theme={null}
from openreward import AsyncOpenReward
client = AsyncOpenReward(api_key="your-api-key")
# Get environment
environment = client.environments.get(name="username/my-environment")
# List available tasks
tasks = await environment.list_tasks(split="train")
# Or fetch a single task by index
task = await environment.get_task(split="train", index=0)
# Or get a range of tasks by index (supports negative indices and None)
# Task ordering is consistent across calls, so ranges are safe to partition
tasks = await environment.get_task_range(split="train", start=0, stop=10)
# Create session
async with environment.session(task=tasks[0]) as session:
# Get task prompt
prompt = await session.get_prompt()
# Call tools
result = await session.call_tool("bash", {"command": "ls"})
# Environment returns results directly
print(result)
```
### 4. Automatic Scaling
Environments automatically scale based on concurrent sessions. Autoscaling settings are configurable when making the environment and can be changed in the environment settings page.
**Benefits:**
* Pay only for active usage
* No manual scaling needed
* Always ready when agents connect
### 5. Updates
Pushing to the connected GitHub repository will automatically build and deploy a new version of the environment.
**Via GitHub (automatic):**
```text theme={null}
1. Push new code to GitHub
↓
2. OpenReward detects changes
↓
3. Builds new version
↓
4. Deploys the new version of the environment server
```
## Sessions
A **session** is a durable, stateful connection between an agent and your environment:
* **Unique**: Each session has a unique ID, this is internally used to make sure you're connecting to the correct environment server.
* **Stateful**: Environment can maintain state between tool calls
* **Time-limited**: Sessions expire after inactivity or when the session is deleted.
* **Sticky**: Session stays connected to the same server instance
### Using Sessions
When creating a session, you can pass a task object directly or reference a task by its split and index:
```python theme={null}
# Using a task object
async with environment.session(task=tasks[0]) as session:
...
# Using split and index
async with environment.session(split="train", index=0) as session:
...
```
Using split and index avoids fetching the full task list. See the [ways to access tasks](/environments/ways-to-access-tasks) guide for more detail.
**Python SDK:**
```python theme={null}
# Create session with context manager (automatic cleanup)
with environment.session(task=task) as session:
# Get initial prompt
prompt = session.get_prompt()
# Call tools multiple times
result1 = session.call_tool("read_file", {"path": "README.md"})
result2 = session.call_tool("bash", {"command": "python test.py"})
# Session automatically cleaned up on exit
```
Conversation-shaped tasks (a dialogue history the model must continue) can serve the prompt as role-tagged messages instead of a single block of text — see [Multi-turn Conversations](/environments/multi-turn-conversations). `get_prompt()` remains the default surface.
**Session lifecycle:**
```text theme={null}
1. Agent creates session → Connected to environment
2. Agent calls tools → Results returned directly
3. Context manager exits → Session automatically cleaned up
```
Sessions are **episodes** in the reinforcement learning sense - they persist across multiple tool calls until the caller breaks out of the session.
In ORS, a tool result with `finished=True` signals that the episode is complete and the agent should end the session. It is the caller's responsibility to check the `finished` flag and break out of the loop accordingly.
An environment may additionally implement **group scoring**: after a trainer collects a group of completed rollouts for one task (as GRPO-style algorithms do), it can submit all of the candidate trajectories to the environment in a single call and receive a comparative reward for each — see [Preference Rewards](/environments/preference-rewards). Within a session, an environment still only ever sees its own rollout.
### Tool Calling
In ORS, the only way agents interact with environments is by calling tools. This leverages existing function calling support from LLM providers, creating a natural action space for language model agents. It also enforces a clean boundary between agent and environment - environments make no assumptions about the agent interacting with them, which keeps them robust to changes in how agents are built.
**Example tool call:**
```python theme={null}
result = session.call_tool(
tool_name="bash",
input={"command": "ls -la"}
)
# Result returned directly from environment
print(result.blocks) # Tool output (list of TextBlock/ImageBlock)
print(result.finished) # Task completion status
print(result.reward) # Performance feedback
```
**Tool results structure:**
* `blocks`: The tool's return value (list of TextBlock/ImageBlock)
* `metadata`: Optional metadata dictionary
* `finished`: Whether the task is complete
* `reward`: Numeric feedback signal
## Storage
### Cloud Storage Integration
Each environment includes cloud storage accessible at `/orwd_data/`:
**Usage in environment server:**
```python theme={null}
from pathlib import Path
# Storage mounted at /orwd_data/
storage = Path("/orwd_data")
# Read dataset
with open(storage / "datasets" / "train.csv") as f:
data = f.read()
# Write results
with open(storage / "results" / "metrics.json", "w") as f:
json.dump(metrics, f)
```
Data in `/orwd_data/` can also be made accessible to sandboxes, depending on how you configure the mount.
See [where environment data lives](/environments/where-environment-data-lives) for a conceptual overview, and [Storage & Buckets](/storage/buckets) for lower-level configuration.
## Next Steps
Build and deploy your first environment
Add isolated code execution to your environment
Set up automatic deployments from GitHub
Configure persistent storage for your environment
# Sandboxes
Source: https://docs.openreward.ai/concepts/sandboxes
Execution containers for agent code
A **sandbox** is an isolated container for executing arbitrary code. Sandboxes provide secure, temporary compute resources for running untrusted agent code in an [ORS environment](https://openrewardstandard.io).
**Key characteristics:**
* **On-Demand**: Created when needed, not pre-provisioned
* **Isolated**: Strong security boundaries with optional network blocking
* **Flexible**: Configurable image, resources, and environment variables
## Why do environments use sandboxes?
Sandboxes are used in conjunction with environments when an agent needs access to a computer. For example, a software engineering task requires interacting with a filesystem, writing and executing files, and more. We use sandboxes, rather than the same compute as the environment, to isolate the agent's actions and prevent them from interfering with the running of the environment.
In the context of ORS, a sandbox is usually initialised when a new session is created and torn down when the session ends. But the exact use can vary - some environments may only use a sandbox for a particular tool execution instead of the entire session.
## How to use sandboxes with ORS
ORS environments work with any sandbox provider. We have detailed guides for how to use popular providers, such as:
* **Using E2B**: See [E2B Sandboxes](../sandboxes/e2b)
* **Using Daytona**: See [Daytona Sandboxes](../sandboxes/daytona)
* **Using Modal**: See [Modal Sandboxes](../sandboxes/modal)
OpenReward also provides an internal sandbox solution - **OpenReward Sandboxes** - which we detail in the rest of this documentation.
## OpenReward Sandboxes
OpenReward provides sandbox infrastructure that integrates seamlessly with environment file storage. The sections below show how to create and use OpenReward sandboxes in your environments. Much of the initial environments on the site were built using this solution, so we detail it in full below.
## Sandbox Lifecycle
### 1. Creation
OpenReward Sandboxes are tied to an environment workspace on OpenReward which provisions compute. This means it is not a general sandbox compute solution: it is meant to be tied to environment use.
An environment workspace is just an environment on the OpenReward platform. For example, `GeneralReasoning/CTF` has its own workspace. This [CTF environment](https://openreward.ai/GeneralReasoning/CTF) also uses sandboxes as part of the design of the environment.
A typical workflow is as follows:
1. You want to create a new ORS environment and host it on OpenReward
2. You know your environment needs a sandbox for the agent to access a computer
3. To provision compute, you create the environment on [OpenReward](https://openreward.ai) - e.g. `username/myenvironment`.
4. When developing your code locally, you reference this namespace in your sandbox settings
Note this happens before you push any code to GitHub. Your code is still local at this point in time. The key point is you still need to set up the environment workspace on OpenReward to get access to the sandbox compute for local development.
In the Python SDK, the Sandbox API looks as follows:
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings
client = AsyncOpenReward(api_key="your-api-key")
settings = SandboxSettings(
environment="username/environment-name", # Required: your environment namespace
image="python:3.11-slim", # Required: container image
machine_size="1:2", # Required: CPU:Memory
env={"DEBUG": "true"}, # Optional: environment variables
block_network=False, # Optional: network isolation
)
sandbox = client.sandbox(
settings=settings,
)
await sandbox.start() # Creates the sandbox
```
You are then billed based on the time spent in the sandbox. Note that when you host an environment, you do not pay the sandbox costs for someone using your environment; this will be billed to their account. However, if you are calling that hosted environment yourself - or you are developing with a sandbox locally - then it will be billed to you.
Billing works on the basis of your API key. You will need to set this; either by passing in `api_key` to the `OpenReward` client (or `AsyncOpenReward` client as is recommended), or by setting an environment variable:
```bash theme={null}
export OPENREWARD_API_KEY='your-api-key'
```
In the context of an ORS environment, a popular design pattern is as follows.
First, we initialise the sandbox within the **init** of the Environment:
```python theme={null}
from openreward.environments import Environment
from openreward import SandboxSettings, SandboxBucketConfig, AsyncOpenReward
...
class MyEnvironment(Environment):
def __init__(self, task_spec: JSONObject, secrets: dict[str, str] = {}) -> None:
super().__init__(task_spec, secrets=secrets)
self.sandbox_settings = SandboxSettings(
environment="Username/MyEnvironment",
image="generalreasoning/python-ds:3.12-tools",
machine_size="2:2",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/datasets",
read_only=True,
only_dir="agent_data",
)
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key", ""))
self.sandbox = or_client.sandbox(self.sandbox_settings)
```
Then we define `setup` and `teardown` methods so the sandbox starts when a new session begins, and ends when it finishes:
```python theme={null}
async def setup(self) -> None:
await self.sandbox.start()
```
```python theme={null}
async def teardown(self) -> None:
await self.sandbox.stop()
```
This ties the sandbox's lifetime to that of the agent's session. This pattern is extremely common and will cover the majority of usecases with ORS environments.
However, there may be exceptions. For example, if the sandbox compute is expensive and is only needed with specific tools, you may want to design the sandbox lifetime so it only exists during the invocation of that specific tool. A specific example would be an environment that needs GPU access - e.g. evaluating a GPU kernel in a kernel optimisation environment would not need the GPU to be available while, e.g. the agent was reasoning or writing code.
### 2. Command Execution
Once a sandbox is created, you can execute commands within it:
```python theme={null}
# Run a command
output, exit_code = await sandbox.run("python script.py")
if exit_code == 0:
print(f"Success: {output}")
# Run with check (raises on non-zero exit)
output = await sandbox.check_run("pytest tests/")
# Upload file
await sandbox.upload(
local_path="./data.csv",
container_path="/app/input.csv"
)
# Download file
content = await sandbox.download("/app/output.json")
```
**Available operations:**
* `run()`: Execute command and return output
* `check_run()`: Execute command, raise on error if it returns a non-zero exit code
* `upload()`: Upload files to sandbox
* `download()`: Download files from sandbox
Usually when developing an environment, you will use these operations inside a tool method for your environment. For example, here is a bash tool that uses these methods:
```python theme={null}
@tool
async def bash(self, params: BashParams) -> ToolOutput:
"""Execute bash commands using the computer instance."""
try:
output, code = await self.sandbox.run(params.command.strip())
return ToolOutput(
blocks=[TextBlock(text=f"{output}\n\n(exit {code})")],
metadata={"output": output, "exit_code": code},
reward=0.0,
finished=False,
)
except Exception as e:
return ToolOutput(
metadata={"error": str(e)},
blocks=[TextBlock(text=f"Error executing command: {str(e)}")],
finished=False
)
```
### 3. Deletion
To delete an OpenReward Sandbox, we use the `stop` method:
```python theme={null}
await sandbox.stop()
```
For example, this is often used as part of a teardown method in an ORS environment:
```python theme={null}
async def teardown(self) -> None:
await self.sandbox.stop()
```
which will terminate the sandbox as soon as the session ends.
## Storage Integration
### Cloud Storage Mounts
Sandboxes can access files uploaded to your OpenReward environment. To do this, you'll need to use the `SandboxBucketConfig` class. For example:
```python theme={null}
from openreward import SandboxSettings, SandboxBucketConfig
settings = SandboxSettings(
environment="username/env-name",
image="python:3.11-slim",
machine_size="1:2",
bucket_config=SandboxBucketConfig(
mount_path="/workspace", # Where to mount in container
only_dir="datasets/subset", # Optional: mount only subdirectory
)
)
```
Here the `/workspace` is the mount\_path on the agent's computer. In other words, the agent will find the files in that location if it explores the filesystem.
The `only_dir` argument specifies that we only mount a specific directory from the `OpenReward` storage for that environment. This is important to ensure agents do not see files we don't want them to see - e.g. ground truth labels.
See [where environment data lives](/environments/where-environment-data-lives) for a conceptual overview, and [Storage & Buckets](/storage/buckets) for lower-level configuration.
## Other configuration options
### Machine sizes
As part of the `SandboxSettings`, you need to specify a machine size - i.e. the CPU and RAM available to the agent.
You can iterate with these settings, but it's important to consider a number of things:
1. **Is the agent dealing with big files?**. If the RAM isn't big enough then the sandbox could suffer from out-of-memory errors (OOMs).
2. **Does the agent need to conduct expensive workloads?**. For example if the agent is training machine learning models, then it might have need for more CPUs (and perhaps GPUs too).
3. **Am I underelicitating agent performance?**. If you don't give the agents enough resources, you may underelicit the agent's true capabilities, because it can do less with less compute. This is particularly important for evaluation.
A list of different machine configurations is provided below:
```python theme={null}
machine_size="CPU:Memory"
# Available sizes:
"0.5:0.5" # Light balanced — simple services, low traffic
"1:1" # Baseline balanced — small production services
"2:2" # Solid balanced — steady traffic, moderate load
"4:4" # Heavy balanced — CPU-bound or busy services
"0.5:1" # Memory-biased light — small services with caching
"1:2" # Memory-biased baseline — typical web apps
"2:4" # Memory-biased heavy — APIs, JVMs, Python services
"4:8" # Memory-heavy — large caches, in-memory workloads
"0.5:2" # Low CPU, high memory — background workers, queues
"1:4" # High memory — analytics, batch jobs
"2:8" # Very high memory — data processing, embeddings
"4:16" # Maxed memory — serious data or model workloads
"nvidia-l4" # NVIDIA L4 GPU
```
### Network Isolation
In some cases you may want to limit network access. By default all egress - all internet access - is allowed, but there may be use cases where this is problematic:
1. **Web acccess could allow the agent to cheat** - e.g. by finding the ground truth answers to the dataset.
2. **Web access could leak future information** - for example, some environments rely on backtests on past data, whereas web access would give it access to information from the future.
3. **Safety concerns from web access** - in some scenarios, especially safety testing, we may not want the agent to have access to the internet.
To enable network isolation, set `block_network=True`:
```python theme={null}
settings = SandboxSettings(
environment="username/env",
image="python:3.11",
machine_size="1:2",
block_network=True
)
```
### Sidecar Containers
In some situations, you may want to add additional containers to run alongside your main container. This is possible through sidecars:
```python theme={null}
from openreward import SandboxSidecarContainer
settings = SandboxSettings(
environment="username/env",
image="python:3.11",
machine_size="1:2",
sidecars=[
SandboxSidecarContainer(
name="redis",
image="redis:7-alpine",
env={"REDIS_PASSWORD": "secret"},
command=["redis-server"],
args=["--maxmemory", "256mb"],
ports=[6379]
)
]
)
```
**Use cases:**
* Running databases (Redis, PostgreSQL)
* Starting services (web servers, message queues)
* Running monitoring tools
### Host Aliases
You can add custom DNS entries to `/etc/hosts`:
```python theme={null}
from openreward import SandboxHostAlias
settings = SandboxSettings(
environment="username/env",
image="python:3.11",
machine_size="1:2",
host_aliases=[
SandboxHostAlias(
ip="127.0.0.1",
hostnames=["myapp.local", "api.local"]
)
]
)
```
## Next Steps
Use E2B, Daytona, or Modal with your environments
Complete API documentation for OpenReward sandboxes
Configure cloud storage access in sandboxes
Understand sandbox providers and when to use them
# GitHub Deployment
Source: https://docs.openreward.ai/deployment/github-integration
Connect and deploy environments from GitHub repositories
## Overview
OpenReward integrates seamlessly with GitHub to provide continuous deployment for your environments. When you push code to a connected repository, OpenReward automatically:
1. Builds your container image
2. Deploys the new version
3. Makes your environment available
**Prerequisite**: You must create an environment before connecting a GitHub repository. See [Your First Environment](/environments/your-first-environment) for setup instructions.
## Connecting a Repository
### Via OpenReward Web UI
1. **Navigate to Environment Page**
* Go to `https://openreward.ai/{username}/{environment-name}`
2. **Connect GitHub Repository**
* Click "Connect GitHub" button
* You will need to login to github, and grant read access to the desired repository
* Select the repository from dropdown
3. **Configure Build Settings**
* **Resource Allocation**: CPU and memory for environment
* **Scaling settings**: Minimum and maximum number of instances, and the number of sessions per instance
4. **Deploy**
* Click "Deploy" button
* First build starts immediately
## Deployment Flow
### Automatic Deployment on Push
Once connected, deployments happen automatically:
```text theme={null}
1. Developer pushes to GitHub
↓
2. GitHub webhook notifies OpenReward
↓
3. OpenReward validates the push event
↓
4. OpenReward builds container image
↓
5. New version deployed automatically
↓
6. Environment ready with new code
```
## Monitoring Deployments
### Deployment Status
**Via Web UI:**
* Navigate to environment page
* Click "Deployments" tab
* See list of recent deployments with status:
* ⏳ **Building**: Build in progress
* ✅ **Deployed**: Successfully deployed
* ❌ **Failed**: Build or deployment failed
**Deployment Information:**
* Git commit SHA and message
* Triggered by (push or manual)
* Build duration
* Deployment timestamp
### Build Logs
**View in OpenReward:**
1. Go to "Deployments" tab
2. Click on deployment
3. View build logs inline
### Environment Logs
**View in OpenReward:**
1. Go to environment page
2. Click "Logs" tab
3. See real-time logs from your environment server
### Resource Configuration
**CPU and Memory:**
Configure via Web UI:
* Choose appropriate size for your workload
* Options like "1:2" (1 CPU, 2GB RAM)
**Scaling Configuration:**
* **Minimum instances**: Scales to zero when idle (saves cost)
* **Maximum instances**: Limit concurrent environment servers
* **Sessions per instance**: How many agents per server
### Branch and Tag Filtering
**Deploy from Specific Branch:**
* Set in repository connection settings
* Only pushes to this branch trigger deployments
* Default: `main`
**Deploy from Tags:**
```bash theme={null}
# Tag a commit
git tag v1.0.0
git push origin v1.0.0
# Triggers deployment if configured
```
## Best Practices
Reduce image size and build time:
```dockerfile theme={null}
# Build stage
FROM python:3.11 AS builder
COPY requirements.txt .
RUN pip install --user -r requirements.txt
# Runtime stage
FROM python:3.11-slim
COPY --from=builder /root/.local /root/.local
COPY server.py .
ENV PATH=/root/.local/bin:$PATH
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8080"]
```
Catch issues early:
```bash theme={null}
docker build -t test-image .
docker run -p 8080:8080 test-image
curl http://localhost:8080/health
```
Exclude unnecessary files from build context:
```
# .dockerignore
.git
.env
*.pyc
__pycache__
tests/
docs/
*.md
```
Faster builds, smaller images
* Use slim base images (python:3.11-slim vs python:3.11)
* Remove unnecessary dependencies
* Use multi-stage builds
* Clean up after installations
```dockerfile theme={null}
RUN pip install --no-cache-dir -r requirements.txt
```
Commit messages appear in deployment history:
```bash theme={null}
# Good
git commit -m "Add caching to reduce response time"
# Bad
git commit -m "fix stuff"
```
## Next Steps
Complete tutorial for deploying your first environment
Test environments locally before deploying
Understand environment architecture
# Local Development
Source: https://docs.openreward.ai/deployment/local-development
Test environments locally before deploying to OpenReward
## Overview
You can develop and test OpenReward environments locally before deploying to production. This enables:
* **Rapid Iteration**: Test changes instantly without waiting for deployments
* **Debugging**: Use debuggers and inspect environment behavior
* **Offline Development**: Work without internet connection
* **Cost Savings**: No cloud resources used during development
## Quick Start: Testing Environments Locally
### 1. Install Dependencies
```bash theme={null}
pip install openreward
```
### 2. Write Your Environment Server
```python theme={null}
# server.py
from openreward.environments import Environment, Server, Split, TextBlock, ToolOutput, tool
from pydantic import BaseModel
class AnswerParams(BaseModel):
answer: str
class MyEnvironment(Environment):
"""A simple math environment"""
@classmethod
def list_tasks(cls, split: str):
return [
{"id": "task-1", "problem": "What is 2+2?", "answer": 4},
{"id": "task-2", "problem": "What is 5*3?", "answer": 15},
]
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
def get_prompt(self):
return [TextBlock(type="text", text=self.task_spec["problem"])]
@tool
def answer(self, params: AnswerParams) -> ToolOutput:
"""Submit your answer"""
correct = str(self.task_spec["answer"]) == params.answer
return ToolOutput(
blocks=[TextBlock(type="text", text="Correct!" if correct else "Wrong!")],
reward=1.0 if correct else 0.0,
finished=True
)
if __name__ == "__main__":
Server([MyEnvironment]).run()
```
### 3. Run the Server
```bash theme={null}
python server.py
```
Output:
```
INFO: Started server process [12345]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080
```
### 4. Test the Environment
```python theme={null}
# test_environment.py
from openreward import OpenReward
or_client = OpenReward(base_url="http://localhost:8080") # you must point towards the environment if running locally
# Connect to local server (no namespace = localhost)
environment = or_client.environments.get(name="myenvironment")
# List tasks
tasks = environment.list_tasks(split="train")
print(f"Found {len(tasks)} tasks")
# Create session and test tool
with environment.session(task=tasks[0]) as session:
prompt = session.get_prompt()
print(f"Prompt: {prompt[0].text}")
result = session.call_tool("answer", {"answer": "4"})
print(f"Result: {result.blocks[0].text}")
print(f"Reward: {result.reward}")
```
Alternatively, if you don't want the whole client to point to `http://localhost:8080`
just change the base\_url when getting the environment, like this:
```python theme={null}
environment = or_client.environments.get(name="myenvironment", base_url="http://localhost:8080")
```
Run:
```bash theme={null}
python test_environment.py
```
Output:
```
Found 2 tasks
Prompt: What is 2+2?
Result: Correct!
Reward: 1.0
```
## Local Development with Docker
### Build and Run Locally
```bash theme={null}
# Build Docker image
docker build -t my-environment:local .
# Run container
docker run -p 8080:8080 my-environment:local
# Test from another terminal
python test_environment.py
```
## Next Steps
Complete tutorial for building an environment
Deploy your local environment to OpenReward
Manage secrets in local and production environments
Understand environment architecture
# Using the Async Client
Source: https://docs.openreward.ai/environments/async-client
Learn to use the asynchronous features of OpenReward
## Goals
* Understand when and where to use the asynchronous client features
* Use the async client to efficiently serve environments that use sandboxes
* Use the async client to run multiple agents in parallel
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
## Basics
Previously, we've seen basic examples using the synchronous client. For example, interacting with an environment:
```python theme={null}
from openreward import OpenReward
client = OpenReward()
env = client.environments.get("GeneralReasoning/counter")
tasks = env.list_tasks(split="train")
with env.session(tasks[0]) as session:
# your agent logic here
result = session.call_tool("increment", {"amount": 1})
```
This simple synchronous interface is problematic when we need to interact with OpenReward features concurrently,
such as when running an evaluation over a suite of environments and tasks. To remedy this, an asynchronous version of the
`OpenReward` client is provided, called `AsyncOpenReward`. Using `AsyncOpenReward`, the above example would become
```python theme={null}
import asyncio
from openreward import AsyncOpenReward
client = AsyncOpenReward()
async def main():
env = client.environments.get("GeneralReasoning/counter")
tasks = await env.list_tasks(split="train")
async with env.session(tasks[0]) as session:
# your agent logic here
result = await session.call_tool("increment", {"amount": 1})
if __name__ == "__main__":
asyncio.run(main())
```
Currently, code using `AsyncOpenReward` is *required* to be run inside of `if __name__ == "__main__"`. This restriction may be lifted in the future.
## Serving Sandbox Environments
Many environments use [sandboxes](../concepts/sandboxes.mdx) to remotely execute agent actions inside of a virtual machine.
When creating such an environment, it is **critical** to use the asynchronous client. On the backend, your environments
are wrapped in a [FastAPI](https://fastapi.tiangolo.com/) server and can process many requests concurrently. Without the
use of `AsyncOpenReward`, requests that execute long-running sandbox actions in your environment can block all other connections,
drastically reducing concurrency of your environment and increasing costs.
Do not use the synchronous sandboxes API inside of an environment
```python theme={null}
from openreward import OpenReward, SandboxSettings
from openreward.environments import Environment, JSONObject, ToolOutput, tool
class MyEnvironment(Environment):
def __init__(self, task_spec: JSONObject, secrets: dict[str, str] = {}) -> None:
self.client = OpenReward()
self.compute_settings = SandboxSettings(...)
self.computer = self.client.sandbox(self.compute_settings)
@tool
def bash(...) -> ToolOutput:
result = self.computer.check_run(...)
```
Instead, always use the asynchronous sandboxes API
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings
from openreward.environments import Environment, JSONObject, ToolOutput, tool
class MyEnvironment(Environment):
def __init__(self, task_spec: JSONObject, secrets: dict[str, str] = {}) -> None:
self.client = AsyncOpenReward()
self.compute_settings = SandboxSettings(...)
self.computer = self.client.sandbox(self.compute_settings)
@tool
async def bash(...) -> ToolOutput:
# Important: we used the asynchronous API, so this call to check_run
# does not block other requests
result = await self.computer.check_run(...)
```
## Running Agents in Parallel
When using OpenReward for things like agentic reinforcement learning or evaluation, it is often desirable
to run many agents in parallel. To do this, a common design pattern is to define the agent to accept an
*asynchronous* session object, allowing the use of builtin python `asyncio` library operations to easily
run agents in parallel.
For example, the interface to the agent could look something like
```python theme={null}
from openreward import AsyncSession
async def run_agent(session: AsyncSession):
while True:
llm_output = await call_llm(...)
tool_name, tool_input = parse_tool_call(llm_output)
tool_output = await session.call_tool(tool_name, tool_input)
# bookkeeping around messages, tool output, etc goes here
```
We can then run many agents in parallel using `asyncio.gather` like
```python theme={null}
import asyncio
from agent import run_agent
from openreward import AsyncOpenReward
async def main():
client = AsyncOpenReward()
env = client.environments.get("my-env")
async def run(task):
async with env.session(task) as session:
return await run_agent(session)
tasks = await env.list_tasks(split="test")
coros = [run(task) for task in tasks]
agent_outputs = await asyncio.gather(*coros)
if __name__ == "__main__":
asyncio.run(main())
```
### Fetching a Subset of Tasks
For large environments, you may want to fetch only a subset of tasks rather than the full list.
`get_task_range` accepts `start` and `stop` parameters following Python range/slice conventions — `start` is inclusive,
`stop` is exclusive, and both support negative indices (resolved relative to the total number of tasks) and `None`
(defaults to `0` and `num_tasks` respectively). The ordering of tasks returned is guaranteed to be consistent across
calls, so you can safely partition work across workers using non-overlapping ranges.
```python theme={null}
async def main():
client = AsyncOpenReward()
env = client.environments.get("my-env")
# First 100 tasks
tasks = await env.get_task_range(split="test", start=0, stop=100)
# Last 50 tasks
tasks = await env.get_task_range(split="test", start=-50)
# All tasks (equivalent to list_tasks)
tasks = await env.get_task_range(split="test")
if __name__ == "__main__":
asyncio.run(main())
```
# Backdated Web Tools
Source: https://docs.openreward.ai/environments/backdated-web-tools
Give agents web search and fetch bounded to a cutoff date, with no post-cutoff leakage
## Introduction
`BackSearchToolset` gives an agent **web search and page fetch bounded to a
cutoff date** — "the web as it existed on or before *date X*". Every request
carries an `as_of` date and the backing corpus only returns documents published
on or before it. This is what makes the tools leakage-safe for evaluation and
prediction tasks: an agent researching a question dated to the past never sees
anything published after the cutoff.
There are two ways to use it:
* **As an environment toolset** — declare `BackSearchToolset` and the agent gets
`web_search` / `web_fetch` tools.
* **Standalone** — call `Backsearch` / `Backfetch` directly from any Python or
agent loop, no environment required.
Requires `openreward >= 0.1.134`. The web tools authenticate with your standard
`OPENREWARD_API_KEY` — no separate key is needed.
`BackSearchToolset` is **always** backed by the backdated corpus — it ignores
`OPENREWARD_SEARCH_BACKEND`, so its point-in-time guarantee cannot be switched
off by configuration. If you want the same two tools with a swappable provider
(Tavily and others), use [`WebToolset`](/environments/web-tools) instead.
## Configuration
The only required setting is your OpenReward API key (the same key the rest of
the SDK uses). In a hosted environment it is forwarded automatically as a
session secret.
```bash theme={null}
export OPENREWARD_API_KEY="or_xxxxxxxx"
```
| Variable | Required | Purpose |
| ------------------------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------- |
| `OPENREWARD_API_KEY` | **yes** | Your OpenReward API key (the web service uses the same key) |
| `OPENREWARD_WEB_AS_OF` | no | Default cutoff date (ISO `YYYY-MM-DD`); a per-call or per-env `as_of` overrides it |
| `OPENREWARD_WEB_SEARCH_URL` / `OPENREWARD_WEB_FETCH_URL` / `OPENREWARD_WEB_DOMAIN_INFO_URL` | no | Override backend endpoints (default: `https://search.openreward.ai/*`) |
| `OPENREWARD_WEB_SKIP_PREFLIGHT` | no | Set to `1` to skip the domain-policy preflight |
| `OPENREWARD_WEB_PREAPPROVED_HOSTS` | no | Comma-separated hosts whose fetched content is returned verbatim |
Search runs over a fixed historical corpus. Choose `as_of` dates inside the
covered window — dates outside it return no results. Confirm the current
coverage with the OpenReward team.
## Using the toolset in an environment
Declare `BackSearchToolset` at class level. Unlike the [harness
toolsets](/harnesses/harness-toolsets), it is **not** sandbox-backed — it talks
to the web service over HTTP — so your environment does not need a `self.sandbox`.
```python theme={null}
from openreward.environments import Environment, Split, TextBlock
from openreward.toolsets import BackSearchToolset
class ResearchEnv(Environment):
toolsets = [BackSearchToolset]
# Optional cutoff. May be a plain attribute, a @property, or a zero-arg
# callable — it is read live on every tool call (see "Progressing the
# cutoff" below).
web_as_of = "2025-12-01"
@classmethod
def list_splits(cls):
return [Split(name="test", type="test")]
@classmethod
def list_tasks(cls, split):
return [{"id": "1", "question": "What happened with X in late 2025?"}]
def get_prompt(self):
return [TextBlock(text="Research the question using web_search and web_fetch, then answer with sources.")]
```
The agent then sees two tools:
| Tool | Input | Output |
| ------------ | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `web_search` | `query` (≥ 2 chars), optional `allowed_domains` **or** `blocked_domains` (never both) | A `Links:` list of `{title, url}` sources plus a reminder to cite them |
| `web_fetch` | `url`, `prompt` | Extracted page text (or a `REDIRECT DETECTED` notice to re-call with the new URL) |
### Selecting it by name in a session
`BackSearchToolset` is a registered built-in, so a client can also attach it to
a session by name:
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
env = or_client.environments.get(name="MyOrg/ResearchEnv")
tasks = env.list_tasks(split="test")
with env.session(task=tasks[0], toolset="backsearch") as session:
result = session.call_tool("web_search", {"query": "US interest rate decision December 2025"})
print(result.blocks[0].text)
```
### Composing with other toolsets
Because it needs no sandbox, `BackSearchToolset` composes cleanly with
sandbox-backed toolsets — e.g. web research plus a file workspace:
```python theme={null}
from openreward.toolsets import BackSearchToolset, CLIToolset
class ResearchAndCodeEnv(Environment):
toolsets = [BackSearchToolset, CLIToolset]
# ... CLIToolset needs self.sandbox; BackSearchToolset does not ...
```
All tools from both toolsets are exposed, and the framework rejects duplicate
tool names. `BackSearchToolset` uses `web_search` / `web_fetch`, which do not
collide with the file/CLI toolsets.
## Progressing the cutoff
The cutoff is resolved **on every tool call**, in priority order:
1. an explicit `as_of=` passed to the toolset constructor (a static pin), else
2. `env.web_as_of`, else
3. the `OPENREWARD_WEB_AS_OF` environment variable, else
4. the backend default (today).
Because `env.web_as_of` is read live — not snapshotted when the toolset is
built — an environment whose simulated clock advances stays correctly bounded to
its current day. Expose it as a `@property` or a callable:
```python theme={null}
class TimeSteppedEnv(Environment):
toolsets = [BackSearchToolset]
@property
def web_as_of(self) -> str:
return self.sim_clock.today().isoformat() # advances as the task steps
```
This matters because the framework builds and caches one toolset instance per
environment for the whole session — a construction-time snapshot would freeze
the date at the first call.
## Standalone usage
The same backdated search and fetch are available without an environment:
```python theme={null}
import asyncio
from openreward.tools import Backsearch, Backfetch
async def main():
bs = Backsearch(as_of="2025-12-01")
result = await bs.run("US interest rate decision December 2025")
if result.ok:
print(result.output) # citation-ready text
for hit in result.data["hits"]:
print(hit["title"], "->", hit["url"])
bf = Backfetch(as_of="2025-12-01")
page = await bf.run("https://example.com/article", "Summarise the key points")
if page.ok:
print(page.data["content"])
asyncio.run(main())
```
Both `run(...)` calls return a `WebToolResult`:
| Field | Meaning |
| -------------------------------- | ----------------------------------------------------------------------------------- |
| `.ok` | `True` on success, `False` on a recoverable error |
| `.output` | The string to show a model (the result, or an error message) |
| `.data` | Structured payload — search: `{query, hits, mode}`; fetch: `{url, prompt, content}` |
| `.error_code` / `.error_message` | Set when `.ok` is `False` (e.g. `not-configured`, `invalid-url`, `domain-blocked`) |
`Backsearch.run` accepts `allowed_domains` / `blocked_domains` and a per-call
`as_of`; `Backfetch.run` takes a `url`, a `prompt`, and a per-call `as_of`.
## Notes
* **Citations.** Search output is formatted so the model is reminded to cite the
returned URLs as markdown links in its final answer.
* **Caching.** Fetches are cached in-process for 15 minutes, keyed by
`(cutoff, url)`, so repeated fetches at the same cutoff are fast and never mix
content across dates.
* **Soft errors.** Bad inputs (unparseable URL, blocked domain, missing key)
come back as a normal result with `.ok == False` and a clear message rather
than as an exception, so an agent can read it and recover.
## Next Steps
Compose document toolsets and build custom ones
Expose agent-native CLI tool surfaces backed by a sandbox
Create sandbox-based environments from scratch
How API keys and secrets reach your environment
# Building Agentic Environments
Source: https://docs.openreward.ai/environments/building-agentic-environments
Make and deploy your first agentic environment
## Goals
* Make an accounting environment using the OpenReward library.
* Deploy the environment to OpenReward.
* Sample from the environment using a model of your choice.
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
* You have completed the [Your First Environment](/your-first-environment) tutorial and understand the main components of [ORS](https://openrewardstandard.io).
## Setup
First, make sure you've installed the openreward library:
```bash theme={null}
pip install openreward
```
In the [Your First Environment](/environments/your-first-environment) tutorial, we made a mathematics environment. But the environment was simple; the agent did not, for example, have access to a computer to perform a task. In this tutorial we will make an environment that requires an agent to use a computer. More specifically, we give the agent access to a [Sandbox](/concepts/sandboxes).
Sandboxes support an extremely wide range of possible environments. In this tutorial we'll setup a simple accountancy task where an agent has access to files and has use sandbox tools to find the answer.
To begin we'll initialise our project with a `sandbox` template:
```bash theme={null}
orwd init accountantenv --template sandbox
cd accountantenv && ls
```
The template contains `server.py`, `sandbox_env.py`, `Dockerfile` and `requirements.txt`.
If you look inside `sandbox_env.py`, you can see a `SandboxEnv` is defined.
It looks very similar to any other [ORS](https://openrewardstandard.io) environment, but the key addition is a sandbox:
```bash theme={null}
self.sandbox_settings = SandboxSettings(
environment="YourUsername/SandboxEnv",
image="generalreasoning/python-ds:3.12-tools",
machine_size="0.5:1",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/sandbox/",
read_only=True
)
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(self.sandbox_settings)
```
You should read the [Sandboxes](/concepts/sandboxes) docs for a full rundown of how this works. Note that you **must** use the `AsyncOpenReward` client for sandbox environments.
The key thing to note is that a sandbox is tied to an environment on [OpenReward](https://openreward.ai), so unlike the ordinary ORS servers, we'll need to connect our server to an OpenReward environment space to utilise this compute.
To do this, let's create our environment on OpenReward. We'll create an environment called AccountantEnv.
Once you've done this, let's alter our `sandbox_env.py` (make sure to replace with your username):
```bash theme={null}
self.sandbox_settings = SandboxSettings(
environment="YourUsername/AccountantEnv",
image="generalreasoning/python-ds:3.12-tools",
machine_size="0.5:1",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/sandbox/",
read_only=True,
only_dir="agent"
)
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(self.sandbox_settings)
```
Now we can run the ORS server locally. First, install the requirements and run `server.py`:
```bash theme={null}
pip install -r requirements.txt
python server.py
```
```txt theme={null}
INFO: Started server process [92466]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
```
This environment is now running on port `8080`. We will leave this running.
Now let's see how we can interact with the `SandboxEnv`. Choose your model provider of choice and write the following file to `test_agent.py`:
Make sure you have API keys for [OpenAI](https://platform.openai.com/api-keys) and [OpenReward](https://openreward.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `test_agent.py`:
```python theme={null}
import json
import asyncio
import os
from openai import AsyncOpenAI
from openreward import AsyncOpenReward
async def main():
or_client = AsyncOpenReward()
oai_client = AsyncOpenAI()
MODEL_NAME = "gpt-5.4"
ENV_NAME = "SandboxEnvironment"
SPLIT = "test"
OR_API_KEY = os.getenv("OPENREWARD_API_KEY")
environment = or_client.environments.get(name=ENV_NAME, base_url="http://localhost:8080")
tasks = await environment.list_tasks(split=SPLIT)
tools = await environment.list_tools(format="openai")
print(f"Found {len(tasks)} tasks")
# Test first scenario
for task in tasks[:1]:
async with environment.session(
task=task,
secrets={"api_key": OR_API_KEY}
) as session:
prompt = await session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(f"\n=== Initial Prompt ===")
print(f"Role: {input_list[0]['role']}")
print(f"Content: {input_list[0]['content'][:100]}..." if len(input_list[0]['content']) > 100 else f"Content: {input_list[0]['content']}")
print(f"Finished: {finished}\n")
while not finished:
response = await oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
last_output = response.output[-1]
print(f"=== Model Response ===")
print(f"Type: {last_output.type}")
if hasattr(last_output, 'name'):
print(f"Function: {last_output.name}")
if hasattr(last_output, 'content'):
print(f"Content: {last_output.content[:100]}..." if len(str(last_output.content)) > 100 else f"Content: {last_output.content}")
print()
input_list += response.output
for item in response.output:
if item.type == "function_call":
print(f"Tool: {item.name}")
print(f"Arguments: {item.arguments}")
tool_result = await session.call_tool(
item.name,
json.loads(str(item.arguments))
)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": tool_result.blocks[0].text
})
print(f"=== Tool Result Logged ===")
print(f"Call ID: {input_list[-1]['call_id']}")
print(f"Output: {input_list[-1]['output'][:100]}..." if len(input_list[-1]['output']) > 100 else f"Output: {input_list[-1]['output']}")
print(f"Reward: {reward:.4f} | Finished: {finished}\n")
if finished:
print(f"FINISHED!")
break
if __name__ == "__main__":
asyncio.run(main())
```
```bash theme={null}
python test_agent.py
```
Make sure you have API keys for [Anthropic](https://console.anthropic.com/settings/keys) and [OpenReward](https://openreward.ai/keys), and set these as environment variables:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `test_agent.py`:
```python theme={null}
import json
import asyncio
import os
import anthropic
from openreward import AsyncOpenReward
async def main():
or_client = AsyncOpenReward()
ant_client = anthropic.AsyncAnthropic()
MODEL_NAME = "claude-sonnet-4-6"
ENV_NAME = "SandboxEnvironment"
SPLIT = "test"
OR_API_KEY = os.getenv("OPENREWARD_API_KEY")
environment = or_client.environments.get(name=ENV_NAME, base_url="http://localhost:8080")
tasks = await environment.list_tasks(split=SPLIT)
tools = await environment.list_tools(format="anthropic")
print(f"Found {len(tasks)} tasks")
# Test first scenario
for task in tasks[:1]:
async with environment.session(
task=task,
secrets={"api_key": OR_API_KEY}
) as session:
prompt = await session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(f"\n=== Initial Prompt ===")
print(f"Role: {messages[0]['role']}")
print(f"Content: {messages[0]['content'][:100]}..." if len(messages[0]['content']) > 100 else f"Content: {messages[0]['content']}")
print(f"Finished: {finished}\n")
while not finished:
response = await ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
print(f"=== Model Response ===")
print(f"Stop reason: {response.stop_reason}")
print(f"Content: {response.content}\n")
messages.append({
"role": "assistant",
"content": response.content,
})
if response.stop_reason == "tool_use":
tool_uses = [b for b in response.content if getattr(b, "type", None) == "tool_use"]
if not tool_uses:
raise RuntimeError("stop_reason was tool_use but no tool_use blocks found")
tool_result_blocks = []
for tu in tool_uses:
tool_name = tu.name
tool_input = tu.input
print(f"Tool: {tool_name}")
print(f"Arguments: {tool_input}")
try:
tr = await session.call_tool(tool_name, tool_input)
text_parts = []
for b in getattr(tr, "blocks", []) or []:
t = getattr(b, "text", None)
if t is not None:
text_parts.append(t)
else:
text_parts.append(str(b))
tool_text = "".join(text_parts)
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": tool_text,
})
reward = tr.reward
finished = getattr(tr, "finished", False)
print(f"=== Tool Result Logged ===")
print(f"Tool use ID: {tu.id}")
print(f"Output: {tool_text[:100]}..." if len(tool_text) > 100 else f"Output: {tool_text}")
print(f"Reward: {reward:.4f} | Finished: {finished}\n")
if finished:
print(f"FINISHED!")
break
except Exception as e:
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": f"Tool execution failed: {type(e).__name__}: {e}",
"is_error": True,
})
messages.append({
"role": "user",
"content": tool_result_blocks
})
if finished:
break
else:
break
if __name__ == "__main__":
asyncio.run(main())
```
```bash theme={null}
python test_agent.py
```
Make sure you have API keys for [Gemini](https://aistudio.google.com/app/apikey) and [OpenReward](https://openreward.ai/keys), and set these as environment variables:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `test_agent.py`:
```python theme={null}
import json
import asyncio
import os
from google import genai
from google.genai import types
from openreward import AsyncOpenReward
async def main():
or_client = AsyncOpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
ENV_NAME = "SandboxEnvironment"
SPLIT = "test"
OR_API_KEY = os.getenv("OPENREWARD_API_KEY")
environment = or_client.environments.get(name=ENV_NAME, base_url="http://localhost:8080")
tasks = await environment.list_tasks(split=SPLIT)
tools = await environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=tools)]
genai_config = types.GenerateContentConfig(tools=genai_tools)
print(f"Found {len(tasks)} tasks")
# Test first scenario
for task in tasks[:1]:
async with environment.session(
task=task,
secrets={"api_key": OR_API_KEY}
) as session:
prompt = await session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(f"\n=== Initial Prompt ===")
print(f"Role: user")
print(f"Content: {prompt[0].text[:100]}..." if len(prompt[0].text) > 100 else f"Content: {prompt[0].text}")
print(f"Finished: {finished}\n")
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(f"=== Model Response ===")
print(f"Content: {response.candidates[0].content}\n")
contents.append(response.candidates[0].content)
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
print(f"Tool: {tool_call.name}")
print(f"Arguments: {tool_call.args}")
tool_result = await session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": tool_result.blocks[0].text},
)
contents.append(types.Content(role="user", parts=[function_response_part]))
print(f"=== Tool Result Logged ===")
print(f"Tool: {tool_call.name}")
print(f"Output: {tool_result.blocks[0].text[:100]}..." if len(tool_result.blocks[0].text) > 100 else f"Output: {tool_result.blocks[0].text}")
print(f"Reward: {reward:.4f} | Finished: {finished}\n")
if finished:
print(f"FINISHED!")
break
if finished:
break
if __name__ == "__main__":
asyncio.run(main())
```
```bash theme={null}
python test_agent.py
```
Make sure you have API keys for [OpenRouter](https://openrouter.ai/keys) and [OpenReward](https://openreward.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `test_agent.py`:
```python theme={null}
import json
import asyncio
import os
from openai import AsyncOpenAI
from openreward import AsyncOpenReward
async def main():
or_client = AsyncOpenReward()
oai_client = AsyncOpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
ENV_NAME = "SandboxEnvironment"
SPLIT = "test"
OR_API_KEY = os.getenv("OPENREWARD_API_KEY")
environment = or_client.environments.get(name=ENV_NAME, base_url="http://localhost:8080")
tasks = await environment.list_tasks(split=SPLIT)
tools = await environment.list_tools(format="openrouter")
print(f"Found {len(tasks)} tasks")
# Test first scenario
for task in tasks[:1]:
async with environment.session(
task=task,
secrets={"api_key": OR_API_KEY}
) as session:
prompt = await session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(f"\n=== Initial Prompt ===")
print(f"Role: {messages[0]['role']}")
print(f"Content: {messages[0]['content'][:100]}..." if len(messages[0]['content']) > 100 else f"Content: {messages[0]['content']}")
print(f"Finished: {finished}\n")
while not finished:
response = await oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=messages
)
print(f"=== Model Response ===")
print(f"Message: {response.choices[0].message}\n")
messages.append({
"role": "assistant",
"content": response.choices[0].message.content,
"tool_calls": response.choices[0].message.tool_calls
})
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
break
for tool_call in tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
print(f"Tool: {tool_name}")
print(f"Arguments: {tool_args}")
tool_result = await session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(f"=== Tool Result Logged ===")
print(f"Tool call ID: {tool_call.id}")
print(f"Output: {tool_result.blocks[0].text[:100]}..." if len(tool_result.blocks[0].text) > 100 else f"Output: {tool_result.blocks[0].text}")
print(f"Reward: {reward:.4f} | Finished: {finished}\n")
if finished:
print(f"FINISHED!")
break
if finished:
break
if __name__ == "__main__":
asyncio.run(main())
```
```bash theme={null}
python test_agent.py
```
```
Found 1 tasks
=== Initial Prompt ===
Role: user
Content: You are participating in task 0. Please explore the file system.
Finished: False
=== Model Response ===
Type: function_call
Function: bash
Tool: bash
=== Tool Result Logged ===
Call ID: call_X9la8q01zbfCBcaXyhhUpl1D
Output: total 60
drwxr-xr-x 1 root root 4096 Jan 18 10:28 .
drwxr-xr-x 1 root root 4096 Jan 18 10:28 ..
...
Reward: 0.0000 | Finished: False
...
```
As we can see the agent utilises the `bash` tool and is able to explore a file system with the environment.
But as we can see, there is nothing in the file system right now.
Let's now give the agent access to a file in the file system. We'll use this [transactions.csv](https://drive.google.com/file/d/1Bw5eoHy2-A6VP21PbiWaehApHWUm5r2O/view?usp=sharing).
Let's upload it to a folder called `agent` in the Files tab of our environment:
Now we want to mount this specific folder to the sandbox for the agent to use, we can do this as follows:
```bash theme={null}
self.sandbox_settings = SandboxSettings(
environment="YourUsername/AccountantEnv",
image="generalreasoning/python-ds:3.12-tools",
machine_size="0.5:1",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/sandbox/",
read_only=True,
only_dir="agent"
)
)
```
We'll also update the prompt to let the agent know where to find the files:
```
async def get_prompt(self) -> List[TextBlock]:
"""Return the challenge prompt."""
full_prompt = f"""You are participating in task {self.task_id}. Please explore the file system. Check /tmp/sandbox/."""
return [TextBlock(text=full_prompt)]
```
Now let's rerun the `test_agent.py` file:
```
python test_agent.py
```
```
Found 1 tasks
=== Initial Prompt ===
Role: user
Content: You are participating in task 0. Please explore the file system. Check /tmp/sandbox/.
Finished: False
=== Model Response ===
Type: function_call
Function: bash
Tool: bash
=== Tool Result Logged ===
Call ID: call_D3sLk8NNb5mikqDMkpucD2UY
Output: total 2
-rw-r--r-- 1 root root 1039 Jan 18 11:47 transactions.csv
/tmp/sandbox/transactions.csv
(ex...
Reward: 0.0000 | Finished: False
=== Model Response ===
Type: function_call
Function: bash
Tool: bash
=== Tool Result Logged ===
Call ID: call_2qnpGUf5cUNRSkliExhEfaxl
Output: ID,Date,Transaction Type,Category,Amount
1,30-Nov-22,Paycheck,Salary," $ 5,500.00 "
2,30-Nov-22,Pr...
Reward: 0.0000 | Finished: False
=== Model Response ===
Type: message
Content: [ResponseOutputText(annotations=[], text='In `/tmp/sandbox/` there is a single file:\n\n- `/tmp/sandbox/transactions.csv` (1039 bytes)\n\nContents preview (CSV with 21 rows + header):\n\nHeader:\n- `ID, Date, Transaction Type, Category, Amount`\n\nSample rows:\n- `1, 30-Nov-22, Paycheck, Salary, "$ 5,500.00 "`\n- `2, 30-Nov-22, Profit from Food Business, Profit/Loss, "$ 1,600.00 "`\n- Expenses are negative formatted like `$(...)`, e.g.\n - `7, 04-Dec-22, Rent, Rent, "$ (1,700.00)"`\n\nCategories include: Salary, Profit/Loss, Insurances, Taxes, Rent, Loans, Food, Entertainment, Utilities, Memberships, Personal, Dining Out.', type='output_text', logprobs=[])]...
```
As we can see, the agent now has access to the `transactions.csv` file and managed to open to see its contents.
Now the agent has access to this file, we can construct a task for the agent that has a reward.
In the transactions spreadsheet, we have a list of transactions. We'll task the agent with calculating the total balance.
The reward in this case is easy to verify: the agent either gets the right balance, or it doesn't.
Here is an implementation for this type of environment. Copy it (replacing with your username) to sandbox\_env.py:
```python theme={null}
from typing import List
from openreward import AsyncOpenReward, SandboxBucketConfig, SandboxSettings
from openreward.environments import (Environment, JSONObject, Split, TextBlock,
ToolOutput, tool)
from pydantic import BaseModel
class BashParams(BaseModel, extra="forbid"):
command: str
class SubmitAnswerParams(BaseModel, extra="forbid"):
answer: int
class EnvironmentSpec(BaseModel):
task_id: str
class AccountantEnv(Environment):
def __init__(self, task_spec: JSONObject, secrets: dict[str, str] = {}) -> None:
super().__init__(task_spec)
self.validated = EnvironmentSpec.model_validate(task_spec)
self.task_id = self.validated.task_id
if not secrets.get("api_key"):
raise ValueError("OpenReward API key is required")
self.sandbox_settings = SandboxSettings(
environment="YourUsername/AccountantEnv",
image="generalreasoning/python-ds:3.12-tools",
machine_size="0.5:1",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/sandbox/",
read_only=True
)
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(self.sandbox_settings)
async def setup(self) -> None:
await self.sandbox.start()
async def teardown(self) -> None:
await self.sandbox.stop()
@tool
async def bash(self, params: BashParams) -> ToolOutput:
"""Executes a bash command in the environment."""
output, code = await self.sandbox.run(params.command.strip())
return ToolOutput(
blocks=[TextBlock(text=f"{output}\n\n(exit {code})")],
metadata={"output": output, "exit_code": code},
reward=0.0,
finished=False,
)
@tool
async def submit_answer(self, params: SubmitAnswerParams) -> ToolOutput:
"""Submit an integer answer for the current task."""
# Define correct answers for each task
correct_answers = {
"0": 1625 # Balance for transactions.csv
}
correct = correct_answers.get(self.task_id)
is_correct = params.answer == correct
result_text = f"Submitted answer: {params.answer}"
if is_correct:
result_text += "\nCorrect! Task completed."
reward = 1.0
else:
result_text += f"\nIncorrect. Expected: {correct}"
reward = 0.0
return ToolOutput(
blocks=[TextBlock(text=result_text)],
metadata={"answer": params.answer, "correct": is_correct},
reward=reward,
finished=True,
)
async def get_prompt(self) -> List[TextBlock]:
"""Return the challenge prompt."""
if self.task_id == "0":
full_prompt = """You are an accountant assistant. Your task is to calculate the final balance from a transactions spreadsheet.
The file is located at: /tmp/sandbox/transactions.csv
Calculate the total balance and submit your answer using the submit_answer tool with an integer (dollars).
"""
else:
full_prompt = f"""You are participating in task {self.task_id}. Please explore the file system. Check /tmp/sandbox/."""
return [TextBlock(text=full_prompt)]
@classmethod
def list_tasks(cls, split: str) -> List[JSONObject]:
"""Get all available tasks."""
if split == "test":
return [{"task_id": "0"}]
elif split == "train":
return [] # No training tasks
else:
raise ValueError(f"Unknown split: {split}")
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
```
We'll also update `server.py` with the new name:
```python theme={null}
from openreward.environments import Server
from sandbox_env import AccountantEnv
if __name__ == "__main__":
server = Server([AccountantEnv])
server.run()
```
Run the new server and then run `test_agent.py`:
```
Found 1 tasks
=== Initial Prompt ===
Role: user
Content: You are an accountant assistant. Your task is to calculate the final balance from a transactions spr...
Finished: False
=== Model Response ===
Type: function_call
Function: bash
Tool: bash
Arguments: {"command":"python3 - <<'PY'\nimport pandas as pd\npath='/tmp/sandbox/transactions.csv'\ndf=pd.read_csv(path)\nprint(df.head())\nprint(df.columns)\n# try find amount column\nfor c in df.columns:\n if 'amount' in c.lower() or 'amt' in c.lower() or 'value' in c.lower():\n print('amount col candidate',c)\nprint('rows',len(df))\nprint(df.dtypes)\nPY"}
=== Tool Result Logged ===
Call ID: call_gItEykpyH3Gqa6xrZN6VeNhw
Output: ID Date Transaction Type Category Amount
0 1 30-Nov-22 ...
Reward: 0.0000 | Finished: False
=== Model Response ===
Type: function_call
Function: bash
Tool: bash
Arguments: {"command":"python3 - <<'PY'\nimport pandas as pd, re\npath='/tmp/sandbox/transactions.csv'\ndf=pd.read_csv(path)\n\ndef parse_amt(s):\n if pd.isna(s):\n return 0.0\n s=str(s).strip()\n neg=False\n # parentheses indicate negative\n if '(' in s and ')' in s:\n neg=True\n # remove currency symbols and commas and parentheses\n s=re.sub(r'[^0-9.\\-]', '', s)\n if s=='':\n val=0.0\n else:\n val=float(s)\n if neg:\n val=-abs(val)\n return val\n\ndf['amt']=df['Amount'].apply(parse_amt)\nprint(df[['Amount','amt']].head(10))\nprint('sum',df['amt'].sum())\nPY"}
=== Tool Result Logged ===
Call ID: call_DkhOWr5mooHXIUmN8vy60FDs
Output: Amount amt
0 $ 5,500.00 5500.0
1 $ 1,600.00 1600.0
2 $ (250.00) -250.0
...
Reward: 0.0000 | Finished: False
=== Model Response ===
Type: function_call
Function: bash
Tool: bash
Arguments: {"command":"python3 - <<'PY'\nimport pandas as pd, re\npath='/tmp/sandbox/transactions.csv'\ndf=pd.read_csv(path)\n\ndef parse_amt(s):\n s=str(s).strip()\n neg=('(' in s and ')' in s)\n s=re.sub(r'[^0-9.\\-]', '', s)\n val=float(s) if s else 0.0\n if neg:\n val=-abs(val)\n return val\n\ndf['amt']=df['Amount'].apply(parse_amt)\nprint(df[['ID','Date','Transaction Type','Category','Amount','amt']])\nprint('total',df['amt'].sum())\nPY"}
=== Tool Result Logged ===
Call ID: call_JgjX0YkVT2GZhpGYwHy2fpe3
Output: ID Date ... Amount amt
0 1 30-Nov-22 ... $ 5,500.00 5500.0
1 2 ...
Reward: 0.0000 | Finished: False
=== Model Response ===
Type: function_call
Function: submit_answer
Tool: submit_answer
Arguments: {"answer":1625}
=== Tool Result Logged ===
Call ID: call_c4wen4IYNovyw88gIQCbd0se
Output: Submitted answer: 1625
Correct! Task completed.
Reward: 1.0000 | Finished: True
FINISHED!
```
Great! We can see that the agent successfully managed to process the file and calculate the correct answer.
## Conclusion
This concludes this tutorial. If you want to host your environment on OpenReward, then the steps are the same as with any other environment.
# Chat Backends
Source: https://docs.openreward.ai/environments/chat-backends
A composable channel for grader model calls
## Goals
* Learn how to build composable chat backends for environments, for use cases like grading and user simulations
* Understand why this traffic should not silently follow `OPENAI_*` environment variables
## Prerequisites
* `openreward>=0.1.153`
* Familiarity with [LLM graders](/environments/using-llm-graders)
## Introduction
Environments can use a regular provider SDK (OpenAI, Anthropic, Google, and so on) directly, as the [LLM graders](/environments/using-llm-graders) tutorial shows. For a model call that only ever runs against one provider, that works fine. This applies to any model the environment itself calls: judges, rubric graders, simulated users, and scripted opponents. (The [preference-rewards judging toolkit](/environments/preference-rewards) is built on this channel.)
The problem appears when your environment runs inside a larger system that *reconfigures* provider clients through the process environment. Training harnesses commonly inject `OPENAI_BASE_URL` (and `OPENAI_API_KEY`) into the environment container so that **policy** traffic is served by an internal inference gateway. The `openai` SDK reads those variables implicitly, so a grader built as a plain `openai.AsyncClient(api_key=...)` gets silently repointed too. If the gateway does not serve your judge model, the judge fails, the failure is usually swallowed by retry logic, and every rollout quietly scores zero.
`openreward.chat_backends` makes grader clients **compose** with that kind of orchestration instead of fighting it:
* the *environment author* writes one call (`resolve_backend(secrets=secrets)`) and never hardcodes a provider client;
* the *operator* decides at runtime where grader traffic goes, separately from policy traffic, using `CHAT_*` settings;
* the *provider* is swappable (`CHAT_PROVIDER`, `register_backend`) without touching grading code.
## Basic usage
Build the grader client in your environment's `__init__` and call it in your grading code:
```python theme={null}
from openreward.chat_backends import resolve_backend
class MyEnvironment(Environment):
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
# Raises ValueError if no key is available: a grader without
# credentials should fail loudly at session start.
self.judge_client = resolve_backend(secrets=secrets)
```
```python theme={null}
resp = await self.judge_client.create(
model="gpt-5-mini",
messages=messages, # OpenAI chat format, including image_url blocks
max_completion_tokens=1200,
)
verdict = parse(resp.text) # normalised ChatCompletion: .text, .model,
# .finish_reason, .usage, .raw
```
`create()` accepts extra provider kwargs verbatim (for example `response_format={"type": "json_object"}`) and raises provider exceptions unchanged, so your existing retry and fail-closed logic keeps working.
## How resolution works
The OpenAI backend has two modes, decided by whether **any** `CHAT_*` setting is present:
| Mode | Trigger | Base URL | API key |
| --------- | ------------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------- |
| Legacy | No `CHAT_*` setting anywhere | `OPENAI_BASE_URL` (or the OpenAI default) | `openai_api_key` secret, then `OPENAI_API_KEY` |
| Dedicated | `chat_api_key` secret, `CHAT_API_KEY`, or `CHAT_BASE_URL` set | `CHAT_BASE_URL` (default `https://api.openai.com/v1`) | `chat_api_key` secret, then `CHAT_API_KEY`, then the `openai_api_key` secret |
With nothing configured, behaviour is identical to a plain client: existing sessions keep working unchanged. The moment an operator sets one `CHAT_*` variable, the grader ignores the `OPENAI_*` environment entirely. For a training run that overrides `OPENAI_BASE_URL`, a single extra override is enough to send judges back to the real endpoint:
```bash theme={null}
CHAT_BASE_URL=https://api.openai.com/v1 # key still comes from the openai_api_key secret
```
Session secrets always beat process environment variables, and the host-pinned `(value, [allowed_hosts])` secret form is understood.
## Swapping and adding providers
`CHAT_PROVIDER` selects the backend by name (default `openai`). An unrecognised name logs a warning and falls back to the default rather than raising: a config typo should degrade the provider choice, not kill a rollout mid-flight.
To add a provider, implement the small ABC and register it:
```python theme={null}
from openreward.chat_backends import ChatBackend, ChatCompletion, register_backend
class AnthropicBackend(ChatBackend):
name = "anthropic"
def __init__(self, *, config=None, secrets=None):
...
async def create(self, *, model, messages, max_completion_tokens=None, **kwargs):
...
return ChatCompletion(text=..., raw=...)
register_backend("anthropic", AnthropicBackend)
```
Grading code written against `create()` and `.text` now runs against either provider, selected per deployment with `CHAT_PROVIDER=anthropic`, with no code change. The same pattern works for any model role the environment owns: a simulated user or opponent built on `create()` can be pointed at a different provider or endpoint per deployment in exactly the same way.
## Escape hatches
If you need the full provider surface (streaming, embeddings), `OpenAIChatBackend.raw_client` exposes the underlying pinned `AsyncOpenAI` client. For synchronous scripts, `OpenAIChatConfig.from_env(secrets=...).client_kwargs()` gives you the resolved `api_key`/`base_url` to construct any client with the same resolution rules.
## Next Steps
Learn how to integrate language model graders into an environment
Pairwise judges and group ranking, built on chat backends
# Debugging Environments
Source: https://docs.openreward.ai/environments/debugging-environments
Tips and techniques for debugging your environments
## Viewing Logs
Your first stop when debugging production environments should be the **Deployments** tab on your environment's page. This shows the build history for your environment, including whether the build completed or failed.
Click into a specific deployment row to access its logs. There are two types:
* **Build Logs** — If your deployment's status is **Failed**, check these first. They contain output from the Docker image build process and will show you exactly where the build broke.
* **Runtime Logs** — If your deployment built successfully but you can't access it or it's misbehaving, check these. They contain the live output from your running environment server.
## Common Errors
### Data in the wrong location
If your environment works locally but fails after deploying, one common source of failure is how your data is referenced. Check for these two things:
1. **You didn't upload your data to OpenReward.** Locally, your files sit next to your code. In production, the environment server reads from `/orwd_data/`, which is populated from the **Files** tab on your environment's page. If you haven't uploaded your data there, the files simply won't exist at runtime.
2. **Your code points to the wrong path.** Even if your data is uploaded, hardcoded local paths like `Path(__file__).parent / "answers.json"` won't resolve to `/orwd_data/` in production. Use a path check so your code works in both contexts:
```python theme={null}
import json
import os
from pathlib import Path
if os.path.exists("/orwd_data"):
DATA_PATH = Path("/orwd_data")
else:
DATA_PATH = Path(__file__).parent
with open(DATA_PATH / "answers.json") as f:
answers = json.load(f)
```
See [Where Environment Data Lives](/environments/where-environment-data-lives) for the full explanation of how `/orwd_data/` works and how to structure your files.
### Out of memory (OOM) crashes
An error like this indicates your environment container was killed for exceeding its memory limit:
```
SessionTerminatedError: Session terminated: 503, message='Pod unavailable:
Container crashed (exit code 137). Check the environment's startup command
and configuration.'
```
Exit code 137 means the process was killed by the OS, typically due to OOM. Start by diagnosing where the memory pressure is coming from, then work through these options:
1. **Reduce memory usage in your environment code.** Look for places where you load large amounts of data into memory at once — for example, reading an entire parquet file with millions of rows. Load only what you need for the current task, or stream data instead of loading it all upfront.
2. **Avoid `list_tasks` with large task sets.** `list_tasks` loads every task into memory. If you have a large number of tasks, override `get_task` and `num_tasks` to use index-based access instead, so only one task is loaded at a time. See [Ways to Access Tasks](/environments/ways-to-access-tasks) for details.
3. **Increase the memory allocation.** If your environment genuinely needs more RAM, you can increase it under **Settings > Compute Settings** on your environment's page.
# Deploying Harbor Environments
Source: https://docs.openreward.ai/environments/deploying-harbor-environments
Deploy environments that use the Harbor task specification
## Goals
* Understand the [Harbor task specification](https://www.harborframework.com/docs) and how OpenReward supports it
* Deploy a Harbor environment from a GitHub repository
* Configure tasks using `task.toml`
* Monitor per-task image builds and debug failures
## Prerequisites
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* A GitHub repository containing [Harbor](https://harborframework.com)-structured tasks
* Familiarity with [environments](/concepts/environments) and [GitHub deployment](/deployment/github-integration)
## What are Harbor environments?
[Harbor](https://www.harborframework.com/docs) is an open specification for defining agent tasks. Each task is a self-contained directory with an instruction, a container environment, and a verification script. Many benchmarks and evaluation suites publish their tasks in this format.
OpenReward has native support for the Harbor specification. When you mark an environment as a Harbor environment, OpenReward scans your connected GitHub repository for task directories, builds a Docker image for each task's sandbox, and generates the environment server code from the task definitions. You don't need to write a `server.py` or `Dockerfile` for the environment itself — OpenReward produces these from your Harbor tasks.
## Task directory structure
Each task in your repository must follow the Harbor layout:
```text theme={null}
my-task/
instruction.md # The prompt shown to the agent
task.toml # Task configuration (resources, timeouts, secrets)
tests/
test.sh # Verification script — determines reward
environment/
Dockerfile # Optional — sandbox image for this task
```
If a task has no `environment/Dockerfile` and no `docker_image` in `task.toml`, it defaults to `python:3.11-slim`.
You can scope the scan to a subdirectory of your repository. This is useful for monorepos where tasks live under a specific folder like `tasks/` or `benchmarks/`.
## Configuring tasks with task.toml
Each task's `task.toml` controls its sandbox resources, timeouts, and environment variables:
```toml theme={null}
split = "test" # "train", "test", or "validation"
[environment]
docker_image = "python:3.11" # Use a prebuilt image instead of building
cpus = 2
memory = "4G"
storage = "10G"
[environment.env]
MY_API_KEY = "${MY_API_KEY}" # Secret reference — see below
[agent]
timeout_sec = 1800
[verifier]
timeout_sec = 300
```
Resources are mapped to the nearest valid machine size. For example, 2 CPUs and 4 GB maps to machine size `"2:4"`.
### Secrets in task.toml
Values in `[environment.env]` or `[verifier.env]` wrapped in `${...}` are treated as secret references. During the build, OpenReward automatically detects these and populates the environment's secrets configuration with placeholder entries. You then fill in the actual values under **Settings > Secrets** on your environment's page.
See [Keeping Secrets Secret](/environments/keeping-secrets-secret) for more on managing secrets.
## Creating a Harbor environment
Go to [openreward.ai/new](https://openreward.ai/new) and enable the **Harbor Environment** toggle. Give the environment a name and create it.
On your environment's page, click **Connect GitHub** and select the repository containing your Harbor tasks. If your tasks live in a subdirectory, specify it in the **Subdirectory** field.
Then configure your compute and scaling settings and click **Deploy**.
Harbor deployments go through four phases:
1. **Building task images** — OpenReward scans your repo, detects tasks, and submits a Docker build for each one. The **Deployments** tab shows a progress counter (e.g. "12/47 images").
2. **Uploading data** — Task instructions, test scripts, and metadata are uploaded.
3. **Building server** — The generated environment server image is built.
4. **Deployed** — The environment is live and ready to accept sessions.
Click into a deployment to see the **Task Images** tab. This shows every task detected in your repo with its build status. Click a task to expand its Cloud Build logs.
Tasks with a `docker_image` set in `task.toml` skip the build step entirely and show as successful immediately.
## Enabling Harbor on an existing environment
You can convert an existing environment to Harbor mode under **Settings** on your environment's manage page. Toggle **Harbor Environment** on and trigger a new deployment. OpenReward will scan the connected repository for Harbor tasks on the next build.
The environment name is used as the class name in the generated server code. Stick to alphanumeric names with hyphens or underscores.
## How verification works
When an agent calls `submit_answer`, the environment uploads the task's `tests/` directory into the sandbox and runs `tests/test.sh`. The reward is read from one of:
1. `/logs/verifier/reward.txt` — a plain float (e.g. `1.0`)
2. `/logs/verifier/reward.json` — a JSON object with a `reward` field
3. Pytest output — if neither file exists, the environment parses pytest results as a fallback
## Tools
Harbor environments include the [ClaudeCodeToolset](/harnesses/harness-toolsets#claudecodetoolset) at the class level, which provides `bash`, `glob`, `grep`, `read`, `write`, `edit`, and `todo_write`. The environment also exposes a `submit_answer` tool for running verification and returning the reward.
## Change detection
On subsequent deployments, OpenReward only rebuilds task images whose `environment/` directory has changed since the last successful build. Unchanged tasks reuse their previous image. This makes incremental deploys fast — even for repositories with hundreds of tasks.
## Debugging failed builds
If a deployment fails during the **Building task images** phase:
1. Go to the deployment's **Task Images** tab to see which tasks failed.
2. Expand a failed task to view its Cloud Build logs — these show the full Docker build output.
3. Common issues: missing dependencies in the Dockerfile, syntax errors in `test.sh`, or invalid `task.toml` configuration.
If the **Building server** phase fails, check the **Build Logs** tab. This is the same build log experience as a standard environment.
See [Debugging Environments](/environments/debugging-environments) for general debugging techniques.
## Next Steps
Convert Harbor tasks locally with the harbor2or CLI tool.
Learn more about connecting repositories and deployment flow.
Manage secrets referenced in your task.toml files.
# Docker Images
Source: https://docs.openreward.ai/environments/docker-images
Choose and configure Docker images for sandbox environments
## Goals
* Understand the pre-made Docker images available for OpenReward sandboxes
* Use any public Docker Hub image in a sandbox environment
* Build and push custom Docker images for specialized needs
* Configure the `image` field in `SandboxSettings` for different use cases
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* [Docker](https://docs.docker.com/get-docker/) installed locally (for custom images)
* Completion of the [Building Agentic Environments](/environments/building-agentic-environments) tutorial
## Introduction
When you create an [OpenReward Sandbox](/concepts/sandboxes), the `image` field in `SandboxSettings` determines what software, libraries, and runtimes are available inside the container. Picking the right image means your agent has the tools it needs without installing packages at runtime.
OpenReward provides pre-made images for common use cases, but you can also use any public image from Docker Hub, or build and push your own. This page covers all three options and how they connect to your `SandboxSettings` configuration.
**Note**: OpenReward's sandbox infrastructure runs on `linux/amd64`. If you're building custom images on an ARM machine (e.g. Apple Silicon), you'll need to target this platform explicitly — more on this below.
## Pre-Made Images
We maintain two images that cover the most common agentic workloads:
| Image | Description | Use Case |
| --------------------------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------- |
| `generalreasoning/python-ds:3.12-tools` | Python 3.12 with data science libraries (pandas, numpy, etc.) | Data analysis, file processing, general Python tasks |
| `generalreasoning/knowledge-worker` | Document processing support (PDF, Excel, Word, PowerPoint) | Environments using pre-built [Toolsets](/environments/using-toolsets) |
### `generalreasoning/python-ds:3.12-tools`
This is the default image used by the `orwd init --template sandbox` scaffold. It includes Python 3.12, pip, and core data science libraries like pandas and numpy. Use this when your agent needs to write and execute Python code for data analysis, file manipulation, or general-purpose tasks.
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings, SandboxBucketConfig
self.sandbox_settings = SandboxSettings(
environment="YourUsername/AccountantEnv",
image="generalreasoning/python-ds:3.12-tools",
machine_size="0.5:1",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/sandbox/",
read_only=True,
)
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(self.sandbox_settings)
```
See the [Building Agentic Environments](/environments/building-agentic-environments) tutorial for a full walkthrough using this image.
### `generalreasoning/knowledge-worker`
This image includes all the dependencies needed for the pre-built toolsets: pdfplumber, pypdf, reportlab, pdf2image, python-docx, openpyxl, and python-pptx. If your environment uses `PDFToolset`, `ExcelToolset`, `WordToolset`, or `PowerPointToolset`, this is the image to use.
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings
self.sandbox_settings = SandboxSettings(
environment="YourUsername/DocProcessorEnv",
image="generalreasoning/knowledge-worker",
machine_size="0.5:1",
block_network=False,
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(self.sandbox_settings)
```
See [Using Toolsets](/environments/using-toolsets) for a full guide on composing toolsets with this image.
## Using Any Docker Hub Image
The `image` field in `SandboxSettings` accepts any public Docker Hub image. This is useful when you need a specific language runtime, a minimal base image, or something outside the Python ecosystem entirely.
Some common options:
| Image | Use Case |
| ------------------ | ---------------------------- |
| `python:3.11-slim` | Minimal Python environment |
| `python:3.12` | Full Python with build tools |
| `node:20` | Node.js environments |
| `ubuntu:22.04` | General-purpose Linux |
```python theme={null}
settings = SandboxSettings(
environment="username/my-env",
image="node:20",
machine_size="1:2",
block_network=False,
)
```
**Note**: Larger images take longer to pull on first sandbox creation. If startup time matters, prefer slim or minimal images when you don't need the full set of system packages.
## Building Custom Images
When the pre-made images don't include the dependencies you need, you can build your own image, push it to Docker Hub, and reference it in `SandboxSettings`.
### Writing a Dockerfile
Start with a base image and add your dependencies. Here's a complete example for a sandbox that needs specific ML libraries:
```dockerfile theme={null}
FROM python:3.12-slim
RUN apt-get update && apt-get install -y \
curl \
git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
```
With a `requirements.txt`:
```txt theme={null}
pandas==2.2.0
numpy==1.26.3
scikit-learn==1.4.0
matplotlib==3.8.2
```
**Important**: This Dockerfile is for your **sandbox image** — the container where the agent executes code. This is separate from the environment server Dockerfile used to [deploy your environment](/deployment/github-integration). The sandbox image is what gets specified in the `image` field of `SandboxSettings`.
### Building and Pushing
OpenReward's sandbox infrastructure runs on `linux/amd64`. You must target this platform when building, even if you're developing on an Apple Silicon Mac or another ARM machine:
```bash theme={null}
docker build --platform linux/amd64 -t yourusername/my-sandbox-image:latest .
```
**Important**: If you omit `--platform linux/amd64` and build on an ARM machine, the image will fail to run on OpenReward's infrastructure.
Once built, push to Docker Hub:
```bash theme={null}
docker login
docker push yourusername/my-sandbox-image:latest
```
The image must be publicly accessible on Docker Hub. OpenReward pulls images from public repositories.
### Using Your Custom Image
Once pushed, reference it in `SandboxSettings` just like any other image:
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings, SandboxBucketConfig
from openreward.environments import Environment, TextBlock, ToolOutput, tool
from pydantic import BaseModel
class BashParams(BaseModel):
command: str
class CustomImageEnv(Environment):
def __init__(self, task_spec, secrets):
super().__init__(task_spec, secrets)
self.sandbox_settings = SandboxSettings(
environment="YourUsername/CustomImageEnv",
image="yourusername/my-sandbox-image:latest", # Your custom image
machine_size="1:2",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/workspace",
read_only=True,
),
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(self.sandbox_settings)
async def setup(self):
await self.sandbox.start()
async def teardown(self):
await self.sandbox.stop()
@tool
async def bash(self, params: BashParams) -> ToolOutput:
"""Execute a bash command in the sandbox."""
output, code = await self.sandbox.run(params.command.strip())
return ToolOutput(
blocks=[TextBlock(text=f"{output}\n\n(exit {code})")],
reward=0.0,
finished=False,
)
```
The only thing that changes compared to using a pre-made image is the `image` value — everything else in `SandboxSettings` works the same way.
## Best Practices
**Start with pre-made images**: Use `generalreasoning/python-ds:3.12-tools` or `generalreasoning/knowledge-worker` when they cover your needs. Only build custom images when you need dependencies they don't include.
**Keep images small**: Use slim base images, clean up after `apt-get install` with `rm -rf /var/lib/apt/lists/*`, and use `pip install --no-cache-dir`. Smaller images pull faster and your sandboxes start sooner.
**Always build for `linux/amd64`**: Use `docker build --platform linux/amd64` regardless of your local machine architecture.
**Pin dependency versions**: Use specific versions in `requirements.txt` (e.g. `pandas==2.2.0`) rather than unpinned packages. This ensures reproducible builds.
**Test locally before pushing**: Run your image locally to verify it has everything you need:
```bash theme={null}
docker run --platform linux/amd64 -it yourusername/my-sandbox-image:latest bash
```
## Next Steps
Learn more about sandbox configuration, lifecycle, and machine sizes
Build a complete agentic environment using sandbox Docker images
Use the knowledge-worker image with pre-built document processing toolsets
# Environment Cards
Source: https://docs.openreward.ai/environments/environment-cards
Document your environment so users know what it does
## What is an Environment Card?
An **environment card** is a structured description of your environment - similar to a [model card](https://arxiv.org/abs/1810.03993), but for environments. It tells users what your environment evaluates, what tools are available, how rewards work, and what compute or safety considerations apply.
On OpenReward, the environment card is your environment's `README.md`. It renders on your environment's overview page and is the first thing users see when they visit your environment.
## Template
Below is a template you can copy and fill in for your own environment. Each section is explained in detail afterwards.
````markdown theme={null}
# YourEnvironmentName
[](https://openreward.ai/YourOrg/YourEnvironmentName)
## Description
## Capabilities
- Capability 1
- Capability 2
- Capability 3
## Compute Requirements
## License
## Tasks
## Reward Structure
## Data
## Tools
## Time Horizon
## Environment Difficulty
## Other Environment Requirements
## Safety
## Citations
```bibtex
@dataset{YourCitation,
author = {Your Name or Team},
title = {YourEnvironmentName},
year = {2026},
publisher = {OpenReward},
url = {https://openreward.ai/YourOrg/YourEnvironmentName}
}
```
````
## Section Reference
### Badge + Title
```markdown theme={null}
# YourEnvironmentName
[](https://openreward.ai/YourOrg/YourEnvironmentName)
```
The badge links to your environment on OpenReward and gives viewers on GitHub an immediate visual indicator that this is an OpenReward environment. Replace `YourOrg/YourEnvironmentName` with your environment's path.
### Description
A short paragraph (2-4 sentences) explaining what the environment evaluates. Cover:
* The domain or task type (e.g. molecular generation, code repair, math reasoning)
* How answers are verified (e.g. RDKit validation, unit tests, exact match)
* The source dataset, with a link
### Capabilities
A bulleted list of the specific skills or competencies the environment tests. This helps users quickly decide whether the environment is relevant to their work.
### Compute Requirements
State whether the environment requires a sandbox, GPU, or additional memory. If it has minimal requirements and runs without a sandbox, say so explicitly - this is useful information for users planning their compute budget.
### License
The license that covers the use of the environment, as well as any underlying data or code it depends on. Link to the full license text. If the environment and its dependencies have different licenses, list both.
### Tasks
Describe the task structure:
* How many tasks in each split (e.g. 1,000 train / 100 test)
* What each task contains (e.g. a prompt, constraints, reference answer)
* Any notable properties (e.g. percentage with constraints, number of unique categories)
### Reward Structure
Explain how rewards are assigned. Key things to cover:
* **Sparse vs dense**: Is reward only given at the end, or at intermediate steps?
* **Binary vs continuous**: Is the reward 0/1, or a value on a scale?
* **Grading method**: Is verification programmatic (exact match, test suite) or does it use an LLM grader? Does it use [rubrics](/environments/using-rubrics)? Does it use [preference-based group scoring](/environments/preference-rewards) (`score_group`)?
* **Pipeline**: If there are multiple validation steps, list them in order
### Data
Where the task data comes from and how it's stored. Link to the source dataset if applicable.
### Tools
Explain the tools available to the agent.
### Time Horizon
State whether the environment is **single-turn** (one tool call) or **multi-turn** (multiple tool calls over a conversation). If multi-turn, give a rough indication of how many tool calls a typical task requires. If the task's prompt is itself a [conversation history](/environments/multi-turn-conversations) the model must continue, say so here too.
### Environment Difficulty
Report any statistics on environment difficulty - for example, solve rates for baseline models.
### Other Environment Requirements
List any external dependencies: API keys, secrets, third-party services.
### Safety
Describe any safety considerations:
* Does the agent have access to external systems (network, file system, APIs)?
* Are there dual-use risks in the domain (e.g. chemistry, cybersecurity)?
* Is there a possibility of goal misspecification in this environment?
* What mitigations are in place?
Even if the risks are low, include this section - it signals that you've considered safety.
### Citations
Provide BibTeX entries for:
* The environment itself
* Any underlying datasets or papers the environment is built on
This makes it easy for researchers to cite your work.
# Your first evaluation
Source: https://docs.openreward.ai/environments/evaluation
Learn how to evaluate models with OpenReward
## Goals
In this documentation, you will:
* Make an AIME 2024 evaluation environment.
* Deploy the environment to OpenReward (optional).
* Write an evaluation script.
* Evaluate from a model of your choice on this evaluation.
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
## Setup
Environments in OpenReward are written using the [OpenReward Python library](https://www.github.com/OpenReward/openreward-python).
You can install this library using pip or uv:
```bash theme={null}
pip install openreward
```
## Our evaluation: AIME 2024
The American Invitational Mathematics Examination (AIME) is a selective 15-question, 3-hour test given since 1983 to those who rank in the top 5% on the AMC 12 high school mathematics examination. Historically it has been used to evaluate the reasoning performance of language models.
An example question and answer from the AIME 2024 examination:
> Let $x,y$ and $z$ be positive real numbers that satisfy the following system of equations:
> $\log_2\left({x \over yz}\right) = {1 \over 2}$
> $\log_2\left({y \over xz}\right) = {1 \over 3}$
> $\log_2\left({z \over xy}\right) = {1 \over 4}$
> Then the value of $\left|\log_2(x^4y^3z^2)\right|$ is $\tfrac{m}{n}$ where $m$ and $n$ are relatively prime positive integers. Find $m+n$.
We'll implement this as an evaluation using OpenReward.
## Building the evaluation
Initialise a project using the OpenReward cli:
```bash theme={null}
orwd init aime2024 --template basic
cd aime2024 && ls aime2024
```
We'll replace the `server.py` file with the following:
```python theme={null}
from math_verify import parse, verify
import pandas as pd
from pydantic import BaseModel
from openreward.environments import Environment, JSONObject, Server, Split, TextBlock, ToolOutput, tool
class AIME2024TaskSpec(BaseModel):
id: str
problem: str
answer: str
class AnswerParams(BaseModel):
answer: str
test_tasks = pd.read_parquet("aime_2024_problems.parquet").to_dict(orient="records")
for i, task in enumerate(test_tasks):
task.pop('ID')
task.pop('Solution')
task['id'] = str(i)
task['Answer'] = str(task['Answer'])
keys_to_change = [key for key in task.keys() if key != "id"]
for key in keys_to_change:
if key.lower() != key:
task[key.lower()] = task.pop(key)
class AIME2024(Environment):
"""
An environment for the AIME 2024 dataset
"""
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = AIME2024TaskSpec.model_validate(task_spec)
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
if split == "train":
return []
elif split == "test":
return test_tasks
raise ValueError(f"Unknown split: {split}")
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
def get_prompt(self) -> str:
return [TextBlock(type="text", text=self.config.problem)]
@tool
async def answer(self, params: AnswerParams) -> ToolOutput:
"""
The answer tool can be used to submit your final answer. Note that this finishes the episode.
"""
gold = parse(self.config.answer)
answer = parse(params.answer)
is_correct = verify(gold, answer)
if is_correct:
agent_message = "Correct!"
reward = 1.0
else:
agent_message = "Wrong!"
reward = 0.0
return ToolOutput(
blocks=[TextBlock(type="text", text=agent_message)],
reward=reward,
finished=True
)
if __name__ == "__main__":
Server([AIME2024]).run()
```
We'll update the `requirements.txt`:
```txt theme={null}
fastapi>=0.115.12
openreward
pandas
pyarrow
uvicorn>=0.34.3
math-verify[antlr4_13_2]
```
And update the `Dockerfile`:
```txt theme={null}
FROM python:3.11-slim
RUN apt update && apt upgrade -y && apt install -y \
curl
WORKDIR /app
# Copy requirements and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application code
COPY server.py .
# Expose port
EXPOSE 8000
RUN curl -L -o aime_2024_problems.parquet \
https://huggingface.co/datasets/Maxwell-Jia/AIME_2024/resolve/main/aime_2024_problems.parquet
# Start the server
CMD ["python", "server.py"]
```
Now create an environment on [OpenReward](https://openreward.ai), upload your code to GitHub and connect your repository. Alternatively, you can do your evaluations locally
by running the server locally, i.e. `python server.py`.
## Evaluating a model
Now let's evaluate our model.
Make sure you have an API key for [OpenAI](https://platform.openai.com/api-keys) and [OpenReward](https://openreward.ai/keys), and set the environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save the following to `evaluate.py`:
```python theme={null}
import asyncio
import json
import copy
from typing import cast
from openai import AsyncOpenAI
from openai.types.responses import ResponseInputItemParam
from openai.types.responses.response_input_item_param import FunctionCallOutput
from openai.types.responses.tool_param import ToolParam
from openreward import OpenReward
from openreward.api.environments.client import Environment, Task, ToolSpec
CLIENT = AsyncOpenAI()
NUM_SEEDS = 2
OAI_MODEL = "gpt-4o"
async def process_aime_task(environment: Environment, task: Task, tools: list[ToolSpec] | list[dict]) -> float | None:
async with environment.session(task=task) as session:
prompt = await session.get_prompt()
prompt = "".join([str(block) for block in prompt])
previous_response_id = None
latest_input: list[ResponseInputItemParam] = [{"role": "user", "content": prompt}]
while True:
response = await CLIENT.responses.create(
model=OAI_MODEL,
tools=cast(list[ToolParam], tools),
input=latest_input,
previous_response_id=previous_response_id,
tool_choice="required",
)
previous_response_id = response.id
# parse and execute tool calls
latest_input = []
for out in response.output:
if out.type != "function_call":
continue
tool_result = await session.call_tool(out.name, json.loads(str(out.arguments)))
if tool_result.finished:
print(tool_result.reward)
return tool_result.reward
item: FunctionCallOutput = {
"type": "function_call_output",
"call_id": out.call_id,
"output": "".join([str(block) for block in tool_result.blocks])
}
latest_input.append(item)
async def main():
or_client = OpenReward()
environment = or_client.environments.get(name="RJT1990/AIME2024")
tasks = await environment.list_tasks(split="test")
# Or fetch a subset: tasks = await environment.get_task_range(split="test", start=0, stop=10)
tools = await environment.list_tools(format="openai")
original_tasks: list[Task] = list(tasks)
tasks_with_seeds: list[Task] = [
copy.deepcopy(task) for task in original_tasks for _ in range(NUM_SEEDS)
]
semaphore = asyncio.Semaphore(10)
async def run_task(task: Task) -> float | None:
async with semaphore:
return await process_aime_task(environment, task, tools)
results = await asyncio.gather(*[run_task(task) for task in tasks_with_seeds])
num_samples = len(results)
num_correct = sum(1 for r in results if r is not None and r == 1)
pass_at_1 = num_correct / num_samples if num_samples > 0 else 0.0
print(pass_at_1)
if __name__ == "__main__":
asyncio.run(main())
```
Now, run the code to evaluate:
```bash theme={null}
python evaluate.py
```
```
0.05
```
So the pass\@1 is calculated a `5.00%` for `gpt-4o` for these seeds.
# Making GPU Environments
Source: https://docs.openreward.ai/environments/gpu-environments
Configure environments that use GPU-accelerated sandboxes
## Goals
* Understand when an environment needs GPU access
* Configure `machine_size` to provision a GPU-enabled sandbox
## Prerequisites
* Completion of the [Your First Environment](/environments/your-first-environment) tutorial
* Familiarity with [Sandboxes](/concepts/sandboxes) and `SandboxSettings`
## Introduction
Some tasks require GPU access - for example, evaluating CUDA kernels or training models inside the sandbox. By default, sandboxes use CPU-only machine sizes like `"0.5:1"`. To give the agent access to a GPU, set `machine_size` to `"nvidia-l4"`.
## Configuration
Specify `machine_size="nvidia-l4"` in your `SandboxSettings`:
```python theme={null}
self.sandbox_settings = SandboxSettings(
environment="YourUsername/YourEnv",
image="nvidia/cuda:12.4.0-devel-ubuntu22.04",
machine_size="nvidia-l4",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/sandbox/",
read_only=True,
)
)
```
Your Docker image must include CUDA drivers and any GPU libraries your tasks depend on. The official `nvidia/cuda` images are a good starting - pick a tag that matches your CUDA version and build on top of it with your own dependencies.
This gives the agent access to an NVIDIA L4 GPU inside the sandbox.
GPU environments have limited capacity and are more expensive than CPU-only environments.
## Managing GPU cost with sandbox lifetime
Because GPU sandboxes are expensive, consider scoping the sandbox lifetime so the GPU is only allocated while it's actually needed. For example, in a kernel-optimization environment the agent spends most of its time reasoning and writing code, it only needs the GPU when executing the kernel. You can design the sandbox so it is created at the start of the tool call that runs the kernel and destroyed when that call finishes, rather than keeping it alive for the entire session.
See the [Sandbox Lifecycle](/concepts/sandboxes#sandbox-lifecycle) section for more on controlling when sandboxes are created and destroyed.
# Keeping Secrets Secret
Source: https://docs.openreward.ai/environments/keeping-secrets-secret
How OpenReward protects sensitive API keys using placeholder injection and egress-time substitution
When you create a session, you can pass a `secrets` dictionary containing API keys and other sensitive values your environment or sandbox needs at runtime:
```python theme={null}
async with environment.session(
task=task,
secrets={
"OPENAI_API_KEY": "sk-...",
"ANTHROPIC_API_KEY": "sk-ant-...",
}
) as session:
...
```
Your environment's `__init__` method receives these via the `secrets` parameter:
```python theme={null}
class MyEnvironment(Environment):
def __init__(self, task_spec: JSONObject, secrets: dict[str, str] = {}) -> None:
super().__init__(task_spec, secrets=secrets)
```
Rather than exposing raw secret values inside your environment or sandbox, OpenReward replaces them with placeholders before they reach your server. The actual values are injected on egress - when outbound HTTP requests leave the environment or sandbox, OpenReward swaps the placeholders back in for allowed destination hosts only.
This means your environment code never handles raw API keys directly, and secrets cannot be leaked to unauthorised hosts.
## How It Works
1. The client passes secrets when creating a session
2. OpenReward stores the real values and replaces them with opaque placeholders
3. Your environment receives only the placeholders
4. When an outbound request leaves your environment or sandbox, OpenReward injects the real value - but only if the destination host is allowed
## Allowed Outbound Hosts
Each secret has a list of allowed hosts that it can be sent to. For common API keys, OpenReward provides sensible defaults:
| Secret Name | Default Allowed Hosts |
| ------------------------ | ------------------------------------------------ |
| `OPENAI_API_KEY` | `api.openai.com` |
| `ANTHROPIC_API_KEY` | `api.anthropic.com` |
| `GEMINI_API_KEY` | `generativelanguage.googleapis.com` |
| `GOOGLE_API_KEY` | `generativelanguage.googleapis.com` |
| `TAVILY_API_KEY` | `api.tavily.com` |
| `MISTRAL_API_KEY` | `api.mistral.ai` |
| `COHERE_API_KEY` | `api.cohere.com` |
| `GROQ_API_KEY` | `api.groq.com` |
| `TOGETHER_API_KEY` | `api.together.xyz` |
| `REPLICATE_API_TOKEN` | `api.replicate.com` |
| `HUGGINGFACE_API_KEY` | `api-inference.huggingface.co`, `huggingface.co` |
| `HF_TOKEN` | `huggingface.co`, `api-inference.huggingface.co` |
| `HUGGING_FACE_HUB_TOKEN` | `huggingface.co`, `api-inference.huggingface.co` |
| `PERPLEXITY_API_KEY` | `api.perplexity.ai` |
| `FIREWORKS_API_KEY` | `api.fireworks.ai` |
| `DEEPSEEK_API_KEY` | `api.deepseek.com` |
| `KAGGLE_KEY` | `www.kaggle.com` |
| `KAGGLE_USERNAME` | `www.kaggle.com` |
| `KAGGLE_API_KEY` | `www.kaggle.com` |
| `E2B_API_KEY` | `api.e2b.app` |
| `MODAL_TOKEN_ID` | `api.modal.com` |
| `MODAL_TOKEN_SECRET` | `api.modal.com` |
| `DAYTONA_API_KEY` | `app.daytona.io` |
For secrets without defaults, or to override the defaults, pass a `(value, [hosts])` tuple instead of a plain string:
```python theme={null}
secrets = {
# Uses default allowed hosts (api.openai.com)
"OPENAI_API_KEY": "sk-...",
# Custom hosts for a secret without defaults
"CUSTOM_API_KEY": ["my-secret-value", ["api.example.com", "api2.example.com"]],
}
```
Secrets with default allowed hosts can be passed as plain strings. Secrets without defaults must use the `(value, [hosts])` format — otherwise the request will be rejected.
# Multi-turn Conversations
Source: https://docs.openreward.ai/environments/multi-turn-conversations
Serving a conversation history as the prompt
## Goals
* Learn how to serve a task's prompt as a role-tagged conversation instead of a single block of text
* Understand how harnesses consume conversation-shaped prompts, and how the feature degrades gracefully everywhere else
## Prerequisites
* `openreward>=0.1.157`
* Completing the [Your First Environment](/environments/your-first-environment) tutorial
## Introduction
Most environments serve their prompt with `get_prompt()`: a list of content blocks that every harness wraps in a single `{"role": "user"}` message. That remains the default — and for most tasks, the right — way to serve a prompt.
Some tasks, though, *are* conversations. A dialogue dataset where the model must write the next reply. A preference task built on chat transcripts. A support conversation the policy must continue. For those, the task carries the conversation so far — user turns, assistant turns, maybe a system prompt — and squeezing it into one user message loses the structure the model was trained on.
`get_messages()` is an additional, opt-in prompt surface for exactly these tasks. The environment returns role-tagged messages, the harness feeds them to the model as a real conversation, and the model writes the next assistant turn. Everything else about the episode — the tool loop, the terminal answer, rewards — proceeds exactly as it always has.
This is **not** user simulation. The conversation is task data, served once at the start of the episode. The environment does not generate new user turns mid-episode. (If you need a simulated user, see the pattern in [Chat Backends](/environments/chat-backends) — a simulated user is a model role the environment owns, called from inside a tool.)
## Serving a conversation
Implement `get_messages()` instead of `get_prompt()` — one of the two is required, and each defaults to delegating to the other:
```python theme={null}
from openreward.environments import Environment, JSONObject, Message, TextBlock
class DialogueEnv(Environment):
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.conversation = task_spec["conversation"]
def get_messages(self) -> list[Message]:
return [
Message(role=turn["role"], blocks=[TextBlock(text=turn["text"])])
for turn in self.conversation
]
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
return [
{
"id": "support-1",
"conversation": [
{"role": "system", "text": "You are a support agent for Acme."},
{"role": "user", "text": "My order hasn't arrived."},
{"role": "assistant", "text": "Sorry to hear that — what's the order number?"},
{"role": "user", "text": "It's #4411, ordered two weeks ago."},
],
}
]
```
A `Message` has a `role` (`"system"`, `"user"`, or `"assistant"`) and content `blocks` — `TextBlock` / `ImageBlock`, plus the [rich block types](#rich-conversations-reasoning-and-tool-history) below, so multimodal and agentic conversations work unchanged.
Note that the `{"role": ..., "text": ...}` dicts in this task spec are just this example's own convention — task specs are plain JSON, and the SDK doesn't prescribe how a conversation is stored in them. `get_messages()` is the translation point where your task data becomes `Message` objects; store whatever shape suits your data and build the blocks there.
Rewards are unaffected: grade the model's next turn however you would grade any response — a submit tool, a [terminal tool](/environments/using-terminal-tools), an [LLM grader](/environments/using-llm-graders) or [rubrics](/environments/using-rubrics).
## Consuming a conversation
On the client, `session.get_messages()` returns the conversation, with optional provider formatting:
```python theme={null}
with environment.session(task=task) as session:
# Provider-shaped, ready to send:
input_messages = session.get_messages(format="openai")
response = oai_client.chat.completions.create(
model="gpt-5.2", messages=input_messages, tools=tools,
)
```
The `format` argument mirrors `list_tools(format=...)`:
| Format | Returns |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `None` | `list[Message]` — the raw conversation |
| `"openai"` / `"openrouter"` | a chat-completions message list |
| `"anthropic"` | `{"system": Optional[str], "messages": [...]}` — leading system turns hoisted into the separate system parameter |
| `"google"` | `{"system_instruction": Optional[str], "contents": [...]}` — the assistant role mapped to `"model"` |
## Rich conversations: reasoning and tool history
Requires `openreward>=0.1.157`.
A conversation the policy must continue often contains more than text: past reasoning (interleaved-thinking models carry reasoning across turns, and OpenReward reasoning is plain text meant to be passed back in) and tool interactions (an agent transcript is a conversation). Three further block types cover this, following **Anthropic message semantics** — everything is a content block, and no `role: "tool"` exists:
| Block | Lives in | Fields |
| ----------------- | ----------------- | ----------------------------------------------------------------------------------- |
| `ReasoningBlock` | `assistant` turns | `text`, optional `signature` (passthrough for provider-signed thinking) |
| `ToolCallBlock` | `assistant` turns | `id`, `name`, `input` |
| `ToolResultBlock` | `user` turns | `tool_call_id`, `blocks` (ordinary text/image content — images welcome), `is_error` |
Placement is validated on the `Message` model itself — a `tool_result` in an assistant turn fails at construction with a clear error, not at the provider.
```python theme={null}
from openreward.environments import (Message, ReasoningBlock, TextBlock,
ToolCallBlock, ToolResultBlock)
def get_messages(self) -> list[Message]:
return [
Message(role="user", blocks=[TextBlock(text="My order #4411 hasn't arrived.")]),
Message(role="assistant", blocks=[
ReasoningBlock(text="Two weeks late — check the carrier first."),
ToolCallBlock(id="tc_1", name="lookup_order", input={"id": "4411"}),
]),
Message(role="user", blocks=[
ToolResultBlock(tool_call_id="tc_1",
blocks=[TextBlock(text='{"status": "stalled_at_hub"}')]),
]),
]
```
The example above builds blocks by hand; if you'd rather keep conversations in your task data, store them in the canonical `Message` wire shape and let pydantic parse them — all block types, placement validation included:
```python theme={null}
def get_messages(self) -> list[Message]:
return [Message.model_validate(m) for m in self.task_spec["conversation"]]
```
```json theme={null}
{
"conversation": [
{"role": "user", "blocks": [{"type": "text", "text": "My order #4411 hasn't arrived."}]},
{"role": "assistant", "blocks": [
{"type": "reasoning", "text": "Two weeks late — check the carrier first."},
{"type": "tool_call", "id": "tc_1", "name": "lookup_order", "input": {"id": "4411"}}
]},
{"role": "user", "blocks": [
{"type": "tool_result", "tool_call_id": "tc_1",
"blocks": [{"type": "text", "text": "{\"status\": \"stalled_at_hub\"}"}]}
]}
]
}
```
`format` conversion handles each provider's shape wherever its schema can represent the block:
| Format | Reasoning | Tool calls / results |
| --------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"anthropic"` | native `thinking` blocks (`signature` included only when set) | native `tool_use` / `tool_result` |
| `"openai"` / `"openrouter"` | `reasoning_content` field on the assistant message (the DeepSeek/vLLM/SGLang convention) | native `tool_calls`; each tool result becomes its own `role: "tool"` message. Images inside tool results raise — tool messages are text-only there |
| `"google"` | raises — no input representation for unencrypted reasoning | native `function_call` / `function_response` parts |
Where a schema can't represent a block, conversion raises a clear `ValueError` rather than silently dropping content; `format=None` always returns the full canonical messages.
## Compatibility
Both prompt surfaces coexist, and `/prompt` remains the primary one:
* **Existing environments and harnesses change nothing.** A prompt-based environment behaves byte-identically to before.
* **`get_messages()` works against every environment.** Against a prompt-based environment it returns the prompt wrapped as a single user message; against a server older than `0.1.157` (which has no `/messages` route) the client falls back the same way. A harness can adopt `get_messages()` unconditionally.
* **Old harnesses still work against conversation environments.** `/prompt` is served for every environment; a conversation flattens to `[role]`-tagged text blocks (rich blocks render to text: reasoning as `…`, tool calls as `[tool_call name] {args}` lines). A conversation consisting of exactly one plain user message flattens to its blocks verbatim.
* **Rich blocks are versioned.** A server's supported block set is advertised as `message_block_types` in the tool list (absent on servers before `0.1.157`, meaning text/image only). Clients skip block types they don't know when reading `/messages` (lossy but safe), so future block types degrade the same way.
## Next Steps
Conversation-shaped tasks pair naturally with preference-based (A ≻ B) rewards
# Preference Rewards
Source: https://docs.openreward.ai/environments/preference-rewards
Constitutional-AI-style rewards from A ≻ B comparisons
## Goals
* Understand when relative (preference-based) rewards beat absolute scoring
* Learn the judging toolkit: pairwise judges with constitutional principles, and group ranking
* Implement `score_group` so a trainer can have the environment judge a whole GRPO group
## Prerequisites
* `openreward>=0.1.157`
* Completing the [Using LLM Graders](/environments/using-llm-graders), [Chat Backends](/environments/chat-backends) and [Using Rubrics](/environments/using-rubrics) tutorials
## Introduction
[Rubrics](/environments/using-rubrics) score one response at a time against criteria — the *pointwise* half of constitutional AI, where a judge checks a response against principles. But the reinforcement-learning half of constitutional AI (RLAIF) is inherently *pairwise*: the judge is shown a conversation and **two** candidate responses, and picks the better one against a principle. The preference signal is relative, not absolute.
Relative rewards suit group-based RL algorithms like GRPO unusually well. GRPO normalizes rewards within a group of rollouts, so only their *ordering* matters — and pointwise judge scores tend to cluster tightly within a group, letting the normalization amplify judge noise into illusory advantages. Win-rates over pairwise comparisons spread by construction.
Two things ship for this:
1. **A judging toolkit** — `openreward.judging` — with a pairwise judge, constitutional principles, and group-ranking helpers. It's a plain library built on [chat backends](/environments/chat-backends), usable anywhere.
2. **A `score_group` capability** — an optional method on your environment that a trainer can call with a whole group's candidate trajectories after the rollouts finish, receiving one reward per candidate.
## The pairwise judge
```python theme={null}
from openreward.chat_backends import resolve_backend
from openreward.judging import PairwiseJudge
CONSTITUTION = [
"Choose the response that is more helpful to the customer's actual problem.",
"Choose the response that is more honest about what the assistant can and cannot do.",
"Choose the response less likely to frustrate or mislead the customer.",
]
judge = PairwiseJudge(
model="gpt-5.2",
backend=resolve_backend(secrets=secrets), # the dedicated judge channel
principles=CONSTITUTION,
)
verdict = await judge.compare(candidate_a, candidate_b, context=conversation)
verdict.winner # "a" | "b" | "tie"
verdict.principle # which principle this comparison was judged against
```
Candidates can be plain strings (final answers) or role-tagged transcripts (`list[Message]`). By default the judge sees only a transcript's final output — see [`view`](#what-the-judge-sees-view) below. `context` is the task conversation the candidates respond to.
The defaults follow the constitutional-AI literature rather than convenience:
* **One principle is sampled per comparison** from `principles`. Ensembling over principles gives a more robust preference signal than judging everything against one fixed rubric string.
* **Presentation order is shuffled per comparison** (`shuffle=True`). Pairwise judges show model-specific position bias; shuffling decorrelates it. `both_orders=True` upgrades this to judging both presentations (2× cost): agreement keeps the winner, disagreement becomes a tie.
* **A broken judge raises, it never scores.** An unparseable verdict gets a bounded corrective retry (`max_retries`), then raises `JudgeFailure`. Silent zeros corrupt training; catch the exception at the reward boundary and fail loudly.
## Ranking a group
`rank_group` turns pairwise verdicts over a group into one scalar per candidate:
```python theme={null}
from openreward.judging import rank_group
rewards = await rank_group(
candidates, judge,
context=conversation,
topology="round_robin", # or "knockout"
aggregator="winrate", # or "rank"
)
```
The **topology** decides which pairs get judged — the cost/fidelity knob:
| Topology | Judge comparisons (group of N) | Character |
| ---------------- | ------------------------------ | ---------------------------------------------------------------------- |
| `GroupwiseJudge` | 1 call | All candidates scored in one call. Cheapest, weakest discrimination. |
| `knockout` | N − 1 | Elimination bracket. Linear cost, coarse resolution among non-winners. |
| `round_robin` | N(N−1)/2 | Every pair judged. The gold standard. |
The **aggregator** decides how verdicts become scalars: `winrate` (wins / games, ties count half) or `rank` (ordering only, evenly spaced — for when you trust the judge's ordering but not its magnitudes).
### What the judge sees: `view`
By default judges **reward the output, not the chain of thought**: with `view="output"` (the default), a transcript candidate renders as only the final assistant turn's visible content — no reasoning, no tool history, no earlier turns. This judges the behavior you actually deploy, and it keeps optimization pressure off the reasoning itself.
Do not train on judged chain of thought. Putting reward on reasoning trains it to *look* good to the judge rather than to be a faithful record of the model's process: models learn to write judge-pleasing thoughts and to move behavior the judge would penalize out of them, which both invites reward hacking and degrades the chain of thought's value for monitoring. Keep reward on outcomes.
`view="trajectory"` is an advanced opt-in for the narrower case where the *actions* matter: agentic tasks where the judge should see tool calls and their results — did the candidate verify before answering, did it take a destructive shortcut (requires `openreward>=0.1.157` for candidates carrying [rich blocks](/environments/multi-turn-conversations#rich-conversations-reasoning-and-tool-history)). Be aware that everything shown to the judge is under optimization pressure, and the trajectory view includes any reasoning in the transcript — so if you use it, write principles that reference actions and outcomes, never reasoning style or thoroughness, and consider sending candidates without reasoning blocks at all. Expect the judge bill to grow with transcript length. The task `context` always renders in full under either view; it's the shared prompt, not the thing being judged.
At N=8 rollouts a round robin is 28 comparisons per task per step — budget accordingly, or drop to `knockout` (7) or the single-call `GroupwiseJudge`.
## Group scoring: `score_group`
Within a session, an environment only ever sees its own rollout — sibling rollouts live in sibling sessions. `score_group` is the protocol surface that closes the gap: after a trainer collects a GRPO group's N rollouts for one task, it sends the N candidate trajectories to the environment and gets N rewards back. The environment owns the constitution and the judging.
Override the method to enable it:
```python theme={null}
from openreward.chat_backends import resolve_backend
from openreward.environments import (Environment, GroupCandidate, GroupScore,
JSONObject, Message)
from openreward.judging import PairwiseJudge, rank_group
class SupportEnv(Environment):
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.secrets = secrets
def get_messages(self) -> list[Message]:
... # the conversation so far — see Multi-turn Conversations
async def score_group(self, candidates: list[GroupCandidate]) -> list[GroupScore]:
judge = PairwiseJudge(
model="gpt-5.2",
backend=resolve_backend(secrets=self.secrets),
principles=CONSTITUTION,
)
rewards = await rank_group(
[c.messages for c in candidates], judge,
context=self.get_messages(),
topology="knockout",
aggregator="winrate",
)
return [GroupScore(reward=r) for r in rewards]
```
Each `GroupCandidate` carries the candidate as role-tagged `messages` plus optional trainer-owned `metadata`, echoed back for correlation. With the default `view="output"` only the final assistant turn is judged, so output-only candidates suffice; send fuller trajectories (optionally with reasoning/tool blocks) only for `view="trajectory"` environments. Return exactly one score per candidate, index-aligned; bare floats are coerced to `GroupScore`. If the judge fails, let the exception propagate — the trainer receives a clean failure for the group instead of fabricated rewards.
Support is advertised automatically: environments that override `score_group` report `supports_score_group: true` in their tool list (the same optional-field pattern as the [terminal tool](/environments/using-terminal-tools)), so harnesses can detect it before spending a single rollout.
## Calling it from a trainer
```python theme={null}
async with environment.session(task=task) as session:
if await session.supports_score_group():
# ... collect the group's N rollouts (their own sessions) ...
scores = await session.score_group(candidate_trajectories)
group_rewards = [s.reward for s in scores]
```
The call is served over the same resumable streaming as tool calls, so a slow tournament (minutes of judge calls) survives disconnects without re-judging. Two failure modes are distinguished: `ScoreGroupUnsupported` (the environment doesn't implement it, or the server predates `0.1.157`) and `ScoreGroupFailed` (the judging raised). Neither kills the session — scoring reads the candidates without advancing environment state, so a trainer can retry or fall back to the pointwise rewards it already holds.
Comparative rewards arrive **alongside** the pointwise rewards the sessions emitted, never instead of them. How the two combine — sum, replace, gate — is trainer policy, deliberately outside the environment's contract.
## Limitations of Preference Rewards
Preference judging inherits every LLM-as-judge failure mode, plus a few of its own:
* **Judge gaming.** Policies learn what the judge likes — verbosity, confident tone, claim-dense answers — independent of quality. Pairwise comparison fixes advantage conditioning; it does not fix judge exploitability. Keep prescriptive anti-hacking principles in the constitution ("do not reward length", "penalize unsupported claims"), and evaluate checkpoints against judges *other* than the training judge.
* **Position bias.** Left on by default (`shuffle`), upgradeable to `both_orders`. Don't turn it off to save calls; the bias is real and model-specific.
* **Non-transitivity.** Judges produce cycles (A ≻ B ≻ C ≻ A). Win-rate aggregation over more pairs is more robust than single-elimination when cycles are common; if the same group keeps producing near-tied win-rates, the judge is at its resolution limit and the signal is mostly noise.
* **Cost.** Comparisons are judge calls over full trajectories. Round robin is quadratic in group size; budget with the topology table above, and consider escalating to pairwise only for groups where cheaper signals produce zero variance.
## Next Steps
Serve the conversation the candidates respond to as a role-tagged prompt
The dedicated judge-model channel the judging toolkit is built on
# Using a Custom Inference Endpoint
Source: https://docs.openreward.ai/environments/using-a-custom-inference-endpoint
Redirect a hosted environment session at your own OpenAI- or Anthropic-compatible inference server
By default, environment sessions hit the inference endpoints they were built against. When you want a session to instead talk to your own inference server — for example a self-hosted vLLM cluster exposing an OpenAI-compatible API — pass `env_overrides` to `Environment.session(...)`.
```python theme={null}
from openreward import OpenReward
client = OpenReward()
env = client.environments.get("GeneralReasoning/counter")
with env.session(
task=task,
env_overrides={"OPENAI_BASE_URL": "https://my-vllm.example.com/v1"},
) as session:
...
```
The overrides are upserted onto the main container of the session pod at session-create time, before any tool calls run.
## Accepted keys
If you are the owner of the environment, you may override **any** env var on the main container — handy when developing or testing your own environment.
Other callers are restricted to the following keys; anything else returns a `400`:
| Key | Purpose |
| -------------------- | ------------------------------------------- |
| `OPENAI_BASE_URL` | Override the OpenAI-compatible endpoint URL |
| `OPENAI_API_KEY` | Override the OpenAI API key |
| `ANTHROPIC_BASE_URL` | Override the Anthropic endpoint URL |
| `ANTHROPIC_API_KEY` | Override the Anthropic API key |
For API keys you do not need to redirect to a custom endpoint, prefer the `secrets=` channel — it never exposes the raw value to your environment code. See [Keeping Secrets Secret](/environments/keeping-secrets-secret). Use `env_overrides` for the API key only when it is paired with a custom `*_BASE_URL` whose host the secrets egress allowlist would otherwise reject.
## Pool isolation
Sessions created with overrides run in a separate pod pool from default sessions, keyed by a hash of the overrides map. Two callers passing the same `env_overrides` will share a pool; callers passing different overrides will not.
# Using Environment Variants
Source: https://docs.openreward.ai/environments/using-environment-variants
Serve multiple environments from a single server and select them by variant
## Goals
* Understand when and why to host multiple environments in a single project
* Define multiple Environment subclasses and serve them from one server
* Select a specific environment variant from the client
## Prerequisites
* Completion of the [Your First Environment](/environments/your-first-environment) tutorial
## Introduction
So far, each project we've built has had a single Environment class served by a single server. But sometimes you have a family of related environments that share logic, data, or infrastructure. For example, an arithmetic benchmark might have a basic variant (addition and subtraction) and a bitwise variant (AND, OR, XOR). These environments share common patterns - task loading, answer verification, data formats - so it makes sense to keep them in the same codebase rather than maintaining separate projects.
An [ORS](https://openrewardstandard.io) server can host multiple Environment classes. When it does, the relationship between server and environment is no longer one-to-one. Clients need to specify which **variant** they want to interact with.
## Defining multiple environments
Let's build two arithmetic environments that share a common `AnswerParams` model but define different tasks and verification logic.
```python theme={null}
from pydantic import BaseModel
from openreward.environments import Environment, JSONObject, Server, Split, TextBlock, ToolOutput, tool
class AnswerParams(BaseModel):
answer: str
# --- Basic Arithmetic ---
class BasicArithmeticTaskSpec(BaseModel):
id: str
problem: str
answer: int
basic_tasks = [
{"id": "0", "problem": "What is 7 + 3?", "answer": 10},
{"id": "1", "problem": "What is 15 - 8?", "answer": 7},
]
class BasicArithmetic(Environment):
"""Addition and subtraction problems."""
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = BasicArithmeticTaskSpec.model_validate(task_spec)
@classmethod
def list_splits(cls):
return [Split(name="train", type="train")]
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
if split == "train":
return basic_tasks
raise ValueError(f"Unknown split: {split}")
def get_prompt(self):
return [TextBlock(type="text", text=self.config.problem)]
@tool
async def answer(self, params: AnswerParams) -> ToolOutput:
"""Submit your final answer."""
try:
is_correct = int(params.answer) == self.config.answer
except ValueError:
is_correct = False
return ToolOutput(
blocks=[TextBlock(type="text", text="Correct!" if is_correct else "Wrong!")],
reward=1.0 if is_correct else 0.0,
finished=True,
)
# --- Bitwise Arithmetic ---
class BitwiseArithmeticTaskSpec(BaseModel):
id: str
problem: str
answer: int
bitwise_tasks = [
{"id": "0", "problem": "What is 5 AND 3? (bitwise)", "answer": 1},
{"id": "1", "problem": "What is 5 OR 3? (bitwise)", "answer": 7},
{"id": "2", "problem": "What is 5 XOR 3? (bitwise)", "answer": 6},
]
class BitwiseArithmetic(Environment):
"""Bitwise operation problems."""
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = BitwiseArithmeticTaskSpec.model_validate(task_spec)
@classmethod
def list_splits(cls):
return [Split(name="train", type="train")]
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
if split == "train":
return bitwise_tasks
raise ValueError(f"Unknown split: {split}")
def get_prompt(self):
return [TextBlock(type="text", text=self.config.problem)]
@tool
async def answer(self, params: AnswerParams) -> ToolOutput:
"""Submit your final answer."""
try:
is_correct = int(params.answer) == self.config.answer
except ValueError:
is_correct = False
return ToolOutput(
blocks=[TextBlock(type="text", text="Correct!" if is_correct else "Wrong!")],
reward=1.0 if is_correct else 0.0,
finished=True,
)
```
Both environments share `AnswerParams` and follow the same structure. The only differences are the tasks and the domain.
## Serving multiple environments
To serve both environments from a single server, pass them as a list to `Server`:
```python theme={null}
ENVIRONMENTS = [
BasicArithmetic,
BitwiseArithmetic,
]
if __name__ == "__main__":
Server(ENVIRONMENTS).run()
```
Each environment class is registered by its lowercased class name:
* `BasicArithmetic` → `"basicarithmetic"`
* `BitwiseArithmetic` → `"bitwisearithmetic"`
The first environment in the list is the default. If a client doesn't specify a variant, it will interact with `BasicArithmetic`.
Run the server:
```bash theme={null}
python server.py
```
## Selecting a variant from the client
When a server hosts a single environment, you don't need to specify a variant:
```python theme={null}
# Single environment — no variant needed
environment = or_client.environments.get(name="gsm8k", base_url="http://localhost:8080")
```
When a server hosts multiple environments, you need to pass the `variant` parameter to target a specific one:
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
# Get the basic arithmetic variant
basic_env = or_client.environments.get(
name="ArithmeticEnv",
variant="basicarithmetic",
base_url="http://localhost:8080"
)
# Get the bitwise arithmetic variant
bitwise_env = or_client.environments.get(
name="ArithmeticEnv",
variant="bitwisearithmetic",
base_url="http://localhost:8080"
)
```
From here, each environment works exactly as before. You can list tasks, start sessions, and call tools independently:
```python theme={null}
# List tasks for each variant
basic_tasks = basic_env.list_tasks(split="train")
bitwise_tasks = bitwise_env.list_tasks(split="train")
print(f"Basic tasks: {len(basic_tasks)}") # 2
print(f"Bitwise tasks: {len(bitwise_tasks)}") # 3
# Run a session on the basic variant
with basic_env.session(task=basic_tasks[0]) as session:
prompt = session.get_prompt()
print(prompt[0].text) # "What is 7 + 3?"
result = session.call_tool("answer", {"answer": "10"})
print(result.reward) # 1.0
# Run a session on the bitwise variant
with bitwise_env.session(task=bitwise_tasks[0]) as session:
prompt = session.get_prompt()
print(prompt[0].text) # "What is 5 AND 3? (bitwise)"
result = session.call_tool("answer", {"answer": "1"})
print(result.reward) # 1.0
```
## Organizing your code
The environment classes can live in the same file or in separate modules. For a small number of variants, a single file is fine. As the number of variants grows, splitting them into separate files keeps things manageable:
```python theme={null}
from basic_arithmetic import BasicArithmetic
from bitwise_arithmetic import BitwiseArithmetic
from modular_arithmetic import ModularArithmetic
from roman_numeral_arithmetic import RomanNumeralArithmetic
ENVIRONMENTS = [
BasicArithmetic,
BitwiseArithmetic,
ModularArithmetic,
RomanNumeralArithmetic,
]
if __name__ == "__main__":
Server(ENVIRONMENTS).run()
```
Shared logic - task spec models, grading utilities, data loading - can go in common modules that each environment imports. This is the main benefit of keeping related environments in one project: you write the shared code once.
# Using LLM Graders
Source: https://docs.openreward.ai/environments/using-llm-graders
Use language models to assign rewards
## Goals
* Learn why we often need language model graders
* Understand how to integrate them into an environment
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
## Introduction
Many tasks can be easily verified with rules-based parsers. For example, in a task where the answer is an integer (e.g. `4`) and there is a `submit_answer` tool, we can simply compare the model answer to the ground truth, e.g. `model_answer == ground_truth_answer` and assign a reward accordingly.
But other tasks are harder to grade with simple rules. Consider a medical question where the model answer is `acetaminophen` but the ground truth answer is `paracetamol`. If we did string matching, we wouldn't find a match - but it turns out these are two different names for the same drug.
For this reason, we often use **LLM graders** to assign reward. We pass the question, the model answer and the ground truth - along with any other instructions - into a language model and ask it to identify whether the answer is correct. This is more expensive than a rules-based parser, but it is more general in that it uses the knowledge of a language model to help assign a reward.
Graders can give a binary response, such as yes or no, but can also give partial credit. Often partial credit is helpful in reinforcement learning as it helps the model improve without getting the full solution.
## Example
In this tutorial we'll see how to set up a binary grader with the drug name example mentioned in the introduction.
First, make sure you have the `openreward` library installed:
```bash theme={null}
pip install openreward
```
To begin we'll initialise our project with a `basic` template:
```bash theme={null}
orwd init medicalenv --template basic
cd medicalenv && ls
```
We'll replace the `train_tasks` and `test_tasks` with the following:
```python theme={null}
train_tasks = []
test_tasks = [{"id": "train-0", "problem": "What is the drug sold under the brand name Tylenol?", "solution": "acetaminophen"}]
```
Now we'll need to define an LLM grader. For this tutorial we'll use an OpenAI grader, but you can choose any grader of your choice.
We'll import the `AsyncOpenAI` client and rewrite our `__init__.py`:
```python theme={null}
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = BasicTaskSpec.model_validate(task_spec)
self.grader_client = AsyncOpenAI(api_key=secrets["OPENAI_API_KEY"])
```
Now let's make a function outside our environment client:
```python theme={null}
async def grade_model_response(client: AsyncOpenAI, model_response: str, solution: str) -> float:
"""
Grades the model response against the solution. Return a reward.
"""
prompt = [
{
"role": "system",
"content": "You are an expert grader. If the reference and model answer match or are equivalent, output a score of 1. Otherwise, give a score of 0. Only respond with a JSON object with a single key 'score' and a value between 0 and 1."
},
{
"role": "user",
"content": f"Reference: {solution}. Model answer: {model_response}"
}
]
response = await client.chat.completions.create(
model="gpt-5-mini",
messages=prompt
)
return float(json.loads(response.choices[0].message.content)["score"])
```
Here we pass in a `model_response` and a `solution`, ask a model to grade it, and then return a JSON response with a score key.
Next we can rewrite the answer tool:
```python theme={null}
@tool
async def answer(self, params: AnswerParams) -> ToolOutput:
"""
The answer tool can be used to submit your final answer. Note that this finishes the episode.
"""
reward = await grade_model_response(self.grader_client, params.answer, self.config.solution)
return ToolOutput(
blocks=[TextBlock(type="text", text=f"Your answer was graded with a score of {reward}")],
reward=reward,
finished=True
)
```
And now we can test. Our full `server.py` is:
```python theme={null}
from pydantic import BaseModel
from openreward.environments import Environment, JSONObject, Server, Split, TextBlock, ToolOutput, tool
from openai import AsyncOpenAI
import json
class BasicTaskSpec(BaseModel):
"""
Each environment has a list of tasks. A task is a dict which contains information about a particular problem in an environment.
Examples:
- A math environment might have a problem and a solution
"""
id: str
problem: str
solution: str
class AnswerParams(BaseModel):
"""
Each tool takes in arguments, and these are specified using types.
Examples:
- An answer tool might have an answer argument
- A bash tool might have a command argument
"""
answer: str
async def grade_model_response(client: AsyncOpenAI, model_response: str, solution: str) -> float:
"""
Grades the model response against the solution. Return a reward.
"""
prompt = [
{
"role": "system",
"content": "You are an expert grader. If the reference and model answer match or are equivalent, output a score of 1. Otherwise, give a score of 0. Only respond with a JSON object with a single key 'score' and a value between 0 and 1."
},
{
"role": "user",
"content": f"Reference: {solution}. Model answer: {model_response}"
}
]
response = await client.chat.completions.create(
model="gpt-5-mini",
messages=prompt
)
return float(json.loads(response.choices[0].message.content)["score"])
train_tasks = []
test_tasks = [{"id": "train-0", "problem": "What is the drug sold under the brand name Tylenol?", "solution": "acetaminophen"}]
class BasicEnvironment(Environment):
"""
A BasicEnvironment showing the main methods needed to define a working environment.
"""
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = BasicTaskSpec.model_validate(task_spec)
self.grader_client = AsyncOpenAI(api_key=secrets["OPENAI_API_KEY"])
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
"""
This method is used to find the available environment tasks for a particular split in the dataset.
"""
if split == "train":
return train_tasks
elif split == "test":
return test_tasks
raise ValueError(f"Unknown split: {split}")
@classmethod
def list_splits(cls):
"""
This method is used to list all the splits in the dataset.
"""
return [Split(name="train", type="train"), Split(name="test", type="test")]
def get_prompt(self) -> str:
"""
This method is used to obtain the prompt that should be used when the agent is starting an episode in the environment.
"""
return [TextBlock(type="text", text=self.config.problem)]
@tool
async def answer(self, params: AnswerParams) -> ToolOutput:
"""
The answer tool can be used to submit your final answer. Note that this finishes the episode.
"""
reward = await grade_model_response(self.grader_client, params.answer, self.config.solution)
return ToolOutput(
blocks=[TextBlock(type="text", text=f"Your answer was graded with a score of {reward}")],
reward=reward,
finished=True
)
if __name__ == "__main__":
Server([BasicEnvironment]).run()
```
To sample, run the following code:
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenAI](https://platform.openai.com/api-keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="medicalenv", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="test")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"OPENAI_API_KEY": os.getenv("OPENAI_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'What is the drug sold under the brand name Tylenol?'}]
[ResponseOutputMessage(id='msg_0c452008c87c52ac006976148ab2fc81a2b15b31b79ab6e53d', content=[ResponseOutputText(annotations=[], text='Tylenol is the brand name for **acetaminophen** (also called **paracetamol** in many countries).', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"Tylenol is the brand name for **acetaminophen** (also called **paracetamol** in many countries)."}', call_id='call_txGlADVzH3YNuBzGZ4tnrjoJ', name='answer', type='function_call', id='fc_0c452008c87c52ac006976148bbba481a2830e9cd56d63799d', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_txGlADVzH3YNuBzGZ4tnrjoJ', 'output': '{"result": "Your answer was graded with a score of 1.0"}'}
```
You should note that because the grader requires an OpenAI API key, we pass in the key as follows:
```python theme={null}
with environment.session(task=example_task, secrets={"OPENAI_API_KEY": os.getenv("OPENAI_API_KEY")}) as session:
```
If you are using a grader from another provider, your code will follow the same pattern. You will need to capture this secret server-side as well so the key from the client can be passed into the grader on the server backend:
```python theme={null}
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = BasicTaskSpec.model_validate(task_spec)
self.grader_client = YourGraderClient(api_key=secrets["YOUR_GRADER_KEY"])
```
## Advanced Graders
In the previous example, we asked the grader to output a score alone. In some cases we may want the agent to do some limited thinking before outputting this score, or at least offer transparent reasoning that leads to its score.
To see how we can do this, let's first rewrite the grader:
```python theme={null}
import re
async def grade_model_response(client: AsyncOpenAI, model_response: str, solution: str) -> tuple[float, str | None]:
"""
Grades the model response against the solution. Return a reward and reasoning.
"""
prompt = [
{
"role": "system",
"content": "You are an expert grader. If the reference and model answer match or are equivalent, output a score of 1. Otherwise, give a score of 0. You should first provide some reasoning in and then afterwrds output your score inside which contains only a JSON object with a single key 'score' and a value between 0 and 1."
},
{
"role": "user",
"content": f"Reference: {solution}. Model answer: {model_response}"
}
]
response = await client.chat.completions.create(
model="gpt-5-mini",
messages=prompt
)
# The model response should contain ... (optional) and ...
# Extract the content between and and between and
content = response.choices[0].message.content
reasoning = None
reasoning_match = re.search(r"(.*?)", content, re.DOTALL | re.IGNORECASE)
if reasoning_match:
reasoning = reasoning_match.group(1).strip()
answer_json = None
answer_match = re.search(r"(.*?)", content, re.DOTALL | re.IGNORECASE)
if answer_match:
answer_text = answer_match.group(1).strip()
try:
answer_json = json.loads(answer_text)
except Exception:
json_match = re.search(r"\{.*\}", answer_text, re.DOTALL)
if json_match:
answer_json = json.loads(json_match.group(0))
else:
raise ValueError("Could not extract score JSON from answer tag.")
else:
# Fallback to old behavior if not found
answer_json = json.loads(content)
return float(answer_json["score"]), reasoning
```
And let us also rewrite the `answer` tool:
```python theme={null}
@tool
async def answer(self, params: AnswerParams) -> ToolOutput:
"""
The answer tool can be used to submit your final answer. Note that this finishes the episode.
"""
reward, reasoning = await grade_model_response(self.grader_client, params.answer, self.config.solution)
return ToolOutput(
blocks=[TextBlock(type="text", text=f"Your answer was graded with a score of {reward}. Reasoning: {reasoning}")],
reward=reward,
finished=True
)
```
Now rerun `quickstart.py`:
```python theme={null}
python quickstart.py
```
```
[{'role': 'user', 'content': 'What is the drug sold under the brand name Tylenol?'}]
[ResponseFunctionToolCall(arguments='{"answer":"Tylenol is the brand name for **acetaminophen** (also called **paracetamol**)."}', call_id='call_2L1hRzCMF4id67XzrGBDEDju', name='answer', type='function_call', id='fc_0d546a6135c94ead006976166700088193b7223a1e2b0021d9', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_2L1hRzCMF4id67XzrGBDEDju', 'output': '{"result": "Your answer was graded with a score of 1.0. Reasoning: The model answer names Tylenol as the brand name for acetaminophen and notes the synonym paracetamol, which matches the reference term \\"acetaminophen.\\" The content is equivalent."}'}
```
As we can see we have the same result but now the agent is incentivised to spend tokens doing some reasoning before answering.
In general, you should test different graders and observe their outputs and decide which one better matches human judgement. The direct grader is cheaper in that it spends fewer tokens, but if you need higher grader accuracy you may want to opt for graders like this.
A more expensive grader entirely involves using a reasoning model to judge the output. This is essentially a more extreme version of the grader we have just made, where a model will spend many more tokens thinking before giving a reward. These types of graders are also called "generative verifiers".
We can use one of these graders by changing our original grader to the following:
```python theme={null}
async def grade_model_response(client: AsyncOpenAI, model_response: str, solution: str) -> tuple[float, str]:
"""
Grades the model response against the solution. Return a reward and reasoning.
"""
prompt = [
{
"role": "system",
"content": "You are an expert grader. If the reference and model answer match or are equivalent, output a score of 1. Otherwise, give a score of 0. Only respond with a JSON object with a single key 'score' and a value between 0 and 1."
},
{
"role": "user",
"content": f"Reference: {solution}. Model answer: {model_response}"
}
]
response = await client.responses.create(
model="gpt-5.4",
reasoning={"effort": "medium"},
input=prompt
)
return float(json.loads(response.output_text)["score"]), ""
```
```python theme={null}
python quickstart.py
```
```
[{'role': 'user', 'content': 'What is the drug sold under the brand name Tylenol?'}]
[ResponseOutputMessage(id='msg_0b1df4bd5a513b1400697617e0d75481978e84348c273060dc', content=[ResponseOutputText(annotations=[], text='Tylenol is the brand name for **acetaminophen** (also called **paracetamol** in many countries).', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"Tylenol is the brand name for **acetaminophen** (also called **paracetamol** in many countries)."}', call_id='call_YwKT2KrCOeqJJCRzB7dFjuXa', name='answer', type='function_call', id='fc_0b1df4bd5a513b1400697617e1d6c0819786ba7acf559fdee6', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_YwKT2KrCOeqJJCRzB7dFjuXa', 'output': '{"result": "Your answer was graded with a score of 1.0. Reasoning: "}'}
```
Most model providers hide the reasoning, so it won't be visible in this case.
Note that a full reasoning model is not needed for verification given the simplicity of this task, but there may be harder verification problems where they may be useful. The [Rubrics](/environments/using-rubrics) tutorial gives some examples of these type of problems.
## Next Steps
Make graders composable: pin them to the real provider endpoint even when training infrastructure overrides OPENAI\_BASE\_URL
Learn how to use rubrics for harder-to-verify domains and partial credit scoring
# Using Rubrics
Source: https://docs.openreward.ai/environments/using-rubrics
Assigning rewards for harder-to-verify domains
## Goals
* Learn why rubrics are useful for assigning reward
* Learning how to define rubrics for an environment
* Understand how to integrate them into an ORS environment
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
* Completing the [Your First Environment](/environments/your-first-environment) and [Using LLM Graders](/environments/using-llm-graders) tutorials.
## Introduction
In the [Using LLM Graders](/environments/using-llm-graders) tutorial, we saw we could use language models to assign rewards where it was hard to match a model and ground truth answer using string matching. But this still relied on the notion of a "reference answer" to compare against.
What about domains where a reference answer isn't available? Consider a task where we task a model to write an essay on the causes of World War One. There isn't a "correct" answer in this case, but a teacher would still need to grade an essay based on the quality of reasoning, the evidence presented, and more.
A grading **Rubric** is a criterion for grading a model response. A rubric can be:
* **Binary**: for example, did the answer mention a particular event or not? Did it pass a unit test or not?
* **Point-based**: for example, it could allocate 0 points for not mentioning an important factor, 0.5 points for mentioning it but not specific details, and 1.0 points for mentioning the factor and the details.
* **Weighted**: we might weight one rubric higher than another when considering the total score.
Typically, we calculate the final reward as a weighted sum of rubrics, where:
$r = \frac{\sum_{i}^{K}w_{i}s_{i}}{\sum_{i}w_{i}}$
where $w_i$ is the weight on a rubric, and $s_i$ is its score.
Multiple rubrics, and/or point-based reward, is useful for reinforcement learning as it enables a more continuous reward signal to distinguish between different responses.
Rubrics are particularly useful for long-form language model responses. For example, they power the training of [DeepResearch](https://openai.com/index/introducing-deep-research/) type models.
Now let's see how rubrics work in action.
## Worked Example
Let's make an ORS environment locally. First, if you haven't already:
```bash theme={null}
pip install openreward
```
We'll initialise a new project:
```bash theme={null}
orwd init historyenv --template basic
cd historyenv && ls
```
First, we'll add some additional imports to our `server.py` file:
```python theme={null}
import asyncio
import re
from typing import List
import openai
```
Next we'll alter train\_tasks and test\_tasks. We'll focus on the question we asked earlier in this tutorial:
```python theme={null}
train_tasks = [
{
"id": "wwi-causes",
"problem": "What were the causes of World War One?",
"rubrics": WWI_RUBRICS
}
]
# Empty test split - only train split has tasks
test_tasks = []
```
We'll need to define the rubrics above. We'll use ten separate rubrics with a point scale:
```python theme={null}
# WWI Essay Rubrics (10 criteria, each worth 1.0 point)
WWI_RUBRICS = [
{
"criterion": "Direct answer + defensible thesis about 'cause'. 0: No clear answer; descriptive list. +0.5: Vague thesis that shifts. +1.0: Clear, arguable thesis defining causation that stays consistent.",
"points": 1.0
},
{
"criterion": "Chronological command of 1908–14 and July Crisis. 0: Major errors. +0.5: Broad chronology right but key steps missing. +1.0: Accurate tight chronology showing decision turning points.",
"points": 1.0
},
{
"criterion": "Explains why 1914 (timing problem). 0: Doesn't address why earlier crises didn't produce war. +0.5: Mentions earlier crises without analysis. +1.0: Uses pre-1914 crises analytically to explain escalation.",
"points": 1.0
},
{
"criterion": "Structural causes: alliances/diplomacy as mechanisms. 0: Alliances as automatic dominoes without explanation. +0.5: Describes alliances but under-specifies mechanism. +1.0: Shows how commitments shaped incentives and constraints.",
"points": 1.0
},
{
"criterion": "Militarism and war planning. 0: 'Countries liked war' with no institutional explanation. +0.5: Mentions arms race/plans without crisis connection. +1.0: Demonstrates how timetables/plans accelerated escalation.",
"points": 1.0
},
{
"criterion": "Nationalism and the Balkans. 0: Balkan nationalism as background noise. +0.5: Acknowledges volatility but agency stays with great powers. +1.0: Explains how Balkan politics interacted with great-power calculations.",
"points": 1.0
},
{
"criterion": "Great-power agency. 0: One-sided blame or vague 'everyone equally'. +0.5: Covers actors unevenly; responsibility asserted not demonstrated. +1.0: Weighs actors with specific motives/constraints; responsibility argued.",
"points": 1.0
},
{
"criterion": "Historiography. 0: Little engagement or name-dropping. +0.5: Uses interpretations but evaluation is thin. +1.0: Engages competing interpretations as arguments and shows thesis relation.",
"points": 1.0
},
{
"criterion": "Evidence and scholarship. 0: Sparse/unsuitable evidence. +0.5: Uses works but cherry-picks or doesn't distinguish evidence from interpretation. +1.0: Uses evidence discriminately with acknowledged limits.",
"points": 1.0
},
{
"criterion": "Argumentative craft. 0: Unclear structure, weak paragraphs. +0.5: Readable and structured but argument gets lost occasionally. +1.0: Coherent architecture, strong paragraphs, professional apparatus.",
"points": 1.0
}
]
```
Next we'll need a template for the LLM grader:
```python theme={null}
GRADER_TEMPLATE = """You are an expert history evaluator specializing in World War One historiography. Evaluate the following essay response against a specific criterion.
QUESTION:
{question}
SUBMITTED ESSAY:
{response}
EVALUATION CRITERION:
{criterion}
Instructions:
1. Analyze how well the essay meets this specific criterion
2. The criterion specifies scores: 0, +0.5, or +1.0 - choose the appropriate score
3. Provide a brief explanation (2-3 sentences) justifying your score
4. Consider the quality of argument, evidence, and historical analysis
Output format:
Analysis: [Your 2-3 sentence explanation]
Score: [One of: 0, 0.5, or 1.0]
"""
```
We'll also need to update `BasicTaskSpec` to include the rubrics field:
```python theme={null}
class BasicTaskSpec(BaseModel):
id: str
problem: str
rubrics: list[dict]
```
We'll change the **init** to include the grader. We'll use an OpenAI model as the grader:
```python theme={null}
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = BasicTaskSpec.model_validate(task_spec)
api_key = secrets.get("OPENAI_API_KEY")
if not api_key:
raise ValueError("OpenAI API key required in secrets parameter")
self.client = openai.AsyncClient(api_key=api_key)
# Store rubrics for grading
self.rubrics = self.config.rubrics
```
And we'll define some methods for the new answer tool:
```python theme={null}
def _parse_score(self, grading_response: str, max_points: float) -> float:
"""Parse score from grading response with fallback logic"""
# Look for "Score: X" pattern
match = re.search(r"Score:\s*([\d.]+)", grading_response, re.IGNORECASE)
if match:
score = float(match.group(1))
return max(0.0, min(max_points, score))
# Fallback: find any decimal number
numbers = re.findall(r"\b(\d+(?:\.\d+)?)\b", grading_response)
if numbers:
score = float(numbers[-1])
return max(0.0, min(max_points, score))
return 0.0 # Default if parsing fails
async def _grade_single_criterion(self, response: str, rubric: dict) -> dict:
"""Grade response against a single criterion"""
grader_prompt = GRADER_TEMPLATE.format(
question=self.config.problem,
response=response,
criterion=rubric["criterion"]
)
res = await self.client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": grader_prompt}]
)
grading_response = res.choices[0].message.content or ""
score = self._parse_score(grading_response, rubric["points"])
return {
"criterion": rubric["criterion"],
"max_points": rubric["points"],
"score": score,
"feedback": grading_response
}
async def _grade_all_rubrics(self, response: str) -> List[dict]:
"""Grade response against all 10 rubrics in parallel"""
grading_tasks = [
self._grade_single_criterion(response, rubric)
for rubric in self.rubrics
]
return await asyncio.gather(*grading_tasks)
@tool
async def answer(self, params: AnswerParams) -> ToolOutput:
"""
Submit your essay answer for grading. Your essay will be evaluated against 10 historical rubrics covering thesis, chronology, causation, historiography, evidence, and argumentative craft. Note that this finishes the episode.
"""
essay = params.answer
# Grade all rubrics in parallel
rubric_results = await self._grade_all_rubrics(essay)
# Calculate total score
total_score = sum(r["score"] for r in rubric_results)
total_possible = sum(r["max_points"] for r in rubric_results)
# Normalize reward to 0-1 scale (sum / 10)
reward = total_score / total_possible if total_possible > 0 else 0.0
# Format detailed feedback
feedback_lines = [
f"",
f"**Total Score: {total_score:.1f} / {total_possible:.1f}**",
f"**Normalized Reward: {reward:.3f}**",
f"",
f"## Rubric-by-Rubric Feedback",
f""
]
for i, result in enumerate(rubric_results, 1):
feedback_lines.append(f"### Rubric {i}: {result['score']:.1f} / {result['max_points']:.1f}")
feedback_lines.append(f"**Criterion**: {result['criterion'][:100]}...")
feedback_lines.append(f"**Feedback**: {result['feedback']}")
feedback_lines.append("")
feedback_text = "\n".join(feedback_lines)
return ToolOutput(
blocks=[TextBlock(type="text", text=feedback_text)],
metadata={
"task_id": self.config.id,
"total_score": total_score,
"total_possible": total_possible,
"normalized_reward": reward,
"rubric_results": rubric_results
},
reward=reward,
finished=True
)
```
Once we've made these changes, let's run the environment. In your terminal run:
```bash theme={null}
python server.py
```
To sample, run the following code:
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenAI](https://platform.openai.com/api-keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `test_rubrics.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="historyenv", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"OPENAI_API_KEY": os.getenv("OPENAI_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": tool_result.blocks[0].text
})
print(input_list[-1])
print(reward)
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python test_rubrics.py
```
Example output:
```bash theme={null}
[ResponseOutputMessage(id='msg_00b7c91bdcd6757f0069920e0cd92c81a3ad06e75fa90d7c09', content=[ResponseOutputText(annotations=[], text='World War I (1914–1918) was caused by a combination of long-term structural tensions in Europe and a short-term crisis that spiraled into a general war.\n\n## Long-term causes (underlying conditions)\n\n### 1) Alliance systems and bloc politics\nEurope was divided into two major, increasingly rigid camps:\n- **Triple Alliance**: Germany, Austria-Hungary, Italy (Italy later stayed neutral and then joined the other side)\n- **Triple Entente**: Britain, France, Russia\n\nThese alliances were partly defensive, but they encouraged **chain reactions**: a conflict involving one state could quickly pull in its partners.\n\n### 2) Militarism and arms races\nMajor powers expanded their armies and developed detailed war plans:\n- Germany’s **Schlieffen Plan** envisioned a rapid strike against France if war broke out with Russia.\n- Russia, France, and others had their own mobilization timetables.\n- Britain and Germany competed in a major **naval arms race**.\n\nBecause mobilization was seen as critical, crises became “use it or lose it,” making de-escalation harder.\n\n### 3) Nationalism\nNationalism increased both unity and conflict:\n- **French revanchism** after losing Alsace-Lorraine to Germany (1871).\n- **Pan-Slavism** and Serbian nationalism in the Balkans, challenging Austria-Hungary.\n- Nationalist movements inside multi-ethnic empires (especially Austria-Hungary) threatened internal stability.\n\n### 4) Imperial and economic rivalry\nCompetition for colonies, markets, and prestige heightened distrust:\n- Crises such as those over **Morocco** (1905, 1911) worsened relations, particularly between Germany and France/Britain.\n- Economic power shifts (notably Germany’s rapid growth) fed strategic anxiety.\n\n### 5) Instability in the Balkans (“powder keg”)\nThe decline of Ottoman influence and the ambitions of Austria-Hungary and Russia made the Balkans volatile:\n- **Balkan Wars (1912–1913)** intensified territorial disputes and regional militarization.\n- Serbia’s increased power alarmed Austria-Hungary.\n\n## Short-term cause (trigger and escalation)\n\n### Assassination of Archduke Franz Ferdinand (June 28, 1914)\nA Bosnian Serb nationalist linked to networks in Serbia assassinated the heir to Austria-Hungary in Sarajevo. Austria-Hungary used the event to confront Serbia.\n\n### The July Crisis: how a regional conflict became a world war\nKey steps:\n1. **Germany gave Austria-Hungary strong backing** (“blank check”), encouraging a tough stance.\n2. Austria-Hungary issued an **ultimatum** to Serbia; Serbia accepted most terms but not all.\n3. Austria-Hungary **declared war on Serbia** (July 28).\n4. Russia **mobilized** to support Serbia.\n5. Germany **declared war on Russia** (Aug 1) and **France** (Aug 3).\n6. Germany invaded **Belgium** to reach France, leading Britain to **declare war on Germany** (Aug 4) due to treaty commitments and strategic concerns.\n\n## Bottom line\nWorld War I was not caused by a single factor. It resulted from **entangled alliances, militarized planning, nationalism (especially in the Balkans), imperial rivalries, and repeated crises**, with the assassination in 1914 triggering a rapid escalation that leaders could not—or would not—stop.\n\nIf you want, I can compare the “systemic causes” vs. “blame/decision” arguments historians debate (e.g., Germany’s role, miscalculation, or shared responsibility).', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"World War I (1914–1918) had deep long‑term causes and a short‑term trigger that escalated into a general European war.\\n\\nLong‑term causes\\n1) Alliance systems and bloc politics: By 1914 Europe was divided into two rival groupings—Germany and Austria‑Hungary (with Italy loosely tied in the Triple Alliance) versus France, Russia, and Britain (the Triple Entente). These commitments were meant to deter war but in a crisis they encouraged a chain reaction, turning local disputes into continent‑wide conflict.\\n\\n2) Militarism and arms races: States expanded armies, invested heavily in weapons, and treated mobilization as decisive. War plans (e.g., Germany’s Schlieffen Plan) and tight railway timetables created pressure to mobilize quickly, making leaders fear that delay meant defeat and reducing room for diplomacy once a crisis began.\\n\\n3) Nationalism: Aggressive nationalism sharpened rivalries. France resented the loss of Alsace‑Lorraine (1871). In the Balkans, Serbian nationalism and pan‑Slav sentiment challenged Austria‑Hungary, while nationalist movements within multi‑ethnic empires threatened internal cohesion and pushed leaders toward hardline policies.\\n\\n4) Imperial and economic rivalry: Competition for colonies, markets, and status—seen in crises like Morocco (1905, 1911)—worsened distrust, especially between Germany and the Entente powers.\\n\\n5) Balkan instability: The Ottoman retreat and the Balkan Wars (1912–13) left unresolved territorial and ethnic disputes. Serbia’s rise alarmed Austria‑Hungary; Russia viewed itself as Serbia’s patron, making the region a flashpoint.\\n\\nShort‑term trigger and escalation\\nThe immediate spark was the assassination of Archduke Franz Ferdinand in Sarajevo (28 June 1914) by a Bosnian Serb nationalist connected to Serbian networks. Austria‑Hungary, backed by Germany’s “blank check,” issued a harsh ultimatum to Serbia and then declared war (28 July). Russia mobilized to support Serbia; Germany declared war on Russia (1 Aug) and France (3 Aug). Germany’s invasion of Belgium brought Britain into the war (4 Aug). In weeks, alliance commitments, mobilization schedules, and mutual fears transformed a Balkan crisis into World War I."}', call_id='call_reMjWpi1OS8rfBg0j8pp93fX', name='answer', type='function_call', id='fc_00b7c91bdcd6757f0069920e1b0cfc81a3b5b103f065b9e5a2', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_reMjWpi1OS8rfBg0j8pp93fX', 'output': '\\n**Total Score: 6.5 / 10.0**\\n**Normalized Reward: 0.650**\\n\\n## Rubric-by-Rubric Feedback\\n\\n### Rubric 1: 1.0 / 1.0\\n**Criterion**: Direct answer + defensible thesis about \'cause\'. 0: No clear answer; descriptive list. +0.5: Vague t...\\n**Feedback**: Analysis: The essay presents a clear, defensible thesis that World War I resulted from long\\u2011term structural causes (alliances, militarism, nationalism, imperial rivalry, Balkan instability) combined with the short\\u2011term trigger of Franz Ferdinand\\u2019s assassination, and it consistently applies that framework in the body. It goes beyond a mere list by explaining how these factors interacted to produce mobilisation pressures and a chain reaction, so it meets the criterion for a sustained, arguable claim about causation.\\n\\nScore: 1.0\\n\\n### Rubric 2: 0.5 / 1.0\\n**Criterion**: Chronological command of 1908\\u201314 and July Crisis. 0: Major errors. +0.5: Broad chronology right but ...\\n**Feedback**: Analysis: The essay gives a correct broad sequence from the assassination to general war and mentions relevant 1912\\u201313 Balkan instability, but it lacks tighter chronological command of 1908\\u201314 (no mention of the 1908 Bosnian Crisis or the Agadir/1905\\u201311 tensions) and omits key July 1914 turning-point details and dates (e.g. Germany\\u2019s 5 July \\"blank cheque,\\" Austria\\u2019s ultimatum of 23 July, Serbia\\u2019s reply on 25 July, partial vs. general Russian mobilization). Because the overall order is right but important steps and precise timing in the July Crisis are missing, it merits a partial credit. \\nScore: 0.5\\n\\n### Rubric 3: 0.5 / 1.0\\n**Criterion**: Explains why 1914 (timing problem). 0: Doesn\'t address why earlier crises didn\'t produce war. +0.5: ...\\n**Feedback**: Analysis: The essay names pre\\u20111914 crises (the Moroccan incidents and the Balkan Wars) and notes their role in increasing distrust and leaving unresolved disputes, but it does not analyze why those earlier crises were contained or why 1914 produced general war when they had not. It therefore mentions earlier crises without using them analytically to explain the timing of escalation.\\n\\nScore: 0.5\\n\\n### Rubric 4: 0.5 / 1.0\\n**Criterion**: Structural causes: alliances/diplomacy as mechanisms. 0: Alliances as automatic dominoes without exp...\\n**Feedback**: Analysis: The essay clearly describes the opposing alliance blocs and notes that commitments produced a \\u201cchain reaction\\u201d and cites Germany\\u2019s \\u201cblank check,\\u201d but it stops short of explaining how those commitments concretely shaped state incentives and constrained diplomatic choices (e.g. obligation to mobilize, fear of isolation, bargaining leverage and rigidity). Because the mechanism is asserted rather than analysed in detail, the response fits the partial-credit category. \\nScore: 0.5\\n\\n### Rubric 5: 1.0 / 1.0\\n**Criterion**: Militarism and war planning. 0: \'Countries liked war\' with no institutional explanation. +0.5: Menti...\\n**Feedback**: Analysis: The essay explicitly links militarism to specific institutions and practices\\u2014war plans (Schlieffen), mobilization timetables, and railway schedules\\u2014and explains how these created pressure to mobilize quickly and reduced diplomatic room, thereby accelerating escalation. This goes beyond a vague claim that \\"countries liked war\\" and provides the causal mechanism the criterion requires.\\n\\nScore: 1.0\\n\\n### Rubric 6: 1.0 / 1.0\\n**Criterion**: Nationalism and the Balkans. 0: Balkan nationalism as background noise. +0.5: Acknowledges volatilit...\\n**Feedback**: Analysis: The essay clearly treats Balkan nationalism as more than background, detailing Serbian nationalism, the Balkan Wars, and the Ottoman retreat, and it links these developments to great\\u2011power responses\\u2014Austria\\u2011Hungary\\u2019s alarm, Russia\\u2019s patronage of Serbia, and how the Sarajevo assassination produced a chain of mobilizations. This shows how Balkan politics interacted with and helped shape great\\u2011power calculations, so the treatment meets the highest level of the criterion.\\n\\nScore: 1.0\\n\\n### Rubric 7: 1.0 / 1.0\\n**Criterion**: Great-power agency. 0: One-sided blame or vague \'everyone equally\'. +0.5: Covers actors unevenly; re...\\n**Feedback**: Analysis: The essay attributes clear agency to the main great powers\\u2014Germany (Schlieffen Plan, \\u201cblank check\\u201d), Austria\\u2011Hungary (ultimatum), Russia (mobilization to defend Serbia), France and Britain (alliance commitments, response to Belgian invasion)\\u2014and links these actions to specific motives and constraints like alliance obligations, mobilization timetables, and imperial/nationalist pressures. While it could deepen causal weighting, it nevertheless argues responsibility with concrete examples rather than vague equal\\u2011blame assertions.\\n\\nScore: 1.0\\n\\n### Rubric 8: 0.0 / 1.0\\n**Criterion**: Historiography. 0: Little engagement or name-dropping. +0.5: Uses interpretations but evaluation is ...\\n**Feedback**: Analysis: The essay provides a clear, conventional summary of causes but contains no engagement with historiographical debate\\u2014no historians, schools, or competing interpretations are referenced or evaluated (e.g., Fischer, A.J.P. Taylor, Christopher Clark). It therefore qualifies as little engagement with historiography rather than using or weighing rival scholarly arguments.\\n\\nScore: 0\\n\\n### Rubric 9: 0.0 / 1.0\\n**Criterion**: Evidence and scholarship. 0: Sparse/unsuitable evidence. +0.5: Uses works but cherry-picks or doesn\'...\\n**Feedback**: Analysis: The essay gives accurate factual examples (Schlieffen Plan, Moroccan Crises, Balkan Wars, assassination) but offers no engagement with scholarship or citation of historians/primary sources, nor does it acknowledge limits or competing interpretations (e.g., Fischer, A.J.P. Taylor, Christopher Clark). Because it presents assertions without discriminating evidence or historiographical context, it fails the criterion for evidence and scholarship.\\n\\nScore: 0\\n\\n### Rubric 10: 1.0 / 1.0\\n**Criterion**: Argumentative craft. 0: Unclear structure, weak paragraphs. +0.5: Readable and structured but argume...\\n**Feedback**: Analysis: The essay is well organized with a clear architecture\\u2014separate, focused sections on long\\u2011term causes and the short\\u2011term trigger\\u2014and each paragraph/point concisely advances the argument. It deploys relevant evidence (Schlieffen Plan, Balkan Wars, assassination, \\u201cblank check,\\u201d Belgian invasion) and maintains a coherent causal chain from underlying tensions to escalation. \\nScore: 1.0\\n'}
0.65
```
In the example above we got a reward of 0.65, or 6.5/10. Studying the grader response, we see the model answer lost points because it did not engage with the existing scholarship, misses key dates and other details.
This shows an example of how rubrics work in practice, and how we can use them in ORS environments. But we should reflect: there are some issue with the rubrics we have defined for this environment, as well as the way we have just prompted a language model:
* **Hallucination**. The rubrics, as defined, do not punish hallucination directly. For example, what if the model made up an event or got a date wrong? Our rubrics do not punish this directly.
* **Required output format**. It may be acceptable to omit detail if it is a chatbot response; it is less acceptable if it is a university thesis or a undergraduate essay. We should make it clear to the language model what type of output is required so it knows how much detail is needed.
* **Tools and citations**. We have not given the language model access to a web search tool so it can search the literature. We have also not explicitly mentioned a need for citations, nor defined a rubric for citation use. Likewise, we have not given the grader access to tools that could be used for verification (and to identify and punish hallucination).
This is not a problem with rubrics, per se, but it does show we need to be careful to elicit and reward the right behaviour. When using rubrics to train models, you should pay close attention to their responses to ensure they are not [reward hacking](https://en.wikipedia.org/wiki/Reward_hacking), and are appropriately incentivised to exhibit the right behaviours.
## Limitations of Rubrics
We have already touched upon some limitations of our rubric example, but we should note some general themes to look out for:
* **Reward Hacking** - if we do not specify the desired behaviours correctly, or underspecify requirements, then the model may find a way to achieve reward without exhibiting the desired behaviours.
* **Subjectivity** - if an objective metric is available, we should always prefer that. If we use a subjective human measure of quality, then we may limit the creativity of the model to discover better-than-human solutions.
* **Cost** - LLM graders can become costly, especially in large-scale reinforcement learning. With rubrics, we often use multiple graders for each rubric which multiplies costs further. If we can find simple rules-based methods to identify and reward the same behaviour, then this is preferred to LLM graded rubrics.
* **Grader Quality** - We are also bound by the quality of the grader. If a rubric is difficult to judge, or requires deep expertise, then it is recommended to use reasoning models as graders.
## Next Steps
Rubrics score one response at a time. For comparative grading — where a judge is shown *two or more* candidate responses and picks the better one against a constitution of principles — see the preference-based approach.
Constitutional-AI-style rewards from A ≻ B comparisons across a group of rollouts
# Using Task-Specific Tools
Source: https://docs.openreward.ai/environments/using-task-specific-tools
Define tools that are specific to individual tasks
## Goals
* Understand the difference between shared tools and task-specific tools
* Use `@tool(shared=False)` for tools that should only be available within a session
* Use `list_task_tools()` for fully dynamic tools generated based on the current task
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* Completion of the [Your First Environment](/environments/your-first-environment) tutorial
## Introduction
By default, all methods decorated with `@tool` are **shared** — they are returned by `environment.list_tools()` and are available to any client before a session even starts. This works well for tools like `bash` or `read_file` that are the same regardless of the task.
But some environments need to present different tool interfaces depending on the task. For example, an environment that tests an agent's ability across multiple different configurations might give the agent a `bash` tool for some tasks, a `select_option` tool for multiple-choice tasks, and a completely different set of tools for other task types. The tool interface itself is part of what varies across tasks.
Task-specific tools solve this by making tools available only through `session.list_tools()`, which requires an active session with a task loaded. This lets you vary the set of tools an agent sees on a per-task basis.
## Using `@tool(shared=False)`
The simplest way to create a task-specific tool is to pass `shared=False` to the `@tool` decorator. These tools are not returned by `environment.list_tools()` but are returned by `session.list_tools()`.
```python theme={null}
from openreward.environments import Environment, tool, ToolOutput, TextBlock
from pydantic import BaseModel, Field
class SearchParams(BaseModel):
query: str = Field(..., description="Search query")
class SubmitParams(BaseModel):
answer: str = Field(..., description="Your answer")
class ToolVariantEnvironment(Environment):
# Shared tool — returned by environment.list_tools()
@tool
async def submit_answer(self, params: SubmitParams) -> ToolOutput:
"""Submit the final answer"""
correct = params.answer.strip() == self.task_spec["answer"]
return ToolOutput(
blocks=[TextBlock(text="Correct!" if correct else "Incorrect.")],
reward=1.0 if correct else 0.0,
finished=True,
)
# Task-specific tool — only returned by session.list_tools()
# Some tasks give the agent a search tool, others don't
@tool(shared=False)
async def search(self, params: SearchParams) -> ToolOutput:
"""Search a knowledge base"""
results = search_kb(params.query) # your own search implementation
return ToolOutput(
blocks=[TextBlock(text=results)],
)
@classmethod
def list_splits(cls):
return ["train"]
@classmethod
def list_tasks(cls, split):
return [
{"question": "What is the capital of France?", "answer": "Paris", "tools": ["search"]},
{"question": "What is 2+2?", "answer": "4", "tools": []},
]
def get_prompt(self):
return [TextBlock(text=self.task_spec["question"])]
```
In this example, `submit_answer` is shared and always visible. The `search` tool is task-specific — it only appears when a session is active. Combined with `list_task_tools()` (below), you can control exactly which task-specific tools appear for each task.
## Using `list_task_tools()`
For fully dynamic tools whose definitions depend on the task, override the `list_task_tools()` instance method. This lets you generate tool specifications at runtime based on `self.task_spec` — controlling exactly which tools the agent sees for each task.
```python theme={null}
from openreward.environments import Environment, tool, ToolOutput, TextBlock
from openreward.environments.types import ListToolsOutput, ToolSpec
from pydantic import BaseModel, Field
class SearchParams(BaseModel):
query: str = Field(..., description="Search query")
class SelectOptionParams(BaseModel):
option: str = Field(..., description="Selected option (A, B, C, or D)")
class SubmitParams(BaseModel):
answer: str = Field(..., description="Your answer")
class MultiInterfaceEnvironment(Environment):
"""An environment that presents different tool interfaces per task."""
@tool
async def submit_answer(self, params: SubmitParams) -> ToolOutput:
"""Submit the final answer"""
correct = params.answer.strip() == self.task_spec["answer"]
return ToolOutput(
blocks=[TextBlock(text="Correct!" if correct else "Incorrect.")],
reward=1.0 if correct else 0.0,
finished=True,
)
def list_task_tools(self) -> ListToolsOutput:
"""Present different tools depending on the task type."""
tools = []
task_type = self.task_spec.get("type")
if task_type == "multiple_choice":
tools.append(ToolSpec(
name="select_option",
description="Select an answer option (A, B, C, or D)",
input_schema=SelectOptionParams.model_json_schema(),
))
elif task_type == "open_book":
tools.append(ToolSpec(
name="search",
description="Search a knowledge base for relevant information",
input_schema=SearchParams.model_json_schema(),
))
return ListToolsOutput(tools=tools)
@classmethod
def list_splits(cls):
return ["train"]
@classmethod
def list_tasks(cls, split):
return [
{"type": "multiple_choice", "question": "Capital of France?", "answer": "B"},
{"type": "open_book", "question": "Explain the photoelectric effect.", "answer": "..."},
{"type": "closed_book", "question": "What is 2+2?", "answer": "4"},
]
def get_prompt(self):
return [TextBlock(text=self.task_spec["question"])]
```
In this example, the agent sees different tool interfaces depending on the task:
* **Multiple-choice tasks**: `submit_answer` + `select_option`
* **Open-book tasks**: `submit_answer` + `search`
* **Closed-book tasks**: `submit_answer` only
Note that `list_task_tools()` is an **instance method** (not a classmethod) because it needs access to `self.task_spec` to determine which tools to generate.
When `list_task_tools()` is used, the returned tools are combined with any `@tool(shared=False)` tools and all shared tools when a client calls `session.list_tools()`.
For a real-world example of this pattern, see [GeneralReasoning/toolathon-gym](https://openreward.ai/GeneralReasoning/toolathon-gym) — an environment that presents different tool configurations per task to test agents across varied interfaces.
## How Clients Access Task-Specific Tools
The SDK provides two different `list_tools()` methods depending on whether you have a session:
```python theme={null}
from openreward import OpenReward
client = OpenReward(api_key="your-api-key")
environment = client.environments.get(name="username/my-environment")
# Environment-level: returns shared tools only
shared_tools = environment.list_tools()
print(f"Shared tools: {[t.name for t in shared_tools]}")
# → Shared tools: ['submit_answer']
# Session-level: returns shared tools + task-specific tools
with environment.session(split="train", index=0) as session:
all_tools = session.list_tools()
print(f"All tools: {[t.name for t in all_tools]}")
# → All tools: ['submit_answer', 'select_option']
```
Under the hood:
* `environment.list_tools()` calls the `/tools` endpoint, which returns only shared tools
* `session.list_tools()` calls the `/task_tools` endpoint, which returns shared tools combined with task-specific tools (both `@tool(shared=False)` and those from `list_task_tools()`)
## Next Steps
Create reusable tool collections and compose them into environments
# Using Terminal Tools
Source: https://docs.openreward.ai/environments/using-terminal-tools
Grade the final assistant message instead of a submit tool call
## Goals
* Understand when to grade a plain assistant message rather than a tool call
* Mark a tool with `@terminal` so it is hidden from the model
* Use `session.is_assistant_message_final()` and `session.call_terminal_tool()` in a rollout loop
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* Completion of the [Your First Environment](/environments/your-first-environment) tutorial
## Introduction
Most OpenReward environments end an episode on a tool call: the model calls `submit_answer(answer="...")`, the environment grades it, and the rollout stops. The **environment** decides when the episode is over, because it is the environment that sees the final action.
But in many cases the last thing an agent should do is write a message, not call a tool — reporting to the user that the changes have been made to the code, answering a question, summarising what it found. Forcing a `submit` call there is an artificial extra step, and it means the environment never gets to see the thing you actually want to grade.
When the final action is an assistant message, the **termination event happens on the harness side, not the environment side**: the environment can't observe a message with no tool calls, so only the harness knows the episode has ended. **Terminal tools** exist to accommodate this. You mark one tool with `@terminal`, the harness recognises that a message with no tool calls ends the rollout, and it hands that message back to the environment for grading.
Concretely, marking a tool with `@terminal` means:
* The tool is **hidden** from `environment.list_tools()` and `session.list_tools()` — the model never sees it and cannot call it directly.
* `is_assistant_message_final()` returns `True`. A harness checks this once, up front, to learn that in this environment a message with no tool calls is the finished answer — rather than a model that forgot to call `submit` and needs nudging.
* The harness passes that message's text to `session.call_terminal_tool(text)`, which routes it into your grading tool.
The result: the model just writes its answer, and you still get a normal `ToolOutput` with a reward.
## Declaring a terminal tool
`@terminal` sits **on top of `@tool`** — it marks an existing tool as terminal rather than declaring one, so `@tool` is still required. (The two are order-independent, but reading `@terminal` first says what the tool *is* before how it's wired.) The tool accepts one argument: a Pydantic model with a single field, which receives the assistant's message. The field can be named whatever reads best.
```python theme={null}
from openreward.environments import Environment, terminal, tool, ToolOutput, TextBlock
from pydantic import BaseModel, Field
class FinalAnswer(BaseModel):
answer: str = Field(..., description="The assistant's final message")
class QAEnvironment(Environment):
@tool
async def search(self, params: SearchParams) -> ToolOutput:
"""Search the knowledge base"""
...
@terminal
@tool
async def grade(self, params: FinalAnswer) -> ToolOutput:
"""Grade the assistant's final message"""
correct = params.answer.strip().lower() == self.task_spec["answer"].lower()
return ToolOutput(
blocks=[TextBlock(text="Correct!" if correct else "Incorrect.")],
reward=1.0 if correct else 0.0,
finished=True,
)
```
Here `session.list_tools()` returns only `search`. An environment whose *only* tool is the terminal tool is valid too — that is the pure "answer the question" case, with no tools at all shown to the model.
### Terminal tools with no arguments
Sometimes the message itself doesn't matter: the model works in a sandbox, says "done", and grading runs against environment state. Give the terminal tool no arguments and the message is simply dropped.
```python theme={null}
@terminal
@tool
async def finish(self) -> ToolOutput:
"""Run the test suite against whatever the agent left behind"""
passed = await self.sandbox.run("pytest -q")
return ToolOutput(blocks=[TextBlock(text="graded")], reward=..., finished=True)
```
`call_terminal_tool(text)` still works — the text is ignored — and `call_terminal_tool()` is valid too.
### Rules
An environment fails at server startup if any rule is broken:
* **`@tool` is still required.** `@terminal` marks an existing tool as terminal; it does not declare one.
* **At most one terminal tool**, counting tools contributed by [declared toolsets](/environments/using-toolsets). Two `@terminal` tools is an error.
* **At most one argument.** The input model may declare one field (which receives the message) or none. Extra fields like `confidence` are an error — there is only one thing to pass in.
## Using it in a rollout
The client reports the environment's choice, so a harness can support both styles with the same loop:
```python theme={null}
from openreward import AsyncOpenReward
client = AsyncOpenReward()
environment = client.environments.get("YourOrg/qa-env")
tasks = await environment.list_tasks(split="train")
async with environment.session(tasks[0]) as session:
tools = await session.list_tools(format="anthropic") # terminal tool not included
assistant_ends_rollout = await session.is_assistant_message_final()
while True:
response = await call_your_model(messages, tools)
if response.tool_calls:
for call in response.tool_calls:
output = await session.call_tool(call.name, call.input)
messages.append(...)
continue
# No tool calls. In a terminal-tool environment this message is the answer.
if assistant_ends_rollout:
output = await session.call_terminal_tool(response.text)
print(output.reward)
break
```
`is_assistant_message_final()` returns `False` for environments without a terminal tool, in which case a message with no tool calls means whatever your harness already does with it. Environments served by an SDK version older than `0.1.139` also report `False`.
Both methods are available on the sync client (`Session`) and the async client (`AsyncSession`), and on the environment itself:
```python theme={null}
await environment.is_assistant_message_final()
await environment.terminal_tool() # TerminalToolSpec(name=..., arg=..., description=...) or None
```
Prefer the session methods inside a rollout — they reuse the session's cached tool list and account for any [session toolset](/harnesses/harness-toolsets), whose terminal tool takes precedence over the environment's.
## Calling it from environment code
Inside the environment, `call_terminal_tool()` does the same routing:
```python theme={null}
result = await env.call_terminal_tool("Paris")
```
The tool also remains callable by name (`session.call_tool("grade", {"answer": "Paris"})`) — hiding it from the tool list stops the model from discovering it, not your own code from using it.
# Using the CLI
Source: https://docs.openreward.ai/environments/using-the-cli
Manage environments, deployments, and runs from the command line
## Goals
* Install and configure the `orwd` CLI
* Create, deploy, and manage environments without leaving the terminal
* Use structured output formats for scripting and automation
## Prerequisites
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* Python 3.11+
* [GitHub CLI](https://cli.github.com/) (`gh`) — optional, needed for repo creation via `orwd new`
## Installation
Install the `openreward` package, which includes the `orwd` CLI:
```bash theme={null}
pip install openreward
```
Then set your API key:
```bash theme={null}
export OPENREWARD_API_KEY='your-api-key-here'
```
Most commands require `OPENREWARD_API_KEY`. The only exception is `orwd init`, which scaffolds files locally without touching the API.
### Shell alias (for development)
If you're developing the SDK locally and want `orwd` to always point at your working copy:
```bash theme={null}
# Add to ~/.zshrc or ~/.bashrc
alias orwd="uv run --project ~/path/to/python-sdk orwd"
```
Use `--project` (not `--directory`) so `orwd` runs relative to your current directory, not the SDK directory.
## Creating an Environment
### Interactive wizard
The fastest way to go from zero to a deployed environment:
```bash theme={null}
orwd new
```
This walks you through every step: naming, template selection, GitHub repo creation, and linking. At each prompt, you can accept the default (shown in parentheses) or type a different value.
```text theme={null}
OpenReward — new environment
> Environment name: my-bench
> Template
1. basic (default)
2. sandbox
Choice:
> Description: A benchmark for evaluating X
> Private (y/N):
> Harbor (sandbox) mode (y/N):
> Namespace (thomas):
> Directory (my-bench):
> Create a GitHub repo (Y/n):
> GitHub repo (owner/repo) (thomas/my-bench):
> Private repo (y/N):
```
### Non-interactive mode
Pass all arguments as flags and add `-y` to skip prompts:
```bash theme={null}
orwd new my-bench \
--description "A benchmark for evaluating X" \
--namespace GeneralReasoning \
--repo GeneralReasoning/my-bench \
--template basic \
-y
```
This is useful for scripting or CI workflows. Any argument you omit will still be prompted for interactively (unless `-y` is set, in which case it falls back to its default).
### Harbor environments
To create an environment that uses the [Harbor task specification](/environments/deploying-harbor-environments):
```bash theme={null}
orwd new my-harbor-env --harbor --description "Harbor-based benchmark"
```
### Step by step
If you prefer more control, you can run the individual steps yourself:
```bash theme={null}
# 1. Scaffold local files
orwd init my-env --template basic
# 2. Create the environment on OpenReward
orwd create my-env --description "My environment" --namespace my-org
# 3. Create a GitHub repo and push
cd my-env
git init && git add . && git commit -m "Initial scaffold"
gh repo create my-org/my-env --source . --push
# 4. Link to GitHub (triggers first deployment)
orwd link my-org/my-env my-org/my-env
```
## Managing Environments
### Listing environments
```bash theme={null}
# List all environments
orwd list
# Filter by owner
orwd list --owner GeneralReasoning
# Search by name or description
orwd list --search "benchmark"
# Your own environments
orwd list --mine
```
By default, `orwd list` fetches all results (auto-paginating through the API). Use `--limit` to cap the number returned.
### Getting environment details
```bash theme={null}
orwd get GeneralReasoning/CTF
```
```text theme={null}
Name: GeneralReasoning/CTF
ID: abc123
Description: Capture the flag challenges
Private: False
GitHub: connected
Repo: https://github.com/GeneralReasoning/CTF
Compute: 1 CPU, 4 GB
Created: 2025-03-15T12:00:00Z
Updated: 2025-04-01T08:30:00Z
```
### Updating an environment
```bash theme={null}
# Change description
orwd update my-org/my-env --description "Updated description"
# Rename
orwd update my-org/my-env --name new-name
# Toggle privacy
orwd update my-org/my-env --private
orwd update my-org/my-env --public
# Enable or disable harbor mode
orwd update my-org/my-env --harbor
orwd update my-org/my-env --no-harbor
# Set external URLs
orwd update my-org/my-env --arxiv-url "https://arxiv.org/abs/..."
```
## GitHub Integration
### Linking
```bash theme={null}
orwd link my-org/my-env my-org/my-repo
```
This connects an environment to a GitHub repository and triggers the first deployment. If you haven't authorized GitHub yet, the CLI opens your browser to complete OAuth.
Default compute settings (matching the web UI):
| Setting | Default |
| --------------- | -------------------- |
| `--cpu-memory` | `1:4` (1 vCPU, 4 GB) |
| `--concurrency` | `500` |
| `--max-scale` | `10` |
Override any of these:
```bash theme={null}
orwd link my-org/my-env my-org/my-repo --cpu-memory 2:8 --max-scale 5
```
### Unlinking
```bash theme={null}
orwd unlink my-org/my-env
```
### Updating link settings
Change compute or scaling settings on an already-linked environment:
```bash theme={null}
orwd update-link my-org/my-env --cpu-memory 4:16 --concurrency 1000 --max-scale 3
```
## Monitoring Deployments
### Listing deployments
```bash theme={null}
orwd deployments my-org/my-env
```
```text theme={null}
ID STATUS BRANCH COMMIT CREATED
a1b2c3d4 deployed main f8e9a0b1 2025-04-20T10:00:00Z
e5f6g7h8 failed main c3d4e5f6 2025-04-19T15:30:00Z
```
### Viewing logs
```bash theme={null}
# Runtime logs (default)
orwd logs my-org/my-env
# Build logs
orwd logs my-org/my-env --build
# Logs for a specific deployment
orwd logs my-org/my-env --deployment-id a1b2c3d4-full-id-here
# More log entries
orwd logs my-org/my-env --limit 200
```
### Harbor task builds
For [Harbor environments](/environments/deploying-harbor-environments), each task gets its own Docker image build. To check their status:
```bash theme={null}
orwd task-builds my-org/my-harbor-env
```
```text theme={null}
TASK STATUS ID UPDATED
task-1 success a1b2c3d4 2025-04-20T10:05:00Z
task-2 building e5f6g7h8 2025-04-20T10:04:00Z
task-3 error i9j0k1l2 2025-04-20T10:03:00Z
error: Dockerfile syntax error on line 12
```
To view the build logs for a specific task:
```bash theme={null}
orwd task-build-logs
```
Use `orwd task-builds my-org/my-env -o json` to get the full task build IDs for use with `task-build-logs`.
## File Management
Environments have a file store for uploading data files (datasets, ground truth, task inputs). These files are available at `/orwd_data/` on the deployed environment server. See [Where Environment Data Lives](/environments/where-environment-data-lives) for details on how uploaded files are used.
### Uploading files
```bash theme={null}
# Upload a single file
orwd upload my-org/my-env data.json
# Upload multiple files
orwd upload my-org/my-env train.csv test.csv labels.json
# Upload an entire directory (preserves structure)
orwd upload my-org/my-env ./data/
```
By default, files are uploaded using their relative path from the current working directory. Use `--dest` to set a custom destination path:
```bash theme={null}
# Upload to a specific path
orwd upload my-org/my-env train.csv --dest datasets/train.csv
# Upload files into a directory (note the trailing slash)
orwd upload my-org/my-env a.txt b.txt --dest inputs/
# Upload a local directory to a different remote path
orwd upload my-org/my-env ./local-data/ --dest ground_truth/
```
Upload concurrency defaults to 10 parallel transfers. Adjust with `--concurrency`:
```bash theme={null}
orwd upload my-org/my-env ./large-dataset/ --concurrency 20
```
### Listing files
```bash theme={null}
# List all files
orwd files my-org/my-env
# Filter by path prefix
orwd files my-org/my-env --prefix ground_truth/
```
### Deleting files
```bash theme={null}
# Delete a single file
orwd delete-file my-org/my-env data.json
# Delete a folder and all its contents
orwd delete-file my-org/my-env ground_truth/ --folder
```
## Runs and Rollouts
Browse your evaluation runs and their rollouts from the CLI:
```bash theme={null}
# List runs
orwd runs
orwd runs --search "experiment-1"
# Get details for a specific run
orwd run
# List rollouts within a run
orwd rollouts
```
## Structured Output
Every command (except `init` and `new`) supports the `-o` / `--output` flag for machine-readable output. This is useful for piping into `jq`, scripting, or loading into other tools.
| Format | Flag | Description |
| ------ | ---------- | ------------------------------------------------- |
| Table | `-o table` | Human-readable columns (default) |
| JSON | `-o json` | Compact JSON array or object |
| YAML | `-o yaml` | YAML output |
| JSONL | `-o jsonl` | One JSON object per line (for row-based commands) |
### Examples
```bash theme={null}
# Get environment details as JSON
orwd get my-org/my-env -o json
# Pipe environment list into jq
orwd list --owner my-org -o json | jq '.[].name'
# Stream deployments as JSONL for processing
orwd deployments my-org/my-env -o jsonl
# Export all environments to YAML
orwd list --mine -o yaml > my-environments.yaml
```
## Command Reference
| Command | Description |
| ---------------------- | ------------------------------------------------------------------------------- |
| `orwd whoami` | Show the current authenticated user |
| `orwd new` | Interactive wizard (or one-shot with flags) to create and deploy an environment |
| `orwd init` | Scaffold local environment files from a template |
| `orwd create` | Create an environment on OpenReward |
| `orwd list` | List environments |
| `orwd get` | Get environment details |
| `orwd update` | Update environment metadata |
| `orwd link` | Link an environment to a GitHub repository |
| `orwd unlink` | Disconnect from GitHub |
| `orwd update-link` | Update compute/scaling settings for a linked environment |
| `orwd deployments` | List deployments |
| `orwd logs` | View build or runtime logs |
| `orwd task-builds` | List Harbor task image builds |
| `orwd task-build-logs` | View logs for a Harbor task image build |
| `orwd upload` | Upload files or directories to an environment's file store |
| `orwd files` | List files in an environment's file store |
| `orwd delete-file` | Delete a file or folder from an environment's file store |
| `orwd runs` | List runs |
| `orwd run` | Get run details |
| `orwd rollouts` | List rollouts in a run |
Run `orwd --help` for full usage details on any command.
# Using Toolsets
Source: https://docs.openreward.ai/environments/using-toolsets
Create reusable tool collections and compose them into environments
## Goals
* Understand what toolsets are and why they're useful
* Use pre-built toolsets for common tasks (PDF, Excel, Word, PowerPoint)
* Create custom toolsets for your specific needs
* Compose multiple toolsets in a single environment
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* Completion of the [Your First Environment](/environments/your-first-environment) tutorial
* Completion of the [Building Agentic Environments](/environments/building-agentic-environments) tutorial
## Installation
Pre-built toolsets run in sandbox environments, which means your Docker image must include the required dependencies. You need the `openreward` package with the `tools` extra dependency:
```bash theme={null}
pip install openreward[tools]
```
This installs the required libraries for all pre-built toolsets:
* **pdfplumber**, **pypdf**, **reportlab**, **pdf2image** - for PDFToolset
* **python-docx** - for WordToolset
* **openpyxl** - for ExcelToolset
* **python-pptx** - for PowerPointToolset
If you only need specific toolsets, you can install individual dependencies instead:
```bash theme={null}
# For PDFToolset only
pip install pdfplumber pypdf reportlab pdf2image
# For WordToolset only
pip install python-docx
# For ExcelToolset only
pip install openpyxl
# For PowerPointToolset only
pip install python-pptx
```
**Note**: Make sure your sandbox Docker image includes these dependencies, or use the recommended `generalreasoning/knowledge-worker` image which has them pre-installed.
## Introduction
When building multiple environments, you'll often find yourself copying the same tool definitions across different environment classes. For example, if you have three environments that all need to read PDF files, you'd typically copy the same PDF reading tools into each environment.
Toolsets solve this problem by letting you define reusable collections of tools that can be declared with a simple `toolsets = [PDFToolset, ExcelToolset]` statement. Instead of inheritance-based code reuse, toolsets provide a composition-based approach that follows the DRY (Don't Repeat Yourself) principle.
The key benefits:
* **Reusability**: Define tools once, use them in multiple environments
* **Composition**: Mix and match toolsets as needed
* **Maintainability**: Update tools in one place
* **Clean separation**: Keep environment-specific logic separate from general-purpose tools
## Using Pre-Built Toolsets
OpenReward provides production-ready toolsets for common tasks. We'll start with these before showing you how to create custom toolsets.
### Available Toolsets
The openreward library includes four pre-built toolsets:
* **PDFToolset** - PDF creation, reading, searching, merging (11 tools)
* **ExcelToolset** - Spreadsheet manipulation, charts, data reading (11 tools)
* **WordToolset** - Document creation, formatting, content manipulation (12 tools)
* **PowerPointToolset** - Presentation creation and editing (11 tools)
All of these toolsets require sandbox access, as they manipulate files within the sandbox environment.
For web access there are two toolsets, both exposing `web_search` / `web_fetch` and neither needing a sandbox, so they compose with any of the toolsets above:
* [Web Tools](/environments/web-tools) — a `WebToolset` whose search provider is swappable via `OPENREWARD_SEARCH_BACKEND` (backdated corpus by default, Tavily and others available).
* [Backdated Web Tools](/environments/backdated-web-tools) — a `BackSearchToolset` pinned to the backdated corpus, for when the cutoff guarantee must not be configurable.
Declare one or the other, not both — they use the same tool names.
**Important**: All pre-built toolsets require sandbox access and the `openreward[tools]` dependencies. When creating sandboxes for toolset usage, we recommend using the `generalreasoning/knowledge-worker` image which includes all required dependencies pre-installed. Alternatively, ensure your custom image includes the necessary libraries listed in the Installation section above.
### Example: Using PDFToolset
Let's create an environment that uses PDFToolset to analyze PDF documents. Here's a complete working example:
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings, SandboxBucketConfig
from openreward.environments import Environment, Split, tool, ToolOutput, TextBlock
from openreward.toolsets import PDFToolset
from pydantic import BaseModel
class AnswerParams(BaseModel):
answer: str
class PDFAnalysisEnv(Environment):
toolsets = [PDFToolset] # Declare PDFToolset at class level
def __init__(self, task_spec, secrets):
super().__init__(task_spec, secrets)
# Setup sandbox - PDFToolset will automatically access this
self.sandbox_settings = SandboxSettings(
environment="YourUsername/PDFAnalysisEnv",
image="generalreasoning/knowledge-worker:latest",
machine_size="0.5:1",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/sandbox/",
read_only=True,
)
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(self.sandbox_settings)
async def setup(self):
await self.sandbox.start()
async def teardown(self):
await self.sandbox.stop()
def get_prompt(self):
return [TextBlock(text="Analyze the PDF document and extract key information.")]
@tool
async def submit_answer(self, params: AnswerParams) -> ToolOutput:
"""Submit your analysis"""
return ToolOutput(
blocks=[TextBlock(text=f"Analysis submitted: {params.answer}")],
reward=1.0,
finished=True,
)
@classmethod
def list_tasks(cls, split: str):
return [{"id": "1", "doc": "report.pdf"}] if split == "train" else []
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
```
With this setup, your environment automatically gains access to all 11 PDF tools. Here are the key tools available:
| Tool Name | Description |
| ---------------------------- | -------------------------------------------------------------------- |
| `pdfs_read_pdf_pages` | Extract text content from PDF pages with optional layout information |
| `pdfs_search_pdf` | Search for text within PDF pages with context |
| `pdfs_get_metadata` | Get PDF document metadata and properties |
| `pdfs_get_document_overview` | Quick overview of PDF structure (page count, text preview, metadata) |
| `pdfs_merge_pdfs` | Combine multiple PDF files into one |
| `pdfs_extract_pages` | Extract specific pages from a PDF to a new file |
| `pdfs_create_pdf` | Create a new PDF file with optional metadata |
| `pdfs_add_content` | Add text content to an existing PDF page or create a new page |
| `pdfs_read_image` | Extract embedded images metadata from PDF pages |
| `pdfs_read_page_as_image` | Convert PDF page to image (PNG/JPEG) |
| `pdfs_delete_pdf` | Delete a PDF file from the sandbox |
An agent interacting with this environment might call `pdfs_get_document_overview` first to understand the document structure, then `pdfs_read_pdf_pages` to extract specific content, and finally `pdfs_search_pdf` to find key terms before submitting an answer.
### Example: Using ExcelToolset
Similarly, you can use ExcelToolset for spreadsheet manipulation:
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings, SandboxBucketConfig
from openreward.environments import Environment, TextBlock
from openreward.toolsets import ExcelToolset
class SpreadsheetEnv(Environment):
toolsets = [ExcelToolset]
def __init__(self, task_spec, secrets):
super().__init__(task_spec, secrets)
# Setup sandbox for ExcelToolset
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(SandboxSettings(
environment="YourUsername/SpreadsheetEnv",
image="generalreasoning/knowledge-worker:latest",
machine_size="0.5:1",
))
async def setup(self):
await self.sandbox.start()
async def teardown(self):
await self.sandbox.stop()
def get_prompt(self):
return [TextBlock(text="Analyze the spreadsheet and calculate the total.")]
```
ExcelToolset provides these key tools:
| Tool Name | Description |
| -------------------------------- | ----------------------------------------------------------------------- |
| `excel_create_spreadsheet` | Create a new Excel workbook with initial worksheet |
| `excel_list_tabs_in_spreadsheet` | List all worksheet/tab names in the workbook |
| `excel_add_tab` | Add a new worksheet to existing workbook |
| `excel_read_tab` | Read data from a specific worksheet |
| `excel_read_csv` | Read CSV file and convert to Excel format data structure |
| `excel_edit_spreadsheet` | Modify existing cell value in a worksheet |
| `excel_add_content_text` | Write data to a range of cells (batch operation) |
| `excel_delete_content_cell` | Clear cell content (sets to None) |
| `excel_create_chart` | Create a chart in the worksheet (bar, column, line, pie, scatter, area) |
| `excel_delete_tab` | Remove a worksheet from workbook |
| `excel_delete_spreadsheet` | Delete an Excel spreadsheet file from sandbox |
### Example: Composing Multiple Toolsets
You can use multiple toolsets together in a single environment. All tools from all toolsets become available:
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings
from openreward.environments import Environment
from openreward.toolsets import PDFToolset, ExcelToolset, WordToolset
class DocumentProcessorEnv(Environment):
toolsets = [PDFToolset, ExcelToolset, WordToolset]
def __init__(self, task_spec, secrets):
super().__init__(task_spec, secrets)
# Setup sandbox (shared by all toolsets)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(SandboxSettings(...))
# ... rest of environment implementation ...
```
Key points about composition:
* All tools from all toolsets are discovered automatically
* Tools can be called without knowing which toolset they came from
* The framework detects and prevents tool name collisions
### WordToolset and PowerPointToolset
**WordToolset** provides 12 tools for Word document manipulation:
* `word_create_document` - Create new document
* `word_get_document_overview` - Get structure overview
* `word_read_document_content` - Extract text
* `word_add_content_text` - Add paragraphs/headings
* `word_edit_content_text` - Modify paragraphs
* `word_delete_content_text` - Remove paragraphs
* `word_add_image` - Insert images
* `word_apply_formatting` - Bold, italic, fonts, colors
* And more...
**PowerPointToolset** provides tools for presentation manipulation, following similar patterns to the other document toolsets.
All pre-built toolsets use the same pattern: import from `openreward.toolsets`, declare in your environment's `toolsets` list, and ensure your environment has a `self.sandbox` attribute.
## Creating Custom Toolsets
While pre-built toolsets cover common scenarios, you can create custom toolsets for your specific needs.
### Simple Toolset (No Dependencies)
For tools that don't need sandbox access or other dependencies, you can create a simple toolset without extending any base class:
```python theme={null}
from pydantic import BaseModel, Field
from openreward.environments import tool, ToolOutput, TextBlock
class MathParams(BaseModel):
a: float = Field(..., description="First number")
b: float = Field(..., description="Second number")
class MathToolset:
"""Simple toolset with no dependencies"""
@tool
async def add(self, params: MathParams) -> ToolOutput:
"""Add two numbers together"""
result = params.a + params.b
return ToolOutput(
blocks=[TextBlock(text=f"{params.a} + {params.b} = {result}")],
reward=0.0,
finished=False,
)
@tool
async def subtract(self, params: MathParams) -> ToolOutput:
"""Subtract second number from first"""
result = params.a - params.b
return ToolOutput(
blocks=[TextBlock(text=f"{params.a} - {params.b} = {result}")],
reward=0.0,
finished=False,
)
```
Using it in an environment:
```python theme={null}
from openreward.environments import Environment
class CalculatorEnv(Environment):
toolsets = [MathToolset]
# ... implement required methods ...
```
### Sandbox-Based Toolset
For tools that need sandbox access, extend the `Toolset` base class:
```python theme={null}
from openreward.environments import Toolset, tool, ToolOutput, TextBlock
from pydantic import BaseModel, Field
class BashParams(BaseModel):
command: str = Field(..., description="Bash command to execute")
class FileToolset(Toolset):
"""Toolset that requires sandbox access"""
@tool
async def list_files(self, params: BashParams) -> ToolOutput:
"""List files in a directory using bash"""
# self.sandbox is automatically available from Toolset base class
output, exit_code = await self.sandbox.run(params.command)
return ToolOutput(
blocks=[TextBlock(text=f"Output:\n{output}\nExit code: {exit_code}")],
reward=0.0,
finished=False,
)
@tool
async def read_file(self, params: BashParams) -> ToolOutput:
"""Read file content"""
output, exit_code = await self.sandbox.run(f"cat {params.command}")
return ToolOutput(
blocks=[TextBlock(text=output if exit_code == 0 else f"Error: {output}")],
reward=0.0,
finished=False,
)
```
The `Toolset` base class:
* Receives the environment instance in its constructor
* Extracts `self.sandbox` automatically (via `sandbox_attr` parameter)
* Makes the sandbox available to all tool methods
* Supports custom sandbox attribute names if needed
## Key Concepts
### Collision Detection
The framework prevents duplicate tool names. If your environment defines a tool with the same name as a toolset tool, an error is raised. This ensures clarity about which tool is being called and prevents subtle bugs.
Example of a collision:
```python theme={null}
class MyEnv(Environment):
toolsets = [PDFToolset]
@tool
async def pdfs_read_pdf_pages(self, params): # Error! Name collision
...
```
This would raise: `ValueError: Tool name collision: 'pdfs_read_pdf_pages' is defined in both the environment and toolset 'PDFToolset'. Please rename one of them to avoid conflicts.`
### Dependency Injection
When a toolset extends the `Toolset` base class, it automatically receives the environment instance and extracts dependencies like `self.sandbox`. You don't need to manually pass these dependencies - the framework handles this through lazy instantiation.
## Best Practices
**When to use pre-built toolsets:**
* You're working with common file formats (PDF, Excel, Word, PowerPoint)
* You need standard operations (read, write, search, merge)
* You want battle-tested, production-ready tools
**When to create custom toolsets:**
* You have domain-specific tools used across multiple environments
* You're building tools for file formats not covered by pre-built toolsets
* You need specialized operations not available in pre-built toolsets
**Naming conventions:**
* Prefix toolset tools to avoid collisions: `pdf_`, `excel_`, `math_`
* Use descriptive toolset class names: `PDFToolset`, not `DocumentToolset`
* Match parameter model names to tools: `MathParams` for both `add` and `subtract` when they share parameters
**When to use toolsets vs environment tools:**
* **Toolsets**: Reusable tools used across 3+ environments, logical groups with shared dependencies
* **Environment tools (`@tool`)**: Environment-specific logic, environment-specific state, one-off operations
* For tools specific to individual tasks, see [Using Task-Specific Tools](/environments/using-task-specific-tools)
**Organizing toolset files:**
* Keep toolsets in a separate `toolsets/` directory
* One file per toolset: `toolsets/math.py`, `toolsets/file.py`
* Import and declare: `from toolsets.math import MathToolset`
## Troubleshooting
**"AttributeError: 'MyEnv' object has no attribute 'sandbox'"**
Solution: Ensure you initialize `self.sandbox` in your environment's `__init__` method before any tools are called. Toolsets extending `Toolset` look for this attribute.
```python theme={null}
def __init__(self, task_spec, secrets):
super().__init__(task_spec, secrets)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(SandboxSettings(...)) # Must be set
```
**"Tool name collision detected"**
Solution: Rename your environment tool or toolset tool to avoid conflicts. Use prefixes to make names unique.
```python theme={null}
# Bad - collision
@tool
async def read(self, params): # Conflicts with toolset
# Good - unique name
@tool
async def read_task_config(self, params): # No collision
```
**"ImportError: cannot import name 'PDFToolset'"**
Solution: Update your openreward library to the latest version:
```bash theme={null}
pip install --upgrade openreward
```
Pre-built toolsets were added in a recent version. Check that you're using the latest version.
**Toolset tools not appearing in list\_tools()**
Solution: Ensure toolsets are declared at the class level, not instance level:
```python theme={null}
# Correct - class level
class MyEnv(Environment):
toolsets = [MathToolset]
# Incorrect - instance level
class MyEnv(Environment):
def __init__(self, task_spec, secrets):
super().__init__(task_spec, secrets)
self.toolsets = [MathToolset] # Won't work!
```
## Next Steps
Now that you understand toolsets, you can build more sophisticated environments:
Learn advanced sandbox patterns that work great with toolsets
Combine toolsets with intelligent grading for complex evaluations
Use toolsets in evaluation environments to test model capabilities
# Ways to Access Tasks
Source: https://docs.openreward.ai/environments/ways-to-access-tasks
List, count, and fetch tasks from an environment
## Goals
* Learn the different ways to access tasks from an environment
* Understand when to use listing vs index-based access
* Learn how to implement task methods in your environment server
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* Completion of the [Your First Environment](/environments/your-first-environment) tutorial
## Introduction
Tasks are JSON objects that describe a specific problem or challenge for an agent to solve. Each environment organises tasks into **splits** (e.g. `"train"`, `"test"`).
The SDK provides several ways to access tasks depending on your needs — from listing all tasks in a split, to fetching individual tasks by index, to retrieving a range of tasks for partitioned workloads.
## Listing Tasks
You can list all tasks for a split:
```python theme={null}
tasks = await environment.list_tasks(split="train")
```
This returns the full list of task objects for the given split. It's the simplest approach and works well when your task set is small enough to load in one call.
## Index-Based Access
For environments with large task sets, you can query the number of tasks and fetch individual tasks by index without loading the entire list:
```python theme={null}
# Get the number of tasks in a split
num = await environment.num_tasks(split="train")
print(f"{num} tasks available")
# Fetch a single task by index
task = await environment.get_task(split="train", index=0)
```
## Getting a Range of Tasks
You can also fetch a contiguous range of tasks by index. This is useful for partitioning tasks across multiple workers:
```python theme={null}
# Get tasks 0 through 9
tasks = await environment.get_task_range(split="train", start=0, stop=10)
```
Task ordering is consistent across calls, so ranges are safe to use for partitioning workloads.
## Implementing Tasks in Your Environment
When building an environment server, implement `list_tasks`, `list_splits`, and optionally override `get_task` and `num_tasks`:
```python theme={null}
from openreward.environments import Environment, Split, JSONObject
class MyEnvironment(Environment):
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
if split == "train":
return [{"task_id": str(i)} for i in range(1000)]
return [{"task_id": "0"}]
```
The base class provides default implementations of `num_tasks` and `get_task` that call `list_tasks` internally. If your task set is large or expensive to load, you can override these for better performance:
```python theme={null}
@classmethod
async def num_tasks(cls, split: str) -> int:
# Return count without loading all tasks
return db.count_tasks(split)
@classmethod
async def get_task(cls, split: str, index: int) -> JSONObject:
# Fetch a single task without loading the full list
return db.get_task(split, index)
```
## Next Steps
Define tools that vary per task for environments with multiple tool interfaces
# Web Tools
Source: https://docs.openreward.ai/environments/web-tools
One web search and fetch surface for your environment, with a swappable backend
## Introduction
`WebToolset` gives an agent `web_search` and `web_fetch` tools **without tying
your environment to a particular search provider**. Which provider actually
answers is process configuration, not code:
```bash theme={null}
# default — GR's backdated corpus
(nothing to set)
# or route the same tools to Tavily
export OPENREWARD_SEARCH_BACKEND=tavily
export TAVILY_API_KEY="tvly-xxxxxxxx"
```
The environment stays identical either way. Input validation, the citation
envelope the model sees, the fetch cache and the error contract all live in the
SDK and are shared by every backend — so a task written against one backend
behaves the same against another. Only the sources differ.
Requires `openreward >= 0.1.148`. The Tavily backend also needs
`pip install 'openreward[search]'`; the default `backsearch` backend needs
nothing beyond the base install.
### Which toolset should I use?
| | `WebToolset` | [`BackSearchToolset`](/environments/backdated-web-tools) |
| ----------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------- |
| Tools | `web_search`, `web_fetch` | `web_search`, `web_fetch` |
| Backend | Swappable via `OPENREWARD_SEARCH_BACKEND` | Always the backdated corpus |
| Point-in-time guarantee | Only on backdated backends | Always |
| Use when | You want to change search providers without editing environments | The `as_of` cutoff is load-bearing and must not be switchable |
`BackSearchToolset` deliberately ignores `OPENREWARD_SEARCH_BACKEND`. A toolset
whose whole value is the leakage-free guarantee should not be silently
redirected to the live web by an environment variable.
## Configuration
| Variable | Required | Purpose |
| --------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------- |
| `OPENREWARD_SEARCH_BACKEND` | no | `backsearch` (default) or `tavily`. An unrecognised value logs a warning and falls back to `backsearch` |
| `OPENREWARD_API_KEY` | for `backsearch` | Your OpenReward API key |
| `TAVILY_API_KEY` | for `tavily` | Your Tavily API key |
| `TAVILY_SEARCH_DEPTH` | no | `basic` (default) or `advanced` |
| `TAVILY_MAX_RESULTS` | no | Results per search (default `8`, capped at Tavily's limit of 20) |
| `TAVILY_EXTRACT_DEPTH` | no | `basic` (default) or `advanced` |
| `TAVILY_API_BASE_URL` | no | Override the Tavily host (egress proxy, self-hosted, or a test stub) |
| `OPENREWARD_WEB_AS_OF` | no | Default cutoff date (ISO `YYYY-MM-DD`) — honoured by backdated backends only |
A backend that is missing its key is **not** a startup failure — the session
still opens, and the problem surfaces on the first tool call. What happens then
depends on whether the agent could plausibly recover; see below.
## Fatal vs recoverable errors
`WebToolset` splits tool failures in two, and the distinction matters for
training data:
* **Recoverable** — a blocked domain, an unparseable URL, an empty result set.
These come back to the model as ordinary tool output with the code in
`metadata["error"]`, because the agent can act on them by searching
differently or picking another source. An empty result set is a real answer
to the query, not a failure.
* **Fatal** — `not-configured` (no API key) or `quota-exceeded`. These raise
`SearchBackendUnavailable`.
The reason for raising: handed back as text, the agent re-issues a dead call
until it hits the turn cap, never reaches the terminal tool, and the rollout
scores **0.0 — indistinguishable from a model that answered wrongly**. Raising
lets the platform retry the tool call and, if it still fails, end the rollout
with a blank reward, which is excluded from training rather than poisoning it.
Pass `raise_on_fatal=False` to get the everything-is-soft behaviour.
`BackSearchToolset` is unchanged and always soft-errors.
## Backends
| Backend | Sources | `as_of` cutoff | `allowed_domains` / `blocked_domains` | Domain policy preflight |
| ---------------------- | ------------------------------------------------ | -------------------------------------------- | ------------------------------------------------------------- | ----------------------- |
| `backsearch` (default) | GR's backdated corpus — news, SEC filings, arXiv | **Honoured** — results bounded to the cutoff | Yes | Yes |
| `tavily` | The live web | **Ignored** — see the warning below | Best-effort (mapped to `include_domains` / `exclude_domains`) | No |
**Tavily searches the live web, so `as_of` is ignored.** Tavily does offer
publish-date filters, but they filter a live index by *claimed* publish date
rather than serving a frozen crawl, so they cannot give the leakage-free
guarantee an evaluation or prediction task needs. Rather than imply a guarantee
we cannot keep, the cutoff is dropped and a warning is logged the first time one
is requested. If post-cutoff leakage would invalidate your task, stay on
`backsearch`.
**`allowed_domains` is not a hard guarantee on Tavily.** It maps to Tavily's
`include_domains`, which filters correctly for indexed domains — but when the
requested domain is absent from Tavily's index, Tavily silently returns
*unfiltered* results rather than an empty set. Asking for `reuters.com`, for
example, comes back with results from other sites entirely. If your task depends
on sources being restricted to a domain, verify the hostnames yourself, or use
`backsearch`, where the filter is applied by the index.
The tool descriptions the model sees follow the configured backend: on a
live-web backend the "results are limited to documents published on or before
the cutoff date" wording is removed, so the model is never told about a bound
that will not hold.
## Using the toolset in an environment
Declare `WebToolset` at class level. Like `BackSearchToolset` it is **not**
sandbox-backed — it talks to a search provider over HTTP — so your environment
does not need a `self.sandbox`.
```python theme={null}
from openreward.environments import Environment, Split, TextBlock
from openreward.toolsets import WebToolset
class ResearchEnv(Environment):
toolsets = [WebToolset]
@classmethod
def list_splits(cls):
return [Split(name="test", type="test")]
@classmethod
def list_tasks(cls, split):
return [{"id": "1", "question": "What happened with X in late 2025?"}]
def get_prompt(self):
return [TextBlock(text="Research the question using web_search and web_fetch, then answer with sources.")]
```
The agent then sees two tools:
| Tool | Input | Output |
| ------------ | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `web_search` | `query` (≥ 2 chars), optional `allowed_domains` **or** `blocked_domains` (never both) | A `Links:` list of `{title, url}` sources plus a reminder to cite them |
| `web_fetch` | `url`, `prompt` | Extracted page text (truncated to 100 KB) |
### Selecting it by name in a session
`WebToolset` is a registered built-in, so a client can attach it to a session by
name:
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
env = or_client.environments.get(name="MyOrg/ResearchEnv")
tasks = env.list_tasks(split="test")
with env.session(task=tasks[0], toolset="web") as session:
result = session.call_tool("web_search", {"query": "US interest rate decision December 2025"})
print(result.blocks[0].text)
```
### Passing credentials as session secrets
Environments conventionally receive API keys through the session `secrets`
mapping rather than the server's process environment. Expose them to the toolset
with a `search_secrets` hook and the configured backend picks out the key it
needs — `tavily_api_key` for Tavily, `api_key` for backsearch:
```python theme={null}
class ResearchEnv(Environment):
toolsets = [WebToolset]
def __init__(self, task_spec, secrets={}):
super().__init__(task_spec, secrets)
self.search_secrets = secrets
```
```python theme={null}
with env.session(
task=task,
secrets={"tavily_api_key": "tvly-xxxxxxxx"},
) as session:
...
```
Without the hook the backend falls back to the process environment. See
[Keeping Secrets Secret](/environments/keeping-secrets-secret) for how secret
values reach a hosted environment.
### Choosing the backend per environment
`OPENREWARD_SEARCH_BACKEND` is the usual control, but an environment can pin or
override it with a `search_backend` hook — read live on every tool call, like
`web_as_of`:
```python theme={null}
class ResearchEnv(Environment):
toolsets = [WebToolset]
search_backend = "tavily" # attribute, @property, or zero-arg callable
```
Resolution order for each call: an explicit `backend=` passed to the toolset
constructor, then `env.search_backend`, then `OPENREWARD_SEARCH_BACKEND`, then
`backsearch`.
### Composing with other toolsets
Because it needs no sandbox, `WebToolset` composes cleanly with sandbox-backed
toolsets:
```python theme={null}
from openreward.toolsets import WebToolset, CLIToolset
class ResearchAndCodeEnv(Environment):
toolsets = [WebToolset, CLIToolset]
```
Do **not** declare `WebToolset` and `BackSearchToolset` together — they expose
the same tool names and the framework rejects duplicates.
## Standalone usage
The same search and fetch are available without an environment:
```python theme={null}
import asyncio
from openreward.tools import Search, Fetch
async def main():
s = Search() # OPENREWARD_SEARCH_BACKEND picks the provider
result = await s.run("US interest rate decision December 2025")
if result.ok:
print(result.output) # citation-ready text
print(result.data["mode"]) # which backend answered
for hit in result.data["hits"]:
print(hit["title"], "->", hit["url"])
f = Fetch(backend="tavily") # or pin one explicitly
page = await f.run("https://example.com/article", "Summarise the key points")
if page.ok:
print(page.data["content"])
asyncio.run(main())
```
Both `run(...)` calls return a `WebToolResult`:
| Field | Meaning |
| -------------------------------- | ----------------------------------------------------------------------------------- |
| `.ok` | `True` on success, `False` on a recoverable error |
| `.output` | The string to show a model (the result, or an error message) |
| `.data` | Structured payload — search: `{query, hits, mode}`; fetch: `{url, prompt, content}` |
| `.error_code` / `.error_message` | Set when `.ok` is `False` |
Error codes are the same on every backend, so calling code can branch on them
without knowing which provider is configured:
| Code | Meaning |
| -------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `not-configured` | No key for the selected backend, or `tavily-python` is not installed |
| `missing-query` | Query shorter than 2 characters |
| `domain-lists-conflict` | Both `allowed_domains` and `blocked_domains` were passed |
| `invalid-url` / `hostname-too-shallow` / `userinfo-not-allowed` / `url-too-long` | Rejected fetch URL |
| `search-failed` / `fetch-error` | The provider failed or returned nothing usable |
| `quota-exceeded` | The provider reported a usage limit |
## Adding your own backend
Backends live in a registry, so a new provider does not require changes to the
engine or to any environment:
```python theme={null}
from openreward.search_backends import SearchBackend, register_backend
from openreward.tools.web import WebToolResult, format_search_output, validate_search_input
class ExaBackend(SearchBackend):
name = "exa"
supports_as_of = False
def __init__(self, *, config=None, secrets=None):
...
async def search(self, *, query, k=None, as_of=None,
allowed_domains=None, blocked_domains=None) -> WebToolResult:
query, allowed, blocked, error = validate_search_input(query, allowed_domains, blocked_domains)
if error is not None:
return error
hits = [...] # {title, url, snippet, score}
return WebToolResult.success(
format_search_output(query, hits), # shared renderer — do not hand-roll
{"query": query, "hits": hits, "mode": "exa"},
)
async def fetch(self, *, url, prompt, as_of=None) -> WebToolResult:
...
register_backend("exa", ExaBackend)
```
Two rules make backends interchangeable:
* **Render through the shared helpers** (`format_search_output`,
`render_fetch_text`) and **validate through the shared validators**
(`validate_search_input`, `validate_fetch_url`). Models are trained on the
exact output shape, and identical inputs must produce identical error codes.
* **Never raise for a recoverable problem.** Return
`WebToolResult.error(code, message)` so the agent can read it and try
something else.
## Notes
* **Citations.** Search output is formatted so the model is reminded to cite the
returned URLs as markdown links in its final answer.
* **Caching.** Fetches are cached in-process for 15 minutes. Cache keys are
namespaced per backend (and, for backdated backends, per cutoff), so a live
page is never served out of a backdated fetch's entry or vice versa.
* **Soft errors.** Bad inputs come back as a normal result with `.ok == False`
and a clear message rather than as an exception.
## Next Steps
The point-in-time corpus, and the toolset pinned to it
Compose document toolsets and build custom ones
Call the search and fetch endpoints directly over HTTP
How API keys and secrets reach your environment
# Where Environment Data Lives
Source: https://docs.openreward.ai/environments/where-environment-data-lives
Understanding the difference between server-side data and sandbox-mounted data
## Overview
You can upload files to any environment via the **Files** tab in the web UI, or using the [`orwd` CLI](/environments/using-the-cli#uploading-files):
```bash theme={null}
# Upload from the command line
orwd upload my-org/my-env ./data/
```
These files are stored at `/orwd_data/` on the environment server. Your environment server code always has full access to everything in `/orwd_data/`.
Sandboxes, on the other hand, **only see what you explicitly mount**. This separation is critical: it lets your server hold ground truth data (answers, test cases, rubrics) that the agent in the sandbox never sees.
## Two Purposes for Your Files
Your uploaded files serve two distinct roles:
| | Environment Server | Sandbox (Agent) |
| ------------ | ------------------------------------------------ | ---------------------------------------------- |
| **Access** | Full access to all of `/orwd_data/` | Only sees what you mount via `bucket_config` |
| **Purpose** | Ground truth, answers, test data, rubrics | Input data the agent needs to work with |
| **Examples** | `answers.json`, `test_cases.csv`, `rubrics.json` | `transactions.csv`, `input.txt`, `dataset.csv` |
## Server-Side Data
The environment server always has access to everything in `/orwd_data/`. This is where you store data that only your server should see — ground truth answers, expected outputs, rubrics, and evaluation data.
```python theme={null}
import json
from pathlib import Path
# The server reads ground truth directly from /orwd_data/
with open("/orwd_data/answers.json") as f:
answers = json.load(f)
# Use ground truth in your reward function
def compute_reward(self, agent_answer: str) -> float:
expected = answers[self.task_id]
return 1.0 if agent_answer == expected else 0.0
```
The agent never has access to these files — they exist only on the server.
### Local vs Production Paths
`/orwd_data/` only exists when your environment is deployed to OpenReward. When developing locally, your files live alongside your code. Use a simple check to handle both cases:
```python theme={null}
import os
from pathlib import Path
if os.path.exists("/orwd_data"):
DATA_PATH = Path("/orwd_data")
else:
DATA_PATH = Path(__file__).parent # local development — files next to your code
# Now use DATA_PATH everywhere
with open(DATA_PATH / "answers.json") as f:
answers = json.load(f)
```
This way you can test locally with the same data files in your project directory, and everything switches to `/orwd_data/` automatically when deployed.
## Sandbox Data
To give the agent access to files, you mount a directory from `/orwd_data/` into the sandbox using `SandboxBucketConfig`. The key parameter is `only_dir`, which restricts the mount to a specific subdirectory.
```python theme={null}
from openreward import SandboxSettings, SandboxBucketConfig
self.sandbox_settings = SandboxSettings(
environment="YourUsername/MyEnv",
image="python:3.11-slim",
machine_size="0.5:1",
bucket_config=SandboxBucketConfig(
mount_path="/workspace", # Where files appear in the sandbox
read_only=True, # Agent can't modify the files
only_dir="agent", # Mount ONLY the agent/ subdirectory
)
)
```
With this configuration, the sandbox at `/workspace` will contain only the files from `/orwd_data/agent/` — nothing else.
## Organising Your Files
If you mount the entire bucket without using `only_dir`, the agent can see **all** of your files — including ground truth answers, test cases, and rubrics. Try to avoid this!
Structure your uploaded files with separate directories for server data and agent data:
```
/orwd_data/
├── ground_truth/ ← Server only (never mounted)
│ ├── answers.json
│ ├── rubrics.json
│ └── test_cases.csv
└── agent/ ← Mounted to sandbox
├── transactions.csv
└── instructions.txt
```
Then mount only the `agent/` directory:
```python theme={null}
bucket_config=SandboxBucketConfig(
mount_path="/workspace",
read_only=True,
only_dir="agent", # Only this subdirectory is visible to the agent
)
```
Use `only_dir` to mount a specific subdirectory rather than the entire bucket. This ensures a clear boundary between what the server knows and what the agent sees.
## Full Example
Here is a complete environment that keeps ground truth on the server and mounts only input data to the sandbox:
**File structure in `/orwd_data/`:**
```
/orwd_data/
├── answers.json ← ground truth (server only)
└── agent/ ← mounted to sandbox
└── transactions.csv
```
**Environment code:**
```python theme={null}
import json
import os
from typing import List
from pathlib import Path
from openreward import AsyncOpenReward, SandboxBucketConfig, SandboxSettings
from openreward.environments import Environment, JSONObject, TextBlock, ToolOutput, tool
from pydantic import BaseModel
# Production vs local data path
if os.path.exists("/orwd_data"):
DATA_PATH = Path("/orwd_data")
else:
DATA_PATH = Path(__file__).parent
class BashParams(BaseModel, extra="forbid"):
command: str
class SubmitAnswerParams(BaseModel, extra="forbid"):
answer: str
class MyEnvironment(Environment):
def __init__(self, task_spec: JSONObject, secrets: dict[str, str] = {}) -> None:
super().__init__(task_spec, secrets=secrets)
# Server reads ground truth — agent never sees this
with open(DATA_PATH / "answers.json") as f:
all_answers = json.load(f)
self.expected_answer = all_answers[self.task_id]
# Sandbox mounts ONLY the agent/ directory
self.sandbox_settings = SandboxSettings(
environment="YourUsername/MyEnv",
image="python:3.11-slim",
machine_size="0.5:1",
bucket_config=SandboxBucketConfig(
mount_path="/workspace",
read_only=True,
only_dir="agent", # Only agent/ is visible
),
)
or_client = AsyncOpenReward(api_key=secrets.get("api_key", ""))
self.sandbox = or_client.sandbox(self.sandbox_settings)
async def setup(self) -> None:
await self.sandbox.start()
async def get_prompt(self) -> List[TextBlock]:
return [TextBlock(
text=f"Task {self.task_id}: Analyze the data in /workspace and submit your answer."
)]
@tool
async def bash(self, params: BashParams) -> ToolOutput:
"""Run a bash command in the sandbox"""
output, exit_code = await self.sandbox.run(params.command)
return ToolOutput(blocks=[TextBlock(text=output)])
@tool
async def submit_answer(self, params: SubmitAnswerParams) -> ToolOutput:
"""Submit your final answer"""
is_correct = params.answer.strip() == str(self.expected_answer).strip()
return ToolOutput(
blocks=[TextBlock(text="Correct!" if is_correct else "Incorrect.")],
reward=1.0 if is_correct else 0.0,
finished=True,
)
async def teardown(self) -> None:
await self.sandbox.stop()
```
## Common Mistake: Mounting Everything
**Do not do this** if your bucket contains ground truth data.
```python theme={null}
# DANGEROUS: mounts ALL of /orwd_data/ to the sandbox
bucket_config=SandboxBucketConfig(
mount_path="/workspace",
read_only=True,
# no only_dir — the agent sees everything!
)
```
With this configuration the agent can read your answers, rubrics, and test cases directly. Always use `only_dir` to restrict access to a specific subdirectory.
## Next Steps
Full reference for SandboxBucketConfig parameters
End-to-end tutorial for building an environment with sandboxes
# Your First Environment
Source: https://docs.openreward.ai/environments/your-first-environment
Make and deploy your first ORS environment
## Goals
* Make a mathematics environment using the OpenReward library.
* Deploy the environment to OpenReward.
* Sample from the environment using a model of your choice.
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
## Setup
Environments in OpenReward are written using [ORS](https://openrewardstandard.io).
ORS is implemented in the [OpenReward Python library](https://www.github.com/OpenReward/openreward-python), and we will use it for this tutorial.
You can install the library using pip or uv:
```bash theme={null}
pip install openreward
```
## Background: GSM8K
[GSM8K](https://arxiv.org/abs/2110.14168) is a classic language model dataset for math word problems released by OpenAI in 2021. Problems are at the grade school
level and answers are integers. An example problem and answer from this dataset is shown below:
> Dean's mother gave him \$28 to go to the toy store. Dean bought 6 toy cars and 5 teddy bears. Each toy car cost \$2 and each teddy bear cost \$1. His mother then feels generous and decides to give him an extra \$10. How much money does Dean have left?
> **Answer**: 21
We will learn how to build an ORS environment server for GSM8K in this tutorial.
## Understanding ORS servers
To begin we'll initialise our GSM8K project with a `basic` template and investigate how ORS servers work.
Initialise a project using the OpenReward cli:
```bash theme={null}
orwd init gsm8k --template basic
cd gsm8k && ls
```
The template contains `server.py`, `Dockerfile`, `requirements.txt` and `README.md` with a template [environment card](/environments/environment-cards).
If you look inside `server.py`, you can see a `BasicEnvironment` is defined.
To run this ORS server, install the requirements and run `server.py`:
```bash theme={null}
pip install -r requirements.txt
python server.py
```
```txt theme={null}
INFO: Started server process [92466]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
```
This environment is now running on port `8080`. We will leave this running.
Now let's see how we can interact with the `BasicEnvironment`. In a different terminal, write the following file to `test_environment.py`:
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
environment = or_client.environments.get(name="basicenvironment", base_url="http://localhost:8080")
```
Because the environment class is named `BasicEnvironment`, we will pass in the name `basicenvironment`. We've
also passed in localhost `base_url` since we are running locally.
This is the same API that we use to get production environments on OpenReward. The difference is that we have not passed in a namespace (e.g. `OpenAI/SimpleQA`) and we are pointing to a local `base_url`.
Now let's test interacting with this ORS server.
Environments have splits, which are lists of tasks for different purposes such as training and evaluation. Each split is represented as a `Split` object with a `name` and a `type`. The `type` can be one of `"train"`, `"validation"`, or `"test"`, and is used to help index your environment on OpenReward so users can see whether they should use your environment for training, validation, or testing.
To see the available splits on the `test_environment.py`, add the following to `test_environment.py` and run:
```python theme={null}
print(environment.list_splits())
```
```bash theme={null}
python test_environments.py
```
```
['train', 'test']
```
So there are two splits available in this environment.
Next, let's view the tasks that are available for the `train` split.
```python theme={null}
print(environment.list_tasks("train"))
```
```bash theme={null}
python test_environments.py
```
```
[Task(server_name='basicenvironment', environment_name='basicenvironment', task_spec={'id': 'train-0', 'problem': 'What is 2+2?', 'solution': 4}, namespace=None)]
```
We can see that there is a single task. Here the `Task` object specifies the task specification, which is one of the primitives of [ORS](https://openrewardstandard.io).
You can also fetch a subset of tasks by index using `get_task_range`, which follows Python range conventions (inclusive start, exclusive stop, with support for negative indices). Task ordering is guaranteed to be consistent across calls:
```python theme={null}
# Get the first 10 tasks
tasks = environment.get_task_range("train", start=0, stop=10)
# Get the last 5 tasks
tasks = environment.get_task_range("train", start=-5)
```
Next, let us see the available tools in the environment. In ORS, actions are tools, and executing tools is the only way to interact with the environment.
```python theme={null}
print(environment.list_tools())
```
```bash theme={null}
python test_environments.py
```
```
[ToolSpec(name='answer', description='The answer tool can be used to submit your final answer. Note that this finishes the episode.', input_schema={'description': 'Each tool takes in arguments, and these are specified using types.\n\nExamples:\n- An answer tool might have an answer argument\n- A bash tool might have a command argument ', 'properties': {'answer': {'title': 'Answer', 'type': 'string'}}, 'required': ['answer'], 'title': 'AnswerParams', 'type': 'object'})]
```
There is a single tool called `answer`. Note this tool specification is the same as a tool specification in MCP, allowing compatibility with existing model function calling capabilities.
Now let's test calling the `answer` tool. Write the following script `test_tool.py`:
```python theme={null}
environment = or_client.environments.get(name="basicenvironment", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
tool_result = session.call_tool("answer", {"answer": "4"})
print(prompt)
print(tool_result)
```
```python theme={null}
[TextBlock(text='What is 2+2?', detail=None, type='text')]
ToolOutput(blocks=[TextBlock(text='Correct!', detail=None, type='text')], metadata=None, reward=1.0, finished=True)
```
As we can see, `prompt` contains a `TextBlock` with the prompt text for this task.
After calling the tool on the session with `call_tool` - with the tool name `answer` and tool arguments `{"answer": "4"}` - we obtain a `ToolOutput`.
The `ToolOutput` also contains a list of blocks, in this case a TextBlock showing us some text for the agent (`Correct!`). We also obtain a `reward` of `1.0`, as well as an `finished` state of `True`, denoting that the episode has finished.
This shows us the basics of how we can interface with an ORS environment server. In the next section we will build a GSM8K environment.
## Building the GSM8K environment
First we'll download the two `parquet` files from the GSM8K [HuggingFace repository](https://huggingface.co/datasets/openai/gsm8k/tree/main/main) and put them in the root of our project:
```
test-00000-of-00001.parquet
train-00000-of-00001.parquet
```
A single row of data from the train set looks as follows:
```
{'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72', 'id': '0'}
```
Next we'll write a new server file. This will involve:
* Loading the tasks from the `parquet` files
* Verifying the answer is correct - we'll use the [MathVerify](https://github.com/huggingface/Math-Verify) library for this.
```python theme={null}
from math_verify import parse, verify
import pandas as pd
from pydantic import BaseModel
from openreward.environments import Environment, JSONObject, Server, Split, TextBlock, ToolOutput, tool
class GSM8KTaskSpec(BaseModel):
id: str
question: str
answer: str
class AnswerParams(BaseModel):
answer: str
train_tasks = pd.read_parquet("train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("test-00000-of-00001.parquet").to_dict(orient="records")
for i, task in enumerate(train_tasks):
task['id'] = str(i)
for i, task in enumerate(test_tasks):
task['id'] = str(i)
class GSM8K(Environment):
"""
A GSM8K environment
"""
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = GSM8KTaskSpec.model_validate(task_spec)
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
if split == "train":
return train_tasks
elif split == "test":
return test_tasks
raise ValueError(f"Unknown split: {split}")
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
def get_prompt(self) -> str:
return [TextBlock(type="text", text=self.config.question)]
@tool
def answer(self, params: AnswerParams) -> ToolOutput:
"""
The answer tool can be used to submit your final answer. Note that this finishes the episode.
"""
gold = parse(self.config.answer, parsing_timeout=None)
answer = parse(params.answer, parsing_timeout=None)
is_correct = verify(gold, answer, timeout_seconds=None)
if is_correct:
agent_message = "Correct!"
reward = 1.0
else:
agent_message = "Wrong!"
reward = 0.0
return ToolOutput(
blocks=[TextBlock(type="text", text=agent_message)],
reward=reward,
finished=True
)
if __name__ == "__main__":
Server([GSM8K]).run()
```
**Task ordering must be deterministic.** Your `list_tasks` implementation (and any overrides of `get_task` or `get_task_range`) must always return tasks in the same order. Clients rely on stable indexing to fetch tasks by index, partition work across workers, and reproduce results. Avoid non-deterministic data structures like sets or unordered database queries — use ordered lists or sort your data before returning it.
Install the `math-verify` requirement, along with pandas and fastparquet for parsing:
```bash theme={null}
pip install math-verify pandas fastparquet
```
Now we can test this environment. First run the server as before:
```bash theme={null}
python server.py
```
Now choose a model provider of your choice and sample from the environment:
Make sure you have an API key for [OpenAI](https://platform.openai.com/api-keys), and set the environment variable:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export OPENAI_API_KEY='your-openai-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="gsm8k", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": tool_result.blocks[0].text
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
[ResponseReasoningItem(id='rs_000dde94c785f76800693413cbb66c81928b57cca2b5488c2d', summary=[], type='reasoning', content=None, encrypted_content=None, status=None), ResponseOutputMessage(id='msg_000dde94c785f76800693413ce541881928b37ef5a6ec49cf9', content=[ResponseOutputText(annotations=[], text='72', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"72"}', call_id='call_Eaw188yiAYpNCgQXduR1x0lR', name='answer', type='function_call', id='fc_000dde94c785f76800693413cf1be08192995f531548dffa4d', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_Eaw188yiAYpNCgQXduR1x0lR', 'output': 'Correct!'}
```
Make sure you have API key for [Anthropic](https://platform.claude.com/settings/keys), and set the environment variable:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="gsm8k", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
message = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
messages.append({
"role": "assistant",
"content": message.content,
})
print(messages[-1])
if message.stop_reason == "tool_use":
tool_uses = [b for b in message.content if getattr(b, "type", None) == "tool_use"]
if not tool_uses:
raise RuntimeError("stop_reason was tool_use but no tool_use blocks found")
tool_result_blocks = []
finished = False
for tu in tool_uses:
tool_name = tu.name
tool_input = tu.input
try:
tr = session.call_tool(tool_name, tool_input)
# Convert OpenReward blocks -> string safely
text_parts = []
for b in getattr(tr, "blocks", []) or []:
t = getattr(b, "text", None)
if t is not None:
text_parts.append(t)
else:
text_parts.append(str(b))
tool_text = "".join(text_parts)
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": tool_text,
})
# Track termination if the env says we're done
if getattr(tr, "finished", False):
finished = True
except Exception as e:
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": f"Tool execution failed: {type(e).__name__}: {e}",
"is_error": True,
})
messages.append({
"role": "user",
"content": tool_result_blocks
})
print(messages[-1])
continue
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
Message(id='msg_01DJorrT3UPhnPhdN19ugsyT', content=[TextBlock(citations=None, text='I need to find the total number of clips Natalia sold in April and May.\n\nGiven information:\n- In April, Natalia sold clips to 48 friends\n- In May, she sold half as many clips as in April\n\nLet me calculate:\n- April: 48 clips\n- May: 48 ÷ 2 = 24 clips\n- Total: 48 + 24 = 72 clips', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': '72'}, name='answer', type='tool_use')], model='claude-sonnet-4-6-20250929', role='assistant', stop_reason='tool_use', stop_sequence=None, type='message', usage=Usage(cache_creation=CacheCreation(ephemeral_1h_input_tokens=0, ephemeral_5m_input_tokens=0), cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=608, output_tokens=151, server_tool_use=None, service_tier='standard'))
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}
```
Make sure you have an API key for [Gemini](https://aistudio.google.com/app/apikey) and set it as an environment variable:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export GEMINI_API_KEY='your-gemini-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="gsm8k", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=tools)]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": tool_result.blocks[0].text},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'answer': 'Natalia sold 72 clips altogether in April and May.'
},
name='answer'
),
thought_signature=b'\n\xe3\x02\x01r\xc8\xda|#w\x85/\x9e\xd1\xe7mK\xc1\xdd\xf6X\xeayY\x12<\x0fQ\x1608\xedKg\x07\xe3tV\xd0\xe3\xaf\xc6=\xc0.x\x15\x85\xeb\xec+\x84\xad\xf0\x02\xb6\xf8\x88iX\x94\xf8\xcb8g\xf9m\x02\xcd`\xb1\x8fm\x1e\x10\x1a\x14]2\x81H\x7f-\xb6E6\xd8\xf3\xa6kUw\xb7\xc3kr\x14...'
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='answer',
response={
'result': 'Correct!'
}
)
)] role='user'
```
Make sure you have an API key for [OpenRouter](https://openrouter.ai/keys) and set it as an environment variables:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="gsm8k", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
input_list.append({
"role": "assistant",
"content": response.choices[0].message.content,
"tool_calls": response.choices[0].message.tool_calls
})
print(input_list[-1])
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
break
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
ChatCompletion(id='gen-1765024924-6BK4o9Oiefjb7Fe7J7eW', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content="Let's break this down step by step.\n\n1. In April: Natalia sold clips to **48 friends**. This means she sold **48 clips** (assuming one clip per friend).\n2. In May: She sold **half as many clips** as in April.\n - Half of 48 is \\( 48 \\div 2 = 24 \\).\n3. Altogether in April and May:\n \\[\n 48 + 24 = 72\n \\]\n\nSo Natalia sold **72 clips** altogether in April and May.\n\n", refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=[ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "72"}', name='answer'), type='function', index=0)], reasoning=None), native_finish_reason='tool_calls')], created=1765024924, model='deepseek/deepseek-v3.2', object='chat.completion', service_tier=None, system_fingerprint='', usage=CompletionUsage(completion_tokens=153, prompt_tokens=333, total_tokens=486, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, image_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, video_tokens=0), cost=0.00015417, is_byok=False, cost_details={'upstream_inference_cost': None, 'upstream_inference_prompt_cost': 8.991e-05, 'upstream_inference_completions_cost': 6.426e-05}), provider='SiliconFlow')
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': 'Correct!'}
```
Nice one! We have a working ORS environment.
Now we'll see how we can host the environment on OpenReward. The benefits of using OpenReward are:
* **Infrastructure**: you do not have to set up infrastructure and compute to host the environment yourself. We take care of this and you are only charged based on your actual usage of the environment.
* **Discovery**: your environment can be discovered and used by other users of the platform, helping drive adoption and attention to your work.
We'll see how to host our environment on OpenReward next.
## Host on OpenReward
Log into [OpenReward](https://openreward.ai), press the plus icon in the navbar and press **New Environment**:
Next, fill in information about the environment and press **Create Environment**:
You will be redirected to your new environment and will see setup instructions:
### Upload environment files
We will need a way to use the train and test parquet files in our environment. We'll upload these to the environment files:
Click on the **Files** tab and upload each file:
Files are mounted to the environment server at the `/orwd_data` directory.
We'll need to reference this folder in our `server.py`. Make the following change:
```python theme={null}
train_tasks = pd.read_parquet("/orwd_data/train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("/orwd_data/test-00000-of-00001.parquet").to_dict(orient="records")
```
Note: you may want to set an environment variable instead of hardcoding like above so you can continue to test
locally (without the `/orwd_data` prefix).
### Write the Dockerfile and requirements
We'll need a `Dockerfile` in our repository:
```txt theme={null}
FROM python:3.11-slim
RUN apt update && apt upgrade -y && apt install -y \
curl
WORKDIR /app
# Copy requirements and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application code
COPY server.py .
# Expose port
EXPOSE 8000
# Start the server
CMD ["python", "server.py"]
```
We'll need to update `requirements.txt`:
```txt theme={null}
fastapi>=0.115.12
openreward
pandas
pyarrow
uvicorn>=0.34.3
math-verify[antlr4_13_2]
```
### Push to GitHub and connect
Next, [push your environment code to a GitHub repository](https://github.com/new).
Once your GitHub repository is ready, go to your OpenReward environment and connect the repository:
You will be given a choice for how much compute you would like to allocate for it. We'll use a low compute configuration since this is a simple environment.
Press Connect GitHub and your first build will begin.
To check the progress of the build, click the Deployments tab:
You can click on the latest build row to see logs:
The build logs show the progress of building the environment. The runtime logs show any calls to the environment server, and can be useful for diagnosing errors.
### Sample from your environment
Now your environment is hosted on OpenReward, we can sample from it:
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenAI](https://platform.openai.com/api-keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": tool_result.blocks[0].text
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
[ResponseReasoningItem(id='rs_000dde94c785f76800693413cbb66c81928b57cca2b5488c2d', summary=[], type='reasoning', content=None, encrypted_content=None, status=None), ResponseOutputMessage(id='msg_000dde94c785f76800693413ce541881928b37ef5a6ec49cf9', content=[ResponseOutputText(annotations=[], text='72', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"72"}', call_id='call_Eaw188yiAYpNCgQXduR1x0lR', name='answer', type='function_call', id='fc_000dde94c785f76800693413cf1be08192995f531548dffa4d', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_Eaw188yiAYpNCgQXduR1x0lR', 'output': 'Correct!'}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Anthropic](https://platform.claude.com/settings/keys), and set these as environment variables:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
response = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
print(response)
messages.append({"role": "assistant", "content": response.content})
if response.stop_reason == "tool_use":
tool_use = next(block for block in response.content if block.type == "tool_use")
tool_name = tool_use.name
tool_input = tool_use.input
tool_result = session.call_tool(tool_name, tool_input)
reward = tool_result.reward
finished = tool_result.finished
messages.append({
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": tool_result.blocks[0].text
}
]
})
print(messages[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
Message(id='msg_01DJorrT3UPhnPhdN19ugsyT', content=[TextBlock(citations=None, text='I need to find the total number of clips Natalia sold in April and May.\n\nGiven information:\n- In April, Natalia sold clips to 48 friends\n- In May, she sold half as many clips as in April\n\nLet me calculate:\n- April: 48 clips\n- May: 48 ÷ 2 = 24 clips\n- Total: 48 + 24 = 72 clips', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': '72'}, name='answer', type='tool_use')], model='claude-sonnet-4-6-20250929', role='assistant', stop_reason='tool_use', stop_sequence=None, type='message', usage=Usage(cache_creation=CacheCreation(ephemeral_1h_input_tokens=0, ephemeral_5m_input_tokens=0), cache_creation_input_tokens=0, cache_read_input_tokens=0, input_tokens=608, output_tokens=151, server_tool_use=None, service_tier='standard'))
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Gemini](https://aistudio.google.com/app/apikey), and set these as environment variables:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=tools)]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": tool_result.blocks[0].text},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'answer': 'Natalia sold 72 clips altogether in April and May.'
},
name='answer'
),
thought_signature=b'\n\xe3\x02\x01r\xc8\xda|#w\x85/\x9e\xd1\xe7mK\xc1\xdd\xf6X\xeayY\x12<\x0fQ\x1608\xedKg\x07\xe3tV\xd0\xe3\xaf\xc6=\xc0.x\x15\x85\xeb\xec+\x84\xad\xf0\x02\xb6\xf8\x88iX\x94\xf8\xcb8g\xf9m\x02\xcd`\xb1\x8fm\x1e\x10\x1a\x14]2\x81H\x7f-\xb6E6\xd8\xf3\xa6kUw\xb7\xc3kr\x14...'
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='answer',
response={
'result': 'Correct!'
}
)
)] role='user'
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
input_list.append({
"role": "assistant",
"content": response.choices[0].message.content,
"tool_calls": response.choices[0].message.tool_calls
})
print(input_list[-1])
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
break
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'}]
ChatCompletion(id='gen-1765024924-6BK4o9Oiefjb7Fe7J7eW', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content="Let's break this down step by step.\n\n1. In April: Natalia sold clips to **48 friends**. This means she sold **48 clips** (assuming one clip per friend).\n2. In May: She sold **half as many clips** as in April.\n - Half of 48 is \\( 48 \\div 2 = 24 \\).\n3. Altogether in April and May:\n \\[\n 48 + 24 = 72\n \\]\n\nSo Natalia sold **72 clips** altogether in April and May.\n\n", refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=[ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "72"}', name='answer'), type='function', index=0)], reasoning=None), native_finish_reason='tool_calls')], created=1765024924, model='deepseek/deepseek-v3.2', object='chat.completion', service_tier=None, system_fingerprint='', usage=CompletionUsage(completion_tokens=153, prompt_tokens=333, total_tokens=486, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None, image_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, video_tokens=0), cost=0.00015417, is_byok=False, cost_details={'upstream_inference_cost': None, 'upstream_inference_prompt_cost': 8.991e-05, 'upstream_inference_completions_cost': 6.426e-05}), provider='SiliconFlow')
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': 'Correct!'}
```
If you are running with another provider, or using custom models, then here are the main principles to keep in mind.
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
You'll need to use the OpenReward client to access the environment and its main information
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
```
```python theme={null}
async with environment.session(task=example_task) as session:
```
Using a context manager, we can start a session with the environment. This defines a scope in which
you can use the agent to call tools and get tool results.
Above we have selected the first task to sample from, which is a particular problem in the CTF environment.
You can get the prompt for the task as follows:
```python theme={null}
prompt = await session.get_prompt()
```
You will also need to pass in the `tools` into the context window of your model, usually somewhere in the system prompt.
An agent will usually keep interacting with an environment until it hits a termination state associated with the environment, or
some other imposed limit (e.g. maximum number of terms).
In a simple sequential agent model, this could be a while loop like:
```python theme={null}
while not finished:
```
where `finished` is a boolean.
In agentic environments, actions are treated as tool calls. That means you need to have a way to parse tool calls from your model's generations.
The key thing to note is that for OpenReward environment, a tool call requires specifying a name (`str`) and some arguments (`dict`).
To call a tool you will call:
```python theme={null}
tool_result = await session.call_tool(tool_name, tool_arguments)
```
This means that you will need to parse out the `tool_name` and `tool_arguments` from your model's generation and then parse this information into the
`call_tool` method.
If you have executed an available tool correctly, you will receive a `ToolOutput` output. This contains attributes for:
* `reward` : an (optional) `float` denoting reward. For example, submitting a correct math solution through a `submit_solution` tool might give a reward of `1.0`.
* `finished` : a `bool` specifying whether the episode is finished or not. For example, some tools may end the episode (if for example an agent submits a final answer through `submit_solution` in a math task).
* `data` : a `dict` with output of the executing of the tool. For example, the stdout of executing a `bash` tool. This information should be passed to the agent as feedback.
For our agent loop, some example control flow we might want after receiving a `ToolOutput` is:
* Recording `reward` for use in a policy gradient algorithm such as GRPO or PPO
* Breaking out of the core agent loop if `finished=True`
* Adding `data` to the context window as feedback and continuing with the next model generation
## Next Steps
Build a more advanced environment that uses Computers
Learn how to run evals with OpenReward.
Learn how to use OpenReward for reinforcement learning
# Harness Toolsets
Source: https://docs.openreward.ai/harnesses/harness-toolsets
Expose agent-native tool surfaces in your environments using session-scoped toolsets
## Introduction
Different agent CLIs expose different tool surfaces. In OpenReward, you can expose agent-native tool surfaces server-side, backed by your environment's sandbox. When a session is created with a toolset, the agent sees the exact tools it expects and every tool call executes against the sandbox.
## How Toolsets Compose into Sessions
Harness toolsets are **session-scoped**: they are passed when creating a session and their tools are merged with the environment's own tools. The `openreward` package includes a registry of built-in harness toolsets that can be referenced by name:
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
env = or_client.environments.get(name="MyOrg/MyEnv")
tasks = env.list_tasks(split="train")
# Pass a toolset by name — the session exposes agent-native tools
with env.session(task=tasks[0], toolset="claude-code") as session:
tools = session.list_tools(format="anthropic")
# tools now include: bash, glob, grep, read, write, edit, todo_write
# plus any environment-specific tools (e.g. submit_answer)
result = session.call_tool("bash", {"command": "ls /tmp"})
print(result.blocks[0].text)
```
When a session toolset defines a tool with the same name as an environment tool, the **toolset tool takes precedence**. A warning is logged for each shadowed tool. This lets harness toolsets override environment-level `bash`, `read`, etc. with their agent-specific implementations.
## Sandbox Requirement
All harness toolsets extend the `Toolset` base class, which requires the environment to have a `self.sandbox` attribute. When the toolset is instantiated against an environment, it automatically extracts `self.sandbox` and uses it for all tool executions.
If you're building an environment that supports harness toolsets, ensure your environment sets up a sandbox:
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings
from openreward.environments import Environment, tool, ToolOutput, TextBlock
class MyEnv(Environment):
def __init__(self, task_spec, secrets):
super().__init__(task_spec, secrets)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(SandboxSettings(
environment="MyOrg/MyEnv",
image="generalreasoning/knowledge-worker:latest",
machine_size="0.5:1",
))
async def setup(self):
await self.sandbox.start()
async def teardown(self):
await self.sandbox.stop()
# Environment-specific tools
@tool
async def submit_answer(self, params) -> ToolOutput:
...
```
When a client creates a session with `toolset="claude-code"`, the `ClaudeCodeToolset` is instantiated with this environment and gains access to `self.sandbox`. All its tools (`bash`, `read`, `write`, etc.) execute commands against that sandbox.
## Available Harness Toolsets
### ClaudeCodeToolset
**Name:** `"claude-code"` · **Tools:** 7
Mirrors the Claude Code CLI's built-in tool surface. Tool descriptions are sourced from Claude Code's internal prompts.
| Tool | Description |
| ------------ | ------------------------------------------------------------------------------ |
| `bash` | Execute shell commands with full bash support |
| `glob` | Find files matching glob patterns |
| `grep` | Search file contents with regex (ripgrep-style) |
| `read` | Read file contents with optional offset/limit pagination |
| `write` | Write content to a file, creating directories as needed |
| `edit` | Exact string replacement in files (unique match required unless `replace_all`) |
| `todo_write` | Manage a todo list for task planning and progress tracking |
```python theme={null}
with env.session(task=task, toolset="claude-code") as session:
session.call_tool("bash", {"command": "python solve.py"})
session.call_tool("read", {"file_path": "/home/ubuntu/output.txt"})
session.call_tool("edit", {
"file_path": "/home/ubuntu/solve.py",
"old_string": "x = 1",
"new_string": "x = 2"
})
```
### CodexToolset
**Name:** `"codex"` · **Tools:** 1
Codex prefers a single shell tool over discrete file tools. All file operations are done through bash.
| Tool | Description |
| ------ | ----------------------------------------------- |
| `bash` | Run shell commands; use for all file operations |
```python theme={null}
with env.session(task=task, toolset="codex") as session:
session.call_tool("bash", {"command": "cat solve.py"})
session.call_tool("bash", {"command": "sed -i 's/x = 1/x = 2/' solve.py"})
```
### GeminiCliToolset
**Name:** `"gemini-cli"` · **Tools:** 8
Matches the upstream Gemini CLI tool surface. Uses Gemini-specific tool names like `run_shell_command` and `replace`.
| Tool | Description |
| ------------------- | ------------------------------------------------------------------------- |
| `run_shell_command` | Execute shell commands as `bash -c ` |
| `glob` | Find files matching glob patterns, sorted by modification time |
| `grep_search` | Search file contents with regex, max 100 matches |
| `read_file` | Read file contents with optional line range |
| `write_file` | Write content to a file |
| `replace` | Find-and-replace in files (unique match required unless `allow_multiple`) |
| `list_directory` | List files and subdirectories in a path |
| `write_todos` | Track subtasks for complex queries |
```python theme={null}
with env.session(task=task, toolset="gemini-cli") as session:
session.call_tool("run_shell_command", {"command": "python solve.py"})
session.call_tool("read_file", {"file_path": "/home/ubuntu/output.txt"})
session.call_tool("replace", {
"file_path": "/home/ubuntu/solve.py",
"old_string": "x = 1",
"new_string": "x = 2"
})
```
### OpenClawToolset
**Name:** `"openclaw"` · **Tools:** 6
Exposes the OpenClaw coding tool surface, including background process management and structured patch application.
| Tool | Description |
| ------------- | ------------------------------------------------------------------------- |
| `exec` | Execute shell commands with configurable timeout (default 1800s) |
| `process` | Manage background processes: list, poll, log, write stdin, kill, remove |
| `read` | Read file contents with optional offset/limit pagination |
| `write` | Write content to a file, creating parent directories |
| `edit` | Apply targeted text replacements with `oldText`/`newText` pairs |
| `apply_patch` | Apply structured patches with `*** Begin Patch` / `*** End Patch` markers |
```python theme={null}
with env.session(task=task, toolset="openclaw") as session:
session.call_tool("exec", {"command": "python solve.py"})
session.call_tool("edit", {
"path": "/home/ubuntu/solve.py",
"edits": [{"oldText": "x = 1", "newText": "x = 2"}]
})
session.call_tool("apply_patch", {
"input": "*** Begin Patch\n*** Update File: solve.py\n- x = 1\n+ x = 2\n*** End Patch"
})
```
### HermesToolset
**Name:** `"hermes"` · **Tools:** 5
Exposes the Hermes Agent tool surface from Nous Research. Supports both targeted string replacement and V4A multi-file patches.
| Tool | Description |
| -------------- | ---------------------------------------------------------------------------------- |
| `terminal` | Execute shell commands (default timeout 180s, max 600s) |
| `read_file` | Read files with `LINE_NUM\|CONTENT` format, pagination via offset/limit |
| `write_file` | Write content to a file, creating parent directories |
| `search_files` | Search file contents (`target="content"`) or find files by name (`target="files"`) |
| `patch` | Replace mode (find-and-replace) or patch mode (V4A multi-file patches) |
```python theme={null}
with env.session(task=task, toolset="hermes") as session:
session.call_tool("terminal", {"command": "python solve.py"})
session.call_tool("search_files", {
"pattern": "def main",
"path": "/home/ubuntu",
"file_glob": "*.py"
})
session.call_tool("patch", {
"mode": "replace",
"path": "/home/ubuntu/solve.py",
"old_string": "x = 1",
"new_string": "x = 2"
})
```
## Tool Comparison
| Capability | Claude Code | Codex | Gemini CLI | OpenClaw | Hermes |
| ------------------ | ------------ | ---------- | ------------------- | ------------- | ----------------------------- |
| Shell execution | `bash` | `bash` | `run_shell_command` | `exec` | `terminal` |
| File read | `read` | via `bash` | `read_file` | `read` | `read_file` |
| File write | `write` | via `bash` | `write_file` | `write` | `write_file` |
| File edit | `edit` | via `bash` | `replace` | `edit` | `patch` (replace mode) |
| File search | `grep` | via `bash` | `grep_search` | via `exec` | `search_files` |
| File glob | `glob` | via `bash` | `glob` | via `exec` | `search_files` (target=files) |
| Directory listing | via `bash` | via `bash` | `list_directory` | via `exec` | `search_files` (target=files) |
| Task planning | `todo_write` | — | `write_todos` | — | — |
| Process management | — | — | — | `process` | — |
| Patch application | — | — | — | `apply_patch` | `patch` (patch mode) |
## Example: Full Environment with Harness Toolset Support
Here is a complete environment that supports harness toolsets. The environment defines its own task-specific tools (`submit_answer`), while the harness toolset provides the agent's coding tools:
```python theme={null}
from openreward import AsyncOpenReward, SandboxSettings, SandboxBucketConfig
from openreward.environments import Environment, Split, tool, ToolOutput, TextBlock
from pydantic import BaseModel
class AnswerParams(BaseModel):
answer: str
class CodingChallenge(Environment):
def __init__(self, task_spec, secrets):
super().__init__(task_spec, secrets)
or_client = AsyncOpenReward(api_key=secrets.get("api_key"))
self.sandbox = or_client.sandbox(SandboxSettings(
environment="MyOrg/CodingChallenge",
image="generalreasoning/knowledge-worker:latest",
machine_size="4:16",
block_network=False,
bucket_config=SandboxBucketConfig(
mount_path="/tmp/datasets/",
read_only=True,
)
))
async def setup(self):
await self.sandbox.start()
async def teardown(self):
await self.sandbox.stop()
def get_prompt(self):
task = self.task_spec
return [TextBlock(
text=f"Solve the following coding challenge:\n\n{task['problem']}\n\n"
f"Submit your answer using the submit_answer tool."
)]
@tool
async def submit_answer(self, params: AnswerParams) -> ToolOutput:
"""Submit your solution"""
correct = params.answer.strip() == self.task_spec["expected"]
return ToolOutput(
blocks=[TextBlock(text="Correct!" if correct else "Incorrect.")],
reward=1.0 if correct else 0.0,
finished=True,
)
@classmethod
def list_tasks(cls, split: str):
if split == "train":
return [
{"id": "1", "problem": "Write a function that reverses a string.", "expected": "..."},
]
return []
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
```
Clients can then evaluate this environment with any harness toolset:
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
env = or_client.environments.get(name="MyOrg/CodingChallenge")
tasks = env.list_tasks(split="train")
# Evaluate with Claude Code's tool surface
with env.session(task=tasks[0], toolset="claude-code") as session:
tools = session.list_tools(format="anthropic")
# Agent sees: bash, glob, grep, read, write, edit, todo_write, submit_answer
# Evaluate with Codex's tool surface
with env.session(task=tasks[0], toolset="codex") as session:
tools = session.list_tools(format="openai")
# Agent sees: bash, submit_answer
# Evaluate with Hermes's tool surface
with env.session(task=tasks[0], toolset="hermes") as session:
tools = session.list_tools(format="anthropic")
# Agent sees: terminal, read_file, write_file, search_files, patch, submit_answer
```
Or with [firehorse](https://github.com/GeneralReasoning/firehorse) from the command line:
```bash theme={null}
# Each of these uses the corresponding toolset automatically
firehorse --env MyOrg/CodingChallenge --agent claude-code --model anthropic/claude-sonnet-4-6
firehorse --env MyOrg/CodingChallenge --agent codex --model openai/codex-mini
firehorse --env MyOrg/CodingChallenge --agent react --model openrouter/nousresearch/hermes-3-llama-3.1-405b
```
## Next Steps
Learn about document toolsets (PDF, Excel, Word, PowerPoint)
Create sandbox-based environments from scratch
Run agent harnesses with Firehorse
Choose a sandbox provider for your environments
# Harness Quickstart
Source: https://docs.openreward.ai/harnesses/quickstart
Run agent harnesses against OpenReward environments with firehorse
[Firehorse](https://github.com/GeneralReasoning/firehorse) is a library for running agent harnesses against any OpenReward environment. It works by composing the appropriate [harness toolset](/harnesses/harness-toolsets) with the environment, connecting the harness agent, and orchestrating the agent loop.
## Prerequisites
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* An API key for your model provider (Anthropic, OpenAI, Google, or OpenRouter)
* Python 3.10+
## Installation
```bash theme={null}
pip install firehorse-cli
```
For specific agent types, you may also need the corresponding CLI tool installed:
| Agent | Additional requirement |
| ---------------- | ---------------------------------------------------------------------- |
| `claude-code` | [Claude Code](https://docs.anthropic.com/en/docs/claude-code) v2.1.88+ |
| `codex` | [Codex CLI](https://github.com/openai/codex) v0.121.0+ |
| `gemini` | [Gemini CLI](https://github.com/google-gemini/gemini-cli) |
| `resum`, `react` | None (API-only, no CLI needed) |
## Your First Evaluation
Set your API keys:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key'
export ANTHROPIC_API_KEY='your-anthropic-api-key'
```
Run an evaluation:
```bash theme={null}
firehorse \
--env GeneralReasoning/terminal-bench-2-verified \
--agent claude-code \
--model anthropic/claude-sonnet-4-6 \
--split test \
--max-tasks 5 \
--output-dir ./results
```
This launches Claude Code as a subprocess against the terminal-bench-2-verified environment, running 5 tasks from the test split. Results are written to `./results`.
## Available Agents
Firehorse ships five agent harnesses, each with a different architecture:
| Agent | Approach | Description |
| ------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `resum` | API + context compaction | Default agent. Implements a ReAct loop with 3-layer context compaction for long trials. Multi-provider support. |
| `claude-code` | Subprocess + MCP | Launches Claude Code CLI per trial. Disables filesystem builtins and replaces them with sandboxed MCP tools. Supports extended thinking via `--effort`. |
| `codex` | Subprocess + MCP | Launches OpenAI Codex CLI. Uses a single `bash` tool surface. Read-only filesystem sandbox mode. |
| `gemini` | Subprocess + MCP | Launches Google Gemini CLI. Pre-builds tool specs to avoid CLI discovery timeout. |
| `react` | API-direct | Direct LLM API integration with a straightforward reason-act loop. Supports Anthropic, OpenAI, Google, and OpenRouter. No subprocess overhead. |
**Subprocess + MCP agents** (claude-code, codex, gemini) launch the respective CLI as a child process and proxy environment tools via [MCP](https://modelcontextprotocol.io). The agent's built-in filesystem tools are disabled and replaced with sandbox-backed equivalents.
**API-direct agents** (resum, react) call LLM APIs directly and execute tool calls via the OpenReward session. No local CLI is required.
## Key Options
| Option | Description | Default |
| ---------------- | --------------------------------------------------------------------------------------------------------------- | ---------------- |
| `--env` | Environment name (e.g. `GeneralReasoning/CTF`) | Required |
| `--agent` | Agent type (`resum`, `claude-code`, `codex`, `react`, `gemini`) | `resum` |
| `--model` | Model identifier with provider prefix (e.g. `anthropic/claude-sonnet-4-6`, `openrouter/deepseek/deepseek-v3.2`) | Required |
| `--effort` | Reasoning depth: `none`, `low`, `medium`, `high`, `max`, `xhigh` | Provider default |
| `--n-concurrent` | Number of parallel trials | `1` |
| `--max-tasks` | Limit number of tasks to evaluate | All tasks |
| `--max-turns` | Maximum tool calls per trial | `100` |
| `--split` | Data split to evaluate (`train`, `test`, `validation`) | `train` |
| `--variant` | Environment variant | Default variant |
| `--output-dir` | Directory for results and trajectory logs | `./output` |
| `--provider-url` | Custom model provider endpoint | — |
| `--secrets` | Extra secrets as `KEY=VALUE` pairs | — |
### Model Identifiers
Models are specified with a provider prefix:
```bash theme={null}
# Anthropic
--model anthropic/claude-sonnet-4-6
# OpenAI
--model openai/gpt-5.4
# Google
--model google/gemini-2.5-flash
# OpenRouter (any model)
--model openrouter/deepseek/deepseek-v3.2
```
### Effort Levels
The `--effort` flag controls reasoning depth and maps to each provider's native mechanism:
| Effort | Anthropic | OpenAI | Google |
| -------- | -------------------------- | -------------------------- | ------------------------ |
| `low` | Adaptive thinking (low) | `reasoning_effort: low` | `thinking_level: low` |
| `medium` | Adaptive thinking (medium) | `reasoning_effort: medium` | `thinking_level: medium` |
| `high` | Adaptive thinking (high) | `reasoning_effort: high` | `thinking_level: high` |
| `max` | Max thinking tokens | — | — |
| `xhigh` | — | `reasoning_effort: xhigh` | — |
## Understanding the Output
Firehorse produces three types of output per trial:
* **Result JSON** — final metrics: total reward, tool call count, token usage, API cost, duration
* **Trajectory JSONL** — full event log capturing reasoning steps, tool calls, and tool results
* **Aggregate summary** — statistics across all trials
Example output structure:
```
results/
├── trial_0/
│ ├── result.json # Final metrics
│ ├── trajectory.jsonl # Full event log
│ └── rewards.jsonl # Timestamped reward signals
├── trial_1/
│ └── ...
└── summary.json # Aggregate statistics
```
## Examples
### Evaluate with OpenRouter
```bash theme={null}
export OPENROUTER_API_KEY='your-key'
firehorse \
--env GeneralReasoning/CTF \
--agent react \
--model openrouter/moonshotai/kimi-k2.6 \
--n-concurrent 4 \
--max-tasks 20
```
### High-effort Claude Code evaluation
```bash theme={null}
export ANTHROPIC_API_KEY='your-key'
firehorse \
--env Eigent/SETA \
--agent claude-code \
--model anthropic/claude-sonnet-4-6 \
--effort high \
--n-concurrent 8 \
--split train
```
### Codex on a custom environment
```bash theme={null}
export OPENAI_API_KEY='your-key'
firehorse \
--env MyOrg/MyEnv \
--agent codex \
--model openai/gpt-5.4 \
--variant hard \
--max-turns 50
```
## Next Steps
Configure agent-native tool surfaces for your environments
Build a custom evaluation environment
Create sandbox-based environments for agent tasks
Compose reusable tool collections into environments
# What is OpenReward?
Source: https://docs.openreward.ai/index
OpenReward is a platform for hosting environments to train and evaluate language model agents. It is built on [Open Reward Standard (ORS)](https://openrewardstandard.io) which specifies a standard interface by which language models can interact with an environment.
An environment is a simulated space where an agent can perform a task using tools and resources. For example, a Kaggle environment gives an agent tools to interact with a computer, data to build models, and tools to submit a solution.
Environments are used to train agents with reinforcement learning. Actions in an environment yield observations with rewards; for example, submitting a correct solution to a mathematics problem yields a positive reward.
Environments are also used to evaluate agents. For example, to see how well a language model agent does in a company workflow task, you can make an environment to simulate it and test the agent within it.
## What is ORS?
The [Open Reward Standard (ORS)](https://openrewardstandard.io) is an open-source standard for connecting language model agents to environments.
It specifies how a language model agent can interact with an environment to manipulate its state and obtain results and rewards.
In ORS, an environment is a server that a language model agent can interact with.
A key assumption in ORS is that **actions are tools**. The only way the agent can interact with an environment is by
calling tools. This allows ORS environments to utilise existing function calling support by model providers.
An ORS environment server provides access to:
1. **Tools**: the core methods for interacting with an environment, for example a `bash` tool or a `submit_solution` tool.
2. **Tasks**: the core tasks to be accomplished in an environment, for example math problems in a mathematics environment.
3. **Splits**: different lists of tasks for use cases like training, evaluation and development. These task lists are associated with split names, e.g. `train`.
4. **Prompts**: instructions to the agent associated with each task. These are used to prompt the language model at the beginning of a task, for example `Who is the author of the document at /home/ubuntu/the-bitter-lesson.md?`.
### How does ORS compare to MCP?
The Model Context Protocol (MCP) is a standard for connecting language models to external tools, data sources and workflows. It does not specify all the necessary elements for an agentic environment, including reward and when an episode should be terminated. It also does not contain information on tasks and splits that are useful for training and evaluation.
For these reasons, OpenReward uses a new standard ([ORS](https://openrewardstandard.io)) to standardise how an agent should interact with an environment. Much of the specification is strongly aligned to MCP, such as how tools are listed and called. Tool results differ in that they also surface fields such as `reward` and `finished`. Other concepts are not present in MCP at all, such as tasks and splits.
## What is the benefit of OpenReward?
There are several core use cases with OpenReward:
* **Hosting environments** - OpenReward provides infrastructure to host environments and a simple way to write them through [ORS](https://openrewardstandard.io). Developers can focus on writing environments without worrying about infrastructure.
* **Training models** - OpenReward has integrations with major training frameworks allowing for low friction training of community or user-defined environments.
* **Evaluating models** - OpenReward allows for fast evaluation of new and existing models on agentic environments, allowing evaluators to focus on evaluation without setting up infrastructure themselves.
## Start Building
Sample from an environment in minutes
Create and deploy your first OpenReward environment
Learn how workspaces, environments, and sandboxes work together
Automatically deploy environments from your GitHub repository
Use cloud storage with environments and sandboxes
Run code in isolated, ephemeral containers
# Using Harbor Environments
Source: https://docs.openreward.ai/integrations/using-harbor-environments
Convert Harbor tasks into OpenReward environments using harbor2or
## Goals
* Understand what [Harbor](https://harborframework.com) is and how its tasks map to OpenReward environments.
* Install and use the `harbor2or` CLI tool.
* Convert a Harbor task source into a deployable [ORS](https://openrewardstandard.io) environment.
* Build, test, and deploy the generated environment to OpenReward.
## Prerequisites
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* Python 3.11+
* Docker installed locally (for building task images)
* Familiarity with [environments](/concepts/environments) and [deployment](/deployment/github-integration)
## What is Harbor?
[Harbor](https://harborframework.com) is a framework for defining agent tasks using a standardized directory structure. Each Harbor task includes instructions, a container environment (Dockerfile or pre-built image), verification tests, and optional oracle solutions.
The `harbor2or` CLI tool converts Harbor task repositories into [ORS](https://openrewardstandard.io) environments that can be deployed on OpenReward. It handles downloading tasks, generating environment server code, building Docker images, and validating tasks with oracle agents.
## Installation
Install `harbor2or` from GitHub:
```bash theme={null}
pip install git+https://github.com/OpenRewardAI/harbor2or.git
```
If you plan to use HuggingFace datasets as a source, install the optional dependencies:
```bash theme={null}
pip install huggingface_hub pyarrow
```
## Quick Start
Point `harbor2or create` at a Harbor task source. This can be a GitHub repository, a HuggingFace dataset, or a local directory:
```bash theme={null}
harbor2or create
```
For example, to convert Terminal-Bench into an OpenReward environment:
```bash theme={null}
# From GitHub
harbor2or create https://github.com/laude-institute/terminal-bench-2 terminal-bench
# Or from HuggingFace
harbor2or create zai-org/terminal-bench-2-verified terminal-bench
```
This generates a complete [ORS](https://openrewardstandard.io) environment in `./terminal-bench/` containing:
* `server.py` -- the environment server with tools and task logic
* `Dockerfile` and `requirements.txt` -- for containerised deployment
* `splits.json` and `tasks.txt` -- task metadata and split mappings
You can customise the output with flags:
```bash theme={null}
harbor2or create \
--output ./custom-dir \
--image-prefix myorg/env-name \
--environment MyOrg/MyEnvironment
```
Build the Docker images for each task locally:
```bash theme={null}
harbor2or build terminal-bench --local
```
This builds and pushes an image per task, then caches the image digest in each task's `sha.txt` file.
For production workloads, `harbor2or build` also supports Google Cloud Build for parallel, remote image building. See the [harbor2or README](https://github.com/OpenRewardAI/harbor2or) for details.
Run the oracle agent on tasks that include a `solution/solve.sh` file:
```bash theme={null}
export OPENREWARD_API_KEY='your-api-key'
harbor2or test terminal-bench --concurrency 4 --verbose
```
This starts a sandbox for each task, uploads and runs the oracle solution, then reports pass/fail results with a summary.
Push the generated environment to GitHub and connect it via the OpenReward dashboard:
```bash theme={null}
cd terminal-bench
git init && git add -A && git commit -m "Initial environment"
git remote add origin https://github.com/yourorg/terminal-bench.git
git push -u origin main
```
Then follow the [GitHub deployment guide](/deployment/github-integration) to connect the repository and trigger your first build on OpenReward.
## Tools
Harbor environments include the [ClaudeCodeToolset](/harnesses/harness-toolsets#claudecodetoolset) at the class level, which provides `bash`, `glob`, `grep`, `read`, `write`, `edit`, and `todo_write`. The environment also exposes a `submit_answer` tool for running verification and returning the reward.
When an agent calls `submit_answer`, the environment executes the `tests/test.sh` script from the original Harbor task to verify the result. The reward is read from `/logs/verifier/reward.txt` in the sandbox.
## Supported Sources
The `harbor2or create` command accepts several source formats:
* **GitHub repository**: `https://github.com/org/repo`
* **HuggingFace dataset URL**: `https://huggingface.co/datasets/org/name`
* **HuggingFace short ID**: `org/name`
* **Local directory**: `./path/to/tasks`
Harbor tasks are auto-detected within the source. Each valid task must contain an `instruction.md`, `task.toml`, `tests/test.sh`, and either an `environment/Dockerfile` or a `docker_image` reference in `task.toml`.
## Next Steps
Full CLI reference, advanced options, and Cloud Build setup.
Learn the fundamentals of building ORS environments from scratch.
Connect your environment repository to OpenReward for automatic deployment.
# Using Verifiers Environments
Source: https://docs.openreward.ai/integrations/using-verifiers-environments
Convert Verifiers tasks into OpenReward environments using verifiers2or
## Goals
* Understand what [Verifiers](https://docs.primeintellect.ai/verifiers/overview) is and how its environments map to OpenReward.
* Install and use the `verifiers2or` CLI tools.
* Convert a Verifiers environment into a deployable [ORS](https://openrewardstandard.io) environment.
* Build, test, and deploy the generated environment to OpenReward.
## Prerequisites
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* Python 3.11+
* A Verifiers environment package to convert (e.g., `wordle`)
* Familiarity with [environments](/concepts/environments) and [deployment](/deployment/github-integration)
## What is Verifiers?
[Verifiers](https://docs.primeintellect.ai/verifiers/overview) is Prime Intellect's library for creating environments to train and evaluate LLMs. Each environment is a self-contained Python module that packages a dataset of task inputs, a harness for tools and context management, and a reward function for scoring. Verifiers environments support reinforcement learning training, capability evaluation, synthetic data generation, and agent experimentation.
The key architectural difference with ORS is **[Cartesian separation](https://arxiv.org/abs/1902.09469)**. In Verifiers, the environment and agent are entangled: the environment owns the agent loop, calls the model, parses its raw text output, and scores the episode. Environment and agent are one fused unit, so changing either means rewriting the other.
ORS decouples these completely. The environment is an HTTP server that exposes tools and returns structured feedback. The agent is an independent process that calls those tools. Neither knows the other's internals. This means any agent can be paired with any environment - they are orthogonal axes, not coupled components.
* **Verifiers**: Environment ↔ Agent are entangled. The environment drives the loop, calls the model, and parses raw text.
* **ORS**: Environment ⊥ Agent are separated. The environment exposes tools via HTTP; the agent drives the loop and calls tools.
The `verifiers2or` toolkit bridges these paradigms, letting you convert existing Verifiers environments into ORS-compatible servers.
## Installation
Install `verifiers2or` from GitHub:
```bash theme={null}
pip install git+https://github.com/OpenRewardAI/verifiers2or.git
```
You also need the Verifiers environment package you want to convert:
```bash theme={null}
pip install verifiers>=0.1.9
```
## Quick Start
Use `analyzer` to inspect a Verifiers environment before converting:
```bash theme={null}
python -m converter.analyzer wordle
```
This outputs the environment type, system prompt, parser fields, reward functions, dataset structure, and more:
```
=== Verifiers Environment Analysis ===
Class: TextArenaEnv
Type: MultiTurnEnv
System prompt: 'You are a competitive game player...'
--- Parser ---
Type: XMLParser
Fields: ['guess']
Answer field: guess
--- Reward Functions ---
correct_answer (weight=1.0)
partial_answer (weight=1.0)
length_bonus (weight=1.0)
```
The analyzer accepts installed packages (`wordle`, `primeintellect/gsm8k`) or local files (`./path/to/my_env.py`).
You have two conversion options:
**Option A: Quick wrap (zero-code)**
Wrap the environment as an ORS server with one command:
```bash theme={null}
python -m converter.wrap wordle
```
This starts an ORS server on `http://localhost:8080` with tools, tasks, and splits automatically extracted from the Verifiers environment.
To pass arguments to the environment:
```bash theme={null}
python -m converter.wrap wordle --env-args '{"num_train_examples": 500}'
```
The wrapper is convenient for quick testing but may be unreliable for complex environments. It attempts to automatically bridge verifiers' internal state management, message formats, and reward computation, which can break for environments with non-standard patterns. For production use or environments with complex multi-turn logic, user simulation, or custom state handling, we recommend using the `scaffold` command and doing a full manual conversion instead.
**Option B: Generate scaffold (full control)**
Generate ORS environment code for customisation:
```bash theme={null}
python -m converter.scaffolder wordle -o ./my_wordle -n wordle
```
This creates:
* `wordle_env.py` -- ORS Environment subclass with TODOs for game logic
* `server.py` -- Server entry point
* `test_wordle.py` -- Client test script
Start the server and run the test script:
```bash theme={null}
# Terminal 1: Start the server
cd ./my_wordle
python server.py
# Terminal 2: Run the test
python test_wordle.py
```
Verify that splits and tasks are listed, tools appear with correct schemas, and episodes complete with `finished=True` and valid rewards.
Push the generated environment to GitHub and connect it via the OpenReward dashboard:
```bash theme={null}
cd my_wordle
git init && git add -A && git commit -m "Initial environment"
git remote add origin https://github.com/yourorg/my-wordle.git
git push -u origin main
```
Then follow the [GitHub deployment guide](/deployment/github-integration) to connect the repository and trigger your first build on OpenReward.
## Conversion Methods
| Method | Use Case | Output |
| ---------- | -------------------------- | --------------------------------------------- |
| `wrap` | Quick prototyping, testing | ORS server at runtime (no code generated) |
| `scaffold` | Production, customisation | Python files with TODOs for manual completion |
The `wrap` command reuses all original Verifiers logic (game state, feedback, scoring) at runtime. Use `scaffold` when you need full control over the ORS implementation.
## Concept Mapping
This table shows how Verifiers concepts translate to ORS:
| Verifiers | ORS | Notes |
| --------------------- | --------------------------- | --------------------------------------------- |
| `load_environment()` | `__init__()` + `setup()` | ORS creates one instance per session |
| Dataset (HuggingFace) | `list_tasks(split)` | Each row becomes a task JSON object |
| Rubric + reward funcs | `ToolOutput.reward` | Reward returned on terminal tool call |
| Parser (XMLParser) | Pydantic `BaseModel` | Structured tool input replaces text parsing |
| `env_response()` | `@tool` method | Agent calls tools instead of env parsing text |
| `system_prompt` | `get_prompt()` | Prompt delivered via API |
| `State` dict | Instance attributes | ORS env is instantiated per session |
| `@vf.stop` conditions | `ToolOutput(finished=True)` | Termination signaled via tool output |
| train/eval datasets | `list_splits()` + `Split` | Named splits with type annotation |
## Supported Environment Types
| Type | Wrapper Support | Scaffold Support | Notes |
| ----------------- | --------------- | ---------------- | ------------------------------- |
| `SingleTurnEnv` | Full | Full | One `submit` tool |
| `MultiTurnEnv` | Full | Full | Tool from parser fields |
| `ToolEnv` | Full | Full | One ORS tool per Verifiers tool |
| `StatefulToolEnv` | Full | Full | Same as ToolEnv |
| `SandboxEnv` | Partial | Template + TODOs | Requires manual sandbox config |
## Next Steps
Full CLI reference, conversion guide, and worked examples.
Learn the fundamentals of building ORS environments from scratch.
Connect your environment repository to OpenReward for automatic deployment.
# Quickstart
Source: https://docs.openreward.ai/quickstart
Sample from an environment in minutes
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
## Sample an Environment
In this example we'll call the `GeneralReasoning/CTF` environment on OpenReward and sample an agent trace using the Python SDK.
This environment is publicly hosted on OpenReward - i.e. you will be accessing it through the cloud, not locally.
Choose a model provider below and sample your first environment!
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenAI](https://platform.openai.com/api-keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
```bash theme={null}
pip install openreward
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": tool_result.blocks[0].text
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
{'role': 'user', 'content': "You are solving a CTF (Capture The Flag) challenge.\n\nChallenge: pancakes\n\nYou ever just get a craving for pancakes? > nc chal.tuctf.com 30503 [pancakes](pancakes) ', 'A' * 44 + p32(e.plt['puts']) + 'AAAA' + p32(e.symbols['password'])) print p.recvall()\n\nYour goal is to find the flag and submit it using the submit_answer tool.\n\nIf challenge files are available, then they have been uploaded to your environment in /tmp/gr-datasets folder.\n\nSometimes you will be given a link to download from or access.\n\nYou should work from the /home/ubuntu directory.\n\nGood luck!"}]
[ResponseReasoningItem(id='rs_0f6e802a28329b2f00696b6de55018819da22725bb44fba022', summary=[], type='reasoning', content=None, encrypted_content=None, status=None), ResponseFunctionToolCall(arguments='{"path":"/tmp/gr-datasets","show_hidden":false,"recursive":false}', call_id='call_VYpjo87wSocwCsrEc7QOvAY7', name='list_files', type='function_call', id='fc_0f6e802a28329b2f00696b6de68780819d85967ff172cae065', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_VYpjo87wSocwCsrEc7QOvAY7', 'output': '{"result": "total 16\\n-rw-r--r-- 1 root root 15696 Jan 15 10:07 pancakes"}'}
[ResponseReasoningItem(id='rs_0f6e802a28329b2f00696b6de7aa04819d85f6f667d4a1fcbb', summary=[], type='reasoning', content=None, encrypted_content=None, status=None), ResponseFunctionToolCall(arguments='{"command":"cp /tmp/gr-datasets/pancakes /home/ubuntu/ && cd /home/ubuntu && ls -l pancakes && file pancakes && sha256sum pancakes"}', call_id='call_zKDEIePRL8nYrh25TMgN2Gvo', name='bash', type='function_call', id='fc_0f6e802a28329b2f00696b6de86d3c819d9122aaaa02bc1a6f', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_zKDEIePRL8nYrh25TMgN2Gvo', 'output': '{"result": "-rw-r--r-- 1 root root 15696 Jan 17 11:09 pancakes\\npancakes: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=a653a608db5ab4716ca7b1e891ee3b460e097eb8, for GNU/Linux 3.2.0, not stripped\\nbd470b542b111437343079807073089ab024de3d89c45ac0926c6a9ed8c36a4a pancakes\\n\\n(exit 0)"}'}
[ResponseReasoningItem(id='rs_0f6e802a28329b2f00696b6dea067c819d9661f75dec1ea93a', summary=[], type='reasoning', content=None, encrypted_content=None, status=None), ResponseFunctionToolCall(arguments='{"command":"cd /home/ubuntu && ./pancakes << EOF\\nEOF"}', call_id='call_XXITZmGDUiZYOlMfuK4kmEDl', name='bash', type='function_call', id='fc_0f6e802a28329b2f00696b6dea46d0819d8ff38d87410d1962', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_XXITZmGDUiZYOlMfuK4kmEDl', 'output': '{"result": "/bin/bash: line 2: ./pancakes: Permission denied\\n\\n(exit 126)"}'}
...
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Anthropic](https://platform.claude.com/settings/keys), and set these as environment variables:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
```bash theme={null}
pip install openreward
```
Save this as `quickstart.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="GeneralReasoning/KellyBench")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
message = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
messages.append({
"role": "assistant",
"content": message.content,
})
print(messages[-1])
if message.stop_reason == "tool_use":
tool_uses = [b for b in message.content if getattr(b, "type", None) == "tool_use"]
if not tool_uses:
raise RuntimeError("stop_reason was tool_use but no tool_use blocks found")
tool_result_blocks = []
finished = False
for tu in tool_uses:
tool_name = tu.name
tool_input = tu.input
try:
tr = session.call_tool(tool_name, tool_input)
# Convert OpenReward blocks -> string safely
text_parts = []
for b in getattr(tr, "blocks", []) or []:
t = getattr(b, "text", None)
if t is not None:
text_parts.append(t)
else:
text_parts.append(str(b))
tool_text = "".join(text_parts)
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": tool_text,
})
# Track termination if the env says we're done
if getattr(tr, "finished", False):
finished = True
except Exception as e:
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": f"Tool execution failed: {type(e).__name__}: {e}",
"is_error": True,
})
messages.append({
"role": "user",
"content": tool_result_blocks
})
print(messages[-1])
continue
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': "You are solving a CTF (Capture The Flag) challenge.\n\nChallenge: pancakes\n\nYou ever just get a craving for pancakes? > nc chal.tuctf.com 30503 [pancakes](pancakes) ', 'A' * 44 + p32(e.plt['puts']) + 'AAAA' + p32(e.symbols['password'])) print p.recvall()\n\nYour goal is to find the flag and submit it using the submit_answer tool.\n\nIf challenge files are available, then they have been uploaded to your environment in /tmp/gr-datasets folder.\n\nSometimes you will be given a link to download from or access.\n\nYou should work from the /home/ubuntu directory.\n\nGood luck!"}]
{'role': 'assistant', 'content': [TextBlock(citations=None, text="I'll start by exploring the challenge files and understanding what we're dealing with.", type='text'), ToolUseBlock(id='toolu_01RzBfJy28JV4XT7Wh5Et3j2', input={'path': '/tmp/gr-datasets'}, name='list_files', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01RzBfJy28JV4XT7Wh5Et3j2', 'content': 'total 16\n-rw-r--r-- 1 root root 15696 Jan 15 10:07 pancakes'}]}
{'role': 'assistant', 'content': [ToolUseBlock(id='toolu_01XyLJMhidhzfZyEH1VHRFZN', input={'command': 'cd /home/ubuntu && cp /tmp/gr-datasets/pancakes . && file pancakes'}, name='bash', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01XyLJMhidhzfZyEH1VHRFZN', 'content': 'pancakes: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=a653a608db5ab4716ca7b1e891ee3b460e097eb8, for GNU/Linux 3.2.0, not stripped\n\n(exit 0)'}]}
{'role': 'assistant', 'content': [ToolUseBlock(id='toolu_01CHtUSccRsVBYghg353gwon', input={'command': 'chmod +x /home/ubuntu/pancakes'}, name='bash', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01CHtUSccRsVBYghg353gwon', 'content': '\n\n(exit 0)'}]}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Gemini](https://aistudio.google.com/app/apikey), and set these as environment variables:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
```bash theme={null}
pip install openreward
```
Save this as `quickstart.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=tools)]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": tool_result.blocks[0].text},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text="""You are solving a CTF (Capture The Flag) challenge.
Challenge: pancakes
You ever just get a craving for pancakes? > nc chal.tuctf.com 30503 [pancakes](pancakes) ', 'A' * 44 + p32(e.plt['puts']) + 'AAAA' + p32(e.symbols['password'])) print p.recvall()
Your goal is to find the flag and submit it using the submit_answer tool.
If challenge files are available, then they have been uploaded to your environment in /tmp/gr-datasets folder.
Sometimes you will be given a link to download from or access.
You should work from the /home/ubuntu directory.
Good luck!"""
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'path': '/tmp/gr-datasets'
},
name='list_files'
),
thought_signature=b'\n\xbf\x03\x01r\xc8\xda|k\x19\xc0\xe8\x82I\xb0\x14\xedu\xfa`X\x7f\xd6\xf6\xedO\\\x1f\xfb\x14A\xa8m\x84\xb7\x81\xe3\xc0Q\xe9\xad\xadI\xb1.Y\x80\xad\xeb(\x12\xc3)\xbe\x0f\x95[\x86\x0e\xb2\xb7\xf1\x9b\x8a0t\x90+7\xf0\xe7\x92\x19\xce\xf6\xa1\xe3\x95\xe5\x198\xad.\xea\xf5\x84u\xddv\xcao\xea\x89\xea\x96\xfd\xe8...'
), Part(
function_call=FunctionCall(
args={},
name='list_files'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='list_files',
response={
'result': '{"result": "total 16\\n-rw-r--r-- 1 root root 15696 Jan 15 10:07 pancakes"}'
}
)
)] role='user'
parts=[Part(
function_response=FunctionResponse(
name='list_files',
response={
'result': '{"result": "total 54\\nlrwxrwxrwx 1 root root 7 Jun 10 2025 bin -> usr/bin\\ndrwxr-xr-x 1 root root 4096 May 9 2025 boot\\ndrwxr-xr-x 5 root root 360 Jan 17 11:48 dev\\ndrwxr-xr-x 1 root root 4096 Jan 17 11:48 etc\\ndrwxr-xr-x 1 root root 4096 Jun 25 2025 home\\nlrwxrwxrwx 1 root root 7 Jun 10 2025 lib -> usr/lib\\nlrwxrwxrwx 1 root root 9 Jun 10 2025 lib64 -> usr/lib64\\ndrwxr-xr-x 1 root root 4096 Jun 10 2025 media\\ndrwxr-xr-x 1 root root 4096 Jun 10 2025 mnt\\ndrwxr-xr-x 1 root root 4096 Jun 10 2025 opt\\ndr-xr-xr-x 365 root root 0 Jan 17 11:48 proc\\ndrwx------ 1 root root 4096 Jul 16 2025 root\\ndrwxr-xr-x 1 root root 4096 Jan 17 11:48 run\\nlrwxrwxrwx 1 root root 8 Jun 10 2025 sbin -> usr/sbin\\ndrwxrwxrwx 2 root root 4096 Jan 17 11:48 shared\\ndrwxr-xr-x 1 root root 4096 Jun 10 2025 srv\\ndr-xr-xr-x 13 root root 0 Jan 17 11:48 sys\\ndrwxrwxrwt 1 root root 4096 Jan 17 11:48 tmp\\ndrwxr-xr-x 1 root root 4096 Jun 10 2025 usr\\ndrwxr-xr-x 1 root root 4096 Jun 10 2025 var"}'
}
)
)]
...
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
```
```bash theme={null}
pip install openreward
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
input_list.append({
"role": "assistant",
"content": response.choices[0].message.content,
"tool_calls": response.choices[0].message.tool_calls
})
print(input_list[-1])
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
break
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": tool_result.blocks[0].text
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': "You are solving a CTF (Capture The Flag) challenge.\n\nChallenge: pancakes\n\nYou ever just get a craving for pancakes? > nc chal.tuctf.com 30503 [pancakes](pancakes) ', 'A' * 44 + p32(e.plt['puts']) + 'AAAA' + p32(e.symbols['password'])) print p.recvall()\n\nYour goal is to find the flag and submit it using the submit_answer tool.\n\nIf challenge files are available, then they have been uploaded to your environment in /tmp/gr-datasets folder.\n\nSometimes you will be given a link to download from or access.\n\nYou should work from the /home/ubuntu directory.\n\nGood luck!"}]
{'role': 'assistant', 'content': "I'll start by exploring the environment and understanding the challenge. Let's first check what files are available.\n\n", 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='call_d7ee86af89a74282a53b5688', function=Function(arguments='{"command": "cd /home/ubuntu && ls -la"}', name='bash'), type='function', index=0)]}
{'role': 'tool', 'tool_call_id': 'call_d7ee86af89a74282a53b5688', 'content': '{"result": "total 8\\ndrwxr-xr-x 2 root root 4096 Jun 25 2025 .\\ndrwxr-xr-x 1 root root 4096 Jun 25 2025 ..\\n\\n(exit 0)"}'}
{'role': 'assistant', 'content': '', 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='019bcbd6c70ac06ecf0c1ff215bba68f', function=Function(arguments='{"command": "find /tmp/gr-datasets -type f 2>/dev/null | head -20"}', name='bash'), type='function', index=0)]}
{'role': 'tool', 'tool_call_id': '019bcbd6c70ac06ecf0c1ff215bba68f', 'content': '{"result": "/tmp/gr-datasets/pancakes\\n\\n(exit 0)"}'}
```
If you are running with another provider, or using custom models, then here are the main principles to keep in mind.
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
```bash theme={null}
pip install openreward
```
You'll need to use the OpenReward client to access the environment and its main information
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
```
```python theme={null}
with environment.session(task=example_task) as session:
```
Using a context manager, we can start a session with the environment. This defines a scope in which
you can use the agent to call tools and get tool results.
Above we have selected the first task to sample from, which is a particular problem in the CTF environment.
You can get the prompt for the task as follows:
```python theme={null}
prompt = session.get_prompt()
```
You will also need to pass in the `tools` into the context window of your model, usually somewhere in the system prompt.
An agent will usually keep interacting with an environment until it hits a termination state associated with the environment, or
some other imposed limit (e.g. maximum number of terms).
In a simple sequential agent model, this could be a while loop like:
```python theme={null}
while not finished:
```
where `finished` is a boolean.
In agentic environments, actions are treated as tool calls. That means you need to have a way to parse tool calls from your model's generations.
The key thing to note is that for OpenReward environment, a tool call requires specifying a name (`str`) and some arguments (`dict`).
To call a tool you will call:
```python theme={null}
tool_result = session.call_tool(tool_name, tool_arguments)
```
This means that you will need to parse out the `tool_name` and `tool_arguments` from your model's generation and then parse this information into the
`call_tool` method.
If you have executed an available tool correctly, you will receive a `ToolOutput` output. This contains attributes for:
* `reward` : an (optional) `float` denoting reward. For example, submitting a correct math solution through a `submit_solution` tool might give a reward of `1.0`.
* `finished` : a `bool` specifying whether the episode is finished or not. For example, some tools may end the episode (if for example an agent submits a final answer through `submit_solution` in a math task).
* `data` : a `dict` with output of the executing of the tool. For example, the stdout of executing a `bash` tool. This information should be passed to the agent as feedback.
For our agent loop, some example control flow we might want after receiving a `ToolOutput` is:
* Recording `reward` for use in a policy gradient algorithm such as GRPO or PPO
* Breaking out of the core agent loop if `finished=True`
* Adding `data` to the context window as feedback and continuing with the next model generation
## Next steps
Now that you know how to sample from OpenReward environments, explore these key features:
Learn how to run evals with OpenReward
Build your first environment
Getting the most out of OpenReward
# Recording Rollouts
Source: https://docs.openreward.ai/rollouts/recording-rollouts
Record and upload agent rollouts to OpenReward for viewing and analysis
## Goals
* Understand what rollouts are and why you would record them.
* Record rollouts from an agent interacting with an OpenReward environment.
* Use provider-specific logging methods for OpenAI, Anthropic, Google Gemini, and OpenRouter.
* View recorded rollouts in the OpenReward dashboard.
## Prerequisites
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
* Familiarity with the [Your First Environment](/environments/your-first-environment) tutorial
## Setup
Install the OpenReward Python library:
```bash theme={null}
pip install openreward
```
## What are Rollouts?
A **rollout** is a recorded trace of an agent interacting with an environment. It captures every message exchange -- user prompts, assistant responses, reasoning steps, tool calls, and tool results -- along with associated rewards and metadata.
Recording rollouts is useful for:
* **Debugging**: Inspect exactly what your agent did step-by-step.
* **Analysis**: Compare performance across models, prompts, or environment configurations.
* **Training data**: Collect trajectories for supervised finetuning or midtraining.
* **Collaboration**: Your team can view runs on your organisation's OpenReward page.
The OpenReward Python SDK provides provider-specific logging methods that automatically serialize messages from OpenAI, Anthropic, and Google Gemini into a normalized format. For OpenRouter (which uses the OpenAI SDK), you use the OpenAI completions logger.
## Key Concepts
### Runs and Rollouts
A **run** groups related rollouts together under a single name (e.g. `"gpt-5.4-ctf-eval"`). Each **rollout** within a run represents one episode or conversation with the environment.
### Creating a Rollout
```python theme={null}
import openreward
or_client = openreward.OpenReward()
rollout = or_client.rollout.create(
run_name="my-eval-run", # Required: groups rollouts together
rollout_name="episode-1", # Optional: name for this specific rollout
environment="GeneralReasoning/CTF", # Optional: environment identifier
split="train", # Optional: data split
metadata={"model": "gpt-5.4"}, # Optional: custom key-value pairs
task_spec={"id": "task-0"} # Optional: task specification
)
```
### Logging Methods
The SDK provides these logging methods on a `Rollout` object:
| Method | Provider | Input Type |
| ----------------------------------------- | -------------------------------------- | --------------------------------------------------- |
| `rollout.log_openai_response(message)` | OpenAI (Responses API) | Response object or individual output items |
| `rollout.log_openai_completions(message)` | OpenAI (Chat Completions) / OpenRouter | Chat message dicts |
| `rollout.log_anthropic_message(message)` | Anthropic | Message dicts (Anthropic format) |
| `rollout.log_gdm_message(message)` | Google Gemini | `google.genai.types.Content` objects |
| `rollout.log(message)` | Generic | `UserMessage`, `AssistantMessage`, `ToolCall`, etc. |
Each method also accepts these parameters:
* `reward` (`Optional[float]`): Reward signal for this step
* `is_finished` (`Optional[bool]`): Whether this step ends the episode
* `metadata` (`Optional[dict]`): Arbitrary metadata for this step
### Flushing Events
When you are done logging, call `or_client.rollout.close()` to ensure all pending events are flushed and uploaded:
```python theme={null}
or_client.rollout.close()
```
## Recording Rollouts by Provider
In this example we will sample the `GeneralReasoning/CTF` environment and record the full rollout trace to OpenReward.
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenAI](https://platform.openai.com/api-keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `record_rollout.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
# Create a rollout to record this episode
rollout = or_client.rollout.create(
run_name="openai-ctf-rollouts",
rollout_name="episode-0",
environment="GeneralReasoning/CTF",
split="train",
metadata={"model": MODEL_NAME},
task_spec=example_task.task_spec
)
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
# Log the initial user prompt
rollout.log_openai_response(input_list[0])
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
# Log the full model response (handles reasoning, text, tool calls)
rollout.log_openai_response(response)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
tool_output = {
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps({
"result": tool_result.blocks[0].text
})
}
input_list.append(tool_output)
# Log the tool result with reward info
rollout.log_openai_response(
tool_output,
reward=reward,
is_finished=finished
)
if finished:
break
# Flush all pending events
or_client.rollout.close()
print("Rollout recorded successfully!")
```
```bash theme={null}
python record_rollout.py
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Anthropic](https://platform.claude.com/settings/keys), and set these as environment variables:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `record_rollout.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
# Create a rollout to record this episode
rollout = or_client.rollout.create(
run_name="anthropic-ctf-rollouts",
rollout_name="episode-0",
environment="GeneralReasoning/CTF",
split="train",
metadata={"model": MODEL_NAME},
task_spec=example_task.task_spec
)
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
# Log the initial user prompt
rollout.log_anthropic_message(messages[0])
while not finished:
response = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
assistant_msg = {"role": "assistant", "content": response.content}
messages.append(assistant_msg)
# Log the assistant response (handles text, thinking, tool_use blocks)
rollout.log_anthropic_message(assistant_msg)
if response.stop_reason == "tool_use":
tool_use = next(block for block in response.content if block.type == "tool_use")
tool_name = tool_use.name
tool_input = tool_use.input
tool_result = session.call_tool(tool_name, tool_input)
reward = tool_result.reward
finished = tool_result.finished
user_msg = {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": tool_result.blocks[0].text
}
]
}
messages.append(user_msg)
# Log the tool result with reward info
rollout.log_anthropic_message(
user_msg,
reward=reward,
is_finished=finished
)
if finished:
break
else:
break
# Flush all pending events
or_client.rollout.close()
print("Rollout recorded successfully!")
```
```bash theme={null}
python record_rollout.py
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Gemini](https://aistudio.google.com/app/apikey), and set these as environment variables:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `record_rollout.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=tools)]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
# Create a rollout to record this episode
rollout = or_client.rollout.create(
run_name="gemini-ctf-rollouts",
rollout_name="episode-0",
environment="GeneralReasoning/CTF",
split="train",
metadata={"model": MODEL_NAME},
task_spec=example_task.task_spec
)
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
# Log the initial user prompt
rollout.log_gdm_message(contents[0])
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
model_content = response.candidates[0].content
contents.append(model_content)
# Log the model response (handles text, thinking, function calls)
rollout.log_gdm_message(model_content)
for part in model_content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": json.dumps({
"result": tool_result.blocks[0].text
})},
)
user_content = types.Content(role="user", parts=[function_response_part])
contents.append(user_content)
# Log the tool result with reward info
rollout.log_gdm_message(
user_content,
reward=reward,
is_finished=finished
)
if finished:
break
# Flush all pending events
or_client.rollout.close()
print("Rollout recorded successfully!")
```
```bash theme={null}
python record_rollout.py
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Save this as `record_rollout.py`.
Since OpenRouter uses the OpenAI SDK (Chat Completions format), we use `log_openai_completions` for logging:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
# Create a rollout to record this episode
rollout = or_client.rollout.create(
run_name="openrouter-ctf-rollouts",
rollout_name="episode-0",
environment="GeneralReasoning/CTF",
split="train",
metadata={"model": MODEL_NAME},
task_spec=example_task.task_spec
)
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
# Log the initial user prompt
rollout.log_openai_completions(input_list[0])
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
assistant_msg = response.choices[0].message
input_list.append(assistant_msg)
# Log the assistant response (handles text + tool calls)
rollout.log_openai_completions({
"role": "assistant",
"content": assistant_msg.content,
"tool_calls": assistant_msg.tool_calls
})
tool_calls = assistant_msg.tool_calls
if not tool_calls:
break
for tool_call in tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
tool_msg = {
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
}
input_list.append(tool_msg)
# Log the tool result with reward info
rollout.log_openai_completions(
tool_msg,
reward=reward,
is_finished=finished
)
if finished:
break
# Flush all pending events
or_client.rollout.close()
print("Rollout recorded successfully!")
```
```bash theme={null}
python record_rollout.py
```
If you are using a different provider, a custom model, or want full control over what gets logged, you can use the generic `rollout.log()` method with the built-in message types.
### Generic Message Types
The SDK exports the following types from the `openreward` package:
| Type | Fields | Description |
| ------------------ | -------------------------------------------------------------------------------------- | ----------------------------------- |
| `UserMessage` | `content: str` | A user/human message |
| `AssistantMessage` | `content: str` | An assistant/model response |
| `SystemMessage` | `content: str` | A system prompt |
| `ReasoningItem` | `content: Optional[str]`, `content_reference: Optional[str]`, `summary: Optional[str]` | Hidden reasoning / chain-of-thought |
| `ToolCall` | `name: str`, `content: str`, `call_id: str` | A tool/function invocation |
| `ToolResult` | `content: str`, `call_id: str` | The result of a tool call |
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
Set any additional API keys required by your model provider.
Save this as `record_rollout.py`:
```python theme={null}
from openreward import (
OpenReward,
UserMessage,
AssistantMessage,
SystemMessage,
ReasoningItem,
ToolCall,
ToolResult,
)
import json
or_client = OpenReward()
environment = or_client.environments.get(name="GeneralReasoning/CTF")
tasks = environment.list_tasks(split="train")
example_task = tasks[0]
# Create a rollout to record this episode
rollout = or_client.rollout.create(
run_name="custom-ctf-rollouts",
rollout_name="episode-0",
environment="GeneralReasoning/CTF",
split="train",
metadata={"model": "my-custom-model"},
task_spec=example_task.task_spec
)
with environment.session(task=example_task) as session:
prompt = session.get_prompt()
finished = False
# Log the user prompt
rollout.log(UserMessage(content=prompt[0].text))
while not finished:
# --- Replace this section with your model's inference ---
model_response = "The answer is 42."
# --------------------------------------------------------
# Log the assistant response
rollout.log(AssistantMessage(content=model_response))
# If your model produces reasoning/thinking, log it:
# rollout.log(ReasoningItem(content="Let me think about this..."))
# If your model makes a tool call, log the call and result:
tool_name = "submit_answer"
tool_args = {"answer": model_response}
call_id = "call_001"
rollout.log(ToolCall(
name=tool_name,
content=json.dumps(tool_args),
call_id=call_id
))
tool_result = session.call_tool(tool_name, tool_args)
rollout.log(
ToolResult(
content=tool_result.blocks[0].text,
call_id=call_id
),
reward=tool_result.reward,
is_finished=tool_result.finished
)
finished = tool_result.finished
# Flush all pending events
or_client.rollout.close()
print("Rollout recorded successfully!")
```
```bash theme={null}
python record_rollout.py
```
## Viewing Rollouts
After running your code, head to [OpenReward](https://openreward.ai) and your profile page. Go to the **Runs** tab to see your recorded runs:
Click on a run to see its rollouts:
Click on a specific rollout to inspect the full message timeline, including reasoning steps, tool calls, and rewards:
## Next Steps
Use recorded rollouts as part of a reinforcement learning training pipeline.
Run systematic evaluations across environments and models.
Create custom environments for your use case.
Record rollouts asynchronously for better performance.
# Using Daytona Sandboxes
Source: https://docs.openreward.ai/sandboxes/daytona
Make and deploy an ORS environment with Daytona Sandboxes
## Goals
* Make a mathematics code execution environment using OpenReward and Daytona.
* Deploy the environment to OpenReward.
* Sample from the environment using a model of your choice.
## Prerequisites
* You have completed the [Your First Environment](../environments/your-first-environment) tutorial
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* A Daytona [API key](https://app.daytona.io/dashboard/keys)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
## Setup
Environments in OpenReward are written using [ORS](https://openrewardstandard.io).
ORS is implemented in the [OpenReward Python library](https://www.github.com/OpenReward/openreward-python), and we will use it for this tutorial.
You can install the library using pip or uv:
```bash theme={null}
pip install openreward
```
You should also install the `daytona` Python SDK:
```bash theme={null}
pip install daytona
```
## Introduction
[ORS](https://openrewardstandard.io) environments can be configured to work with any sandbox provider. In this tutorial we will show how to initialise a Daytona sandbox within an ORS environment. We'll setup a simple mathematics environment and give an agent access to a sandbox for executing code. By the end of this tutorial, you will understand how to use Daytona as an integration for making environments with ORS and OpenReward.
## Getting Started
In the [Your First Environment](../environments/your-first-environment) tutorial, we built a mathematics environment using the [GSM8K](https://arxiv.org/abs/2110.14168) dataset.
We will use a similar dataset here, but this time we will give an agent an access to a Daytona sandbox for code execution.
First, let's initialise our environment `gsm8ksandbox`:
```bash theme={null}
orwd init gsm8ksandbox --template basic
cd gsm8k && ls
```
Next we'll download the two `parquet` files from the GSM8K [HuggingFace repository](https://huggingface.co/datasets/openai/gsm8k/tree/main/main) and put them in the root of our project:
```
test-00000-of-00001.parquet
train-00000-of-00001.parquet
```
A single row of data from the train set looks as follows:
```
{'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72', 'id': '0'}
```
Next we'll write a new server file. This will involve:
* Loading the tasks from the `parquet` files
* Verifying the answer is correct - we'll use the [MathVerify](https://github.com/huggingface/Math-Verify) library for this.
```python theme={null}
from math_verify import parse, verify
import pandas as pd
from pydantic import BaseModel
from typing import Optional
from daytona import Daytona, DaytonaConfig
from openreward.environments import Environment, JSONObject, Server, Split, TextBlock, ToolOutput, tool
class GSM8KTaskSpec(BaseModel):
id: str
question: str
answer: str
class AnswerParams(BaseModel):
answer: str
train_tasks = pd.read_parquet("train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("test-00000-of-00001.parquet").to_dict(orient="records")
for i, task in enumerate(train_tasks):
task['id'] = str(i)
for i, task in enumerate(test_tasks):
task['id'] = str(i)
class GSM8KSandbox(Environment):
"""
A GSM8K sandbox environment
"""
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = GSM8KTaskSpec.model_validate(task_spec)
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
if split == "train":
return train_tasks
elif split == "test":
return test_tasks
raise ValueError(f"Unknown split: {split}")
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
def get_prompt(self) -> str:
return [TextBlock(type="text", text=self.config.question)]
@tool
def answer(self, params: AnswerParams) -> ToolOutput:
"""
The answer tool can be used to submit your final answer. Note that this finishes the episode.
"""
gold = parse(self.config.answer)
answer = parse(params.answer)
is_correct = verify(gold, answer)
if is_correct:
agent_message = "Correct!"
reward = 1.0
else:
agent_message = "Wrong!"
reward = 0.0
return ToolOutput(
blocks=[TextBlock(type="text", text=agent_message)],
reward=reward,
finished=True
)
if __name__ == "__main__":
Server([GSM8KSandbox]).run()
```
Install the `math-verify` requirement:
```bash theme={null}
pip install math-verify
```
Now we are going to modify our environment by including a Daytona sandbox. First, we'll adjust our `__init__`:
```python theme={null}
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = GSM8KTaskSpec.model_validate(task_spec)
self.api_key = secrets.get("daytona_api_key")
if not self.api_key:
raise ValueError("Daytona API key must be provided via secrets parameter")
self.sandbox = None
```
Next, we'll add a `setup` and `teardown` method:
```python theme={null}
def setup(self) -> None:
daytona = Daytona(DaytonaConfig(api_key=self.api_key))
self.sandbox = daytona.create()
def teardown(self) -> None:
if self.sandbox is not None:
self.sandbox.delete()
```
These methods ensure that when a session begins, we create a sandbox, and when a session ends, we delete it.
Lastly, we can add a tool that relies on code execution. We'll add a `bash` tool. First we'll add the Pydantic model:
```python theme={null}
class BashParams(BaseModel, extra="forbid"):
command: str
timeout: Optional[float] = 30.0
```
and then we'll add the `bash` tool to the class:
```python theme={null}
@tool
def bash(self, params: BashParams) -> ToolOutput:
"""Execute bash commands using the computer instance."""
try:
response = self.sandbox.process.exec(params.command.strip())
result_dict = {
"stdout": response.result if response.exit_code == 0 else "",
"stderr": response.result if response.exit_code != 0 else "",
"exit_code": response.exit_code,
}
# What the model sees as tool text output
text_out = result_dict["stdout"] or result_dict["stderr"] or str(result_dict)
return ToolOutput(
blocks=[TextBlock(type="text", text=text_out)],
metadata={"output": result_dict}, # JSON-safe now
reward=0.0,
finished=False,
)
except Exception as e:
return ToolOutput(
metadata={"error": str(e)},
blocks=[TextBlock(text=f"Error executing command: {str(e)}")],
finished=False
)
```
Now we can test this environment. First run the server as before:
```bash theme={null}
python server.py
```
Now choose a model provider of your choice and sample from the environment:
Make sure you have an API key for [OpenAI](https://platform.openai.com/api-keys), and set the environment variable:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export DAYTONA_API_KEY='your-daytona-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import os
import json
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"daytona_api_key": os.getenv("DAYTONA_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
[ResponseFunctionToolCall(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april/2\\ntotal=april+may\\nprint(int(total))\\nPY","timeout":100000}', call_id='call_kshkbrFlQtJcnu7pAR0x4aGH', name='bash', type='function_call', id='fc_0a2052fb4b543cf400699aeb18dc288192b0d106902d87d69b', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_kshkbrFlQtJcnu7pAR0x4aGH', 'output': '{"result": "72"}'}
[ResponseOutputMessage(id='msg_0a2052fb4b543cf400699aeb1b3c78819289bfc6c76a734ee3', content=[ResponseOutputText(annotations=[], text='Natalia sold 48 clips in April. In May, she sold half as many: \\(48 \\div 2 = 24\\). \nAltogether, she sold \\(48 + 24 = 72\\) clips.', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"Natalia sold 48 clips in April. In May, she sold half as many: \\\\(48 \\\\div 2 = 24\\\\). \\nAltogether, she sold \\\\(48 + 24 = 72\\\\) clips."}', call_id='call_zC6RbMlTkH7Ws0OnZeQCPi9t', name='answer', type='function_call', id='fc_0a2052fb4b543cf400699aeb1cfd7c8192a72ecf312a5d2440', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_zC6RbMlTkH7Ws0OnZeQCPi9t', 'output': '{"result": "Correct!"}'}
```
Make sure you have API key for [Anthropic](https://platform.claude.com/settings/keys), and set the environment variable:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export DAYTONA_API_KEY='your-daytona-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
import os
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"daytona_api_key": os.getenv("DAYTONA_API_KEY")}) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
message = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
messages.append({
"role": "assistant",
"content": message.content,
})
print(messages[-1])
if message.stop_reason == "tool_use":
tool_uses = [b for b in message.content if getattr(b, "type", None) == "tool_use"]
if not tool_uses:
raise RuntimeError("stop_reason was tool_use but no tool_use blocks found")
tool_result_blocks = []
finished = False
for tu in tool_uses:
tool_name = tu.name
tool_input = tu.input
try:
tr = session.call_tool(tool_name, tool_input)
# Convert OpenReward blocks -> string safely
text_parts = []
for b in getattr(tr, "blocks", []) or []:
t = getattr(b, "text", None)
if t is not None:
text_parts.append(t)
else:
text_parts.append(str(b))
tool_text = "".join(text_parts)
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": tool_text,
})
# Track termination if the env says we're done
if getattr(tr, "finished", False):
finished = True
except Exception as e:
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": f"Tool execution failed: {type(e).__name__}: {e}",
"is_error": True,
})
messages.append({
"role": "user",
"content": tool_result_blocks
})
print(messages[-1])
continue
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': [TextBlock(text="I'll help you solve this math problem using the bash tool to calculate the answer.", type='text'), ToolUseBlock(id='toolu_01XbashToolExample123', input={'command': 'python3 - << \'PY\'\napril=48\nmay=april/2\ntotal=april+may\nprint(int(total))\nPY', 'timeout': 100000}, name='bash', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01XbashToolExample123', 'content': '72'}]}
{'role': 'assistant', 'content': [TextBlock(text='Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': 'Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.'}, name='answer', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}
```
Make sure you have an API key for [Gemini](https://aistudio.google.com/app/apikey) and set it as an environment variable:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export DAYTONA_API_KEY='your-daytona-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
import os
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=tools)]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task, secrets={"daytona_api_key": os.getenv("DAYTONA_API_KEY")}) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": json.dumps({
"result": tool_result.blocks[0].text
})},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'command': "python3 - << 'PY'\napril=48\nmay=april/2\ntotal=april+may\nprint(int(total))\nPY",
'timeout': 100000
},
name='bash'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='bash',
response={
'result': '{"result": "72"}'
}
)
)] role='user'
parts=[Part(
text='Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.'
)] role='model'
parts=[Part(
function_call=FunctionCall(
args={
'answer': 'Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.'
},
name='answer'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='answer',
response={
'result': '{"result": "Correct!"}'
}
)
)] role='user'
```
Make sure you have an API key for [OpenRouter](https://openrouter.ai/keys) and set it as an environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export DAYTONA_API_KEY='your-daytona-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"daytona_api_key": os.getenv("DAYTONA_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
input_list.append({
"role": "assistant",
"content": response.choices[0].message.content,
"tool_calls": response.choices[0].message.tool_calls
})
print(input_list[-1])
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
break
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': None, 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='call_bash_example_001', function=Function(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april/2\\ntotal=april+may\\nprint(int(total))\\nPY","timeout":100000}', name='bash'), type='function')]}
{'role': 'tool', 'tool_call_id': 'call_bash_example_001', 'content': '{"result": "72"}'}
{'role': 'assistant', 'content': 'Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.', 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \\nAltogether, she sold 48 + 24 = 72 clips."}', name='answer'), type='function')]}
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': '{"result": "Correct!"}'}
```
Nice one! We have a working ORS environment.
Now we'll see how we can host the environment on OpenReward. The benefits of using OpenReward are:
* **Infrastructure**: you do not have to set up infrastructure and compute to host the environment yourself. We take care of this and you are only charged based on your actual usage of the environment.
* **Discovery**: your environment can be discovered and used by other users of the platform, helping drive adoption and attention to your work.
## Host on OpenReward
Log into [OpenReward](https://openreward.ai), press the plus icon in the navbar and press **New Environment**:
Next, fill in information about the environment and press **Create Environment**:
You will be redirected to your new environment and will see setup instructions:
### Upload environment files
We will need a way to use the train and test parquet files in our environment. We'll upload these to the environment files:
Click on the **Files** tab and upload each file:
Files are mounted to the environment server at the `/orwd_data` directory.
We'll need to reference this folder in our `server.py`. Make the following change:
```python theme={null}
train_tasks = pd.read_parquet("/orwd_data/train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("/orwd_data/test-00000-of-00001.parquet").to_dict(orient="records")
```
Note: you may want to set an environment variable instead of hardcoding like above so you can continue to test
locally (without the `/orwd_data` prefix).
### Write the Dockerfile and requirements
We'll need a `Dockerfile` in our repository:
```txt theme={null}
FROM python:3.11-slim
RUN apt update && apt upgrade -y && apt install -y \
curl
WORKDIR /app
# Copy requirements and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application code
COPY server.py .
# Expose port
EXPOSE 8000
# Start the server
CMD ["python", "server.py"]
```
We'll need to update `requirements.txt`:
```txt theme={null}
fastapi>=0.115.12
openreward
pandas
pyarrow
uvicorn>=0.34.3
math-verify[antlr4_13_2]
daytona
```
### Push to GitHub and connect
Next, [push your environment code to a GitHub repository](https://github.com/new).
Once your GitHub repository is ready, go to your OpenReward environment and connect the repository:
You will be given a choice for how much compute you would like to allocate for it. We'll use a low compute configuration since this is a simple environment.
Press Connect GitHub and your first build will begin.
To check the progress of the build, click the Deployments tab:
You can click on the latest build row to see logs:
The build logs show the progress of building the environment. The runtime logs show any calls to the environment server, and can be useful for diagnosing errors.
### Sample from your environment
Now your environment is hosted on OpenReward, we can sample from it:
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenAI](https://platform.openai.com/api-keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export DAYTONA_API_KEY='your-daytona-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"daytona_api_key": os.getenv("DAYTONA_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
[ResponseFunctionToolCall(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april/2\\ntotal=april+may\\nprint(int(total))\\nPY","timeout":100000}', call_id='call_kshkbrFlQtJcnu7pAR0x4aGH', name='bash', type='function_call', id='fc_0a2052fb4b543cf400699aeb18dc288192b0d106902d87d69b', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_kshkbrFlQtJcnu7pAR0x4aGH', 'output': '{"result": "72"}'}
[ResponseOutputMessage(id='msg_0a2052fb4b543cf400699aeb1b3c78819289bfc6c76a734ee3', content=[ResponseOutputText(annotations=[], text='Natalia sold 48 clips in April. In May, she sold half as many: \\(48 \\div 2 = 24\\). \nAltogether, she sold \\(48 + 24 = 72\\) clips.', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"Natalia sold 48 clips in April. In May, she sold half as many: \\\\(48 \\\\div 2 = 24\\\\). \\nAltogether, she sold \\\\(48 + 24 = 72\\\\) clips."}', call_id='call_zC6RbMlTkH7Ws0OnZeQCPi9t', name='answer', type='function_call', id='fc_0a2052fb4b543cf400699aeb1cfd7c8192a72ecf312a5d2440', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_zC6RbMlTkH7Ws0OnZeQCPi9t', 'output': '{"result": "Correct!"}'}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Anthropic](https://platform.claude.com/settings/keys), and set these as environment variables:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export DAYTONA_API_KEY='your-daytona-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
import os
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"daytona_api_key": os.getenv("DAYTONA_API_KEY")}) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
response = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
print(message)
messages.append(message)
if message.stop_reason == "tool_use":
tool_use = next(block for block in message.content if block.type == "tool_use")
tool_name = tool_use.name
tool_input = tool_use.input
tool_result = session.call_tool(tool_name, tool_input)
reward = tool_result.reward
finished = tool_result.finished
messages.append({
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": tool_result.blocks[0].text
}
]
})
print(messages[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': [TextBlock(text="I'll help you solve this math problem using the bash tool to calculate the answer.", type='text'), ToolUseBlock(id='toolu_01XbashToolExample123', input={'command': 'python3 - << \'PY\'\napril=48\nmay=april/2\ntotal=april+may\nprint(int(total))\nPY', 'timeout': 100000}, name='bash', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01XbashToolExample123', 'content': '72'}]}
{'role': 'assistant', 'content': [TextBlock(text='Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': 'Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.'}, name='answer', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Gemini](https://aistudio.google.com/app/apikey), and set these as environment variables:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export DAYTONA_API_KEY='your-daytona-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
import os
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=[f]) for f in tools]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task, secrets={"daytona_api_key": os.getenv("DAYTONA_API_KEY")}) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": json.dumps({
"result": tool_result.blocks[0].text
})},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'command': "python3 - << 'PY'\napril=48\nmay=april/2\ntotal=april+may\nprint(int(total))\nPY",
'timeout': 100000
},
name='bash'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='bash',
response={
'result': '{"result": "72"}'
}
)
)] role='user'
parts=[Part(
text='Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.'
)] role='model'
parts=[Part(
function_call=FunctionCall(
args={
'answer': 'Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.'
},
name='answer'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='answer',
response={
'result': '{"result": "Correct!"}'
}
)
)] role='user'
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export DAYTONA_API_KEY='your-daytona-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"daytona_api_key": os.getenv("DAYTONA_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
print(response)
input_list.append(response)
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
return
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': None, 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='call_bash_example_001', function=Function(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april/2\\ntotal=april+may\\nprint(int(total))\\nPY","timeout":100000}', name='bash'), type='function')]}
{'role': 'tool', 'tool_call_id': 'call_bash_example_001', 'content': '{"result": "72"}'}
{'role': 'assistant', 'content': 'Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips.', 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "Natalia sold 48 clips in April. In May, she sold half as many: 48 ÷ 2 = 24. \\nAltogether, she sold 48 + 24 = 72 clips."}', name='answer'), type='function')]}
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': '{"result": "Correct!"}'}
```
If you are running with another provider, or using custom models, then here are the main principles to keep in mind.
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
You'll need to use the OpenReward client to access the environment and its main information
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
```
```python theme={null}
with environment.session(task=example_task) as session:
```
Using a context manager, we can start a session with the environment. This defines a scope in which
you can use the agent to call tools and get tool results.
Above we have selected the first task to sample from, which is a particular problem in the CTF environment.
You can get the prompt for the task as follows:
```python theme={null}
prompt = session.get_prompt()
```
You will also need to pass in the `tools` into the context window of your model, usually somewhere in the system prompt.
An agent will usually keep interacting with an environment until it hits a termination state associated with the environment, or
some other imposed limit (e.g. maximum number of terms).
In a simple sequential agent model, this could be a while loop like:
```python theme={null}
while not finished:
```
where `finished` is a boolean.
In agentic environments, actions are treated as tool calls. That means you need to have a way to parse tool calls from your model's generations.
The key thing to note is that for OpenReward environment, a tool call requires specifying a name (`str`) and some arguments (`dict`).
To call a tool you will call:
```python theme={null}
tool_result = session.call_tool(tool_name, tool_arguments)
```
This means that you will need to parse out the `tool_name` and `tool_arguments` from your model's generation and then parse this information into the
`call_tool` method.
If you have executed an available tool correctly, you will receive a `ToolOutput` output. This contains attributes for:
* `reward` : an (optional) `float` denoting reward. For example, submitting a correct math solution through a `submit_solution` tool might give a reward of `1.0`.
* `finished` : a `bool` specifying whether the episode is finished or not. For example, some tools may end the episode (if for example an agent submits a final answer through `submit_solution` in a math task).
* `data` : a `dict` with output of the executing of the tool. For example, the stdout of executing a `bash` tool. This information should be passed to the agent as feedback.
For our agent loop, some example control flow we might want after receiving a `ToolOutput` is:
* Recording `reward` for use in a policy gradient algorithm such as GRPO or PPO
* Breaking out of the core agent loop if `finished=True`
* Adding `data` to the context window as feedback and continuing with the next model generation
Importantly, if you are publishing your environment, you should update your SDK example and let users know that your environment needs a `DAYTONA_API_KEY`. Press Edit Settings near the SDK example then insert the code for environment variables:
# Using E2B Sandboxes
Source: https://docs.openreward.ai/sandboxes/e2b
Make and deploy an ORS environment with E2B Sandboxes
## Goals
* Make a mathematics code execution environment using OpenReward and E2B.
* Deploy the environment to OpenReward.
* Sample from the environment using a model of your choice.
## Prerequisites
* You have completed the [Your First Environment](../environments/your-first-environment) tutorial
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* An E2B [API key](https://e2b.dev/dashboard/)
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
## Setup
Environments in OpenReward are written using [ORS](https://openrewardstandard.io).
ORS is implemented in the [OpenReward Python library](https://www.github.com/OpenReward/openreward-python), and we will use it for this tutorial.
You can install the library using pip or uv:
```bash theme={null}
pip install openreward
```
You should also install the `e2b` Python SDK:
```bash theme={null}
pip install e2b
```
\## Introduction
[ORS](https://openrewardstandard.io) environments can be configured to work with any sandbox provider. In this tutorial we will show how to initialise an E2B sandbox within an ORS environment. We'll setup a simple mathematics environment and give an agent access to a sandbox for executing code. By the end of this tutorial, you will understand how to use E2B as an integration for making environments with ORS and OpenReward.
## Getting Started
In the [Your First Environment](../environments/your-first-environment) tutorial, we built a mathematics environment using the [GSM8K](https://arxiv.org/abs/2110.14168) dataset.
We will use a similar dataset here, but this time we will give an agent an access to an E2B sandbox for code execution.
First, let's initialise our environment `gsm8ksandbox`:
```bash theme={null}
orwd init gsm8ksandbox --template basic
cd gsm8k && ls
```
Next we'll download the two `parquet` files from the GSM8K [HuggingFace repository](https://huggingface.co/datasets/openai/gsm8k/tree/main/main) and put them in the root of our project:
```
test-00000-of-00001.parquet
train-00000-of-00001.parquet
```
A single row of data from the train set looks as follows:
```
{'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72', 'id': '0'}
```
Next we'll write a new server file. This will involve:
* Loading the tasks from the `parquet` files
* Verifying the answer is correct - we'll use the [MathVerify](https://github.com/huggingface/Math-Verify) library for this.
```python theme={null}
from math_verify import parse, verify
import pandas as pd
from pydantic import BaseModel
from typing import Optional
from e2b import AsyncSandbox
from openreward.environments import Environment, JSONObject, Server, Split, TextBlock, ToolOutput, tool
class GSM8KTaskSpec(BaseModel):
id: str
question: str
answer: str
class AnswerParams(BaseModel):
answer: str
train_tasks = pd.read_parquet("train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("test-00000-of-00001.parquet").to_dict(orient="records")
for i, task in enumerate(train_tasks):
task['id'] = str(i)
for i, task in enumerate(test_tasks):
task['id'] = str(i)
class GSM8KSandbox(Environment):
"""
A GSM8K sandbox environment
"""
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = GSM8KTaskSpec.model_validate(task_spec)
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
if split == "train":
return train_tasks
elif split == "test":
return test_tasks
raise ValueError(f"Unknown split: {split}")
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
def get_prompt(self) -> str:
return [TextBlock(type="text", text=self.config.question)]
@tool
def answer(self, params: AnswerParams) -> ToolOutput:
"""
The answer tool can be used to submit your final answer. Note that this finishes the episode.
"""
gold = parse(self.config.answer)
answer = parse(params.answer)
is_correct = verify(gold, answer)
if is_correct:
agent_message = "Correct!"
reward = 1.0
else:
agent_message = "Wrong!"
reward = 0.0
return ToolOutput(
blocks=[TextBlock(type="text", text=agent_message)],
reward=reward,
finished=True
)
if __name__ == "__main__":
Server([GSM8KSandbox]).run()
```
Install the `math-verify` requirement:
```bash theme={null}
pip install math-verify
```
Now we are going to modify our environment by including an E2B sandbox. First, we'll adjust our `__init__`:
```python theme={null}
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = GSM8KTaskSpec.model_validate(task_spec)
self.api_key = secrets.get("e2b_api_key")
if not self.api_key:
raise ValueError("E2B API key must be provided via secrets parameter")
self.sandbox = None
```
Next, we'll add a `setup` and `teardown` method:
```python theme={null}
async def setup(self) -> None:
self.sandbox = await AsyncSandbox.create(
api_key=self.api_key,
timeout=60 * 60, # 1 hour
)
async def teardown(self) -> None:
if self.sandbox is not None:
await self.sandbox.kill()
```
These methods ensure that when a session begins, we create a sandbox, and when a session ends, we kill it.
Lastly, we can add a tool that relies on code execution. We'll add a `bash` tool. First we'll add the Pydantic model:
```python theme={null}
class BashParams(BaseModel, extra="forbid"):
command: str
timeout: Optional[float] = 30.0
```
and then we'll add the `bash` tool to the class:
```python theme={null}
@tool
async def bash(self, params: BashParams) -> ToolOutput:
"""Execute bash commands using the computer instance."""
try:
result = await self.sandbox.commands.run(params.command.strip(), timeout=params.timeout)
result_dict = {
"stdout": getattr(result, "stdout", ""),
"stderr": getattr(result, "stderr", ""),
"exit_code": getattr(result, "exit_code", None),
"error": getattr(result, "error", None),
}
# What the model sees as tool text output
text_out = result_dict["stdout"] or result_dict["stderr"] or str(result_dict)
return ToolOutput(
blocks=[TextBlock(type="text", text=text_out)],
metadata={"output": result_dict}, # JSON-safe now
reward=0.0,
finished=False,
)
except Exception as e:
return ToolOutput(
metadata={"error": str(e)},
blocks=[TextBlock(text=f"Error executing command: {str(e)}")],
finished=False
)
```
Now we can test this environment. First run the server as before:
```bash theme={null}
python server.py
```
Now choose a model provider of your choice and sample from the environment:
Make sure you have an API key for [OpenAI](https://platform.openai.com/api-keys), and set the environment variable:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export E2B_API_KEY='your-e2b-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import os
import json
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"e2b_api_key": os.getenv("E2B_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
[ResponseFunctionToolCall(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april/2\\nprint(april+may)\\nPY","timeout":100000}', call_id='call_TTS3bI1XFo0Gl88If18JZZ6l', name='bash', type='function_call', id='fc_04efdcdc3efb56c300699ada2d1fac81908551b7ab626fffd2', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_TTS3bI1XFo0Gl88If18JZZ6l', 'output': '{"result": "72.0\\n"}'}
[ResponseOutputMessage(id='msg_04efdcdc3efb56c300699ada2f46d48190a6234d05fdc92310', content=[ResponseOutputText(annotations=[], text='Natalia sold 48 clips in April. In May she sold half of that: \\(48 \\div 2 = 24\\).\n\nAltogether, she sold \\(48 + 24 = 72\\) clips in April and May.', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24. Altogether, she sold 48 + 24 = 72 clips."}', call_id='call_QPFUtfAviwYsozYSKadq78gZ', name='answer', type='function_call', id='fc_04efdcdc3efb56c300699ada30ddfc81909c7bd18e1398f223', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_QPFUtfAviwYsozYSKadq78gZ', 'output': '{"result": "Correct!"}'}
```
Make sure you have API key for [Anthropic](https://platform.claude.com/settings/keys), and set the environment variable:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export E2B_API_KEY='your-e2b-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
import os
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"e2b_api_key": os.getenv("E2B_API_KEY")}) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
message = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
messages.append({
"role": "assistant",
"content": message.content,
})
print(messages[-1])
if message.stop_reason == "tool_use":
tool_uses = [b for b in message.content if getattr(b, "type", None) == "tool_use"]
if not tool_uses:
raise RuntimeError("stop_reason was tool_use but no tool_use blocks found")
tool_result_blocks = []
finished = False
for tu in tool_uses:
tool_name = tu.name
tool_input = tu.input
try:
tr = session.call_tool(tool_name, tool_input)
# Convert OpenReward blocks -> string safely
text_parts = []
for b in getattr(tr, "blocks", []) or []:
t = getattr(b, "text", None)
if t is not None:
text_parts.append(t)
else:
text_parts.append(str(b))
tool_text = "".join(text_parts)
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": tool_text,
})
# Track termination if the env says we're done
if getattr(tr, "finished", False):
finished = True
except Exception as e:
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": f"Tool execution failed: {type(e).__name__}: {e}",
"is_error": True,
})
messages.append({
"role": "user",
"content": tool_result_blocks
})
print(messages[-1])
continue
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': [TextBlock(text="I'll help you solve this math problem using the bash tool to calculate the answer.", type='text'), ToolUseBlock(id='toolu_01XbashToolExample123', input={'command': 'python3 - << \'PY\'\napril=48\nmay=april/2\nprint(april+may)\nPY', 'timeout': 100000}, name='bash', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01XbashToolExample123', 'content': '72.0\n'}]}
{'role': 'assistant', 'content': [TextBlock(text='Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24.\n\nAltogether, she sold 48 + 24 = 72 clips in April and May.', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': 'Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24. Altogether, she sold 48 + 24 = 72 clips.'}, name='answer', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}
```
Make sure you have an API key for [Gemini](https://aistudio.google.com/app/apikey) and set it as an environment variable:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export E2B_API_KEY='your-e2b-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
import os
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=tools)]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task, secrets={"e2b_api_key": os.getenv("E2B_API_KEY")}) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": json.dumps({
"result": tool_result.blocks[0].text
})},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'command': "python3 - << 'PY'\napril=48\nmay=april/2\nprint(april+may)\nPY",
'timeout': 100000
},
name='bash'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='bash',
response={
'result': '{"result": "72.0\\n"}'
}
)
)] role='user'
parts=[Part(
text='Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24.\n\nAltogether, she sold 48 + 24 = 72 clips in April and May.'
)] role='model'
parts=[Part(
function_call=FunctionCall(
args={
'answer': 'Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24. Altogether, she sold 48 + 24 = 72 clips.'
},
name='answer'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='answer',
response={
'result': '{"result": "Correct!"}'
}
)
)] role='user'
```
Make sure you have an API key for [OpenRouter](https://openrouter.ai/keys) and set it as an environment variables:
```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export E2B_API_KEY='your-e2b-api-key-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"e2b_api_key": os.getenv("E2B_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
input_list.append({
"role": "assistant",
"content": response.choices[0].message.content,
"tool_calls": response.choices[0].message.tool_calls
})
print(input_list[-1])
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
break
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': None, 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='call_bash_example_001', function=Function(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april/2\\nprint(april+may)\\nPY","timeout":100000}', name='bash'), type='function')]}
{'role': 'tool', 'tool_call_id': 'call_bash_example_001', 'content': '{"result": "72.0\\n"}'}
{'role': 'assistant', 'content': 'Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24.\n\nAltogether, she sold 48 + 24 = 72 clips in April and May.', 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24. Altogether, she sold 48 + 24 = 72 clips."}', name='answer'), type='function')]}
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': '{"result": "Correct!"}'}
```
Nice one! We have a working ORS environment.
Now we'll see how we can host the environment on OpenReward. The benefits of using OpenReward are:
* **Infrastructure**: you do not have to set up infrastructure and compute to host the environment yourself. We take care of this and you are only charged based on your actual usage of the environment.
* **Discovery**: your environment can be discovered and used by other users of the platform, helping drive adoption and attention to your work.
## Host on OpenReward
Log into [OpenReward](https://openreward.ai), press the plus icon in the navbar and press **New Environment**:
Next, fill in information about the environment and press **Create Environment**:
You will be redirected to your new environment and will see setup instructions:
### Upload environment files
We will need a way to use the train and test parquet files in our environment. We'll upload these to the environment files:
Click on the **Files** tab and upload each file:
Files are mounted to the environment server at the `/orwd_data` directory.
We'll need to reference this folder in our `server.py`. Make the following change:
```python theme={null}
train_tasks = pd.read_parquet("/orwd_data/train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("/orwd_data/test-00000-of-00001.parquet").to_dict(orient="records")
```
Note: you may want to set an environment variable instead of hardcoding like above so you can continue to test
locally (without the `/orwd_data` prefix).
### Write the Dockerfile and requirements
We'll need a `Dockerfile` in our repository:
```txt theme={null}
FROM python:3.11-slim
RUN apt update && apt upgrade -y && apt install -y \
curl
WORKDIR /app
# Copy requirements and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application code
COPY server.py .
# Expose port
EXPOSE 8000
# Start the server
CMD ["python", "server.py"]
```
We'll need to update `requirements.txt`:
```txt theme={null}
fastapi>=0.115.12
openreward
pandas
pyarrow
uvicorn>=0.34.3
math-verify[antlr4_13_2]
```
### Push to GitHub and connect
Next, [push your environment code to a GitHub repository](https://github.com/new).
Once your GitHub repository is ready, go to your OpenReward environment and connect the repository:
You will be given a choice for how much compute you would like to allocate for it. We'll use a low compute configuration since this is a simple environment.
Press Connect GitHub and your first build will begin.
To check the progress of the build, click the Deployments tab:
You can click on the latest build row to see logs:
The build logs show the progress of building the environment. The runtime logs show any calls to the environment server, and can be useful for diagnosing errors.
### Sample from your environment
Now your environment is hosted on OpenReward, we can sample from it:
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenAI](https://platform.openai.com/api-keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export E2B_API_KEY='your-e2b-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"e2b_api_key": os.getenv("E2B_API_KEY")}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
[ResponseFunctionToolCall(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april/2\\nprint(april+may)\\nPY","timeout":100000}', call_id='call_TTS3bI1XFo0Gl88If18JZZ6l', name='bash', type='function_call', id='fc_04efdcdc3efb56c300699ada2d1fac81908551b7ab626fffd2', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_TTS3bI1XFo0Gl88If18JZZ6l', 'output': '{"result": "72.0\\n"}'}
[ResponseOutputMessage(id='msg_04efdcdc3efb56c300699ada2f46d48190a6234d05fdc92310', content=[ResponseOutputText(annotations=[], text='Natalia sold 48 clips in April. In May she sold half of that: \\(48 \\div 2 = 24\\).\n\nAltogether, she sold \\(48 + 24 = 72\\) clips in April and May.', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24. Altogether, she sold 48 + 24 = 72 clips."}', call_id='call_QPFUtfAviwYsozYSKadq78gZ', name='answer', type='function_call', id='fc_04efdcdc3efb56c300699ada30ddfc81909c7bd18e1398f223', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_QPFUtfAviwYsozYSKadq78gZ', 'output': '{"result": "Correct!"}'}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Anthropic](https://platform.claude.com/settings/keys), and set these as environment variables:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export E2B_API_KEY='your-e2b-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
import os
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"e2b_api_key": os.getenv("E2B_API_KEY")}) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
response = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
print(message)
messages.append(message)
if message.stop_reason == "tool_use":
tool_use = next(block for block in message.content if block.type == "tool_use")
tool_name = tool_use.name
tool_input = tool_use.input
tool_result = session.call_tool(tool_name, tool_input)
reward = tool_result.reward
finished = tool_result.finished
messages.append({
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": tool_result.blocks[0].text
}
]
})
print(messages[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': [TextBlock(text="I'll help you solve this math problem using the bash tool to calculate the answer.", type='text'), ToolUseBlock(id='toolu_01XbashToolExample123', input={'command': 'python3 - << \'PY\'\napril=48\nmay=april/2\nprint(april+may)\nPY', 'timeout': 100000}, name='bash', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01XbashToolExample123', 'content': '72.0\n'}]}
{'role': 'assistant', 'content': [TextBlock(text='Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24.\n\nAltogether, she sold 48 + 24 = 72 clips in April and May.', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': 'Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24. Altogether, she sold 48 + 24 = 72 clips.'}, name='answer', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Gemini](https://aistudio.google.com/app/apikey), and set these as environment variables:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export E2B_API_KEY='your-e2b-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
import os
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=[f]) for f in tools]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task, secrets={"e2b_api_key": os.getenv("E2B_API_KEY")}) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": json.dumps({
"result": tool_result.blocks[0].text
})},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'command': "python3 - << 'PY'\napril=48\nmay=april/2\nprint(april+may)\nPY",
'timeout': 100000
},
name='bash'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='bash',
response={
'result': '{"result": "72.0\\n"}'
}
)
)] role='user'
parts=[Part(
text='Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24.\n\nAltogether, she sold 48 + 24 = 72 clips in April and May.'
)] role='model'
parts=[Part(
function_call=FunctionCall(
args={
'answer': 'Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24. Altogether, she sold 48 + 24 = 72 clips.'
},
name='answer'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='answer',
response={
'result': '{"result": "Correct!"}'
}
)
)] role='user'
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export E2B_API_KEY='your-e2b-api-key-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task, secrets={"e2b_api_key": os.getenv("E2B_API_KEY")}) as session:
prompt = await session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
print(response)
input_list.append(response)
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
return
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = await session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': None, 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='call_bash_example_001', function=Function(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april/2\\nprint(april+may)\\nPY","timeout":100000}', name='bash'), type='function')]}
{'role': 'tool', 'tool_call_id': 'call_bash_example_001', 'content': '{"result": "72.0\\n"}'}
{'role': 'assistant', 'content': 'Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24.\n\nAltogether, she sold 48 + 24 = 72 clips in April and May.', 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "Natalia sold 48 clips in April. In May she sold half of that: 48 ÷ 2 = 24. Altogether, she sold 48 + 24 = 72 clips."}', name='answer'), type='function')]}
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': '{"result": "Correct!"}'}
```
If you are running with another provider, or using custom models, then here are the main principles to keep in mind.
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
You'll need to use the OpenReward client to access the environment and its main information
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
```
```python theme={null}
async with environment.session(task=example_task) as session:
```
Using a context manager, we can start a session with the environment. This defines a scope in which
you can use the agent to call tools and get tool results.
Above we have selected the first task to sample from, which is a particular problem in the CTF environment.
You can get the prompt for the task as follows:
```python theme={null}
prompt = await session.get_prompt()
```
You will also need to pass in the `tools` into the context window of your model, usually somewhere in the system prompt.
An agent will usually keep interacting with an environment until it hits a termination state associated with the environment, or
some other imposed limit (e.g. maximum number of terms).
In a simple sequential agent model, this could be a while loop like:
```python theme={null}
while not finished:
```
where `finished` is a boolean.
In agentic environments, actions are treated as tool calls. That means you need to have a way to parse tool calls from your model's generations.
The key thing to note is that for OpenReward environment, a tool call requires specifying a name (`str`) and some arguments (`dict`).
To call a tool you will call:
```python theme={null}
tool_result = await session.call_tool(tool_name, tool_arguments)
```
This means that you will need to parse out the `tool_name` and `tool_arguments` from your model's generation and then parse this information into the
`call_tool` method.
If you have executed an available tool correctly, you will receive a `ToolOutput` output. This contains attributes for:
* `reward` : an (optional) `float` denoting reward. For example, submitting a correct math solution through a `submit_solution` tool might give a reward of `1.0`.
* `finished` : a `bool` specifying whether the episode is finished or not. For example, some tools may end the episode (if for example an agent submits a final answer through `submit_solution` in a math task).
* `data` : a `dict` with output of the executing of the tool. For example, the stdout of executing a `bash` tool. This information should be passed to the agent as feedback.
For our agent loop, some example control flow we might want after receiving a `ToolOutput` is:
* Recording `reward` for use in a policy gradient algorithm such as GRPO or PPO
* Breaking out of the core agent loop if `finished=True`
* Adding `data` to the context window as feedback and continuing with the next model generation
Importantly, if you are publishing your environment, you should update your SDK example and let users know that your environment needs an `E2B_API_KEY`. Press Edit Settings near the SDK example then insert the code for environment variables:
# Using Modal Sandboxes
Source: https://docs.openreward.ai/sandboxes/modal
Make and deploy an ORS environment with Modal Sandboxes
## Goals
* Make a mathematics code execution environment using OpenReward and Modal.
* Deploy the environment to OpenReward.
* Sample from the environment using a model of your choice.
## Prerequisites
* You have completed the [Your First Environment](../environments/your-first-environment) tutorial
* An OpenReward [account](https://openreward.ai/)
* An OpenReward [API key](https://openreward.ai/keys)
* A Modal account with [Service User tokens](https://modal.com/docs/guide/service-users) for programmatic access
* An API key and SDK for your model provider of choice (e.g. OpenAI, Anthropic, Google, OpenRouter)
## Setup
Environments in OpenReward are written using [ORS](https://openrewardstandard.io).
ORS is implemented in the [OpenReward Python library](https://www.github.com/OpenReward/openreward-python), and we will use it for this tutorial.
You can install the library using pip or uv:
```bash theme={null}
pip install openreward
```
You should also install the `modal` Python SDK:
```bash theme={null}
pip install modal
```
## Introduction
[ORS](https://openrewardstandard.io) environments can be configured to work with any sandbox provider. In this tutorial we will show how to initialise a Modal sandbox within an ORS environment. We'll setup a simple mathematics environment and give an agent access to a sandbox for executing code. By the end of this tutorial, you will understand how to use Modal as an integration for making environments with ORS and OpenReward.
Modal uses an App-based architecture, where sandboxes are created within the context of a Modal App. We'll use `App.lookup()` with `create_if_missing=True` to automatically handle app creation.
## Getting Started
In the [Your First Environment](../environments/your-first-environment) tutorial, we built a mathematics environment using the [GSM8K](https://arxiv.org/abs/2110.14168) dataset.
We will use a similar dataset here, but this time we will give an agent an access to a Modal sandbox for code execution.
First, let's initialise our environment `gsm8ksandbox`:
```bash theme={null}
orwd init gsm8ksandbox --template basic
cd gsm8k && ls
```
Next we'll download the two `parquet` files from the GSM8K [HuggingFace repository](https://huggingface.co/datasets/openai/gsm8k/tree/main/main) and put them in the root of our project:
```
test-00000-of-00001.parquet
train-00000-of-00001.parquet
```
A single row of data from the train set looks as follows:
```
{'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72', 'id': '0'}
```
Next we'll write a new server file. This will involve:
* Loading the tasks from the `parquet` files
* Verifying the answer is correct - we'll use the [MathVerify](https://github.com/huggingface/Math-Verify) library for this.
```python theme={null}
from math_verify import parse, verify
import pandas as pd
from pydantic import BaseModel
from typing import Optional
import modal
from openreward.environments import Environment, JSONObject, Server, Split, TextBlock, ToolOutput, tool
class GSM8KTaskSpec(BaseModel):
id: str
question: str
answer: str
class AnswerParams(BaseModel):
answer: str
train_tasks = pd.read_parquet("train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("test-00000-of-00001.parquet").to_dict(orient="records")
for i, task in enumerate(train_tasks):
task['id'] = str(i)
for i, task in enumerate(test_tasks):
task['id'] = str(i)
class GSM8KSandbox(Environment):
"""
A GSM8K sandbox environment
"""
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = GSM8KTaskSpec.model_validate(task_spec)
@classmethod
def list_tasks(cls, split: str) -> list[JSONObject]:
if split == "train":
return train_tasks
elif split == "test":
return test_tasks
raise ValueError(f"Unknown split: {split}")
@classmethod
def list_splits(cls):
return [Split(name="train", type="train"), Split(name="test", type="test")]
def get_prompt(self) -> str:
return [TextBlock(type="text", text=self.config.question)]
@tool
def answer(self, params: AnswerParams) -> ToolOutput:
"""
The answer tool can be used to submit your final answer. Note that this finishes the episode.
"""
gold = parse(self.config.answer)
answer = parse(params.answer)
is_correct = verify(gold, answer)
if is_correct:
agent_message = "Correct!"
reward = 1.0
else:
agent_message = "Wrong!"
reward = 0.0
return ToolOutput(
blocks=[TextBlock(type="text", text=agent_message)],
reward=reward,
finished=True
)
if __name__ == "__main__":
Server([GSM8KSandbox]).run()
```
Install the `math-verify` requirement:
```bash theme={null}
pip install math-verify
```
Now we are going to modify our environment by including a Modal sandbox. First, we'll adjust our `__init__`:
```python theme={null}
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.config = GSM8KTaskSpec.model_validate(task_spec)
self.token_id = secrets.get("modal_token_id")
self.token_secret = secrets.get("modal_token_secret")
if not self.token_id or not self.token_secret:
raise ValueError("Modal token ID and secret must be provided via secrets parameter")
self.app = None
self.sandbox = None
```
Next, we'll add a `setup` and `teardown` method:
```python theme={null}
async def setup(self) -> None:
import os
os.environ["MODAL_TOKEN_ID"] = self.token_id
os.environ["MODAL_TOKEN_SECRET"] = self.token_secret
self.app = await modal.App.lookup.aio("gsm8ksandbox", create_if_missing=True)
self.sandbox = await modal.Sandbox.create.aio(app=self.app)
async def teardown(self) -> None:
if self.sandbox is not None:
await self.sandbox.terminate.aio()
```
These methods ensure that when a session begins, we create a Modal app and sandbox, and when a session ends, we terminate the sandbox. Note that we use Modal's async interfaces (`aio`) since OpenReward runs in an async context.
Lastly, we can add a tool that relies on code execution. We'll add a `bash` tool. First we'll add the Pydantic model:
```python theme={null}
class BashParams(BaseModel, extra="forbid"):
command: str
timeout: Optional[float] = 30.0
```
and then we'll add the `bash` tool to the class:
```python theme={null}
@tool
async def bash(self, params: BashParams) -> ToolOutput:
"""Execute bash commands using the computer instance."""
try:
process = await self.sandbox.exec.aio("bash", "-c", params.command.strip(), bufsize=1)
# Read output from process
stdout_lines = []
async for line in process.stdout:
stdout_lines.append(line)
stdout_text = "".join(stdout_lines)
# Wait for process to complete
await process.wait.aio()
exit_code = process.returncode if hasattr(process, 'returncode') else 0
result_dict = {
"stdout": stdout_text,
"stderr": "",
"exit_code": exit_code,
}
# What the model sees as tool text output
text_out = result_dict["stdout"] or str(result_dict)
return ToolOutput(
blocks=[TextBlock(type="text", text=text_out)],
metadata={"output": result_dict}, # JSON-safe now
reward=0.0,
finished=False,
)
except Exception as e:
return ToolOutput(
metadata={"error": str(e)},
blocks=[TextBlock(text=f"Error executing command: {str(e)}")],
finished=False
)
```
Now we can test this environment. First run the server as before:
```bash theme={null}
python server.py
```
Now choose a model provider of your choice and sample from the environment:
Make sure you have an API key for [OpenAI](https://platform.openai.com/api-keys), and set the environment variable:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export MODAL_TOKEN_ID='your-modal-token-id-here'
export MODAL_TOKEN_SECRET='your-modal-token-secret-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import os
import json
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task, secrets={
"modal_token_id": os.getenv("MODAL_TOKEN_ID"),
"modal_token_secret": os.getenv("MODAL_TOKEN_SECRET")
}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
[ResponseFunctionToolCall(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april//2\\nprint(april+may)\\nPY","timeout":100000}', call_id='call_Rw5I79bwylStMxdlup0aE8ut', name='bash', type='function_call', id='fc_023997d138873f4a00699aefcd70ac819fa4ae3910d0b23fc9', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_Rw5I79bwylStMxdlup0aE8ut', 'output': '{"result": "72\\n"}'}
[ResponseOutputMessage(id='msg_023997d138873f4a00699aefd0539c819f8d4a46ded9cdf39e', content=[ResponseOutputText(annotations=[], text='Natalia sold 48 clips in April and half as many in May: \\(48 \\div 2 = 24\\). \nAltogether, she sold \\(48 + 24 = 72\\) clips in April and May.', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"Natalia sold 48 clips in April and half as many in May: \\\\(48 \\\\div 2 = 24\\\\).\\nAltogether, she sold \\\\(48 + 24 = 72\\\\) clips in April and May."}', call_id='call_Rzy1nsOTzb7d2OWJpA0phOf0', name='answer', type='function_call', id='fc_023997d138873f4a00699aefd1f850819f80f481022033bc70', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_Rzy1nsOTzb7d2OWJpA0phOf0', 'output': '{"result": "Correct!"}'}
```
Make sure you have API key for [Anthropic](https://platform.claude.com/settings/keys), and set the environment variable:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export MODAL_TOKEN_ID='your-modal-token-id-here'
export MODAL_TOKEN_SECRET='your-modal-token-secret-here'
```
Save this as `sample_agent.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
import os
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task, secrets={
"modal_token_id": os.getenv("MODAL_TOKEN_ID"),
"modal_token_secret": os.getenv("MODAL_TOKEN_SECRET")
}) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
message = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
messages.append({
"role": "assistant",
"content": message.content,
})
print(messages[-1])
if message.stop_reason == "tool_use":
tool_uses = [b for b in message.content if getattr(b, "type", None) == "tool_use"]
if not tool_uses:
raise RuntimeError("stop_reason was tool_use but no tool_use blocks found")
tool_result_blocks = []
finished = False
for tu in tool_uses:
tool_name = tu.name
tool_input = tu.input
try:
tr = session.call_tool(tool_name, tool_input)
# Convert OpenReward blocks -> string safely
text_parts = []
for b in getattr(tr, "blocks", []) or []:
t = getattr(b, "text", None)
if t is not None:
text_parts.append(t)
else:
text_parts.append(str(b))
tool_text = "".join(text_parts)
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": tool_text,
})
# Track termination if the env says we're done
if getattr(tr, "finished", False):
finished = True
except Exception as e:
tool_result_blocks.append({
"type": "tool_result",
"tool_use_id": tu.id,
"content": f"Tool execution failed: {type(e).__name__}: {e}",
"is_error": True,
})
messages.append({
"role": "user",
"content": tool_result_blocks
})
print(messages[-1])
continue
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': [TextBlock(text="I'll help you solve this math problem using the bash tool to calculate the answer.", type='text'), ToolUseBlock(id='toolu_01XbashToolExample123', input={'command': 'python3 - << \'PY\'\napril=48\nmay=april//2\nprint(april+may)\nPY', 'timeout': 100000}, name='bash', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01XbashToolExample123', 'content': '72\n'}]}
{'role': 'assistant', 'content': [TextBlock(text='Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips in April and May.', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': 'Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24.\nAltogether, she sold 48 + 24 = 72 clips in April and May.'}, name='answer', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}
```
Make sure you have an API key for [Gemini](https://aistudio.google.com/app/apikey) and set it as an environment variable:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export MODAL_TOKEN_ID='your-modal-token-id-here'
export MODAL_TOKEN_SECRET='your-modal-token-secret-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
import os
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=tools)]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task, secrets={
"modal_token_id": os.getenv("MODAL_TOKEN_ID"),
"modal_token_secret": os.getenv("MODAL_TOKEN_SECRET")
}) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": json.dumps({
"result": tool_result.blocks[0].text
})},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'command': "python3 - << 'PY'\napril=48\nmay=april//2\nprint(april+may)\nPY",
'timeout': 100000
},
name='bash'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='bash',
response={
'result': '{"result": "72\\n"}'
}
)
)] role='user'
parts=[Part(
text='Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips in April and May.'
)] role='model'
parts=[Part(
function_call=FunctionCall(
args={
'answer': 'Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24.\nAltogether, she sold 48 + 24 = 72 clips in April and May.'
},
name='answer'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='answer',
response={
'result': '{"result": "Correct!"}'
}
)
)] role='user'
```
Make sure you have an API key for [OpenRouter](https://openrouter.ai/keys) and set it as an environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export MODAL_TOKEN_ID='your-modal-token-id-here'
export MODAL_TOKEN_SECRET='your-modal-token-secret-here'
```
Save this as `sample_agent.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="gsm8ksandbox", base_url="http://localhost:8080")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task, secrets={
"modal_token_id": os.getenv("MODAL_TOKEN_ID"),
"modal_token_secret": os.getenv("MODAL_TOKEN_SECRET")
}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
input_list.append({
"role": "assistant",
"content": response.choices[0].message.content,
"tool_calls": response.choices[0].message.tool_calls
})
print(input_list[-1])
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
break
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python sample_agent.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': None, 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='call_bash_example_001', function=Function(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april//2\\nprint(april+may)\\nPY","timeout":100000}', name='bash'), type='function')]}
{'role': 'tool', 'tool_call_id': 'call_bash_example_001', 'content': '{"result": "72\\n"}'}
{'role': 'assistant', 'content': 'Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips in April and May.', 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24.\\nAltogether, she sold 48 + 24 = 72 clips in April and May."}', name='answer'), type='function')]}
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': '{"result": "Correct!"}'}
```
Nice one! We have a working [ORS](https://openrewardstandard.io) environment.
Now we'll see how we can host the environment on OpenReward. The benefits of using OpenReward are:
* **Infrastructure**: you do not have to set up infrastructure and compute to host the environment yourself. We take care of this and you are only charged based on your actual usage of the environment.
* **Discovery**: your environment can be discovered and used by other users of the platform, helping drive adoption and attention to your work.
## Host on OpenReward
Log into [OpenReward](https://openreward.ai), press the plus icon in the navbar and press **New Environment**:
Next, fill in information about the environment and press **Create Environment**:
You will be redirected to your new environment and will see setup instructions:
### Upload environment files
We will need a way to use the train and test parquet files in our environment. We'll upload these to the environment files:
Click on the **Files** tab and upload each file:
Files are mounted to the environment server at the `/orwd_data` directory.
We'll need to reference this folder in our `server.py`. Make the following change:
```python theme={null}
train_tasks = pd.read_parquet("/orwd_data/train-00000-of-00001.parquet").to_dict(orient="records")
test_tasks = pd.read_parquet("/orwd_data/test-00000-of-00001.parquet").to_dict(orient="records")
```
Note: you may want to set an environment variable instead of hardcoding like above so you can continue to test
locally (without the `/orwd_data` prefix).
### Write the Dockerfile and requirements
We'll need a `Dockerfile` in our repository:
```txt theme={null}
FROM python:3.11-slim
RUN apt update && apt upgrade -y && apt install -y \
curl
WORKDIR /app
# Copy requirements and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application code
COPY server.py .
# Expose port
EXPOSE 8000
# Start the server
CMD ["python", "server.py"]
```
We'll need to update `requirements.txt`:
```txt theme={null}
fastapi>=0.115.12
openreward
pandas
pyarrow
uvicorn>=0.34.3
math-verify[antlr4_13_2]
modal
```
### Push to GitHub and connect
Next, [push your environment code to a GitHub repository](https://github.com/new).
Once your GitHub repository is ready, go to your OpenReward environment and connect the repository:
You will be given a choice for how much compute you would like to allocate for it. We'll use a low compute configuration since this is a simple environment.
Press Connect GitHub and your first build will begin.
To check the progress of the build, click the Deployments tab:
You can click on the latest build row to see logs:
The build logs show the progress of building the environment. The runtime logs show any calls to the environment server, and can be useful for diagnosing errors.
### Sample from your environment
Now your environment is hosted on OpenReward, we can sample from it:
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenAI](https://platform.openai.com/api-keys), and set these as environment variables:
```bash theme={null}
export OPENAI_API_KEY='your-openai-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export MODAL_TOKEN_ID='your-modal-token-id-here'
export MODAL_TOKEN_SECRET='your-modal-token-secret-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI()
MODEL_NAME = "gpt-5.4"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
with environment.session(task=example_task, secrets={
"modal_token_id": os.getenv("MODAL_TOKEN_ID"),
"modal_token_secret": os.getenv("MODAL_TOKEN_SECRET")
}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.responses.create(
model=MODEL_NAME,
tools=tools,
input=input_list
)
print(response.output)
input_list += response.output
for item in response.output:
if item.type == "function_call":
tool_result = session.call_tool(item.name, json.loads(str(item.arguments)))
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
[ResponseFunctionToolCall(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april//2\\nprint(april+may)\\nPY","timeout":100000}', call_id='call_Rw5I79bwylStMxdlup0aE8ut', name='bash', type='function_call', id='fc_023997d138873f4a00699aefcd70ac819fa4ae3910d0b23fc9', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_Rw5I79bwylStMxdlup0aE8ut', 'output': '{"result": "72\\n"}'}
[ResponseOutputMessage(id='msg_023997d138873f4a00699aefd0539c819f8d4a46ded9cdf39e', content=[ResponseOutputText(annotations=[], text='Natalia sold 48 clips in April and half as many in May: \\(48 \\div 2 = 24\\). \nAltogether, she sold \\(48 + 24 = 72\\) clips in April and May.', type='output_text', logprobs=[])], role='assistant', status='completed', type='message')]
[ResponseFunctionToolCall(arguments='{"answer":"Natalia sold 48 clips in April and half as many in May: \\\\(48 \\\\div 2 = 24\\\\).\\nAltogether, she sold \\\\(48 + 24 = 72\\\\) clips in April and May."}', call_id='call_Rzy1nsOTzb7d2OWJpA0phOf0', name='answer', type='function_call', id='fc_023997d138873f4a00699aefd1f850819f80f481022033bc70', status='completed')]
{'type': 'function_call_output', 'call_id': 'call_Rzy1nsOTzb7d2OWJpA0phOf0', 'output': '{"result": "Correct!"}'}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Anthropic](https://platform.claude.com/settings/keys), and set these as environment variables:
```bash theme={null}
export ANTHROPIC_API_KEY='your-anthropic-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export MODAL_TOKEN_ID='your-modal-token-id-here'
export MODAL_TOKEN_SECRET='your-modal-token-secret-here'
```
Save this as `quickstart.py`:
```python theme={null}
import anthropic
from openreward import OpenReward
import json
import os
or_client = OpenReward()
ant_client = anthropic.Anthropic()
MODEL_NAME = "claude-sonnet-4-6"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="anthropic")
example_task = tasks[0]
with environment.session(task=example_task, secrets={
"modal_token_id": os.getenv("MODAL_TOKEN_ID"),
"modal_token_secret": os.getenv("MODAL_TOKEN_SECRET")
}) as session:
prompt = session.get_prompt()
messages = [{"role": "user", "content": prompt[0].text}]
finished = False
print(messages)
while not finished:
response = ant_client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
messages=messages
)
print(message)
messages.append(message)
if message.stop_reason == "tool_use":
tool_use = next(block for block in message.content if block.type == "tool_use")
tool_name = tool_use.name
tool_input = tool_use.input
tool_result = session.call_tool(tool_name, tool_input)
reward = tool_result.reward
finished = tool_result.finished
messages.append({
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": tool_result.blocks[0].text
}
]
})
print(messages[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': [TextBlock(text="I'll help you solve this math problem using the bash tool to calculate the answer.", type='text'), ToolUseBlock(id='toolu_01XbashToolExample123', input={'command': 'python3 - << \'PY\'\napril=48\nmay=april//2\nprint(april+may)\nPY', 'timeout': 100000}, name='bash', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01XbashToolExample123', 'content': '72\n'}]}
{'role': 'assistant', 'content': [TextBlock(text='Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips in April and May.', type='text'), ToolUseBlock(id='toolu_01HEJromRn1bMcU8HM5jQZDF', input={'answer': 'Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24.\nAltogether, she sold 48 + 24 = 72 clips in April and May.'}, name='answer', type='tool_use')]}
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_01HEJromRn1bMcU8HM5jQZDF', 'content': 'Correct!'}]}
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [Gemini](https://aistudio.google.com/app/apikey), and set these as environment variables:
```bash theme={null}
export GEMINI_API_KEY='your-gemini-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export MODAL_TOKEN_ID='your-modal-token-id-here'
export MODAL_TOKEN_SECRET='your-modal-token-secret-here'
```
Save this as `quickstart.py`:
```python theme={null}
from google import genai
from google.genai import types
from openreward import OpenReward
import json
import os
or_client = OpenReward()
gem_client = genai.Client()
MODEL_NAME = "gemini-2.5-flash"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="google")
genai_tools = [types.Tool(function_declarations=[f]) for f in tools]
genai_config = types.GenerateContentConfig(tools=genai_tools)
example_task = tasks[0]
with environment.session(task=example_task, secrets={
"modal_token_id": os.getenv("MODAL_TOKEN_ID"),
"modal_token_secret": os.getenv("MODAL_TOKEN_SECRET")
}) as session:
prompt = session.get_prompt()
contents = [
types.Content(
role="user", parts=[types.Part(text=prompt[0].text)]
)
]
finished = False
print(contents)
while not finished:
response = gem_client.models.generate_content(
model=MODEL_NAME,
config=genai_config,
contents=contents
)
print(response.candidates[0].content)
contents.append(response.candidates[0].content) # Append the content from the model's response.
for part in response.candidates[0].content.parts:
if part.function_call:
tool_call = part.function_call
tool_result = session.call_tool(tool_call.name, tool_call.args)
reward = tool_result.reward
finished = tool_result.finished
function_response_part = types.Part.from_function_response(
name=tool_call.name,
response={"result": json.dumps({
"result": tool_result.blocks[0].text
})},
)
contents.append(types.Content(role="user", parts=[function_response_part])) # Append the function response
print(contents[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[Content(
parts=[
Part(
text='Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'
),
],
role='user'
)]
parts=[Part(
function_call=FunctionCall(
args={
'command': "python3 - << 'PY'\napril=48\nmay=april//2\nprint(april+may)\nPY",
'timeout': 100000
},
name='bash'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='bash',
response={
'result': '{"result": "72\\n"}'
}
)
)] role='user'
parts=[Part(
text='Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips in April and May.'
)] role='model'
parts=[Part(
function_call=FunctionCall(
args={
'answer': 'Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24.\nAltogether, she sold 48 + 24 = 72 clips in April and May.'
},
name='answer'
)
)] role='model'
parts=[Part(
function_response=FunctionResponse(
name='answer',
response={
'result': '{"result": "Correct!"}'
}
)
)] role='user'
```
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
export MODAL_TOKEN_ID='your-modal-token-id-here'
export MODAL_TOKEN_SECRET='your-modal-token-secret-here'
```
Save this as `quickstart.py`:
```python theme={null}
from openai import OpenAI
from openreward import OpenReward
import json
import os
or_client = OpenReward()
oai_client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ.get("OPENROUTER_API_KEY")
)
MODEL_NAME = "deepseek/deepseek-v3.2"
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openrouter")
example_task = tasks[0]
with environment.session(task=example_task, secrets={
"modal_token_id": os.getenv("MODAL_TOKEN_ID"),
"modal_token_secret": os.getenv("MODAL_TOKEN_SECRET")
}) as session:
prompt = session.get_prompt()
input_list = [{"role": "user", "content": prompt[0].text}]
finished = False
print(input_list)
while not finished:
response = oai_client.chat.completions.create(
model=MODEL_NAME,
tools=tools,
messages=input_list
)
print(response)
input_list.append(response)
tool_calls = response.choices[0].message.tool_calls
if not tool_calls:
return
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = session.call_tool(tool_name, tool_args)
reward = tool_result.reward
finished = tool_result.finished
input_list.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({
"result": tool_result.blocks[0].text
})
})
print(input_list[-1])
if tool_result.finished:
finished = True
break
```
```bash theme={null}
python quickstart.py
```
Example output:
```bash theme={null}
[{'role': 'user', 'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nUse the bash tool to execute commands.'}]
{'role': 'assistant', 'content': None, 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='call_bash_example_001', function=Function(arguments='{"command":"python3 - << \'PY\'\\napril=48\\nmay=april//2\\nprint(april+may)\\nPY","timeout":100000}', name='bash'), type='function')]}
{'role': 'tool', 'tool_call_id': 'call_bash_example_001', 'content': '{"result": "72\\n"}'}
{'role': 'assistant', 'content': 'Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24. \nAltogether, she sold 48 + 24 = 72 clips in April and May.', 'tool_calls': [ChatCompletionMessageFunctionToolCall(id='019af3af119248e0f1cd239559746033', function=Function(arguments='{"answer": "Natalia sold 48 clips in April and half as many in May: 48 ÷ 2 = 24.\\nAltogether, she sold 48 + 24 = 72 clips in April and May."}', name='answer'), type='function')]}
{'role': 'tool', 'tool_call_id': '019af3af119248e0f1cd239559746033', 'content': '{"result": "Correct!"}'}
```
If you are running with another provider, or using custom models, then here are the main principles to keep in mind.
Make sure you have API keys for [OpenReward](https://openreward.ai/keys) and [OpenRouter](https://openrouter.ai/keys), and set these as environment variables:
```bash theme={null}
export OPENROUTER_API_KEY='your-openrouter-api-key-here'
export OPENREWARD_API_KEY='your-openreward-api-key-here'
```
You'll need to use the OpenReward client to access the environment and its main information
```python theme={null}
from openreward import OpenReward
or_client = OpenReward()
environment = or_client.environments.get(name="yourusername/gsm8k")
tasks = environment.list_tasks(split="train")
tools = environment.list_tools(format="openai")
example_task = tasks[0]
```
```python theme={null}
with environment.session(task=example_task) as session:
```
Using a context manager, we can start a session with the environment. This defines a scope in which
you can use the agent to call tools and get tool results.
Above we have selected the first task to sample from, which is a particular problem in the CTF environment.
You can get the prompt for the task as follows:
```python theme={null}
prompt = session.get_prompt()
```
You will also need to pass in the `tools` into the context window of your model, usually somewhere in the system prompt.
An agent will usually keep interacting with an environment until it hits a termination state associated with the environment, or
some other imposed limit (e.g. maximum number of terms).
In a simple sequential agent model, this could be a while loop like:
```python theme={null}
while not finished:
```
where `finished` is a boolean.
In agentic environments, actions are treated as tool calls. That means you need to have a way to parse tool calls from your model's generations.
The key thing to note is that for OpenReward environment, a tool call requires specifying a name (`str`) and some arguments (`dict`).
To call a tool you will call:
```python theme={null}
tool_result = session.call_tool(tool_name, tool_arguments)
```
This means that you will need to parse out the `tool_name` and `tool_arguments` from your model's generation and then parse this information into the
`call_tool` method.
If you have executed an available tool correctly, you will receive a `ToolOutput` output. This contains attributes for:
* `reward` : an (optional) `float` denoting reward. For example, submitting a correct math solution through a `submit_solution` tool might give a reward of `1.0`.
* `finished` : a `bool` specifying whether the episode is finished or not. For example, some tools may end the episode (if for example an agent submits a final answer through `submit_solution` in a math task).
* `data` : a `dict` with output of the executing of the tool. For example, the stdout of executing a `bash` tool. This information should be passed to the agent as feedback.
For our agent loop, some example control flow we might want after receiving a `ToolOutput` is:
* Recording `reward` for use in a policy gradient algorithm such as GRPO or PPO
* Breaking out of the core agent loop if `finished=True`
* Adding `data` to the context window as feedback and continuing with the next model generation
Importantly, if you are publishing your environment, you should update your SDK example and let users know that your environment needs `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`. Press Edit Settings near the SDK example then insert the code for environment variables:
# Using OpenReward Sandboxes
Source: https://docs.openreward.ai/sandboxes/openreward
Use OpenReward native sandbox infrastructure with ORS environments
## Overview
[ORS](https://openrewardstandard.io) environments work with any sandbox provider - E2B, Daytona, Modal, or others. OpenReward also provides a native sandbox option that integrates directly with the platform's environment file storage.
## When to Use OpenReward Sandboxes
OpenReward sandboxes may be a good fit if you:
* **Already use OpenReward**: Same API key for environments and sandboxes
* **Use environment file storage**: Direct mounting of environment buckets
For specialised needs, external providers like E2B, Daytona, or Modal may offer better solutions.
## Features
* **Multiple machine sizes**: From `0.1:0.1` (minimal) to `4:16` (high-memory workloads)
* **Flexible configuration**: Custom container images, resources, and environment variables
* **Network isolation**: Optional network blocking for secure code execution
* **Storage mounting**: Access environment cloud storage with bucket mounting
* **Sidecar support**: Run additional services (Redis, PostgreSQL, etc.)
## Full Documentation
**Complete OpenReward Sandboxes Documentation**
For comprehensive documentation including setup, configuration, lifecycle management, and advanced features, see the [OpenReward Sandboxes section](../concepts/sandboxes#openreward-sandboxes) in the Sandboxes concepts guide.
## Next Steps
* **[Complete Sandbox Documentation](../concepts/sandboxes#openreward-sandboxes)** - Full guide with examples and API reference
* **[Building Agentic Environments](../environments/building-agentic-environments)** - Tutorial showing OpenReward Sandboxes in action
* **[Storage & Buckets](../storage/buckets)** - Configure cloud storage for sandbox access
## Other Sandbox Providers
[ORS](https://openrewardstandard.io) environments work with any sandbox provider. Other popular options include:
* **[E2B Sandboxes](./e2b)** - Cloud-native sandbox environment
* **[Daytona Sandboxes](./daytona)** - Development environment sandboxes
* **[Modal Sandboxes](./modal)** - Serverless compute platform
# Why use Sandboxes?
Source: https://docs.openreward.ai/sandboxes/why-sandboxes
Understand what sandboxes are and why environments need them
## What are Sandboxes?
Sandboxes are isolated execution environments. They are temporary, disposable computers that are spin up when needed and shut down when finished.
When you run AI-generated code, sandboxes provide:
* **Isolation**: Code runs in its own container, separated from your system
* **Security**: Malicious code can't access your files or network
* **Reproducibility**: Each sandbox starts fresh with a clean state
* **Scalability**: Spin up hundreds of sandboxes simultaneously
## Sandboxes in Environments
Environments can be powerful with custom tools alone, but sandboxes unlock a new level of capability by giving agents access to a full computer environment.
Sandboxes enable agents to:
* **Execute code**: Run Python, JavaScript, shell scripts, and more
* **Use command-line tools**: Install packages, run git commands, process files
* **Test solutions**: Write code, run it, debug errors, and iterate
* **Access file systems**: Read, write, and manipulate files across multiple tool calls
For example, in a mathematics environment, an agent might:
1. Write Python code to solve an equation
2. Execute the code in a sandbox
3. Get the result back
4. Submit the final answer
This workflow requires code execution, which sandboxes provide. Without sandboxes, you'd need to define custom tools for mathematical operations instead.
## Sandbox Providers
ORS environments work with any sandbox provider.\*\*
[OpenReward](https://openreward.ai) provides a sandbox solution that plays nicely with mounting agent filesystems. We show how to use this in
the [Sandboxes documentation](../concepts/sandboxes), as well as the [Building agentic environments](../environments/building-agentic-environments) tutorial.
But ORS environments are built to be vendor-agnostic, and work just as well with standard sandbox providers in the community. Some examples include:
* **Using E2B**: See [Using E2B Sandboxes](./e2b)
* **Using Daytona**: See [Using Daytona Sandboxes](./daytona)
* **Using Modal**: See [Using Modal Sandboxes](./modal)
All three providers work the same way from your agent's perspective. The only difference is how you initialize the sandbox in your environment.
**Work in Progress**. Right now OpenReward environment files can only be mounted to OpenReward sandboxes, but we are working on a solution that allows you to mount it
to any sandbox provider. Likewise, we are working on a way to mount external filesystems (e.g. AWS S3, GCP, Azure) to OpenReward sandboxes.
# Cloud Storage
Source: https://docs.openreward.ai/storage/buckets
Using cloud storage with environments and sandboxes
## Overview
OpenReward provides cloud storage integration for both environments and sandboxes.
**Key Concept**: The same storage is accessible in **both** environments and sandboxes.
### Automatic Setup
Each environment automatically includes isolated cloud storage:
* **Created automatically** with your environment
* **Isolated** from other environments
* **Accessible** in both environment server and sandboxes
## Using Storage in Environments
### Accessing Data
Inside your environment server code, access storage at `/orwd_data/`:
```python theme={null}
import os
import json
from pathlib import Path
# Storage is mounted at /orwd_data/
data_dir = Path("/orwd_data")
# List files
files = list(data_dir.glob("*.json"))
print(f"Found {len(files)} JSON files")
# Read file
with open(data_dir / "tasks.json") as f:
tasks = json.load(f)
```
### Common Use Cases
**1. Large Datasets:**
```python theme={null}
# Don't include large datasets in Docker image
# Instead, access from cloud storage
import pandas as pd
dataset = pd.read_csv("/orwd_data/datasets/large_dataset.csv")
```
**2. Configuration Files:**
```python theme={null}
import yaml
# Store environment config in cloud storage
config_path = Path("/orwd_data/config.yaml")
if config_path.exists():
with open(config_path) as f:
config = yaml.safe_load(f)
```
## Using Storage in Sandboxes
### Configuration
Configure storage access via `bucket_config` in `SandboxSettings`:
```python theme={null}
from openreward import OpenReward, SandboxSettings, SandboxBucketConfig
client = OpenReward(api_key="your-api-key")
settings = SandboxSettings(
environment="username/env-name",
image="python:3.11-slim",
machine_size="1:2",
bucket_config=SandboxBucketConfig(
mount_path="/workspace", # Where to mount in container
read_only=True, # Buckets are always read-only
only_dir="datasets/subset", # Optional: mount only subdirectory
implicit_dirs=False # Optional: show all subdirectories
)
)
async with client.sandbox(settings) as sandbox:
# Storage is mounted at /workspace
output, _ = await sandbox.run("ls -la /workspace")
print(output)
```
### SandboxBucketConfig Parameters
| Parameter | Type | Required | Default | Description |
| --------------- | --------------- | -------- | ------- | ---------------------------------------------- |
| `mount_path` | `str` | Yes | - | Path inside container where storage is mounted |
| `read_only` | `Literal[True]` | No | `True` | Buckets are always mounted read-only |
| `only_dir` | `str \| None` | No | `None` | Mount only this subdirectory |
| `implicit_dirs` | `bool` | No | `False` | Show all subdirectories in listings |
### Accessing Data
Inside the sandbox, access files at the configured mount path:
```python theme={null}
# List files
output, _ = await sandbox.run("ls -la /workspace")
# Read file
output, _ = await sandbox.run("cat /workspace/data.csv")
# Process file with Python
output, _ = await sandbox.run("""
python -c "
import pandas as pd
df = pd.read_csv('/workspace/data.csv')
print(f'Loaded {len(df)} rows')
"
""")
```
### Mounting Subdirectories
Mount only a specific subdirectory with `only_dir`:
```python theme={null}
# Storage structure:
# /orwd_data/
# ├── datasets/
# │ ├── train/
# │ │ └── data.csv
# │ └── test/
# │ └── data.csv
# └── models/
# └── checkpoint.pt
# Mount only datasets/train
bucket_config=SandboxBucketConfig(
mount_path="/data",
only_dir="datasets/train"
)
# Inside sandbox:
# /data/
# └── data.csv (only files from datasets/train/)
```
This is useful if you want to mount only a specific subdirectory of the bucket, for example if you want to mount only the training data of a dataset.
### Implicit Directories
Control how directory listings work:
```python theme={null}
# Without implicit_dirs (default, faster):
bucket_config=SandboxBucketConfig(
mount_path="/workspace",
implicit_dirs=False # Default
)
# Shows only directories that explicitly exist
# Better performance
# With implicit_dirs (slower, more complete):
bucket_config=SandboxBucketConfig(
mount_path="/workspace",
implicit_dirs=True
)
# Shows all directories implied by file paths
# More complete directory tree
```
## Next Steps
Learn how to use storage in sandboxes
Understand storage access in environments
Learn about automatic workspace storage
Complete SandboxBucketConfig API documentation
# Debugging Training
Source: https://docs.openreward.ai/training/debugging-training
Common errors encountered during training runs and how to handle them
When running large-scale RL training against OpenReward environment endpoints, you may encounter errors related to capacity limits or resource constraints. This page covers the most common ones and how to handle them.
## Max capacity errors
During training you may see errors like this in your rollout logs:
```
ClientResponseError(429): Environment at maximum capacity.
Ask the environment owner to increase the max pods or sessions per pod.
```
This means the environment can't handle the number of concurrent sessions your training run is requesting. There are two ways to address this:
1. **Increase capacity on the environment.** If you own the environment (or can contact the owner), increase the max pods or sessions per pod in the environment's settings.
2. **Lower your max concurrency.** Reduce the max concurrency in your training settings so fewer sessions are requested at once.
## Memory allocation errors
You may also see errors like this:
```
ClientResponseError(503): Pod unavailable: Container ran out of memory.
Ask the environment owner to increase the memory allocation.
```
This means the environment server itself crashed due to OOM during your rollouts. Unlike a max capacity error, this requires the environment to restart before it can serve requests again. To handle this:
1. **Retry the affected rollouts.** Your training code should detect these failures and redo the rollouts once the environment server comes back up.
2. **Reduce memory pressure on the environment.** This is the longer-term fix. See [Out of memory (OOM) crashes](/environments/debugging-environments#out-of-memory-oom-crashes) in the environment debugging guide for specific strategies - loading less data, using index-based task access, or increasing the environment's memory allocation.
# Training with Miles
Source: https://docs.openreward.ai/training/training-with-miles
Train language models with reinforcement learning using Miles and OpenReward environments
**Research Preview.** Training is in a research preview, meaning we are gaining feedback before a fully supported release. This means capacity is limited in this trial period, so large-scale runs may be unreliable at this point in time. Please give us feedback on our [Discord](https://discord.gg/openreward).
## Goals
* Set up distributed RL training with Miles
* Configure an OpenReward environment for training
* Monitor training progress with WandB
* Train a model on the WhoDunIt environment
## Prerequisites
* [Miles](https://github.com/radixark/miles) installed locally (`pip install -e /path/to/miles`)
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* A [WandB](https://wandb.ai/) account and API key
* Python 3.11+
* NVIDIA GPUs (tested on H100/H200)
## Setup
Miles is a fork of [Slime](https://github.com/THUDM/slime) that adds production-grade stability features for RL post-training. It uses SGLang for fast inference and supports FSDP or Megatron backends for distributed training. Key improvements over Slime include:
* **Graceful OOM recovery** — benign OOMs from variable-length multi-turn rollouts are caught and propagated instead of crashing the job
* **True on-policy with FSDP** — zero train-inference mismatch via aligned numerics (FlashAttention-3, DeepGEMM, batch-invariant kernels)
* **FSDP memory fixes** — reduced excessive memory usage, move-based offloading, host peak memory savings
* **Partial rollout & over-sampling** — handles the long-tail effect in multi-turn RL by over-sampling and recycling half-finished trajectories
In this tutorial, we'll use Miles to train a language model on an OpenReward environment using reinforcement learning with GRPO.
First, clone the [OpenReward cookbook repository](https://github.com/OpenRewardAI/openreward-cookbook) and navigate to the Miles training example:
```bash theme={null}
git clone https://github.com/OpenRewardAI/openreward-cookbook.git
cd openreward-cookbook/training/miles
```
Install the required packages:
```bash theme={null}
pip install -r requirements.txt
```
Or using uv:
```bash theme={null}
uv pip install -r requirements.txt
```
Next, set the required environment variables:
```bash theme={null}
export OPENREWARD_API_KEY=your_openreward_key_here
export WANDB_API_KEY=your_wandb_key_here
export OPENAI_API_KEY=your_openai_key_here # If environments use LLM-based graders
```
## Understanding the Training Pipeline
The training pipeline combines three services:
* **Miles** provides the distributed compute infrastructure for running training (FSDP or Megatron backend) and SGLang for fast inference during rollouts, with production-grade stability features
* **OpenReward** provides the environments and tasks for the agent to learn from
* **WandB** tracks metrics, logs, and training progress
As training runs, Miles will sample multi-turn rollouts from your OpenReward environment, compute rewards using GRPO advantage estimation, and update the model using reinforcement learning. Per-token log probabilities are tracked for importance sampling, and trajectories are uploaded to OpenReward for visualization. Miles' graceful OOM recovery means that if a rare batch exceeds memory, the job won't crash — training continues automatically.
## Selecting an Environment
Browse available environments at [OpenReward](https://openreward.ai/environments):
Let's use the `GeneralReasoning/WhoDunIt` environment for this tutorial. This environment challenges agents to solve mystery scenarios.
Click the copy button to copy the identifier `GeneralReasoning/WhoDunIt` for use in your config.
## Configuration
Training is configured via two files:
### `train_config.yaml` — Environment & agent settings
Open `train_config.yaml` and update the environment configuration to use `GeneralReasoning/WhoDunIt`:
```yaml theme={null}
environments:
GeneralReasoning/WhoDunIt:
splits:
- train
nonterminal_reward: 0.0
reward_reduction: sum
max_turns: 20
```
You can train on multiple environments simultaneously by adding entries:
```yaml theme={null}
environments:
GeneralReasoning/WhoDunIt:
splits: [train]
reward_reduction: sum
max_turns: 20
MATH/GSM8K:
splits: [train]
reward_reduction: mean
max_turns: 10
```
### `run.sh` — Training hyperparameters
All training, optimizer, cluster, and rollout settings are passed via `run.sh` CLI flags:
| Flag | Default | Description |
| ---------------------- | -------------------- | ---------------------------------------------- |
| `--model` | `Qwen/Qwen3-30B-A3B` | HuggingFace checkpoint |
| `--lr` | `1e-5` | Learning rate |
| `--n-samples` | `16` | Rollouts per prompt (for GRPO) |
| `--rollout-batch-size` | `32` | Prompts per rollout batch |
| `--max-response-len` | `4096` | Max response tokens per generation call |
| `--max-tokens-per-gpu` | `8192` | Token cap per GPU in training (OOM prevention) |
| `--temperature` | `1.0` | Sampling temperature |
| `--train-backend` | `fsdp` | `fsdp` or `megatron` |
## Running Training
Training is a two-step process. First, fetch tasks from OpenReward and write a Miles-compatible JSONL dataset:
```bash theme={null}
python prepare_tasks.py --config train_config.yaml --output tasks.jsonl
```
Then, from the Miles repo root, launch training:
```bash theme={null}
cd /path/to/miles
bash /path/to/openreward-cookbook/training/miles/run.sh
```
Common overrides:
```bash theme={null}
# Different model
bash run.sh --model Qwen/Qwen3-4B
# Adjust GPU allocation
bash run.sh --actor-gpus 4 --rollout-gpus 4 --tp 4
# Tune training
bash run.sh --lr 5e-6 --n-samples 8 --rollout-batch-size 16
# Pass arbitrary Miles args after --
bash run.sh -- --context-parallel-size 2 --use-kl-loss --kl-loss-coef 0.01
```
To resume from a checkpoint:
```bash theme={null}
bash run.sh --load /path/to/checkpoints/
```
Miles auto-resumes from the latest checkpoint in `--load` if one exists.
Training will begin and you'll see output in your terminal:
The training process will:
1. Load your model and prepare for distributed training
2. Connect to SGLang for inference
3. Sample multi-turn rollouts from the WhoDunIt environment
4. Compute rewards and update the model using GRPO
5. Log metrics to WandB
6. Save checkpoints periodically
## Monitoring Training
Your training metrics will appear in your WandB dashboard. You can track rewards, response lengths and other key metrics in real-time.
To view your WandB dashboard, go to [https://wandb.ai/](https://wandb.ai/) and navigate to your project. You'll see charts showing:
* Training loss over time
* Average reward per episode
* Success rate on tasks
* Learning rate schedule
Detailed rollout data is uploaded to your OpenReward runs page:
## Additional tips
Some environments require additional secrets, for example environments that use LLM graders or environments that use external search APIs. You can configure these in the `secrets` section of `train_config.yaml`:
```yaml theme={null}
secrets:
openai_api_key: null # null = read from OPENAI_API_KEY env var
```
### Memory considerations
Multi-turn agent rollouts produce long sequences (system prompt + tools + N turns of generation + tool responses). This can cause OOM during training. Key levers:
* **`--max-tokens-per-gpu N`** + **`--use-dynamic-batch-size`**: Caps tokens packed per GPU per training step. Start at `max_response_len` and increase for throughput.
* **`--gradient-checkpointing`**: Trades \~10% speed for significantly less activation memory. Enabled by default in `run.sh`. Recommended for models with large vocabularies (e.g. Qwen3's 152k vocab).
* **`--context-parallel-size N`**: Splits long sequences across N GPUs (requires N actor GPUs).
* **`max_turns` in `train_config.yaml`**: Fewer turns = shorter sequences.
Miles' graceful OOM recovery means that if a rare batch does exceed memory, the job won't crash — the error is propagated and training continues. This is particularly valuable for multi-turn rollouts where sequence length variance is high.
### Known issue: FSDP logging crash
When using a custom generate function with FSDP, you may encounter an `Attribute tokens is not found in packed batch` error. Workaround: wrap the logging call in a try/except in `miles/backends/fsdp_utils/actor.py`:
```python theme={null}
# around line 560, change:
self._log_rollout_data(rollout_id, rollout_data, packed_batches)
# to:
try:
self._log_rollout_data(rollout_id, rollout_data, packed_batches)
except Exception as e:
import logging
logging.getLogger(__name__).warning(f"Failed to log rollout data: {e}")
```
This preserves all training behavior and reward logging - you only lose some per-step rollout metrics in WandB for affected batches.
## Next Steps
Learn how to run evaluations on your trained model
Create custom environments for training
Learn more about Miles' capabilities
# Training with SkyRL
Source: https://docs.openreward.ai/training/training-with-skyrl
Train language models with reinforcement learning using SkyRL and OpenReward environments
Thanks to [@tyfeng1997](https://x.com/tyfeng1997) for contributing the SkyRL × OpenReward integration upstream in [SkyRL PR #1458](https://github.com/NovaSky-AI/SkyRL/pull/1458).
## Goals
* Set up distributed RL training with SkyRL
* Configure an OpenReward environment for training
* Monitor training progress with WandB and OpenReward rollouts
* Train a model on the WhoDunit environment using GRPO
## Prerequisites
* The [SkyRL](https://github.com/NovaSky-AI/SkyRL) repository cloned locally
* A [Modal](https://modal.com/) account and the Modal CLI (`pip install modal && modal setup`)
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* *(Optional)* A [WandB](https://wandb.ai/) account and API key — pass `LOGGER=console` to skip
* Python 3.11+
* NVIDIA GPUs (the example targets 4× A100 via Modal)
## Setup
[SkyRL](https://github.com/NovaSky-AI/SkyRL) is NovaSky-AI's modular full-stack RL training framework for LLMs. It uses Ray for distributed orchestration, vLLM for fast rollout generation, and FSDP2 for distributed training. SkyRL ships with a ready-to-run OpenReward integration under `examples/train_integrations/openreward`, which trains agents on OpenReward environments using GRPO and is set up to launch on Modal-provisioned A100s.
First, clone the SkyRL repository and install the Modal CLI:
```bash theme={null}
git clone https://github.com/NovaSky-AI/SkyRL.git
cd SkyRL
pip install modal && modal setup
```
The example uses [`uv`](https://github.com/astral-sh/uv) to install SkyRL and the OpenReward client inside the Modal container at run time, so there's nothing else to install on your machine.
Export the API keys you'll forward into Modal in the next steps:
```bash theme={null}
export OPENREWARD_API_KEY=your_openreward_key_here
export WANDB_API_KEY=your_wandb_key_here # optional
export OPENREWARD_UPLOAD_ROLLOUT=true # upload rollouts to the OpenReward dashboard
export OPENREWARD_RUN_NAME=skyrl-openreward-whodunit # groups uploads from one training run
```
Each Modal command below passes these variables through inside the `--command` string so the training container can read them.
## Understanding the Training Pipeline
The training pipeline combines three services:
* **SkyRL** provides the GRPO trainer, Ray-based orchestration, vLLM rollout engine, and FSDP2 training backend. The example is configured to run on Modal's GPU infrastructure (4× A100 by default).
* **OpenReward** provides the environments and tasks. A `BaseTextEnv` adapter (`OpenRewardEnv` in `env.py`) wraps OpenReward's session API into SkyRL-Gym, with exponential-backoff retries for transient API errors.
* **WandB** tracks metrics, logs, and training progress.
During training, SkyRL samples multi-turn tool-use rollouts from your OpenReward environment via vLLM, computes GRPO advantages with KL regularization against a reference policy, and updates the model. Trajectories are optionally uploaded to OpenReward so you can inspect each step's tool call, tool result, and reward on the OpenReward dashboard.
## Selecting an Environment
Browse available environments at [OpenReward](https://openreward.ai/environments):
Let's use the `GeneralReasoning/WhoDunit` environment for this tutorial. This environment challenges agents to solve murder mystery puzzles by gathering information about suspects, weapons, and locations.
Click the copy button to copy the identifier `GeneralReasoning/WhoDunit` for use in your dataset preparation command.
## Configuration
Training is controlled by two things: the task dataset produced by `prepare_tasks.py`, and CLI overrides passed through `run_openreward.sh` to SkyRL's config system. The script also reads a few env vars (`MODEL`, `NUM_GPUS`, `LOGGER`, `RUN_NAME`) for the most common knobs.
### `prepare_tasks.py` — task dataset
This script queries OpenReward for tasks, opens a temporary session per task to fetch the initial prompt and tool specs, and writes a Parquet dataset that SkyRL loads at training time. It runs as a one-shot job on a small Modal GPU:
```bash theme={null}
MODAL_GPU=L4:1 modal run examples/train_integrations/modal/main.py \
--command "OPENREWARD_API_KEY=$OPENREWARD_API_KEY \
uv run --isolated --with openreward --with pyarrow \
python examples/train_integrations/openreward/prepare_tasks.py \
--env GeneralReasoning/WhoDunit \
--split train \
--max-tasks 50 \
--output /root/data/openreward/train.parquet"
```
You can train on multiple environments simultaneously by passing `--env` more than once — the dataset row's `env_name` column tells `OpenRewardEnv` which environment to open at rollout time:
```bash theme={null}
MODAL_GPU=L4:1 modal run examples/train_integrations/modal/main.py \
--command "OPENREWARD_API_KEY=$OPENREWARD_API_KEY \
uv run --isolated --with openreward --with pyarrow \
python examples/train_integrations/openreward/prepare_tasks.py \
--env GeneralReasoning/WhoDunit \
--env GeneralReasoning/CTF \
--split train \
--output /root/data/openreward/train.parquet"
```
### `run_openreward.sh` — trainer flags
All training, optimizer, generator, and placement settings are overridable. The script forwards any positional args (`$@`) to SkyRL's config system, so you can append `key=value` overrides to the `bash run_openreward.sh ...` line. The most common ones:
| Flag | Default | Description |
| ------------------------------------------------- | -------------------------- | ------------------------------------------- |
| `MODEL` (env var) | `Qwen/Qwen2.5-3B-Instruct` | HuggingFace checkpoint (set via env var) |
| `NUM_GPUS` (env var) | `4` | GPUs for colocated policy + ref + inference |
| `trainer.epochs` | `3` | Training epochs over the dataset |
| `trainer.train_batch_size` | `16` | Unique prompts per training step |
| `trainer.policy.optimizer_config.lr` | `1.0e-6` | Learning rate |
| `trainer.algorithm.advantage_estimator` | `grpo` | RL advantage estimator |
| `trainer.algorithm.kl_loss_coef` | `0.001` | KL regularization coefficient |
| `trainer.strategy` | `fsdp2` | Distributed training strategy |
| `trainer.max_prompt_length` | `2048` | Max prompt tokens |
| `generator.inference_engine.num_engines` | `4` | Number of vLLM engines |
| `generator.inference_engine.tensor_parallel_size` | `1` | TP size per engine |
| `generator.n_samples_per_prompt` | `4` | Rollouts per prompt (GRPO group size) |
| `generator.max_turns` | `10` | Max agent-environment turns per episode |
| `generator.sampling_params.temperature` | `1.0` | Sampling temperature |
| `generator.sampling_params.max_generate_length` | `1024` | Max generation tokens per turn |
| `environment.env_class` | `openreward` | Always `openreward` for this example |
Total rollouts per step = `train_batch_size × n_samples_per_prompt = 16 × 4 = 64`.
## Running Training
Training is a two-step process. First, fetch tasks from OpenReward into a Parquet dataset (see [Configuration](#configuration) above for the full command). Then launch training on a 4× A100 Modal job:
```bash theme={null}
MODAL_GPU=A100:4 modal run examples/train_integrations/modal/main.py \
--command "OPENREWARD_API_KEY=$OPENREWARD_API_KEY \
WANDB_API_KEY=$WANDB_API_KEY \
OPENREWARD_UPLOAD_ROLLOUT=true \
OPENREWARD_RUN_NAME=$OPENREWARD_RUN_NAME \
bash examples/train_integrations/openreward/run_openreward.sh"
```
Common overrides — append `key=value` arguments to the `bash` line and they'll be forwarded to SkyRL's config system:
```bash theme={null}
# Shorter run, fewer turns per episode
MODAL_GPU=A100:4 modal run examples/train_integrations/modal/main.py \
--command "OPENREWARD_API_KEY=$OPENREWARD_API_KEY \
bash examples/train_integrations/openreward/run_openreward.sh \
trainer.epochs=2 generator.max_turns=8"
# Larger model (use the MODEL env var the script reads)
MODAL_GPU=A100:4 modal run examples/train_integrations/modal/main.py \
--command "OPENREWARD_API_KEY=$OPENREWARD_API_KEY MODEL=Qwen/Qwen2.5-7B-Instruct \
bash examples/train_integrations/openreward/run_openreward.sh"
# Tune GRPO group size and batch
MODAL_GPU=A100:4 modal run examples/train_integrations/modal/main.py \
--command "OPENREWARD_API_KEY=$OPENREWARD_API_KEY \
bash examples/train_integrations/openreward/run_openreward.sh \
generator.n_samples_per_prompt=8 trainer.train_batch_size=32"
```
The training process will:
1. Spin up the Modal container, install SkyRL + OpenReward via `uv`, and initialize Ray
2. Register `OpenRewardEnv` with SkyRL-Gym inside each Ray worker
3. Load the policy with FSDP2 and start the colocated vLLM inference engines
4. Sample multi-turn tool-use rollouts from the WhoDunit environment
5. Compute GRPO advantages and update the policy with KL regularization against the reference model
6. Log metrics to WandB and upload rollouts to OpenReward
7. Save FSDP2 checkpoints periodically
## Monitoring Training
Your training metrics will appear in your WandB dashboard. You can track rewards, episode length, and pass-rate metrics in real time.
Key SkyRL + OpenReward metrics:
* `reward/avg_pass_at_4` — success rate across the 4 GRPO rollouts per prompt
* `reward/avg_raw_reward` — mean raw reward across all episodes
* `reward/mean_positive_reward` — mean reward on successful episodes
* `environment/turns` — average number of turns per episode
* `environment/total_reward` / `environment/num_rewards` — cumulative environment signal
In the reference PR run (Qwen2.5-3B-Instruct, 100 WhoDunit tasks, 3 epochs = 18 steps), `avg_pass_at_4` climbed from \~0.70 to \~0.90 and `mean_positive_reward` improved from \~0.09 to \~0.16 — clear learning signal even from a small 3B model with limited data.
Detailed rollout data is uploaded to your OpenReward runs page so you can inspect each trajectory:
Click a rollout to see every tool call, tool result, and per-step reward:
## Additional tips
### Rollout visualization
Rollout upload is controlled by the `OPENREWARD_UPLOAD_ROLLOUT` environment variable and `OPENREWARD_RUN_NAME` groups rollouts from a single training run together. Set `OPENREWARD_UPLOAD_ROLLOUT=false` to skip uploads.
### Retry & resilience
`OpenRewardEnv` wraps OpenReward API calls in an exponential-backoff retry (handling 502/503/429 and connection errors), so transient service hiccups won't crash a long training run.
### Memory considerations
Multi-turn tool-use rollouts produce long sequences (system prompt + tool specs + N turns of generation + tool responses). If you hit OOM during training:
* Reduce `trainer.micro_forward_batch_size_per_gpu` and `trainer.micro_train_batch_size_per_gpu` (default `4` each in `run_openreward.sh`)
* Reduce `trainer.max_prompt_length` or `generator.sampling_params.max_generate_length`
* The reference model already runs with `trainer.ref.fsdp_config.cpu_offload=true` by default; you can also offload the policy with `trainer.policy.fsdp_config.cpu_offload=true`
* For larger models, increase tensor parallelism: `generator.inference_engine.tensor_parallel_size=2`
### Disabling WandB
If you don't want to use WandB, pass `LOGGER=console` and the script will print metrics to stdout instead:
```bash theme={null}
MODAL_GPU=A100:4 modal run examples/train_integrations/modal/main.py \
--command "OPENREWARD_API_KEY=$OPENREWARD_API_KEY LOGGER=console \
bash examples/train_integrations/openreward/run_openreward.sh"
```
### Secrets for environments with external services
Some environments require additional secrets (e.g. `OPENAI_API_KEY` for LLM graders, search API keys). Forward them through the Modal `--command` string the same way as `OPENREWARD_API_KEY` so the training container picks them up.
## Next Steps
Learn how to run evaluations on your trained model
Create custom environments for training
Learn more about SkyRL's capabilities
# Training with Slime
Source: https://docs.openreward.ai/training/training-with-slime
Train language models with reinforcement learning using Slime and OpenReward environments
**Research Preview.** Training is in a research preview, meaning we are gaining feedback before a fully supported release. This means capacity is limited in this trial period, so large-scale runs may be unreliable at this point in time. Please give us feedback on our [Discord](https://discord.gg/openreward).
## Goals
* Set up distributed RL training with Slime
* Configure an OpenReward environment for training
* Monitor training progress with WandB
* Train a model on the WhoDunIt environment
## Prerequisites
* [Slime](https://github.com/THUDM/slime) installed locally (`pip install -e /path/to/slime`)
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* A [WandB](https://wandb.ai/) account and API key
* Python 3.11+
* NVIDIA GPUs (tested on H100/H200)
## Setup
Slime is an RL post-training framework from Tsinghua University. It uses SGLang for fast inference and supports FSDP or Megatron backends for distributed training. In this tutorial, we'll use it to train a language model on an OpenReward environment using reinforcement learning with GRPO.
First, clone the [OpenReward cookbook repository](https://github.com/OpenRewardAI/openreward-cookbook) and navigate to the Slime training example:
```bash theme={null}
git clone https://github.com/OpenRewardAI/openreward-cookbook.git
cd openreward-cookbook/training/slime
```
Install the required packages:
```bash theme={null}
pip install -r requirements.txt
```
Or using uv:
```bash theme={null}
uv pip install -r requirements.txt
```
Next, set the required environment variables:
```bash theme={null}
export OPENREWARD_API_KEY=your_openreward_key_here
export WANDB_API_KEY=your_wandb_key_here
export OPENAI_API_KEY=your_openai_key_here # If environments use LLM-based graders
```
## Understanding the Training Pipeline
The training pipeline combines three services:
* **Slime** provides the distributed compute infrastructure for running training (FSDP or Megatron backend) and SGLang for fast inference during rollouts
* **OpenReward** provides the environments and tasks for the agent to learn from
* **WandB** tracks metrics, logs, and training progress
As training runs, Slime will sample multi-turn rollouts from your OpenReward environment, compute rewards using GRPO advantage estimation, and update the model using reinforcement learning. Per-token log probabilities are tracked for importance sampling, and trajectories are uploaded to OpenReward for visualization.
## Selecting an Environment
Browse available environments at [OpenReward](https://openreward.ai/environments):
Let's use the `GeneralReasoning/WhoDunIt` environment for this tutorial. This environment challenges agents to solve mystery scenarios.
Click the copy button to copy the identifier `GeneralReasoning/WhoDunIt` for use in your config.
## Configuration
Training is configured via two files:
### `train_config.yaml` — Environment & agent settings
Open `train_config.yaml` and update the environment configuration to use `GeneralReasoning/WhoDunIt`:
```yaml theme={null}
environments:
GeneralReasoning/WhoDunIt:
splits:
- train
nonterminal_reward: 0.0
reward_reduction: sum
max_turns: 20
```
You can train on multiple environments simultaneously by adding entries:
```yaml theme={null}
environments:
GeneralReasoning/WhoDunIt:
splits: [train]
reward_reduction: sum
max_turns: 20
MATH/GSM8K:
splits: [train]
reward_reduction: mean
max_turns: 10
```
### `run.sh` — Training hyperparameters
All training, optimizer, cluster, and rollout settings are passed via `run.sh` CLI flags:
| Flag | Default | Description |
| ---------------------- | -------------------- | ---------------------------------------------- |
| `--model` | `Qwen/Qwen3-30B-A3B` | HuggingFace checkpoint |
| `--lr` | `1e-5` | Learning rate |
| `--n-samples` | `16` | Rollouts per prompt (for GRPO) |
| `--rollout-batch-size` | `32` | Prompts per rollout batch |
| `--max-response-len` | `4096` | Max response tokens per generation call |
| `--max-tokens-per-gpu` | `8192` | Token cap per GPU in training (OOM prevention) |
| `--temperature` | `1.0` | Sampling temperature |
| `--train-backend` | `fsdp` | `fsdp` or `megatron` |
## Running Training
Training is a two-step process. First, fetch tasks from OpenReward and write a Slime-compatible JSONL dataset:
```bash theme={null}
python prepare_tasks.py --config train_config.yaml --output tasks.jsonl
```
Then, from the Slime repo root, launch training:
```bash theme={null}
cd /path/to/slime
bash /path/to/openreward-cookbook/training/slime/run.sh
```
Common overrides:
```bash theme={null}
# Different model
bash run.sh --model Qwen/Qwen3-4B
# Adjust GPU allocation
bash run.sh --actor-gpus 4 --rollout-gpus 4 --tp 4
# Tune training
bash run.sh --lr 5e-6 --n-samples 8 --rollout-batch-size 16
# Enable gradient checkpointing (reduces memory, ~10% slower)
bash run.sh -- --gradient-checkpointing
# Pass arbitrary Slime args after --
bash run.sh -- --context-parallel-size 2 --use-kl-loss --kl-loss-coef 0.01
```
To resume from a checkpoint:
```bash theme={null}
bash run.sh --load /path/to/checkpoints/
```
Training will begin and you'll see output in your terminal:
The training process will:
1. Load your model and prepare for distributed training
2. Connect to SGLang for inference
3. Sample multi-turn rollouts from the WhoDunIt environment
4. Compute rewards and update the model using GRPO
5. Log metrics to WandB
6. Save checkpoints periodically
## Monitoring Training
Your training metrics will appear in your WandB dashboard. You can track rewards, response lengths and other key metrics in real-time.
To view your WandB dashboard, go to [https://wandb.ai/](https://wandb.ai/) and navigate to your project. You'll see charts showing:
* Training loss over time
* Average reward per episode
* Success rate on tasks
* Learning rate schedule
Detailed rollout data is uploaded to your OpenReward runs page:
## Additional tips
Some environments require additional secrets, for example environments that use LLM graders or environments that use external search APIs. You can configure these in the `secrets` section of `train_config.yaml`:
```yaml theme={null}
secrets:
openai_api_key: null # null = read from OPENAI_API_KEY env var
```
### Memory considerations
Multi-turn agent rollouts produce long sequences (system prompt + tools + N turns of generation + tool responses). This can cause OOM during training. Key levers:
* **`--max-tokens-per-gpu N`** + **`--use-dynamic-batch-size`**: Caps tokens packed per GPU per training step. Start at `max_response_len` and increase for throughput.
* **`--gradient-checkpointing`**: Trades \~10% speed for significantly less activation memory. Recommended for models with large vocabularies (e.g. Qwen3's 152k vocab).
* **`--context-parallel-size N`**: Splits long sequences across N GPUs (requires N actor GPUs).
* **`max_turns` in `train_config.yaml`**: Fewer turns = shorter sequences.
## Next Steps
Learn how to run evaluations on your trained model
Create custom environments for training
Learn more about Slime's capabilities
# Training with Tinker
Source: https://docs.openreward.ai/training/training-with-tinker
Train language models with reinforcement learning using Tinker and OpenReward environments
**Research Preview.** Training is in a research preview, meaning we are gaining feedback before a fully supported release. This means capacity is limited in this trial period, so large-scale runs may be unreliable at this point in time. Please give us feedback on our [Discord](https://discord.gg/openreward).
## Goals
* Set up distributed RL training with Tinker
* Configure an OpenReward environment for training
* Monitor training progress with WandB
* Train a model on the WhoDunIt environment
## Prerequisites
* A [Tinker](https://tinker-docs.thinkingmachines.ai/) account and API key
* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* A [WandB](https://wandb.ai/) account and API key
* Python 3.11+
## Setup
Tinker is a flexible API for efficiently fine-tuning open source models with LoRA. In this tutorial, we'll use it to train a language model on an OpenReward environment using reinforcement learning.
First, clone the [OpenReward cookbook repository](https://github.com/OpenRewardAI/openreward-cookbook) and navigate to the Tinker training example:
```bash theme={null}
git clone https://github.com/OpenRewardAI/openreward-cookbook.git
cd openreward-cookbook/training/tinker
```
Install the required packages:
```bash theme={null}
pip install -r requirements.txt
```
Or using uv:
```bash theme={null}
uv pip install -r requirements.txt
```
Next, create a `.env` file with your API credentials:
```bash theme={null}
TINKER_API_KEY=your_tinker_key_here
OPENREWARD_API_KEY=your_openreward_key_here
WANDB_API_KEY=your_wandb_key_here
```
## Understanding the Training Pipeline
The training pipeline combines three services:
* **Tinker** provides the distributed compute infrastructure for running training
* **OpenReward** provides the environments and tasks for the agent to learn from
* **WandB** tracks metrics, logs, and training progress
As training runs, Tinker will sample rollouts from your OpenReward environment, compute rewards, and update the model using reinforcement learning. All metrics are logged to WandB for monitoring.
## Selecting an Environment
Browse available environments at [OpenReward](https://openreward.ai/environments):
Let's use the `GeneralReasoning/WhoDunIt` environment for this tutorial. This environment challenges agents to solve mystery scenarios.
Click the copy button to copy the identifier `GeneralReasoning/WhoDunIt` for use in your config.
## Configuration
Now we'll update the configuration file for the training run. Open `tinker-config.yaml` and update the environment configuration to use `GeneralReasoning/WhoDunIt`:
```yaml theme={null}
...
# Environment Configuration
environments:
GeneralReasoning/WhoDunIt:
splits:
train:
shuffle: true
num_samples: null
num_rollouts: 16
max_failing_rollouts: 2
temperature: 1.0
reward_reduction: "sum"
nonterminal_reward: 0.0
```
There are other settings you can modify here, including:
* `wandb_project_name`: The WandB project where metrics will be logged
* `wandb_run_name`: A name for this specific training run
* `model_name`: The base model to fine-tune (using LoRA)
* `lora_rank`: The rank for LoRA adapters (lower = fewer parameters)
* `batch_size`: Number of samples per training batch
* `learning_rate`: Learning rate for the optimizer
* `save_every`: Save a checkpoint every N batches
* `num_rollouts`: Number of rollouts to collect per environment per batch
* `temperature`: Sampling temperature for the model (1.0 = default)
## Running Training
Now we can start training. The cookbook includes a `main.py` file with the complete training loop.
Run training with:
```bash theme={null}
python main.py --config_path tinker-config.yaml
```
Training will begin and you'll see output in your terminal:
The training process will:
1. Load your model and initialize LoRA adapters
2. Connect to Tinker's infrastructure
3. Sample rollouts from the WhoDunIt environment
4. Compute rewards and update the model
5. Log metrics to WandB
6. Save checkpoints periodically
## Monitoring Training
Your training metrics will appear in your WandB dashboard. You can track rewards, response lengths and other key metrics in real-time.
To view your WandB dashboard, go to [https://wandb.ai/](https://wandb.ai/) and navigate to your project. You'll see charts showing:
* Training loss over time
* Average reward per episode
* Success rate on tasks
* Learning rate schedule
Detailed rollout data is uploaded to your OpenReward runs page:
Local logs are saved to the path specified in your config (`/tmp/tinker-rl` in our example). These contain detailed information about each training step.
## Additional tips
Some environments require additional secrets, for example environments that use LLM graders or environments that use external search APIs.
You can configure these additional secrets in the tinker-config.yaml:
```bash theme={null}
# secrets:
# openai_api_key: "sk-different-key-for-this-env"
```
## Next Steps
Learn how to run evaluations on your trained model
Create custom environments for training
Learn more about Tinker's capabilities