> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openreward.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Harness Quickstart

> Run agent harnesses against OpenReward environments with firehorse

[Firehorse](https://github.com/GeneralReasoning/firehorse) is a library for running agent harnesses against any OpenReward environment. It works by composing the appropriate [harness toolset](/harnesses/harness-toolsets) with the environment, connecting the harness agent, and orchestrating the agent loop.

## Prerequisites

* An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys)
* An API key for your model provider (Anthropic, OpenAI, Google, or OpenRouter)
* Python 3.10+

## Installation

```bash theme={null}
pip install firehorse-cli
```

For specific agent types, you may also need the corresponding CLI tool installed:

| Agent | Additional requirement |
| - | - |
| `claude-code` | [Claude Code](https://docs.anthropic.com/en/docs/claude-code) v2.1.88+ |
| `codex` | [Codex CLI](https://github.com/openai/codex) v0.121.0+ |
| `gemini` | [Gemini CLI](https://github.com/google-gemini/gemini-cli) |
| `resum`, `react` | None (API-only, no CLI needed) |

## Your First Evaluation

Set your API keys:

```bash theme={null}
export OPENREWARD_API_KEY='your-openreward-api-key'
export ANTHROPIC_API_KEY='your-anthropic-api-key'
```

Run an evaluation:

```bash theme={null}
firehorse \
  --env GeneralReasoning/terminal-bench-2-verified \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6 \
  --split test \
  --max-tasks 5 \
  --output-dir ./results
```

This launches Claude Code as a subprocess against the terminal-bench-2-verified environment, running 5 tasks from the test split. Results are written to `./results`.

## Available Agents

Firehorse ships five agent harnesses, each with a different architecture:

| Agent | Approach | Description |
| - | - | - |
| `resum` | API + context compaction | Default agent. Implements a ReAct loop with 3-layer context compaction for long trials. Multi-provider support. |
| `claude-code` | Subprocess + MCP | Launches Claude Code CLI per trial. Disables filesystem builtins and replaces them with sandboxed MCP tools. Supports extended thinking via `--effort`. |
| `codex` | Subprocess + MCP | Launches OpenAI Codex CLI. Uses a single `bash` tool surface. Read-only filesystem sandbox mode. |
| `gemini` | Subprocess + MCP | Launches Google Gemini CLI. Pre-builds tool specs to avoid CLI discovery timeout. |
| `react` | API-direct | Direct LLM API integration with a straightforward reason-act loop. Supports Anthropic, OpenAI, Google, and OpenRouter. No subprocess overhead. |

**Subprocess + MCP agents** (claude-code, codex, gemini) launch the respective CLI as a child process and proxy environment tools via [MCP](https://modelcontextprotocol.io). The agent's built-in filesystem tools are disabled and replaced with sandbox-backed equivalents.

**API-direct agents** (resum, react) call LLM APIs directly and execute tool calls via the OpenReward session. No local CLI is required.

## Key Options

| Option | Description | Default |
| - | - | - |
| `--env` | Environment name (e.g. `GeneralReasoning/CTF`) | Required |
| `--agent` | Agent type (`resum`, `claude-code`, `codex`, `react`, `gemini`) | `resum` |
| `--model` | Model identifier with provider prefix (e.g. `anthropic/claude-sonnet-4-6`, `openrouter/deepseek/deepseek-v3.2`) | Required |
| `--effort` | Reasoning depth: `none`, `low`, `medium`, `high`, `max`, `xhigh` | Provider default |
| `--n-concurrent` | Number of parallel trials | `1` |
| `--max-tasks` | Limit number of tasks to evaluate | All tasks |
| `--max-turns` | Maximum tool calls per trial | `100` |
| `--split` | Data split to evaluate (`train`, `test`, `validation`) | `train` |
| `--variant` | Environment variant | Default variant |
| `--output-dir` | Directory for results and trajectory logs | `./output` |
| `--provider-url` | Custom model provider endpoint | — |
| `--secrets` | Extra secrets as `KEY=VALUE` pairs | — |

### Model Identifiers

Models are specified with a provider prefix:

```bash theme={null}
# Anthropic
--model anthropic/claude-sonnet-4-6

# OpenAI
--model openai/gpt-5.4

# Google
--model google/gemini-2.5-flash

# OpenRouter (any model)
--model openrouter/deepseek/deepseek-v3.2
```

### Effort Levels

The `--effort` flag controls reasoning depth and maps to each provider's native mechanism:

| Effort | Anthropic | OpenAI | Google |
| - | - | - | - |
| `low` | Adaptive thinking (low) | `reasoning_effort: low` | `thinking_level: low` |
| `medium` | Adaptive thinking (medium) | `reasoning_effort: medium` | `thinking_level: medium` |
| `high` | Adaptive thinking (high) | `reasoning_effort: high` | `thinking_level: high` |
| `max` | Max thinking tokens | — | — |
| `xhigh` | — | `reasoning_effort: xhigh` | — |

## Understanding the Output

Firehorse produces three types of output per trial:

* **Result JSON** — final metrics: total reward, tool call count, token usage, API cost, duration
* **Trajectory JSONL** — full event log capturing reasoning steps, tool calls, and tool results
* **Aggregate summary** — statistics across all trials

Example output structure:

```
results/
├── trial_0/
│   ├── result.json          # Final metrics
│   ├── trajectory.jsonl     # Full event log
│   └── rewards.jsonl        # Timestamped reward signals
├── trial_1/
│   └── ...
└── summary.json             # Aggregate statistics
```

## Examples

### Evaluate with OpenRouter

```bash theme={null}
export OPENROUTER_API_KEY='your-key'

firehorse \
  --env GeneralReasoning/CTF \
  --agent react \
  --model openrouter/moonshotai/kimi-k2.6 \
  --n-concurrent 4 \
  --max-tasks 20
```

### High-effort Claude Code evaluation

```bash theme={null}
export ANTHROPIC_API_KEY='your-key'

firehorse \
  --env Eigent/SETA \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6 \
  --effort high \
  --n-concurrent 8 \
  --split train
```

### Codex on a custom environment

```bash theme={null}
export OPENAI_API_KEY='your-key'

firehorse \
  --env MyOrg/MyEnv \
  --agent codex \
  --model openai/gpt-5.4 \
  --variant hard \
  --max-turns 50
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Harness Toolsets" icon="toolbox" href="/harnesses/harness-toolsets">
    Configure agent-native tool surfaces for your environments
  </Card>

  <Card title="Your First Evaluation" icon="trophy" href="/environments/evaluation">
    Build a custom evaluation environment
  </Card>

  <Card title="Building Agentic Environments" icon="robot" href="/environments/building-agentic-environments">
    Create sandbox-based environments for agent tasks
  </Card>

  <Card title="Using Toolsets" icon="wrench" href="/environments/using-toolsets">
    Compose reusable tool collections into environments
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.