> ## Documentation Index > Fetch the complete documentation index at: https://docs.openreward.ai/llms.txt > Use this file to discover all available pages before exploring further. # Harness Quickstart > Run agent harnesses against OpenReward environments with firehorse [Firehorse](https://github.com/GeneralReasoning/firehorse) is a library for running agent harnesses against any OpenReward environment. It works by composing the appropriate [harness toolset](/harnesses/harness-toolsets) with the environment, connecting the harness agent, and orchestrating the agent loop. ## Prerequisites * An OpenReward [account](https://openreward.ai/) and [API key](https://openreward.ai/keys) * An API key for your model provider (Anthropic, OpenAI, Google, or OpenRouter) * Python 3.10+ ## Installation ```bash theme={null} pip install firehorse-cli ``` For specific agent types, you may also need the corresponding CLI tool installed: | Agent | Additional requirement | | ---------------- | ---------------------------------------------------------------------- | | `claude-code` | [Claude Code](https://docs.anthropic.com/en/docs/claude-code) v2.1.88+ | | `codex` | [Codex CLI](https://github.com/openai/codex) v0.121.0+ | | `gemini` | [Gemini CLI](https://github.com/google-gemini/gemini-cli) | | `resum`, `react` | None (API-only, no CLI needed) | ## Your First Evaluation Set your API keys: ```bash theme={null} export OPENREWARD_API_KEY='your-openreward-api-key' export ANTHROPIC_API_KEY='your-anthropic-api-key' ``` Run an evaluation: ```bash theme={null} firehorse \ --env GeneralReasoning/terminal-bench-2-verified \ --agent claude-code \ --model anthropic/claude-sonnet-4-6 \ --split test \ --max-tasks 5 \ --output-dir ./results ``` This launches Claude Code as a subprocess against the terminal-bench-2-verified environment, running 5 tasks from the test split. Results are written to `./results`. ## Available Agents Firehorse ships five agent harnesses, each with a different architecture: | Agent | Approach | Description | | ------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `resum` | API + context compaction | Default agent. Implements a ReAct loop with 3-layer context compaction for long trials. Multi-provider support. | | `claude-code` | Subprocess + MCP | Launches Claude Code CLI per trial. Disables filesystem builtins and replaces them with sandboxed MCP tools. Supports extended thinking via `--effort`. | | `codex` | Subprocess + MCP | Launches OpenAI Codex CLI. Uses a single `bash` tool surface. Read-only filesystem sandbox mode. | | `gemini` | Subprocess + MCP | Launches Google Gemini CLI. Pre-builds tool specs to avoid CLI discovery timeout. | | `react` | API-direct | Direct LLM API integration with a straightforward reason-act loop. Supports Anthropic, OpenAI, Google, and OpenRouter. No subprocess overhead. | **Subprocess + MCP agents** (claude-code, codex, gemini) launch the respective CLI as a child process and proxy environment tools via [MCP](https://modelcontextprotocol.io). The agent's built-in filesystem tools are disabled and replaced with sandbox-backed equivalents. **API-direct agents** (resum, react) call LLM APIs directly and execute tool calls via the OpenReward session. No local CLI is required. ## Key Options | Option | Description | Default | | ---------------- | --------------------------------------------------------------------------------------------------------------- | ---------------- | | `--env` | Environment name (e.g. `GeneralReasoning/CTF`) | Required | | `--agent` | Agent type (`resum`, `claude-code`, `codex`, `react`, `gemini`) | `resum` | | `--model` | Model identifier with provider prefix (e.g. `anthropic/claude-sonnet-4-6`, `openrouter/deepseek/deepseek-v3.2`) | Required | | `--effort` | Reasoning depth: `none`, `low`, `medium`, `high`, `max`, `xhigh` | Provider default | | `--n-concurrent` | Number of parallel trials | `1` | | `--max-tasks` | Limit number of tasks to evaluate | All tasks | | `--max-turns` | Maximum tool calls per trial | `100` | | `--split` | Data split to evaluate (`train`, `test`, `validation`) | `train` | | `--variant` | Environment variant | Default variant | | `--output-dir` | Directory for results and trajectory logs | `./output` | | `--provider-url` | Custom model provider endpoint | — | | `--secrets` | Extra secrets as `KEY=VALUE` pairs | — | ### Model Identifiers Models are specified with a provider prefix: ```bash theme={null} # Anthropic --model anthropic/claude-sonnet-4-6 # OpenAI --model openai/gpt-5.4 # Google --model google/gemini-2.5-flash # OpenRouter (any model) --model openrouter/deepseek/deepseek-v3.2 ``` ### Effort Levels The `--effort` flag controls reasoning depth and maps to each provider's native mechanism: | Effort | Anthropic | OpenAI | Google | | -------- | -------------------------- | -------------------------- | ------------------------ | | `low` | Adaptive thinking (low) | `reasoning_effort: low` | `thinking_level: low` | | `medium` | Adaptive thinking (medium) | `reasoning_effort: medium` | `thinking_level: medium` | | `high` | Adaptive thinking (high) | `reasoning_effort: high` | `thinking_level: high` | | `max` | Max thinking tokens | — | — | | `xhigh` | — | `reasoning_effort: xhigh` | — | ## Understanding the Output Firehorse produces three types of output per trial: * **Result JSON** — final metrics: total reward, tool call count, token usage, API cost, duration * **Trajectory JSONL** — full event log capturing reasoning steps, tool calls, and tool results * **Aggregate summary** — statistics across all trials Example output structure: ``` results/ ├── trial_0/ │ ├── result.json # Final metrics │ ├── trajectory.jsonl # Full event log │ └── rewards.jsonl # Timestamped reward signals ├── trial_1/ │ └── ... └── summary.json # Aggregate statistics ``` ## Examples ### Evaluate with OpenRouter ```bash theme={null} export OPENROUTER_API_KEY='your-key' firehorse \ --env GeneralReasoning/CTF \ --agent react \ --model openrouter/moonshotai/kimi-k2.6 \ --n-concurrent 4 \ --max-tasks 20 ``` ### High-effort Claude Code evaluation ```bash theme={null} export ANTHROPIC_API_KEY='your-key' firehorse \ --env Eigent/SETA \ --agent claude-code \ --model anthropic/claude-sonnet-4-6 \ --effort high \ --n-concurrent 8 \ --split train ``` ### Codex on a custom environment ```bash theme={null} export OPENAI_API_KEY='your-key' firehorse \ --env MyOrg/MyEnv \ --agent codex \ --model openai/gpt-5.4 \ --variant hard \ --max-turns 50 ``` ## Next Steps Configure agent-native tool surfaces for your environments Build a custom evaluation environment Create sandbox-based environments for agent tasks Compose reusable tool collections into environments