Chat Backends

Goals

Learn how to build composable chat backends for environments, for use cases like grading and user simulations
Understand why this traffic should not silently follow OPENAI_* environment variables

Prerequisites

openreward>=0.1.153
Familiarity with LLM graders

Introduction

Environments can use a regular provider SDK (OpenAI, Anthropic, Google, and so on) directly, as the LLM graders tutorial shows. For a model call that only ever runs against one provider, that works fine. This applies to any model the environment itself calls: judges, rubric graders, simulated users, and scripted opponents. The problem appears when your environment runs inside a larger system that reconfigures provider clients through the process environment. Training harnesses commonly inject OPENAI_BASE_URL (and OPENAI_API_KEY) into the environment container so that policy traffic is served by an internal inference gateway. The openai SDK reads those variables implicitly, so a grader built as a plain openai.AsyncClient(api_key=...) gets silently repointed too. If the gateway does not serve your judge model, the judge fails, the failure is usually swallowed by retry logic, and every rollout quietly scores zero. openreward.chat_backends makes grader clients compose with that kind of orchestration instead of fighting it:

the environment author writes one call (resolve_backend(secrets=secrets)) and never hardcodes a provider client;
the operator decides at runtime where grader traffic goes, separately from policy traffic, using CHAT_* settings;
the provider is swappable (CHAT_PROVIDER, register_backend) without touching grading code.

Basic usage

Build the grader client in your environment’s __init__ and call it in your grading code:

from openreward.chat_backends import resolve_backend

class MyEnvironment(Environment):
    def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
        super().__init__(task_spec)
        # Raises ValueError if no key is available: a grader without
        # credentials should fail loudly at session start.
        self.judge_client = resolve_backend(secrets=secrets)

resp = await self.judge_client.create(
    model="gpt-5-mini",
    messages=messages,                  # OpenAI chat format, including image_url blocks
    max_completion_tokens=1200,
)
verdict = parse(resp.text)              # normalised ChatCompletion: .text, .model,
                                        # .finish_reason, .usage, .raw

create() accepts extra provider kwargs verbatim (for example response_format={"type": "json_object"}) and raises provider exceptions unchanged, so your existing retry and fail-closed logic keeps working.

How resolution works

The OpenAI backend has two modes, decided by whether any CHAT_* setting is present:

Mode	Trigger	Base URL	API key
Legacy	No `CHAT_*` setting anywhere	`OPENAI_BASE_URL` (or the OpenAI default)	`openai_api_key` secret, then `OPENAI_API_KEY`
Dedicated	`chat_api_key` secret, `CHAT_API_KEY`, or `CHAT_BASE_URL` set	`CHAT_BASE_URL` (default `https://api.openai.com/v1`)	`chat_api_key` secret, then `CHAT_API_KEY`, then the `openai_api_key` secret

With nothing configured, behaviour is identical to a plain client: existing sessions keep working unchanged. The moment an operator sets one CHAT_* variable, the grader ignores the OPENAI_* environment entirely. For a training run that overrides OPENAI_BASE_URL, a single extra override is enough to send judges back to the real endpoint:

CHAT_BASE_URL=https://api.openai.com/v1    # key still comes from the openai_api_key secret

Session secrets always beat process environment variables, and the host-pinned (value, [allowed_hosts]) secret form is understood.

Swapping and adding providers

CHAT_PROVIDER selects the backend by name (default openai). An unrecognised name logs a warning and falls back to the default rather than raising: a config typo should degrade the provider choice, not kill a rollout mid-flight. To add a provider, implement the small ABC and register it:

from openreward.chat_backends import ChatBackend, ChatCompletion, register_backend

class AnthropicBackend(ChatBackend):
    name = "anthropic"

    def __init__(self, *, config=None, secrets=None):
        ...

    async def create(self, *, model, messages, max_completion_tokens=None, **kwargs):
        ...
        return ChatCompletion(text=..., raw=...)

register_backend("anthropic", AnthropicBackend)

Grading code written against create() and .text now runs against either provider, selected per deployment with CHAT_PROVIDER=anthropic, with no code change. The same pattern works for any model role the environment owns: a simulated user or opponent built on create() can be pointed at a different provider or endpoint per deployment in exactly the same way.

Escape hatches

If you need the full provider surface (streaming, embeddings), OpenAIChatBackend.raw_client exposes the underlying pinned AsyncOpenAI client. For synchronous scripts, OpenAIChatConfig.from_env(secrets=...).client_kwargs() gives you the resolved api_key/base_url to construct any client with the same resolution rules.

Next Steps

Using LLM Graders

Learn how to integrate language model graders into an environment

Get started

Core Concepts

Making Environments

Advanced Environment Topics

Training

Evaluation

Harnesses

Sandbox Providers

Standalone APIs

Integrations

Rollouts

Deployment

Storage & Data

Goals

Prerequisites

Introduction

Basic usage

How resolution works

Swapping and adding providers

Escape hatches

Next Steps

Using LLM Graders

​Goals

​Prerequisites

​Introduction

​Basic usage

​How resolution works

​Swapping and adding providers

​Escape hatches

​Next Steps

Using LLM Graders

Goals

Prerequisites

Introduction

Basic usage

How resolution works

Swapping and adding providers

Escape hatches

Next Steps