> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openreward.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Preference Rewards
> Constitutional-AI-style rewards from A ≻ B comparisons
## Goals
* Understand when relative (preference-based) rewards beat absolute scoring
* Learn the judging toolkit: pairwise judges with constitutional principles, and group ranking
* Implement `score_group` so a trainer can have the environment judge a whole GRPO group
## Prerequisites
* `openreward>=0.1.157`
* Completing the [Using LLM Graders](/environments/using-llm-graders), [Chat Backends](/environments/chat-backends) and [Using Rubrics](/environments/using-rubrics) tutorials
## Introduction
[Rubrics](/environments/using-rubrics) score one response at a time against criteria — the *pointwise* half of constitutional AI, where a judge checks a response against principles. But the reinforcement-learning half of constitutional AI (RLAIF) is inherently *pairwise*: the judge is shown a conversation and **two** candidate responses, and picks the better one against a principle. The preference signal is relative, not absolute.
Relative rewards suit group-based RL algorithms like GRPO unusually well. GRPO normalizes rewards within a group of rollouts, so only their *ordering* matters — and pointwise judge scores tend to cluster tightly within a group, letting the normalization amplify judge noise into illusory advantages. Win-rates over pairwise comparisons spread by construction.
Two things ship for this:
1. **A judging toolkit** — `openreward.judging` — with a pairwise judge, constitutional principles, and group-ranking helpers. It's a plain library built on [chat backends](/environments/chat-backends), usable anywhere.
2. **A `score_group` capability** — an optional method on your environment that a trainer can call with a whole group's candidate trajectories after the rollouts finish, receiving one reward per candidate.
## The pairwise judge
```python theme={null}
from openreward.chat_backends import resolve_backend
from openreward.judging import PairwiseJudge
CONSTITUTION = [
"Choose the response that is more helpful to the customer's actual problem.",
"Choose the response that is more honest about what the assistant can and cannot do.",
"Choose the response less likely to frustrate or mislead the customer.",
]
judge = PairwiseJudge(
model="gpt-5.2",
backend=resolve_backend(secrets=secrets), # the dedicated judge channel
principles=CONSTITUTION,
)
verdict = await judge.compare(candidate_a, candidate_b, context=conversation)
verdict.winner # "a" | "b" | "tie"
verdict.principle # which principle this comparison was judged against
```
Candidates can be plain strings (final answers) or role-tagged transcripts (`list[Message]`). By default the judge sees only a transcript's final output — see [`view`](#what-the-judge-sees-view) below. `context` is the task conversation the candidates respond to.
The defaults follow the constitutional-AI literature rather than convenience:
* **One principle is sampled per comparison** from `principles`. Ensembling over principles gives a more robust preference signal than judging everything against one fixed rubric string.
* **Presentation order is shuffled per comparison** (`shuffle=True`). Pairwise judges show model-specific position bias; shuffling decorrelates it. `both_orders=True` upgrades this to judging both presentations (2× cost): agreement keeps the winner, disagreement becomes a tie.
* **A broken judge raises, it never scores.** An unparseable verdict gets a bounded corrective retry (`max_retries`), then raises `JudgeFailure`. Silent zeros corrupt training; catch the exception at the reward boundary and fail loudly.
## Ranking a group
`rank_group` turns pairwise verdicts over a group into one scalar per candidate:
```python theme={null}
from openreward.judging import rank_group
rewards = await rank_group(
candidates, judge,
context=conversation,
topology="round_robin", # or "knockout"
aggregator="winrate", # or "rank"
)
```
The **topology** decides which pairs get judged — the cost/fidelity knob:
| Topology | Judge comparisons (group of N) | Character |
| ---------------- | ------------------------------ | ---------------------------------------------------------------------- |
| `GroupwiseJudge` | 1 call | All candidates scored in one call. Cheapest, weakest discrimination. |
| `knockout` | N − 1 | Elimination bracket. Linear cost, coarse resolution among non-winners. |
| `round_robin` | N(N−1)/2 | Every pair judged. The gold standard. |
The **aggregator** decides how verdicts become scalars: `winrate` (wins / games, ties count half) or `rank` (ordering only, evenly spaced — for when you trust the judge's ordering but not its magnitudes).
### What the judge sees: `view`
By default judges **reward the output, not the chain of thought**: with `view="output"` (the default), a transcript candidate renders as only the final assistant turn's visible content — no reasoning, no tool history, no earlier turns. This judges the behavior you actually deploy, and it keeps optimization pressure off the reasoning itself.
Do not train on judged chain of thought. Putting reward on reasoning trains it to *look* good to the judge rather than to be a faithful record of the model's process: models learn to write judge-pleasing thoughts and to move behavior the judge would penalize out of them, which both invites reward hacking and degrades the chain of thought's value for monitoring. Keep reward on outcomes.
`view="trajectory"` is an advanced opt-in for the narrower case where the *actions* matter: agentic tasks where the judge should see tool calls and their results — did the candidate verify before answering, did it take a destructive shortcut (requires `openreward>=0.1.157` for candidates carrying [rich blocks](/environments/multi-turn-conversations#rich-conversations-reasoning-and-tool-history)). Be aware that everything shown to the judge is under optimization pressure, and the trajectory view includes any reasoning in the transcript — so if you use it, write principles that reference actions and outcomes, never reasoning style or thoroughness, and consider sending candidates without reasoning blocks at all. Expect the judge bill to grow with transcript length. The task `context` always renders in full under either view; it's the shared prompt, not the thing being judged.
At N=8 rollouts a round robin is 28 comparisons per task per step — budget accordingly, or drop to `knockout` (7) or the single-call `GroupwiseJudge`.
## Group scoring: `score_group`
Within a session, an environment only ever sees its own rollout — sibling rollouts live in sibling sessions. `score_group` is the protocol surface that closes the gap: after a trainer collects a GRPO group's N rollouts for one task, it sends the N candidate trajectories to the environment and gets N rewards back. The environment owns the constitution and the judging.
Override the method to enable it:
```python theme={null}
from openreward.chat_backends import resolve_backend
from openreward.environments import (Environment, GroupCandidate, GroupScore,
JSONObject, Message)
from openreward.judging import PairwiseJudge, rank_group
class SupportEnv(Environment):
def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
super().__init__(task_spec)
self.secrets = secrets
def get_messages(self) -> list[Message]:
... # the conversation so far — see Multi-turn Conversations
async def score_group(self, candidates: list[GroupCandidate]) -> list[GroupScore]:
judge = PairwiseJudge(
model="gpt-5.2",
backend=resolve_backend(secrets=self.secrets),
principles=CONSTITUTION,
)
rewards = await rank_group(
[c.messages for c in candidates], judge,
context=self.get_messages(),
topology="knockout",
aggregator="winrate",
)
return [GroupScore(reward=r) for r in rewards]
```
Each `GroupCandidate` carries the candidate as role-tagged `messages` plus optional trainer-owned `metadata`, echoed back for correlation. With the default `view="output"` only the final assistant turn is judged, so output-only candidates suffice; send fuller trajectories (optionally with reasoning/tool blocks) only for `view="trajectory"` environments. Return exactly one score per candidate, index-aligned; bare floats are coerced to `GroupScore`. If the judge fails, let the exception propagate — the trainer receives a clean failure for the group instead of fabricated rewards.
Support is advertised automatically: environments that override `score_group` report `supports_score_group: true` in their tool list (the same optional-field pattern as the [terminal tool](/environments/using-terminal-tools)), so harnesses can detect it before spending a single rollout.
## Calling it from a trainer
```python theme={null}
async with environment.session(task=task) as session:
if await session.supports_score_group():
# ... collect the group's N rollouts (their own sessions) ...
scores = await session.score_group(candidate_trajectories)
group_rewards = [s.reward for s in scores]
```
The call is served over the same resumable streaming as tool calls, so a slow tournament (minutes of judge calls) survives disconnects without re-judging. Two failure modes are distinguished: `ScoreGroupUnsupported` (the environment doesn't implement it, or the server predates `0.1.157`) and `ScoreGroupFailed` (the judging raised). Neither kills the session — scoring reads the candidates without advancing environment state, so a trainer can retry or fall back to the pointwise rewards it already holds.
Comparative rewards arrive **alongside** the pointwise rewards the sessions emitted, never instead of them. How the two combine — sum, replace, gate — is trainer policy, deliberately outside the environment's contract.
## Limitations of Preference Rewards
Preference judging inherits every LLM-as-judge failure mode, plus a few of its own:
* **Judge gaming.** Policies learn what the judge likes — verbosity, confident tone, claim-dense answers — independent of quality. Pairwise comparison fixes advantage conditioning; it does not fix judge exploitability. Keep prescriptive anti-hacking principles in the constitution ("do not reward length", "penalize unsupported claims"), and evaluate checkpoints against judges *other* than the training judge.
* **Position bias.** Left on by default (`shuffle`), upgradeable to `both_orders`. Don't turn it off to save calls; the bias is real and model-specific.
* **Non-transitivity.** Judges produce cycles (A ≻ B ≻ C ≻ A). Win-rate aggregation over more pairs is more robust than single-elimination when cycles are common; if the same group keeps producing near-tied win-rates, the judge is at its resolution limit and the signal is mostly noise.
* **Cost.** Comparisons are judge calls over full trajectories. Round robin is quadratic in group size; budget with the topology table above, and consider escalating to pairwise only for groups where cheaper signals produce zero variance.
## Next Steps
Serve the conversation the candidates respond to as a role-tagged prompt
The dedicated judge-model channel the judging toolkit is built on