Preference Rewards

​

Goals

• Understand when relative (preference-based) rewards beat absolute scoring
• Learn the judging toolkit: pairwise judges with constitutional principles, and group ranking
• Implement score_group so a trainer can have the environment judge a whole GRPO group

​

Prerequisites

• openreward>=0.1.157
• Completing the Using LLM Graders, Chat Backends and Using Rubrics tutorials

​

Introduction

1. A judging toolkit — openreward.judging — with a pairwise judge, constitutional principles, and group-ranking helpers. It’s a plain library built on chat backends, usable anywhere.
2. A score_group capability — an optional method on your environment that a trainer can call with a whole group’s candidate trajectories after the rollouts finish, receiving one reward per candidate.

​

The pairwise judge

from openreward.chat_backends import resolve_backend
from openreward.judging import PairwiseJudge

CONSTITUTION = [
    "Choose the response that is more helpful to the customer's actual problem.",
    "Choose the response that is more honest about what the assistant can and cannot do.",
    "Choose the response less likely to frustrate or mislead the customer.",
]

judge = PairwiseJudge(
    model="gpt-5.2",
    backend=resolve_backend(secrets=secrets),  # the dedicated judge channel
    principles=CONSTITUTION,
)

verdict = await judge.compare(candidate_a, candidate_b, context=conversation)
verdict.winner      # "a" | "b" | "tie"
verdict.principle   # which principle this comparison was judged against

• One principle is sampled per comparison from principles. Ensembling over principles gives a more robust preference signal than judging everything against one fixed rubric string.
• Presentation order is shuffled per comparison (shuffle=True). Pairwise judges show model-specific position bias; shuffling decorrelates it. both_orders=True upgrades this to judging both presentations (2× cost): agreement keeps the winner, disagreement becomes a tie.
• A broken judge raises, it never scores. An unparseable verdict gets a bounded corrective retry (max_retries), then raises JudgeFailure. Silent zeros corrupt training; catch the exception at the reward boundary and fail loudly.

​

Ranking a group

from openreward.judging import rank_group

rewards = await rank_group(
    candidates, judge,
    context=conversation,
    topology="round_robin",   # or "knockout"
    aggregator="winrate",     # or "rank"
)

​

What the judge sees: view

​

Group scoring: score_group

from openreward.chat_backends import resolve_backend
from openreward.environments import (Environment, GroupCandidate, GroupScore,
                                     JSONObject, Message)
from openreward.judging import PairwiseJudge, rank_group

class SupportEnv(Environment):
    def __init__(self, task_spec: JSONObject = {}, secrets: dict[str, str] = {}):
        super().__init__(task_spec)
        self.secrets = secrets

    def get_messages(self) -> list[Message]:
        ...  # the conversation so far — see Multi-turn Conversations

    async def score_group(self, candidates: list[GroupCandidate]) -> list[GroupScore]:
        judge = PairwiseJudge(
            model="gpt-5.2",
            backend=resolve_backend(secrets=self.secrets),
            principles=CONSTITUTION,
        )
        rewards = await rank_group(
            [c.messages for c in candidates], judge,
            context=self.get_messages(),
            topology="knockout",
            aggregator="winrate",
        )
        return [GroupScore(reward=r) for r in rewards]

​

Calling it from a trainer

async with environment.session(task=task) as session:
    if await session.supports_score_group():
        # ... collect the group's N rollouts (their own sessions) ...
        scores = await session.score_group(candidate_trajectories)
        group_rewards = [s.reward for s in scores]

​

Limitations of Preference Rewards

• Judge gaming. Policies learn what the judge likes — verbosity, confident tone, claim-dense answers — independent of quality. Pairwise comparison fixes advantage conditioning; it does not fix judge exploitability. Keep prescriptive anti-hacking principles in the constitution (“do not reward length”, “penalize unsupported claims”), and evaluate checkpoints against judges other than the training judge.
• Position bias. Left on by default (shuffle), upgradeable to both_orders. Don’t turn it off to save calls; the bias is real and model-specific.
• Non-transitivity. Judges produce cycles (A ≻ B ≻ C ≻ A). Win-rate aggregation over more pairs is more robust than single-elimination when cycles are common; if the same group keeps producing near-tied win-rates, the judge is at its resolution limit and the signal is mostly noise.
• Cost. Comparisons are judge calls over full trajectories. Round robin is quadratic in group size; budget with the topology table above, and consider escalating to pairwise only for groups where cheaper signals produce zero variance.

​

Next Steps