Skip to main content

​
Goals

  • Learn why rubrics are useful for assigning reward
  • Learning how to define rubrics for an environment
  • Understand how to integrate them into an ORS environment

​
Prerequisites

​
Introduction

In the Using LLM Graders tutorial, we saw we could use language models to assign rewards where it was hard to match a model and ground truth answer using string matching. But this still relied on the notion of a “reference answer” to compare against. What about domains where a reference answer isn’t available? Consider a task where we task a model to write an essay on the causes of World War One. There isn’t a “correct” answer in this case, but a teacher would still need to grade an essay based on the quality of reasoning, the evidence presented, and more. A grading Rubric is a criterion for grading a model response. A rubric can be:
  • Binary: for example, did the answer mention a particular event or not? Did it pass a unit test or not?
  • Point-based: for example, it could allocate 0 points for not mentioning an important factor, 0.5 points for mentioning it but not specific details, and 1.0 points for mentioning the factor and the details.
  • Weighted: we might weight one rubric higher than another when considering the total score.
Typically, we calculate the final reward as a weighted sum of rubrics, where: r=∑iKwisi∑iwir = \frac{\sum_{i}^{K}w_{i}s_{i}}{\sum_{i}w_{i}} where wiw_i is the weight on a rubric, and sis_i is its score. Multiple rubrics, and/or point-based reward, is useful for reinforcement learning as it enables a more continuous reward signal to distinguish between different responses. Rubrics are particularly useful for long-form language model responses. For example, they power the training of DeepResearch type models. Now let’s see how rubrics work in action.

​
Worked Example

Let’s make an ORS environment locally. First, if you haven’t already:
We’ll initialise a new project:
First, we’ll add some additional imports to our server.py file:
Next we’ll alter train_tasks and test_tasks. We’ll focus on the question we asked earlier in this tutorial:
We’ll need to define the rubrics above. We’ll use ten separate rubrics with a point scale:
Next we’ll need a template for the LLM grader:
We’ll also need to update BasicTaskSpec to include the rubrics field:
We’ll change the init to include the grader. We’ll use an OpenAI model as the grader:
And we’ll define some methods for the new answer tool:
Once we’ve made these changes, let’s run the environment. In your terminal run:
To sample, run the following code:
1

Set your API keys

Make sure you have API keys for OpenReward and OpenAI, and set these as environment variables:
2

Create your code

Save this as test_rubrics.py:
3

Run your code

Example output:
In the example above we got a reward of 0.65, or 6.5/10. Studying the grader response, we see the model answer lost points because it did not engage with the existing scholarship, misses key dates and other details. This shows an example of how rubrics work in practice, and how we can use them in ORS environments. But we should reflect: there are some issue with the rubrics we have defined for this environment, as well as the way we have just prompted a language model:
  • Hallucination. The rubrics, as defined, do not punish hallucination directly. For example, what if the model made up an event or got a date wrong? Our rubrics do not punish this directly.
  • Required output format. It may be acceptable to omit detail if it is a chatbot response; it is less acceptable if it is a university thesis or a undergraduate essay. We should make it clear to the language model what type of output is required so it knows how much detail is needed.
  • Tools and citations. We have not given the language model access to a web search tool so it can search the literature. We have also not explicitly mentioned a need for citations, nor defined a rubric for citation use. Likewise, we have not given the grader access to tools that could be used for verification (and to identify and punish hallucination).
This is not a problem with rubrics, per se, but it does show we need to be careful to elicit and reward the right behaviour. When using rubrics to train models, you should pay close attention to their responses to ensure they are not reward hacking, and are appropriately incentivised to exhibit the right behaviours.

​
Limitations of Rubrics

We have already touched upon some limitations of our rubric example, but we should note some general themes to look out for:
  • Reward Hacking - if we do not specify the desired behaviours correctly, or underspecify requirements, then the model may find a way to achieve reward without exhibiting the desired behaviours.
  • Subjectivity - if an objective metric is available, we should always prefer that. If we use a subjective human measure of quality, then we may limit the creativity of the model to discover better-than-human solutions.
  • Cost - LLM graders can become costly, especially in large-scale reinforcement learning. With rubrics, we often use multiple graders for each rubric which multiplies costs further. If we can find simple rules-based methods to identify and reward the same behaviour, then this is preferred to LLM graded rubrics.
  • Grader Quality - We are also bound by the quality of the grader. If a rubric is difficult to judge, or requires deep expertise, then it is recommended to use reasoning models as graders.