coco
268 lines
17 KiB
Markdown
268 lines
17 KiB
Markdown
---
|
||
title: "OpenClaw RL, explained clearly"
|
||
source: "https://x.com/neural_avb/status/2032489770648256956"
|
||
author:
|
||
- "[[Unknown]]"
|
||
published: 2026-03-13
|
||
created: 2026-03-14
|
||
description: "ai research engineer • designing agent runtimes, memory & retrieval systems for autonomous agents"
|
||
tags:
|
||
- "clippings"
|
||
---
|
||
The premise of this new research paper is actually very simple. **What if your AI agent got smarter every time you talk to it?** To test this, they designed a very practical experiment:
|
||
|
||
> A student uses Openclaw and he wants his LLM to complete homework on a personal computer, **while trying to avoid being perceived as relying on AI**.
|
||
|
||
> Basically, the student **does not want to be found using AI,** and wants the AI to copy the student's **personal preferences and writing style.**
|
||
|
||
> The archaic way of solving this problem would be doing supervised finetuning on the student's own written notes
|
||
|
||
> The even more archaic thing would be to write long prompts to teach the model rules of writing like this student.
|
||
|
||
> **But we are in 2026 so we are going to learn how to do this in a way cooler way - OpenClaw RL.**
|
||
|
||
But first, let's start with some terminologies to set the record straight for the rest of this article.
|
||
|
||
|
||
|
||
# Background
|
||
|
||
1. **What is RL?**
|
||
|
||
> RL is a Machine Learning framework where an **agent** learns to make better **decisions** by interacting with an **environment**.
|
||
|
||
> It observes a state, takes a decision, receives feedback from the environment about how favourable it's action were, and then it slowly improves it's decision making abilities (policy).
|
||
|
||
**2\. What is Policy Distillation?**
|
||
|
||
While RL trains LLMs by making it take actions and collect experiences in an environment, policy distillation takes a different approach.
|
||
|
||
> Policy Distillation comes from the broader concept of **Knowledge Distillation** in deep learning.
|
||
|
||
> We transfer knowledge from a more **capable teacher model** into a **smaller, faster student model** - without the student having to learn from scratch.
|
||
|
||
In other words, we take a model that is more capable (either a larger model, or a model that has more information) and distill it's behavior into a smaller, less capable model.
|
||
|
||
We will soon see how OpenClaw-RL uses this concept.
|
||
|
||
**3\. RLVR vs RLRF**
|
||
|
||
> **RLVR (RL with Verifiable Rewards)** is RL applied to tasks where the reward signal is **verifiable**. Meaning there is a simple deterministic way to verify if a task was successful or not. Example: math problems (is the answer correct?) or code (does the test pass?). These require no human annotation, so you can pretty much automate these rewards.
|
||
|
||
> There is one more term you should know about - **RLRF (RL with Rich Feedback)**. RLRF design provides richer feedback compared (ex: stack trace of generated code, or just feedback from an LLM judge) compared to just a scalar reward. This rich feedback can be used to train LLM models to generate better rollouts.
|
||
|
||
**4\. Process Reward Models (PRMs)**
|
||
|
||
RLVR gives feedback at the very end of a trajectory (a sequence of action that lead to a success/failure state). Traditionally this has led to some scary situations in RL training history resulting in people quitting their research careers at a young age. Google: "credit assignment in RL" and "sparse reward environments in RL" ... you'll get the gist.
|
||
|
||
> A **PRM** provides **step-level** feedback rather than outcome-level feedback. Instead of waiting until the end, it scores each intermediate step atat of the agent's reasoning chain.
|
||
|
||
Originally studied in **mathematical reasoning**, PRMs have been shown to dramatically outperform outcome-only rewards in long-horizon tasks.
|
||
|
||
OpenClaw-RL extends this idea to the **online, live, continual setting -** where process rewards are inferred from live next-state signals rather than pre-collected ground truth.
|
||
|
||
**5\. States and Next State-Signal**
|
||
|
||
This is the most central Openclaw-RL concept you must grasp. After every action a, the environment responds with a **next-state signal** st+1. Think of it as the "reaction" to what the agent just did **0**:
|
||
|
||
- A **user reply** after a chatbot response (are they thankful or pissed off?)
|
||
- A **terminal output** after a shell command (did it work? did it cause error?)
|
||
- A **test output** after code is submitted (did it compile? did it cause error?)
|
||
|
||
> This next-state signal is **implicit feedback**. It tells you both how well the action performed and, often, how it should have been different.
|
||
|
||
It is kind of a "rich-text" feedback. The Agent can infer both the reward and the next state from paying attention to this.
|
||
|
||
**6\. Two Types of RL Supervision**
|
||
|
||
This is another crucial thing for understanding what makes OpenClaw-RL different:
|
||
|
||
**Evaluative Signals → Scalar Rewards**
|
||
|
||
> A signal that **scores** the action. Did it work? How well? This is the traditional form of RL supervision - a scalar reward or a boolean flag that says "good" or "bad."
|
||
|
||
**Directive Signals → Token-Level Guidance**
|
||
|
||
> A signal that **directs** the action. Tells it exactlywhat should have been different and how. For example: "You should have checked the file first" tells the agent exactly which tokens to reconsider
|
||
|
||
You can read more about token level guidance in some cool papers that came out this year like
|
||
|
||
[Experiential Reinforcement Learning](https://x.com/neural_avb/status/2024874677609660831?s=20)
|
||
|
||
and
|
||
|
||
[SDPO (Self-Distillation via Policy Optimization)](https://x.com/neural_avb/status/2021216205684781094?s=20)
|
||
|
||
# The Goal
|
||
|
||
> OpenClaw-RL aims to show that **every agent interaction naturally produces a "next-state signal"** (a user reply, tool output, or screen state change) that can be used as a live training source.
|
||
|
||
The framework unifies these signals across diverse settings - personal conversations, terminal, GUI, SWE, and tool-call, to train a **single policy continuously and online**, without any separate data collection phase.
|
||
|
||
> It's a recipe for lifelong learning through experience collection.
|
||
|
||
|
||
|
||
## The Main Observation: You're Already Collecting the Data!
|
||
|
||
Here is the insight the paper opens with:
|
||
|
||
> "Every deployed AI agent is already collecting the data it needs to improve - and discarding it."
|
||
|
||
Every single time an agent takes an action a, the environment fires back a **next-state signal** s'. Like I mentioned in point 2 in my epic terminology section: this could be a user's reply, a terminal error trace, a test result. Most systems treat this signal as nothing more than **context for the next action.**
|
||
|
||
Meaning the AI simply takes the "next step" decision based off of this latest context, without realizing it can also learn from it.
|
||
|
||
The paper's boldest claim is the following:
|
||
|
||
> Personal conversations, terminal executions, GUI interactions, SWE tasks, and tool-call traces are not separate training problems. They are all interactions that generate next-state signals, and a single policy can learn from all of them simultaneously.
|
||
|
||
OpenClaw-RL's core claim is that this is a "**massive WASTE"**, and it identifies exactly **two forms of recoverable information** sitting inside every next state signal:
|
||
|
||
**Waste 1: Evaluative Signals (The "****How Good?****" Signal)**
|
||
|
||
The next-state signal **implicitly scores** the preceding action:
|
||
|
||
- A **user re-query** (e.g., "That's not what I meant") → dissatisfaction signal
|
||
- A **passing test** → success signal
|
||
- An **error trace** → failure signal
|
||
|
||
This is a **natural process reward.** It arises for **free** at every step, requires **zero annotation**, and provides the dense per-step credit assignment that long-horizon tasks desperately need. Yet existing systems either ignore it entirely or only exploit it in **offline, pre-collected form** - after the fact, on fixed datasets.
|
||
|
||
**Waste 2: Directive Signals (The "****How to Fix It?****" Signal)**
|
||
|
||
Beyond scoring, next-state signals often carry **directional information.** I.e. how the action should have been different:
|
||
|
||
- A user saying "you should have checked the file first" specifies the **exact correction direction** at the token level
|
||
- A detailed SWE error trace **implies a concrete fix**
|
||
|
||
Current RLVR methods compress everything into a **scalar reward**, which throws this rich directional information away entirely. OpenClaw-RL recovers it through a mechanism called "**Hindsight-Guided OPD"** (oooooo we'll cover in a later section).
|
||
|
||
**The Unifying Framework: One Policy, All Streams** Let's look at this more closely.
|
||
|
||
## The Architecture: A Four-Engine Machine
|
||
|
||
|
||
|
||
At the heart of OpenClaw-RL is one radical architectural principle: **break everything apart**. Complete decoupling. Trad RL training are tightly coupled: the model waits for an environment response, the environment waits for a reward, the reward waits for the trainer. Every component blocks the next. This slows down real-world agents that need to keep serving live users.
|
||
|
||
OpenClaw-RL's answer? **Four completely independent, asynchronous loops** - none that blocks the other.
|
||
|
||
The four decoupled, asynchronous components ("engines") of OpenClaw-RL are:
|
||
|
||
1. **Environment Server :** Hosts the agent's environment (user devices for personal agents, cloud services for general agents). It collects interaction samples and feeds them into the training pipeline.
|
||
2. **PRM / Judge:** Evaluates the quality of each action by computing rewards from the next-state signal. It runs independently, judging previous responses while the model is already serving new ones.
|
||
3. **Megatron (Policy Trainer):** Applies gradient updates to the policy using the rewards computed by the PRM. Built on the Megatron framework for large-scale training. Megatron-LM is a high-performance library for training very large language models by the good folks at Nvidia. It is designed to handle the massive computational demands through several parallelism strategies like tensor parallelism, pipeline parallelism, data parallelism.
|
||
4. **SGLang (Policy Server):** Serves the live policy to users, handling inference requests. It supports **graceful weight updates**, meaning the policy can be updated without interrupting ongoing inference.
|
||
|
||
The key insight is that **none of these four components waits for the others** they all run asynchronously, which is what makes continuous, online learning from live interactions practical.
|
||
|
||
These four engines spin simultaneously, **with zero blocking dependencies between them**.
|
||
|
||
**How Data Flows Through the System**
|
||
|
||
1. A user (or environment) sends a message → the **SGLang Policy Server** generates a response in real-time.
|
||
2. The response lands in the environment (a user's device, a terminal, a GUI) → the **Environment Server** captures the next-state signal (user reply, terminal output, GUI change).
|
||
3. That interaction is logged asynchronously → the **PRM Judge** scores the quality of the action.
|
||
4. Scored trajectories accumulate in a replay buffer → the **Megatron Trainer** pulls batches and updates the policy weights.
|
||
5. Updated weights are **gracefully pushed** back to the serving layer without interrupting live inference.
|
||
|
||
Many RL projects are switching to exactly this kind of system design including the GLM-5 and Minimax training stack.
|
||
|
||
> 2月19日
|
||
>
|
||
> LLM research in 2026 is basically all about 3 things: - scaling Asynchronous RL at a system design level - harnesses/recipes to train independent (clawdbot style) agentic systems - make low-level architectural choices that achieve this cheaply
|
||
|
||
In the next section let's talk about how the actual RL stuff happens in more detail.
|
||
|
||
# The PRM Judge
|
||
|
||
**PRM** stands for **Process Reward Model**. Unlike a standard outcome reward model that only tells you "did you succeed at the end?", a PRM assigns a **reward at every step** of the agent's trajectory providing much richer training signal.
|
||
|
||
|
||
|
||
In OpenClaw-RL, the PRM Judge is an **LLM-based evaluator** that looks at:
|
||
|
||
- The agent's action a (at a step t)
|
||
- The **next state** s\` (the user reply, tool output, terminal state, etc.)
|
||
- And outputs a scalar reward score: **R** (usually +1, 0, -1)
|
||
- They run this multiple times prompts and take the **majority vote reward** as the final.
|
||
|
||
Problem with scalar reward for whole sequence is that it pushes **all tokens** in the response in the same direction. If the response was bad (r=−1), every single token gets penalized equally, even the tokens that were actually fine
|
||
|
||
# Hindsight-Guided On-Policy Distillation (OPD)
|
||
|
||
Along with PRM Judge, we also use rich reward text feedback to do training.
|
||
|
||
The basic gist of OPD is simple:
|
||
|
||
> If you augment the original prompt with a **textual hint** extracted from the next state s', the same model will produce a **different token distribution** - one that "knows" what the response should have been.
|
||
|
||
|
||
|
||
Basically, suppose you ask the AI a difficult question and record it's answer.... Then you asked the AI the same difficult question BUT ALSO give a hint what went wrong previously.
|
||
|
||
The **gap** between this hint-enhanced distribution and the original student distribution gives a **per-token directional advantage**: positive where the model should upweight a token, negative where it should downweight.
|
||
|
||
This is fundamentally different from:
|
||
|
||
- **RLHF** - uses scalar preference signals
|
||
- **DPO** - requires paired preferences (often annotated by human or another AI)
|
||
- **Standard distillation** - requires a separate, stronger teacher model
|
||
|
||
OPD uses **the model itself** as its own teacher, just with extra context!
|
||
|
||
The policy is run under enhanced with the original response atat as forced input. The per-token log-probability gap gives the advantage. If teacher assigns a token higher probability, student would upweight this token, If teacher assigns a lower probability, student should downweight.
|
||
|
||
> **Basically we try to update the model by moving the student's token log-probs closer to the teacher's log-prob.** The teacher's log-probs are generated using the hint, and the student's log-probs were generated without the hint.
|
||
|
||
> With distillation, we are training the student to get to the correct solution in one attempt WITHOUT requiring the hint!
|
||
|
||
## PRM + OPD: Better Together
|
||
|
||
|
||
|
||
Process Reward Models (generating scalar/ternary rewards) and OPD (generating rich feedback) are used together during training.
|
||
|
||
> The advantage of each token is basically the global advantage of the entire sequence (coming from PRM) plus the distillation lift from the OPD.
|
||
|
||
So the combination is simply a **weighted sum of the two advantages**:
|
||
|
||
# Final Thoughts
|
||
|
||
|
||
|
||
OpenClaw-RL is useful in two contexts:
|
||
|
||
- **Personal Agents** run on a single user's device. Interactions are sparse, session-based, and deeply personalized. Agent trains on interaction with it's user (for example, personalizing to a student who uses OpenClaw to do homework)
|
||
- **General Agents** are learning agentic tasks. **Terminal**, **GUI**, **SWE (software engineering)**, and **Tool-call** agents - covering virtually every real-world deployment setting.
|
||
|
||
They do experiments with small Qwen3 models (4B, 8B, and 32B variants). Here are a list of their main takeaways:
|
||
|
||
1. **Binary RL alone** barely moves the needle - marginal improvement only
|
||
2. **OPD alone** starts slow (sparse hints early on) but jumps significantly later
|
||
3. **Combined** (Binary + OPD) wins convincingly at both personal and general agents
|
||
4. PRM gains is especially dramatic in the **tool-call** setting (250 steps long) - a 76% jump. This makes sense intuitively: the longer the horizon, the more the agent suffers from sparse outcome-only rewards, and the more dense PRM step signals help!
|
||
5. The Qwen Models learned very fast. **36 problem-solving interactions** in the student setting (described in the intro).
|
||
|
||
The paper provides prompts how these tasks were learnt, and how PRM and OPD rewards were extracted. It is all very cool, I encourage yall to read the main paper for their experiments and results!
|
||
|
||
**Their Repo:** [https://github.com/Gen-Verse/OpenClaw-RL](https://github.com/Gen-Verse/OpenClaw-RL) **Read paper on your own at Arxiv:** [arxiv.org/abs/2603.10165](https://arxiv.org/abs/2603.10165)
|
||
|
||
**Study the paper with an AI that highlights relevant sections for you as it answers your questions:** [http://paperbreakdown.com/abs/2603.10165](http://paperbreakdown.com/abs/2603.10165)
|
||
|
||
If you are into links, here is where to find me:
|
||
|
||
**Follow me for more**:
|
||
|
||
[@neural\_avb](https://x.com/@neural_avb)
|
||
|
||
**My YouTube channel:**
|
||
|
||
[https://www.youtube.com/@avb\_fj](https://www.youtube.com/@avb_fj)
|
||
|
||
**Patreon:**
|
||
|
||
[https://www.patreon.com/NeuralBreakdownwithAVB](https://www.patreon.com/NeuralBreakdownwithAVB)
|
||
|
||
A bit of kindness goes a long way, RT or comment to support! 💙 |