Obsidian/openclaw/OpenClaw RL, explained clearly.md

title, source, author, published, created, description, tags

title	source	author	published	created	description	tags
OpenClaw RL, explained clearly	https://x.com/neural_avb/status/2032489770648256956	Unknown	2026-03-13	2026-03-14	ai research engineer • designing agent runtimes, memory & retrieval systems for autonomous agents	clippings

The premise of this new research paper is actually very simple. What if your AI agent got smarter every time you talk to it? To test this, they designed a very practical experiment:

A student uses Openclaw and he wants his LLM to complete homework on a personal computer, while trying to avoid being perceived as relying on AI.

Basically, the student does not want to be found using AI, and wants the AI to copy the student's personal preferences and writing style.

The archaic way of solving this problem would be doing supervised finetuning on the student's own written notes

The even more archaic thing would be to write long prompts to teach the model rules of writing like this student.

But we are in 2026 so we are going to learn how to do this in a way cooler way - OpenClaw RL.

But first, let's start with some terminologies to set the record straight for the rest of this article.

Background

1. What is RL?

RL is a Machine Learning framework where an agent learns to make better decisions by interacting with an environment.

It observes a state, takes a decision, receives feedback from the environment about how favourable it's action were, and then it slowly improves it's decision making abilities (policy).

2. What is Policy Distillation?

While RL trains LLMs by making it take actions and collect experiences in an environment, policy distillation takes a different approach.

Policy Distillation comes from the broader concept of Knowledge Distillation in deep learning.

We transfer knowledge from a more capable teacher model into a smaller, faster student model - without the student having to learn from scratch.

In other words, we take a model that is more capable (either a larger model, or a model that has more information) and distill it's behavior into a smaller, less capable model.

We will soon see how OpenClaw-RL uses this concept.

3. RLVR vs RLRF

RLVR (RL with Verifiable Rewards) is RL applied to tasks where the reward signal is verifiable. Meaning there is a simple deterministic way to verify if a task was successful or not. Example: math problems (is the answer correct?) or code (does the test pass?). These require no human annotation, so you can pretty much automate these rewards.

There is one more term you should know about - RLRF (RL with Rich Feedback). RLRF design provides richer feedback compared (ex: stack trace of generated code, or just feedback from an LLM judge) compared to just a scalar reward. This rich feedback can be used to train LLM models to generate better rollouts.

4. Process Reward Models (PRMs)

RLVR gives feedback at the very end of a trajectory (a sequence of action that lead to a success/failure state). Traditionally this has led to some scary situations in RL training history resulting in people quitting their research careers at a young age. Google: "credit assignment in RL" and "sparse reward environments in RL" ... you'll get the gist.

A PRM provides step-level feedback rather than outcome-level feedback. Instead of waiting until the end, it scores each intermediate step atat​ of the agent's reasoning chain.

Originally studied in mathematical reasoning, PRMs have been shown to dramatically outperform outcome-only rewards in long-horizon tasks.

OpenClaw-RL extends this idea to the online, live, continual setting - where process rewards are inferred from live next-state signals rather than pre-collected ground truth.

5. States and Next State-Signal

This is the most central Openclaw-RL concept you must grasp. After every action a​, the environment responds with a next-state signal st+1​. Think of it as the "reaction" to what the agent just did 0:

• A user reply after a chatbot response (are they thankful or pissed off?)
• A terminal output after a shell command (did it work? did it cause error?)
• A test output after code is submitted (did it compile? did it cause error?)

This next-state signal is implicit feedback. It tells you both how well the action performed and, often, how it should have been different.

It is kind of a "rich-text" feedback. The Agent can infer both the reward and the next state from paying attention to this.

6. Two Types of RL Supervision

This is another crucial thing for understanding what makes OpenClaw-RL different:

Evaluative Signals → Scalar Rewards

A signal that scores the action. Did it work? How well? This is the traditional form of RL supervision - a scalar reward or a boolean flag that says "good" or "bad."

Directive Signals → Token-Level Guidance

A signal that directs the action. Tells it exactlywhat should have been different and how. For example: "You should have checked the file first" tells the agent exactly which tokens to reconsider

You can read more about token level guidance in some cool papers that came out this year like

Experiential Reinforcement Learning

and

SDPO (Self-Distillation via Policy Optimization)

The Goal

OpenClaw-RL aims to show that every agent interaction naturally produces a "next-state signal" (a user reply, tool output, or screen state change) that can be used as a live training source.

The framework unifies these signals across diverse settings - personal conversations, terminal, GUI, SWE, and tool-call, to train a single policy continuously and online, without any separate data collection phase.

It's a recipe for lifelong learning through experience collection.

The Main Observation: You're Already Collecting the Data!

Here is the insight the paper opens with:

"Every deployed AI agent is already collecting the data it needs to improve - and discarding it."

Every single time an agent takes an action a​, the environment fires back a next-state signal s'​. Like I mentioned in point 2 in my epic terminology section: this could be a user's reply, a terminal error trace, a test result. Most systems treat this signal as nothing more than context for the next action.

Meaning the AI simply takes the "next step" decision based off of this latest context, without realizing it can also learn from it.

The paper's boldest claim is the following:

Personal conversations, terminal executions, GUI interactions, SWE tasks, and tool-call traces are not separate training problems. They are all interactions that generate next-state signals, and a single policy can learn from all of them simultaneously.

OpenClaw-RL's core claim is that this is a "massive WASTE", and it identifies exactly two forms of recoverable information sitting inside every next state signal​:

Waste 1: Evaluative Signals (The "How Good?" Signal)

The next-state signal implicitly scores the preceding action:

• A user re-query (e.g., "That's not what I meant") → dissatisfaction signal
• A passing test → success signal
• An error trace → failure signal

This is a natural process reward. It arises for free at every step, requires zero annotation, and provides the dense per-step credit assignment that long-horizon tasks desperately need. Yet existing systems either ignore it entirely or only exploit it in offline, pre-collected form - after the fact, on fixed datasets.

Waste 2: Directive Signals (The "How to Fix It?" Signal)

Beyond scoring, next-state signals often carry directional information. I.e. how the action should have been different:

• A user saying "you should have checked the file first" specifies the exact correction direction at the token level
• A detailed SWE error trace implies a concrete fix

Current RLVR methods compress everything into a scalar reward, which throws this rich directional information away entirely. OpenClaw-RL recovers it through a mechanism called "Hindsight-Guided OPD" (oooooo we'll cover in a later section).

The Unifying Framework: One Policy, All Streams Let's look at this more closely.

The Architecture: A Four-Engine Machine

At the heart of OpenClaw-RL is one radical architectural principle: break everything apart. Complete decoupling. Trad RL training are tightly coupled: the model waits for an environment response, the environment waits for a reward, the reward waits for the trainer. Every component blocks the next. This slows down real-world agents that need to keep serving live users.

OpenClaw-RL's answer? Four completely independent, asynchronous loops - none that blocks the other.

The four decoupled, asynchronous components ("engines") of OpenClaw-RL are:

1. Environment Server : Hosts the agent's environment (user devices for personal agents, cloud services for general agents). It collects interaction samples and feeds them into the training pipeline.
2. PRM / Judge: Evaluates the quality of each action by computing rewards from the next-state signal. It runs independently, judging previous responses while the model is already serving new ones.
3. Megatron (Policy Trainer): Applies gradient updates to the policy using the rewards computed by the PRM. Built on the Megatron framework for large-scale training. Megatron-LM is a high-performance library for training very large language models by the good folks at Nvidia. It is designed to handle the massive computational demands through several parallelism strategies like tensor parallelism, pipeline parallelism, data parallelism.
4. SGLang (Policy Server): Serves the live policy to users, handling inference requests. It supports graceful weight updates, meaning the policy can be updated without interrupting ongoing inference.

The key insight is that none of these four components waits for the others they all run asynchronously, which is what makes continuous, online learning from live interactions practical.

These four engines spin simultaneously, with zero blocking dependencies between them.

How Data Flows Through the System

1. A user (or environment) sends a message → the SGLang Policy Server generates a response in real-time.
2. The response lands in the environment (a user's device, a terminal, a GUI) → the Environment Server captures the next-state signal (user reply, terminal output, GUI change).
3. That interaction is logged asynchronously → the PRM Judge scores the quality of the action.
4. Scored trajectories accumulate in a replay buffer → the Megatron Trainer pulls batches and updates the policy weights.
5. Updated weights are gracefully pushed back to the serving layer without interrupting live inference.

Many RL projects are switching to exactly this kind of system design including the GLM-5 and Minimax training stack.

2月19日

LLM research in 2026 is basically all about 3 things: - scaling Asynchronous RL at a system design level - harnesses/recipes to train independent (clawdbot style) agentic systems - make low-level architectural choices that achieve this cheaply

In the next section let's talk about how the actual RL stuff happens in more detail.

The PRM Judge

PRM stands for Process Reward Model. Unlike a standard outcome reward model that only tells you "did you succeed at the end?", a PRM assigns a reward at every step of the agent's trajectory providing much richer training signal.

In OpenClaw-RL, the PRM Judge is an LLM-based evaluator that looks at:

• The agent's action a​ (at a step t)
• The next state s`​ (the user reply, tool output, terminal state, etc.)
• And outputs a scalar reward score: R (usually +1, 0, -1)
• They run this multiple times prompts and take the majority vote reward as the final.

Problem with scalar reward for whole sequence is that it pushes all tokens in the response in the same direction. If the response was bad (r=−1), every single token gets penalized equally, even the tokens that were actually fine

Hindsight-Guided On-Policy Distillation (OPD)

Along with PRM Judge, we also use rich reward text feedback to do training.

The basic gist of OPD is simple:

If you augment the original prompt with a textual hint extracted from the next state s'​, the same model will produce a different token distribution - one that "knows" what the response should have been.

Basically, suppose you ask the AI a difficult question and record it's answer.... Then you asked the AI the same difficult question BUT ALSO give a hint what went wrong previously.

The gap between this hint-enhanced distribution and the original student distribution gives a per-token directional advantage: positive where the model should upweight a token, negative where it should downweight.

This is fundamentally different from:

• RLHF - uses scalar preference signals
• DPO - requires paired preferences (often annotated by human or another AI)
• Standard distillation - requires a separate, stronger teacher model

OPD uses the model itself as its own teacher, just with extra context!

The policy is run under enhanced​ with the original response atat​ as forced input. The per-token log-probability gap gives the advantage. If teacher assigns a token higher probability, student would upweight this token, If teacher assigns a lower probability, student should downweight.

Basically we try to update the model by moving the student's token log-probs closer to the teacher's log-prob. The teacher's log-probs are generated using the hint, and the student's log-probs were generated without the hint.

With distillation, we are training the student to get to the correct solution in one attempt WITHOUT requiring the hint!

PRM + OPD: Better Together

Process Reward Models (generating scalar/ternary rewards) and OPD (generating rich feedback) are used together during training.

The advantage of each token is basically the global advantage of the entire sequence (coming from PRM) plus the distillation lift from the OPD.

So the combination is simply a weighted sum of the two advantages:

Final Thoughts

OpenClaw-RL is useful in two contexts:

• Personal Agents run on a single user's device. Interactions are sparse, session-based, and deeply personalized. Agent trains on interaction with it's user (for example, personalizing to a student who uses OpenClaw to do homework)
• General Agents are learning agentic tasks. Terminal, GUI, SWE (software engineering), and Tool-call agents - covering virtually every real-world deployment setting.

They do experiments with small Qwen3 models (4B, 8B, and 32B variants). Here are a list of their main takeaways:

1. Binary RL alone barely moves the needle - marginal improvement only
2. OPD alone starts slow (sparse hints early on) but jumps significantly later
3. Combined (Binary + OPD) wins convincingly at both personal and general agents
4. PRM gains is especially dramatic in the tool-call setting (250 steps long) - a 76% jump. This makes sense intuitively: the longer the horizon, the more the agent suffers from sparse outcome-only rewards, and the more dense PRM step signals help!
5. The Qwen Models learned very fast. 36 problem-solving interactions in the student setting (described in the intro).

The paper provides prompts how these tasks were learnt, and how PRM and OPD rewards were extracted. It is all very cool, I encourage yall to read the main paper for their experiments and results!

Their Repo: https://github.com/Gen-Verse/OpenClaw-RL Read paper on your own at Arxiv: arxiv.org/abs/2603.10165

Study the paper with an AI that highlights relevant sections for you as it answers your questions: http://paperbreakdown.com/abs/2603.10165

If you are into links, here is where to find me:

Follow me for more:

@neural_avb

My YouTube channel:

https://www.youtube.com/@avb_fj

Patreon:

https://www.patreon.com/NeuralBreakdownwithAVB

A bit of kindness goes a long way, RT or comment to support! 💙