Why Real-World RL Struggles to Generalize Beyond a Single Task?

Lightweight policy models trained through real-world reinforcement learning at customer sites typically follow a pipeline of on-site data collection, reward model training, expert-trajectory warm-up, and iterative real-robot rollouts. This workflow is direct and closely aligned with the target task. However, its limitation is equally clear: the model often becomes specialized to a single task and fails to generalize effectively to new settings.

For example, in a task that requires placing an object within a designated region, changes in the background, object pose, or camera viewpoint at a customer site may affect both the Reward Model’s assessment and the Policy Model’s action generation. This highlights the need for both the Reward Model and the Policy Model to generalize reliably across environmental variations. In particular, the stability of the Reward Model is critical: even if the Policy Model remains sensitive to such variations, a Reward Model that can accurately assess task progress and completion can provide reliable feedback for real-world reinforcement learning, thereby enabling the Policy Model to progressively improve its adaptability and generalization.

We summarize the main limitations of the current workflow in three areas.

• High cross-task transfer cost: Each new task typically requires collecting a new set of successful and failed examples and preparing new expert demonstrations for warm-starting.
• Weak cross-scene generalization: Short-term on-site data are often limited in diversity, causing the model to memorize the environment rather than understand task progress.
• High cost of real-world sampling: Each rollout requires a physical robot, while failures consume time, require intervention, wear hardware, and introduce safety risks.

Core challenge: reduce the cost of on-site data collection and annotation, minimize real-robot trial-and-error, and improve generalization across new tasks, new scenes, and new robot embodiments.

System Design

A World Model (WM) provides a more general foundation for task-state understanding in robot learning. By learning object relations, future state transitions, and action-induced changes from large-scale multimodal data, a WM develops representations of how objects and environments evolve. Existing world models, however, are primarily optimized for future-state prediction. Their understanding of physical interactions such as contact, force, and collision remains limited, and they cannot directly support cross-task reward estimation or policy generation.

Building on this foundation, we introduce a unified framework that derives two complementary components from the World Model: a World Reward Model for reward estimation and a World Policy Model for policy initialization.

• World Reward Model (WRM): translates the World Model’s understanding of state evolution, task progress, and physical interaction into reward assessments.
• World Policy Model (WPM): translates world understanding into an initial policy capability, allowing a robot to begin exploring a new task from actions that are already closer to a feasible solution.

WM → WRM

The World Reward Model extends the World Model beyond state prediction by converting state changes into task feedback for reinforcement learning. Rather than exposing a large set of task-specific engineering metrics, WRM focuses on the three operational outcomes that matter most in real-world training: Success, In Progress, and Failure.

• Success: the task has reached its target state. WRM determines whether the current scene truly satisfies the task specification. A successful assessment can be converted directly into positive reward, reinforcing effective behavior in the current policy.
• Failure: the task has clearly departed from the goal or entered a state from which recovery is unlikely. Detecting failure allows the system to terminate low-value rollouts early, reduce real-world sampling cost, and provide evidence for policy correction.
• In Progress: WRM evaluates whether the current state is moving the task toward the goal, such as approaching a valid interaction region or producing object changes consistent with successful completion.

Beyond task-state classification, WRM can use the physical dynamics learned by the World Model to evaluate interaction quality throughout execution. From continuous state transitions, it reasons about the relationship between an action and the resulting object response: whether the action caused an effective change in the object state, whether contact remained stable, and whether force and displacement were physically consistent.

In this way, WRM turns the World Model’s implicit knowledge of contact, collision, force, displacement, and spatial relations into physical-interaction value estimates for reinforcement learning. It then organizes task progress and interaction quality into reward signals that provide a clearer supervisory direction for WPM and downstream policies.

WM → WPM

Humans do not approach a new task through completely random exploration. Even when encountering an unfamiliar instruction, they can anticipate how contact and applied forces may change an object’s state, position, and motion. They also possess basic motor skills that allow them to translate this imagined interaction into a plausible initial attempt. By observing the outcome and refining subsequent actions, they gradually learn to complete the task. In essence, this is reinforcement learning grounded in prior physical knowledge, predictive imagination, and motor competence.

The World Policy Model is designed to provide the same kind of foundation. WPM is pretrained on the dynamics and physical-interaction experience captured by the World Model, producing a policy with useful initial task-execution capability. Given a task instruction, WPM generates a physically plausible initial action sequence. The robot therefore enters the task through informed exploration rather than random trial-and-error.

This capability is particularly important for real-world RL, where random exploration causes costly failures, hardware wear, and safety risks. By completing part of a task or moving closer to success at deployment, WPM allows subsequent training to focus on critical failures and fine-grained correction rather than relearning basic motion patterns.

WRM + WPM → Real-world RL

Once WRM and WPM have been pretrained, the system enters real-robot deployment. General WRM and WPM capabilities are maintained in a centralized model layer, while WPM can be distilled or adapted into compact deployable policies for different robots and tasks, from Policy model 1 through Policy model N. These task policies inherit WPM’s initial execution capability and can begin with informed attempts rather than random exploration. The overall training paradigm consists of five stages:

1. Cloud-based foundation models: The system maintains a general WRM and WPM. WPM initializes task execution, WRM evaluates outcomes and provides reward feedback, and a human-in-the-loop coordinator determines when a rollout should be terminated or escalated for intervention.
2. Distillation into deployable task policies: For different deployment sites, robot embodiments, and tasks, WPM can be distilled or compressed into multiple lightweight policies. Each policy begins with a useful level of task-execution capability and continuously generates real interaction data in its target setting.
3. Real-world rollouts and WRM reward assessment: The deployed policy interacts directly with the physical environment, generating and executing actions conditioned on the task instruction. In parallel, WRM determines whether the task is successful, in progress, or failed, and returns the corresponding reward signal.
4. Failure attribution and intervention scheduling: When WRM detects failure or abnormal interaction, the system further identifies the cause. Severe events, such as collision with an obstacle, can terminate the rollout immediately. Recoverable deviations, pose errors, or stalls at a particular stage instead trigger an intervention request. Each request contains the current reward, task stage, failure cause, and anomaly type. A priority evaluator ranks requests by expected intervention value and task priority, identifying which robot is closest to success and would benefit most from human intervention. Limited operator capacity is therefore allocated to the highest-value cases.

5. Continuous data feedback and optimization: Successful, in-progress, failed, anomalous, and human-assisted trajectories from multiple robots and tasks are continuously returned to the cloud system. Reward-labeled interaction data improve WRM’s task assessment and reward feedback. Real-world interaction sequences are used to update WPM and deployed policies, while intervention trajectories, failure attributions, and priority labels strengthen anomaly handling and takeover decisions.

This framework transforms real-world RL from isolated task-specific training into a scalable and generalizable learning system that transfers experience across tasks and robots. WPM provides each robot with a useful initial policy, while WRM delivers continuous reward supervision during physical interaction. Human intervention is required only at critical failure points. The system can then accumulate experience, refine policies, and improve task completion directly in the environments where the robots operate.

Experimental Design and Results

To evaluate the effectiveness and practical utility of WRM and WPM in real robotic manipulation, we deployed the framework on physical robot systems and tested five representative tasks: box closing, plug insertion, chip placement, IMU loading and unloading, and calibration-weight grasping. Together, these tasks cover alignment, contact, insertion, transport, and grasping, enabling a broad evaluation of task-completion assessment, physical-interaction understanding, and policy execution.

The results show that WPM provides a useful initial policy, while WRM supplies reliable reward signals by consistently evaluating task progress and real-world interaction quality. This enables the robot to complete a range of real-world manipulation tasks without requiring a separately engineered complex reward function for each task.

We will soon release the model code as open source, along with the accompanying paper and complete technical details. We believe that continuous generalization represents the next stage of real-world robot learning. This goal cannot be achieved by designing complex reward functions and training a separate policy for each task. It requires transferable initial policies, robust reward assessment, and rapid feedback from physical interaction. WRM and WPM are only the beginning, but they establish a foundation for real-world reinforcement learning that can scale, adapt, and continuously improve.

Citation

If you found this work useful in your research, please consider citing it as:

@article{wrm2026worldrewardmodel,
  author  = {RimBot Research Team},
  title   = {World Reward Model: The Scaling Path for Real-World RL},
  journal = {RimBot Robotics Technical Report},
  year    = {2026},
  note    = {A World Model based reward framework for real-world reinforcement learning}
}