Judging a run

A run ends with two files under run/: what the robot says happened, and what the simulator saw.

The task's own account

run/task_result.json is written by the task runner:

• the status: success, failure, timeout, error or not_started;
• the start and end times, in simulation seconds;
• one event per action, with its parameters, times and outcome;
• the final pose the robot believes it has, and the distance its odometry measured;
• unstopped_actions, when a Nav2 action the run canceled had not stopped 5 s later.

It is the robot's belief, as good as its localization and odometry.

Three verdicts

While the task runs, the runner also records the robot's pose as the simulator publishes it, the joint states, and every bumper's reports. When the task ends it writes run/evaluation.json with three verdicts, each resting on different evidence:

Verdict	Rests on	Fails when
task	The task runner's status	The task did not succeed
physical	The simulator's pose alone	The robot ended away from its goal, or touched an obstacle
checks	Each check of the .eval file	A check's value left its bounds

The physical verdict asks two questions:

• Did the robot end near its goal? The last goal it navigated to, within twice the controller's goal tolerance. The controller stops on its own belief, and the ground truth differs from that belief by the localization error.
• Did it touch an obstacle? Its closest approach to any Obstacle over the run, measured between the collision shapes seen from above; within 5 mm is a touch.

The three can disagree, and that is the point. A task that reports success while the robot stopped short of the goal, or brushed a box on the way, fails the physical verdict.

How checks are measured

Checks	Read
Check_vel, Check_acc, Check_jerk, Check_effort	The named wheel's joint states, every sample judged by magnitude
Check_distance, Check_clearance	The recorded poses and the planar shapes in task/evaluation.json
Check_contact	A bumper's own reports; for any other part, the poses and shapes
Check_time	The task's duration

A check with no samples to judge is unknown, never pass. Nothing is read from odometry or from Nav2.

Shapes

The shapes the evaluation measures are the collision shapes the robot is simulated with, seen from above: the chassis and sensor collision boxes, each wheel and each ground contact a part declares as a circle of its radius, and each scene entity as the region its shape covers seen from above, tilted or not. A part's ground contacts count as that part, so a check on the chassis also measures the skid under it. A part without a collision box counts as a point, which is one more reason to declare one.