DOC 06AUTONOMY DEVELOPMENT

Reinforcement learning and evaluation

LEVIATHAN provides marine control tasks as learning environments, batched dynamics for training at scale, observation and actuator models taken from hardware datasheets, and an evaluation layer. The evaluation layer reports not just whether a policy succeeded, but at which stage it failed.

Tasks

Fourteen tasks are registered as Gymnasium environments, each on four thruster-driven vehicles: BlueROV2 Heavy, BlueROV2 Heavy (faithful), Hovering AUV and Work-class ROV. That makes 56 environment ids, named like MarineLab-StationKeep-BlueROV2Heavy-v0.

The Dashboard's Reinforcement learning and Grade suite panels also include scene-specific variants: reef avoidance, deep-sea and harbour transits, and dam inspection.

Observations, actions and models

• Actions are per-thruster commands in [−1, 1], scaled to force and mapped to a wrench through the vehicle's thruster configuration, so the policy drives thrusters directly. Observations are each task's state features, such as the goal vector in the body frame, body velocities and attitude, with optional noise and partial observability.
• DVL observation model. A DVL is modelled from the datasheet, not as a coin flip. It reports at its own rate (2–15 Hz for a Water Linked A50, 8 Hz by default) and loses bottom lock outside its altitude envelope (0.05–50 m) or beyond 25° of tilt. During an outage the solution dead-reckons and drifts rather than reporting zero. Presets include benign conditions and a soft-bottom envelope.
• Actuator faults. Thruster degradation and failure, with fault-aware allocation available to controllers.
• Domain randomisation. Mass, inertia, added mass, linear and quadratic damping, volume, current and JONSWAP sea state can be randomised. Transport delay models latency.
• Policy architectures. MLP, GRU, LSTM and transformer (GTrXL) policies, and a CNN encoder for sonar images.

Training

Training runs

The Training runs panel launches a training job in one of three ways: as a local process on this machine's GPU, as a job submitted to a Slurm cluster, or as a dry run that only produces the job script for inspection. The job can be a Code project's run command, or a generated job for the Isaac Lab and MarineGym cluster setup. The panel follows the run's status, metrics and checkpoints, and a checkpoint can be exported into the portable policy format the evaluation panels accept. Cluster runs need that cluster and its software; they are not bundled.

Environments run batched on the GPU through the vectorised API. A Stable-Baselines3 adapter is also provided. Policies export as TorchScript, as a dependency-free JSON MLP that the evaluation panels load directly, and as ONNX, verified by running the exported model.

Evaluation

KRAKEN is organised in three parts. K1 holds trajectory data: the schema, ingest of LEVIATHAN rollouts and of recorded training rollouts, and a catalogue. K2 holds the stage verifiers, the multi-seed runner and regression baselines that the Dashboard's grading panels use. K3 covers fidelity: degradation curves across randomisation axes and comparison of simulated with real trajectories. It is available from the library and its command line.

LEVIATHAN's evaluation layer, KRAKEN, grades every episode by stage, not only by success. An episode ends in one of four stages: never engaged, engaged but not acquired, acquired then lost or succeeded. A policy that never engages has a different problem from one that reaches the goal and drifts off, and the stage mix shows which.

Aggregate results follow the practice recommended by Agarwal et al. (NeurIPS 2021) for few-seed reporting: medians, interquartile means and stratified bootstrap intervals, rather than a bare mean and standard deviation.