DOC 04PERCEPTION
Perception and computer vision
Beyond ground truth, LEVIATHAN includes perception algorithms and the means to evaluate them: classical and learned vision, sonar detection and tracking, camera–sonar association, and benchmarks with frozen, held-out splits. This page states what each component is and how far it has been demonstrated.
Perception algorithms run from LEVIATHAN's Python library and its benchmark harnesses. Their inputs are canonical sensor products: PerceptionFrames, sonar measurements and IMU and DVL measurements. They are evaluated against the ground truth described in Perception ground truth. They are not exposed as application panels or service endpoints, and no trained model weights ship with LEVIATHAN; the model definitions, training code and evaluators do.
Four kinds of perception work
| Kind | What it means | Where |
|---|---|---|
| Ground-truth generation | Exact, simulation-derived answers: geometry, labels, motion, tracks, sonar contributors | Perception ground truth |
| Synthetic data | Exports of that ground truth in native and common formats | Synthetic data and export |
| Evaluation tooling | Metrics and harnesses that score any algorithm's output against ground truth | This page |
| Algorithms | Baseline and learned methods that produce perception outputs from sensor data | This page |
Algorithms and their maturity
Each capability below is graded by the evidence behind it, all of it from simulated held-out data unless real data is named.
| Capability | Method | Grade |
|---|---|---|
| Camera geometry | Canonical rays, projection and lens handling shared by every vision method | Mature |
| Feature matching | Classical detectors and matchers, graded against surface correspondence; turbidity limits characterised | Credible |
| Two-view geometry, PnP | Essential-matrix and pose estimation with explicit degeneracy and failure states. Planar seabed views are often refused as ambiguous; IMU rotation can resolve the ambiguity | Credible |
| Visual odometry | Monocular, feature-based, with keyframes and local bundle adjustment; scale-free | Partial |
| Visual–inertial–DVL odometry | Loosely coupled fusion of camera, IMU and DVL on moving-vehicle scenes | Partial |
| Pose reliability | Decides whether a visual pose estimate is trustworthy enough to aid the state estimator | Mature (simulation) |
| Depth | Dense depth-model integration, learned monocular depth graded on structured scenes, metric scale anchored by DVL ranges on flat seabed | Partial |
| Object detection | Contrast-blob baseline; a learned Faster R-CNN fine-tuned on LEVIATHAN synthetic data | Credible (sim) Weak (real) |
| Segmentation | Semantic (DeepLabV3) and instance (Mask R-CNN) models trained on synthetic data | Credible (sim) Weak (real) |
| Multi-object tracking | IoU and Hungarian assignment, with an appearance re-identification variant; bounded by detection quality | Partial |
| Optical flow | Classical baselines (DIS, Farnebäck) and RAFT, graded against motion ground truth | Credible (sim) |
| Underwater restoration | Classical baselines and a small learned U-Net that estimate the clean image from the underwater one | Partial |
| Sonar detection and tracking | 2-D CFAR detection on the canonical sonar image, object and shadow extraction, multi-ping tracking on a moving vehicle | Credible (sim) |
| Camera–sonar association | Geometric association of sonar detections with camera detections along the sonar's elevation arc, including asynchronous captures compensated for vehicle motion | Mature (simulation) |
| Runtime contracts | A common result contract (ok, refused or failed, with timestamps and provenance), an algorithm registry and device management | Partial |
Learned models record their provenance: the code version, the initial weights and their licence, and the training split. Restoration is not placed in front of flow or odometry, because its gains in image fidelity come with artefacts that harm motion estimation.
Evaluation tooling
Every evaluator scores outputs against simulation ground truth, and several are cross-checked against reference implementations.
| Task | Ground truth and metrics |
|---|---|
| Detection | Boxes from object-track observations; COCO-style AP and AR, cross-checked against pycocotools |
| Segmentation | Semantic channel and track rasters; semantic and panoptic metrics |
| Tracking | Track identities from the simulation; CLEAR-MOT and IDF1, verified against py-motmetrics |
| Feature matching | Surface correspondence answered for arbitrary sub-pixel keypoints |
| Optical flow | Motion ground truth with non-occluded and occluded regions |
| Depth | Range and z-depth channels, with each model's declared output (metric, relative or inverse) respected |
| Trajectories | Absolute and relative trajectory error with SE(3) or Sim(3) alignment; monocular scale is reported, not claimed |
| Sonar detection | Detection truth from sonar contributors; cells without valid acoustics are excluded |
| Restoration | The clean sensor image from the same capture as the target |
Benchmarks
Seeded scene families with designed occlusion, turbidity and seabed variation are split into training, validation and test groups. The test split was frozen before any tuning, and uncertainty is reported with a bootstrap over scene groups. Procedural seabed families (sand, mud, rock, mixed), parametric debris such as bottles, cans, tyres and pipes, surface biofouling cover and geometry-anchored textures supply the scene variation. Splits are assigned by a recorded, salted rule over scene groups, so a scene never appears in two splits.
Real data
Learned perception trained on LEVIATHAN's synthetic data has been evaluated on one real underwater dataset, the SeaClear marine-debris set, for 2-D detection. Zero-shot performance on real images is weak. After fine-tuning on a few hundred real images it is not worse than a COCO-pretrained detector, but that does not establish an advantage. Further real-data evaluation is blocked until fresh real test data is available. LEVIATHAN does not claim real-world perception performance.
Camera domain adapter
A separate post-processing step can apply consumer-camera automatic white balance to a canonical camera image, to resemble the colour handling of typical ROV cameras. It produces a new, labelled image. The canonical sensor image is never modified.