DOC 04PERCEPTION
Synthetic data and export
LEVIATHAN writes ground truth in its own native format, which is lossless and content-addressed. Adapters derive views for common tools: COCO, KITTI, depth maps and optical-flow formats. Each adapter records what it loses.
Formats at a glance
| Format | Contains | Lossless | From |
|---|---|---|---|
| Native frame / dataset | Every PerceptionFrame channel, frame records, object tables, calibration, provenance; optionally sonar, DVL, IMU, pressure, magnetometer and USBL measurements | Yes | Viewport Export → Frame / Dataset; service |
| COCO | Instance boxes and segmentation masks with canonical class ids | No | Viewport Export → COCO; service |
| KITTI | 3D object boxes for the camera, as KITTI camera 2 | No | Viewport Export → KITTI; service |
| Depth | Z-depth or range maps in float64 | Yes, for that channel | Viewport Export → Z-depth / Range; service |
| Native motion dataset | Surface correspondence, forward and backward optical flow, scene displacement, motion fields | Yes | Service |
| Sintel / Middlebury .flo | Optical flow (u, v) | No (float32) | Service |
| KITTI 2015 flow PNG | Optical flow with a validity channel | No (1/64 px steps) | Service |
| PFM | Flow, scene displacement, motion field or point velocity, with a JSON sidecar | No (float32) | Service |
| Native track dataset | Tracks, observations, events and boxes for one camera | Yes | Service |
| Native cross-modal dataset | Camera–sonar contributor tables, observability and query indexes | Yes | Service |
| Native temporal camera–sonar dataset | Pairings of camera and sonar measurements taken at different times, with time-compensated ground truth per direction Under review | Yes | Service |
Native datasets
The native dataset is the authoritative form. Writing and reading never regenerate provenance, change a hash or round an array.
| Entry | Content |
|---|---|
| manifest.json | Dataset schema, ordered frames, channel registry with units and frames, conventions, dataset hash |
| frames.jsonl | One canonical-JSON record per frame: camera calibration, pose, objects, completeness, provenance, identity |
| channels/<channel>/<frame>.npy | Scientific arrays, NumPy format, little-endian, in each channel's own type (float64, uint16, bool) |
| representations/rgb_sensor/…png | 16-bit PNG of the sensor codes, exact |
| representations/rgb_display/…png | 8-bit sRGB PNG, for viewing only |
| measurements.jsonl, measurements/… | Optional sonar, DVL, IMU, pressure, magnetometer and USBL measurement records with their arrays |
- Missing values. A missing float, such as a pixel with no hit, is written as NaN and read back as null. A NaN that is actually present in the data is refused at write time, so the mapping is exact.
- Dataset hash. The SHA-256 of the ordered list of frame content hashes. It does not depend on file names, timestamps, compression or display images.
- Capture-group datasets. These package all sensors captured at the same instants, cameras and measurements together, in one dataset.
COCO
- Category ids are LEVIATHAN's canonical semantic ids; names are the canonical labels.
- There is one annotation per instance visible in the frame. The box [x, y, w, h] comes from the visible mask, with pixel centres at integers and w = umax − umin + 1. The segmentation is uncompressed COCO RLE, column-major, starting with background.
- Extra keys keep the persistent object identity and the frame's content hash.
- Not representable: range and z-depth, normals, radiometry, 3D pose, water and seabed as "stuff", missing pixels, amodal extent.
KITTI
A PerceptionFrame has one camera, exported as KITTI camera 2. Each visible object gets a 3D box (height, width, length, location, rotation and observation angle) in the camera frame.
KITTI boxes are produced for spheres, and for meshes whose rotation is the identity with one object axis parallel to the camera's y axis, seen by an undistorted pinhole camera. Other objects are refused, or skipped when you ask for that, rather than given an invented box. Calibration entries for other cameras, LiDAR and IMU are reported as unavailable.
Depth maps
A depth export is a set of float64 maps whose definition is always recorded: z_depth (distance along the optical axis, the usual computer-vision convention) or range (Euclidean distance along the ray). Missing pixels are NaN, and the export says so.
Motion and flow formats
The native motion dataset holds correspondence, motion ground truth and motion fields without loss. External formats are one-way projections with stated losses.
| Format | Encoding | Options and losses |
|---|---|---|
| Sintel / Middlebury .flo | "PIEH" header, width, height, then float32 (u, v) pairs, row-major from the top. Unknown pixels are written as 1010 | Mask valid (visible only, the default) or geometric (every finite flow, including occluded points). Float64 is reduced to float32, and the maximum rounding error is recorded. Status codes go to a sidecar PNG |
| KITTI 2015 flow PNG | 16-bit RGB PNG: u = (R − 215)/64, v likewise, B = valid | Variant noc (visible only) or occ (every finite geometric flow). Rounded to the nearest 1/64 px, and the maximum error is recorded. Values outside ±512 px are refused, never clamped |
| PFM | Float32, rows bottom to top, little-endian; three-channel for vectors, one-channel for masks | Quantity: flow, scene displacement (metres, NED), motion field or point velocity. A JSON sidecar is mandatory because PFM does not record channel meaning. NaN preserved |
A request may hold up to 16 frames and 65,536 pixels per frame.
Conventions in every export
| Item | Convention |
|---|---|
| Pixels | Centres at integer (u, v); principal point ((W−1)/2, (H−1)/2); row 0 at the top |
| Frames | World NED, body FRD, camera OpenCV optical (see Coordinate frames) |
| Units | Metres, radians, seconds or integer nanoseconds; radiometry relative to surface irradiance |
| Depth | Range and z-depth are always separate; no channel is called "depth" |
| Labels | Canonical semantic ids; 65535 means unannotated geometry, to be ignored |
| Determinism | Bitwise reproducible from the same spec, seed and frame index on the same build and platform |
Legacy synthetic-data generator
The Dashboard's Synthetic data panel runs LEVIATHAN's earlier generator. It produces domain-randomised scenes of labelled primitives on a flat seabed, with segmentation or detection labels. It can grade the result and export COCO, KITTI, depth and normals. A related generator turns imaging-sonar and side-scan images into COCO or KITTI detection labels. Legacy These generators are deterministic for a seed, but they use simplified scenes and do not carry the canonical identity, completeness and provenance described above. Use PerceptionFrame datasets for new work.
Other data products
- Learning datasets: offline reinforcement-learning datasets of transitions (see Learning and evaluation).
- Replay: ROS 2 bags and pose or odometry logs can be ingested and played back in the Viewport.
- Charts and surveys: gridded bathymetry and chart tiles from the hydrographic tools (see Mapping and navigation).