DOC 04PERCEPTION

Camera and underwater optics

The PerceptionFrame camera is LEVIATHAN's scientific camera. A single geometry pass produces every per-pixel channel, so the images, depths, normals and labels cannot disagree with each other. A physically based underwater light model forms the images from that pass.

Viewport with the Sensors menu open: Camera mode buttons RGB, Depth and Seg, FOV and range sliders, the Perception section with Output buttons, Resolution and Capture; a camera frustum drawn from the vehicle toward a wreck and a semantic preview in the monitor. — Figure 4.1 Camera controls in the Viewport. The upper section drives the live display feed; the **Perception** section captures a PerceptionFrame from the simulation service and shows its channels in the monitor.

Display feed versus PerceptionFrame

The live RGB, Depth and Seg modes are rendered by the Viewport's 3D view for display, using the camera's field of view and range. A PerceptionFrame is computed by the simulation service from the same camera pose. It is the scientific product, with declared units, frames, validity and identity, and it is the one exports and datasets use.

Camera geometry

The camera is a pinhole in the OpenCV optical frame: x right, y down, z forward along the optical axis. The field of view is horizontal. Intrinsics follow from the image size and that angle, with square pixels and pixel centres at integer coordinates:

f_x = f_y = (W/2) / tan(hfov/2) c_x = (W − 1)/2 c_y = (H − 1)/2(1)

For example, a 248 × 164 frame with a 90° field of view has f = 124 px and principal point (123.5, 81.5). The camera matrix K, the distortion model and the valid range interval (0, max range) are written into every frame's calibration record.

Lens distortion

Distortion is optional and follows the Brown–Conrady model on normalised coordinates (x_n, y_n), with r² = x_n² + y_n²:

x_d = x_n(1 + k₁r² + k₂r⁴ + k₃r⁶) + 2p₁x_ny_n + p₂(r² + 2x_n²)(2)

The y component is symmetric. Under the opencv convention, a pixel is a distorted image point and its ray is undistort(K⁻¹[u, v, 1]), as in OpenCV's calibration model. A legacy_v0 convention reproduces datasets made before PerceptionFrame. Inversion is iterative; motion products mark pixels outside the lens model's invertible domain as distortion undefined (see Surface correspondence).

Ground-truth channels

Figure 4.2 Range and z-depth are distinct channels and are never interchanged. Range is the Euclidean distance along the pixel ray; z-depth is its component along the optical axis.

Table 4.1 PerceptionFrame geometry channels. H × W arrays, row 0 at the top. Missing values (no hit within the range limit) are null in float channels.
Channel	Type	Units · frame	Meaning
`valid`	bool	—	The ray hit geometry within (0, max range)
`semantic`	uint16	—	Canonical class id; open water is the water class; 65535 marks exact geometry with no semantic annotation
`instance`	uint16	—	Instance id from the frame's object table; 0 for water and seabed
`range`	float64	m · camera centre	Euclidean distance from the camera centre to the hit along the ray
`z_depth`	float64	m · camera	Distance of the hit along the optical axis
`normals_world`	float64 × 3	unit · NED	Surface normal, oriented toward the viewer
`normals_camera`	float64 × 3	unit · camera	The same normal rotated into the camera frame

Semantic labels use a fixed marine vocabulary, which includes water, seabed, diver, plant, fish, reef, wreck, robot, ship hull, anode, bilge keel, propeller, sea-chest grating, marine growth, corrosion, defect, pipe, cable, manifold, wellhead, mine, debris, boulder, mooring and buoy. Each class has a fixed id and display colour, and common synonyms (for example rock, riser, fouling) map onto it.

Underwater image formation

Figure 4.3 The radiometric chain. All radiometric quantities are expressed relative to the surface irradiance E_s, so they are dimensionless or in sr⁻¹.

The model is the revised underwater image formation model of Akkaynak and Treibitz (CVPR 2018). It separates the coefficient that governs direct transmission, β_D, which depends on range, from the coefficient that governs backscatter, β_B. The older atmospheric dehazing model uses one coefficient for both, and that is wrong underwater.

Ambient illumination. Light from the sky arrives with an upper-hemisphere radiance distribution and is attenuated with depth. The irradiance on each surface depends on its depth and its viewer-oriented normal.
Clean radiance. A Lambertian surface of reflectance ρ leaves radiance J = ρE_n/π. This is rgb_clean: the object as lit underwater, without the water between it and the camera.
Water transport. Along the ray of length r, the direct signal keeps a fraction t = exp(−β_D(r)r), and the water adds single-scattered path radiance B. The radiance at the camera is I = J·t + B. This is rgb_degraded, unclamped.
Sensor response. LEVIATHAN's normalised, quantised linear camera response turns radiance into a fraction of full scale. It has exposure (an EV value or automatic), knee, bit depth, read noise, shot noise and black level. This is rgb_sensor, which is not a Bayer or RAW image. rgb_sensor_clean is the clean radiance through the same exposure, without noise.
Display. rgb_display applies the sRGB transfer function for viewing only. It is never ground truth, and it has its own identity hash.

Table 4.2 PerceptionFrame radiometric channels.
Channel	Units	Meaning
`albedo`	—	Lambertian reflectance ρ of the surface hit; its source is recorded per object
`irradiance`	E / E_s	Ambient irradiance on the surface at its depth
`rgb_clean`	sr⁻¹ (radiance / E_s)	Clean surface-leaving radiance J
`transmission`	—	Direct transmission t over the camera ray
`path_radiance`	sr⁻¹	Single-scattering water-path radiance B
`rgb_degraded`	sr⁻¹	Radiance at the camera, I = J·t + B
`rgb_sensor`	fraction of full scale	Camera-response image with noise
`rgb_sensor_clean`	fraction of full scale	Clean radiance through the same exposure, no noise
`rgb_display`	sRGB [0, 1]	Visualisation only
`background`, `surface_not_modeled`	bool	The ray looks into open water, or rises toward the sea surface without a hit
`radiometric_valid`, `radiometric_reason`	bool, uint16	Whether the pixel's whole radiometric chain is modelled, and if not, why

Water types

Water is homogeneous and is specified by its beam attenuation coefficient c per colour channel, in m⁻¹. You can pick a named type or give your own non-negative [r, g, b].

Table 4.3 Named water types: beam attenuation c in m⁻¹ for the red, green and blue channels. The Jerlov types follow the classical oceanic (I–III) and coastal (1C–9C) classes.
Type	c_R	c_G	c_B	Type	c_R	c_G	c_B
clear	0.42	0.09	0.03	Jerlov II	0.52	0.12	0.07
coastal	0.55	0.12	0.18	Jerlov III	0.64	0.16	0.12
turbid	0.88	0.30	0.45	Jerlov 1C	0.78	0.23	0.24
murky	1.90	0.90	1.20	Jerlov 3C	1.05	0.34	0.40
Jerlov I	0.34	0.07	0.021	Jerlov 5C	1.35	0.48	0.62
Jerlov IA	0.39	0.083	0.028	Jerlov 7C	1.75	0.68	0.90
Jerlov IB	0.44	0.095	0.038	Jerlov 9C	2.30	0.98	1.35

Ambient illumination

An ambient model is required: LEVIATHAN has no default, because the choice changes every radiometric value. Two models are available. A power-cosine sky has radiance proportional to cos^m of the zenith angle, and you must state the exponent m. An isotropic sky is the case m = 0.

Outputs in the application

One capture produces every output from the same camera, pose and scene, so the channels line up pixel for pixel. Figure 4.4 shows five of them from a single Pipeline capture: the camera looks down at 32° over the manifold, the flowlines and a flanged pipe section, with coastal water and an isotropic sky.

Underwater channel of the Pipeline capture: green seabed fading into a grey-green water veil toward the top, with the manifold, flowlines and pipe drawn in magenta where radiometry is invalid; the concrete pipe supports appear as dark, validly lit blocks. — Underwater RGB

Semantic channel in canonical label colours: seabed white, pipe and flowlines cyan, an anode orange, and the manifold and other structure parts in their own label colours. — Underwater RGB

The same capture also holds the Clean, Sensor and Albedo images, Range, and normals in the NED world frame. Each is selected under Output in the camera panel and shown in the camera monitor.

Magenta in the image channels marks pixels without valid radiometry. The Pipeline structure's material map establishes reflectance only for its concrete pipe supports. The pipe, manifold, anodes and markers have geometry and labels but no established reflectance, so their radiometric reason is material_unknown. This capture used Estimated reflectance for the seabed, which the frame records as unmeasured.

Radiometric validity

A pixel's radiometry is reported only where every link of the chain is modelled. Otherwise the radiometric channels hold null, never a plausible-looking value such as black, and radiometric_reason records why:

Table 4.4 Radiometric reason codes. The highest applicable code is recorded.
Code	Reason	Effect
0	valid	All radiometric channels present
1	material_unknown	No scientific reflectance: albedo, clean, underwater and sensor values are null
2	interface_crossing_unsupported	The camera-to-surface path crosses the air–water surface, which is not modelled
3	above_water_unsupported	The path or surface is in air; there is no topside illumination model

Where reflectance comes from

Scene models carry display colours and textures, which are not evidence of how a surface reflects light. Reflectance is resolved in this order:

an explicit reflectance given for the object;
a material from LEVIATHAN's scientific material library, through a structure material map;
only if you allow it, the display palette, recorded in the frame as an unmeasured estimate.

The scientific material library holds reflectances reduced from source spectra in the ECOSTRESS spectral library, currently a set of concretes. One built-in assignment exists: the pipeline site's concrete saddles. It is declared a proxy, because the values are dry-laboratory measurements not corrected for immersion. Every other surface is unknown until reflectance is supplied.

Completeness

Every frame states whether it is complete. When something physically present cannot be represented, the frame lists it with a category and a reason, and is marked incomplete. Examples are the ego vehicle, which is drawn by a display mesh only and has no scientific geometry, and entities without semantic labels. Omission categories include unsupported geometry, unsupported semantic labels, non-metric geometry, unsupported dynamic entities, unmapped entities, and participating media, illumination effects and interfaces that are not modelled.

Capture settings and limits

Table 4.5 Capture settings and service limits.
Setting	Value
In-app resolutions	124 × 82, 186 × 123, 248 × 164 (the feed's aspect)
Service limits	Up to 1024 px on a side and 65,536 pixels per frame; 16 frames and 262,144 pixels per dataset request
Scene objects	Spheres, boxes, meshes, mesh assets and structures; flat or absent seabed; up to 64 objects
Seed and frame index	Root seed and frame index; per-source random streams derive from them
Arrays	IEEE-754 binary64, unrounded

Not modelled

Direct sunlight, artificial lights, the sea surface (reflection and refraction), multiple scattering, forward-scatter blur, ambient light varying along the ray, discrete suspended particles and stratified water. Pixels that depend on these are marked rather than approximated.