Troubleshooting

Symptom first, because in the field you have a symptom, not a status code. If you do have a code, go to Status codes.

Licensing#

Every camera fails with BAD_SIGNATURE, all at once#

Almost never a corrupted token. The licensing server's signing key and the public key compiled into the SDK no longer match — typically after a server redeployment that generated a fresh key.

curl -s https://licensing.xcaicx.com/healthz

Compare public_key against the key in your SDK build (core/src/license/root_key.inc). If they differ, restore the original signing key on the server; do not re-sign the fleet.

A single camera fails with WRONG_DEVICE#

Its fingerprint changed, or its token came from elsewhere.

# On the camera
echo "$XCAICX_DEVICE_ID"
# and what the token is bound to
xcaicxctl inspect /var/lib/xcaicx/unit.token

ACTIVATION_REQUIRED on a camera that is online#

The camera is holding a batch key, not a unit token, and has not completed the exchange.

1. Is cfg.cloud_endpoint set, and cfg.offline zero?
2. Can the camera actually reach the authority? curl -s $ENDPOINT/healthz from the unit.
3. Is state_dir writable? An activation that succeeds but cannot be persisted repeats forever — and consumes a unit each time.

That third case is worth checking early: it is silent, and it eats the batch.

Activation returns 402#

The batch is exhausted.

xcaicxctl batch-status B-2026-1C4417D5

If units_activated is higher than the number of cameras the customer has actually built, the fingerprint is unstable — every reflash is registering as a new device. Fix the fingerprint before selling more units, or you will sell the same cameras twice and hear about it.

Activation returns 429#

Edge rate limit: 20 requests per 10 seconds per IP. A fast provisioning line will trip it. Allowlist the line's egress IP rather than raising the global limit.

CLOCK_ROLLBACK after a power cut#

The RTC battery is dead, so the camera boots with a clock behind the persisted high-water mark. Fix NTP or the battery. This is the SDK behaving correctly — but tell the technician what it means, because "clock rollback" sounds like an accusation.

Licence expired but the camera still works#

That is XCAICX_LIC_GRACE. Expired, inside the grace window, everything still runs. Surface it as a warning. It is not a bug, and it exists so a renewal that slips by a week does not stop a production line.

Detection quality#

Nothing is detected at all#

1. Is the module entitled? xcaicx_engine_active_modules() tells you what is actually running.
2. Is detect_threshold too high? Drop it to 0.20 temporarily.
3. Are frames arriving? Check result.total_ms is non-zero and frame_id advancing.
4. Is the image what you think it is? Dump one frame to disk and look at it. A stride bug produces a sheared image in which nothing is recognisable.

Detections are offset, doubled or sheared#

stride is wrong. ISPs align rows to 16/32/64 bytes; passing 0 when the buffer is padded produces exactly this. Set stride to the ISP's bytesperline.

PPE reports violations on everyone#

Check in this order:

1. Format. BGR passed as RGB inverts the colours, and hi-vis orange becomes blue.
2. Greyscale. On GRAY8 the colour tests have nothing to work with. Do not run PPE on a monochrome sensor.
3. Lighting. Sodium or heavily tinted lighting moves hi-vis outside the saturation band.
4. The backend. The reference backend's PPE is a colour-ratio heuristic. On real imagery it is not production accuracy, and no amount of tuning makes it so.

Defect inspection flags everything, or nothing#

The defect module assumes a uniform product. On a textured or patterned surface, every tile is an outlier and the MAD floor stops being meaningful.

• Flags everything: the surface is not uniform, or the lighting is uneven across the frame.
• Flags nothing: the defect is smaller than one tile, or its contrast is below the z ≥ 6 gate. Tighter framing raises the effective resolution of the inspection.

ANPR returns boxes but no text#

Expected on the reference backend — it localises plates and cannot read them. text is empty and detail is plate_candidate. Reading characters needs a production build with the text model. See Backends.

Tracking and events#

Track ids change constantly#

• Frames arriving out of order, or pts_us not monotonic.
• detect_every_n too high for the object speed — the tracker coasts too far.
• Detections flickering below detect_threshold; lower it slightly.

Events fire twice, or not at all#

Performance#

The pipeline cannot keep up#

In order of effectiveness: raise detect_every_n; subset modules per stream; feed a smaller analysis stream; move to an accelerated backend. Full detail in Performance & tuning.

QUEUE_FULL constantly#

The worker is behind. This is backpressure — drop frames and count them. If the rate keeps climbing, the board has outgrown the scene.

Memory grows over time#

• Retaining xcaicx_result pointers past the next call on that stream. They are stream-owned; copy what you keep.
• max_queue_depth too deep — each queued frame is a converted full-resolution copy.

Metering and billing#

The customer's usage looks too low#

Unshipped spool. Compare the camera's local counters against what the server recorded:

cat /var/lib/xcaicx/usage.json
xcaicxctl usage LIC-…

A gap means the spool has not been acknowledged yet — the usage is not lost, it is pending. It ships on the next successful heartbeat.

Usage resets to zero#

state_dir is not persistent. Almost always /tmp, a tmpfs, or a read-only rootfs with no writable overlay. This also means the camera re-activates on every boot and consumes a unit each time, so check batch-status for the damage.

Counts look double what was expected#

Check how many modules are enabled. Metering is one inference per module per frame — detect plus ppe at 30fps is 60 calls a second, not 30.

Build and load#

Getting help#

Include these, and the first reply will be an answer rather than a question:

xcaicx_version_string()                    # SDK version
xcaicx_status_str(st), xcaicx_last_error() # the exact failure
xcaicxctl inspect unit.token               # the licence, minus nothing
curl -s $ENDPOINT/healthz                  # the authority's public key

Plus the engine's INFO log from startup — it names the backend that was actually selected, which answers the most common question before it is asked.

[email protected]