Performance & tuning

Fitting the pipeline into an SoC budget without losing the events you are being paid to produce.

Where the time goes#

Per frame, in order:

1. Convert — YUV→RGB plus luma, over the full frame. Cost scales with input resolution.
2. Downscale — box filter to 320px on the long side.
3. Infer — once per enabled module.
4. NMS — negligible.
5. Track + rules — negligible; tens of objects at most.

xcaicx_result reports both inference_ms and total_ms, so you can see the split directly:

printf("%.1fms inference of %.1fms total\n", res->inference_ms, res->total_ms);

If total_ms - inference_ms is large, you are paying for conversion — feed a smaller or better-matched format. If inference_ms dominates, use the levers below.

The levers, in the order to try them#

1. detect_every_n#

The single most effective control.

sc.detect_every_n = 3;   /* infer on every third frame */

Inference runs on every Nth frame; tracking coasts across the gaps on each track's smoothed velocity, so zone and tripwire events still fire. Metered inferences drop by roughly the same factor, which also reduces the customer's bill.

2. Subset modules per stream#

sc.modules = XCAICX_MODULE_DETECT;   /* not the engine's full set */

Every enabled module costs one inference per frame whether or not it finds anything. A gate camera does not need PPE; an inspection camera does not need ANPR.

3. Input resolution#

Analysis happens at 320px on the long side regardless. Feeding 4K instead of 1080p costs conversion and downscale time and buys nothing from the pipeline. If your ISP can emit a second, smaller stream, use it for analysis and keep the full-resolution stream for recording.

4. Format#

NV12 or I420 straight from the ISP. Converting to RGB yourself doubles the copy. See Feeding frames.

5. Threads#

cfg.num_threads = 0;   /* 0 = auto */

Auto is usually right. On a camera where the video path must never be starved, pin the SDK lower than the core count so encoding keeps its headroom.

6. Queue depth#

cfg.max_queue_depth = 2;   /* per stream; default 4 */

Each queued frame holds a converted copy. On a memory-tight board, 1–2 with the synchronous xcaicx_stream_process is often better than a deep queue that adds latency without adding throughput.

Synchronous or asynchronous#

With submit/poll, XCAICX_ERR_QUEUE_FULL is flow control. Drop the frame; do not retry in a loop and do not sleep in the ISP callback.

if (xcaicx_stream_submit(stream, &f) == XCAICX_ERR_QUEUE_FULL) {
    ++stats.analysis_frames_dropped;   /* count it, then move on */
}

Counting drops is worth doing: a rising drop rate is the earliest signal that a site's scene has become busier than the board can handle.

Thresholds#

Neither reduces inference cost — the work is already done by the time they apply. They control what reaches the tracker.

A worked budget#

A 30fps 1080p NV12 stream with detect and ppe enabled:

• Metered calls: 2 modules × 30fps = 60/second, 5.18M/day.
• With detect_every_n = 3: 2 × 10 = 20/second, 1.73M/day.

If a board can run the pipeline in 60ms, then at detect_every_n = 1 it cannot keep up with 30fps (33ms/frame) and will either block the ISP or fill the queue. At detect_every_n = 3 it has 100ms of wall time per inference and fits comfortably.

Work the arithmetic that way round — from the measured total_ms on your board with your model — rather than from a target frame rate.

Measuring on the target#

uint64_t n = 0; double acc = 0;
...
if (xcaicx_stream_process(stream, &f, &res) == XCAICX_OK) {
    acc += res->total_ms; ++n;
    if ((n % 300) == 0)
        log_info("mean %.1fms over %llu frames", acc / n, (unsigned long long)n);
}

Measure with the real model, the real resolution and a realistic scene. An empty corridor is cheap: the reference backend finds nothing, connected-component labelling has nothing to label, and the number you get is not the number you will see at shift change.

Memory#

Roughly:

• Analysis image: 320px long side, RGB + luma.
• Per queued frame: one converted full-resolution copy.
• Tracker and rule state: tens of objects; negligible.
• state_dir on disk: ~64 KB, growing only during long offline periods.

The model bundle dominates a production image. Budget for it first.

When none of it is enough#

• Move to an accelerated backend — RKNN, Hailo, TensorRT, OpenVINO. That is one interface each; see Backends & models.
• Reduce the analysis stream resolution at the ISP rather than in software.
• Split modules across cameras rather than running all four on one weak unit.

Next#

• Feeding frames — the copies to avoid
• Backends & models — accelerator paths
• Metering & billing — how these settings change the bill