Feeding frames

Formats, strides, ownership, and the conversion you should not perform. Getting this right is most of the difference between a pipeline that fits the SoC budget and one that does not.

Ownership#

xcaicx_frame.data is a borrowed view of caller-owned pixels. The SDK never takes ownership and never frees it.

That means you can hand over an ISP buffer and release it as soon as the call returns. You do not need to keep it alive until the result arrives from poll().

Results run the other way: xcaicx_result and everything it points at — class_name, zone_name, detail, the arrays themselves — are owned by the stream and valid only until the next call on that stream. Copy anything you need to retain.

Supported formats#

GRAY8 and the colour modules#

PPE compliance keys on hi-vis saturation and helmet colour. On a GRAY8 frame those tests have nothing to work with and will report violations indiscriminately. Detection and defect inspection are luma-based and are unaffected.

If your sensor is monochrome, do not license PPE.

Strides#

xcaicx_frame f = {
    .struct_size = sizeof f,
    .data     = isp_buffer,
    .data_len = isp_buffer_len,
    .width    = 1920,
    .height   = 1080,
    .stride   = isp_stride,   /* bytes per row of plane 0; 0 = tightly packed */
    .format   = XCAICX_PIX_NV12,
    .pts_us   = frame_pts,
    .frame_id = seq++,
};

stride is the bytes per row of plane 0. ISPs commonly align rows to 16, 32 or 64 bytes, so a 1920-wide NV12 frame may well have a 1984-byte stride. Passing 0 when the buffer is actually padded produces a sheared image and detections in the wrong places — and it looks like a model problem, not a plumbing problem, which is what makes it expensive to debug.

The engine validates that data_len is large enough for the declared geometry and returns XCAICX_ERR_FRAME_FORMAT with a detail string naming both numbers:

frame.data_len is 3110400 but the declared geometry needs 3214080 bytes

Timestamps and frame ids#

Resolution#

Frames are box-filter downscaled internally so the longest side is 320px for analysis.

This means:

• Feeding 4K instead of 1080p costs conversion and downscale time, and buys nothing in detection quality from the reference backend.
• Small objects that are only a few pixels tall at 320px will not be detected. If you need small-object detection at distance, that is a model and a tiling strategy — talk to us rather than pushing resolution up.
• Detection boxes come back normalised to [0,1], so they map onto whatever resolution you display at.

Box filtering rather than nearest-neighbour is deliberate: nearest aliases thin structures — wires, plate glyphs, hairline scratches — in and out between frames, which destabilises the tracker.

Synchronous vs asynchronous#

Synchronous#

const xcaicx_result *res = NULL;
if (xcaicx_stream_process(stream, &f, &res) == XCAICX_OK) {
    handle(res);
}

Simplest, and correct for most cameras. Inference runs on the calling thread, so your ISP callback blocks for the pipeline duration. Fine when you are running detect_every_n high enough to fit the frame budget.

Asynchronous#

xcaicx_status s = xcaicx_stream_submit(stream, &f);
if (s == XCAICX_ERR_QUEUE_FULL) {
    ++dropped;      /* flow control, not an error */
}

const xcaicx_result *res = NULL;
while (xcaicx_stream_poll(stream, /*timeout_ms=*/0, &res) == XCAICX_OK) {
    handle(res);
}

submit() converts on your thread and runs inference on a worker. XCAICX_ERR_QUEUE_FULL means the worker is behind.

Queue depth is max_queue_depth in the engine config, default 4 per stream. Each queued frame costs a converted copy, so on a memory-tight board set it to 1 or 2.

poll() returns XCAICX_ERR_NO_RESULT when nothing is ready.

Threading rules#

• xcaicx_engine is thread-safe for submit/poll from different threads.
• A single xcaicx_stream must be driven from one thread at a time. Submitting from two threads to the same stream is undefined.
• One stream per video source; that is also how metering attributes usage.

A zero-copy V4L2 loop#

struct v4l2_buffer buf = {
    .type = V4L2_BUF_TYPE_VIDEO_CAPTURE, .memory = V4L2_MEMORY_MMAP };

while (running) {
    if (ioctl(fd, VIDIOC_DQBUF, &buf) < 0) continue;

    xcaicx_frame f = {
        .struct_size = sizeof f,
        .data     = mmap_planes[buf.index],
        .data_len = buf.bytesused,
        .width    = fmt.fmt.pix.width,
        .height   = fmt.fmt.pix.height,
        .stride   = fmt.fmt.pix.bytesperline,
        .format   = XCAICX_PIX_NV12,
        .pts_us   = buf.timestamp.tv_sec * 1000000LL + buf.timestamp.tv_usec,
        .frame_id = seq++,
    };

    const xcaicx_result *res = NULL;
    if (xcaicx_stream_process(stream, &f, &res) == XCAICX_OK)
        handle(res);

    ioctl(fd, VIDIOC_QBUF, &buf);   /* safe: the SDK is done with the pixels */
}

Note bytesperline going straight into stride, and the buffer being requeued immediately after the call returns.

Common mistakes#

Next#

• Zones, tripwires & events — turning detections into events
• Performance & tuning — frame budget, detect_every_n, threads
• C ABI reference — the exact struct definitions