Image Classifier Architecture
The Image Classifier runs object-detection models on video for two workloads. Alarm clip classification looks at the recorded clip of an alarm and returns one verdict for that alarm. Live-stream analytics classifies a camera's live video frame by frame while someone watches it, or continuously when the camera is pinned. Both workloads run in the imageclassifier service and share one inference engine per model on a host. Setup, configuration and the alarm codes are on Image Classifier.
The two paths
| Alarm clip classification | Live-stream analytics | |
|---|---|---|
| Trigger | An alarm on a camera whose alarm binding has analytics turned on | A viewer turns analytics on, or the camera is pinned for always-on classification |
| Input | The recorded clip of the alarm, or the media the camera pushed with it | The camera's live H.264 stream |
| Frames | 1 per second, up to 30 per clip | Every Nth frame (imageclassifier.frame.interval, default 5) |
| Output | One verdict per alarm, a snapshot and an optional alarm-log signal | Detection boxes on the viewer's video and one detection event per new track |
| Turned on by | clipacquirer.enabled, imageclassifier.mediaReady.enabled and their tenant lists | imageclassifier.enabled with imageclassifier.devices, or imageclassifier.alwaysOn.enabled |
System context
The Image Classifier runs inside sbn-media. A fault-tolerant deployment runs sbn-media on two hosts that share the classifier work, and one NATS cluster of three nodes: one on each host and a quorum node that runs no products. The cluster keeps working when either host fails. Alarm classification starts from an alarm event, either pushed by the camera or polled from SBN live traffic through the API Engine, and ends with a signal posted to the SBN alarm log through the API Engine. Live analytics starts when a viewer turns analytics on and ends with the detection overlay on that viewer's video. Each arrow points from the side that starts the exchange.
Alarm clip classification
- Alarm receipt. Every alarm that sbn-media queues produces a receipt on the alarm receipt stream. Alarms arrive from cameras that push events, and from SBN live-traffic rows of bound cameras, which the live-traffic receiver files for the tenants in
linereceiver.liveTraffic.tenants. - Clip acquisition. The clip acquirer reads the receipts of each tenant in
clipacquirer.tenants. It skips system, caller ID and operator receipts, scene check receipts and the classifier's own signals. For a qualifying alarm it requests the recorded clip around the alarm time fromdevicestreamer, which fetches it in the way that suits the camera type. Media the camera pushed with the alarm is used as well, up to four items. The clip acquirer then publishes one media-ready message per alarm. The message is deduplicated by the alarm's event id and kept for 24 hours. - Gating. With
imageclassifier.mediaReady.enabledset, the classifier runs one durable consumer per tenant inimageclassifier.mediaReady.tenantsand takes one message at a time from each. It classifies only when the camera's alarm binding has analytics turned on and the camera has a classifier group. A camera without a classifier group is skipped with a warning; the default model is never used for an alarm. - Classification of each media item. The classifier retrieves each item from the file server; images are capped at 10 MB and video at 100 MB. An item already classified with the same group and model reuses the stored result without inference. A video is decoded by the GStreamer sidecar at one frame per second, up to 30 frames; a still image is one frame. The frames pass through optional contrast enhancement (CLAHE), the model and a tracker. The camera's detection areas are applied, and classification stops early when the group's stop class is found.
- Verdict. The classifier combines the items into one outcome: objects, empty, inconclusive or failed. It picks the best frame by class priority and confidence, stores the snapshot and object crops on the file server, and publishes the verdict on the detection events stream, where the recorder indexes it against the alarm. A message is delivered up to four times; the last delivery publishes a failed outcome when classification did not finish.
- Alarm signal. With
imageclassifier.alarmSignals.enabledset, the classifier injects one signal per final classification into the signal queue.sbnfrontenddelivers it to the SBN alarm log on the account and zone bound to the camera. Codes and latching: Image Classifier.
Live-stream analytics
- Start. When a viewer turns analytics on, the WebRTC peer of that view asks the classifier to attach to the camera, and asks it to detach when the viewer turns analytics off or leaves. Live video of a camera is classified once for all its viewers and stops when the last one detaches. With
imageclassifier.alwaysOn.enabled, the classifier also keeps workers running without viewers for cameras with history enabled, up toimageclassifier.alwaysOn.maxWorkers(default 2). Pinned workers count within the instance's capacity. - Capacity. One instance runs up to
imageclassifier.maxConcurrentstreams (default 4). At capacity it stops accepting new streams and another instance takes them. - Frame stages. Every frame is decoded from H.264. The interval gate passes every Nth frame. The frame is cropped to the camera's include areas, when it has any. The motion gate, when the camera's profile turns it on, skips a frame without change inside the crop. The shared inference engine resizes the frame to the model input, runs the model and suppresses overlapping boxes. The detections are mapped back to full-frame coordinates and filtered by the detection areas, and the tracker gives each object a stable id and a direction of movement.
- Viewer overlay. Each classified frame publishes its detections. The WebRTC peer sends them to the browser on the analytics data channel, where Media Console draws the boxes.
- Detection events. With
imageclassifier.detectionEvents.enabledset, the classifier publishes one event per track the first time it passes the group's static filter. It uploads the track's best crop once to the file server. The recorder indexes the event in the camera's event log. Only live classifier tracks produce these events; alarm, APSL and playback results never do.
Shared inference engine
- One engine per model. A host keeps one inference engine per model file. Alarm clips, on-request classification and every live stream that use the same model share it, so memory stays flat as streams are added.
- Scheduling. The engine runs inference on a single scheduling loop, because an ONNX Runtime session is not safe for concurrent use. Each cycle it first serves waiting clip and on-request frames, then the live streams in priority order. Clips that use one model on one host are therefore classified one at a time.
- Overload. The engine targets a 250 ms scheduling cycle. When a cycle takes 2, 4 or 8 times that long, the engine enters the elevated, high or critical tier. Elevated is reported in the pool status only. High classifies every second live stream per cycle, alternating between them. Critical classifies at most three live streams per cycle. The engine steps back down after the load has stayed lower for 5 seconds and 10 cycles.
- Models and runtimes. The parsers cover YOLOv8, YOLO11, YOLOX, RF-DETR and D-FINE. Inference runs on ONNX Runtime with the execution provider set in
imageclassifier.execution.provider: CPU by default, or CUDA, CoreML, TensorRT, OpenVINO or DirectML.imageclassifier.sessionTuning.intraOpThreadssets the threads per inference; the default of 0 keeps the ONNX Runtime default, which uses every physical core.
On-request classification
POST /api/classify classifies one JPEG image or MP4 clip and answers within 30 seconds. APSL automation uses the same request. Instances share these requests through a queue group. A request without a classifier group uses the default model. A request that names an alarm event also sends the classification alarm signal.
Sizing
| Workload | Measured | Figure |
|---|---|---|
| Alarm clip | D-FINE-S, CPU only, 2 vCPU (one physical core) | 12.3 s per clip on average, 0.46 s per frame, at most about 300 clips per hour |
| Alarm clip | Alarm to published verdict | 37 s on average, of which classification takes 12 s |
| Live stream | D-FINE-S, 8-core laptop CPU, 1 stream | 147 ms per frame, about 550 MB resident |
| Live stream | YOLOv8n, same host, 4 streams on one engine | 167 ms per frame, about 380 MB resident |
For up to about 2,000 alarm clips per day, the recommended host has 4 vCPU that are not burstable, 16 GB of memory and an SSD sized for the clip retention period. No GPU is needed at that volume. Size a host from its busiest hour, because clips queue behind each other on one engine.