Catalogskills.sh · nvidia

deepstream-sop

Use this skill when building, deploying, evaluating, debugging, or measuring latency for the DeepStream SOP Inference Microservice — a GPU-accelerated FastAPI service that detects whether operators perform assembly-line steps in order via event boundary detection (GEBD) plus VLM classification. Trigger even if the user does not name it: verify operator step sequence, detect missing or out-of-order SOP steps, score factory/work-cell video for procedure compliance, run VLM-based SOP checking on industrial cameras, or call /v1/chat/completions with a file, RTSP, or Basler camera. Also trigger for its internals: SOPVideoProcessor, DeepStream GEBD model (e.g. DDM) via Triton CAPI, nvds_custom_postprocess, Cosmos Reason 1/2 vLLM, SSE streaming, Kafka NvProto/JSON output, Basler/Pylon camera + emulation, Docker compose, chunk-level latency. Do NOT trigger for generic DeepStream pipelines, object detection/tracking, NIM imports, or video summarization.

Review advisedVerified hub696 installs

Savant verdict: Review advised

Security analysis or source-hub review identified patterns requiring attention.

Live evaluation

Not evaluated yet. Workspaces can request a live evaluation.

Security assessment

Scan pending.

Structure

  • 1 referenced file isn't in the packagereferences/sop_architecture.svg. They may be binary, too large to catalog, or missing upstream.
  • Ships 14 executable scriptsreferences/copy_sources_reference.sh, references/ddm_net_reference.py, references/export_ddm_to_tensorrt_reference.py, references/ext_pb2_reference.py, references/messager_reference.py, …. Review what they do before enabling the skill for agents with tool access.
  • The source hub's security checks flagged this skillGen Agent Trust Hub: warn (This skill provides a framework for building an industrial SOP monitoring microservice. It uses multi-stage processing, including Vision-Language Models and custom logic. Security observations include dynamic code generation based on configuration files, execution of build scripts via subprocess, and reliance on external research repositories for model weights and code.); Socket: warn (2 alerts: gptSecurity, gptAnomaly)

SKILL.md

---
name: "deepstream-sop"
description: >
  Use this skill when building, deploying, evaluating, debugging, or measuring
  latency for the DeepStream SOP Inference Microservice — a GPU-accelerated
  FastAPI service that detects whether operators perform assembly-line steps in
  order via event boundary detection (GEBD) plus VLM classification. Trigger
  even if the user does not name it: verify operator step sequence, detect
  missing or out-of-order SOP steps, score factory/work-cell video for
  procedure compliance, run VLM-based SOP checking on industrial cameras, or
  call /v1/chat/completions with a file, RTSP, or Basler camera. Also trigger
  for its internals: SOPVideoProcessor, DeepStream GEBD model (e.g. DDM) via
  Triton CAPI, nvds_custom_postprocess, Cosmos Reason 1/2 vLLM, SSE streaming,
  Kafka NvProto/JSON output, Basler/Pylon camera + emulation, Docker compose,
  chunk-level latency. Do NOT trigger for generic DeepStream pipelines, object
  detection/tracking, NIM imports, or video summarization.
owner: "windy@nvidia.com"
service: "deepstream-sop"
version: "1.0.0"
license: "CC-BY-4.0 AND Apache-2.0"
reviewed: "2026-04-08"
metadata:
  author: "Wind Yuan <windy@nvidia.com>"
  tags:
    - deepstream
    - sop
    - vlm
    - triton
    - gpu
  languages:
    - python
  frameworks:
    - deepstream
    - triton
    - fastapi
  domain: video-analytics
---
# DeepStream SOP Inference Microservice Skill

This skill guides AI coding assistants in building, extending, and debugging the
**NVIDIA DeepStream SOP (Standard Operating Procedure) Inference Microservice** —
a GPU-accelerated pipeline for temporal action detection and VLM-based SOP compliance
monitoring on industrial video feeds.

**Reference repository**: https://github.com/NVIDIA/sop-monitoring-blueprints/tree/main/microservices/sop-inference-bp
**Local reference code**: `sop-inference-bp/` directory (from a local clone of the repository)

---

## Models

Model-agnostic at both inference stages — swap via env var (and Triton dir for GEBD).

| Stage | Role | Model class | Default | Swap via |
|------|------|-------------|---------|----------|
| Stage 1 (CV) | Per-frame boundary scoring → chunk segmentation | **Generic Event Boundary Detection (GEBD)** | **DDM** ([MCG-NJU/DDM](https://github.com/MCG-NJU/DDM)) via Triton Python backend | Replace `triton_model_repo/<model>/` + `DDM_MODEL_PATH` (§ 5) |
| Stage 3 (VLM) | Per-chunk action classification | Vision-language model via vLLM | **Cosmos Reason 1 7B** (Reason 2 also supported) | Set `VLLM_MODEL_PATH` to a different HF ID or local path |

"GEBD" = swappable Stage-1 slot; "DDM" = the default architecture (terms used interchangeably).

**Chunking is selectable per request** (§ 2): default `ddm-net` uses GEBD; `uniform` produces fixed-length chunks and **bypasses Stage-1 GEBD** (§ 3, § 6). DDM temporal window is configurable via `FRAMES_PER_SIDE` / `SEQUENCE_BATCH` (§ 4, § 5), with optional **TensorRT** (§ 5).

---

## Architecture Overview

Runs in a Docker container (`nvds-action-sop`) alongside a Kafka container. Full diagram: [`references/sop_architecture.svg`](references/sop_architecture.svg).

**Data flow through the 4-stage `SOPVideoProcessor` pipeline (per-request):**

```
Input Sources                    Docker Container: nvds-action-sop
─────────────                    ──────────────────────────────────────────────────
Video Files ──┐                  FastAPI Server (port 8300)
RTSP Streams ─┤── base64/       ├─ /v1/chat/completions → SOPProcessManager
Basler Camera ┘   file/rtsp/       │
                  camera           │ ModelInitializer: VLM first, then DDM dummy pipeline
                                   │ 4 Thread Pools: cv(32), clip(32), vlm(64), vlm_req(64)
                                   │
                                   ▼ SOPVideoProcessor (per-request)
                                   ┌────────────────────────────────────────────────┐
                                   │ Stage 1: DeepStream Pipeline (GPU)             │
                                   │   Source → nvstreammux → tee1                  │
                                   │    ├─[inference] queue1 → nvdspreprocess       │
                                   │    │  → nvinferserver (Triton CAPI + DDM)      │
                                   │    │  → InferOutputTensorParser → score_queue  │
                                   │    ├─[frames]  queue3 → nvvideoconvert         │
                                   │    │  → capsfilter → appsink                   │
                                   │    │  → DecodedFrameRetriever → frame_queue    │
                                   │    └─[RTSP out] queue → convert → H.264 enc    │  (optional, § 18)
                                   │       → rtppay → udpsink → RTSPServer (§ 18)   │  opt-in only
                                   │              │ boundary scores                 │
                                   │              ▼                                 │
                                   │ Stage 2: Clip Post-Process                     │
                                   │   Boundary detection → chunk segmentation      │
                                   │              │ video frames + timestamps        │
                                   │              ▼                                 │
                                   │ Stage 3: VLM Inference                         │
                                   │   Embedded vLLM (Cosmos Reason 1/2)            │
                                   │   Frame sampling at VLM_FPS → classification   │
                                   │              │ action labels                    │
                                   │              ▼                                 │
                                   │ Stage 4: SOP Checker                           │
                                   │   Sequence validation → missing/misordered     │
                                   │              │ chunk results                    │
                                   │              ▼                                 │
                                   │         final_queue                            │
                                   └────────────────────────────────────────────────┘
                                          │
Output                                    ▼
──────                             ┌─────────────────┐
SSE Stream (chat.completion.chunk) │ Kafka Messages   │
Non-streaming (chat.completion)    │ (JSON/Protobuf)  │
Prometheus metrics (/v1/metrics)   └────────┬────────┘
                                            ▼
                                   Docker Container: kafka
                                   (apache/kafka:3.7.0)
```

---

## Section Index

Each section is a standalone file in `references/` — load only what your task needs.

| § | File | Responsibility |
|---|------|---------------|
| 1 | [`skill_01_fastapi_endpoints.md`](references/skill_01_fastapi_endpoints.md) | FastAPI endpoints, server init, Prometheus metrics |
| 2 | [`skill_02_pydantic_schemas.md`](references/skill_02_pydantic_schemas.md) | Request/response Pydantic models (`api_types.py`) |
| 3 | [`skill_03_deepstream_pipeline.md`](references/skill_03_deepstream_pipeline.md) | DeepStream pyservicemaker pipeline, tensor parser, dummy pipeline |
| 4 | [`skill_04_config_templates.md`](references/skill_04_config_templates.md) | nvdspreprocess / nvinferserver config templates + rendering |
| 5 | [`skill_05_triton_ddm_model.md`](references/skill_05_triton_ddm_model.md) | Triton model repo, config.pbtxt, model.py, ddm_net.py |
| 5b | [`skill_05b_custom_postprocess.md`](references/skill_05b_custom_postprocess.md) | C++ postprocess plugin, Makefile, IOptions API |
| 6 | [`skill_06_sop_process_manager.md`](references/skill_06_sop_process_manager.md) | SOPProcessManager, SOPVideoProcessor, VLLMInference, Kafka |
| 6b | [`skill_06b_sop_checker.md`](references/skill_06b_sop_checker.md) | SOP sequence and checker compliance: MissingNumberDetector, SopCheckerCache, SopCheckerRequest/Response |
| 7 | [`skill_07_sse_streaming.md`](references/skill_07_sse_streaming.md) | SSE generator, stream response formatting, dummy test mode |
| 8 | [`skill_08_basler_camera.md`](references/skill_08_basler_camera.md) | Basler camera support, Pylon SDK, emulation, formats |
| 9 | [`skill_09_docker_build_deploy.md`](references/skill_09_docker_build_deploy.md) | Docker build, deploy, .env configuration |
| 10 | [`skill_10_test_suite.md`](references/skill_10_test_suite.md) | Test suite coverage, assertions, running tests |
| 11 | [`skill_11_env_variables.md`](references/skill_11_env_variables.md) | All environment variables reference |
| 12 | [`skill_12_evaluation_workflow.md`](references/skill_12_evaluation_workflow.md) | End-to-end eval workflow: static checks, build, launch, tests, API/camera/Kafka checks, report |
| 13 | [`skill_13_verification_curl.md`](references/skill_13_verification_curl.md) | Verification steps and curl examples |
| 14 | [`skill_14_implementation_checklist.md`](references/skill_14_implementation_checklist.md) | Implementation checklist: file copy list, generated files, Docker prereqs, verification |
| 15 | [`skill_15_latency_measurement.md`](references/skill_15_latency_measurement.md) | TTFC and C2C latency measurement for file input via SSE streaming |
| 16 | [`skill_16_message_schema.md`](references/skill_16_message_schema.md) | Kafka message schema selection (`JSON` default vs `NvProtoSchema`) and extending messages with custom data |
| 17 | [`skill_17_camera_latency_measurement.md`](references/skill_17_camera_latency_measurement.md) | Camera / live-stream chunk_e2e latency measurement using internal pipeline timestamps |
| 18 | [`skill_18_rtsp_streaming_output.md`](references/skill_18_rtsp_streaming_output.md) | **OPT-IN** RTSP streaming output: `tee1`-tap re-stream, `RTSPStreamingServer`, `SW_ENCODER` toggle. Generate **only** when user explicitly requests RTSP |

For end-to-end evaluation, read § 12 first; load build/test/curl/latency/camera/Kafka as needed.

**§ 18 is opt-in** — generate only when the user explicitly requests RTSP output; otherwise skip § 18 and the `RTSP_*` rules below.

---

## Key Files Map

The full source-to-target file mapping lives in
[`skill_14_implementation_checklist.md`](references/skill_14_implementation_checklist.md):

- **Files copied verbatim** from `references/` (non-trivial algorithms — cycle
  detection, qwen_vl_utils preprocessing, DeepStream `IOptions` API, protobuf
  sources) with the rationale per file.
- **Files copied as adaptable templates** (Dockerfile, compose.yaml, Triton
  config and `model.py`, `ddm_net.py`, Pylon emulation config, etc.).
- **Files generated from skill sections** — each annotated with the Critical
  Rules below that the generation must follow exactly.
- Docker build prerequisites and post-build verification checklist.

Config files (`nvds_preprocess_template.txt`, `nvds_inference_template.txt`,
`vlm_prompts.txt`) are used as-is from `configs/`.

When `skill_06b` is loaded, read `configs/actions.json` from the project root and run the
§ 6b-G generation workflow to produce `nvds_action_detector/missing_number_detector.py`.
If `configs/actions.json` is absent or invalid, fall back to copying the reference file.

---

## Critical Rules

> Each rule's full detail lives in the linked `skill_NN_*.md` reference file.

| Tag | Rule summary | Details in |
|-----|---|---|
| `MANAGER_INIT_IN_MAIN` | `SOPProcessManager` init in `main()` before `uvicorn.run()` — not inside `lifespan()` | `skill_01_fastapi_endpoints.md` |
| `NAMED_KWARGS` | `create_video_processor()` uses named kwargs; camera args as separate kwargs | `skill_06_sop_process_manager.md` |
| `LIVE_REQUIRES_STREAM_TRUE` | `stream: true` required for live inputs (RTSP / camera) | `skill_08_basler_camera.md` |
| `VLM_DISABLED_DISABLES_SOP_CHECKER` | `DISABLE_VLM_INFERENCE=true` auto-disables SOP checker at import | `skill_06_sop_process_manager.md` |
| `CHUNK_PARAMS_MAX_LENGTH` | `ChunkParams.max_length_sec` = 10s internal; 60s API default | `skill_06_sop_process_manager.md` |
| `VLM_WARMUP_BEFORE_DDM` | `ModelInitializer`: VLM warmup FIRST, then CV dummy pipeline | `skill_06_sop_process_manager.md` |
| `VLM_WARMUP_3_FRAMES` | VLM warmup needs 3 frames (`torch.zeros`) — Qwen3VL hangs on < 3 | `skill_06_sop_process_manager.md` |
| `THREAD_POOL_SIZES` | 4 thread pools: `cv`(32), `clip`(32), `vlm_inference`(64), `vlm_request`(64) | `skill_06_sop_process_manager.md` |
| `MEDIA_INFO_PYMEDIAINFO` | Media info via `pymediainfo`; live sources set fps=30/duration=inf directly | `skill_06_sop_process_manager.md` |
| `CAMERA_EMULATION_PYLON_CAMEMU` | `PYLON_CAMEMU=1` for camera emulation (serial `0815-0000`) | `skill_08_basler_camera.md` |
| `DEEPSTREAM_LIB_HIDE` | DeepStream lib hide trick: rename lib → lib.tmp during gst-plugin-pylon build | `skill_08_basler_camera.md` |
| `VLM_REAL_GPU_FRAMES` | VLM uses real GPU frames via `DecodedFrameRetriever`; never `torch.zeros` for inference | `skill_06_sop_process_manager.md` |
| `BUFFER_RETRIEVER_STATIC_BASE` | `DecodedFrameRetriever` MUST inherit `BufferRetriever` statically via `super().__init__()`; runtime `__class__.__bases__` mutation hangs `pipeline.attach()` | `skill_06_sop_process_manager.md` |
| `FRAME_RETRIEVER_PRIORITY` | `create_inference_pipeline`: `frame_retriever=` kwarg takes priority over `frame_queue` | `skill_03_deepstream_pipeline.md` |
| `MUX_ORIGINAL_RESOLUTION` | nvstreammux uses original resolution (not 224); pass `mux_width/mux_height` from `get_media_info()` (probe live RTSP for non-camera inputs; camera path unaffected) | `skill_03_deepstream_pipeline.md`, `skill_06_sop_process_manager.md` |
| `FILE_URI_NO_DOUBLE_PREFIX` | `create_inference_pipeline` file source: check `file_path.startswith("file://")` before prepending — API passes `file://` URLs directly | `skill_03_deepstream_pipeline.md` |
| `CLEANUP_ON_DISCONNECT` | Pipeline cleanup on client disconnect via `trigger_stop_processors` in `try/finally` | `skill_07_sse_streaming.md` |
| `UNIFIED_CLIP_POST_PROCESS` | Unified `clip_post_process()` for file + live; `stop()` puts `None` in `_score_queue` | `skill_06_sop_process_manager.md` |
| `ABORT_INFLIGHT_VLM` | Abort in-flight VLM requests on `stop()` via `llm.abort(req_id)` | `skill_06_sop_process_manager.md` |
| `LOGGER_EXPORT_GET_LOGGER` | `ds_logger.py` must export `get_logger` | `skill_06_sop_process_manager.md` |
| `KAFKA_USE_CREATE_PRODUCER` | Kafka: use `create_producer()` from `messager.py`; no `Messager` class | `skill_06_sop_process_manager.md` |
| `USER_PROMPT_PRIORITY` | User request text takes priority over `VLM_PROMPT_PATH` file; `{"type":"text"}` in the request overrides the config-file prompt | `skill_06_sop_process_manager.md` |
| `EVAL_USE_CONFIG_PROMPT` | Eval/latency requests omit request text by default so the VLM uses `VLM_PROMPT_PATH` | `skill_12_evaluation_workflow.md`, `skill_13_verification_curl.md`, `skill_15_latency_measurement.md`, `skill_17_camera_latency_measurement.md` |
| `CHUNK_SCHEMA_FIELD_NAMES` | Chunk schema: `chunk_idx`, `cv_boundary_score`, `checker_result`; summary `chunk_idx=-1` | `skill_06_sop_process_manager.md` |
| `SEQUENTIAL_FRAME_DRAIN` | Drain `decoded_frame_queue` (FIFO, shared across chunks) in a SINGLE thread and submit VLM per chunk incrementally; parallel drain steals frames → 0-frame chunks / wrong VLM input | `skill_06_sop_process_manager.md` |
| `WALL_CLOCK_BEFORE_GPU` | `DecodedFrameRetriever.consume()`: capture `wall_clock_entry = time.time()` BEFORE GPU dlpack; queue 3-tuple `(timestamp, wall_clock_entry, tensor)` | `skill_06_sop_process_manager.md`, `skill_17_camera_latency_measurement.md` |
| `CHUNK_E2E_PIPELINE_TIMESTAMPS` | Write `pipeline_chunk_end_timestamp` (last frame wall_clock) and `pipeline_vlm_ready_timestamp` (`tm_e2e.now()`) into `chunk_info` for camera latency (§ 17) | `skill_06_sop_process_manager.md`, `skill_17_camera_latency_measurement.md` |
| `VLM_INFERENCE_REQUIRED_KWARGS` | Every `VLLMInference.inference()` call must pass `video_fps`, `system_prompt`, `max_completion_tokens` | `skill_06_sop_process_manager.md` |
| `UNIFORM_CHUNKING_BYPASSES_DDM` | `chunking_options.algorithm="uniform"` → fixed-length chunks; `create_inference_pipeline(uniform_chunk=True)` skips DDM but keeps `tee1` fanout; Stage 2 uses `uniform_clip_post_process` | `skill_02_pydantic_schemas.md`, `skill_03_deepstream_pipeline.md`, `skill_06_sop_process_manager.md` |
| `DDM_TEMPORAL_CONFIGURABLE` | `SLIDING_WINDOWS_SIZE = 2*FRAMES_PER_SIDE + SEQUENCE_BATCH` rendered into preprocess/nvinferserver (no hard-coded 18); Triton `config.pbtxt` sequence dim `-1` | `skill_04_config_templates.md`, `skill_05_triton_ddm_model.md` |
| `DDM_TRT_OPTIONAL_PATH` | `DDM_TRT_OPTIMIZATION=true` runs DDM via TensorRT (per-thread contexts, fixed batch = SEQUENCE_BATCH); PyTorch fallback; never both. PyTorch is default | `skill_05_triton_ddm_model.md` |
| `DDM_TRT_STREAM_ORDERING` | `DDMTensorRTEngine.infer()`: `wait_stream(current)` → `execute_async_v3` → **`torch.cuda.synchronize(device)`** (NOT per-stream). Per-stream sync leaves TRT aux-stream work in flight → gst-CV SIGSEGV (NVBug 6289256) | `skill_05_triton_ddm_model.md` |
| `METADATA_LICENSE_FROM_FILE` | `/v1/metadata` reads `licenseInfo` from `DS_SOP_LICENSE_PATH` (default `/opt/nvidia/nvds_sop/license.txt`); never hard-code license text | `skill_01_fastapi_endpoints.md` |
| `CAMERA_EMULATION_FRAMES_RGB` | Pylon emulation PNGs must be explicit 3-channel RGB (matches `Emulation_0815-0000.pfs PixelFormat=RGB8Packed`); generate via `nvvideoconvert ! videoconvert ! "video/x-raw,format=RGB" ! pngenc` | `skill_08_basler_camera.md` |
| `COMPOSE_ENV_PASSTHROUGH` | `docker compose` only substitutes `${VAR}` references; every runtime env var must be explicitly listed under `environment:` to reach the container. | `skill_09_docker_build_deploy.md` |

> The four `RTSP_*` rules below apply **only when the optional RTSP streaming-output feature
> (§ 18) is requested**. They do not apply to the default build — skip them if the user did
> not ask for RTSP output.

| `RTSP_OUTPUT_TAPS_TEE1` | RTSP output branch links from the existing `tee1` (added after the main inference link) only when `rtsp_port` is present. | `skill_18_rtsp_streaming_output.md` |
| `RTSP_LEAKY_QUEUE_TINY` | RTSP branch queue must be `leaky=2` + tiny cap (`max-size-buffers=2`) to prevent backpressure and NVMM pool exhaustion. | `skill_18_rtsp_streaming_output.md` |
| `RTSP_KEYINT_MAX_30` | RTSP H.264 encoder must set `key-int-max=30` (and B-frames disabled) to allow downstream seeking. | `skill_18_rtsp_streaming_output.md` |
| `RTSP_ENCODER_FALLBACK` | Select software/hardware H.264 encoder based on `SW_ENCODER` with MJPEG fallback. | `skill_18_rtsp_streaming_output.md` |

---