openclaw-debugging

Savant verdict: Analyzed

Scanned or evaluated by Savant; not yet both passing.

Live evaluation

Not evaluated yet. Workspaces can request a live evaluation.

Safety (NVIDIA SkillSpector)

1 pattern found

• Rogue Agent: systemd installmedium · SKILL.md:101 · Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Structure

• No license declaredConfirm you may reuse this skill before importing it into your repository.

SKILL.md

---
name: openclaw-debugging
description: Debug OpenClaw model, provider, tool-surface, code-mode, streaming, and live/Crabbox behavior by choosing the right logs, probes, and proof path before changing code, including fetching stored sessions, transcripts, and attachments as evidence.
---

# OpenClaw Debugging

Use this skill when OpenClaw behavior differs between local tests, live models,
providers, code mode, Tool Search, Crabbox, or CI, and the next move should be a
debug signal rather than a guess.

## Read First

- `docs/logging.md` for log files, `openclaw logs`, and targeted debug flags.
- `docs/reference/test.md` for local test commands.
- `docs/tools/code-mode.md` for code-mode exec/wait and tool catalog rules.
- Use `$openclaw-testing` for choosing test lanes.
- Use `$crabbox` for broad, Docker, package, Linux, live-key, or CI-parity proof.

## Default Loop

1. State the suspected boundary: config, tool construction, provider payload,
   fetch, stream/SSE, transcript replay, worker/runtime, package/dist, or CI.
2. Add or enable the narrowest signal that proves that boundary.
3. Reproduce with the same provider/model/config. Do not randomly switch models
   unless the model itself is the variable being tested.
4. Compare configured state with actual run activation.
5. Patch the root cause.
6. Rerun the exact failing probe, then broaden only if the contract requires it.

## Model Transport Logs

Use targeted env flags instead of global debug when the model request shape or
stream timing matters:

```bash
OPENCLAW_DEBUG_MODEL_TRANSPORT=1 openclaw gateway
OPENCLAW_DEBUG_MODEL_PAYLOAD=tools OPENCLAW_DEBUG_SSE=events openclaw gateway
OPENCLAW_DEBUG_MODEL_PAYLOAD=full-redacted OPENCLAW_DEBUG_SSE=peek openclaw gateway
```

Useful flags:

- `OPENCLAW_DEBUG_MODEL_TRANSPORT=1`: request start, fetch response, SDK
  headers, first SSE event, stream done, and transport errors at `info`.
- `OPENCLAW_DEBUG_MODEL_PAYLOAD=summary`: bounded payload summary.
- `OPENCLAW_DEBUG_MODEL_PAYLOAD=tools`: all model-facing tool names.
- `OPENCLAW_DEBUG_MODEL_PAYLOAD=full-redacted`: capped, redacted JSON payload.
  Use only while debugging; prompts/message text may still appear.
- `OPENCLAW_DEBUG_SSE=events`: first-event and stream-completion timing.
- `OPENCLAW_DEBUG_SSE=peek`: first five redacted SSE events.
- `OPENCLAW_DEBUG_CODE_MODE=1`: code-mode tool-surface diagnostics.

Watch logs with:

```bash
openclaw logs --follow
```

## Common Boundaries

- **Config vs activation:** config can be enabled while the run disables tools,
  is raw, has an empty allowlist, or lacks model tool support. Check the actual
  visible tools before enforcing provider payload invariants.
- **Tool surface:** inspect final model-visible tool names, not only the tool
  registry or config. Code mode means exactly `exec` and `wait` only after it
  actually activates.
- **Provider payload:** log fields, model id, service tier, reasoning, input
  size, metadata keys, prompt-cache key presence, and tool names before SDK
  call.
- **Fetch vs SSE:** fetch response proves HTTP headers arrived; first SSE event
  proves provider body progress. A gap here is a stream/body/provider issue, not
  tool execution.
- **Worker/dist:** run `pnpm build` when touching workers, dynamic imports,
  package exports, lazy runtime boundaries, or published paths.
- **Live keys:** use the configured secret workflow for missing provider keys
  before saying live proof is blocked. Env checks are presence-only; never print
  secrets.

## Fetching Sessions and Transcripts

Use these paths when a bug report references a chat session and you need the
actual transcript, sender attribution, or attachments as evidence.

CLI first (needs a configured install; safe against a live gateway):

```bash
openclaw sessions list --agent <agentId> --json
openclaw sessions tail
openclaw sessions export-trajectory
```

Docs: `docs/reference/database-schemas.md` for the store layout,
https://docs.openclaw.ai/cli/sessions for the CLI.

Raw store (when the CLI is unavailable, e.g. inspecting a remote host over
SSH, or you need event-level detail):

- Per-agent data plane: `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`.
  Canonical schema: `src/state/openclaw-agent-schema.sql`.
- Hosted/systemd installs keep state under the service user's home (e.g.
  `/home/openclaw/.openclaw/...`), not root's — root may carry a separate
  stray install with different agents. If the expected agent dir is missing,
  locate the real DB: `find / -maxdepth 6 -name openclaw-agent.sqlite`.
- Web chat URLs end in a session-id fragment: `/chat/<agentId>/<slug>-<hex>`.
  Resolve it in `session_nodes`: `session_key LIKE '%<hex>%'` →
  `current_session_id`, `display_name`. Key shape is
  `agent:<agentId>:<surface>:<uuid>`; subagent sessions use surface `subagent`.
- Transcript: `transcript_events` (`session_id`, `seq`, `event_json`).
  `event_json.message` has `role` (`user`/`assistant`/`toolResult`) and
  `content` (string, or parts of type `text`/`toolCall`/`image`).
- Sender provenance: real user messages carry `message.__openclaw`
  (`senderId`, `senderName`, `senderIsOwner`); runtime-synthesized inputs do
  not. Use this to separate operator-authored text from injected prompts.
- Full-text search across transcripts: `session_transcript_fts`.
- Attachments: `media://inbound/<file>` URLs map to
  `<state-dir>/media/inbound/<file>`.
- User-attached images are not `image` content parts. They ride the message
  envelope: `message.__openclaw.media[]` entries with `url`
  (`media://inbound/<file>`), `contentType`, `kind`, `fileName`. A parts-only
  extractor misses every image — grep raw `event_json` for `media://`. On a
  remote host, scp the files locally (one remote path per scp argument) and
  read them there.

Hosts without a `sqlite3` binary still have Node: `node:sqlite` needs no
dependencies.

```bash
node -e 'const {DatabaseSync}=require("node:sqlite");
const db=new DatabaseSync(process.argv[1],{readOnly:true});
console.log(JSON.stringify(db.prepare(
  "SELECT seq,event_json FROM transcript_events WHERE session_id=? ORDER BY seq"
).all(process.argv[2])))' <db-path> <session-id>
```

Always open live stores `readOnly: true`; never write a running gateway's
state (see Validation rules in the root `AGENTS.md`). For realistic-data
work, copy the DB into a dev state dir first.

## Code Pointers

- Model payload + Responses stream:
  `src/agents/openai-transport-stream.ts`
- Guarded fetch/timing:
  `src/agents/provider-transport-fetch.ts`
- OpenAI/Codex provider wrappers:
  `src/llm/providers/stream-wrappers/openai.ts`
- Tool construction, Tool Search, code-mode activation:
  `src/agents/embedded-agent-runner/run/attempt.ts`
- Code-mode runtime and worker:
  `src/agents/code-mode.ts`
  `src/agents/code-mode.worker.ts`
- Tool Search catalog:
  `src/agents/tool-search.ts`

## Proof Choice

- Single helper/payload bug: local targeted Vitest.
- Docs/logging-only: `pnpm check:docs` and `git diff --check`.
- Worker/dist/lazy import/package surface: targeted tests plus `pnpm build`.
- Live provider/model behavior: same provider/model with debug flags and a real
  key if available.
- Docker/package/Linux/CI-parity: current dedicated Linux worker when capable;
  otherwise `$crabbox`.
- CI failure: exact SHA, relevant job only, logs only after failure/completion.

## Output Habit

Report:

- boundary tested
- exact command/env shape, redacted
- observed signal, such as tool names or first SSE event timing
- fix location
- narrow proof and any remaining risk