eve-troubleshooting
Troubleshoot common Eve deploy and job failures using CLI-first diagnostics.
Savant verdict: Review advised
Security analysis or source-hub review identified patterns requiring attention.
Live evaluation
Not evaluated yet. Workspaces can request a live evaluation.
Security assessment
Scan pending.
Structure
- 1 referenced file isn't in the packagereferences/manifest.md. They may be binary, too large to catalog, or missing upstream.
- No license declaredConfirm you may reuse this skill before importing it into your repository.
- The source hub's security checks flagged this skillRunlayer: warn (1/1 file flagged)
SKILL.md
---
name: eve-troubleshooting
description: Troubleshoot common Eve deploy and job failures using CLI-first diagnostics.
---
# Eve Troubleshooting
Use CLI-first diagnostics. Do not assume cluster access.
## Quick Triage Checklist
```bash
eve system health
eve auth status
eve job list --phase active
```
## Common Issues and Fixes
### Auth Fails or "Not authenticated"
```bash
eve auth logout
eve auth login
eve auth status
```
If SSH key is missing, register it with the admin or follow the CLI prompt to fetch from GitHub.
### Secret Missing / Interpolation Error
```bash
eve secrets list --project proj_xxx
eve secrets set MISSING_KEY "value" --project proj_xxx
```
Verify `.eve/dev-secrets.yaml` exists for local interpolation.
### Deploy Job Failed
```bash
eve job follow <job-id>
eve job diagnose <job-id>
eve job result <job-id>
eve env diagnose <project> <env> # structured DeployFailure + cluster snapshot
```
Check for registry auth errors, missing secrets, or healthcheck failures.
`eve env diagnose` surfaces a typed `last_deploy_failure` (kind, service, pod, namespace, message) plus the live K8s state. Read the failure `kind` first — the CLI no longer collapses real causes into a bare `HTTP request failed`. Compare the diagnose `manifest_hash` against the latest sync to spot applied-release drift.
### Sentinel Slack Alerts
Slack pings from the platform sentinel point at a degraded env. Don't act on the alert text alone — pull the project/env out of it and re-confirm with `eve env diagnose <project> <env>`. If the env reads healthy, the alert was a transient that has self-healed; if not, follow the standard deploy-failure triage above.
### Missing Magic-Link or Invite Email
Pre-flight checks usually catch SES misconfig, but if a recipient swears the mail never landed:
```bash
eve admin email bounces list --recipient user@example.com --limit 20
eve admin email bounces list --event-type Bounce
eve admin email bounces list --event-type Complaint
```
The `email_delivery_events` table is the source of truth — `Bounce` / `Complaint` / `Reject` rows explain silent SES drops. No row at all means the send never happened; re-check `x-eve.branding` reply-to/support emails and the project's `x-eve.auth` policy.
### Magic-Link "Already Used" on First Click
Mail-security scanners (Defender SafeLinks, Mimecast, Proofpoint, Barracuda, IronPort) prefetch URLs and burn single-use OTPs. The platform now wraps every action link behind an SSO interstitial that requires an explicit click, so prefetches no longer consume the token. If a user still hits "already used":
- The wrap has expired (1h TTL) — have them request a fresh magic link.
- They genuinely clicked twice — the second click hits a consumed wrap.
- Telemetry: rows with `get_count > 1` in `magic_link_wraps` are expected for protected mailboxes; only `consumed_at` matters.
### Magic-Link or Invite Lands on the Generic SSO Page
The redirect allowlist rejected the app's custom-domain origin, so SSO fell back to the cluster landing page. Confirm the project's `x-eve.auth.allowed_redirect_origins` (or its registered `custom_domains` rows) include the app origin — origin only, `scheme://host[:port]`, https unless local. Then re-sync:
```bash
eve project sync
eve project auth-context <project_id> # shows the resolved allowlist
```
Paths, queries, fragments, and userinfo are rejected at sync time, so a stray `https://app.example.com/callback` in the manifest will surface there.
### `/session` 401 on a Custom-Domain App
Browsers strip `SameSite=Lax` cookies on cross-site requests, so a custom-domain app hitting `sso.<cluster>` will see no `eve_sso_rt` and a 401. The SSO broker now sets `SameSite=None; Secure` when `EVE_SSO_SECURE_COOKIES=true`. Verify in the broker logs:
```
[eve-sso] Secure cookies: true (SameSite=none)
```
In the browser devtools, the `eve_sso_rt` cookie on `.<EVE_DEFAULT_DOMAIN>` must show `SameSite=None; Secure; HttpOnly`. If it shows `Lax`, the broker is running in local-http mode — set `EVE_SSO_SECURE_COOKIES=true` and redeploy. Local k3d on `*.lvh.me` keeps `Lax` deliberately (same parent site).
### Domain-Signup Manifest Rejected After Upgrade
The v1 shape (`domains: [string]` plus a block-level `target_org`) is no longer accepted as of 2026-05-12 — manifest sync rejects on first sight. Migrate to v2:
```yaml
x-eve:
auth:
org_access:
domain_signup:
enabled: true
domains:
- { domain: acme.com, target_org: org_acme, role: member }
- { domain: partner.example, target_org: org_partner }
```
Each rule must declare its own `target_org`, and each `target_org` must already appear in the project's `allowed_orgs`. Free-email domains (`free-mail.example`, `outlook.com`, ...) emit a coherence warning, not a reject — declaring them is almost always wrong. See `references/manifest.md` and `docs/system/auth.md#domain-based-signup` for the full v2 shape.
### Workflow Step Token "Resource Denied"
Workflow step jobs now carry a `token_scope` claim derived from workflow- and step-level `scope` blocks (intersected with any API-supplied scope). If a step gets `denied` on an org filesystem path, env DB table, or Cloud FS mount, the scope is the suspect:
```bash
eve job show <job-id> --json | jq '.token_scope'
```
Compare the claim against the manifest's workflow/step `scope`. Step scope intersects workflow scope — it can only narrow, never widen. If the step needs a path the workflow forbids, broaden the workflow `scope` (or drop it). Request-supplied `scope` requires the `jobs:harness_override` permission; there is no `--scope-*` CLI flag yet.
### Custom Domain Issues
```bash
eve domain verify <hostname> # DNS check + cert state + next steps
eve domain status <hostname> # which env owns it
eve domain list --env <env> # everything bound to this env
```
Common causes:
- **DNS not resolved**: `verify` will print the expected target and `dns_result.verified: false` — fix the CNAME/A record before re-running deploy.
- **Cert pending**: cert-manager HTTP-01 challenge in flight; re-run `verify` after a minute.
- **First-bind-wins conflict**: another env already claimed the hostname. Use `eve domain transfer <host> --to <env>` and redeploy, or scope the domain per-env via `environments.<env>.overrides`.
### Registry Push Fails with UNAUTHORIZED
If build jobs fail with `UNAUTHORIZED: authentication required` when pushing:
1. Verify secrets are set: `eve secrets list --project proj_xxx`
2. If using a custom BYO registry, verify credentials map to `registry.host`
3. Confirm the imagePull metadata in your manifest is correct
4. Add OCI source label to Dockerfile: `LABEL org.opencontainers.image.source="https://github.com/ORG/REPO"`
Some registries require repository-linked package metadata or workspace-level auth alignment.
### Build Failures
#### Symptoms
- Pipeline fails at build step
- `eve build diagnose` shows run status = `failed`
#### Triage
```bash
eve build list --project <id> # Find recent builds
eve build diagnose <build_id> # Full state dump
eve build logs <build_id> # Raw build output
```
#### Common Causes
**Registry authentication:**
- If using custom registry mode, verify `REGISTRY_USERNAME` and `REGISTRY_PASSWORD` secrets are set (or provider-equivalent registry credentials). With managed registry (`registry: "eve"`), this step is usually not required.
- Ensure credentials can access the configured registry account and namespace
- Check: `eve secrets list --project <id>`
**Dockerfile issues:**
- Service must have `build.context` in manifest pointing to directory with Dockerfile
- Dockerfile path defaults to `<context>/Dockerfile`
- Multi-stage builds work with BuildKit; may fail with Kaniko
**Workspace/clone errors:**
- Build requires workspace at the correct git SHA
- Check `eve build diagnose` for workspace preparation errors
**Image push failures:**
- OCI labels help link packages to repos: add `LABEL org.opencontainers.image.source="https://github.com/OWNER/REPO"` to Dockerfile
- Ensure registry host and auth match manifest `registry.host` when using BYO/custom registry
### Job Stuck or Blocked
```bash
eve job show <job-id>
eve job dep list <job-id>
```
Resolve dependencies or update phase with `eve job update` if appropriate.
Stale-attempt recovery now covers all assignees and graceful agent-runtime shutdown, and agent-runtime pod death no longer strands attempts. If a job still looks stuck after a recent restart, give the recovery loop one cycle before manually intervening — it usually reclaims on its own.
### Chat Route Doesn't Match
Chat route matching is now case-insensitive on the regex side. If a route still fails to fire, the pattern itself is the suspect — not the casing. Re-check the `match` regex in your chat config, then resync the manifest.
### Agent-Runtime Org Discovery
Agent-runtime auto-discovers orgs at startup; you no longer need an `org_default` config entry. If a fresh deploy can't see an org's jobs, check that the org actually has at least one project + agent registered, then check agent-runtime logs — don't go looking for a missing `org_default`.
### App Not Reachable After Deploy
- Confirm deploy job succeeded (`eve job result`).
- Validate ingress host pattern: `{service}.{orgSlug}-{projectSlug}-{env}.{domain}`.
- Ensure service port matches `x-eve.ingress.port`.
## Escalation
If CLI output is insufficient, collect:
- `eve system health`
- `eve job diagnose <job-id>`
- manifest diff (recent changes)
Then hand off to the platform operator.