Technical article

Runtime diagnostics: remote troubleshooting that never touches the business

LightESB-Camel exposes its live runtime state through one read-only management API and a matching set of CLI commands. The diagnostics snapshot never reloads routes, cleans data, closes connections, or changes log levels — it collects evidence and nothing else.

Docs: docs/runtime-diagnostics-api.mdRepository: github.com/nghxni/lightesb-camel

Evidence collection without side effects

When a production ESB route misbehaves, the classic playbook — SSH in, grep logs, restart something — is slow and risky: every evidence-gathering step can itself change the system under observation. The diagnostics API takes the opposite posture — GET /api/diagnostics/runtime-snapshot is a formal remote management API that reports what the runtime already knows, and the CLI mirrors it one-to-one so Codex and automation collect the same evidence with --output json.

One snapshot, seven components

The snapshot accepts serviceName, serviceVersion, and component filters. Each component reports a status, a summary, and warnings — all computed on the server; the CLI only displays them, never infers locally. When the server starts with lightesb.route.enabled=false, the route-runtime component is not registered and filtering by it returns an empty component list.

route-runtime

Internal and public service-version status, route files, routeId, CamelContext status, failure stage, diagnostic id, sanitized error summary, and dynamic watcher state — the first stop when a route is not running.

service-log and instance-log

Logger/config/path consistency, plus writer storage mode, H2 fallback state, query stores, queue and batch state, rejected tasks, and the last flush and errors of the instance log pipeline.

ai-route-cache and ai-model-session

AI route cache entry counts, TTL, hit and cleanup statistics, and local model session counts, TTL, and limits — AI route issues are observable without reading prompts or payloads.

external-datasource and robot-command

DataSource cache size, beanName, type, and signature hash; plus the robot command ledger, audit, outbox, dispatcher, state snapshots, compensation, denylist, recent error-code distribution, and a read-only doctor.

Real commands, real output

Three commands cover most remote evidence collection. A profile carries the server address and token, so the CLI sends Authorization: Bearer automatically:

lightesb diagnostics snapshot --server http://localhost:8080 --output json
lightesb diagnostics warnings --server http://localhost:8080 --output json
lightesb diagnostics snapshot --server http://localhost:8080 --component route-runtime --output json

The response uses the standard management API envelope. A failed route service shows up with its failure stage and a sanitized error summary — enough to act on without seeing any configuration values:

{
  "success": true,
  "data": {
    "runtime": { "generatedAt": "2026-06-30T08:00:00Z", "uptimeMs": 1000, "javaVersion": "21" },
    "filters": { "serviceName": "DemoSrv", "serviceVersion": "v1.0.0", "component": "route-runtime" },
    "components": [
      {
        "component": "route-runtime",
        "status": "WARN",
        "summary": {
          "totalServices": 1,
          "failedServices": 1,
          "files": [
            { "fileKey": "DemoSrv@v1.0.0@route.xml", "internalStatus": "FAILED", "serviceStatus": "STOPPED",
              "contextStatus": "NOT_CREATED", "failureStage": "CONFIGURATION", "errorSummary": "missing required configuration" }
          ]
        },
        "warnings": [ "one or more route services are in FAILED state" ]
      }
    ]
  }
}

Start from the symptom, end with a controlled recovery

The support runbook turns symptoms into short read-only command paths before anyone touches a route file:

Route not running: route status, route detail, then diagnostics snapshot --component route-runtime for fileKey, routeId, CamelContext status, and warnings.
Missing configuration: route config and route detail return only the missing key name — never the values.
No instance logs: log health plus diagnostics snapshot --component instance-log shows writer and query storage state and the H2 fallback flags.
CLI cannot connect or 401/403: profile current and doctor report the server, exit code, and requestId — profile JSON only reports tokenConfigured flags, never token values.

Recovery is equally narrow: reload the affected route file or service config, then re-check diagnostics warnings. Restart the backend only when Java code, dependencies, Spring beans, global configuration, or startup parameters changed — or when hot reload fails and state is inconsistent.

Sanitized by design, not by discipline

Diagnostics output contains summaries only: no passwords, tokens, full prompts, payloads, XML, properties, connection strings, or business message bodies. Error summaries from route, instance-log, and robot-command are always redacted — regardless of the service-level log.redaction.enabled switch or caller permissions. This API is not a privileged log-reading channel.

It is still a formal management API: put it behind your deployment's authentication, gateway audit, and access-logging policies. On the wire, 401/403 means authorization refused, 5xx means a diagnostics component failed server-side, and the CLI maps failures to exit code 69 (HTTP or success:false) and 78 (no --server or profile configured). Redact secrets and customer payloads before sharing any collected evidence.

Diagnostics API guide Support diagnostics runbook