Handbook
Operational runbook
Each section lists symptoms, likely causes, commands, remediation, and when to escalate. Align taxonomy with errors.md and tutorial 301 production readiness.
Updated
Transport timeouts / flaky gateways
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
Repeated TransportFailure, slow Ok |
Gateway saturation, aggressive timeouts | reproduce with pytest -q -m "not granite" on pinned SHA; raise timeouts in LlmEnvProfile temporarily |
Enable backoff in consumer; reduce parallelism | SLA breached across regions |
| Connect resets mid-stream | DNS/TLS intermediaries | curl -v to gateway base URL (no secrets in logs) |
Failover profile / alternate egress | Vendor outage |
401 / 403 gateway auth
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
TransportFailure with HTTP 401/403 |
Rotated keys, wrong profile path | verify read_certificator_profile / env mirrors CI secrets layout |
Rotate credential via orchestrator; re-pin env files locally | Keys leaked or unexplained forbidden |
HTTP 200 with JSON error object
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
GatewayFailure despite HTTP success |
Vendor wraps provider errors | Inspect Err payload fields logged redacted |
Map gateway-specific retry vs fatal codes | Vendor regression persists |
Empty assistant content
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
ParseFailure or malformed choices |
Model censorship / truncation quirks | Capture fake_chat minimal replay offline |
Adjust ChatParameters / model route docs MODEL-ROUTING |
Cross-model recurrence |
Non-JSON assistant response
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
ParseFailure |
Prompt drift / refusal prose | Tighten contract.md examples + deterministic fake_chat |
Use repair loops orchestration hooks | Needs prompt/product redesign |
Schema failure after HTTP OK JSON
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
SchemaFailure |
Sidecar drift vs runner | python3 scripts/audit_contract_docs.py --strict --strict-md |
bump contract.json + targeted pytest |
Silent partial adoption |
Model ignores enum / routing labels
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
Wrong routes branch (pw_*) |
Weak discriminators | add deterministic probes tests fixtures | enrich interaction_probe / cues consumer-side |
Requires upstream UX redesign |
Contract sidecar mismatch
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
audit_contract_docs.py fails in CI |
manual edits skipped regeneration | run audit_contract_docs.py --require-contract-identity |
sync contract_markdown_path, task_id, version |
cross-repo fork divergence |
Website handbook stale vs library mainline
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
| Public handbook misses docs merged locally | Submodule bump forgotten | git -C forge-lcdl-website/forge-lcdl rev-parse HEAD vs forge-lcdl main |
bump submodule + rebuild WEBSITE-PUBLISHING | Marketing SLA misses GA |
Firebase deploy failure
| Symptoms | Likely causes | Commands | Remediation | Escalate when |
|---|---|---|---|---|
firebase deploy non-zero |
token expiry bad firebase.json |
bash scripts/verify-hosting-target.sh |
rotate FIREBASE_TOKEN secret confirm lcdl-542d8 project |
Hosting outage |
Periodic baseline
git pull
pip install -e ".[dev]"
python3 scripts/audit_contract_docs.py --strict --strict-md --require-contract-identity
python3 scripts/audit_docs_site.py --strict
pytest -q -m "not granite"