Handbook

LLM priority registry

Use forge_lcdl.llm_priority when a product needs several OpenAI-compatible endpoints (primary gateway, backup host, local Granite) with automatic transport failover. The registry picks endpoints; ModelRouter still picks…

Updated

Discovery

Source Precedence
Explicit path= to load_registry() Highest when passed
FORGE_LCDL_LLM_PRIORITY_FILE Env override
<cwd>/config/llm-priority.json Repo-local default
LLM_* environment only Compat: single entry from read_certificator_profile()

When no JSON file exists and profiles is empty, behavior matches today's single-profile path (one env-derived entry, no extra HTTP attempts).

JSON schema

{
  "profiles": [
    {
      "id": "granite-primary",
      "priority": 0,
      "base_url": "https://gateway.example/v1",
      "api_key": "optional",
      "model": "ctx-unlim-qwen3-8b:latest",
      "timeout_sec": 120,
      "json_mode": true,
      "ngrok_bypass": false
    }
  ]
}
  • priority: ascending order (0 first). Duplicate model ids on multiple entries: first by priority wins for resolve_profile_for_model.
  • id: stable label for logs and aggregated errors (required, unique by convention).
  • Malformed JSON fails loud; it does not silently pick a wrong endpoint.

Failover semantics

chat_with_failover(messages, registry, policy_fn) calls policy_fn(profile) per entry in priority order.

Failure class Failover?
Timeout, connection error, HTTP 5xx (incl. 502/503/504/524) Yes — try next profile
JSON parse, missing content, HTTP 4xx, model-content errors No — inner policy owns retries (chat_with_json_mode_then_plain)

Per-endpoint urllib retries still apply via FORGE_LCDL_CHAT_HTTP_RETRIES inside chat_completion_sync.

Aggregated errors list profile id, host, and cause — never api_key.

Chat policy integration

Optional registry= on chat_once and chat_with_json_mode_then_plain delegates to chat_with_failover. Default registry=None preserves existing call sites.

from forge_lcdl.llm_priority import load_registry
from forge_lcdl.generic.chat_policy import chat_with_json_mode_then_plain

registry = load_registry()
result = chat_with_json_mode_then_plain(
    messages,
    profile=registry.entries[0].profile,
    temperature=0.2,
    registry=registry,
)

Model router mapping

ModelRouter.choose returns RoutingDecision with primary_model / fallback_model ids. Map them to endpoints:

from forge_lcdl.llm_priority import apply_routing_decision, load_registry

registry = load_registry()
decision = ModelRouter.default().choose("my_task", TaskComplexity.medium)
primary_entry, fallback_entry = apply_routing_decision(registry, decision)

Router picks models; registry picks endpoints.

Lenses concept mapping

Forge Lenses uses .lenses-local/llm-settings.json (no code dependency here):

Lenses LCDL registry
routing_mode: single One registry entry / env fallback
routing_mode: smart Priority list + model-based pick
routing_mode: advanced Explicit per-profile priority + task router model ids
fallback_provider / fallback_model Lower-priority entry or RoutingDecision.fallback_model
pools Multiple entries sharing model id (first priority wins)

Governed runs (lmeta / forgeLcdlRun)

Automation flows that call forgeLcdlRun resolve to FlowExecutor.run_lcdl → LcdlClient.execute. Pass registry at client construction:

from forge_lcdl.execution.client import LcdlClient

client = LcdlClient.from_env()  # loads registry when config file exists
client.execute("task_id", {"key": "value"})

Env/registry plumbing follows the same discovery table as load_registry().

See also