forge-lcdl

Handbook

Verification registry (`forge_lcdl.verification`)

Cheap models can sound confident while wrong. Verification is separate from Ok / Err task results: it produces evidence (schema checks, subprocess exit codes, etc.) so callers can require machine-checkable proof before…

Verification registry (forge_lcdl.verification)

Purpose

Cheap models can sound confident while wrong. Verification is separate from Ok / Err task results: it produces evidence (schema checks, subprocess exit codes, etc.) so callers can require machine-checkable proof before treating a run as “done.”

Core types

Type Meaning
VerificationStatus pass, fail, warn, skipped (enum values are wire strings).
VerificationResult status, verifier_id, summary, evidence (dict), errors / warnings (tuples), optional elapsed_seconds.

Serialize with verification_result_to_dict or verification_results_to_jsonable for JSON.

Verifiers

  • JsonObjectVerifier (verifier_id: json.object): requires input_obj is a dict; if output_obj is not None, it must be a dict.
  • ContractSchemaVerifier (verifier_id: contract.schema): uses validate_contract_input / validate_contract_output from contracts_api.py. Pass context={"contract": contract_spec}. If output_obj is None, output schema is not checked.
  • PytestVerifier (verifier_id: pytest.subprocess when you register it yourself): runs subprocess.run with an explicit argv list you provide (e.g. [sys.executable, "-m", "pytest", …]). No default argv — empty argv yields skipped. Captures stdout/stderr truncated with truncate_utf8_bytes; supports timeout and max_output_bytes per stream.

Security: Never pass user-controlled strings into argv without your own policy. The default registry does not register pytest — only safe json.object + contract.schema.

Registry

  • VerifierRegistry: register(verifier_id, verifier), get(verifier_id), list_ids() (sorted registered ids).
  • default_registry(): pre-registers json.object and contract.schema only.

Local verification

Use python3 (many systems have no python symlink). From the forge-lcdl repo root, set PYTHONPATH=src for ad-hoc scripts unless you use pip install -e ".[dev]" (same pattern as CONTEXT-PACKS.md).

./scripts/verify-verification-sprint.sh

Manual shortcuts:

export PYTHONPATH=src
python3 -m pytest -q tests/test_verification_registry.py
python3 -m compileall -q src/forge_lcdl/verification

Policy: verification_policy["verifier_ids"]

ContractSpec includes verification_policy. Optional verifier_ids: a list of strings run in order, e.g.:

"verification_policy": {
  "level": "task_internal",
  "verifier_ids": ["json.object", "contract.schema"]
}

If verifier_ids is missing or invalid, verify_task_result uses ("json.object", "contract.schema").

Unknown verifier_id values produce a FAIL result for that slot (strict: no silent skip).

Helper

from forge_lcdl.contracts_api import load_contract_spec
from forge_lcdl.verification import verify_task_result, verification_results_to_jsonable

spec = load_contract_spec("llm_boolean_gate", "v1")
results = verify_task_result(spec, input_obj, output_obj)
payload = verification_results_to_jsonable(results)

Benchmarks

benchmarks uses string verification_status. Mapping VerificationResult.status to those strings can be added in consumers when wiring benchmarks to this module.