Handbook

PDF extractor parity (certificator pm-cert)

LCDL tasks and workbench harness for synthesizing extract_questions(pdf_path) Python modules and scoring them against a committed pm-cert-questions-run-v1 oracle bank.

Updated

Tasks

Synthesis loop (LLM — Cursor / Granite / lmeta-full)

Task LLM Purpose
pdf_extractor_synthesize yes Emit extractor_python from pdftotext sample + exemplar rows
pdf_extractor_parity_evaluate no Exec extractor; return score / feedback
pdf_extractor_refine yes Re-synthesize with evaluation/repair hints

Explore loop (deterministic compile — no LLM in lmeta)

Task LLM Purpose
pdf_layout_spec_from_profile no Profile row → catalog LayoutSpec
pdf_extractor_compile no LayoutSpec → blessed extractor_python + YAML
pdf_layout_spec_refine no Parity eval → catalog-safe spec patch → re-compile

Instruction hierarchy for spec refine and Cursor handoff: forge_lcdl.tasks.pdf_explore_instructions (catalog > profile > eval feedback > history).

Generic PDF primitives (format-agnostic)

Task LLM Purpose
pdf_text_extract no pdftotext stdout + optional pypdf page metadata (forge_lcdl.ingest.pdf_text)
pdf_profile_detect optional Deterministic OLH/PSC/TSP anchors; optional LLM fallback
pdf_structure_discover yes Layout anchors, option style, answer-key pattern → structure_notes
pdf_deterministic_parse no Wrap certificator pmp_pdf_ingest parsers when profile known
pdf_question_block_judge yes Edge-case block resolution (multi-answer, footer bleed, option parse)

LayoutSpec catalog

forge_lcdl.ingest.pm_cert_pdf_layout loads pdf_layout_catalog.yaml:

layout_id parser_profile
tsp_inline_answer TSP
psc_split_answer_sheet PSC
olh_zoned_numeric_answer OLH

parse_with_layout and compile_extractor_python dispatch to certificator PARSERS for parity.

Parity library

forge_lcdl.ingest.pm_cert_parity — stem/options/answer fingerprint compare (aligned with certificator bank_diff).

Workbench

See forge-composer-workbench/certificator-question-bank-scripts/docs/pipeline/pdf_llm_extract_parity.md for CLI, profile registry (profiles/pdf-profiles.yaml), and flows:

  • flows/pdf-extractor-parity.lmeta — LLM synthesize optimize
  • flows/pdf-extractor-explore.lmeta — deterministic compile / spec-refine optimize (embedded in forge_lcdl/flows/)

Certificator consumer

forge_lcdl.ingest.explore_runner.run_extractor_explore is the shared entry point for workbench CLIs and forge-certificators Bank Health (runner.type: lcdl_explore, layout_id from registry, artifacts under sources/artifacts/explore-ingest/<job_id>/). UI: layout catalog picker + explore phase rail (layout spec → compile → parity → refine → promote).

Note on lmeta

lmeta orchestrates LCDL tasks; it does not compile to Python. The consumer hook or pdf_extractor_parity_evaluate writes/executes the extractor_python string.

Pipeline order in pdf_lmeta_parity_runner.py:

  1. pdf_text_extract → 2. pdf_deterministic_parse → parity evaluate (fast path)
  2. optional pdf_question_block_judge on skipped blocks
  3. optional pdf_structure_discover + forgeOptimizeLoop (synthesize / refine)

Explore runner (pdf_lmeta_explore_runner.py — thin wrapper → run_extractor_explore):

  1. pdf_layout_spec_from_profile → 2. pdf_extractor_compile → parity evaluate
  2. pdf_layout_spec_refine on miss (bounded optimize loop)