Configuring Evaluator Gates
edikt enforces completion claims through three layered gates: the evaluator agent (LLM-mediated, judgment-heavy), the verify runner (mechanical, exit-code-driven), and the stop-hook detector (runtime safety net). This page covers how they compose, what each one decides, and how to tune them via .edikt/config.yaml.
The short version:
| Gate | What it judges | How |
|---|---|---|
| Evaluator | "Is this acceptance criterion satisfied?" | LLM verdict (PASS / FAIL / BLOCKED) per criterion |
| Verify runner | "Does the declared shell check actually pass?" | bash -c "<command>", exit 0 = pass |
| Stop-hook detector | "Did the agent claim done without evidence?" | Regex on completion phrases + plan-state lookup, soft systemMessage |
The verify runner is the hardest gate — exit code 1 blocks state transitions. The evaluator is the most nuanced — it can return BLOCKED to surface ambiguity rather than guess. The stop-hook is purely advisory — it never blocks, only warns.
The three gates in detail
Evaluator agent
Dispatched by /edikt:sdlc:plan at two points in a phase's lifecycle:
- Pre-flight — before the phase starts. Validates that every acceptance criterion is testable, not vague ("works well", "feels right"). Testable means the criterion states what would prove it wrong and declares how it is checked: by a command (mechanical), by someone watching a procedure (observed), or by a named person reading named evidence (judged). It does not mean "has a shell command". Returns BLOCKED when criteria need rewriting; the plan command refuses to start the phase until they're addressed.
- Phase-end — after the generator agent finishes. Reads the evidence (test outputs, file content, git diff) and emits a PASS / FAIL / BLOCKED verdict per criterion.
The evaluator is configured under evaluator: in .edikt/config.yaml:
evaluator:
preflight: true # gate the phase start on criteria quality
phase_end: true # gate phase completion on per-criterion verdicts
mode: headless # "headless" (CLI, no Task tool) | "subagent" (Task fork)
model: claude-opus-4-7 # which model to usemode: headless is the default for CI-friendly determinism. subagent enables interactive subagent dispatch — useful when you want the evaluator to surface context from sibling files.
Verify runner
Configured per-sidecar via the optional verify: field. Each verify: is a single shell command run by bin/edikt verify:
# In a gov sidecar (ADR, invariant, guideline)
directives:
- text: "..."
source_excerpt: {...}
verify: "! rg -P 'echo.*\\{.*\\}' templates/hooks/*.sh"
# In a PRD or SPEC sidecar
acceptance_criteria:
- id: AC-001-1
given: "..."
when: "..."
then: "..."
verify: "bash test/integration/renewal_reminder.sh"See Falsifiable Verification for the full verify-runner contract — execution model, timeout, exit codes, and verify_kind.
Wired into every claim-bearing path:
bin/edikt gov compilerunsverify allafter Phase B./edikt:adr:new,/edikt:invariant:new,/edikt:guideline:newrunverify gov <ID>post-write./edikt:sdlc:prd shipandsupersederunverify prd <PRD-ID>first; refuse on failure./edikt:sdlc:driftinvokesverify spec <SPEC-ID>and folds failures into the drift report./edikt:doctorreports coverage as a soft signal.
There is no config switch for the verify runner — it's always on. The opt-out is per-call (gov compile --skip-verify, prd supersede --force-verify) and each opt-out is loud (recorded in revision_history for supersede; mentioned in compile output).
Stop-hook completion-claim detector
Runtime safety net in templates/hooks/stop-hook.sh. Detects completion phrases (✓, Done!, All tests pass, Phase N complete, …) during in-progress plan phases and emits a non-blocking systemMessage suggesting bin/edikt verify <plan-id> --phase <N>.
Activation predicate (all three must hold):
features.quality-gates: truein.edikt/config.yaml(the default).- The last assistant message matches a completion phrase.
- At least one plan under
paths.planshas a row withstatus: in-progress.
The detector is advisory — systemMessage only, never decision: block. The user is the final judge. Setting features.quality-gates: false opts out entirely.
See Gates → Completion-claim detector for the full predicate.
How the gates compose
The three gates fire at different points in the lifecycle:
Plan phase starts Phase ends
───────────────── ──────────
│ │
├── pre-flight evaluator ├── phase-end evaluator
│ (BLOCKED → fix criteria) │ (FAIL → blocks done flip)
│ │
└─→ generator agent runs └─→ verify runner (--phase N)
(exit 1 → blocks done flip)
Any moment during a turn:
─────────────────────────
stop-hook detector ── advisory only ─→ systemMessage
Other completion paths:
───────────────────────
gov compile ───→ verify all (post-merge)
adr/inv/guideline:new ───→ verify gov <ID> (post-write)
prd ship ───→ verify prd <PRD-ID> (pre-transition)
drift ───→ verify spec <SPEC-ID> (folded into report)The evaluator and the verify runner are independent gates — both must pass for a plan phase to flip done. A phase with a hand-written verify command that exits 0 still needs the evaluator to confirm the AC is satisfied; a phase the evaluator passes still needs verify to confirm the mechanical check holds. This is intentional: the LLM's judgment and the shell's exit code catch different classes of regression.
Tuning
Disable the stop-hook detector
features:
quality-gates: falseThis also disables the agent-driven specialist gates (security, dba, etc.) — see features.quality-gates. If you only want the completion-claim detector off, file an issue; we'll add a separate switch.
Disable evaluator pre-flight
evaluator:
preflight: falseUseful when prototyping a plan structure and you don't want the AC-quality gate firing on every iteration. Phase-end stays on by default.
Bypass a verify gate (deliberate)
bin/edikt gov compile --skip-verify— explicit opt-out, loud in the output./edikt:sdlc:prd PRD-NNN supersede --force-verify— recorded inrevision_historyso the audit trail captures the override.
There is no opt-out for verify gov after /edikt:adr:new — failures are surface-only (warning, no rollback), so there's no transition to bypass. The user fixes the directive or removes the bad verify: line.
When the gates conflict
You'll occasionally see an evaluator PASS alongside a verify FAIL (or vice versa). The resolution depends on which is right:
- Evaluator PASS, verify FAIL — usually means the verify command is mis-written (wrong path, brittle regex, missing dependency). Read the captured stderr in
.edikt/state/verify/<run>.json, fix the command, re-run. - Evaluator FAIL, verify PASS — usually means the verify command is too lenient (e.g.,
! grep BAD-PATTERN .passes when the directory is empty). Tighten the predicate. - Both BLOCKED — the criterion is genuinely ambiguous. Rewrite the AC, then re-run pre-flight.
The two gates are deliberately redundant. If you find yourself disabling one to make the other quieter, something is off — the right move is to fix whichever gate is producing the false signal.
Related
- Falsifiable Verification — the verify-runner contract,
verify_kind, and the pre-edit gate edikt verify— the runner that backs the verify gates/edikt:sdlc:plan— orchestrates the evaluator gates- Gates — specialist agents + completion-claim detector
- Features — config switches for the gate stack