Calibration

Calibration tells you whether a pack is ready. It asks the pack a bank of test questions with verified answer keys. It grades the answers. It saves a verified plan for each question the pack answers correctly. Because of these plans, most real questions arrive Known. Known answers are fast, cheap, and consistent, however the question is phrased.

What it does

1. Prepares. Compiles taught documents into facts (see How a pack reads documents) and studies the pack’s data models.
2. Writes and binds questions. Builds the test bank and binds each question to a verified answer key.
3. Builds plans ahead of time. Writes plans from the pack’s facts and data models. Each test question then already has a known path to its answer.
4. Tests. Asks the questions, grades each answer against its key, and keeps the plans that pass.

A plan is the pack’s saved method for one kind of question. It lists which facts to read, and which queries to run against which data models. When a question has a verified plan, the pack runs the plan directly. It does not work out the answer from the start.

Where the test questions come from

Source	How they get in
Objectives, the questions your team writes from real work	the brainstorm conversation, run in the edisyl app or through your AI client’s brainstorm tool, or structure and add in the CLI. See Objectives. Always tested first.
Generated questions	Calibration writes them from the pack’s documents and data, about 10 to 14 per data model. A larger pack gets more.

Generated questions are usually simpler than real ones. Real questions often combine several parts. For example, a figure plus the policy that explains it, or a ranking that depends on an earlier step. Objectives make the pack’s readiness meaningful for the work people do.

When it runs

Calibration runs automatically. You don’t need to start it.

• After every teach or import.
• When you load objectives.
• After a Lexicon edit. Calibration rereads the questions whose terms the edit moved.

After a change, the next full test run follows within 12 to 24 hours. Run edisyl calibrate test <pack> to measure a change sooner. This is CLI only.

What it costs

Calibration is the most expensive thing a pack does. For every test question, top-tier models write a plan, run it, and grade the result over several rounds. An ask costs a little each time. A calibration run costs more, all at once. It runs only when something has changed.

Spend has limits:

• Each pack has a daily calibration budget. calibrate test and sweeps don’t start when the budget can’t cover them. --dry-run shows the cost first.
• A document compile estimated at more than $80 or 30,000 claims waits for approval with edisyl calibrate approve-compile. Approval is CLI only.
• edisyl calibrate pause <pack>, or calibration_pause in chat, stops calibration from starting new work. For example, use it while a warehouse is rebuilt. Answers are not affected.

Bigger packs need more of it

Every table or document you add increases the number of ways a question can be built. There are more joins, measures, and passages that a question might combine. A pack that answers well when it is small can start to miss answers as it grows. When you add to a pack, add objectives for the new material. Calibration then tests the combinations that people will ask.

Reading where it stands

MCP

Say it in chat. The assistant calls pack_health, calibration_status, or target_questions_list.

“How ready is the growth-metrics pack, and what should we fix next?”

“What is calibration doing on growth-metrics now?”

“What state is each target question on growth-metrics in?”

CLI

edisyl pack health <pack>          # readiness against targets, and what to fix next
edisyl calibrate status <pack>     # what calibration is doing now
edisyl pack objectives list <pack> # each objective and its state

A pack is ready when its answers meet accuracy, time, and cost targets across the whole test bank. It does not need every objective to pass. Open the pack to people after the first calibration. Before it, answers are slower and less consistent.

See edisyl calibrate for every command, and Improving answers for fixing what calibration finds.