Maintaining a healthy pack

Most of a pack’s upkeep runs on its own. As documents are added, data changes and definitions move, calibration re-tests the pack and keeps it to its targets. Your team checks where the pack stands, approves the changes that need a person, and keeps edisyl itself up to date.

On its ownHuman in the loop

1

Something changes

A document is taught, data is imported, a term is edited or objectives are loaded

→

2

The pack re-tests itself

Writes test questions about what's new and runs a test against its accuracy, speed and cost targets

↑A fix is a change too

↻

↓

4

Your team fixes what it flags

A Lexicon term, a PACK.md rule or a document, each with a yes

←

3

Pack health shows where it stands

Readiness against target, and findings ranked with the next step for each

↻ Back to 1: a fix is a change too

Steps 2 and 3 run on their own after every change. A person steps in at step 4, and only where health points.

What runs on its own

• Calibration after every change. After every teach or import, and whenever people load objectives, the pack compiles the new material, writes test questions about it, and runs a test. Every document you add adds to the test questions the pack is measured on.
• Re-checks after a Lexicon edit. When a term’s meaning changes, the pack re-reads the test cases that term affects. Edits made within a few minutes of each other share one re-check.
• Measured on every test question. The pack is held to accuracy, speed and cost targets across every test question, not only the ones your team wrote.
• Standing work, on a schedule. Give an ask or a task a time and the pack runs it without being asked, such as a weekly digest or a Monday check. See Schedule.

See Calibration for how a pack is tested.

Check where it stands

Pack health is the one place to look. It shows accuracy, time and cost against target, then findings ranked by what each could return, each naming the next step.

edisyl pack health <pack>

From an agent, call the pack_health tool.

Pack health

Supplier risk

Example

AccuracyOn target

92%

target 90%

SpeedOn target

11 s

target < 15 s

Cost per askOver target

$0.13

target < $0.10

Findings, most to gain first

1. 3 questions misread "Q3"→ Define Q3 in the Lexicon
2. 2 questions have no source for supplier tiers→ Teach the supplier tier policy
3. 1 test's expected answer looks wrong→ Review and correct the test case

An example readout. Each pack's health shows its own targets and findings, from edisyl pack health or the pack_health tool.

A pack is ready when its answers meet targets across all of its test questions. That is not the same as every objective passing: a pack can be ready with a few objectives still failing. Compare one test run with the run before it rather than reading a single moment, since each run regrades its cases.

Fix what it flags

Each miss has a cause, and each cause has its own fix. Fix one thing at a time, then re-ask the question.

What went wrong	The fix
The pack read a word wrong: an acronym or house term landed on the wrong data, or on nothing	Define or merge the term in the Lexicon
It used the wrong source, the wrong period or the wrong style	Set it in PACK.md
No source in the pack states the fact	Teach the document that does
It couldn’t find or resolve the data	Task the pack to model it
The test’s expected answer is itself wrong	Correct the test case. See Calibrate

Most wrong answers on a pack that already holds the fact are misreads, so check the Lexicon first. Adding a note to move a number adds a document and new test questions, and usually leaves the misread in place.

Changes that need a person

The pack proposes and your team decides. Nothing below happens without a yes:

• Writes to the pack. Teaching a document, editing the Lexicon or changing PACK.md. A wrong document affects every answer after it.
• Changes to how the pack is measured. Correcting a test’s expected answer or overruling a grade changes the score, not what the pack knows. Each one names the case it moves, and none should be made only to make a number pass.
• Objectives as the work changes. When the work the pack supports changes, add the new questions it must answer and archive the ones that no longer matter. See Objectives.

Spend is bounded

Calibration is the most expensive thing a pack does, so it has limits:

• Each pack has a daily calibration budget. A run the budget can’t cover does not start, and --dry-run shows the cost first.
• A document compile estimated at more than $80 or 30,000 claims waits for your approval.
• You can pause calibration, for example while a warehouse is rebuilt. Answers are unaffected, and runs already in flight finish.

edisyl calibrate pause <pack> --reason "warehouse rebuild"
edisyl calibrate resume <pack>

Resuming needs the admin role in your organization.

Keep edisyl up to date

Your platform team upgrades edisyl in your environment. Each version’s release notes say what changed for the people building packs and give the steps to upgrade. See Upgrading.