Semantic layer: pre-enrich or blank slate

When you connect a database, edisyl does two different things to it. The first is discovery. The connector lists the databases, schemas, tables, and columns that the connecting user can see, and imports them as models. Discovery is deterministic and fast, and it uses no LLM. The result is a catalog of names.

The second is enrichment. This step makes a source answerable. A study runs over the imported tables. It learns what the columns mean, how the tables join, which measures and dimensions they support, and which tables a question is about. This learned understanding is the semantic layer. The study also searches the pack’s documents to describe each table in your organization’s terms. For this reason, add documents before data. Until a table has a semantic layer, the pack knows only the table’s name. A question that depends on that table is declined, as if the data were never connected.

The last step of the Connect a source wizard asks how you want enrichment to run. The two options differ only in when and how much enrichment runs. They connect the same data.

How should we set up your semantic layer?

• Pre-enrich modelsRecommendedWhy?At this step, you have usually already limited the source to the tables that matter. Studying them all up front costs little, and the first ask can already be answered. Use a blank slate for wide schemas you could not limit.Every selected table is indexed and enriched up front. The study proposes joins, measures, and example questions for you to review.
• Start with a blank slateOnly the schema is indexed. The layer grows over time. Learning candidates appear as your team asks questions.

Pre-enrich models

Every table you selected is studied up front, in one pass, as soon as the source connects. The study proposes joins between tables, measures and dimensions on them, and example questions the source can answer. You review those proposals.

What you get: the source can answer questions about any selected table from the first ask. The proposals show you what the pack thinks the data means before anyone relies on it.

What it costs: one LLM-driven study per table, paid immediately. This includes tables that nobody asks about.

Start with a blank slate

Only the schema is indexed. A table is studied only when a question needs it. The semantic layer grows as people use the pack. When someone asks a question that the pack can’t ground in modeled data, that is a data-miss. The data-miss appears as a learning candidate: a specific piece of modeling that the pack has found it needs. To close it, task the pack to curate that part of the layer (see Improving answers). The next ask then gets an answer.

What you get: enrichment is spent only on tables that real questions use, in the order people need them. The layer matches how your team uses the data, not everything in the schema.

What it costs: the first ask about any unstudied table is declined. Someone has to route the gap before the question can be answered.

Choosing between them

	Pre-enrich models	Start with a blank slate
First ask on a table	Answers	Declined until curated
Enrichment spend	All selected tables, up front	Only tables that asks reach, over time
Review work	Proposals to review at connect time	Learning candidates to act on as they appear
Layer reflects	The schema you selected	The questions your team asks

Pick pre-enrich when

• You already narrowed the source to the tables that matter, on the Database and Tables steps. A small selection is cheap to study in full, with little waste.
• You know which questions are coming and want them answered on day one. For example, for a demo, an onboarding, or a team that won’t accept “not yet” on its first try.
• You want to see the pack’s proposed joins and measures before the data is used. You can then correct them early.

Pick a blank slate when

• The schema is wide and most of it is not useful. A common case is a warehouse with hundreds of tables where only a dozen matter for the questions. Pre-enriching all of it pays for understanding that nobody will use.
• You don’t yet know what people will ask. When the layer grows from real questions, you learn which tables matter, and the rest are not modeled.
• You want to control spend and increase it step by step, curating one gap at a time.
• The schema is still changing. Enriching tables that will be renamed or dropped next week is wasted work.

Narrow first, then decide

Most of the cost decision happens on the Tables step. If you can untick the tables that don’t matter, pre-enriching the rest is usually the better choice. Use a blank slate for sources where you can’t or don’t want to make that choice up front.

The choice isn’t final

You can enrich a blank slate source in full at any time. A pre-enriched source needs re-enrichment when its schema changes. Both routes end with the same result: a semantic layer over the tables you care about.

• Blank slate, later enriched in bulk. Run edisyl data-sources enrich <id> on the source (CLI only), or task the pack with a broader curation objective. This studies every imported table, exactly as pre-enrich would have.
• New tables added to the warehouse. The daily refresh adds and enriches them, or holds them as pending for you to admit. When you admit a table, it is studied. Adding tables to a connected source explains the steps.
• Specific tables, with direction. edisyl study <pack> <data-source-id> <model-name> --guidance "<what matters>" studies the tables you pass. The guidance tells the study what to focus on, such as which columns you report on. It changes the data models, not the pack’s documents. study is CLI only. See edisyl study.
• Either route, one gap at a time. edisyl task <pack> "...", or a task in chat, with a specific objective curates the part of the layer that objective needs, whichever starting point the source began with.

How this maps to the CLI

In the CLI, data-sources create discovers the tables and imports them by name. This is the blank slate. enrich is the pre-enrich step. Run it right after create to study every table up front. Until a table is studied, questions that depend on it are declined.

CLI only

# Discovery and import: table and column names only
DS=$(edisyl data-sources create --name "Warehouse Production" --type snowflake \
       --config-file sf.json --database PROD -j | jq -r '.id')

# Study every table, now and again after a schema change
edisyl data-sources enrich "$DS"

Enrichment can take up to an hour on a large source. See Connect a data source for the full steps, Adding tables to a connected source for what happens after, and Data sources for the command reference.