Sources, readers and chunkers

How a folder is registered as a source with its owner and its hosted-model switch, how each source's files are read and cut into passages, and what a run reports it did not read.

01Sources

prem sources add <name> <folder> [--prefix p] [--chunker name] [--okf-bundle] [--undeclared-as-machine] registers a folder, and prem ingest --source <name> reads it with its registered settings. Two sources may not share or nest a prefix. prem sources status <name> shows the last run. Removing a source leaves its documents indexed; ingest an empty folder over the same prefix with --allow-empty-source, or rebuild the index, to remove them. prem ingest <folder> still works and registers nothing. Every run of either form is recorded.

Each source also carries an owner and a switch. The owner is the person answerable for those documents, the one the review queue and the sources pages name; it decides nothing, because what anybody may read is the folder rule and there is only one place that decides that.

The switch is "may be served to hosted models", and it is not a flag: it is one deny entry, for the reserved hosted-model agents group, at the top of the source's folder rule. Keeping it in the rule means the gate that already decides every read decides this too, rather than a second flag some code path could forget to consult.

prem sources set handbook --owner dana
prem sources set handbook --hosted no      (never leaves the network)
prem sources list                          (both, in words)

Turning it off on a folder with no rule is refused rather than inventing one: a rule holding only a denial reaches nobody at all, which is not what "never leaves the network" means to the person asking for it. Set who may read it first. A missing reserved group is refused too, in both reading and setting: it is created with the tenant and cannot be removed, so its absence means the configuration database was edited by hand, and answering "may be served" there would be a reassuring sentence about a question this cannot answer.

A source a profile ADDS is held back unless the profile says otherwise; a source already registered is left as it is. A profile that sets the switch has to state the rule it is changing, because the switch is an entry of that rule. Existing sources are untouched by the migration, which means they may be served until somebody turns the switch, and prem sources list says so for each of them.

prem sources list tells three states apart, not two: never leaves, may be served, and a denial somebody wrote into the rule by hand rather than with the switch. The third is reported in its own words because the documents do not leave, and calling that "may be served" would be false.

02Chunkers

Each source is cut into passages by a chunker, chosen by name. markdown is built in and is the default: it cuts at headings and splits a long section at blank lines. prem sources add <name> <folder> --chunker <name> chooses one when a source is registered, prem sources set <name> --chunker <name> changes it, prem sources chunkers lists the ones this installation has, and the portal's source forms offer the same list. A name the installation does not have is refused when the source is added or changed; if a source names one anyway (a chunker removed from the build, or a second machine built without it), its run stops before it reads a file and is recorded as failed with the name, and the index is left as it was. Changing a source's chunker cuts, embeds and stores every one of its documents again at its next run.

To add a chunker, implement IChunker (a name, and a method that returns a document's chunks, each one text taken from the document) and ship it as an extension (see Extensions), which every host takes from the same composition point. A chunker whose output changes for the same text takes a new name, since the name is what tells the next run to cut again.

03Readers

Before a document is cut it is read, by a reader chosen by the file's extension: Markdown and plain text are built in, and a reader an extension brings (the sample's .csv reader; the Word, PDF and Excel readers planned as the first extensions) reaches the CLI, the API and the portal alike. A file no reader claims is counted as skipped, by extension, and never guessed at. A reader returns the document's text and, when it has one, its frontmatter; the content hash is the SHA-256 of the file's bytes.

04What a run did not read

A file in a format this version does not read (PDF, Word, a spreadsheet) is not indexed, and every run now says so, by extension, so a folder of them does not look like an empty success. Hidden files and folders are left out and not counted.