Contents

Extensions

How a reader, a chunker, an embedding provider, a way of signing in, a command, a setting or a principal mapper is added from a folder without a change to this repository, allowed by hash, loaded from the bytes that were checked, and refused with a reason.

A reader, a chunker, an embedding provider, a way of signing in, a prem command, a setting or a principal mapper can be added without a change to this repository. An extension is a folder holding one assembly and an extension.json beside it:

json01
{
  "name": "sentence-chunker",
  "version": "1.0.0",
  "assemblyFile": "SentenceChunker.dll",
  "sha256": "0a1b...",
  "seams": { "chunker": 1, "reader": 1, "signin": 1 },
  "files": [ { "file": "SentenceRules.dll", "sha256": "9c2d..." } ]
}

files is optional: every other file the extension loads, managed or unmanaged, by name in the extension's own folder, each with its SHA-256. A manifest with no files list loads its assembly alone.

A manifest is read to 64 KB at most; a larger one is refused as a bad manifest, extension.json is larger than 64 KB, the most a manifest may be.

seams gives, for each seam the extension uses, the version it was built for, and a version is the least the extension needs: a host that offers version 2 of a seam loads an extension built for version 1 or 2 of it. One built for a later version than the host offers, or for a seam the host does not have, is refused as seam too new, with a sentence naming the seam and, when the host has it, the version it was built for and the version the host offers. A seam's version goes up when its contract changes in a way an extension built for the older one cannot meet, and also when a member is added that an extension may call: an extension that uses the new member declares the new version, so an older host refuses it with a reason instead of failing it on the missing member partway through a run. Adding a member an extension neither implements nor calls leaves the version as it is. The PremAgentic an extension was built against is checked too: one built against a newer PremAgentic than the host is refused as core too new, with both versions named, and one built against the same or an older version is given the host's own contracts library. This version offers reader 2, and version 1 of chunker, source, embedding, signin, reminder, command, setting and principals.

Reader version 2 added two members a reader may use. ReadDocument.Skipped("reason") is for a file the reader recognizes and will not index: the run counts it under its extension and the reason, as .pdf (no text layer). UnreadableDocumentException("reason") is for a file that exists and cannot be read, such as a password-protected or damaged one: the run reports the file with the reason and keeps its existing index entry. The reason is shown to an administrator, so it says what is wrong with the file and quotes none of its content. Whatever else a reader throws on a file is contained to that file, which is reported unreadable, and the next file is read. A reader that uses either member declares reader 2. One built for reader 1 loads on this version and reads as before: the sentence-chunker sample still declares reader 1, and this repository's tests load its built manifest.

An extension may ship native libraries, one per platform, under runtimes/<rid>/native/ in its folder: runtimes/win-x64/native/helper.dll, runtimes/linux-x64/native/libhelper.so. Each is listed in the manifest's files with its SHA-256, by its path under the folder, with forward slashes. The host loads the one for the platform it runs on (<os>-<arch>, then <os>, then beside the assembly), measures it again just before loading it, and refuses the extension if a file under runtimes/ is not listed. A path that leaves the folder, or that uses a backslash, a drive or .., is a bad manifest. samples/extensions/native-lines is the shape, with its two small binaries committed and reproducible from one C file by scripts/build-native-sample.sh.

Managed libraries are loaded from beside the extension's assembly only. A package with one managed assembly per platform lays them out under runtimes/<rid>/lib/<tfm>/; an extension that carries such a library, listed or not, is refused as a bad manifest with a sentence naming the file. Publish the extension for its platform (dotnet publish -r linux-x64, for example), which puts that platform's assembly beside the extension's own, and list it in files like any other.

prem extensions allow refuses such a folder with the same sentence and allows nothing, so the refusal comes when the extension is allowed rather than at the next start.

A reminder sink (seam reminder, version 1) receives each delivered run of prem reminders run, after the built-in sink has kept it for the portal. The mail-reminders sample sends one message per owner over SMTP; its settings are mail.json beside its manifest, and its password is in a credentials file, never in mail.json, the manifest or a setting. --plan delivers to no sink.

A command (seam command, version 1) adds to prem: a command of the extension's own, such as prem greeting, or subcommands under prem groups, the one built-in command that takes them. It adds and never replaces: a built-in command, a built-in subcommand, and a command another loaded extension already brought are refused with a sentence naming it. Its usage text is written as a built-in one is, with a prem <command> <subcommand> line for every subcommand, and an extension whose usage names other subcommands than it registers is refused. prem <command> --help prints the text with the extension that added it, and a flag the text does not name is refused with it before anything runs. Help about a built-in command needs no database; help about an extension's command is answered once the extension loads, which needs the database, since the allow list is kept there. A subcommand that changes something says so, and makes its change and its row in the change record in one transaction, by the command line's actor, as a built-in one does.

A setting (seam setting, version 1) is listed, read, checked and changed with prem settings like a built-in one, and every change is in the change record. The extension says what its key means, what it takes, what applies when nothing is stored, and the check a value must pass before it is stored. A built-in key, and a key under trust. or extensions., are refused. The key exists only while the extension is loaded: without it prem settings takes no value for it, a value already stored is kept and not used, and a profile does not set an extension's setting in this version.

An extension that registers a command or a setting and does not declare that seam in its manifest is refused, so an older PremAgentic refuses it by the seam's name. prem extensions list names what each loaded extension adds. samples/extensions/greeting-command adds a command of its own, a subcommand under prem groups and a setting, and this repository's tests load it through the host and run it through the command line.

A principal mapper (seam principals, version 1) says what principals from outside mean here: the groups a sign-in adapter reports, and the principals a connector names for a file. It answers with the ids of PremAgentic groups, and the host keeps an answer only for a principal it asked about and only when it names a live group an administrator made in this deployment, so a mapper can never hand anybody everyone, a user, a role or a group PremAgentic maintains itself. A deployment has one mapper: a second extension that brings one is refused as name taken, with a sentence naming the first. With no mapper, which is what PremAgentic ships with, every such group and principal means nothing: a reported group is ignored, a file readable only by outside principals is readable by nobody, and nothing reads an outside name as a PremAgentic one. A mapper that throws fails the request or the ingest run that asked; it is never read as mapping everything or nothing. An extension that registers a mapper declares the principals seam, as one that registers a command or a setting declares that seam, and is refused if it does not.

The folder goes under the extensions folder, which is the setting extensions.folder or the environment variable PREM_EXTENSIONS_DIR, the setting winning. Neither one means no extensions, which is not an error.

Nothing loads until an administrator allows it:

bash02
prem extensions allow /opt/premagentic/extensions/sentence-chunker
prem extensions list
prem extensions disallow sentence-chunker

Documents an extension's reader indexed stay indexed after it is disallowed: an ingest keeps them and says so, and one ingest with --remove-unread removes them.

allow reads the manifest, computes the SHA-256 of the assembly and of every listed file itself, prints each, and writes the name and the allowed hash to the setting extensions.allowed, with an entry in the change record. The allowed hash covers every listed file, and is the assembly's hash alone when nothing is listed, so entries written before files could be listed still match. It refuses when the manifest's hashes and the files disagree, rather than writing a pair that could never match.

At startup the host reads the assembly once, hashes those bytes, and loads the assembly from the same bytes, so the file cannot be exchanged between the check and the load. A listed managed file is loaded the same way; an unmanaged library can only be loaded by path, so it is measured again just before it is loaded. A file in the folder the manifest does not list is never loaded, and a listed file whose bytes differ refuses the whole extension with a sentence naming the file. The pair of name and allowed hash must be in extensions.allowed. An extension is refused, with a reason, when its manifest cannot be read or used, when it names an assembly outside its own folder, when the hash does not match, when nobody allowed the pair, when it needs a seam, or a version of a seam, that this version does not offer, when it will not run, when what it registers is named the same as something already loaded, or when it brings a second principal mapper. A refusal never stops the deployment: it starts with the built-in readers, chunkers, embedding providers and ways of signing in, and no principal mapper, and prem extensions list and the health page say what was refused and why.

Each extension loads into its own context and takes the contracts it implements from the process, so two extensions can carry different versions of one library and neither can bring its own copy of the core. An extension may add an embedding provider under a new name and may not take a built-in one, so naming a provider can never quietly stand in for the local, offline model. An extension that brings a way of signing in is named in the log at every start, the way the trusted-header mode is, and so is a principal mapper. With a way of signing in from an extension and no principal mapper, the start says so too: any group those ways report means nothing here and is ignored, and the log names each such group once, the first time it is reported.

An upgrade is a new assembly with a new hash, so it is allowed again on purpose. Changing either setting applies when the service is next started, which is also the only safe moment to replace an assembly on disk.

samples/extensions/sentence-chunker is a whole extension that registers a chunker, a reader for .csv files and a sign-in adapter that claims nothing; it is loaded in this repository's tests to prove the refusals and that each seam is used end to end. Three first-party extensions, extensions/pdf-reader, extensions/docx-reader and extensions/xlsx-reader, read PDF, Word and Excel files; what they read, their limits and how to allow them are on Sources, readers and chunkers.

The conformance kit, tests/Premagentic.Conformance, holds xunit fixtures to inherit that prove a reader, a chunker, a connector, an embedding provider, a sign-in adapter or a reminder sink keeps its contract; the built-ins and the PDF, Word and Excel readers are held to them here. It packs as the package Premagentic.Conformance. Its version is the product version it is built with, the MSBuild property Version (from Directory.Build.props, or -p:Version=<version> given to dotnet pack), and it carries no copy of the core: the contracts come from the PremAgentic the extension builds against. The kit records the seam versions it proves, read from SeamVersions.cs when it is built, and each fixture has a test, The_kit_proves_the_seams_this_extension_builds_against, that fails when the PremAgentic the extension builds against offers another version of that seam, with a sentence naming the kit's version and the version to match. No package feed carries the kit; Writing an extension packs it into a local folder and restores it from there.

The connector fixture proves on every machine that one item a connector cannot read does not throw and does not become a document. Left alone, WithAnUnreadableItemAsync wraps the connector's source in the kit's OneUnreadableItem, which holds the first item the connector hands to a reader unreadable. Returning null from it fails the test rather than passing it over, and so does a connector that reads its items itself, since the kit then has nothing to hold; such a connector overrides it with a source that holds a real unreadable item.