Compilation Pipeline (High-Level)
This chapter details the internal physics of the Go compiler, defining how abstract YAML is translated into physical configurations across the compilation pipeline, the Intermediate Representation (IR) Graph, and WebAssembly (Wasm) execution sockets.
The Compilation Pipeline & Execution Flow
Aggrega runs a strict, fixed-order Ahead-of-Time (AOT) compiler pipeline rather than hardcoded orchestration logic. Triggered by aggrega build, the compiler passes the flat graph between isolated WebAssembly (Wasm) sockets to evaluate the architecture before rendering physical artifacts.
The Wasm sockets run on an embedded host — the Extism toolkit on top of the Wazero engine — behind an AOT-purity firewall that denies a plugin any runtime-access capability (network, filesystem, wall-clock, environment, standard I/O), so a plugin cannot break the deterministic, ahead-of-time guarantee. The host runs the publishing linter out of band, via aggrega lint. The in-band gates described below — the governance socket’s label-transform role at topology construction and its IR-verdict role at static analysis, plus the plugin pipeline that closes rendering — run in every build: a governance role wherever the workspace attaches a plugin to it, and the plugin pipeline for every Recipe that declares steps. Status of the Language tracks the status of each pipeline phase.
The pipeline executes entirely in-memory in a strict, fixed order:
Resolution — The Resolver & Syntax Validator
- Mechanism: Inheritance Merge, Go Template Interpolation, and JSON Schema 2020-12 Validation.
- Action: The CLI fetches all requested entities (Components, Features, Releases). It first checks these definitions for the
extendskeyword. For the Catalog entities, it deep-merges the YAML metadata, and the two kinds part company on the data pillars: a Feature’sschema:composes with its base’s under JSON Schema 2020-12 composition, while an extending Component composes structure only — its attachment slots, topology sockets, self-inject allowlist and named assertions — and receives its base’sschema:block exactly as the base authored it. An extendingReleasefolds by the same rule: the child’s value wins key by key, down to the first indivisible value — a scalar, a list, a template — which replaces whole, and a key the child writes asnullis removed. So its whole authoring surface travels — the boundary tree with its per-componentinfraand Feature attachments, theparametersblock, the annotations, and the Manifest pointer — while nometadatafield does, since identity and provenance are per-Release. Next, it performs an in-memory interpolation in three sub-steps: parameter contract validation against the Aggregate’s typedparameters:declarations; interpolation of the resolved values into the surroundingvalues:strings; withdrawal of the.parametersmap from every value-template context, leaving it readable to operator-authored path templates alone. Finally, it injects the resolvedinfrablocks into the target component payloads. Once the blueprint is fully concrete, the compiler uses the unifiedschema:declaration from the Catalog (compact form or full JSON Schema 2020-12) to strictly validate the developer’svaluesandinfraconfigurations locally.- Release Feature schema validation: Features attached via the Release (both boundary-wide and component-specific) are resolved against their Catalog entities during this step. The compiler validates the Release-provided
infrablocks against each Feature’sschema:declaration immediately after catalog resolution, ensuring type safety before the topology is built. - Annotation-driven Feature expansion sub-phase: After Manifest resolution, Recipe/Provider binding, and Manifest binding-target validation (paths declared under
mappings.features.annotations.<key>.mapsexist in the referenced Feature’sspec.schema), the compiler runs an annotation-expansion sub-phase before the topology builder. The sub-phase walks Release boundaries outer-to-inner and Aggregate subsystems outer-to-inner, computing the effective per-component annotation set through cascade + override + null-opt-out merge; parses each annotation value as YAML into a scalar, a structured value or null; routes each value (scalar or structured-per-subkey) through the Manifest binding’smapsdeclaration to the target Feature schema pillar with side-vs-pillar enforcement (Aggregate-side firesmaps.values; Release-side firesmaps.infra); validates per-leaf types against the Feature’sspec.schema; and injects synthetic Feature attachments, which the IR graph marks withorigin: annotation. Schemas are compiled once per build and reused across cascade walks. The expansion runs after binding-target validation and before topology construction, so downstream stages treat synthetic attachments structurally identically to explicit ones (except via provenance inspection). See 13-catalog-and-registry.md §“Annotation-driven Feature expansion” andlanguage-reference/providers/well-known-annotations.mdfor the binding shape, cascade semantics, and cross-side composition rules.
- Release Feature schema validation: Features attached via the Release (both boundary-wide and component-specific) are resolved against their Catalog entities during this step. The compiler validates the Release-provided
Topology construction — The Topology Builder & Mutation Gate
- Mechanism: AST Expansion + the governance socket’s
label-transformrole. - Action: The compiler natively reads the explicitly declared
usesblocks from the validated components to deterministically draw topological edges between nodes in the graph, leaving thevaluesblock completely opaque. It then evaluates the Mutation Gate, passing a projection of the graph — each node’s labels and thevaluesits instance declares, which the compiler hands over without reading — with the Release’s name and environment, to a governance plugin attached at thelabel-transformrole. The plugin evaluates the context and returns JSONPatches to dynamically auto-tag the graph (e.g., injecting the labelcompliance.aggrega.ai/data-class: restrictedbased on the Release’s environment, ordatastore.aggrega.ai/engine: vectorbased on a database’s declared extensions); the patches reach labels alone.
Graph finalization
- Mechanism: Normalization.
- Action: The compiler finalizes the flat Intermediate Representation (IR) Graph, preserving the explicit topological edges the topology builder drew. It then normalizes the attached Features and their
valuespayloads into a flat dictionary mapped by node ID, preparing the capability-agnostic graph for validation.- IR Payload Normalization: The IR’s structural shape enforces a Single Source of Truth for configuration: the topology layer carries only the graph structure (node identity, edges, metadata labels), and the normalized configuration state lives in a separate in-memory dictionary the compiler passes to downstream plugins. Mutators that change configuration change it in one place; the provider reads the normalized state without risk of drift from redundant storage.
Static analysis — The Static Analyzer & Validation Gate
- Mechanism: Topological Purity Checks + the governance socket’s
ir-verdictrole. - Action: The compiler runs abstract static analysis over the IR Graph. First, it extracts the
selectorstrings from theusesSchemadefinitions and evaluates them using a native Go label-matching engine (supporting Kubernetes-styleAND,ORviain(), andNOTvia!=semantics) across the explicitly defined graph edges to guarantee cross-node type safety over declared sockets. An edge on an ad-hoc socket — one an Aggregate author named and the catalog did not declare — has no catalog entry and therefore no selector, so there is nothing for this check to evaluate and nothing missing; each edge records which of the two it is, and what an ad-hoc edge may carry is bounded by the target’s declaredprovidesinstead. It also evaluates FeatureappliesToandappliesToAnyOfselectors to verify that attached features are compatible with their target component’s labels, and each filled attachment slot’s ownselectoragainst the labels of the Feature filling it. Second, it evaluates every presence mandate a node’s Component declares — eachfeatures.aggrega.ai/requiredType.<label>annotation — against that node’s final attachment set. The analyzer is where this check belongs because it is the first stage that holds both halves of the rule: the resolved Component entity’s annotations and the attachment set the 5-Tier cascade produced, counting Aggregate-attached, subsystem-attached, Release-attached and annotation-expanded attachments alike. A mandate no attachment satisfies halts the build; a mandate whose declaration is malformed halts with a distinct diagnostic, so a broken declaration never reads as a satisfied requirement. Third, it executes the Validation Gate, passing the complete graph structure (including all nodes, edges, attached features, and precalculated DFS paths) to a governance plugin attached at their-verdictrole. The enterprise’s custom Wasm policy evaluates the topology. If the Wasm plugin returns a failure array (e.g., detecting a continuous path from apublicnode to arestrictednode without a specific proxy feature), the build halts.
Rendering
- Mechanism: Pre-Render Interception (
injectMapping), Late Binding (valuesMapping), and Wasm Rendering Sockets (pipeline). - Action: The final artifact generation phase is split into strict temporal sub-phases that run in a fixed order, which keeps execution deterministic:
- Inject mapping (pre-render interception): The Go compiler resolves the Developer’s abstract
.context.injectcontract. It then executes theinjectMappingGo templates from all attachedFeatureRecipesfollowing the strict 5-Tier Onion Model execution order (Sub-Aggregate — a module component’s own attachments, then those its module’s subsystems declare, outer-to-inner → Root Aggregate component → Root Aggregate subsystem → Release boundary outer-to-inner → Release component-specific). The compiler natively deep-merges the resulting dictionaries over the Caller’s.context.injectmemory pipe. This safely rewrites environment variables and connection strings for the Caller before the Base Recipe ever sees them. A mapping may rewrite what the Caller’s owninject:block or one of its connections delivered; a key a different attached Feature already delivered — through its own mapping or its owninject:block — halts the build naming both attachments, so the execution order never decides between two attachments’ values. - Locals preamble: Before any
providestemplate runs, the compiler evaluates the Recipe’s.localsentries in the order they are written, each against a self-contained node-local context —.context.instance_id,.values,.infra, and the entries of the same block resolved so far, but not.context.injector.context.uses. An entry may therefore read a sibling defined above it and receives that sibling’s resolved value; author order is part of the contract wherever one entry reads another, and the block is never reordered to make a reference resolve. Resolving locals first is what lets aprovidestemplate read.locals. The Recipe’s.sourcetemplate is evaluated against the same context once those locals are resolved, which keeps.localsreusable as shorthand across a Recipe’s templates and lets authors parameterise.source(chart version, path, registry) per boundary. - Base rendering (base Recipe
valuesMapping): The compiler evaluates the Base Recipe’svaluesMappingtemplate. The template reads the intercepted strings from the.context.injectpipe (rewritten during inject mapping), and the resulting values dictionary is fed to the provider the Recipe’scapabilitynames — the helm and helm-chart providers both render through Helm — to render the base infrastructure artifacts. - Workload parsing and validation: A Recipe that declares
workload:— inline block content, orworkload: { $import: <path> }naming a file — has that document evaluated as a template against the node’s render context, then parsed as YAML into the Workload shape and checked: the rendered bytes are validated against the Workload schema, the decoded value is validated against it as well (the bytes carry keys the decode discards, and the decode elides a field supplied empty, so each rejects what the other cannot see), everyattachmentsentry is checked against the slots the bound Component’sfeaturesSchemadeclares, and every resource reference is checked against the resolvedprovidesof the slot it names. A failure here halts the build. The parsed Workload is placed on the template context as.workload, so a template evaluated later on the same node can read it. - Artifacts evaluation: Every entry a Recipe declares under
artifacts:— inline content, or a file named by that entry’s own$import:— is read, template-parsed, evaluated against the full template context, and carries the build’s provenance. The resulting projection is handed to the provider, which decides the output format, where provenance lands, the output path and the final bytes. Each emitted file records which authored form it came from, so a filename collision can name every contributor. - Workload synthesis: The parsed Workload is handed to the provider the Recipe’s
capabilitynames, which translates it into that provider’s native resources. A Recipe that declares noworkload:has nothing to synthesize here; its files come from the source render instead. - Feature rendering (FeatureRecipe Resource Generation): Every attachment on the node whose bound FeatureRecipe carries its own
source,workload,artifactsand/orvaluesMapping— the Resource Generation pattern, used for auxiliary manifests like a ServiceMonitor, a NetworkPolicy, a sidecar ConfigMap, or a credential-delivery resource — is evaluated in one pass, in the same 5-Tier Onion Model order inject mapping follows. There is one sequence over one attachment set: an operational Feature and a credential Feature travel the same path, and what differs between them is what their FeatureRecipe emits. Each attachment’s.valuesand.infraare swapped to its own scope; it inherits the base Recipe’s evaluated.locals;.attachment_namenames the instance being rendered and.providescarries the attachment’s own resolved provides, the same bag the consumer reads. The resulting values dictionary is fed to the provider named by the FeatureRecipe’s own capability, and the rendered files are appended to the boundary’s output carrying two coordinates — the component the Feature is attached to, then the name it is attached under — so one Feature name attached to several components keeps each separate. Which form those coordinates take follows from the host provider’s own layout: a<component>--<instance>/directory beside the component’s own, where the target gives each contributor a position of its own, and a<component>--<instance>--filename prefix where it reads only the files sitting directly in a directory. Inject-only FeatureRecipes — those carrying onlyinjectMapping, already handled during inject mapping — are skipped here. Like the sub-phases before it, feature rendering runs for one component at a time, so an attachment is normally evaluated right after the component it is attached to. An attachment whose FeatureRecipe declaresreadsCallerRenderFacts: trueorreadsTargetRenderFacts: trueis evaluated instead once every component, and every attachment that declares neither, has rendered — so each component connecting to its host, and each its host is wired to, has rendered by then — and its output lands where it would otherwise have landed. - The plugin pipeline (Wasm step execution): The compiler reads the Recipe’s
pipelinearray and executes its steps in declared order. It dispatches each step’s named Wasm plugin against the node’s rendered documents, applies the RFC 6902 JSON Patch operations the step returns to the documents those operations address, and emits the step’s side effects alongside them — the surface behind surgical mutations such as injecting an Envoy sidecar or mounting a secret VolumeMount. A step whosewhen:gate renders falsy is skipped as a pass-through, leaving the document set unchanged for the next step. Each step’spluginURI resolves ahead of execution to the verified artifact bytes the lockfile pins by digest. A FeatureRecipe’spipelinesteps execute the same way, over the attachment’s own rendered documents and, for a component-scope attachment, the host component’s alongside them.
- Inject mapping (pre-render interception): The Go compiler resolves the Developer’s abstract
All three Recipe input kinds reach the compiler: source: + valuesMapping: through base rendering, workload: through workload parsing and workload synthesis, and artifacts: through artifacts evaluation. Which of them a given provider may be handed is not decided sub-stage by sub-stage — it is settled once per node, ahead of any render dispatch, from the provider’s own declared contract, so a Recipe whose input kind the bound provider does not accept is refused there rather than part-way through rendering. See 16-recipe-input-composition.md for the three-input-kinds composition model, the Workload kind, and the Artifacts mechanism, and Status of the Language for which provider accepts which kind.
GitOps-Native Output
To remain strictly unopinionated about how its output is applied, the rendering plugins output the generated artifacts into a clean, boundary-isolated directory structure (e.g., dist/boundaries/aws-core-cluster/, dist/boundaries/gcp-data-cluster/). This allows external GitOps controllers (like ArgoCD or Flux) to sync and apply the infrastructure without the Aggrega compiler ever needing live cluster API credentials.
graph TD
subgraph CLI_Orchestrator ["The Aggrega Compiler Pipeline"]
direction TB
A["Resolution — The Resolver & Syntax Validator<br/>(Inheritance Merge + Parameter Interpolation + JSON Schema 2020-12 Validation)"] --> B["Topology construction — The Topology Builder<br/>(AST Expansion + governance label-transform)"]
B -->|"Emits IR Graph"| C["Graph finalization<br/>(Normalization)"]
C -->|"Flat IR Graph"| D["Static analysis — The Static Analyzer<br/>(Selectors + governance ir-verdict Gate)"]
D -->|"Validated Topology"| E["Rendering<br/>Inject mapping: Pre-Render Interception (injectMapping)<br/>Base rendering (valuesMapping)<br/>The plugin pipeline: Wasm Mutation (pipeline)"]
end
graph LR
E["Rendering"]
E -->|"Stringified Artifacts"| F["Helm Charts"]
E -->|"Stringified Artifacts"| G["Docker Compose"]
E -->|"Stringified Artifacts"| H["Kubernetes Manifests"]
E -->|"Stringified Artifacts"| I["Terraform Configuration"]
E -->|"Stringified Artifacts"| J["CloudFormation Templates"]
Infrastructure Source Map
Aggrega compiles a ~30-line Aggregate.yaml into thousands of lines of physical JSON or YAML configuration. Without a way to trace a running resource back to its source, debugging a failure at “main.tf.json line 4,032” forces developers to reconstruct the mapping manually. Aggrega’s rendering plugins address this by merging trace identifiers into the provider’s native metadata surface — metadata.labels and metadata.annotations for Kubernetes, a provenance comment for Terraform, and equivalent per-provider conventions — so each rendered object carries a pointer back to the source it came from. The aggrega.ai/compiler-version label (documented in governance-and-compliance §5) is one trace key in this family; beside it the stamp carries aggrega.ai/aggregate-name, aggrega.ai/component-id, aggrega.ai/boundary-path, aggrega.ai/recipe-uri and aggrega.ai/environment, and the helm, helm-chart, compose, terraform and cloudformation providers each emit it — the helm-chart provider on the template files a Recipe adds to a chart, since it copies the chart’s own templates unchanged.
The stamp carries no key identifying the feature origin or the catalog type.
Two further debug surfaces ship with the compiler. aggrega build --trace writes each node’s intermediate state, pass by pass, to .trace/ beside dist/, for a Recipe author inspecting what each stage received and produced; and a halting diagnostic carries the category of the authoring surface it belongs to — DEVELOPER_ERROR, OPS_ERROR, CATALOG_ERROR and the rest — wherever the failure has one, so it points at the right author. Errors that trace a failing generated manifest field back to the template line that produced it are not part of the compiler.
The Validator Pipeline
The schema-validation phase of resolution rests on a validator pipeline that converts entity-spec content (compact form, JSON Schema YAML, or mixed) into a unified JSON Schema 2020-12 document with per-leaf provenance, then validates developer-authored instance content against it. Five stages run as pure transformations in order; each is a deterministic function over its declared inputs.
Stage 1 — Compact-form compilation. The compiler walks the YAML tree of schema:, parameters:, and SchemaLibrary content; dispatches per node (compact form leaf vs JSON Schema YAML vs $ref: reference vs composition); emits resolved JSON Schema 2020-12. Compact form and JSON Schema YAML may mix at any node boundary via the $jsonschema: escape; the compiler handles the mixed case by walking the tree and dispatching per node. The transformation is a pure function: the same compact-form input always produces the same JSON Schema output. See language-reference/vocabulary/compact-form-authoring.md for the keyword vocabulary.
Stage 2 — spec.templates and $instantiate: resolution. For SchemaLibrary entries authored as template references, the compiler walks every library entry that invokes $instantiate: with $args:, substitutes the parameter values as literal-scalar replacement, and produces concrete schema entries. The substitution runs once per library load; resolved entries are cached. See language-reference/entities/schemalibrary.md for the library entity’s authoring shape.
Stage 3 — Cross-document $ref: expansion. For SchemaLibrary references via $ref: (the identity form <Library>#/schemas/<entry>), the compiler walks the reference chain at compile time via resolution-stack threading; the referenced content inlines into the consumer’s schema. The expansion preserves cycle detection (cycles halt the build with a chain-naming error), the identity-not-location guardrail (a reference carrying a registry host halts with a steer to the identity form — references address libraries by identity, not physical location), and single-source composition ($ref: blocks accept only the bare reference; consumer-side overrides compose via JSON Schema’s allOf:). See language-reference/vocabulary/cross-document-ref.md for the resolution semantics.
Stage 4 — Bridge synthesis for featuresSchema:. The compiler walks each entity’s featuresSchema: content in author order and emits a JSON Schema fragment describing the shape of a fill that may bind each attachment slot, with an x-aggrega-inject: extension keyword on a slot that declares an inject allowlist. That fragment is compiled and checked on its own, against the authored attachment block, rather than being folded into the developer-facing schema: content; the extension keyword is metadata that JSON Schema validation ignores, and no later stage reads it back off the fragment — a check wanting a slot’s allowlist reads the slot. The usesSchema: socket table is projected into no fragment at all: a socket’s selector: and inject: are read as the typed YAML the Architect wrote, and a socket’s per-edge valueSchema: / infraSchema: is compiled on its own. The synthesis is a pure function over its inputs.
Stage 5 — Custom merger with provenance. For entities that declare extends:, the compiler walks the parent chain; the parent’s resolved JSON Schema becomes the base; the child’s overrides apply on top. Every leaf in the resolved schema carries a per-leaf provenance tag (file path, line number, parent-vs-child origin). The merger’s output is what the validator engine consumes; the validator engine is unmodified third-party JSON Schema 2020-12 conformance with a deterministic walker around it for ordered finding emission.
The five stages run before the validator engine sees the schema. The engine receives a single resolved JSON Schema 2020-12 document plus the developer’s instance content; it produces a validation report. The error formatter consults the provenance tags to emit user-facing diagnostics that name source positions clearly — including the originating file for a constraint a Feature inherited from its base rather than authored itself.
The validator pipeline runs where each payload is first complete — a component’s values and infra during resolution, a sub-aggregate’s components as topology construction brings them in, and an attachment’s payload at graph finalization — so a schema finding halts the build before static analysis evaluates any selector, and before assertion evaluation, which lets the assertion templates rely on validated payloads.
Determinism preservation across the pipeline
Every stage of the compilation pipeline preserves the Deterministic Output Invariant: given the same source files plus the same lockfile plus the same compiler version, the build produces byte-identical output. The invariant is foundational — content-addressing in the lockfile requires stable artifact digests; reproducible builds require that a release authored on developer-A’s laptop compiles to the same output on developer-B’s laptop and on CI; diff-reviewability requires that source changes show up as the only changes in compiled output.
Determinism is preserved by a small number of disciplines that every transformation respects:
- Fixed iteration order. Every walk over map-shaped YAML iterates in a fixed order, because Go’s native map iteration is randomized. The compiler sorts before iterating at most stages (compact-form compiler, merger,
spec.templatesresolver, error formatter, YAML serializer); the attachment-slot projection instead carries the author’s own order through, so a slot table’s property order is the order it was written in. - No time embedding. Compilation does not embed timestamps in output. A build at 2:00 PM produces the same artifact as a build at 2:00 AM. The merger’s provenance metadata uses file:line, not file:timestamp.
- Workspace-relative paths. All paths in resolved output are workspace-relative; the path-resolution layer normalises before paths enter any output.
- No random sources, no external state. Transformations are pure functions over their declared inputs (entity content, parent chain, lockfile). External state is consulted only during resolution (OCI artifact fetching), and resolved content is content-addressed by digest.
- Canonical YAML serialisation. Output is written in one fixed form for one input: the same key order at every depth on every run, with arrays and
enum:arrays in input order; 2-space indentation; Unix LF line endings; unquoted strings where YAML allows;true/falseboolean form; null omitted for absent fields. $defs:alphabetical order. Schema$defs:entries are emitted in alphabetical key order in resolved JSON Schema.$defs:is conceptually a dictionary, not an ordered list; alphabetical order makes diffs predictable across reorganisations of the source.- Parallelism does not surface to output. The compiler can parallelise independent units (different entities can compile concurrently). Parallel results are gathered into a deterministic structure (e.g., sorted by entity name) before being serialised.
The compiler’s end-to-end tests compile each fixture several times within one test and assert that every run emits the same files with byte-identical contents. Repeating the compilation is what catches a map-iteration-order leak or an iteration over an unsorted set, which yields stable output on most runs and diverges occasionally. The tests run with the rest of the compiler’s test suite.
The Deterministic Output Invariant applies at build time: given the same source files + the same lockfile + the same compiler version, byte-identical output. The companion property at lock time (given the same source files + the same registry state, the same lockfile) is weaker — registries are mutable (new versions can be published; sidecars can be added; per-component metadata can evolve), so two lock invocations against the same source files at different times can produce different lockfiles. That is the lockfile’s purpose, mirroring conventions in npm, Cargo, and Go. From a fresh lockfile, the build produces deterministic output. The two-moment separation is what makes registry-side mutability tolerable: a sidecar published at time T2 changes the next lock invocation; it does not change builds against the lockfile generated at T1.