Skip to content
AggregaAggrega
Select language: English

Fluentd — a log aggregator, two targets

This example compiles a Fluentd log aggregator — the receiving end of a log pipeline — from one Developer-authored Aggregate to two realizations:

  • Docker Compose for local development, and
  • the synthesized helm Recipe running the official upstream fluent/fluentd image directly on Kubernetes (no external chart).

The synthesized helm Recipe fetches nothing at build time; helm/fluentd-chart renders the Fluent project’s own chart as the same aggregator, for a build that can reach a chart repository.

The same aggregate.yaml drives both. Only the Recipe layer selected by the Manifest changes per realization.

The aggregator role

A log pipeline has two ends. A forwarder (such as Fluent Bit) collects log records and ships them downstream. An aggregator is the sink at the other end: it listens on the forward input protocol, receives the records upstream forwarders send it, and routes them to its output. Fluentd here plays the aggregator role.

Because the aggregator receives rather than forwards, it declares no downstream uses socket. Instead it exports the host and port of its forward endpoint through provides, so a forwarder can bind it on a forward socket and ship records to it. The forward endpoint listens on port 24224 — the conventional Fluentd forward port.

Files

FileRole
aggregate.yamlDeveloper intent: an aggregator component (core/observability/fluentd) listening on the forward input.
manifest-compose.yamlMaps the Fluentd type to the Compose Recipe (compose/fluentd).
manifest-helm-synth.yamlMaps the Fluentd type to the synthesized helm Recipe (helm/fluentd).
release-compose.yaml / release-helm-synth.yamlOperator records for the two realizations.
release-helm-synth-buffer.yamlThe synthesized helm realization with durable buffering enabled (infra.storage.size set).

Because both a Compose and a synthesized helm Recipe carry the name fluentd, the Manifests reference them provider-scoped as compose/fluentd and helm/fluentd.

What it demonstrates

  • One authoring surface, many targets. A single Aggregate is the source of truth for what the stack is. Each Manifest says how to realize it.
  • The receiving end of the String Pipe. The aggregator exports its forward endpoint through provides — host (the aggregator’s service) and port (24224) on the forward protocol. A forwarder that binds this component pipes those strings into its own forward output; the compiler routes them without interpreting them.
  • Workload synthesis (Kubernetes). The synthesized helm Recipe declares a Workload — a Deployment plus a ClusterIP Service running the official fluent/fluentd image — that the helm provider translates into Kubernetes manifests with no external chart. The rendered resource set is exactly what the Workload declares.
  • Developer-controlled routing. The developer’s output match selects which received records are routed to the aggregator’s output. It defaults to **, which routes every received record.
  • Ephemeral buffer by default, durable on request. The aggregator buffers in memory, so the default render carries no persistent volume. Setting infra.storage.size (see release-helm-synth-buffer.yaml) backs the buffer with a durable volume mounted at /fluentd/log; the workload stays a single-replica Deployment.
  • Reproducible images. infra.engine_version pins the Fluentd image tag, so the rendered workload references a fixed image and the synthesized Recipe renders no floating tag.

Scaffold the example

You can build this example with the commands below, or have your AI assistant build it for you.

Write the example into a new folder named after it:

Terminal window
aggrega scaffold --example fluentd-aggregator

It then prints the commands to run next. The first changes into the new folder:

Terminal window
cd fluentd-aggregator

Build

This workspace carries no .aggrega-config.yaml, so the standard-library entities resolve from the hub registry, hub.aggrega.ai, and no catalog flag is needed; --auto-pull writes the lockfile a fresh clone lacks during the build:

Terminal window
# Local development (Docker Compose)
aggrega build --release release-compose.yaml --auto-pull
# Kubernetes (synthesized helm Recipe, official image)
aggrega build --release release-helm-synth.yaml --auto-pull
# Kubernetes with a durable buffer volume
aggrega build --release release-helm-synth-buffer.yaml --auto-pull

Rendered artifacts land under dist/.

With an AI assistant

An AI assistant set up with Aggrega’s tools can do the same: paste this prompt into it.

Use the Aggrega tools only, and create no folder yourself. If the Workspace folder holds no `fluentd-aggregator` folder yet, call the `scaffold` tool once with `example` set to `fluentd-aggregator` and no `dir`: it writes the example into a new `fluentd-aggregator` folder. Then call the `build` tool once for each Release, in this order — `release-compose.yaml`, `release-helm-synth.yaml`, `release-helm-synth-buffer.yaml` — each time with `workspace_root` set to `fluentd-aggregator`, `release` set to that file, `auto_pull` set to true and `write` set to true. Tell me which files each build wrote; they are under `fluentd-aggregator/dist/`.

What the forward endpoint looks like

On Compose, the aggregator writes a fluent.conf with a forward input source bound to the forward port, and routes matched records to standard output:

<source>
@type forward
bind 0.0.0.0
port 24224
</source>
<match **>
@type stdout
</match>

On Kubernetes, the synthesized helm Recipe writes the same fluent.conf at start-up, and the ClusterIP Service exposes the forward port:

spec:
type: ClusterIP
[… omitted: the selector matching the aggregator's Pods]
ports:
- name: forward
port: 24224
targetPort: forward

The Deployment references the pinned image and exposes the forward port on the container:

image: fluent/fluentd:v1.19.3-debian-2.1
[… omitted: the start-up command that writes the fluent.conf shown above]
ports:
- name: forward
containerPort: 24224

When infra.storage.size is set, the Recipe additionally renders a PersistentVolumeClaim and mounts it at /fluentd/log, backing a file buffer so buffered records survive restarts.

Run it (Compose)

Terminal window
aggrega build --release release-compose.yaml --auto-pull
docker compose -f dist/app/compose/docker-compose.yaml up -d

The aggregator listens for forwarded records on localhost:24224. Send it a test record with any Fluentd or Fluent Bit forwarder pointed at that port, and the received records appear on the aggregator’s standard output. Tear down with docker compose -f dist/app/compose/docker-compose.yaml down.