跳转到内容
AggregaAggrega
选择语言: 简体中文
本页面由机器翻译。阅读英文原文。

按需运行的任务:一个 Aggregate,四个目标

一个携带 core/manual-trigger 的任务在 Kubernetes、Docker Compose、Terraform 和 CloudFormation 上部署而不启动,旁边是一个不携带任何内容的任务,并附有在每个目标上启动一次运行的命令。

有些工作只有在有人要求时才会运行至完成:一次性的数据修复、按需的备份。core/manual-trigger 就是用于此的 Feature。挂接到一个没有调度的 core/compute/task 上时,它会部署该任务而不启动它,每次运行都由某人使用目标自己的命令手动启动。

此工作区为四个目标构建一个 Aggregate——通过 Helm 的 Kubernetes、Docker Compose、Terraform 和 CloudFormation——并在每个目标上展示该挂接在渲染输出中改变了什么,以及启动一次运行的命令。第二个任务不携带挂接,因此差异在同一次构建中即可看到。另外两个 Releases 预期会中止,本页引用了它们各自打印的内容。

它构建了什么

实例目录中的 Component挂接由什么启动运行
data-fixcore/compute/taskcore/manual-trigger某人,手动,在每个目标上
reportcore/compute/task无它的调度,或者在 Docker Compose 上为栈的启动

挂接位于 Aggregate 中,紧挨着它所改变的任务:按需运行是 data-fix 本身的一部分,因此由开发者编写,并且该 Feature 不接受任何值,也不接受 infra。Release 不为它编写任何内容。

report 的调度是一个参数 report_schedule,Aggregate 以空默认值声明它,并将其读入任务的 schedule 值:

aggregate.yaml
spec:
parameters:
report_schedule:
type: string
default: ""
[… omitted: the parameter's description]
components:
[… omitted: data-fix with its attachment, and the comment above each task]
report:
type: core/compute/task
values:
schedule: "{{ .parameters.report_schedule }}"

它之所以是参数,是因为四个目标对没有挂接的任务处理方式不一致。Kubernetes 在有调度时将其作为 CronJob 按调度运行,没有调度时则在部署时运行一次。Docker Compose 没有调度器,因此每次栈启动时运行该任务一次,并拒绝调度。在 AWS 上,Terraform 或 CloudFormation 应用时没有任何东西会运行 Amazon ECS 任务,因此那里的任务需要调度或该挂接,两者都没有时会被拒绝。所以 Kubernetes、Terraform 和 CloudFormation 的 Releases 填写 report_schedule,而 Docker Compose 的 Release 将其留空。

data-fix 在全部四个目标上以相同方式构建,没有调度,也没有参数。

六个 Releases

ReleaseManifestreport_schedule构建结果
release-helm.yamlmanifest-helm.yaml30 2 * * 1-5完成
release-compose.yamlmanifest-compose.yaml空完成
release-terraform.yamlmanifest-terraform.yaml30 2 * * 1-5完成
release-cloudformation.yamlmanifest-cloudformation.yaml30 2 * * 1-5完成
release-compose-scheduled.yamlmanifest-compose.yaml30 2 * * 1-5中止
release-terraform-unscheduled.yamlmanifest-terraform.yaml空中止

每个 Manifest 将 core/compute/task 映射到其目标的 Recipe,并将 core/manual-trigger 绑定到其目标的 FeatureRecipe:helm/task 和 helm/manual-trigger、compose/task 和 compose/manual-trigger、terraform/task-ecs 和 terraform/manual-trigger,以及 cloudformation/task-ecs 和 cloudformation/manual-trigger。未为该挂接绑定 FeatureRecipe 的 Manifest 会中止构建,并指出该挂接。

每个 Release 在 infra.image 下指定每个任务运行的镜像,因为 core/compute/task 不打包任何软件。AWS 的 Releases 还在 infra.overrides 下指定每个任务运行所用的集群、子网、安全组和执行角色,并且在 report 有调度的情况下,指定调度启动每次运行时所用的角色。其中每个特定于账户的值都是占位符。

生成示例

你可以用下面的命令构建这个示例,也可以让你的 AI 助手替你构建。

将示例写入一个以其命名的新文件夹:

Terminal window
aggrega scaffold --example task-run-on-demand

然后它会打印接下来要运行的命令。第一条命令进入新文件夹:

Terminal window
cd task-run-on-demand

构建

工作区从 .aggrega-config.yaml 指定的 hub 注册表读取标准库目录。这里的每个 Recipe 都仅从目录渲染,因此一旦锁文件解析完毕,每次构建都可离线运行。

Terminal window
aggrega build --release release-helm.yaml --auto-pull # one Release

在文档仓库的检出中(它包含由 aggrega scaffold --example 写出的副本所不具备的共享构建辅助脚本),./check.sh 会构建全部六个 Releases 并检查每个结果。它为每个 Release 打印一行,并且仅当每个都按本页所述结束时才以 0 退出:

RELEASE EXIT EXPECTED VERDICT
----------------------------------------------------------------------------------------------------
release-helm.yaml 0 complete OK: completed, the files the README shows
release-compose.yaml 0 complete OK: completed, the files the README shows
release-terraform.yaml 0 complete OK: completed, the files the README shows
release-cloudformation.yaml 0 complete OK: completed, the files the README shows
release-compose-scheduled.yaml 1 halt OK: halted with the expected refusal
release-terraform-unscheduled.yaml 1 halt OK: halted with the expected refusal
check: every build ended as expected.

借助 AI 助手

已配置 Aggrega 工具的 AI 助手也能完成同样的操作:把下面这条提示粘贴给它。

仅使用 Aggrega 工具,不要自行创建任何文件夹。如果工作区文件夹中还没有 `task-run-on-demand` 文件夹,请调用一次 `scaffold` 工具,将 `example` 设为 `task-run-on-demand`,且不设 `dir`:它会把示例写入新的 `task-run-on-demand` 文件夹。然后调用 `build` 工具,将 `workspace_root` 设为 `task-run-on-demand`,`release` 设为 `release-helm.yaml`,`auto_pull` 设为 true,`write` 设为 true。告诉我构建写入了哪些文件;它们位于 `task-run-on-demand/dist/` 下。

Kubernetes

release-helm.yaml 写入两个文件,每个任务一个 CronJob,此外不写任何内容。data-fix 是一个保持挂起的 CronJob,其调度是开发者从未编写过的:

dist/helm/data-fix-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: data-fix
[… omitted: the labels and annotations the build writes on every resource]
spec:
schedule: 0 0 30 2 *
suspend: true
jobTemplate:
spec:
template:
[… omitted: the pod template's metadata, the same labels again]
spec:
containers:
- name: main
image: registry.example.com/orders-data-fix:2.0.1
[… omitted: the container's CPU and memory requests and limits]
restartPolicy: Never

CronJob 需要调度,而 0 0 30 2 * 指的是 2 月 30 日,任何年份都没有这个日期:挂起时,该 CronJob 不启动任何运行;恢复后,它仍然没有可以启动运行的时间。如果没有该挂接,没有调度的任务会是一个 Job,它在部署后立即运行。

report 是一个按 Release 所填写的调度运行的 CronJob,且未被挂起:

dist/helm/report-cronjob.yaml
[… omitted: apiVersion, kind and metadata, as in data-fix-cronjob.yaml under report's name]
spec:
schedule: 30 2 * * 1-5
jobTemplate:
[… omitted: the job template, as in data-fix-cronjob.yaml with report's image]

data-fix 的一次运行是从其 CronJob 创建的 Job,名称由你选择:

Terminal window
kubectl create job data-fix-run-1 --from=cronjob/data-fix

同一命令在给定 --from=cronjob/report 时,会在 report 的计划运行之间启动一次额外运行。

Docker Compose

release-compose.yaml 写入一个 Compose 文件。每个任务都是一个服务,退出后 Compose 不会再次启动它;data-fix 还携带 profile manual,而 report 不携带任何 profile:

dist/compose/docker-compose.yaml
services:
data-fix:
image: registry.example.com/orders-data-fix:2.0.1
[… omitted: the labels the build writes on every service, and the service's network]
restart: "no"
[… omitted: the CPU and memory limits and reservations]
profiles:
- manual
report:
image: registry.example.com/orders-report:1.4.2
[… omitted: the labels and the network, as on data-fix]
restart: "no"
[… omitted: the CPU and memory limits and reservations, as on data-fix]
[… omitted: the network the services join]

Compose 只有在某个 profile 被启用或该服务在命令行中被指定时,才会启动携带该 profile 的服务,因此 docker compose up 会运行 report 一次,并让 data-fix 保持停止。data-fix 的一次运行通过指定其服务来启动,--rm 会在运行结束后删除其容器:

Terminal window
docker compose -f dist/compose/docker-compose.yaml run --rm data-fix

Terraform

release-terraform.yaml 写入一个根配置,每个任务一个 module 块,并在其旁边写入每个任务的模块。两个块的区别在于调度:

dist/terraform/aggrega.modules.tf
# Root configuration for this boundary. Each component rendered here is one
# module block below, and the directory that block points at holds that
# component's own files. One configuration means one dependency graph, which is
# what lets a reference or a depends_on reach from one component to another.
module "data-fix" {
source = "./modules/data-fix"
assign_public_ip = false
cluster = "arn:aws:ecs:eu-west-1:123456789012:cluster/example-cluster"
cpu = "256"
environment = {}
execution_role_arn = "arn:aws:iam::123456789012:role/example-ecs-task-execution"
image = "registry.example.com/orders-data-fix:2.0.1"
memory = "512"
name = "data-fix"
security_groups = ["sg-0fedcba9876543210"]
subnet_ids = ["subnet-0123456789abcdef0"]
}
module "report" {
source = "./modules/report"
assign_public_ip = false
cluster = "arn:aws:ecs:eu-west-1:123456789012:cluster/example-cluster"
cpu = "256"
environment = {}
execution_role_arn = "arn:aws:iam::123456789012:role/example-ecs-task-execution"
image = "registry.example.com/orders-report:1.4.2"
memory = "512"
name = "report"
schedule_expression = "cron(30 2 ? * 2,3,4,5,6 *)"
schedule_expression_timezone = "UTC"
scheduler_role_arn = "arn:aws:iam::123456789012:role/example-scheduler"
security_groups = ["sg-0fedcba9876543210"]
subnet_ids = ["subnet-0123456789abcdef0"]
}

该模块只有在被给定表达式时才声明 EventBridge Scheduler 调度,因此 data-fix 的任务定义是独立存在的:

dist/terraform/modules/data-fix/main.tf
[… omitted: the provider requirement, the Region lookup, the task definition and the comments on each]
resource "aws_scheduler_schedule" "this" {
count = var.schedule_expression == null ? 0 : 1
[… omitted: the rest of the schedule]

Recipe 以 EventBridge Scheduler 自己的形式写入 report 的调度:30 2 * * 1-5 变为 cron(30 2 ? * 2,3,4,5,6 *),星期几从星期日为 1 开始编号,并使用 UTC,因为 Release 未指定时区。

CloudFormation

release-cloudformation.yaml 写入一个栈模板。它为每个任务包含一个任务定义,并仅为 report 包含一个调度:

dist/cloudformation/cloudformation-stack.json
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"DataFixTaskDefinition": {
[… omitted: the provenance metadata the build writes on every resource]
"Properties": {
"ContainerDefinitions": [
{
"Essential": true,
"Image": "registry.example.com/orders-data-fix:2.0.1",
"Name": "main"
}
],
"Cpu": "256",
"ExecutionRoleArn": "arn:aws:iam::123456789012:role/example-ecs-task-execution",
"Family": "data-fix",
"Memory": "512",
"NetworkMode": "awsvpc",
"RequiresCompatibilities": [
"FARGATE"
]
},
"Type": "AWS::ECS::TaskDefinition"
},
"ReportSchedule": {
[… omitted: the provenance metadata]
"Properties": {
"FlexibleTimeWindow": {
"Mode": "OFF"
},
"ScheduleExpression": "cron(30 2 ? * 2,3,4,5,6 *)",
"ScheduleExpressionTimezone": "UTC",
"Target": {
"Arn": "arn:aws:ecs:eu-west-1:123456789012:cluster/example-cluster",
"EcsParameters": {
[… omitted: the launch type, the network configuration and the task count each run starts with]
"TaskDefinitionArn": {
"Ref": "ReportTaskDefinition"
}
},
"RoleArn": "arn:aws:iam::123456789012:role/example-scheduler"
}
},
"Type": "AWS::Scheduler::Schedule"
},
"ReportTaskDefinition": {
[… omitted: the same shape as DataFixTaskDefinition, with report's image and family]
}
}
}

栈中没有任何内容引用 DataFixTaskDefinition,因此没有任何东西会启动它。

在 Amazon ECS 上启动运行

在两个 AWS 目标上,data-fix 的一次运行都通过 Amazon ECS RunTask 操作启动,传入渲染出的任务定义族,以及 Release 在 infra.overrides 下指定的集群和网络配置:

Terminal window
aws ecs run-task \
--cluster arn:aws:ecs:eu-west-1:123456789012:cluster/example-cluster \
--task-definition data-fix \
--launch-type FARGATE \
--network-configuration 'awsvpcConfiguration={subnets=[subnet-0123456789abcdef0],securityGroups=[sg-0fedcba9876543210],assignPublicIp=DISABLED}'

仅按族指定时,任务定义解析为其最新的活动修订版。

构建不做的事

构建写入文件后即停止。它不会在任何目标上启动任何运行:它不连接任何集群,不启动任何 Compose 栈,也不调用任何 AWS API,挂接只改变文件的内容。它也不写入任何脚本:本页上的每条命令都是目标自己的命令,由运维该目标的人运行,且没有一条位于 dist/ 中。挂接的各个实现在任何目标上都不会添加自己的文件;每个实现所做的改变都在任务自身的输出中,如上所示。

会中止的两个 Releases

release-compose-scheduled.yaml 在 Docker Compose 上填写 report_schedule,并且同时将 core/manual-trigger 挂接到 report。在 Kubernetes 和 AWS 上,有调度的任务无论是否有该挂接都会按调度运行;在没有调度器的 Docker Compose 上,有调度的任务无论是否挂接都没有实现。aggrega build --release release-compose-scheduled.yaml --auto-pull 会中止:

[… omitted: what resolving the lockfile prints, the same halt among it]
[CATALOG_ERROR E0033]
Recipe/FeatureRecipe assertions
caused by: Recipe "task" instance "report" assertion "schedule_refused": [… omitted: the template position and the assertion's own source]: error calling fail: assertion failed: the component "report" is scheduled "30 2 * * 1-5", and Docker Compose has no scheduler: nothing in a compose file starts a service at a time, so the run would happen at none of the times written. Map core/compute/task to helm/task in the Manifest, which runs it on its schedule as a Kubernetes CronJob; or remove the schedule from the component in the Aggregate to run it once at each docker compose up, or attach core/manual-trigger to it there to start each run with docker compose run --rm

release-terraform-unscheduled.yaml 在 Terraform 上将 report_schedule 留空,因此 report 既没有调度也没有该挂接,AWS 上没有任何东西会启动它。这正是 core/manual-trigger 存在的用途,aggrega build --release release-terraform-unscheduled.yaml --auto-pull 会中止并指出它:

[… omitted: what resolving the lockfile prints, the same halt among it]
[CATALOG_ERROR E0033]
Recipe/FeatureRecipe assertions
caused by: Recipe "task-ecs" instance "report" assertion "schedule_translates": [… omitted: the template position and the assertion's own source]: error calling fail: assertion failed: the component "report" has no schedule and carries no core/manual-trigger attachment, so it runs once, and nothing on AWS starts a run that happens once: neither Terraform nor CloudFormation runs an Amazon ECS task when it applies, so the run would never happen. Give the component a schedule in the Aggregate; attach core/manual-trigger to it there to deploy its task definition alone, for the operator to start each run with aws ecs run-task; or map core/compute/task to helm/task in the Manifest, which runs it once on Kubernetes

这里调度通过 Release 填写的参数到达任务,因此给该组件一个调度就意味着填写 report_schedule,正如 release-terraform.yaml 所做的那样。