Skip to content

This guide shows you how a task reports data to Sluice, and how later tasks and other flows use it. A task can emit three kinds of data:

Kind What it is Where it goes
Output A named JSON value, for example a row count or a file name. Templates of later tasks, the flow outputs, the Outputs tab.
Metric A named number with a unit and tags. The Metrics tab and the metric chart of the flow page.
Artifact A file that the task wrote, for example a report. Later tasks that declare it, and the Artifacts tab, for download.

Sluice puts the path of an empty file into the variable SLUICE_OUTPUTS of each script and command task. The task appends one JSON object per line to that file. The runner reads the file after the process ends and sends the data to the server.

{"type":"output","key":"rows","value":1234}
{"type":"metric","name":"rows_loaded","value":1234,"unit":"rows","tags":{"table":"orders"}}
{"type":"artifact","path":"out/report.html","name":"report.html","content_type":"text/html"}
import json, os
def emit(line):
with open(os.environ["SLUICE_OUTPUTS"], "a") as f:
f.write(json.dumps(line) + "\n")
emit({"type": "output", "key": "rows", "value": 1234})
emit({"type": "metric", "name": "rows_loaded", "value": 1234,
"unit": "rows", "tags": {"table": "orders"}})
emit({"type": "artifact", "path": "out/report.html", "content_type": "text/html"})

The task can write the lines at any time while it runs. The runner reads them only once, after the process ends. An http task writes no file: Sluice records its outputs status, headers and body. A subflow task gets the outputs of its child.

Line type Fields Limits
output key, value (any JSON value) key has 1 to 256 characters. All outputs of a task together: at most 1 MiB.
metric name, value (a number), unit, tags name matches ^[a-z][a-z0-9_.]{0,99}$. At most 8 tags. A tag value has at most 128 characters. At most 10 000 metrics per task.
artifact path, name, content_type path is relative to the working directory. name defaults to the file name of path and matches ^[A-Za-z0-9][A-Za-z0-9._-]{0,199}$. The file has at most SLUICE_MAX_ARTIFACT_BYTES, 100 MiB by default.

A line that breaks a rule does not change the result of the task. The runner skips the line and writes a warning to the task log, for example:

[sluice] warning: outputs line 4: invalid JSON
[sluice] warning: outputs line 7: invalid metric name "Rows"

When an output key appears twice, the later line wins.

Sluice masks secret values in output values, metric tag values and artifact files before it stores them. Metric names and numbers stay as they are.

A task reads an output of another task with ${{ tasks.<task_id>.outputs.<key> }}. The other task must be a dependency, direct or through other tasks. Otherwise validation fails with output_reference_not_dependency.

id: load-and-check
tasks:
- id: extract
type: script
file: pipelines/extract.py
- id: check
type: command
depends_on: [extract]
command: ["sh", "-c", "test \"$ROWS\" -gt 0"]
env:
ROWS: ${{ tasks.extract.outputs.rows }}
- id: notify
type: http
depends_on: [check]
method: POST
url: https://hooks.example.com/loaded
body: '{"rows": ${{ tasks.extract.outputs.rows }}}'

A string value renders as it is. A number, boolean, object or list renders as compact JSON. Thus body above sends {"rows": 1234}.

A task that reads an output which the dependency did not emit fails with reason template_error, and its process does not start. When a dependency ran several attempts, the template reads the outputs of the last attempt.

Each task run gets its own working directory, and Sluice removes it when the task run ends. All outputs of a task together have at most 1 MiB. To give a file to a later task, emit the file as an artifact and declare it in artifacts of the task that reads it.

id: report
tasks:
- id: transform
type: script
file: scripts/transform.py
- id: render
type: script
file: scripts/render.py
depends_on: [transform]
artifacts:
- from: transform
name: report.duckdb
path: data/report.duckdb

Here scripts/transform.py writes a file and emits the line {"type":"artifact","path":"out/report.duckdb","name":"report.duckdb"}. Before scripts/render.py starts, the runner downloads the artifact and writes it to data/report.duckdb in the working directory of render.

Field Rule
from The ID of a script or command task of the same flow. The task must be in depends_on of this task. Otherwise validation fails with artifact_source_not_dependency.
name The artifact name that the task in from emits.
path Optional. The destination, relative to the working directory. The default is name. The artifact replaces a namespace file at the same path.

The three fields are literal text. A ${{ block is not allowed in them. Only script and command tasks take artifacts. A task cannot read an artifact of a child execution of a subflow task.

The artifact works the same on the process, docker and kubernetes executors, and when the two tasks run on different instances. Each task can then have its own retry policy, its own logs and its own bar in the timeline.

When the artifact does not exist, the task that reads it fails with reason artifact_not_found, and its process does not start. This occurs when the task in from did not emit that name, or when its upload failed. The task in from keeps its state. Look for a warning in its log, for example a file above SLUICE_MAX_ARTIFACT_BYTES.

When the task in from ran several attempts, the runner reads the artifact of the last attempt. sluice executions restart gives the new execution the artifacts of each reused task. They show on its Artifacts tab, and they stay when retention deletes the old execution.

When the task in from has each, the task gets one file for each item. See Run a task for each item.

The outputs map of a flow names the results of the whole execution. Each value is a template, and it can read the outputs of any task.

id: nightly-load
tasks:
- id: extract_orders
type: script
file: pipelines/extract.sh
args: ["orders"]
outputs:
orders: ${{ tasks.extract_orders.outputs.rows }}

Sluice renders the flow outputs when the execution succeeds. A JSON number, boolean, object or list becomes a typed value. Other text stays a string. A template that fails to render ends the execution FAILED with reason output_error.

Flow outputs reach three places:

  • A downstream flow reads them as trigger.outputs. See Chain flows.
  • A parent subflow task gets them as its own outputs.
  • The execution API and sluice executions get --output json return them as outputs.

The execution page has an inspector with four tabs: Logs, Outputs, Metrics and Artifacts. Select a task in the Timeline to filter the tabs to that task. Click the bar again, or click All tasks, to see all tasks.

A succeeded execution of sales/nightly-load: the timeline with four tasks, and the inspector with the tabs Logs, Outputs, Metrics and Artifacts.A succeeded execution of sales/nightly-load: the timeline with four tasks, and the inspector with the tabs Logs, Outputs, Metrics and Artifacts.
Tab Shows
Outputs Without a selected task: Execution outputs, that is the flow outputs. Then the outputs of each attempt as JSON, for example extract_orders #1.
Metrics One row for each metric line: task, name, value, unit and tags.
Artifacts One row for each artifact: name, task, size and content type, with Download.

The Overview tab of a flow page has a metric chart next to the duration chart. It shows one point for each of the last 50 executions of the flow.

The Overview tab of the flow sales/nightly-load: the last five executions, the duration chart, and the chart of the metric rows_loaded with Sum selected.The Overview tab of the flow sales/nightly-load: the last five executions, the duration chart, and the chart of the metric rows_loaded with Sum selected.
  1. Select the metric in the list. The list holds every metric name that the flow has reported.

  2. Select the aggregation: Sum, Avg or Max. Sluice combines all lines of that metric in one execution, over all tasks and attempts.

  3. Optional: type a tag key in Group by tag, for example table. The chart then draws one line for each tag value.

The same data comes from GET /api/v1/flows/{namespace}/{flowId}/metrics?name=rows_loaded&agg=sum&group_by=table.