Run tasks on Kubernetes

Copy page

This guide shows you how to run script and command tasks as Kubernetes Jobs. Sluice creates one Job for each task attempt. The Job runs your image and the Sluice runner.

http and subflow tasks always run inside the server. They cannot use the kubernetes executor. For the reasons to choose Kubernetes, see Executors, pools and the runner.

Before you start

The Sluice server runs in the cluster, or it has a kubeconfig for the cluster.
The server can create Jobs in the Job namespace. The Helm chart grants the permissions below.
Pods in the Job namespace can reach the server over HTTP.
Your task image can run on the nodes of the cluster.

The server needs these permissions in the Job namespace:

API group	Resource	Verbs
`batch`	`jobs`	create, get, list, watch, delete
core	`pods`	get, list, watch
core	`pods/log`	get

Enable the kubernetes executor

Give the server access to the cluster.

In a pod, the server uses the in-cluster configuration. Outside the cluster, set SLUICE_K8S_KUBECONFIG to the path of a kubeconfig file.
Choose the Job namespace.

SLUICE_K8S_NAMESPACE sets the namespace of the Jobs. When it is empty, Sluice uses the namespace of the server pod. Outside a pod, it uses default.
Turn the executor on.

With the default SLUICE_EXECUTORS=auto, the server turns the kubernetes executor on when a Job create dry run in the Job namespace succeeds within 5 seconds. With the Helm chart and its Role, this check succeeds. To turn it on without detection, name the executors, for example SLUICE_EXECUTORS=process,kubernetes. A client without a cluster configuration then stops the start.
Set the runner image.

An init container copies the runner from SLUICE_RUNNER_IMAGE into the pod. The Helm chart sets this variable to the server image. Without the chart, a release build uses its own published image, for example ghcr.io/alternayte/sluice:0.2.0. Set the variable to use a mirror in a private registry.
Set the URL that pods call.

The runner calls SLUICE_INTERNAL_URL. The Helm chart sets it to the URL of the Service, for example http://sluice.sluice-system.svc:8080. Without the chart, set it to an address of the server that pods can reach.
Check the executors of the instance.

Open Settings → Instances. The Executors column of your instance lists kubernetes.

Run a task as a Job

Add an executor block with type: kubernetes and an image.

id: k8s-hello
description: Print the Python version in a Kubernetes Job.
executor:
  type: kubernetes
  image: python:3.12-slim
tasks:
  - id: hello
    type: command
    command: ["python3", "-c", "import sys; print(sys.version)"]

Save the flow. On the flow page, click Run, then click Run in the dialog.
Select the task in the timeline. The Executor field of the inspector shows kubernetes · default.

List the Job in the cluster:

kubectl get jobs -l sluice.dev/managed-by=sluice

Set the pod fields

The executor block maps its fields to the pod of the Job:

Flow field	Pod field
`resources.requests`	`resources.requests` of the task container
`resources.limits`	`resources.limits` of the task container
`kubernetes.service_account`	`serviceAccountName`
`kubernetes.node_selector`	`nodeSelector`
`kubernetes.tolerations`	`tolerations`, with `key`, `operator`, `value`, `effect` and `toleration_seconds`
`kubernetes.image_pull_secrets`	`imagePullSecrets`
`kubernetes.labels`	Labels of the Job and the pod
`kubernetes.annotations`	Annotations of the Job and the pod

id: nightly-model
executor:
  type: kubernetes
  pool: default
  image: ghcr.io/acme/elt:1.4.0
  resources:
    requests: { cpu: 500m, memory: 512Mi }
    limits: { cpu: "2", memory: 2Gi }
  kubernetes:
    service_account: elt
    node_selector: { workload: batch }
    tolerations:
      - { key: batch, operator: Equal, value: "true", effect: NoSchedule }
    image_pull_secrets: [ghcr]
    labels: { team: data }
    annotations: { cost-center: "42" }
tasks:
  - id: train
    type: script
    file: pipelines/train.py

A Sluice label replaces a user label with the same key. Sluice does not set imagePullPolicy, so the Kubernetes default applies. The pull field is for the docker executor only. On a kubernetes block, it gives the validation error field_not_allowed.

Know what the Job holds

Property	Value
Name	`sluice-<task_run_id>`
`backoffLimit`	`0`. Sluice retries with the retry policy of the task, not with the Job.
`restartPolicy`	`Never`
`activeDeadlineSeconds`	The task timeout. The default task timeout is 24 hours.
`ttlSecondsAfterFinished`	`SLUICE_K8S_JOB_TTL`, default `600s`.
Init container	`sluice-runner` from `SLUICE_RUNNER_IMAGE`, only with `inject_runner: true`.
Task container	`task` from `executor.image`.
Volumes	An `emptyDir` at `/workdir`. With the runner injection, also an `emptyDir` at `/sluice-bin`.
Environment	`SLUICE_API_URL`, `SLUICE_RUN_TOKEN`, `SLUICE_TASK_RUN_ID` and `SLUICE_WORKDIR`.

The Job holds no secret values. The runner reads the secrets of the task from the server at run time. The run token in the Job is valid for this task run only, and it expires.

With inject_runner: false, the pod has no init container. The task container runs sluice exec from the image PATH.

Limit the number of Jobs

SLUICE_K8S_MAX_JOBS limits the running Jobs of one pool. The default is 50. Sluice counts the running kubernetes task runs of the pool in the database, over all instances. A task over the limit stays QUEUED until a Job ends.

Run tasks in a second cluster

A pool routes tasks to the instances that serve it. To run tasks in a second cluster, deploy a Sluice instance there with its own pool:

SLUICE_POOLS=cluster-b SLUICE_EXECUTORS=kubernetes sluice server

With the Helm chart, set pools: [cluster-b] in the values.

The instance uses the same database, the same object storage and the same master keys as the other instances. Then set pool: cluster-b in the executor block of the tasks. Only instances with this pool claim them.

Know what happens on a restart or a cancel

A Job continues when the server stops. The runner sends its data to any instance.
A cancel deletes the Job with background propagation. Kubernetes then removes the pod.
A Job that ends stays until ttlSecondsAfterFinished passes.

Fix common failures

Reason	Cause	Fix
`image_pull_failed`	A container of the pod waits with `ErrImagePull`, `ImagePullBackOff`, `InvalidImageName` or `ErrImageNeverPull`. Sluice deletes the Job.	Check the image name and `image_pull_secrets`.
`pod_pending_timeout`	The pod stayed `Pending` longer than `SLUICE_K8S_PENDING_TIMEOUT`, default `10m`.	Check the node selector, the tolerations and the free capacity of the cluster.
`lost`	The Job disappeared before the runner sent a result, for example after an eviction or a manual delete.	Read the events of the pod. The retry policy applies.
`no_instance_for_pool`	No online instance serves the pool with the kubernetes executor. The task stays `QUEUED`.	Start an instance with this pool and the kubernetes executor.
`executor_error`	The Job create failed, or the instance has no kubernetes executor.	Read the task error in the inspector. Check the Role of the server.