Skip to content
Try Gitea Cloud ☁️ for 30 days → Accelerate your Development & Deploys!
Support

Kubernetes Backend

With the Kubernetes backend, the runner pods run no Docker daemon. Instead, the runner creates a pod for each job (a job pod) in the namespace of the Runner, through the Kubernetes API, and deletes it when the job ends. Neither the runner pods nor the job pods are privileged, and each runner pod runs several jobs at once.

A Runner uses this backend when spec.kubernetesRuntime is set. Exactly one of spec.runtime and spec.kubernetesRuntime must be set.

  • The cluster does not allow privileged pods.
  • Jobs should be isolated in pods of their own, for example with a sandboxed runtime class such as Kata Containers.

Workflows that need Docker run on a Docker-in-Docker Runner instead, which can be used side by side.

Jobs that use any of the following fail:

Workflow feature Error
Docker actions (runs.using: docker in action.yml), also inside composite actions docker actions and docker:// steps are not supported on Kubernetes
uses: docker://... steps Same as above
options, volumes or credentials of the job container container options, volumes and credentials are not supported on Kubernetes
volumes or credentials of a service service <id>: volumes and credentials are not supported on Kubernetes
options of a service other than --health-cmd, --health-interval, --health-timeout, --health-retries and --health-start-period only --health-* options are supported on Kubernetes
Two services whose ids map to the same container name, such as my_db and my-db, or a service named job service <id> maps to container "<name>", which is empty or named like another container of the pod

Jobs also have no Docker daemon to use, so docker build, docker run, docker compose and tools that start containers, such as Testcontainers, do not work.

Some features work differently than with Docker:

  • Pull credentials for private images go in the imagePullSecrets of the job pod template, and volumes in its volumes, instead of in credentials and volumes of the workflow.
  • Services run as containers of the job pod and share its network. A step reaches a service by its id or by localhost. Two services cannot listen on the same port, and the host port of a ports mapping has no effect: a service is reached on its container port. An id that is not a valid host name is changed into one, for example my_db is reached as my-db.
  • The command of a service is visible in the pod spec, so pass secrets to services in env.
  • Labels map to images only, so jobs cannot run on the host, that is directly in the runner container.
  • Job images are pulled and cached by the nodes. On a cluster that adds nodes on demand, a job on a new node pulls its image again.
  • There is no tool cache volume: the tools that setup-* actions install are lost with the job pod.
  • A job pod that cannot be scheduled waits until the job times out, logging Waiting for pod <pod> to be scheduled in the job log.

How the runner runs jobs as pods is described in its documentation, Install on Kubernetes: Job pods.

  • Runner image 5.3.0 or later, without the -dind suffix; 6.0.0 or later is recommended. The default image is gitea/runner:nightly, pin a version in production.
  • Kubernetes 1.30 or later.
  • Job images must contain sh, tar, tail and env, and node to run JavaScript actions such as actions/checkout. The docker.gitea.com/runner-images images have all of them.

The requirements below come on top of the general requirements and the storage requirements.

  • Runner pods and the built-in cache server meet the restricted Pod Security Standard. So must the init containers of the runner pods.
  • Job pods meet baseline with the default settings, but not restricted, as the processes in job images usually run as root.

A namespace at baseline therefore works:

Terminal window
kubectl label namespace <namespace> pod-security.kubernetes.io/enforce=baseline

For a namespace at restricted, set the security context of the job pods in the job pod template, with job images that can run as a non-root user. The following meets restricted, but whether jobs work as that user depends on the images and the workflows. kubernetesRuntime.runnerRunAsUser sets the user of the runner pods and of the built-in cache server, not that of the job pods.

kubernetesRuntime:
jobPodTemplate:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1001
seccompProfile:
type: RuntimeDefault
containers:
- name: job
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
- name: $services
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
  • Each runner pod has one persistent volume, the action cache. There is no Docker data root volume, so storage.cacheVolumeSize, dockerStorageClassName and dockerStorageMountPath have no effect.
  • The StorageClass must support fsGroup. Runner pods and the built-in cache server run as a non-root user (UID 1000 by default) and obtain write access to their volumes through fsGroup. Some provisioners, often NFS ones, ignore fsGroup, and the runner then fails to download actions. Checking the cluster tests it.
  • Job pods use the disk of their node. The node pulls and caches the job images, and the job workspace is an emptyDir on the node by default. Leave enough disk on the nodes for the jobs that can run at once, as job images are often several GB, or job pods are evicted and their jobs fail. The workspace can also be put on a volume with the job pod template.
  • Jobs use the resources of their job pod. scalability.cpuRequest and memoryRequest (and cpuLimit and memoryLimit when set) apply to the job container of each job pod, 2 CPUs and 4Gi by default. Plan the cluster for the number of jobs that can run at once, up to maxSize × capacity. Service containers are assigned no resources unless the job pod template sets them for $services.
  • Runner pods use kubernetesRuntime.runnerResources, by default a request of 100m CPU and 256Mi with no limit. The runner mainly manages the job pods and needs few resources of its own, but the logs, action sources and cache traffic of every job pass through it, so raise them for a large capacity, verbose jobs or heavy cache use. If a LimitRange of the namespace adds default limits, check that they are high enough, or the runner can be terminated for exceeding its memory limit.
  • Keep runner pods on stable nodes with nodeSelector or affinity, for example a node pool that is not reclaimed often. When a runner pod is terminated abruptly, by a node failure or an eviction, the jobs it runs fail, up to capacity of them, and its job pods are deleted. Runner pods that are deleted in an orderly way, when scaling down or applying a change, finish their jobs first, see Draining and shutdown.
  • nodeSelector, affinity, tolerations and imagePullSecrets of the Runner do not apply to the job pods. Set those in the job pod template.

Allow:

  • job pods to reach Gitea at the URL it is configured with (its ROOT_URL), to check out code and upload artifacts, their image registries, any package registries or services the workflows use, and port 8088 of their runner pod, which serves the workflow cache to them;
  • runner pods to reach Gitea, the sources of the actions that jobs use (https://github.com by default, see Actions), the Kubernetes API server, and the cache server or S3 bucket.

Job pods carry the label app.kubernetes.io/managed-by: gitea-runner, by which a NetworkPolicy can select them. The labels of the runner pods and the cache server are listed in What the controller creates.

Job images are pulled by the container runtime of the nodes, which uses the proxy settings of the nodes, not the runnerEnv of the Runner.

The built-in cache server runs as a non-root user with this backend and listens on port 8080 by default. An explicit cacheServer.port must be 1024 or above while the built-in cache server is used, or the Runner is rejected.

The controller creates no NetworkPolicy.

The runner creates job pods with the permissions of the ClusterRole runner-controller-worker-role, which the installation manifest includes:

Resource Verbs
pods create, get, list, delete
pods/exec create, get
pods/log get
secrets create, delete

For each Runner of this backend, the controller creates a ServiceAccount <name>-worker in its namespace and a RoleBinding that grants it this ClusterRole in that namespace only.

  • To use a ServiceAccount of your own, set kubernetesRuntime.runnerServiceAccountName. The controller then creates neither the ServiceAccount nor the RoleBinding, and you grant the permissions above.
  • The controller does not take over a ServiceAccount or RoleBinding of the same name that it did not create. It creates no runner pods and reports an ApplyRunnerWorkerFailed warning instead. Delete the object or use runnerServiceAccountName.
  • Job pods do not mount a ServiceAccount token. A job pod template that turns on automountServiceAccountToken gives jobs the token of the default ServiceAccount of the namespace, not that of the runner.
  • A user who can create a Runner in a namespace can, through its runner pods, create pods and run commands in them in that namespace. Treat this permission as equivalent to namespace administration.

This manifest checks that the StorageClass provisions volumes, that a non-root pod can write to them through fsGroup, and that the namespace accepts the security settings of a runner pod at the restricted level. Replace <storage class> in it with the StorageClass the Runner will use: className is standard by default, not the default StorageClass of the cluster. When the volumes of the Runner use different StorageClasses, run the check once for each.

Terminal window
kubectl create namespace rc-preflight
kubectl label namespace rc-preflight pod-security.kubernetes.io/enforce=restricted
cat <<'EOF' | kubectl -n rc-preflight apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: preflight
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
storageClassName: <storage class> # the StorageClass to check
---
apiVersion: v1
kind: Pod
metadata:
name: preflight
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: test
image: busybox
command: ["sh", "-c", "touch /data/ok && sleep 3600"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: preflight
EOF
kubectl -n rc-preflight wait --for=condition=Ready pod/preflight --timeout=3m
kubectl -n rc-preflight exec preflight -- ls /data/ok

The check passes when both commands succeed. If the pod keeps restarting and kubectl -n rc-preflight logs preflight shows Permission denied, the StorageClass does not support fsGroup. Otherwise kubectl -n rc-preflight describe pod preflight shows why. Clean up with kubectl delete namespace rc-preflight.

Terminal window
kubectl create namespace gitea-k8s-runners
kubectl label namespace gitea-k8s-runners pod-security.kubernetes.io/enforce=baseline
kubectl -n gitea-k8s-runners create secret generic runner-registration-token \
--from-literal=token=<registration token>

Save the following as runner.yaml, replacing the Gitea URL and the StorageClass:

apiVersion: gitea.com/v1
kind: Runner
metadata:
name: k8s-runner
namespace: gitea-k8s-runners
spec:
gitea:
instanceUrl: "https://<gitea host>"
registrationTokenSecretRef:
name: runner-registration-token
key: token
kubernetesRuntime:
image: gitea/runner:6.0.0 # without -dind
capacity: 4 # jobs each runner pod runs at once
labels: # runs-on name: job image, without docker://
ubuntu-latest: docker.gitea.com/runner-images:ubuntu-latest
scalability:
minSize: 1 # runner pods
maxSize: 5 # runner pods: up to 5 × 4 jobs at once
cpuRequest: "2" # of each job container
memoryRequest: 4Gi
storage: # sizes for a trial
className: "<storage class>"
actionCacheVolumeSize: 10Gi
cacheServerVolumeSize: 20Gi

The volume sizes suit a trial. For production, size them as described in Capacity planning, and keeping the cache in S3-compatible storage is recommended.

Terminal window
kubectl apply -f runner.yaml
kubectl -n gitea-k8s-runners get runner,pod,serviceaccount,rolebinding

A runner pod k8s-runner-0001, a cache server pod k8s-runner-cache-server, the ServiceAccount k8s-runner-worker and a RoleBinding of the same name appear, and a runner named k8s-runner-0001 comes online in Gitea. kubectl -n gitea-k8s-runners describe runner k8s-runner shows RegisterRunner and Create... events on success, warnings otherwise.

Add a workflow with runs-on: ubuntu-latest to a repository that the token covers, such as the one in Creating a Runner of the Docker-in-Docker backend. While the job runs, a job pod named gitea-actions-task-<id>-... appears in the namespace.

All other options are described in Configuration and Storage and caching, and listed in the reference. The rest of this page covers the options of this backend only.

kubernetesRuntime:
labels:
ubuntu-latest: docker.gitea.com/runner-images:ubuntu-latest
sandboxed: docker.gitea.com/runner-images:ubuntu-latest
jobPodTemplate: # applies to every job
spec:
nodeSelector:
node-pool: ci-jobs
containers:
- name: job
resources:
limits:
memory: 8Gi
jobPodTemplates: # keyed by label, for jobs whose runs-on has it
sandboxed:
spec:
runtimeClassName: kata
  • The keys of labels are the names jobs use in runs-on, and the values their job images, without a prefix such as docker://. Names cannot contain :, , or whitespace.
  • jobPodTemplate is the base of every job pod, with metadata and spec only. The runner adds what it generates on top:
    • A container named job merges into the container that runs the steps, one named $services into each service container. Containers of other names are added to the job pod.

    • A volume named workspace replaces the emptyDir that holds the job workspace, for example to give jobs more space on a volume:

      jobPodTemplate:
      spec:
      volumes:
      - name: workspace
      ephemeral:
      volumeClaimTemplate:
      spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: <storage class>
      resources:
      requests:
      storage: 50Gi
    • Pull credentials for private job images go in its spec.imagePullSecrets, see Private job images.

  • jobPodTemplates are merged over jobPodTemplate, each for the jobs whose runs-on contains its key, in the order of runs-on when several match. Each key must be one of labels, or the Runner is rejected, so remove the template along with its label. In the example, jobs with runs-on: sandboxed run in Kata Containers, which needs a RuntimeClass named kata in the cluster.
  • Write a literal ${NAME} in the templates, for example in a command, as $${NAME}: sh -c 'echo $${HOME}'. A bare ${NAME} is rejected, see below.
  • Unless the templates set them, the controller adds the annotation karpenter.sh/do-not-disrupt: "true", which keeps Karpenter from removing the node of a running job, and the resources of the job container from scalability.

The resources from scalability apply to every job. To give some jobs more, add a label for them, and set the resources of the job container in its template:

kubernetesRuntime:
labels:
ubuntu-latest: docker.gitea.com/runner-images:ubuntu-latest
ubuntu-latest-large-mem: docker.gitea.com/runner-images:ubuntu-latest
jobPodTemplates:
ubuntu-latest-large-mem:
spec:
containers:
- name: job
resources:
requests:
memory: 16Gi
limits:
memory: 16Gi
scalability:
cpuRequest: "2" # every job
memoryRequest: 4Gi # other jobs

Jobs with runs-on: ubuntu-latest-large-mem request 16Gi of memory, which is also their limit, and 2 CPUs from scalability, which the template does not change. Other jobs request 2 CPUs and 4Gi.

  • Set both requests and limits. The template is merged over the resources from scalability one quantity at a time, so a request set alone can end up above a memoryLimit of scalability, which makes the template invalid.
  • The controller adds runner pods by the number of waiting jobs, not by the resources they need. Make sure the nodes can fit these job pods, for example with a nodeSelector in the same template that puts them on larger nodes: a job pod that cannot be scheduled waits until the job times out.

Job pods pull their images with the imagePullSecrets of the job pod template. Create a Secret with the credentials of the registry in the namespace of the Runner:

Terminal window
kubectl -n gitea-k8s-runners create secret docker-registry internal-registry \
--docker-server=registry.example.com \
--docker-username=<user name> \
--docker-password=<password>

Then name it in the template of internal, the label whose job image is in that registry:

kubernetesRuntime:
labels:
ubuntu-latest: docker.gitea.com/runner-images:ubuntu-latest
internal: registry.example.com/ci/build-image:1.0
jobPodTemplates:
internal:
spec:
imagePullSecrets:
- name: internal-registry
  • The credentials apply to every image of these job pods: the job image of the label, or the image of a container that the workflow sets instead, and the images of services.
  • To use them for every job, set them in jobPodTemplate instead. The imagePullSecrets of a jobPodTemplates entry are added to those of jobPodTemplate.
  • The imagePullSecrets of the Runner itself, at the top of spec, apply to the runner pods and the built-in cache server, not to job pods.

The controller checks that jobPodTemplate and each entry of jobPodTemplates meet these conditions:

  • Only metadata and spec at the top.
  • spec.containers is a list of containers, each with a name.
  • No bare ${NAME}.
  • The job container requests no more than its limit, counting the resources taken from scalability.

While a template fails these checks, the controller reports InvalidJobPodTemplate warnings, whose message names the template, such as kubernetesRuntime.jobPodTemplates["sandboxed"], and stops acting on the Runner until the template is fixed: it does not scale, apply changes, recreate runner pods that disappear, or delete expired volumes. Runner pods already running are not affected, and keep running jobs with their current configuration. Common mistakes:

Written Problem
jobPodTemplate: {runtimeClassName: kata} spec: is missing
jobPodTemplate: {template: {spec: ...}} One level of template: too many
containers: {job: {...}} containers must be a list
containers: [{resources: ...}] The container has no name
command: ["sh", "-c", "echo ${FOO}"] Write $${FOO}
A jobPodTemplates entry sets only limits: {memory: 2Gi} for job The request still comes from jobPodTemplate or scalability (4Gi by default) and exceeds the limit: set requests too
scalability sets only memoryLimit: 2Gi Without memory in the templates, the request is memoryRequest (4Gi by default) and exceeds the limit: lower memoryRequest too

Fields inside spec are not checked. A wrong field type, a malformed quantity or a missing RuntimeClass is rejected by the API server when the runner creates the job pod, and the job fails with create pod: ....

kubernetesRuntime:
capacity: 4 # jobs per runner pod, 4 by default
scalability:
minSize: 1 # runner pods
maxSize: 5 # runner pods
  • minSize and maxSize count runner pods, so up to maxSize × capacity jobs run at once.
  • Waiting jobs first take the free places of the running runner pods, and the controller adds runner pods for the rest. For example, with capacity: 4, two runner pods that run 3 jobs between them, and 9 waiting jobs, 5 jobs fit on the existing pods and the controller adds one runner pod for the other 4.
  • It scales down one runner pod at a time, starting with the highest number, and only one that runs no job, as described in Scaling.

When a change has to be applied to a runner pod that runs jobs, the pod is drained: it takes no new jobs, finishes the ones it has, and is then rebuilt. One runner pod of a Runner drains at a time. See How configuration changes are applied.

kubernetesRuntime:
shutdownTimeout: 3h # 3h by default
# terminationGracePeriodSeconds: 11100 # shutdownTimeout + 5m by default

The default shutdownTimeout of 3 hours matches the job timeout of the runner, so draining cancels no job. As a trade-off, node maintenance, such as kubectl drain, can wait up to 3 hours and 5 minutes for a runner pod to finish. Lower it for faster maintenance: jobs still running at the timeout are cancelled.

kubernetesRuntime.runnerConfigMapName works as described in Using your own runner configuration, and in addition:

  • runner.labels must be in the form <name>:kubernetes://<image>, with the same names as kubernetesRuntime.labels.
  • Keep runner.capacity equal to kubernetesRuntime.capacity, which the controller uses to plan scaling.
  • Set host.workdir_parent: /cache/act_workdir, and with the cache enabled a cache.dir under /cache. Otherwise both default to the home directory of the runner, which is not writable for its user, and jobs fail to download actions.
  • Do not set kubernetes.namespace or kubernetes.kubeconfig: job pods must be created in the namespace of the Runner. Without them, the runner uses its ServiceAccount and namespace.
  • jobPodTemplate and jobPodTemplates have no effect, and the controller adds neither the scalability resources nor the karpenter.sh/do-not-disrupt annotation to the job pods. Set them in kubernetes.pod_template of the configuration instead.
  • To allow the cache traffic with a NetworkPolicy, set a fixed cache.port. The generated configuration uses 8088.

The controller still manages the ServiceAccount, the RoleBinding and the security context of the runner pods.

Workflows that need Docker can run on a second Runner with the Docker-in-Docker backend. As that one needs privileged pods, put it in a namespace of its own that allows them, so the namespace of the Kubernetes backend keeps a stricter level. The token Secret must be in that namespace too:

Terminal window
kubectl create namespace gitea-docker-runners
kubectl label namespace gitea-docker-runners pod-security.kubernetes.io/enforce=privileged
kubectl -n gitea-docker-runners create secret generic runner-registration-token \
--from-literal=token=<registration token>
apiVersion: gitea.com/v1
kind: Runner
metadata:
name: docker-runner
namespace: gitea-docker-runners
spec:
gitea:
instanceUrl: "https://<gitea host>"
registrationTokenSecretRef:
name: runner-registration-token
key: token
runtime:
image: gitea/runner:6.0.0-dind
labels:
ubuntu-latest-docker: docker://docker.gitea.com/runner-images:ubuntu-latest
storage:
className: "<storage class>"

Workflows that need Docker use runs-on: ubuntu-latest-docker. Give the two Runner resources different labels: with the same label, a waiting job scales up both.

To move workflows from the Docker-in-Docker backend to this one, create a new Runner with this backend, in a namespace of its own, next to the existing one. Once the workflows run on the new Runner, delete the existing one.

The two backends do not take the same options: label values lose their docker:// prefix, the scalability resources apply to the job containers, and options such as pvcMounts and jobTmpfs have no equivalent. Write the new Runner from the reference rather than by copying the existing one.