Kubernetes Backend
With the Kubernetes backend, the runner pods run no Docker daemon. Instead, the runner creates a pod for each job (a job pod) in the namespace of the Runner, through the Kubernetes API, and deletes it when the job ends. Neither the runner pods nor the job pods are privileged, and each runner pod runs several jobs at once.
A Runner uses this backend when spec.kubernetesRuntime is set. Exactly one of spec.runtime and spec.kubernetesRuntime must be set.
When to use it
Section titled “When to use it”- The cluster does not allow privileged pods.
- Jobs should be isolated in pods of their own, for example with a sandboxed runtime class such as Kata Containers.
Workflows that need Docker run on a Docker-in-Docker Runner instead, which can be used side by side.
Limitations
Section titled “Limitations”Jobs that use any of the following fail:
| Workflow feature | Error |
|---|---|
Docker actions (runs.using: docker in action.yml), also inside composite actions |
docker actions and docker:// steps are not supported on Kubernetes |
uses: docker://... steps |
Same as above |
options, volumes or credentials of the job container |
container options, volumes and credentials are not supported on Kubernetes |
volumes or credentials of a service |
service <id>: volumes and credentials are not supported on Kubernetes |
options of a service other than --health-cmd, --health-interval, --health-timeout, --health-retries and --health-start-period |
only --health-* options are supported on Kubernetes |
Two services whose ids map to the same container name, such as my_db and my-db, or a service named job |
service <id> maps to container "<name>", which is empty or named like another container of the pod |
Jobs also have no Docker daemon to use, so docker build, docker run, docker compose and tools that start containers, such as Testcontainers, do not work.
Some features work differently than with Docker:
- Pull credentials for private images go in the
imagePullSecretsof the job pod template, and volumes in itsvolumes, instead of incredentialsandvolumesof the workflow. - Services run as containers of the job pod and share its network. A step reaches a service by its id or by
localhost. Two services cannot listen on the same port, and the host port of aportsmapping has no effect: a service is reached on its container port. An id that is not a valid host name is changed into one, for examplemy_dbis reached asmy-db. - The
commandof a service is visible in the pod spec, so pass secrets to services inenv. - Labels map to images only, so jobs cannot run on the host, that is directly in the runner container.
- Job images are pulled and cached by the nodes. On a cluster that adds nodes on demand, a job on a new node pulls its image again.
- There is no tool cache volume: the tools that
setup-*actions install are lost with the job pod. - A job pod that cannot be scheduled waits until the job times out, logging
Waiting for pod <pod> to be scheduledin the job log.
How the runner runs jobs as pods is described in its documentation, Install on Kubernetes: Job pods.
Version requirements
Section titled “Version requirements”- Runner image 5.3.0 or later, without the
-dindsuffix; 6.0.0 or later is recommended. The default image isgitea/runner:nightly, pin a version in production. - Kubernetes 1.30 or later.
- Job images must contain
sh,tar,tailandenv, andnodeto run JavaScript actions such asactions/checkout. Thedocker.gitea.com/runner-imagesimages have all of them.
Cluster requirements
Section titled “Cluster requirements”The requirements below come on top of the general requirements and the storage requirements.
Pod Security level
Section titled “Pod Security level”- Runner pods and the built-in cache server meet the
restrictedPod Security Standard. So must the init containers of the runner pods. - Job pods meet
baselinewith the default settings, but notrestricted, as the processes in job images usually run as root.
A namespace at baseline therefore works:
kubectl label namespace <namespace> pod-security.kubernetes.io/enforce=baselineFor a namespace at restricted, set the security context of the job pods in the job pod template, with job images that can run as a non-root user. The following meets restricted, but whether jobs work as that user depends on the images and the workflows. kubernetesRuntime.runnerRunAsUser sets the user of the runner pods and of the built-in cache server, not that of the job pods.
kubernetesRuntime: jobPodTemplate: spec: securityContext: runAsNonRoot: true runAsUser: 1001 seccompProfile: type: RuntimeDefault containers: - name: job securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] - name: $services securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"]Storage
Section titled “Storage”- Each runner pod has one persistent volume, the action cache. There is no Docker data root volume, so
storage.cacheVolumeSize,dockerStorageClassNameanddockerStorageMountPathhave no effect. - The StorageClass must support
fsGroup. Runner pods and the built-in cache server run as a non-root user (UID 1000 by default) and obtain write access to their volumes throughfsGroup. Some provisioners, often NFS ones, ignorefsGroup, and the runner then fails to download actions. Checking the cluster tests it. - Job pods use the disk of their node. The node pulls and caches the job images, and the job workspace is an
emptyDiron the node by default. Leave enough disk on the nodes for the jobs that can run at once, as job images are often several GB, or job pods are evicted and their jobs fail. The workspace can also be put on a volume with the job pod template.
Compute
Section titled “Compute”- Jobs use the resources of their job pod.
scalability.cpuRequestandmemoryRequest(andcpuLimitandmemoryLimitwhen set) apply to thejobcontainer of each job pod, 2 CPUs and 4Gi by default. Plan the cluster for the number of jobs that can run at once, up tomaxSize × capacity. Service containers are assigned no resources unless the job pod template sets them for$services. - Runner pods use
kubernetesRuntime.runnerResources, by default a request of 100m CPU and 256Mi with no limit. The runner mainly manages the job pods and needs few resources of its own, but the logs, action sources and cache traffic of every job pass through it, so raise them for a largecapacity, verbose jobs or heavy cache use. If a LimitRange of the namespace adds default limits, check that they are high enough, or the runner can be terminated for exceeding its memory limit. - Keep runner pods on stable nodes with
nodeSelectororaffinity, for example a node pool that is not reclaimed often. When a runner pod is terminated abruptly, by a node failure or an eviction, the jobs it runs fail, up tocapacityof them, and its job pods are deleted. Runner pods that are deleted in an orderly way, when scaling down or applying a change, finish their jobs first, see Draining and shutdown. nodeSelector,affinity,tolerationsandimagePullSecretsof theRunnerdo not apply to the job pods. Set those in the job pod template.
Network
Section titled “Network”Allow:
- job pods to reach Gitea at the URL it is configured with (its
ROOT_URL), to check out code and upload artifacts, their image registries, any package registries or services the workflows use, and port 8088 of their runner pod, which serves the workflow cache to them; - runner pods to reach Gitea, the sources of the actions that jobs use (
https://github.comby default, see Actions), the Kubernetes API server, and the cache server or S3 bucket.
Job pods carry the label app.kubernetes.io/managed-by: gitea-runner, by which a NetworkPolicy can select them. The labels of the runner pods and the cache server are listed in What the controller creates.
Job images are pulled by the container runtime of the nodes, which uses the proxy settings of the nodes, not the runnerEnv of the Runner.
The built-in cache server runs as a non-root user with this backend and listens on port 8080 by default. An explicit cacheServer.port must be 1024 or above while the built-in cache server is used, or the Runner is rejected.
The controller creates no NetworkPolicy.
Permissions
Section titled “Permissions”The runner creates job pods with the permissions of the ClusterRole runner-controller-worker-role, which the installation manifest includes:
| Resource | Verbs |
|---|---|
pods |
create, get, list, delete |
pods/exec |
create, get |
pods/log |
get |
secrets |
create, delete |
For each Runner of this backend, the controller creates a ServiceAccount <name>-worker in its namespace and a RoleBinding that grants it this ClusterRole in that namespace only.
- To use a ServiceAccount of your own, set
kubernetesRuntime.runnerServiceAccountName. The controller then creates neither the ServiceAccount nor the RoleBinding, and you grant the permissions above. - The controller does not take over a ServiceAccount or RoleBinding of the same name that it did not create. It creates no runner pods and reports an
ApplyRunnerWorkerFailedwarning instead. Delete the object or userunnerServiceAccountName. - Job pods do not mount a ServiceAccount token. A job pod template that turns on
automountServiceAccountTokengives jobs the token of thedefaultServiceAccount of the namespace, not that of the runner. - A user who can create a
Runnerin a namespace can, through its runner pods, create pods and run commands in them in that namespace. Treat this permission as equivalent to namespace administration.
Checking the cluster
Section titled “Checking the cluster”This manifest checks that the StorageClass provisions volumes, that a non-root pod can write to them through fsGroup, and that the namespace accepts the security settings of a runner pod at the restricted level. Replace <storage class> in it with the StorageClass the Runner will use: className is standard by default, not the default StorageClass of the cluster. When the volumes of the Runner use different StorageClasses, run the check once for each.
kubectl create namespace rc-preflightkubectl label namespace rc-preflight pod-security.kubernetes.io/enforce=restricted
cat <<'EOF' | kubectl -n rc-preflight apply -f -apiVersion: v1kind: PersistentVolumeClaimmetadata: name: preflightspec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi storageClassName: <storage class> # the StorageClass to check---apiVersion: v1kind: Podmetadata: name: preflightspec: securityContext: runAsNonRoot: true runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 seccompProfile: type: RuntimeDefault containers: - name: test image: busybox command: ["sh", "-c", "touch /data/ok && sleep 3600"] securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: preflightEOF
kubectl -n rc-preflight wait --for=condition=Ready pod/preflight --timeout=3mkubectl -n rc-preflight exec preflight -- ls /data/okThe check passes when both commands succeed. If the pod keeps restarting and kubectl -n rc-preflight logs preflight shows Permission denied, the StorageClass does not support fsGroup. Otherwise kubectl -n rc-preflight describe pod preflight shows why. Clean up with kubectl delete namespace rc-preflight.
Creating a Runner
Section titled “Creating a Runner”kubectl create namespace gitea-k8s-runnerskubectl label namespace gitea-k8s-runners pod-security.kubernetes.io/enforce=baselinekubectl -n gitea-k8s-runners create secret generic runner-registration-token \ --from-literal=token=<registration token>Save the following as runner.yaml, replacing the Gitea URL and the StorageClass:
apiVersion: gitea.com/v1kind: Runnermetadata: name: k8s-runner namespace: gitea-k8s-runnersspec: gitea: instanceUrl: "https://<gitea host>" registrationTokenSecretRef: name: runner-registration-token key: token
kubernetesRuntime: image: gitea/runner:6.0.0 # without -dind capacity: 4 # jobs each runner pod runs at once labels: # runs-on name: job image, without docker:// ubuntu-latest: docker.gitea.com/runner-images:ubuntu-latest
scalability: minSize: 1 # runner pods maxSize: 5 # runner pods: up to 5 × 4 jobs at once cpuRequest: "2" # of each job container memoryRequest: 4Gi
storage: # sizes for a trial className: "<storage class>" actionCacheVolumeSize: 10Gi cacheServerVolumeSize: 20GiThe volume sizes suit a trial. For production, size them as described in Capacity planning, and keeping the cache in S3-compatible storage is recommended.
kubectl apply -f runner.yamlkubectl -n gitea-k8s-runners get runner,pod,serviceaccount,rolebindingA runner pod k8s-runner-0001, a cache server pod k8s-runner-cache-server, the ServiceAccount k8s-runner-worker and a RoleBinding of the same name appear, and a runner named k8s-runner-0001 comes online in Gitea. kubectl -n gitea-k8s-runners describe runner k8s-runner shows RegisterRunner and Create... events on success, warnings otherwise.
Add a workflow with runs-on: ubuntu-latest to a repository that the token covers, such as the one in Creating a Runner of the Docker-in-Docker backend. While the job runs, a job pod named gitea-actions-task-<id>-... appears in the namespace.
All other options are described in Configuration and Storage and caching, and listed in the reference. The rest of this page covers the options of this backend only.
Labels and job pod templates
Section titled “Labels and job pod templates”kubernetesRuntime: labels: ubuntu-latest: docker.gitea.com/runner-images:ubuntu-latest sandboxed: docker.gitea.com/runner-images:ubuntu-latest jobPodTemplate: # applies to every job spec: nodeSelector: node-pool: ci-jobs containers: - name: job resources: limits: memory: 8Gi jobPodTemplates: # keyed by label, for jobs whose runs-on has it sandboxed: spec: runtimeClassName: kata- The keys of
labelsare the names jobs use inruns-on, and the values their job images, without a prefix such asdocker://. Names cannot contain:,,or whitespace. jobPodTemplateis the base of every job pod, withmetadataandspeconly. The runner adds what it generates on top:-
A container named
jobmerges into the container that runs the steps, one named$servicesinto each service container. Containers of other names are added to the job pod. -
A volume named
workspacereplaces theemptyDirthat holds the job workspace, for example to give jobs more space on a volume:jobPodTemplate:spec:volumes:- name: workspaceephemeral:volumeClaimTemplate:spec:accessModes: ["ReadWriteOnce"]storageClassName: <storage class>resources:requests:storage: 50Gi -
Pull credentials for private job images go in its
spec.imagePullSecrets, see Private job images.
-
jobPodTemplatesare merged overjobPodTemplate, each for the jobs whoseruns-oncontains its key, in the order ofruns-onwhen several match. Each key must be one oflabels, or theRunneris rejected, so remove the template along with its label. In the example, jobs withruns-on: sandboxedrun in Kata Containers, which needs a RuntimeClass namedkatain the cluster.- Write a literal
${NAME}in the templates, for example in a command, as$${NAME}:sh -c 'echo $${HOME}'. A bare${NAME}is rejected, see below. - Unless the templates set them, the controller adds the annotation
karpenter.sh/do-not-disrupt: "true", which keeps Karpenter from removing the node of a running job, and the resources of thejobcontainer fromscalability.
More resources for some jobs
Section titled “More resources for some jobs”The resources from scalability apply to every job. To give some jobs more, add a label for them, and set the resources of the job container in its template:
kubernetesRuntime: labels: ubuntu-latest: docker.gitea.com/runner-images:ubuntu-latest ubuntu-latest-large-mem: docker.gitea.com/runner-images:ubuntu-latest jobPodTemplates: ubuntu-latest-large-mem: spec: containers: - name: job resources: requests: memory: 16Gi limits: memory: 16Giscalability: cpuRequest: "2" # every job memoryRequest: 4Gi # other jobsJobs with runs-on: ubuntu-latest-large-mem request 16Gi of memory, which is also their limit, and 2 CPUs from scalability, which the template does not change. Other jobs request 2 CPUs and 4Gi.
- Set both
requestsandlimits. The template is merged over the resources fromscalabilityone quantity at a time, so a request set alone can end up above amemoryLimitofscalability, which makes the template invalid. - The controller adds runner pods by the number of waiting jobs, not by the resources they need. Make sure the nodes can fit these job pods, for example with a
nodeSelectorin the same template that puts them on larger nodes: a job pod that cannot be scheduled waits until the job times out.
Private job images
Section titled “Private job images”Job pods pull their images with the imagePullSecrets of the job pod template. Create a Secret with the credentials of the registry in the namespace of the Runner:
kubectl -n gitea-k8s-runners create secret docker-registry internal-registry \ --docker-server=registry.example.com \ --docker-username=<user name> \ --docker-password=<password>Then name it in the template of internal, the label whose job image is in that registry:
kubernetesRuntime: labels: ubuntu-latest: docker.gitea.com/runner-images:ubuntu-latest internal: registry.example.com/ci/build-image:1.0 jobPodTemplates: internal: spec: imagePullSecrets: - name: internal-registry- The credentials apply to every image of these job pods: the job image of the label, or the image of a
containerthat the workflow sets instead, and the images of services. - To use them for every job, set them in
jobPodTemplateinstead. TheimagePullSecretsof ajobPodTemplatesentry are added to those ofjobPodTemplate. - The
imagePullSecretsof theRunneritself, at the top ofspec, apply to the runner pods and the built-in cache server, not to job pods.
Invalid templates
Section titled “Invalid templates”The controller checks that jobPodTemplate and each entry of jobPodTemplates meet these conditions:
- Only
metadataandspecat the top. spec.containersis a list of containers, each with aname.- No bare
${NAME}. - The
jobcontainer requests no more than its limit, counting the resources taken fromscalability.
While a template fails these checks, the controller reports InvalidJobPodTemplate warnings, whose message names the template, such as kubernetesRuntime.jobPodTemplates["sandboxed"], and stops acting on the Runner until the template is fixed: it does not scale, apply changes, recreate runner pods that disappear, or delete expired volumes. Runner pods already running are not affected, and keep running jobs with their current configuration. Common mistakes:
| Written | Problem |
|---|---|
jobPodTemplate: {runtimeClassName: kata} |
spec: is missing |
jobPodTemplate: {template: {spec: ...}} |
One level of template: too many |
containers: {job: {...}} |
containers must be a list |
containers: [{resources: ...}] |
The container has no name |
command: ["sh", "-c", "echo ${FOO}"] |
Write $${FOO} |
A jobPodTemplates entry sets only limits: {memory: 2Gi} for job |
The request still comes from jobPodTemplate or scalability (4Gi by default) and exceeds the limit: set requests too |
scalability sets only memoryLimit: 2Gi |
Without memory in the templates, the request is memoryRequest (4Gi by default) and exceeds the limit: lower memoryRequest too |
Fields inside spec are not checked. A wrong field type, a malformed quantity or a missing RuntimeClass is rejected by the API server when the runner creates the job pod, and the job fails with create pod: ....
Concurrency and scaling
Section titled “Concurrency and scaling”kubernetesRuntime: capacity: 4 # jobs per runner pod, 4 by defaultscalability: minSize: 1 # runner pods maxSize: 5 # runner podsminSizeandmaxSizecount runner pods, so up tomaxSize × capacityjobs run at once.- Waiting jobs first take the free places of the running runner pods, and the controller adds runner pods for the rest. For example, with
capacity: 4, two runner pods that run 3 jobs between them, and 9 waiting jobs, 5 jobs fit on the existing pods and the controller adds one runner pod for the other 4. - It scales down one runner pod at a time, starting with the highest number, and only one that runs no job, as described in Scaling.
Draining and shutdown
Section titled “Draining and shutdown”When a change has to be applied to a runner pod that runs jobs, the pod is drained: it takes no new jobs, finishes the ones it has, and is then rebuilt. One runner pod of a Runner drains at a time. See How configuration changes are applied.
kubernetesRuntime: shutdownTimeout: 3h # 3h by default # terminationGracePeriodSeconds: 11100 # shutdownTimeout + 5m by defaultThe default shutdownTimeout of 3 hours matches the job timeout of the runner, so draining cancels no job. As a trade-off, node maintenance, such as kubectl drain, can wait up to 3 hours and 5 minutes for a runner pod to finish. Lower it for faster maintenance: jobs still running at the timeout are cancelled.
Using your own runner configuration
Section titled “Using your own runner configuration”kubernetesRuntime.runnerConfigMapName works as described in Using your own runner configuration, and in addition:
runner.labelsmust be in the form<name>:kubernetes://<image>, with the same names askubernetesRuntime.labels.- Keep
runner.capacityequal tokubernetesRuntime.capacity, which the controller uses to plan scaling. - Set
host.workdir_parent: /cache/act_workdir, and with the cache enabled acache.dirunder/cache. Otherwise both default to the home directory of the runner, which is not writable for its user, and jobs fail to download actions. - Do not set
kubernetes.namespaceorkubernetes.kubeconfig: job pods must be created in the namespace of theRunner. Without them, the runner uses its ServiceAccount and namespace. jobPodTemplateandjobPodTemplateshave no effect, and the controller adds neither thescalabilityresources nor thekarpenter.sh/do-not-disruptannotation to the job pods. Set them inkubernetes.pod_templateof the configuration instead.- To allow the cache traffic with a NetworkPolicy, set a fixed
cache.port. The generated configuration uses 8088.
The controller still manages the ServiceAccount, the RoleBinding and the security context of the runner pods.
Using both backends
Section titled “Using both backends”Workflows that need Docker can run on a second Runner with the Docker-in-Docker backend. As that one needs privileged pods, put it in a namespace of its own that allows them, so the namespace of the Kubernetes backend keeps a stricter level. The token Secret must be in that namespace too:
kubectl create namespace gitea-docker-runnerskubectl label namespace gitea-docker-runners pod-security.kubernetes.io/enforce=privilegedkubectl -n gitea-docker-runners create secret generic runner-registration-token \ --from-literal=token=<registration token>apiVersion: gitea.com/v1kind: Runnermetadata: name: docker-runner namespace: gitea-docker-runnersspec: gitea: instanceUrl: "https://<gitea host>" registrationTokenSecretRef: name: runner-registration-token key: token runtime: image: gitea/runner:6.0.0-dind labels: ubuntu-latest-docker: docker://docker.gitea.com/runner-images:ubuntu-latest storage: className: "<storage class>"Workflows that need Docker use runs-on: ubuntu-latest-docker. Give the two Runner resources different labels: with the same label, a waiting job scales up both.
Switching from Docker-in-Docker
Section titled “Switching from Docker-in-Docker”To move workflows from the Docker-in-Docker backend to this one, create a new Runner with this backend, in a namespace of its own, next to the existing one. Once the workflows run on the new Runner, delete the existing one.
The two backends do not take the same options: label values lose their docker:// prefix, the scalability resources apply to the job containers, and options such as pvcMounts and jobTmpfs have no equivalent. Write the new Runner from the reference rather than by copying the existing one.