Storage and Caching
Runner pods keep their caches on persistent volumes, which outlive the pods: when a runner pod is removed its volumes are kept, and when the runner pod is recreated it mounts them again, with the images and actions it downloaded before. This page describes those volumes, how long they live, and the workflow cache.
Storage requirements
Section titled “Storage requirements”- Dynamic provisioning of
ReadWriteOncevolumes. The volumes are created as runner pods are added, so a StorageClass must provision them on demand. Volumes created in advance by hand are not supported. allowVolumeExpansionon the StorageClass is recommended, so that the volumes can grow in place later, see Changing StorageClass or size.volumeBindingMode: WaitForFirstConsumeris recommended, so that a volume is created where its pod is scheduled.- Volumes that can be attached on another node. A runner pod recreated after a scale-down mounts its previous volumes, on whichever node it is scheduled. Node-local provisioners such as
local-path-provisionerpin each runner to one node, and the runner cannot start again once that node is gone. Use them for single-node trials only. - Topology. Volumes bound to a zone, or to a subset of the nodes, keep their runner pods there. If the cluster adds nodes on demand, the component that does so must take the topology of the volumes into account, or pods stay
Pendingwithvolume node affinity conflict. If it cannot, keep the runner pods in one zone withnodeSelectororaffinity.
Each backend adds its own requirements: the Docker data root of the Docker-in-Docker backend must be on block storage, and the volumes of the Kubernetes backend must support fsGroup.
Check the StorageClass with:
kubectl get storageclassALLOWVOLUMEEXPANSION should preferably be true, and VOLUMEBINDINGMODE WaitForFirstConsumer.
Volumes of a runner pod
Section titled “Volumes of a runner pod”| Volume | Claim name | Mounted at | Holds | Size | StorageClass |
|---|---|---|---|---|---|
| Docker data root (Docker-in-Docker only) | <runner pod>-cache |
dockerStorageMountPath, /var/lib/docker by default |
Images and layers pulled by Docker, job workspaces | cacheVolumeSize, 100Gi |
dockerStorageClassName, else className |
| Action cache | <runner pod>-action-cache |
/cache |
The code of the actions jobs use, and the staging area of S3 caching | actionCacheVolumeSize, 10Gi |
actionCacheClassName, else className |
The action cache holds the actions that jobs use with uses:, so they are not downloaded again for every job. The workflow cache is separate, see Workflow cache.
className is the StorageClass named standard by default, not the default StorageClass of the cluster. Set it, or the more specific fields, to a StorageClass that exists: without one, the claims stay Pending.
With the Docker-in-Docker backend, everything a job container writes, such as its workspace and the tools that setup-* actions install, is written to the Docker data root. The action cache only holds what the runner itself downloads, which is light on I/O. Therefore, the action cache can be on network storage, while the Docker data root should be on fast block storage.
Capacity planning
Section titled “Capacity planning”Plan the storage for the peak, not the average:
total ≈ maxSize × (Docker data root + action cache) + cache server- The default sizes are large. The Docker data root is 100Gi for each runner pod and the cache server 500Gi. Size them to the jobs.
- Volumes are not freed when scaling down. The volumes of a runner are kept for
storage.cacheTTLafter their last use, a week by default. ARunnerthat once scaled tomaxSizekeeps that storage for a week. - With S3 caching there is no cache server volume, but the action cache also stages each cache before upload.
Volume lifetime
Section titled “Volume lifetime”storage: cacheTTL: 168h # deleted this long after the last use, 168h by default cacheMaxAge: 720h # recreated empty this long after creation, 720h by default- The volumes of a runner are deleted once they have not been used for
cacheTTL. While its runner pod runs, they are in use and kept. The volume of the built-in cache server followscacheTTLtoo, and is deleted once the cache server has been unused for that long, for example after scaling to 0 or switching to another cache. cacheMaxAgecaps the total age of the volumes of a runner pod. A volume that reaches it is deleted and created again empty, even if the runner is always in use, like the runners kept byminSize. Such a runner pod is rebuilt with the new volume once it is idle, one runner pod of aRunnerat a time, with aCachePVCExpiredevent. A largercacheTTLtherefore does not keep a cache forever. The cache server volume is not affected, as the cache server expires its own entries.- The registrations of the runners are kept in Secrets that do not expire.
- Two cases delete the volumes of a runner immediately. Lowering
maxSizedeletes those of the runners above it, along with their registrations. A runner pod that does not come online is deleted with its volumes, see Runner pods that do not come online.
Changing StorageClass or size
Section titled “Changing StorageClass or size”The StorageClass of a volume cannot be changed, and Kubernetes cannot shrink a volume. To avoid emptying caches by accident, the controller only applies such a change when storage.allowDeleteCachePVC is true:
| Change | allowDeleteCachePVC: false (default) |
allowDeleteCachePVC: true |
|---|---|---|
| Larger size, StorageClass allows expansion | Expanded in place (CachePVCExpand) |
Expanded in place (CachePVCExpand) |
| Larger size, StorageClass does not allow expansion | Ignored, CachePVCExpandUnsupported warning |
Volume deleted and created again, empty (CachePVCRecreate) |
| Smaller size | Ignored, CachePVCShrinkIgnored warning |
Volume deleted and created again, empty (CachePVCRecreate) |
| Different StorageClass | Ignored, CachePVCClassChangeIgnored warning |
Volume deleted and created again, empty (CachePVCRecreate) |
The volumes of a runner pod are recreated once it is idle, as for any configuration change. The volume of the built-in cache server is recreated immediately, together with its pod.
Reclaim policy
Section titled “Reclaim policy”When a volume is deleted, the StorageClass decides whether the storage behind it is freed. With a Retain policy, deleted volumes leave Released persistent volumes behind. To free them, set:
storage: reclaimPolicy: Delete # or Retain; unset keeps the policy of the StorageClassThe controller then sets this policy on the persistent volumes provisioned for the Runner, but only while a volume is bound and in use by a runner pod or the cache server. The volumes of runners already scaled down are not in use, and keep the policy of the StorageClass when the Runner is deleted. Therefore, set it when you create the Runner, not just before deleting it. Statically provisioned volumes are never changed.
Workflow cache
Section titled “Workflow cache”The workflow cache, which actions/cache and actions such as setup-go and setup-node use, needs a cache server. There are three ways to provide it, and S3-compatible storage is recommended for production:
- The cache is kept in a bucket, so neither the pod of the built-in cache server nor its volume, 500Gi by default, is needed.
- Capacity grows with the bucket, without planning or resizing a cache server volume.
- Several
Runnerresources can share a cache by using the same prefix, while the built-in cache server serves a singleRunner.
Read the notes in S3-compatible storage before using it. The built-in cache server needs no configuration, which suits a trial.
| S3-compatible storage (recommended) | Built-in cache server (default) | External cache server | |
|---|---|---|---|
| Configured with | cacheServer.s3 |
Nothing | cacheServer.externalServerURL |
| Where the cache lives | A bucket | A volume of a pod the controller creates | Your cache server |
The cache traffic of jobs goes through the runner, so the cache server only needs to be reachable from the runner pods.
S3-compatible storage
Section titled “S3-compatible storage”The cache can live in a bucket of an S3-compatible service, such as AWS S3, MinIO or RustFS. The runners then read and write the bucket directly, the controller creates no cache server, and runners with the same bucket and prefix share their cache.
Create a Secret with the access key:
kubectl -n <namespace> create secret generic runner-cache-s3 \ --from-literal=access-key=<access key ID> \ --from-literal=secret-key=<secret access key>cacheServer: s3: endpoint: "https://s3.us-west-2.amazonaws.com" # starts with http:// or https:// region: "us-west-2" # see below bucket: "runner-cache" prefix: "" # <namespace>/<name> of the Runner when empty # pathStyle: true accessKeyRef: name: runner-cache-s3 key: access-key secretKeyRef: name: runner-cache-s3 key: secret-keyregionis required, and signs every request. On AWS it is the region of the bucket. MinIO and RustFS usually useus-east-1, Cloudflare R2 usesauto.pathStylecan usually be left unset: the runner then addresses buckets onamazonaws.comby host name and others by path. Set it totrueon AWS for a bucket name with dots, and tofalsefor a service that only supports host names.- Set lifecycle rules on the bucket. The runners never delete caches, and
storage.cacheTTLdoes not apply to the bucket. Add at least a rule that expires the objects under the prefix after some days, for example 7, and one that aborts incomplete multipart uploads, for example after 1 day. Expiry counts from the creation of an object, so caches in use also expire and are created again on the next run. Unlike the built-in cache server, the bucket keeps every cache saved within the expiry, including those replaced by newer ones, and has no size limit per repository. Keep the expiry short. - Size the action cache for the largest cache. A runner writes each cache in full to its action cache volume before uploading it, and a multipart upload briefly takes about twice that while its parts are combined. The action cache should hold the largest single cache twice, plus the downloaded actions. The default 10Gi fits single caches of up to about 4GiB. When it runs out of space the cache is not saved, and the job is not affected. On network storage, saving caches is noticeably slower.
- Only static access keys are supported, not IRSA, EKS Pod Identity or instance roles. The keys need
s3:ListBucketon the bucket, which a condition can limit to the prefix, ands3:GetObject,s3:PutObjectands3:DeleteObjecton the objects under the prefix.s3:AbortMultipartUploadis optional: it lets a runner clean up a failed upload, which the lifecycle rule does otherwise. Changing the Secret, for example to rotate the keys, rebuilds the runner pods so that they read the new keys. - With the Docker-in-Docker backend, including jobs that run on the host, job code may be able to read the access keys. Grant only the permissions above, limited to the prefix, and restrict the keys to the network of the cluster where the storage service allows it. Use separate keys for separate trust domains, such as
Runnerresources of different Gitea instances. - The runner verifies the certificate of the endpoint with the certificates of its image, or the bundle in
SSL_CERT_FILEas in the init container example.gitea.skipTlsVerifydoes not apply. - Only the runners access the bucket, the job containers do not.
- The default prefix differs per
Runner.Runnerresources with the same prefix share the cache. Caches are told apart by repository name only, not by Gitea instance, soRunnerresources of different Gitea instances should not share a prefix. s3cannot be combined withexternalServerURL.hostandporthave no effect with it.- Caches are not moved when switching from the built-in cache server. Until its volume is deleted after
storage.cacheTTL, removings3switches back to the previous cache.
Built-in cache server
Section titled “Built-in cache server”By default, the controller creates a cache server for each Runner: a pod named <name>-cache-server with a Service, a Secret holding the key the runners use to authenticate, and a volume. The cache is kept on the volume <name>-cache-server-data, of storage.cacheServerVolumeSize, 500Gi by default, and of the StorageClass className. Size it to the cache use of the repositories. With S3, an external cache server or your own runner configuration, neither the cache server nor its volume is created.
cacheServer: # host: cache.example.internal # <name>-cache-server-svc.<namespace>.svc.cluster.local when unset # port: 8081 # 80 by default, 8080 with the Kubernetes backend, where it must be 1024 or above- The runners reach it by the host name of its Service, which assumes the cluster domain
cluster.local. On a cluster with another domain, sethost, or the workflow cache misses every time without failing the job. - While a
Runnerhas no runner pods, the pod and Service of the cache server are removed, and its volume is kept forstorage.cacheTTL. - It runs the runner image, with no resource requests or limits.
- The runners authenticate to it with a key that the controller generates and keeps in the Secret
<name>-cache-server-secret, mounted into the runner pods as a file. - To change the key, delete that Secret. The controller generates a new key, rebuilds the cache server immediately, and then the runner pods as for any configuration change. Until a runner pod is rebuilt, the cache reads and writes of its jobs fail, which jobs treat as a cache miss.
- When its configuration changes, for example the runner image or the port, the cache server pod is rebuilt immediately, and its data stays on the volume. Cache reads and writes in flight fail once, which jobs treat as a cache miss.
- Its volume follows
cacheServerVolumeSizeandclassNamelike the volumes of the runner pods. - After switching to an external cache server, S3 or
runnerConfigMapName, the built-in cache server is kept until every runner pod uses the new configuration, so running jobs are not affected, and is then deleted. Its volume is deleted afterstorage.cacheTTL.
External cache server
Section titled “External cache server”To use a cache server of your own, such as one started with gitea-runner cache-server, set its URL and the shared key. The controller then creates no built-in cache server.
cacheServer: externalServerURL: "http://cache.example.com/" externalSecretRef: name: cache-server-secret key: secretexternalSecretRef reads the key from a Secret in the namespace of the Runner. The deprecated externalSecret takes the key as plain text instead, and is written into the generated configuration. Only one of the two can be set.
- When the Secret or its key is missing or empty, the controller creates no runner pod and reports a
BuildRunnerWorkerFailedwarning that mentionscache server external secret. - Changing the Secret, for example to rotate the key, rebuilds the runner pods so that they read the new key.