Scaling cert-manager

Learn how to optimize cert-manager for your cluster.

Overview

The defaults in the Helm chart and YAML manifests are intended for general use. You may want to modify the configuration to suit the size and usage of your Kubernetes cluster.

Set appropriate memory requests and limits

When Certificate resources are the dominant use-case, such as when workloads need to mount the TLS Secret or when gateway-shim is used, the memory consumption of the cert-manager controller will be roughly proportional to the total size of those Secret resources that contain the TLS key pairs. Why? Because the cert-manager controller caches the entire content of these Secret resources in memory. If large TLS keys are used (e.g. RSA 4096) the memory use will be higher than if smaller TLS keys are used (e.g. ECDSA).

The other Secrets in the cluster, such as those used for Helm chart configurations or for other workloads, will not significantly increase the memory consumption, because cert-manager will only cache the metadata of these Secrets.

When CertificateRequest resources are the dominant use-case, such as with csi-driver or with istio-csr, the memory consumption of the cert-manager controller will be much lower, because there will be fewer TLS Secrets and fewer resources to be cached.

📖️ Read What Everyone Should Know About Kubernetes Memory Limits, to learn how to right-size the memory requests.

Disable client-side rate limiting for Kubernetes API requests

By default cert-manager throttles the rate of requests to the Kubernetes API server to 20 queries per second. Historically this was intended to prevent cert-manager from overwhelming the Kubernetes API server, but modern versions of Kubernetes implement API Priority and Fairness, which obviates the need for client side throttling. You can increase the threshold of the client-side rate limiter using the following helm values:

# helm-values.yaml
config:
  apiVersion: controller.config.cert-manager.io/v1alpha1
  kind: ControllerConfiguration
  kubernetesAPIQPS: 10000
  kubernetesAPIBurst: 10000

ℹ️ This does not technically disable the client-side rate-limiting but configures the QPS and Burst values high enough that they are never reached.

🔗 Read cert-manager#6890: Allow client-side rate-limiting to be disabled; a proposal for a cert-manager configuration option to disable client-side rate-limiting.

🔗 Read kubernetes#111880: Disable client-side rate-limiting when AP&F is enabled; a proposal that the kubernetes.io/client-go module should automatically use server-side rate-limiting when it is enabled.

🔗 Read about other projects that disable client-side rate limiting: Flux.

📖 Read API documentation for ControllerConfiguration for a description of the kubernetesAPIQPS and kubernetesAPIBurst configuration options.

Restrict the use of large RSA keys

Certificates with large RSA keys cause cert-manager to use more CPU resources. When there are insufficient CPU resources, the reconcile queue length grows, which delays the reconciliation of all Certificates. A user who has permission to create a large number of RSA 4096 certificates, might accidentally or maliciously cause a denial of service for other users on the cluster.

📖 Learn how to enforce an Approval Policy, to prevent the use of large RSA keys.

📖 Learn how to set Certificate defaults automatically, using tools like Kyverno.

Set revisionHistoryLimit: 1 on all Certificate resources

By default, cert-manager will keep all the CertificateRequest resources that it creates (revisionHistoryLimit):

The maximum number of CertificateRequest revisions that are maintained in the Certificate's history. Each revision represents a single CertificateRequest created by this Certificate, either when it was created, renewed, or Spec was changed. Revisions will be removed by oldest first if the number of revisions exceeds this number. If set, revisionHistoryLimit must be a value of 1 or greater. If unset (nil), revisions will not be garbage collected. Default value is nil.

On a busy cluster these will eventually overwhelm your Kubernetes API server; because of the memory and CPU required to cache them all and the storage required to save them.

Use a tool like Kyverno to override the Certificate.spec.revisionHistoryLimit for all namespaces.

📖 Adapt the Kyverno policies in the tutorial: how to set Certificate defaults automatically, to override rather than default the revisionHistoryLimit field.

📖 Learn how to set revisionHistoryLimit when using Annotated Ingress resources.

🔗 Read cert-manager#3958: Sane defaults for Certificate revision history limit; a proposal to change the default revisionHistoryLimit, which will obviate this particular recommendation.

Enable Server-Side Apply

By default, cert-manager uses Update requests to create and modify resources like CertificateRequest and Secret, but on a busy cluster there will be frequent conflicts as the control loops in cert-manager each try to update the status of various resources.

You will see errors, like this one, in the logs:

I0419 14:11:51.325377 1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-trigger" key="team-864-p6ts6/app-7" error="Operation cannot be fulfilled on certificates.cert-manager.io \"app-7\": the object has been modified; please apply your changes to the latest version and try again"

This error is relatively harmless because the update attempt is retried, but it slows down the reconciliation because each error triggers an exponential back off mechanism, which causes increasing delays between retries.

The solution is to turn on the Server-Side Apply Feature, which causes cert-manager to use HTTP PATCH using Server-Side Apply when ever it needs to modify an API resource. This avoids all conflicts because each cert-manager controller sets only the fields that it owns.

You can enable the server-side apply feature gate with the following Helm chart values:

# helm-values.yaml
config:
  apiVersion: controller.config.cert-manager.io/v1alpha1
  kind: ControllerConfiguration
  featureGates:
    ServerSideApply: true

📖 Read Using Server-Side Apply in a controller, to learn about the advantages of server-side apply for software like cert-manager.