Platform Tools Best Practices

A platform's tooling is a product, and like any product it needs a supported surface, a deprecation policy, and documentation.

These practices cover how to choose, install, secure, and retire the controllers and CLIs your cluster depends on.

How to Use This List

Treat each group as a review gate rather than a checklist to complete in one sitting.

Run groups A and B when you are introducing a new tool, and groups C through E during a quarterly platform review.

If a practice conflicts with your organization's constraints, write down why - an explicit exception beats silent drift.

A - Choose and Bless Tools Deliberately

Pick tools that are CNCF-graded or vendor-supported. Velero, cert-manager, and external-dns all have active maintainers and security processes. A clever one-off controller from a personal GitHub account becomes your team's maintenance burden the day its author changes jobs.

Publish a blessed-tools list. One page naming each tool, its version, its owner, and its purpose eliminates most "which thing do I use for TLS?" questions. Without it, three teams install three ingress controllers.

Prefer one tool per job. Two controllers writing the same DNS zone or the same Secret will fight, and the loser is whoever is on call. Overlap is a bug, not redundancy.

Evaluate the CRD surface before installing. Every tool that ships CRDs is claiming a piece of your API server permanently, and CRDs outlive Helm releases. Ask what happens on uninstall before you install.

Reject tools that need cluster-admin without explanation. A controller that watches Ingresses does not need write access to Secrets in every namespace. If the chart's default RBAC is a wildcard, that is a finding.

B - Install Through Declarative, Reviewable Config

Install platform tools with Helm or Kustomize through GitOps, never kubectl apply from a laptop. An Argo CD Application makes the installed version visible and the upgrade reviewable. A one-off apply is invisible until it breaks.

Pin chart and image versions explicitly. latest in a platform controller means an unreviewed upgrade during an unrelated sync. Pin the chart version, and pin plugin images like velero/velero-plugin-for-aws to a tag.

Manage CRDs deliberately. Decide whether Helm owns them (crds.enabled=true) or whether they are applied separately, and record the choice. Ambiguity here is how a helm uninstall deletes every Certificate in the cluster.

Keep values files in the same repo as the rest of the platform. Tool configuration is infrastructure, and it deserves the same review, history, and rollback as application manifests.

Stage every platform tool in a non-production cluster first. cert-manager against Let's Encrypt staging and external-dns with --dry-run cost nothing and catch the misconfigurations that burn rate limits.

C - Scope Permissions and Blast Radius

Use workload identity, not static cloud credentials. IRSA on EKS, Workload Identity on GKE, and their AKS equivalent remove long-lived keys from Secrets. A leaked Route53 key is a domain takeover.

Scope cloud IAM to specific resources. external-dns needs route53:ChangeResourceRecordSets on one hosted zone ARN, not *. Velero needs its own bucket, not the account's object storage.

Give every cluster a unique identity in shared systems. txtOwnerId for external-dns and a distinct bucket prefix for Velero prevent two clusters from garbage-collecting each other's state.

Start destructive controllers in a safe mode. external-dns --policy=upsert-only and Velero schedules with a generous ttl let you observe behavior before you allow deletion.

Run platform controllers in dedicated namespaces with restricted Pod Security Standards. They are privileged in effect, so treat them as workloads that need NetworkPolicy and admission constraints like any other.

Constrain the reach of anything that mutates admission. A failing cert-manager webhook blocks its own CRDs, so understand what your tools make undeployable when they are down.

D - Make Debugging Repeatable, Not Heroic

Standardize on k9s and stern for triage, and document the workflow. Interactive tools are excellent for humans under pressure and terrible as pipeline dependencies. Say which is which.

Manage kubectl plugins with krew and check the list into the repo. A shared plugin manifest means everyone's kubectl neat, kubectl tree, and kubectl cnpg behave the same during an incident.

Never build automation on interactive tools. If a runbook step is "open k9s and press s", it cannot be tested, reviewed, or run by a script. Turn it into a kubectl command in a repo.

Write runbooks that name exact commands. velero backup logs <name> and kubectl describe challenge are the actual diagnostic steps; "check cert-manager" is not a runbook.

Ban one-off kubectl edit in production. A live edit is reverted by the next GitOps sync, which makes the fix look flaky rather than wrong. Change the source, then sync.

E - Verify, Monitor, and Retire

Test restores on a schedule, not after an incident. A Velero backup marked Completed proves an upload succeeded and nothing else. Restore into a scratch namespace quarterly and time it.

Alert on the tools, not just through them. Certificate expiry, external-dns reconciliation errors, and failed backup schedules all deserve alerts. Silent controllers fail silently.

Scrape the controller metrics you already have. cert-manager, external-dns, and Velero all export Prometheus metrics; wiring them into existing dashboards costs one ServiceMonitor each.

Track upstream Kubernetes version compatibility. Platform controllers pin supported API versions, and a cluster upgrade to 1.36.2 can strand a tool two minors behind. Check the compatibility matrix before the upgrade, not during.

Retire tools explicitly. Announce the deprecation, migrate the consumers, remove the CRDs last, and confirm nothing references them. A half-removed controller with live CRDs is worse than either state.

Review the blessed-tools list every quarter. Tools accumulate, ownership decays, and the list is only useful if it reflects what is actually installed. Diff it against the cluster.

When You Are Done

You should be able to name every controller running in your platform namespaces, its owner, and its pinned version.

Each one should install from Git, hold scoped credentials, export metrics, and have a documented failure mode.

Anything that fails those checks is either a candidate for hardening or a candidate for removal.

FAQs

How many platform tools is too many? The number you cannot staff. If nobody can name the owner of a controller, it is already too many.

Should application teams install their own controllers? Rarely. Cluster-scoped CRDs and webhooks are shared state, so they belong to the platform team even when one team requested them.

Is Helm or Kustomize better for platform tools? Helm, usually, because upstream projects ship charts and pin their own CRDs. Use Kustomize to patch the rendered output when the chart lacks a value you need.

Do we need GitOps for tools we install once? Yes, especially those. The install you never touch is the one nobody remembers configuring when it breaks two years later.

How do we handle a tool that needs cluster-admin? Read its RBAC, understand why, and if the answer is "the chart was lazy," open an upstream issue and scope it yourself in the meantime.

What belongs in the platform namespace versus a per-tool namespace? Give each tool its own namespace. It keeps RBAC, NetworkPolicy, and resource quotas separable, and it makes uninstall a clean operation.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Platform Tools Best Practices

A platform's tooling is a product, and like any product it needs a supported surface, a deprecation policy, and documentation.

These practices cover how to choose, install, secure, and retire the controllers and CLIs your cluster depends on.

How to Use This List

Treat each group as a review gate rather than a checklist to complete in one sitting.

Run groups A and B when you are introducing a new tool, and groups C through E during a quarterly platform review.

If a practice conflicts with your organization's constraints, write down why - an explicit exception beats silent drift.

A - Choose and Bless Tools Deliberately

Prefer one tool per job. Two controllers writing the same DNS zone or the same Secret will fight, and the loser is whoever is on call. Overlap is a bug, not redundancy.

B - Install Through Declarative, Reviewable Config

Keep values files in the same repo as the rest of the platform. Tool configuration is infrastructure, and it deserves the same review, history, and rollback as application manifests.

C - Scope Permissions and Blast Radius

Scope cloud IAM to specific resources. external-dns needs route53:ChangeResourceRecordSets on one hosted zone ARN, not *. Velero needs its own bucket, not the account's object storage.

Give every cluster a unique identity in shared systems. txtOwnerId for external-dns and a distinct bucket prefix for Velero prevent two clusters from garbage-collecting each other's state.

Start destructive controllers in a safe mode. external-dns --policy=upsert-only and Velero schedules with a generous ttl let you observe behavior before you allow deletion.

Constrain the reach of anything that mutates admission. A failing cert-manager webhook blocks its own CRDs, so understand what your tools make undeployable when they are down.

D - Make Debugging Repeatable, Not Heroic

Standardize on k9s and stern for triage, and document the workflow. Interactive tools are excellent for humans under pressure and terrible as pipeline dependencies. Say which is which.

Manage kubectl plugins with krew and check the list into the repo. A shared plugin manifest means everyone's kubectl neat, kubectl tree, and kubectl cnpg behave the same during an incident.

Never build automation on interactive tools. If a runbook step is "open k9s and press s", it cannot be tested, reviewed, or run by a script. Turn it into a kubectl command in a repo.

Write runbooks that name exact commands. velero backup logs <name> and kubectl describe challenge are the actual diagnostic steps; "check cert-manager" is not a runbook.

Ban one-off kubectl edit in production. A live edit is reverted by the next GitOps sync, which makes the fix look flaky rather than wrong. Change the source, then sync.

E - Verify, Monitor, and Retire

Test restores on a schedule, not after an incident. A Velero backup marked Completed proves an upload succeeded and nothing else. Restore into a scratch namespace quarterly and time it.

Alert on the tools, not just through them. Certificate expiry, external-dns reconciliation errors, and failed backup schedules all deserve alerts. Silent controllers fail silently.

Scrape the controller metrics you already have. cert-manager, external-dns, and Velero all export Prometheus metrics; wiring them into existing dashboards costs one ServiceMonitor each.

Review the blessed-tools list every quarter. Tools accumulate, ownership decays, and the list is only useful if it reflects what is actually installed. Diff it against the cluster.

When You Are Done

You should be able to name every controller running in your platform namespaces, its owner, and its pinned version.

Each one should install from Git, hold scoped credentials, export metrics, and have a documented failure mode.

Anything that fails those checks is either a candidate for hardening or a candidate for removal.

FAQs

How many platform tools is too many? The number you cannot staff. If nobody can name the owner of a controller, it is already too many.

Should application teams install their own controllers? Rarely. Cluster-scoped CRDs and webhooks are shared state, so they belong to the platform team even when one team requested them.

Do we need GitOps for tools we install once? Yes, especially those. The install you never touch is the one nobody remembers configuring when it breaks two years later.

How do we handle a tool that needs cluster-admin? Read its RBAC, understand why, and if the answer is "the chart was lazy," open an upstream issue and scope it yourself in the meantime.

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).

Platform Tools Best Practices

Buscar en la documentación

Platform Tools Best Practices