Team & Onboarding Best Practices

The headline rule: no production kubectl write access for developers without training wheels.

Everything below is how you make that rule feel like enablement rather than obstruction.

How to Use This List

Treat each group as a checkpoint in the onboarding arc, from day one through production ownership.

Adopt them in order. Groups A and B pay for themselves in the first week; group E only matters once teams are shipping.

Each practice is a default, not a law. Where you deviate, write down why, because undocumented exceptions become the new default within a quarter.

A - Day One

• Give real access on day one, scoped to one namespace. A new engineer with get, list, watch, and logs in their own dev namespace learns more in an hour than a week of documentation. Access is not the risk; scope is.
• Ship a working kubeconfig, not instructions to build one. OIDC login through your identity provider, one command, in the onboarding script. Nobody's first Kubernetes memory should be a certificate error.
• Deploy something on day one. The golden-path template, one image, one kubectl apply or one PR, and a URL that returns 200. The feedback loop is the lesson.
• Explain the Docker/containerd split immediately. Docker is for building images and Compose for local dev; containerd via the CRI runs the Pods on nodes. Getting this wrong on day one produces months of misdirected debugging.
• Point at one runbook, not a wiki. A single page with the ten commands they will actually use beats a hundred pages nobody maintains.

B - Access & Blast Radius

• Default to namespace-scoped Roles. Role plus RoleBinding bounds a mistake to one tenant. Reserve ClusterRole and ClusterRoleBinding for the platform team and controllers.
• Never grant standing cluster-admin to a human. Use a just-in-time break-glass path that expires and is audited. If break-glass is used weekly, your normal permissions are wrong.
• No direct write access to production. Production changes go through a PR that Argo CD or Flux reconciles, so the GitOps controller's service account is the only prod writer. This is the training wheels.
• Grant read and logs in production freely. Debugging without visibility is what drives people to ask for write access. Reads are cheap; pods/exec in prod is not.
• Verify RBAC from the user's seat. kubectl auth can-i --as-group is the only proof that a Role does what the diagram claims. Test the denials, not just the grants.
• Enforce Pod Security Standards at restricted on every tenant namespace. RBAC decides who may create a Pod; admission decides what a Pod may be. You need both.

C - The Paved Path

• Make the safe way the easy way. A golden-path Helm chart or Kustomize base that already carries probes, requests, non-root securityContext, and a PDB is the highest-leverage artifact a platform team owns.
• Version and migrate the template. Existing manifests do not update themselves. Announce, automate what you can, and pair on the rest.
• Write the self-service contract down. A table of what teams change themselves versus what needs a platform ticket answers the most-asked question in ten seconds. Keep the ticket column short.
• Expose shared resources as constrained primitives. Teams own their HTTPRoute; the platform owns the Gateway. Ownership without the ability to break the front door.
• Keep quotas outside the tenant's write scope. A team that can edit its own ResourceQuota does not have a quota. Same reasoning for PriorityClass.

D - Review & Judgment Transfer

• Automate every mechanical check before asking a human. Schema validation, Kyverno or Gatekeeper policy, and a rendered diff run in CI. Human review is for semantics.
• Pair on the first several manifest PRs. Read the diff together for probes, resources, and disruption. Then graduate the team to spot checks - pairing that never ends is a template failure.
• Comment with the failure mode, not the rule. "This liveness probe restarts every replica during a database blip" teaches; "set failureThreshold: 3" does not.
• Promote your most repeated comment into policy. If you have written "add resource requests" five times, it is a LimitRange default and a Kyverno rule, not a lesson.
• Ship new policy in audit mode first. Measure what it would have blocked, tell the affected teams, then enforce. Policy that breaks Friday deploys does not survive the retro.

E - Growing the Team

• Score competencies against observed behavior. A skills matrix works when it reflects what someone has done under pressure, not what they have read. Re-score twice a year, not quarterly.
• Close gaps with real work plus a reviewer. The path from mid to senior runs through owning an upgrade or an incident, not through a course.
• Watch the bus factor per competency, not per person. One engineer who understands the CNI is a platform risk regardless of how strong the team looks on average.
• Run blameless postmortems that end in platform changes. An incident that produces only a wiki edit will recur. It should produce a template default, a policy rule, or a guardrail.
• Treat the platform as a product with users. Measure adoption and friction, and sunset things deliberately. Teams route around platforms that are slower than the alternative.
• Let developers graduate visibly. Read-only, then dev write, then staging write, then prod PR approval rights. A visible ladder converts "you don't trust me" into "here is the next rung."

When You Are Done

A new engineer should reach a deployed, reachable workload on day one, using their own credentials, without a platform engineer typing anything.

Any developer should be able to answer "can I change this myself?" from a table, not from Slack.

Nobody should hold standing write access to production, and nobody should feel blocked by that - because the PR path is fast and reads are open.

Your most common review comment from six months ago should now be enforced automatically and never typed again.

And when a developer makes their worst plausible mistake, it should stay inside their namespace, hit a quota, and page nobody.

FAQs

Is "no prod kubectl" realistic for a small team? The write half is, from about five engineers onward, because GitOps costs little once configured. Keep reads open regardless.

What if an incident needs immediate prod changes? That is what break-glass is for - just-in-time elevation, audited, expiring. It should be rare enough to notice and reliable enough to trust.

How long should onboarding to production ownership take? Two to six weeks for an experienced engineer new to your platform, most of it spent in staging with paired reviews.

Do we need a service mesh to do this well? No. Namespaces, RBAC, Pod Security Standards, quotas, and default-deny NetworkPolicies cover the boundaries. A mesh adds identity and traffic control on top.

What is the single highest-leverage thing to build first? The golden-path template. Every other guardrail gets cheaper once the default manifest is already correct.

How do we avoid becoming a ticket queue? Watch the platform column of your contract table and treat every recurring ticket type as a self-service primitive waiting to be built.

Related

• Making App Devs Cluster-Safe
• Onboarding Basics
• Developer Self-Service
• Skills Matrix
• Pairing on Manifest PRs

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).