Reference: Multi-Tenant EKS Platform

Summary

• A worked example of one EKS cluster shared by ~12 product teams and ~200 services, isolated by namespace with quotas, RBAC, NetworkPolicy, and GitOps.
• Insight: Soft multi-tenancy works when the namespace is the unit of everything - quota, policy, RBAC, network, cost, and the GitOps sync target. The moment those five boundaries stop lining up, tenants leak into each other.
• Key Concepts: the tenant namespace, ResourceQuota + LimitRange, default-deny NetworkPolicy, Argo CD ApplicationSet, and a shared observability baseline.
• When to Use: trusted-but-independent internal teams on one cloud account, where per-team clusters would be too costly to operate.
• Limitations: namespaces are not a security boundary against hostile tenants. Untrusted code needs separate node pools with a sandboxed runtime, or separate clusters entirely.

Recipe

The shortest path to this platform is five decisions, applied in order.

1. One namespace per team, created only by the platform repo - never by hand.
2. Every namespace gets a ResourceQuota, a LimitRange, and a default-deny NetworkPolicy at creation.
3. Pod Security Standards restricted enforced by namespace label, with baseline allowed only by documented exception.
4. Argo CD owns the cluster: an ApplicationSet generates one Application per team from the tenants directory in Git.
5. A shared observability stack (Prometheus, Grafana, OpenTelemetry Collector) that tenants consume but do not deploy.

Working Example

The tenant namespace is the whole design in one file. The platform repo renders this from a template for every team.

apiVersion: v1
kind: Namespace
metadata:
  name: team-payments
  labels:
    # Pod Security Standards, enforced by the built-in admission controller
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.36
    pod-security.kubernetes.io/warn: restricted
    tenant: payments
    cost-center: "4412"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: team-payments
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.memory: 160Gi
    pods: "150"
    count/services.loadbalancers: "2"
    persistentvolumeclaims: "20"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-defaults
  namespace: team-payments
spec:
  limits:
    - type: Container
      default: { memory: 256Mi }
      defaultRequest: { cpu: 100m, memory: 128Mi }
      max: { cpu: "4", memory: 8Gi }

The LimitRange matters more than it looks. A ResourceQuota on requests.cpu rejects any pod without a CPU request, so without defaults every tenant's first deploy fails with a confusing error.

Network isolation is default-deny per namespace, then explicitly opened.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: team-payments
spec:
  podSelector: {}
  policyTypes: ["Ingress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-gateway
  namespace: team-payments
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: gateway-system }
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

GitOps binds it together. One ApplicationSet replaces twelve hand-maintained Applications.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: tenants
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/acme/platform.git
        revision: main
        directories:
          - path: tenants/*
  template:
    metadata:
      name: "{{path.basename}}"
    spec:
      project: tenants
      source:
        repoURL: https://github.com/acme/platform.git
        targetRevision: main
        path: "{{path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{path.basename}}"
      syncPolicy:
        automated: { prune: true, selfHeal: true }

Deep Dive

Why namespaces and not clusters

Twelve clusters means twelve upgrade cycles, twelve CNI versions, and twelve observability stacks. For teams that already trust each other inside one company, that overhead buys isolation they do not need.

The trade is explicit: you accept that a namespace is a policy boundary, not a kernel boundary.

RBAC scoping

Each team gets a RoleBinding in its own namespace, never a ClusterRoleBinding. Platform engineers hold cluster-admin through a break-glass role that is audited.

Tenants must not be able to create ClusterRole, ValidatingWebhookConfiguration, or PersistentVolume objects - all three are cluster-scoped escape hatches.

Node pools and scheduling

Most tenants share a general-purpose Karpenter NodePool. Two teams with latency SLOs get a dedicated NodePool selected by nodeSelector, with taints so nothing else lands there.

This is the one place where the namespace boundary is deliberately reinforced with a hardware boundary.

Observability baseline

The platform runs Prometheus and an OpenTelemetry Collector as a DaemonSet. Tenants emit OTLP to the node-local collector and get dashboards for free.

Critically, tenants cannot deploy their own Prometheus - a quota on pods alone will not stop a team from doubling cluster scrape load. Admission policy blocks it.

Cost attribution

The cost-center namespace label flows into Kubecost or AWS split-cost allocation. Without a label enforced at namespace creation, showback is guesswork.

Gotchas

Quota without LimitRange breaks every first deploy. Ship both together, always.

selfHeal: true fights kubectl edit. That is the point, but tell tenants before they lose an afternoon debugging a reverted change.

Default-deny egress breaks DNS silently. Pods hang for 5 seconds then fail resolution. Always pair default-deny with an explicit kube-dns egress rule.

PSS restricted rejects most public images. Anything that runs as root or writes to / fails admission. Budget real time for base-image remediation before enforcing.

One tenant's LoadBalancer sprawl hits cloud quotas, not cluster quotas. Use count/services.loadbalancers in the quota and route through a shared Gateway instead.

Noisy neighbors on memory are real. CPU is compressible, memory is not. A tenant without memory limits can trigger node-level OOM kills that hit unrelated pods.

Alternatives

Cluster per team. Strongest isolation, highest operational cost. Pick it when tenants are external customers or when compliance demands separate control planes.

vCluster / virtual control planes. Each tenant gets an apparent API server on shared nodes. Good when teams need CRDs or cluster-scoped objects without real cluster cost.

Account-per-team with EKS in each. The AWS-native blast-radius answer. Correct for regulated or acquisition-heavy orgs, painful for shared platform tooling.

Single namespace, prefix conventions. Only defensible below roughly five services and one team. It collapses as soon as RBAC matters.

FAQs

Is a namespace a security boundary? Not against hostile code. It is a strong policy and RBAC boundary, but tenants still share a kernel per node unless you split node pools.

How many tenants can one EKS cluster hold? Practically, watch API server load and etcd object count, not team count. Hundreds of namespaces are fine; tens of thousands of objects churning per minute is not.

Where do CRDs go? The platform owns them. Cluster-scoped CRDs cannot be safely delegated to tenants in soft multi-tenancy.

Do tenants get their own Argo CD? No. One Argo CD with an AppProject per tenant restricts what each can deploy and where.

How do we onboard a new team? A pull request adding tenants/team-x/ to the platform repo. The ApplicationSet does the rest, which is exactly the golden path this design is selling.

Related

• How to Read a Reference Architecture
• Reference: Worker-Heavy Cluster
• Reference: Regulated Environment
• Before/After: Compose to EKS Migration
• Case Studies Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).