Reference: Regulated Environment

Summary

• A worked example of a cluster under external audit (PCI-DSS and SOC 2 in this case), where every workload is non-root, every image is signed, and every API mutation is logged and retained.
• Insight: Compliance is not a security feature you add - it is evidence generation. The architecture's real job is producing an auditable trail that a human can verify a year later.
• Key Concepts: Pod Security Standards restricted, cosign/Sigstore verification at admission, API server audit policy, default-deny networking, and GitOps as the change-control record.
• When to Use: regulated industries, customer-data platforms, or any environment where "who deployed what, when, and who approved it" must be answerable.
• Limitations: developer velocity drops. Every control here adds a gate, and gates cost time. Do not import this design into an unregulated product team.

Recipe

1. Enforce PSS restricted on every workload namespace via namespace labels.
2. Sign images with cosign in CI, and verify signatures at admission before a pod is ever scheduled.
3. Turn on a real API server audit policy and ship logs off-cluster to immutable storage.
4. Default-deny both ingress and egress, then allow explicitly.
5. Make Git the only path to production, with mandatory review, so the PR history is the change-control record.

Working Example

Pod Security Standards are enforced by labels on the namespace and are non-negotiable here.

apiVersion: v1
kind: Namespace
metadata:
  name: cardholder-data
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.36
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
    compliance-scope: pci

Any pod that runs as root, escalates privileges, or keeps capabilities is rejected at admission. There is no exception process at the pod level - the exception is a different namespace with a documented risk acceptance.

Image provenance is verified in-cluster, not just in CI. Signing an image but never checking the signature is theater.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-ecr-signatures
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "123456789012.dkr.ecr.us-east-1.amazonaws.com/*"
          attestors:
            - entries:
                - keyless:
                    issuer: https://token.actions.githubusercontent.com
                    subject: "https://github.com/acme/*/.github/workflows/release.yaml@refs/heads/main"
                    rekor:
                      url: https://rekor.sigstore.dev

Keyless signing binds the image to the workflow that built it, which is exactly the claim an auditor wants: this artifact came from this reviewed pipeline on this branch.

CI signs with the GitHub OIDC identity, no long-lived keys.

# in CI, after docker buildx build --push
IMAGE=123456789012.dkr.ecr.us-east-1.amazonaws.com/api@${DIGEST}
 
cosign sign --yes "${IMAGE}"
 
# attach the SBOM and the scan result as attestations
syft "${IMAGE}" -o spdx-json > sbom.json
cosign attest --yes --predicate sbom.json --type spdxjson "${IMAGE}"

The audit policy is the evidence engine. A default EKS cluster logs far less than an auditor expects.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  # never log secret contents, but do log that they were touched
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
 
  # full request+response for exec and attach: the classic audit finding
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]
 
  # RBAC changes are always interesting
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["*"]
 
  - level: Metadata
    verbs: ["create", "update", "patch", "delete"]
 
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]

Deep Dive

Why signature verification belongs at admission

CI can be bypassed. A cluster-admin with kubectl run cannot be, unless the cluster itself refuses unsigned images.

Admission is the last gate before scheduling, so it is the only place where the control is complete.

Audit logs must leave the cluster

On EKS, control-plane audit logs go to CloudWatch. Ship them onward to an S3 bucket with Object Lock in compliance mode.

The requirement is not "we have logs" - it is "these logs could not have been altered by the person being audited."

pods/exec is the finding

Interactive shells into production pods are the control auditors probe hardest. Log them at RequestResponse, alert on them, and require a break-glass role to obtain them.

Better: make them unnecessary with good observability, then remove the permission entirely.

Secrets

Kubernetes Secrets are base64, not encrypted, at the API level. Enable EKS envelope encryption with KMS so etcd contents are encrypted with a customer-managed key.

Better still, keep secrets in AWS Secrets Manager and project them with the Secrets Store CSI driver, so they never persist in etcd.

GitOps as change control

Argo CD with selfHeal: true means drift is corrected automatically and every intentional change has a reviewed PR behind it.

That PR - author, reviewer, timestamp, diff - is a better change-control artifact than any ticketing system, and it is generated for free.

Gotchas

PSS restricted breaks most vendor Helm charts. Budget weeks, not days. Many charts still assume root or a writable root filesystem.

Signing without verifying proves nothing. The audit finding is not "did you sign" but "can an unsigned image run." Test it by trying.

level: RequestResponse on secrets leaks secret values into your audit log. Use Metadata for secrets, always.

A too-verbose audit policy costs more than the cluster. Logging every watch from every kubelet generates enormous volume. Exclude system read traffic explicitly.

Namespace labels are the enforcement, so who can edit namespaces matters. If a tenant can relabel their namespace to privileged, PSS is decorative. Lock namespace mutation to the platform.

Digest pinning and mutable tags conflict. :latest cannot be verified meaningfully. Deploy by digest.

Alternatives

OPA Gatekeeper instead of Kyverno. Rego is more expressive and better established in some audit shops; Kyverno's YAML policies are easier for platform teams to review.

Validating Admission Policy (built-in CEL). No extra controller to run, which auditors like. Less capable than either policy engine for image verification.

Cluster-per-compliance-scope. Strongest separation of in-scope and out-of-scope workloads, and it dramatically shrinks the audit surface. Often worth the cost when scope reduction is the goal.

Key-based cosign signing. Simpler to explain than keyless, but you inherit key custody and rotation as new audit findings.

FAQs

Does PSS restricted satisfy a compliance requirement by itself? No. It satisfies the workload-hardening controls. You still need network, audit, provenance, and access controls.

How long do audit logs need to be retained? Depends on the framework, but one year with 90 days immediately searchable is a common baseline. Confirm with your auditor, not with a blog post.

Can developers still debug production? Through logs, metrics, and traces, yes. Through kubectl exec, only via an audited break-glass role with an expiry.

Is keyless signing acceptable to auditors? Increasingly yes, because the identity binding is stronger than a shared key. Be ready to explain Fulcio and the transparency log.

What is the first control to implement? Audit logging. Without evidence you cannot demonstrate any other control is working.

Related

• How to Read a Reference Architecture
• Reference: Multi-Tenant EKS Platform
• Reference: Worker-Heavy Cluster
• Before/After: Compose to EKS Migration
• Case Studies Best Practices

Stack versions: This page was written for Kubernetes 1.36.2, Docker Engine 29.6.1 (BuildKit default), containerd (CRI runtime on nodes), Helm 3, Compose v2, Argo CD (latest - verify at build), and Gateway API (GA - verify controller support at build).