OPA Gatekeeper Issue #3928 Playground

What You Can Do

• Recreate Gatekeeper issue #3928
• Launch separate playgrounds with either a 1.29.12 or 1.32.3 Kind cluster
• Deploy Gatekeeper 3.15.1 (no defect)
• Or deploy Gatekeeper 3.18.3 (defect)
• Apply policy enforcement resources
• Test policy enforcement with the sample resources

Deploying Kind 1.29.12 and Gatekeeper 3.15.1

Deployment Files

Deployment files for reference:

• deploy.sh
• mutating-gatekeeper-3-15-1.yaml
• validating-gatekeeper-3-15-1.yaml
• cert-manager-resources.yaml

Observations

No defect observations for Gatkeeper 3.15:

• Support running seprate operations for each webhook, one for validating and one for mutation.
• Does not have an operation=generate for the audit webhook.
• All mutation and valdation policies work as expected.

Running the Playground

1. Open the playground in a separate browser tab
2. Start playground with kind.version set to 1.29.12 under the Playground Parameters section above.
3. Once the playground fully starts, you control the deployment of the Gatekeeper environement by using make targets under the home directory of the playground.

Running Make Targets

Ensure you are in the home directory of the playground and run the following make targets in this order:

1. Deploy Gatekeeper version 3.15.1.

   make deploy-gk-3-15-1
   

   Copy to clipboard
2. Apply all Gatekeeper policies (mutation and validation).

   make apply-policy
   

   Copy to clipboard
3. Apply all example resources.

   make apply-example-resources
   

   Copy to clipboard
4. Review the running gatekeeper instances and their configured operations.
   • The gatekeeper mutation webhook is configured with only mutation-webhook operation.
   • The gatekeeper validating webhook is configured with only webhook operation.
   • The gatekeeper audit webhook is configured with audit, status, mutation-status operations.

Deploying Kind 1.32.3 and Gatekeeper 3.18.3

Deployment Files

Deployment files for reference:

• deploy.sh
• mutating-gatekeeper-3-18-3.yaml
• validating-gatekeeper-3-18-3.yaml
• cert-manager-resources.yaml

Observations

Defect observations for Gatekeeper 3.18

• Cannot run seprate operations for each webhook, one for validating and one for mutation.
  • As a potential workaround: Add the webhook operation to the mutation gatekeeper webhook in addition to the mutation-webhook operation.
• The audit gatekeeper webhook MUST add the generate operation.
• All mutation and valdation policies seem to work as expected.

Running the Playground

1. Open the playground in a separate browser tab
2. Start the playground with kind.version set to 1.32.3 under the Playground Parameters section above.
3. Once the playground fully starts, you control the deployment of the Gatekeeper environement by using make targets under the home directory of the playground.

Running Make Targets

Ensure you are in the home directory of the playground and run the following make targets in this order:

1. Deploy Gatekeeper version 3.18.3:

   make deploy-gk-3-18-3
   

   Copy to clipboard
   • The gatekeeper-mutating-controller-manager will fail to start due to Gatekeeper issue #3928.
   • A potential workaround is to add the operation=webhook to the gatekeeper-mutating-controller-manager deployment manifest.
   • Update the deployment manifest by running the following command:

   sed -i '/- --operation=mutation-webhook/ a\        - --operation=webhook' deployment/mutating-gatekeeper-3-18-3.yaml
   

   Copy to clipboard
   • Re-run the make deploy-gk-3-18-3 target command
2. Apply all Gatekeeper policies (mutation and validation):

   make apply-policy
   

   Copy to clipboard
3. Apply all example resources:

   make apply-example-resources
   

   Copy to clipboard
4. Review the running gatekeeper instances and their configured operations.
   • The gatekeeper mutation webhook is configured with the potential workaround of mutation-webhook and webhook operations.
   • The gatekeeper validating webhook is configured with only webhook operation.
   • The gatekeeper audit webhook is configured with audit, status, mutation-status, generate operations.

Validating the Fix — PR #4423

PR #4423 fixes issue #3928 by conditionally initializing the constraint/OPA client only when validation operations are enabled (operations.HasValidationOperations()). Since the constraint client is exclusively used for validation (constraints and constraint templates), it is no longer required for mutation-only deployments. The cache manager now handles a nil cfClient gracefully to prevent panics when mutation operations interact with the cache without a constraint client present.

With this fix, Gatekeeper can start successfully with --operation=mutation-webhook only — no workaround of adding --operation=webhook is needed.

Deployment Files

Deployment files for reference:

• deploy.sh
• mutating-gatekeeper-3-22-0.yaml
• validating-gatekeeper-3-22-0.yaml
• cert-manager-resources.yaml

Note: The deployment manifests use the development image openpolicyagent/gatekeeper:70d79e4 which contains the fix from PR #4423. This fix is expected to ship in the next official release (v3.23.0+).

Runtime Flags

Since this setup splits Gatekeeper into separate controllers (validating, mutating, audit), each deployment requires specific runtime flags. The constraint templates in this setup use Rego only (no CEL/K8sNativeValidation engine), so the Kubernetes ValidatingAdmissionPolicy (VAP) integration must be explicitly disabled. In Gatekeeper v3.22.x, several VAP-related flags default to true, which causes errors when the VAP API is not in use.

Validating Webhook (--operation=webhook)

Mutating Webhook (--operation=mutation-webhook)

The mutating webhook does not need VAP flags because it has no validation operations — the constraint controller's Add() method returns early when HasValidationOperations() is false.

Audit Controller (--operation=audit,status,mutation-status,generate)

Note: During startup, the constraint controller unconditionally calls IsVapAPIEnabled() as part of controller registration (source). This produces a one-time error checking VAP API availability log entry at error level in both the validating webhook and audit pods. This is benign — it only determines whether to register VAP binding watchers and does not affect constraint enforcement. No runtime flag can suppress this startup check.

For the full list of Gatekeeper runtime flags, see the Runtime Flags documentation.

Observations

Fix validation observations for Gatekeeper with PR #4423:

• The mutating controller starts successfully with only --operation=mutation-webhook — the previous unable to create client: must specify at least one enforcement point error no longer occurs.
• The --operation=webhook workaround documented in the 3.18.3 section is no longer required.
• Separate operations for each webhook are restored: one for validating (webhook) and one for mutation (mutation-webhook).
• All mutation and validation policies work as expected.

Running the Playground

1. Open the playground in a separate browser tab
2. Start the playground with kind.version set to 1.32.3 under the Playground Parameters section above.
3. Once the playground fully starts, you control the deployment of the Gatekeeper environment by using make targets under the home directory of the playground.

Running Make Targets

Ensure you are in the home directory of the playground and run the following make targets in this order:

1. Deploy Gatekeeper with the fix from PR #4423:

   make deploy-gk-3-22-0
   

   Copy to clipboard
   • The gatekeeper-mutating-controller-manager should now start without errors using only --operation=mutation-webhook.
2. Apply all Gatekeeper policies (mutation and validation):

   make apply-policy
   

   Copy to clipboard
3. Apply all example resources:

   make apply-example-resources
   

   Copy to clipboard
4. Review the running gatekeeper instances and their configured operations.
   • The gatekeeper mutation webhook is configured with only mutation-webhook operation (fix validated).
   • The gatekeeper validating webhook is configured with only webhook operation.
   • The gatekeeper audit webhook is configured with audit, status, mutation-status, generate operations.