From 82b898a7234cb7d788c85704a0206c824bb31b31 Mon Sep 17 00:00:00 2001 From: koba1t Date: Tue, 5 Nov 2024 03:08:07 +0900 Subject: [PATCH] proposal/kustomize_wasm_plugin --- .../4944-kustomize-wasm-plugin/README.md | 809 ++++++++++++++++++ .../4944-kustomize-wasm-plugin/kep.yaml | 41 + 2 files changed, 850 insertions(+) create mode 100644 keps/sig-cli/4944-kustomize-wasm-plugin/README.md create mode 100644 keps/sig-cli/4944-kustomize-wasm-plugin/kep.yaml diff --git a/keps/sig-cli/4944-kustomize-wasm-plugin/README.md b/keps/sig-cli/4944-kustomize-wasm-plugin/README.md new file mode 100644 index 00000000000..a0943c18da9 --- /dev/null +++ b/keps/sig-cli/4944-kustomize-wasm-plugin/README.md @@ -0,0 +1,809 @@ +# KEP-4944: Kustomize Wasm Plugin + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + + +## Release Signoff Checklist + + + +Items marked with (R) are required _prior to targeting to a milestone / +release_. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in + [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and + SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for + [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) + [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) + must be hit by + [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for + publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to + mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +This KEP proposes adding WebAssembly plugin support as a sandboxed execution +environment for the Kubernetes Resource Model (KRM) framework in Kustomize. + +This enhancement enables platform-independent plugin execution in a secure, +sandboxed environment, eliminating runtime dependencies while maintaining strict +isolation. + +And promote kustomize KRM plugins to GA with the addition and stability of Wasm +KRM Function. + +## Motivation + + + +Kustomize currently supports various ways to transform Kubernetes resources, +including built-in transformers and +[container](https://kubectl.docs.kubernetes.io/guides/extending_kustomize/containerized_krm_functions/)/[exec](https://kubectl.docs.kubernetes.io/guides/extending_kustomize/exec_krm_functions/) +plugins. However, these methods have limitations in terms of portability, +security, and performance. + +By introducing Wasm plugin support, we can address these limitations and provide +a more robust solution for custom resource transformations. + +#### Limitation current plugins + +The current Kustomize plugin ecosystem faces several operational challenges that +limit its effectiveness in various environments. Container plugins, while +providing good isolation, require a Docker runtime, which is often unavailable +or restricted in CI/CD environments due to security policies or resource +constraints. Exec plugins are tightly coupled to their underlying operating +system and execution environment, requiring platform-specific compilation and +distribution. This makes maintaining consistency across diverse environments +challenging. Additionally, container plugins introduce architecture-specific +dependencies, requiring builds for different CPU architectures (e.g., amd64, +arm64), which complicates deployment across heterogeneous infrastructure. These +limitations create scenarios where neither exec nor container plugins can be +effectively utilized, particularly in organizations with diverse infrastructure +requirements or strict security policies. + +### Goals + +1. Implement Wasm plugin support in the Kustomize KRM framework +1. Provide a secure sandboxed environment for executing custom transformations +1. Improve portability of custom transformers across different platforms +1. Enhance performance of resource transformations through WebAssembly + +### Non-Goals + +1. Replace existing transformation methods (built-in transformers and exec + plugins) +1. Implement specific Wasm plugins (focus is on the framework support) + +## Proposal + + + +We propose to extend the Kustomize KRM framework to support WASI plugins as an +additional execution environment. This proposal is motivated by several key +factors and aims to provide significant benefits: + +#### Benefits of Wasm Plugin Support + +##### Platform Independence + +WASM allows for writing plugins that are independent of OS and CPU +architecture.This ensures consistent behavior across different environments and +simplifies deployment. + +##### Simplified Distribution + +WASM plugins can leverage OCI registries for distribution that a potential +alternative to current container plugins. This eliminates the need to separately +distribute executable binaries to the execution environment, as is often +required with exec plugins. + +##### Familiar Interface + +Previous KRM plugins used stdin for input parameters from kustomize and stdout +for output. The WebAssembly System Interface (WASI) naturally supports this +model, allowing for a plugin framework that uses stdin and stdout as interfaces, +maintaining consistency with existing patterns and user migration will be +easier. + +##### Enhanced Security + +Wasm provides a sandboxed environment, offering improved security compared to +traditional exec plugins. + +### User Stories (Optional) + + + +#### Story 1 + +A DevOps engineer needs to build a common CI/CD pipeline that operates across +multiple cloud providers and on-premises environments. By using Wasm plugins, +they can implement custom resource transformations that work consistently and +securely across different execution environments, eliminating the overhead of +distributing and managing binaries for each environment. + +#### Story 2 + +A platform team wants to share team-specific resource transformation logic as +Kustomize plugins. By distributing Wasm plugins through OCI registries, team +members can easily obtain plugins and start using them without worrying about +execution environment setup or binary compatibility. + +#### Story 3 + +An application developer wants to use a popular open-source Kustomize plugin +that implements advanced JSON patching, which is distributed through an OCI +registry. Since the plugin is implemented in WebAssembly and runs in a sandboxed +environment, they can confidently use the third-party plugin without worrying +about security vulnerabilities or system access. The developer can easily fetch +the latest version of the plugin from the OCI registry, verify its integrity, +and run it safely within their build pipeline, knowing that the plugin's +capabilities are limited to the intended resource transformations. + +### Notes/Constraints/Caveats (Optional) + + + +- Current kustomize plugin systen that called `Exec/Container KRM functions` is + using stdin/stdout to communicate with kustomize to plugins. + - WASI is support to stdin/stdout, so We reuse the current interface + definitions and users can migrate with minimal changes. +- Now, + [Wasm binary can be distributed with OCI registry](https://tag-runtime.cncf.io/wgs/wasm/deliverables/wasm-oci-artifact/). + +### Risks and Mitigations + + + +1. WASI Specific is not yet GA. Any disruptive changes in the future will incur + user transition costs. +1. The Wasm OCI Artifact specification is a relatively new standard in the OCI + ecosystem. Maybe not mature enough. +1. Containaized KRM function is alpha feature but that used some user. + - Provide container to exec conversion tools. + +## Design Details + + + +TODO(koba1t): + +### Test Plan + + + +[ ] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes +necessary to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- ``: `` - `` + +##### Integration tests + + + + + +- : + +##### e2e tests + + + +- : + +### Graduation Criteria + + + +### Upgrade / Downgrade Strategy + +N/A + +### Version Skew Strategy + +N/A + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +N/A - client-side only + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [ ] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: + - Components depending on the feature gate: +- [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control plane? + - Will enabling / disabling the feature require downtime or reprovisioning of + a node? + +###### Does enabling the feature change any default behavior? + + + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +###### What happens if we reenable the feature if it was previously rolled back? + +###### Are there any tests for feature enablement/disablement? + + + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + +N/A - client-side only + +###### What specific metrics should inform a rollback? + +N/A - client-side only + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + +N/A - client-side only + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +I plan to deprecate +[Containerized KRM Functions](https://kubectl.docs.kubernetes.io/guides/extending_kustomize/containerized_krm_functions/) +due to I believe it was introduced for the reason of redistributing the plugin, +and we can be replaced by OCI redistribution of WASI. + +Current `Containerized KRM Functions` is just only call to `docker` command in +`$PATH`, I feel It can be integrated into the `Exec KRM Finctions`. + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + +N/A - client-side only + +###### How can someone using this feature know that it is working for their instance? + +N/A - client-side only + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +N/A - client-side only + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + +N/A - client-side only + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +N/A - client-side only + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + +No. + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +N/A - client-side only + +###### Will enabling / using this feature result in introducing new API types? + + + +No. Use API types already defined in kustomize repo. + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +N/A - client-side only + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +N/A - client-side only + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +N/A - client-side only + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +N/A - client-side only + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +N/A - client-side only + +### Troubleshooting + +N/A - client-side only + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +N/A - client-side only + +###### What are other known failure modes? + + + +N/A - client-side only + +###### What steps should be taken if SLOs are not being met to determine the problem? + +N/A - client-side only + +## Implementation History + + + +## Drawbacks + + + +The WASI specification is not yet GA. + +I am considering implementing using the `WASI Preview 1` specification, but this +means a transition will be necessary after the WASI specification is GA. + +## Alternatives + + + +It is possible to use existing `KRM Framework` to run a WASI execution +environment such as [wasmtime](https://wasmtime.dev/). However, all users need +to carefully prepare the execution environment. + +## Infrastructure Needed (Optional) + +None. This will be implemented within the existing Kustomize repo. diff --git a/keps/sig-cli/4944-kustomize-wasm-plugin/kep.yaml b/keps/sig-cli/4944-kustomize-wasm-plugin/kep.yaml new file mode 100644 index 00000000000..6312da8c1f9 --- /dev/null +++ b/keps/sig-cli/4944-kustomize-wasm-plugin/kep.yaml @@ -0,0 +1,41 @@ +title: Kustomize Wasm Plugin +kep-number: 4944 +authors: + - "@koba1t" +owning-sig: sig-cli +participating-sigs: + - sig-cli +status: implementable +creation-date: 2024-11-05 +reviewers: + - "@varshaprasad96" +approvers: + - "@varshaprasad96" + +see-also: [] +replaces: + - "/keps/sig-cli/2953-kustomize-plugin-graduation/README.md" + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.34" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + beta: TBD + stable: TBD +# # The following PRR answers are required at alpha release +# # List the feature gate name and the components for which it must be enabled +# feature-gates: +# - name: n/a +# components: +# - n/a +# disable-supported: true + +# # The following PRR answers are required at beta release +# metrics: +# - n/a