Add proposal for simulation provider for testing#1121
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
229b46e to
f103eb4
Compare
|
/kind enhancement |
f103eb4 to
b8bf237
Compare
ashwani2k
left a comment
There was a problem hiding this comment.
Thanks for collating the thoughts here. Just some observations.
| @@ -0,0 +1,116 @@ | |||
| # Simulation Provider for Testing | |||
There was a problem hiding this comment.
Won't Emulator be a better fit. As you emulate the cloud provider here, or may be call it virtual, I think simulation means you will still call the actual provider but you just simulate the tests locally.
or may be this is a like a fake provider.
There was a problem hiding this comment.
@ashwani2k It is exactly the other way around.
An emulator replicates both the hardware and software of a target device, acting as an exact clone (e.g., running an Android game on your PC). A simulator only mimics the software environment and its behavior, without replicating the underlying hardware (e.g., testing how an iOS app interface responds on your `Mac)
So, it is best to call it simulated-provider.
|
|
||
| 1. Initialise control plane i.e. start `kube-apiserver`, `etcd` as local processes. | ||
| 2. Deploy MCM CRDs. | ||
| 3. Start `machine-controller-manager` and `machine-controller-simulation-provider` processes. |
There was a problem hiding this comment.
Even if we keep it as simulation we should still name it as machine-controller-manager-provider-simulator|virtual|fake to be inline with other providers.
|
|
||
| ## Target Scope | ||
|
|
||
| Only offer functionality needed to run IT standalone (as part of CI/CD pipelines), with optional support for adding points of failures. The simulation config skaffold and supported modifications doesn't have to be exhaustive in the first iteration, however it should be extensible enough to allow someone to add their customizations to the simulation provider at their specified modification point to test edge cases (restricted still to modification related to driver interface calls). |
There was a problem hiding this comment.
As currently we don't run IT standalone, it will be good to classify test added as part of this endeavor as mandatory vs non-mandatory for PR merge. Otherwise you might end up with IT test failing and never succeeding esp. if they validate scenarios which have time constraints.
|
|
||
| ### Initialisation | ||
|
|
||
| Cluster creation and lifecycle to be managed via `e2e-framework`, it handles the initialisation and creation of the minimal control plane needed for running the simulation provider. This fetches the specified `kube-apiserver` and `etcd` binaries (if not present) and runs them as local processes. |
There was a problem hiding this comment.
What is this e2e-framework ? Why can't we just use a make target for this ? Ex: make test-integration-setup ?
| "percentageOfMachines": "10%" | ||
| }, | ||
| "preDelete": { | ||
| "rateLimitError": { |
There was a problem hiding this comment.
We need this configurable as a pattern for against MachineDeployments. ie we need a machineDeploymentPattern attribute somewhere in this json.
| The simulation provider would be added as a new package located at `pkg/simulator`. | ||
| Proposed file structure: | ||
| ``` | ||
| pkg/simulator: |
There was a problem hiding this comment.
This should be pkg/simulatedprovider
| - simulation.go | ||
| - config.go | ||
| provider/simulation/injections/ | ||
| - pre_create.go |
There was a problem hiding this comment.
I don't think files like pre_create.go are needed. What would even go inside this file ? If someone wants to inject something outside of the MCM code base, then explicitly define an interface for this.
What this PR does / why we need it:
This PR introduces a proposal to introduce a machine controller simulation provider to enhance developer experience by making manual testing easier and having IT run as part of MCM CI/CD pipelines without the need of real CSP infrastructure.
Which issue(s) this PR fixes:
Fixes part of #1120
Special notes for your reviewer:
Release note: