✨ Support for BoostrapSelfManagedAddons flag for EKS cluster creation

[jas-nik]

What type of PR is this?
/kind feature

What this PR does / why we need it:

Add flag to support BootstrapSelfManagedAddons to provision Bare EKS cluster without default addons (coreDNS, kube-proxy, aws-vpc-cni)

https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html

By default, EKS installs multiple networking add-ons during cluster creation. This includes the Amazon VPC CNI, CoreDNS, and kube-proxy.

If you’d like to disable the installation of these default networking add-ons, use the parameter below. This may be used for alternate CNIs, such as Cilium. Review the EKS API reference for more information.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist:

• squashed commits
• includes documentation
• includes emojis
• adds unit tests
• adds or updates e2e tests

Release note:

Add flag to support BootstrapSelfManagedAddons to provision Bare EKS cluster without default addons (coreDNS, kube-proxy, aws-vpc-cni)

[k8s-ci-robot]

Hi @jas-nik. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[adriananeci]

/ok-to-test

[jas-nik]

"failed to create new managed VPC: failed to create vpc: The maximum number of VPCs has been reached"

😞

[jas-nik]

/retest

[JonnieDoe]

/lgtm

[k8s-ci-robot]

@JonnieDoe: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[JonnieDoe]

/retest

[jas-nik]

/retest

[nrb]

@jas-nik Sorry about the delay here, and thank you for this contribution.

I've been troubleshooting EKS CI issues, and this behavior is very welcome :)

I do ask that you add some tests for this case. We'll need at least a cluster template and to point to that template in the e2e test config.

[jas-nik]

@nrb Apologies, this fell off my radar. I'll get them added

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign neolit123 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[damdo]

Hey @jas-nik CAPA doesn't do merge commits, would you be able to rebase instead?
Thanks!

[nrb]

/test pull-cluster-api-provider-aws-test

Hit VPC limit.

[jas-nik]

Still the same VPC limit issue.

[jas-nik]

/retest

[damdo]

/test pull-cluster-api-provider-aws-test

[jas-nik]

@damdo @nrb @richardcase would you be able to help with the VPC limit issue?

[damdo]

/test pull-cluster-api-provider-aws-test

[nrb]

@jas-nik Unfortunately we can't allocate more VPCs right now. The best we can do is retry the tests at off-peak times.

[nrb]

/retest

[richardcase]

@jas-nik @nrb @damdo - are getting the "VPC limit" error when running just the unit tests or the e2e tests? If it's the unit tests, (i.e. via /test pull-cluster-api-provider-aws-test), which i guess is the case from the discussion, then that should be within our control to change as it shouldn't be fitting AWS and this may be coming from out "resource counting" code.

Also, if it is the e2e then potentially we can increase the service limits.

[k8s-ci-robot]

@jas-nik: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-aws-test	49f825a	link	true	/test pull-cluster-api-provider-aws-test

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[richardcase]

Are we sure thats the real error? We have a test the checks for maximum number of VPCs:

cluster-api-provider-aws/controllers/awscluster_controller_test.go

Line 569 in dcea94f

g.Expect(err.Error()).To(ContainSubstring("The maximum number of VPCs has been reached"))

I will have a look at the logs in the morning to check.

[richardcase]

Looking at the logs it seems to be multiple issues with tests:

• FAIL: TestDefaultingWebhook (0.59s) (and sub tests)
• FAIL: TestCreateCluster/cluster_create_with_2_subnets (0.00s)
• FAIL: TestCreateIPv6Cluster (0.01s)

Its worth running just these tests locally to see whats going on.

[jas-nik]

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/5336/pull-cluster-api-provider-aws-test/1887504193065848832 - One of the latest PRs build passed even after VPC limit failure, so it might be a red herring after all. Thank you for chiming in. Still need to investigate what is the actual failure.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.