[BUG] VPC Nat Gateway iptables doesn't reconcile on node reboot

[abasitt]

Kube-OVN Version

v1.13.1

Kubernetes Version

Client Version: v1.30.10
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.10

Operation-system/Kernel Version

"Rocky Linux 8.10 (Green Obsidian)"
4.18.0-513.5.1.el8_9.x86_64

Description

VPC nat gateway running on a node e.g.

[root@vnode-103-163 abasit]# k get pods -n kube-system -owide | grep gw
vpc-nat-gw-vpc1-nat-gateway-0           1/1     Running   0              16m     192.168.200.254   vnode-103-167   <none>           <none>

if the node on which vpc-gateway reboot ( force reboot for testing). With node reboot, pod reboot e.g. below 1 restart. It seems like with restart it looses all iptable state e.g. EIP/FIP etc.

k get pods -n kube-system -owide | grep gw
vpc-nat-gw-vpc1-nat-gateway-0           1/1     Running   1 (26s ago)    21m     192.168.200.254   vnode-103-167   <none>           <none>

I see below logs in the ovn controller.

I0225 07:39:20.751308       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:39:20.751339       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:39:27.363923       7 endpoint.go:67] handle update endpoint kube-system/kube-ovn-cni
I0225 07:39:27.363975       7 endpoint.go:192] add vip endpoint 10.107.208.20:10665, backends [10.9.103.163:10665 10.9.103.167:10665 10.9.117.39:10665] to LB cluster-tcp-loadbalancer
I0225 07:39:35.080437       7 pod.go:395] enqueue update pod kube-system/kube-ovn-pinger-rm5lx
I0225 07:39:35.080472       7 pod.go:450] handle add/update pod kube-system/kube-ovn-pinger-rm5lx
I0225 07:39:35.084552       7 endpoint.go:67] handle update endpoint kube-system/kube-ovn-pinger
I0225 07:39:35.084599       7 endpoint.go:192] add vip endpoint 10.106.111.108:8080, backends [10.16.0.4:8080 10.16.0.5:8080] to LB cluster-tcp-loadbalancer
I0225 07:39:36.087607       7 pod.go:395] enqueue update pod kube-system/kube-ovn-pinger-rm5lx
I0225 07:39:36.087643       7 pod.go:450] handle add/update pod kube-system/kube-ovn-pinger-rm5lx
I0225 07:39:36.098495       7 endpoint.go:67] handle update endpoint kube-system/kube-ovn-pinger
I0225 07:39:36.098558       7 endpoint.go:192] add vip endpoint 10.106.111.108:8080, backends [10.16.0.2:8080 10.16.0.4:8080 10.16.0.5:8080] to LB cluster-tcp-loadbalancer
I0225 07:39:37.091978       7 pod.go:395] enqueue update pod vpc1/starter-backend-7ff5f85b46-ktfj7
I0225 07:39:37.092008       7 pod.go:450] handle add/update pod vpc1/starter-backend-7ff5f85b46-ktfj7
I0225 07:39:37.103223       7 pod.go:395] enqueue update pod vpc1/starter-backend-7ff5f85b46-lhqdp
I0225 07:39:37.103247       7 pod.go:450] handle add/update pod vpc1/starter-backend-7ff5f85b46-lhqdp
I0225 07:39:37.297235       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:39:37.297269       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:39:38.105201       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:39:38.105239       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0

but inside the nat-gateway-pod, nothing. e.g.

 k exec vpc-nat-gw-vpc1-nat-gateway-0 -n kube-system -it -- iptables-legacy -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

If I manually bounce the pod after node is up, everything comes up correctly.

k delete pod vpc-nat-gw-vpc1-nat-gateway-0 -n kube-system
pod "vpc-nat-gw-vpc1-nat-gateway-0" deleted
[root@vnode-103-163 abasit]# k exec vpc-nat-gw-vpc1-nat-gateway-0 -n kube-system -it -- iptables-legacy -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT_FILTER  all  --  anywhere             anywhere

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SNAT_FILTER  all  --  anywhere             anywhere

Chain DNAT_FILTER (1 references)
target     prot opt source               destination
EXCLUSIVE_DNAT  all  --  anywhere             anywhere
SHARED_DNAT  all  --  anywhere             anywhere

Chain EXCLUSIVE_DNAT (1 references)
target     prot opt source               destination

Chain EXCLUSIVE_SNAT (1 references)
target     prot opt source               destination

Chain SHARED_DNAT (1 references)
target     prot opt source               destination

Chain SHARED_SNAT (1 references)
target     prot opt source               destination
SNAT       all  --  192.168.0.0/16       anywhere             to:10.9.250.4 random-fully

Chain SNAT_FILTER (1 references)
target     prot opt source               destination
EXCLUSIVE_SNAT  all  --  anywhere             anywhere
SHARED_SNAT  all  --  anywhere             anywhere

the controllers logs different also.

I0225 07:42:53.173807       7 ipam.go:73] allocating static ip 192.168.200.254 from subnet internal-subnet-1
I0225 07:42:53.173827       7 ipam.go:103] allocate v4 192.168.200.254, mac f2:56:c2:d0:0a:8a for kube-system/vpc-nat-gw-vpc1-nat-gateway-0 from subnet internal-subnet-1
I0225 07:42:53.180126       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:53.187737       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:53.189789       7 pod.go:671] sync pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0 routed
I0225 07:42:53.189939       7 vpc_nat_gateway.go:290] handle init vpc nat gateway vpc1-nat-gateway
I0225 07:42:53.192018       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:53.195879       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:53.198645       7 pod.go:671] sync pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0 routed
I0225 07:42:53.199566       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:53.202463       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:53.694652       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:53.694687       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:54.476632       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:54.476662       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
E0225 07:42:58.190477       7 vpc_nat_gateway.go:297] failed to get nat gw vpc1-nat-gateway pod: pod is not active now
E0225 07:42:58.190530       7 controller.go:1346] "Unhandled Error" err="error syncing init vpc nat gateway \"vpc1-nat-gateway\": failed to get nat gw vpc1-nat-gateway pod: pod is not active now, requeuing" logger="UnhandledError"
I0225 07:42:59.191084       7 vpc_nat_gateway.go:290] handle init vpc nat gateway vpc1-nat-gateway
I0225 07:42:59.255511       7 vpc_nat_gateway.go:463] handle update vpc dnat vpc1-nat-gateway
I0225 07:42:59.255537       7 vpc_nat_gateway.go:361] handle update vpc fip vpc1-nat-gateway
I0225 07:42:59.255546       7 vpc_nat_gateway.go:429] handle update vpc snat vpc1-nat-gateway
I0225 07:42:59.261654       7 vpc_nat_gw_eip.go:49] enqueue update iptables eip eips01
I0225 07:42:59.261753       7 vpc_nat_gw_eip.go:195] handle update iptables eip eips01
I0225 07:42:59.261914       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:59.261933       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:59.264175       7 vpc_nat_gw_eip.go:244] handle update eip eips01
I0225 07:42:59.269323       7 vpc_nat_gateway.go:540] handle update subnet route for nat gateway vpc1-nat-gateway
I0225 07:42:59.269686       7 vpc_nat_gw_nat.go:641] handle update iptables snat rule snat01
I0225 07:42:59.277104       7 vpc_nat_gateway.go:396] handle update vpc eip vpc1-nat-gateway
I0225 07:42:59.281065       7 pod.go:395] enqueue update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0
I0225 07:42:59.281100       7 pod.go:450] handle add/update pod kube-system/vpc-nat-gw-vpc1-nat-gateway-0

Steps To Reproduce

1- deploy vpc nat gateway
2- config eip/snat
3- reboot the node on which nat-gateway is running
4- once the node back up again, nat-gateway pod looses iptables-state.

Current Behavior

iptables config inside the nat-gateway is lost on node reboot.

Expected Behavior

iptables config should be reconcile correctly even if the node is rebooted, same like when gateway pod is manually rebooted.

[dosubot]

I found some similar closed issues that might be relevant to your problem:

1. LoadBalancer vpc-nat-gateway was closed by a pull request that added a reinit process for the lb-svc pod ^[1].
2. nat-gateway.sh init not exec after k8s cluster reboot was closed by a pull request that added an init container in the vpc-nat-gateway StatefulSet for initialization ^[2].

The Kube-OVN controller handles iptables rule persistence and reconciliation for the VPC NAT Gateway during pod restarts by reapplying the necessary rules and configurations when a NAT Gateway pod is initialized or updated. This involves using the handleInitVpcNatGw function to check if the pod has been initialized by looking for a specific annotation. If not, it executes the initial set of iptables rules using the execNatGwRules function with the natGwInit operation ^[3].

Additionally, the init container in the VPC NAT Gateway StatefulSet plays a crucial role in setting up necessary iptables rules and configurations during initialization. This ensures that the iptables rules are correctly applied and reconciled ^[3].

It seems like the issue you're experiencing might be related to the initialization process not being triggered correctly after a node reboot. You might want to check if the init container is functioning as expected and if the necessary annotations are being applied to the pod.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[oilbeater]

We are missing the mechanism to reconcile VPC NAT Gateway IPTables, and we need to add this functionality. @zhangzujian , do we have a similar issue with the new egress gateway?

[zhangzujian]

We are missing the mechanism to reconcile VPC NAT Gateway IPTables, and we need to add this functionality. @zhangzujian , do we have a similar issue with the new egress gateway?

VPC Egress Gateway uses a different implement. Routes and iptables rules are added/configured on pod initialization.

[abasitt]

@zhangzujian is this a new feature in the incoming release ? If nat-gateway has to stay, can this be fixed because this will have a hug impact if we move to production with current limitation.

[zhangzujian]

@zhangzujian is this a new feature in the incoming release ?

It's a new feature introduced in v1.14.0. Here is the document.

[abasitt]

@zhangzujian thank you, this is great. Really looking forward to it. Will this have in the future EIP/FIP features as well ?

[zhangzujian]

Will this have in the future EIP/FIP features as well ?

No. It focuses on egress traffic only.