Troubleshooting Egress

This message will appear if there are kernel modules that are missing from the system, kube-vip will highlight in the logs which are the missing modules as follows:

missing iptables modules -> nat [false] -> filter [false] mangle -> [false]

To install these modules you can do the following:


          1
          sudo modprobe iptable_filter
        
          2
          sudo modprobe iptable_nat
        
          3
          sudo modprobe iptable_mangle
not set

They should also be added to /etc/modules for reboot persistence.

The Calico CNI by default will always attempt to have its iptables rules as the highest priority, which means that the kube-vip rules can end up being ignored. In order for the kube-vip egress rules to have the precident over any other rules managed by Calico we need to modify its behaviour, which we can do with the following command:


          1
          kubectl patch felixconfigurations.crd.projectcalico.org default --type='merge' -p '{"spec":{"chainInsertMode":"Append"}}'
not set

We can verify the mode of the calcio pods by examining them:


          1
          kubectl logs -n kube-system calico-node- | grep -i chaininsertmode
not set

More information about Calicos behaviour is available here

In the event that kube-vip is being terminated, then it won't be able to clean up existing rules during shutdown. In order for kube-vip to clean those rules we can add the environment variable EGRESS_CLEAN, set to true to the kube-vip configuration. This will ensure that on startup kube-vip will remove any rules that have the comment /* a3ViZS12aXAK=kube-vip */ (used to identify rules kube-vip manages).

In order to view the iptables rules created by kube-vip you may need to use the legacy iptables command, you can view the current configuration with sudo iptables -v. If nf_tables is listed then you will need to use iptables-legacy in order to view the correct rules.

iptables-legacy -t mangle -L


          1
          sudo iptables-legacy -t mangle -L
        
          2
          Chain PREROUTING (policy ACCEPT)
        
          3
          target     prot opt source               destination
        
          4
          KUBE-VIP-EGRESS  all  --  anywhere             anywhere             /* a3ViZS12aXAK=kube-vip */
        
          6
          {...}
        
          8
          Chain KUBE-VIP-EGRESS (1 references)
        
          9
          target     prot opt source               destination
        
          10
          RETURN     all  --  anywhere             10.0.0.0/16          /* a3ViZS12aXAK=kube-vip */
        
          11
          RETURN     all  --  anywhere             10.96.0.0/12         /* a3ViZS12aXAK=kube-vip */
        
          12
          MARK       all  --  172.17.88.129        anywhere             /* a3ViZS12aXAK=kube-vip */ MARK or 0x40
        
          13
          MARK       all  --  172.17.88.19         anywhere             /* a3ViZS12aXAK=kube-vip */ MARK or 0x40
        
          14
          MARK       all  --  172.17.88.190        anywhere             /* a3ViZS12aXAK=kube-vip */ MARK or 0x40
...
not set

          1
          sudo iptables-legacy -t nat -L POSTROUTING
        
          3
          Chain POSTROUTING (policy ACCEPT)
        
          4
          target     prot opt source               destination
        
          5
          SNAT       all  --  172.17.88.129        anywhere             mark match 0x40/0x40 /* a3ViZS12aXAK=kube-vip */ to:192.168.0.217
not set