DaemonSet
Some Kubernetes distributions can bring up a Kubernetes cluster without depending on a pre-existing VIP (but they may be configured to support one). A prime example of this would be K3s which can be configured to start and also sign the certificates to allow incoming traffic to a virtual IP. Given we don't need the VIP to exist before the cluster, we can bring up the K3s node(s) and then add kube-vip as a DaemonSet for all control plane nodes.
If the Kubernetes installer allows for adding a virtual IP as an additional SAN to the API server certificate, we can apply kube-vip to the cluster once the first node has been brought up.
Unlike running kube-vip as a static Pod there are a few more things that may need configuring when running kube-vip as a DaemonSet. This page will cover primarily the differences.
kube-vip as HA, Load Balancer, or both
The functionality of kube-vip depends on the flags used to create the Daemonset manifest. By passing in --controlplane we instruct kube-vip to provide and advertise a virtual IP to be used by the control plane. By passing in --services we tell kube-vip to provide load balancing for Kubernetes Service resources created inside the cluster. With both enabled, kube-vip will manage a virtual IP address that is passed through its configuration for a highly available Kubernetes cluster. It will also watch Services of type LoadBalancer and once their service.metadata.annotations["kube-vip.io/loadbalancerIPs"] or spec.LoadBalancerIP is updated (typically by a cloud controller, including (optionally) the one provided by kube-vip in on-prem scenarios) it will advertise this address using BGP/ARP. In this example, we will use both when generating the manifest.
When in routing table mode, kube-vip can be configured to not use any kind of leader election. In such case every kube-vip instance will watch all the configured services and will configure routing if the node kube-vip instance is running on has an endpoint associated with the service. To achieve this, both leaderElection and servicesElection flags should not be set.
Create the RBAC settings
Since kube-vip as a DaemonSet runs as a regular resource instead of a static Pod, it still needs the correct access to be able to watch Kubernetes Services and other objects. In order to do this, RBAC resources must be created which include a ServiceAccount, ClusterRole, and ClusterRoleBinding and can be applied this with the command:
1kubectl apply -f https://kube-vip.io/manifests/rbac.yaml
Generating a Manifest
In order to create an easier experience of consuming the various functionality within kube-vip, we can use the kube-vip container itself to generate our static Pod manifest. We do this by running the kube-vip image as a container and passing in the various flags for the capabilities we want to enable.
Set configuration details
We use environment variables to predefine the values of the inputs to supply to kube-vip.
Set the VIP address to be used for the control plane:
export VIP=192.168.0.40
Set the INTERFACE name to the name of the interface on the control plane(s) which will announce the VIP. In many Linux distributions this can be found with the ip a command.
export INTERFACE=ens160
Get the latest version of the kube-vip release by parsing the GitHub API. This step requires that jq and curl are installed.
KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
To set manually instead, find the desired release tag:
export KVVERSION=v0.5.0
Creating the manifest
With the input values now set, we can pull and run the kube-vip image supplying it the desired flags and values.
Depending on the container runtime, use one of the two aliased commands to create a kube-vip command which runs the kube-vip image as a container.
For containerd, run the below command:
alias kube-vip="ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION; ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"
For Docker, run the below command:
alias kube-vip="docker run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"
ARP Example for DaemonSet
When creating the kube-vip installation manifest as a DaemonSet, the manifest subcommand takes the value daemonset as opposed to the pod value. The flags --inCluster and --taint are also needed to configure the DaemonSet to use a ServiceAccount and affine the kube-vip Pods to control plane nodes thereby preventing them from running on worker instances.
1kube-vip manifest daemonset \
2 --interface $INTERFACE \
3 --address $VIP \
4 --inCluster \
5 --taint \
6 --controlplane \
7 --services \
8 --arp \
9 --leaderElection
Example ARP Manifest
1apiVersion: apps/v1
2kind: DaemonSet
3metadata:
4 creationTimestamp: null
5 name: kube-vip-ds
6 namespace: kube-system
7spec:
8 selector:
9 matchLabels:
10 name: kube-vip-ds
11 template:
12 metadata:
13 creationTimestamp: null
14 labels:
15 name: kube-vip-ds
16 spec:
17 affinity:
18 nodeAffinity:
19 requiredDuringSchedulingIgnoredDuringExecution:
20 nodeSelectorTerms:
21 - matchExpressions:
22 - key: node-role.kubernetes.io/master
23 operator: Exists
24 - matchExpressions:
25 - key: node-role.kubernetes.io/control-plane
26 operator: Exists
27 containers:
28 - args:
29 - manager
30 env:
31 - name: vip_arp
32 value: "true"
33 - name: port
34 value: "6443"
35 - name: vip_interface
36 value: ens160
37 - name: vip_cidr
38 value: "32"
39 - name: cp_enable
40 value: "true"
41 - name: cp_namespace
42 value: kube-system
43 - name: vip_ddns
44 value: "false"
45 - name: svc_enable
46 value: "true"
47 - name: vip_leaderelection
48 value: "true"
49 - name: vip_leaseduration
50 value: "5"
51 - name: vip_renewdeadline
52 value: "3"
53 - name: vip_retryperiod
54 value: "1"
55 - name: address
56 value: 192.168.0.40
57 image: ghcr.io/kube-vip/kube-vip:v0.4.0
58 imagePullPolicy: Always
59 name: kube-vip
60 resources: {}
61 securityContext:
62 capabilities:
63 add:
64 - NET_ADMIN
65 - NET_RAW
66 - SYS_TIME
67 hostNetwork: true
68 serviceAccountName: kube-vip
69 tolerations:
70 - effect: NoSchedule
71 operator: Exists
72 - effect: NoExecute
73 operator: Exists
74 updateStrategy: {}
75status:
76 currentNumberScheduled: 0
77 desiredNumberScheduled: 0
78 numberMisscheduled: 0
79 numberReady: 0
BGP Example for DaemonSet
This configuration will create a manifest that starts kube-vip providing control plane VIP and Kubernetes Service management. Unlike ARP, all nodes in the BGP configuration will advertise virtual IP addresses.
Note we bind the address to lo as we don't want multiple devices that have the same address on public interfaces. We can specify all the peers in a comma-separated list in the format of address:AS:password:multihop.
export INTERFACE=lo
1kube-vip manifest daemonset \
2 --interface $INTERFACE \
3 --address $VIP \
4 --inCluster \
5 --taint \
6 --controlplane \
7 --services \
8 --bgp \
9 --localAS 65000 \
10 --bgpRouterID 192.168.0.2 \
11 --bgppeers 192.168.0.10:65000::false,192.168.0.11:65000::false
Example BGP Manifest
1apiVersion: apps/v1
2kind: DaemonSet
3metadata:
4 creationTimestamp: null
5 name: kube-vip-ds
6 namespace: kube-system
7spec:
8 selector:
9 matchLabels:
10 name: kube-vip-ds
11 template:
12 metadata:
13 creationTimestamp: null
14 labels:
15 name: kube-vip-ds
16 spec:
17 affinity:
18 nodeAffinity:
19 requiredDuringSchedulingIgnoredDuringExecution:
20 nodeSelectorTerms:
21 - matchExpressions:
22 - key: node-role.kubernetes.io/master
23 operator: Exists
24 - matchExpressions:
25 - key: node-role.kubernetes.io/control-plane
26 operator: Exists
27 containers:
28 - args:
29 - manager
30 env:
31 - name: vip_arp
32 value: "false"
33 - name: port
34 value: "6443"
35 - name: vip_interface
36 value: ens160
37 - name: vip_cidr
38 value: "32"
39 - name: cp_enable
40 value: "true"
41 - name: cp_namespace
42 value: kube-system
43 - name: vip_ddns
44 value: "false"
45 - name: svc_enable
46 value: "true"
47 - name: bgp_enable
48 value: "true"
49 - name: bgp_routerid
50 value: 192.168.0.2
51 - name: bgp_as
52 value: "65000"
53 - name: bgp_peeraddress
54 - name: bgp_peerpass
55 - name: bgp_peeras
56 value: "65000"
57 - name: bgp_peers
58 value: 192.168.0.10:65000::false,192.168.0.11:65000::false
59 - name: address
60 value: 192.168.0.40
61 image: ghcr.io/kube-vip/kube-vip:v0.4.0
62 imagePullPolicy: Always
63 name: kube-vip
64 resources: {}
65 securityContext:
66 capabilities:
67 add:
68 - NET_ADMIN
69 - NET_RAW
70 - SYS_TIME
71 hostNetwork: true
72 serviceAccountName: kube-vip
73 tolerations:
74 - effect: NoSchedule
75 operator: Exists
76 - effect: NoExecute
77 operator: Exists
78 updateStrategy: {}
79status:
80 currentNumberScheduled: 0
81 desiredNumberScheduled: 0
82 numberMisscheduled: 0
83 numberReady: 0
Managing a routerID as a DaemonSet
The routerID needs to be unique on each node that participates in BGP advertisements. In order to do this, we can modify the manifest so that when kube-vip starts it will look up its local address and use that as the routerID. Add the following to the env[] array of the container:
1- name: bgp_routerinterface
2 value: "ens160"
Alternatively, we can use the IP address of the pod itself as a unique value for the routerID:
1- name: bgp_routerid
2 valueFrom:
3 fieldRef:
4 fieldPath: status.podIP
DaemonSet Manifest Overview
Once the manifest for kube-vip as a DaemonSet is generated, these are some of the notable differences over the static Pod manifest and their significance.
nodeSelector: Ensures that DaemonSet Pods only run on control plane nodes.serviceAccountName: kube-vip: Specifies the ServiceAccount name that will be used to get/update Kubernetes Service resources.tolerations: Allows scheduling to control plane nodes that normally specifyNoScheduleorNoExecutetaints.