Skip to content

In this section we will prepare the NKP Cluster with Flow CNI.

Deploy NKP Clusters

This section will take you through install NKP(Kubernetes) on Nutanix cluster as we will be deploying Flow CNI on these kubernetes clusters and testing networking between containers and VMs.

Create a Jumphost VM

Create a Jumphost VM in any available subnet or Nutanix cluster.

  1. See instructions here to prepare your workstation (Mac/PC) with Tools
  2. See instructions here to create a Jumphost VM

Create Externel Subnet

We will create a external subnet to deploy the NKP cluster nodes. This has to be a separate network to the one that VM and containers will be sharing to communicate.

Warning

You must deploy the NKP clusters only on managed VLAN Basic (AHV-managed) subnets.

Note

The IP address schemes are for illustrative purposes. Choose your own subnet details.

Make sure to have enough IP (at least 2) addresses for nkpflow workload NKP cluster. See here for IP address requirement details for workload cluster.

  1. Go to Prism Central > Networking > Subnets

  2. Click on Create Subnet and fill the following details

    • Name: NKP
    • Type: VLAN
    • Virtual Switch: vs0
    • VLAN ID: 203
    • Exretnal connectivity for VPC - Toggle to Yes and select NAT
    • IP Address Management: Nutanix IPAM
    • Network IP Address / Profile: 10.24.160.0/22
    • Gateway IP Address: 10.24.160.1
    • IP Pool
      • Start Address: 10.24.163.45
      • End Address: 10.24.163.60
    • Click on Create

NKP High Level Cluster Design

The nkpflow cluster will be hosting the Flow CNI and the testing application stack.

Management Cluster

Management Cluster called nkpmanage will be essential to deploying a workload nkpflow cluster.

Role No. of Nodes (VM) vCPU RAM Storage
Master 1 12 16 GB 200 GB
Worker 2 12 16 GB 200 GB

Flow Workload Cluster

For nkpflow, we will deploy an NKP Cluster of with the following resources to be able to deploy Flow CNI.

Role No. of Nodes (VM) vCPU RAM Storage
Master 1 12 16 GB 200 GB
Worker 2 12 16 GB 200 GB

Deploy NKP Management Cluster

  1. Follow instructions here to deploy a management cluster.

  2. Ensure to license the Management cluster with at least NKP Pro License

  3. Follow instructions here to generate and license the NKP Management cluster.

Create Rocky Linux Base Image

Warning

You can deploy Flow CNI to extend VPCs with VM configurations to NKP-managed Kubernetes workload clusters only in Nutanix environments with the following Nutanix software versions:

Nutanix Kubernetes Platform: 2.15 or later with Rocky Linux operating system.

  1. Connect to the Jumphost VM using VSCode

  2. In VSCode Explorer pane, click on existing $HOME folder

  3. Click on New Folder name it: flow

  4. On VSCode Explorer plane, click the $HOME/flow folder

  5. On VSCode menu, select Terminal > New Terminal

  6. Browse to nkp directory

    cd $HOME/flow

  7. Create and the following values inside the .env file for the workload nkpflow cluster

    export NUTANIX_USER=_your_nutanix_username
export NUTANIX_PASSWORD=_your_nutanix_password
export NUTANIX_ENDPOINT=_your_prism_central_fqdn
export NUTANIX_CLUSTER=_your_prism_element_cluster_name
export NUTANIX_SUBNET_NAME=_your_ahv_ipam_network_name
export STORAGE_CONTAINER=_your_storage_container_nmae
export SSH_PUBLIC_KEY=_path_to_ssh_pub_key_on_jumphost_vm
export NKP_CLUSTER_NAME=_your_nkp_cluster_name
export CONTROLPLANE_VIP=_your_nkp_cluster_controlplane_ip
export LB_IP_RANGE=_your_range_of_two_ips
export DOCKER_USERNAME=_your_docker_username
export DOCKER_PASSWORD=_your_docker_password_pat
export CONTROL_PLANE_REPLICAS=_no_of_control_plane_replicas
export CONTROL_PLANE_VCPUS=_no_of_control_plane_vcpus
export CONTROL_PLANE_CORES_PER_VCPU=_no_of_control_plane_cores_per_vcpu
export CONTROL_PLANE_MEMORY_GIB=_no_of_control_plane_memory_gib
export WORKER_REPLICAS=_no_of_worker_replicas
export WORKER_VCPUS=_no_of_worker_vcpus
export WORKER_CORES_PER_VCPU=_no_of_worker_cores_per_vcpu
export WORKER_MEMORY_GIB=_no_of_worker_memory_gib
export CSI_FILESYSTEM=_preferred_filesystem_ext4/xfs
export CSI_HYPERVISOR_ATTACHED=_true/false

    export NUTANIX_USER=admin
export NUTANIX_PASSWORD=xxxxxxxx
export NUTANIX_ENDPOINT=pc.example.com
export NUTANIX_CLUSTER=pe
export NUTANIX_SUBNET_NAME=NKP
export STORAGE_CONTAINER=default
export SSH_PUBLIC_KEY=$HOME/.ssh/id_rsa.pub
export NKP_CLUSTER_NAME=nkpflow
export CONTROLPLANE_VIP=10.x.x.220
export LB_IP_RANGE=10.x.x.221-10.x.x.222
export DOCKER_USERNAME=_your_docker_username
export DOCKER_PASSWORD=_your_docker_password_pat
export CONTROL_PLANE_REPLICAS=1
export CONTROL_PLANE_VCPUS=12
export CONTROL_PLANE_CORES_PER_VCPU=1
export CONTROL_PLANE_MEMORY_GIB=16
export WORKER_REPLICAS=2
export WORKER_VCPUS=12
export WORKER_CORES_PER_VCPU=1
export WORKER_MEMORY_GIB=16
export CSI_FILESYSTEM=ext4
export CSI_HYPERVISOR_ATTACHED=true

  8. Ensure to use the rocky-9.6 as the image version in the base image command

    nkp create image nutanix rocky-9.6 \
  --endpoint ${NUTANIX_ENDPOINT} --cluster ${NUTANIX_CLUSTER} \
  --subnet ${NUTANIX_SUBNET_NAME} --insecure

    ---> 100%
Build 'nutanix.kib_image' finished after 4 minutes 55 seconds.
==> Wait completed after 4 minutes 55 seconds

==> Builds finished. The artifacts of successful builds are:
--> nutanix.kib_image: export NKP_IMAGE=nkp-rocky-9.6-1.34.3-20260609005954
--> nutanix.kib_image: export NKP_IMAGE=nkp-rocky-9.6-1.34.3-20260609005954

  9. Populate the .env file with the NKP image name by adding (appending) the following environment variables and save it

    export NKP_IMAGE=nkp-image-name

    export NKP_IMAGE=nkp-ubuntu-24.04-1.34.3-20260328040605

We are now ready to install the workload nkpflow cluster

Reservation of IPs

  1. We need to reserve two static IPs in NKP external subnet for NKP cluster with Flow CNI. See here for instructions to check and reserve static IPs in a Nutanix subnet.

  2. Reserve the first IP for NKP control plane

  3. Reserve the second IP for MetalLB distributed load balancer

Tip

Here is a sample IP reservation

Component IP
NKP Control Plane VIP 10.x.x.70
NKP MetalLB IP Range 10.x.x.71-10.x.x.71

Deploy NKP Workload Cluster

We will create the workload cluster's cluster definition manifest first, modify values and proceed to deploy the workload cluster.

  1. Using VSC Terminal, load the environment variables and its values

    source $HOME/flow/.env

  2. Open nkpmanage management clusters context

    export KUBECONFIG=nkpmanage.conf

  3. Create the cluster configuration file for workload nkpflow cluster

    nkp create cluster nutanix -c ${NKP_CLUSTER_NAME} \
  --control-plane-endpoint-ip ${CONTROLPLANE_VIP} \
  --control-plane-prism-element-cluster ${NUTANIX_CLUSTER} \
  --control-plane-subnets ${NUTANIX_SUBNET_NAME} \
  --control-plane-vm-image ${NKP_IMAGE} \
  --csi-storage-container ${STORAGE_CONTAINER} \
  --endpoint https://${NUTANIX_ENDPOINT}:9440 \
  --worker-prism-element-cluster ${NUTANIX_CLUSTER} \
  --worker-subnets ${NUTANIX_SUBNET_NAME} \
  --worker-vm-image ${NKP_IMAGE} \
  --ssh-public-key-file ${SSH_PUBLIC_KEY} \
  --kubernetes-service-load-balancer-ip-range ${LB_IP_RANGE} \
  --control-plane-disk-size 200 \
  --control-plane-memory ${CONTROL_PLANE_MEMORY_GIB} \
  --control-plane-vcpus ${CONTROL_PLANE_VCPUS} \
  --control-plane-cores-per-vcpu ${CONTROL_PLANE_CORES_PER_VCPU} \
  --worker-disk-size 200 \
  --worker-memory ${WORKER_MEMORY_GIB} \
  --worker-vcpus ${WORKER_VCPUS} \
  --worker-cores-per-vcpu ${WORKER_CORES_PER_VCPU} \
  --csi-file-system ${CSI_FILESYSTEM} \
  --csi-hypervisor-attached-volumes=${CSI_HYPERVISOR_ATTACHED} \
  --registry-mirror-url "https://registry-1.docker.io" \
  --registry-mirror-username ${DOCKER_USERNAME} \
  --registry-mirror-password ${DOCKER_PASSWORD} \
  --kubernetes-pod-network-cidr 192.168.0.0/16     
  --kubernetes-service-cidr 10.96.0.0/12     
  --insecure 
  --dry-run     
  --output=yaml > cluster-nkpflow-install.yaml

    nkp create cluster nutanix -c nkpflow \
  --control-plane-endpoint-ip 10.x.x.70 \
  --control-plane-prism-element-cluster pe \
  --control-plane-subnets NKP \
  --control-plane-vm-image \
  --csi-storage-container default \
  --endpoint https://pc.example.com:9440 \
  --worker-prism-element-cluster pe \
  --worker-subnets NKP \
  --worker-vm-image nkp-rocky-9.6-1.34.3-20260609005954 \
  --ssh-public-key-file ~/.ssh/id_rsa.pub \
  --kubernetes-service-load-balancer-ip-range 10.x.x.71-10.x.x.71 \
  --control-plane-disk-size 200 \
  --control-plane-memory 16 \
  --control-plane-vcpus 12 \
  --control-plane-cores-per-vcpu 1 \
  --worker-disk-size 200 \
  --worker-memory 16 \
  --worker-vcpus 12 \
  --worker-cores-per-vcpu 1 \
  --csi-file-system ext4 \
  --csi-hypervisor-attached-volumes=true \
  --registry-mirror-url "https://registry-1.docker.io" \
  --registry-mirror-username _your_docker_username \
  --registry-mirror-password _your_docker_password \
  --kubernetes-pod-network-cidr 192.168.0.0/16     
  --kubernetes-service-cidr 10.96.0.0/12     
  --insecure 
  --dry-run     
  --output=yaml > cluster-nkpflow-install.yaml

  4. Download the pull secret for Flow CNI container images from Nutanix Portal > Downloads > Flow Networking and Security

  5. Choose Flow CNI form the drop-down menu > Click on Manage Access Tokens

  6. Copy any username and password and append to the .env file

    export FLOW_DOCKER_USERNAME=_your_docker_username
export FLOW_DOCKER_PASSWORD=_your_docker_password_pat

    export FLOW_DOCKER_USERNAME=svcpubflowcni
export FLOW_DOCKER_PASSWORD=xXXXXXXXXXXXXXXXXXXXXXXXX

  7. Create base64 value for the download secret

    export FLOW_SECRET=$(echo '${FLOW_DOCKER_USERNAME}:${FLOW_DOCKER_USERNAME}' | base64)

    export FLOW_SECRET=$(echo -n  'svcpubflowcni:xXXXXXXXXXXXXXXXXXXXXXXXXxXXXXXXXXXXXXXXXXXXXXXXXX'| base64)

  8. Create the HelmChartProxy manifest to append to the cluster-nkpflow-install.yaml manifest file 

    cat <<EOF >> cluster-nkpflow-install.yaml

---
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: flow-cni
  namespace: <namespace-cluster>
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: ${NKP_CLUSTER_NAME}
  repoURL: https://nutanix.github.io/helm-releases/
  chartName: nutanix-flow-cni
  version: 1.0.0
  namespace: flow-cni-system
  options:
    waitForJobs: true
    wait: true
    timeout: 30m
    install:
      createNamespace: true
  valuesTemplate: |
    nutanix-core-flow-ovn-kubernetes:
      k8sAPIServer: "https://${CONTROLPLANE_VIP}:6443" 
      podNetwork: "192.168.0.0/16/24" 
      serviceNetwork: "10.96.0.0/12"
    global:
      dockerConfigSecret:
        registry: docker.io
        auth: ${FLOW_SECRET}
        create: true
      imagePullSecretName: "flow-cni-secret"
    imagePullSecrets:
      - name: flow-cni-secret
EOF

    < SNIPPED FOR BREVITY>
< Contents of cluster-nkpflow-install.yaml file>

            imagePullSecrets:
      - name: flow-cni-Secret
---
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: flow-cni
  namespace: <namespace-cluster>
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: nkpflow
  repoURL: https://nutanix.github.io/helm-releases/
  chartName: nutanix-flow-cni
  version: 1.0.0
  namespace: flow-cni-system
  options:
    waitForJobs: true
    wait: true
    timeout: 30m
    install:
      createNamespace: true
  valuesTemplate: |
    nutanix-core-flow-ovn-kubernetes:
      k8sAPIServer: "https://10.x.x.220:6443" 
      podNetwork: "192.168.0.0/16/24" 
      serviceNetwork: "10.96.0.0/12"
    global:
      dockerConfigSecret:
        registry: docker.io
        auth: xXXXXXXXXXXXXXXXXXXXXXXXXxXXXXXXXXXXXXXXXXXXXXXXXX=
        create: true
      imagePullSecretName: "flow-cni-secret"
    imagePullSecrets:
      - name: flow-cni-secret

  9. Remove the default Cilium definition from the cluster-nkpflow-install.yaml file as we will be installing Flow CNI

    yq eval -i 'select(.kind == "Cluster").spec.topology.variables[] | select(.name == "addons").value.cni del' cluster-${NKP_CLUSTER_NAME}-install.yaml

    yq eval -i 'select(.kind == "Cluster").spec.topology.variables[] | select(.name == "addons").value.cni del' cluster-nkpflow-install.yaml

  10. Open cluster-nkpflow-install.yaml file and visually inspect for any formatting issues or incorrect values

  11. Create the nkpflow workload cluster with Flow CNI

    Warning

    The following workload cluster creation must be done in the management nkpmanage cluster's context.

    Ensure the context for nkpmanage before proceeding. 

    export KUBECONFIG=nkpmanage.conf
kubectl apply -f cluster-nkpflow-install.yaml

    $ kubectl apply -f cluster-nkpflow-install.yaml
#
secret/flow-pc-credentials created
secret/flow-pc-credentials-for-csi created
secret/flow-pc-credentials-for-konnector-agent created
secret/flow-image-registry-credentials created
cluster.cluster.x-k8s.io/flow created
helmchartproxy.addons.cluster.x-k8s.io/flow-cni created

  12. Monitor the progress of the nkpflow cluster creation and wait until all conditions are True

    Note

    This will take about 5 minutes.

    while test true;do nkp describe cluster -c flow -n kommander-default-workspace; sleep 5;clear;done

    Cluster/flow                                                                True                     76s
├─ClusterInfrastructure - NutanixCluster/flow-gp7rc
├─ControlPlane - KubeadmControlPlane/flow-zfck9                             True                     76s
│ └─Machine/flow-zfck9-57vxk                                                True                     2m12s
│   └─MachineInfrastructure - NutanixMachine/flow-zfck9-57vxk
└─Workers
  └─MachineDeployment/flow-md-0-4v9w6                                       True                     20s
    ├─Machine/flow-md-0-4v9w6-b26pn-8qgq6                                   True                     70s
    │ └─MachineInfrastructure - NutanixMachine/flow-md-0-4v9w6-b26pn-8qgq6
    └─Machine/flow-md-0-4v9w6-b26pn-vbqwv                                   True                     67s
      └─MachineInfrastructure - NutanixMachine/flow-md-0-4v9w6-b26pn-vbqwv

  13. Once the cluster creation is finished, get the kubeconfig file for the nkpflow cluster (if it is not already present in the directory)

    Use the workspace namespace where the nkpflow cluster was deployed
    export KUBECONFIG=nkpmanage.conf
nkp get kubeconfig -c nkpflow -n _namespace_ > nkpflow.conf

    export KUBECONFIG=nkpmanage.conf
nkp get kubeconfig -c nkpflow -n kommander-default-workspace > nkpflow.conf

  14. Change context to nkpflow cluster and check status

    export KUBECONFIG=nkpflow.conf
kubectl get nodes

    kubectl get all -n ovn-kubernetes


    $ kubectl get nodes
#
NAME                          STATUS   ROLES           AGE   VERSION
flow-md-0-4v9w6-b26pn-8qgq6   Ready    <none>          28m   v1.34.3
flow-md-0-4v9w6-b26pn-vbqwv   Ready    <none>          28m   v1.34.3
flow-zfck9-57vxk              Ready    control-plane   28m   v1.34.3

    $ kubectl get all -n ovn-kubernetes
#
NAME                                  READY   STATUS    RESTARTS   AGE
pod/ovnkube-db-8cbf9bf6-kclm5         2/2     Running   0          28m
pod/ovnkube-master-64cbc8c88d-nnrbf   2/2     Running   0          28m
pod/ovnkube-node-hvw7f                3/3     Running   0          28m
pod/ovnkube-node-w9qnz                3/3     Running   0          28m
pod/ovnkube-node-xlzqm                3/3     Running   0          28m
pod/ovs-node-29zmx                    1/1     Running   0          28m
pod/ovs-node-dln5j                    1/1     Running   0          28m
pod/ovs-node-g9f2q                    1/1     Running   0          28m

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/ovnkube-db   ClusterIP   None         <none>        6641/TCP,6642/TCP   28m

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/ovnkube-node   3         3         3       3            3           kubernetes.io/os=linux   28m
daemonset.apps/ovs-node       3         3         3       3            3           kubernetes.io/os=linux   28m

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ovnkube-db       1/1     1            1           28m
deployment.apps/ovnkube-master   1/1     1            1           28m

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/ovnkube-db-8cbf9bf6         1         1         1       28m
replicaset.apps/ovnkube-master-64cbc8c88d   1         1         1       28m

We have now deployed nkpflow cluster with Flow CNI. We can now proceed to onboarding this cluster from Nutanix Prism Central and creating required network components.