Skip to content

Deploying Nutanix Enterprise AI

Version 2.8.0

This version of the NAI deployment is based on the Nutanix Enterprise AI (NAI) v2.8.0 release.

Prepare for NAI Deployment

GA Software with NAI v2.8.0

In this lab, we will deploy GA version of the following software to test the following:

  • Nutanix Enterprise AI

    • Unified Endpoints - multiple endpoints for HA and token-based rate limiting
    • Providers - Add remote endpoints from providers to utilize their models in Nutanix Enterprise AI workloads.

Info

Changes in NAI v2.8.0

  • Kserve is of at least of v0.19.0
  • Cert-manager is at least of v1.17.2
  • OpenTelemetry operator is at least of v0.114.1
  • Envoy Gateway is at least of v1.8.1
  • Prometheus Monitoring is at least of 82.13.6
  • CloudNativePG Operator is ar least of 0.28.0 [usually pre-installed with NKP]
  • LeaderWorkerSet is at least of 0.8.0

Enable Pre-requisite Applications

Warning

Make sure to license NKP cluster with at least NKP Pro License to make use of the NKP Applications catalog to provision NAI (and other applications)

Prometheus

The following pre-requisite applications will be enabled on NKP GUI:

Note

In this lab, we will be using the Management Cluster Workspace to deploy our Nutanix Enterprise AI (NAI)

However, in a customer environment, it is recommended to use a separate workload NKP cluster.

Search and Enable the following applications: follow this order to install dependencies for NAI application

  1. In the NKP GUI, Go to Clusters
  2. Click on Management Cluster Workspace

  3. Go to Applications to search and enable the following:

    • Prometheus Monitoring : version 82.13.6 or higher with the following Values configuration

  4. Wait for Deployed state in the GUI

Cert Manager

Cert Manager is pre-installed on all NKP Clusters. If not installed, use the following method to install:

  1. In the NKP GUI, Go to Clusters
  2. Click on Management Cluster Workspace

  3. Go to Applications to search and enable the following:

    • Cert-manager- at least v1.17.2

  4. Wait for Deployed state in the GUI

Envoy Gateway

  1. In the NKP GUI, Go to Clusters
  2. Click on Management Cluster Workspace

  3. Go to Applications to search and enable the following:

    • NAI - Envoy Gateway : version v1.8.1 or higher with the following Values configuration 

      config:
  envoyGateway:
    gateway:
      controllerName: "gateway.envoyproxy.io/gatewayclass-controller"
    logging:
      level:
        default: "info"
    provider:
      kubernetes:
        rateLimitDeployment:
          container:
            image: "docker.io/envoyproxy/ratelimit:1e50889b"
          patch:
            type: "StrategicMerge"
            value:
              spec:
                template:
                  spec:
                    containers:
                      - imagePullPolicy: "IfNotPresent"
                        name: "envoy-ratelimit"
                        image: "docker.io/envoyproxy/ratelimit:1e50889b"
                        env:
                          - name: REDIS_TYPE
                            value: "sentinel"
                          - name: REDIS_PIPELINE_WINDOW
                            value: "150us"
      type: "Kubernetes"
    extensionApis:
      enableEnvoyPatchPolicy: true
      enableBackend: true
    extensionManager:
      maxMessageSize: 11Mi
      backendResources:
        - group: inference.networking.k8s.io
          kind: InferencePool
          version: v1
      hooks:
        xdsTranslator:
          translation:
            listener:
              includeAll: true
            route:
              includeAll: true
            cluster:
              includeAll: true
            secret:
              includeAll: true
          post:
            - "Translation"
            - "Cluster"
            - "Route"
      service:
        fqdn:
          hostname: "ai-gateway-controller.nai-system.svc.cluster.local"
          port: 1063
    rateLimit:
      backend:
        type: "Redis"
        redis:
          url: "mymaster,nai-valkey-sentinel.nai-system.svc.cluster.local:26379"

  4. Check if Envoy Gateway resources are ready either in the GUI by watching for Deployed state or in the commandline as follows:

    Warning

    The envoy-ratelimit- pod will temporarily be in CrashLoopBackOff state and eventually will transition to Running after redis-standalone pod is fully deployed in the upcoming Deploy NAI section.

    Ignore the CrashLoopBackOff state for now and move on to the next section. 

    kubectl get po -n envoy-gateway-system

    $ kubectl get pods -n envoy-gateway-system
#
NAME                               READY   STATUS    RESTARTS      AGE
envoy-gateway-5c8b5fd5fb-zwhwz     1/1     Running   5 (90m ago)   5m
envoy-ratelimit-6b4657bddd-5zzms   1/1     Running   9 (89m ago)   5m

Kserve

Kserve and Kserve LLMInferenceService

Note that Kserve and Kserve LLMInferenceService are bundled in the same Kserve NKP catalog application.

Installing Kserve will also install Kserve LLMInferenceService CRDs and resources.

For more information, see catalog application packaging here on Github.

  1. In the NKP GUI, Go to Clusters
  2. Click on Management Cluster Workspace

  3. Go to Applications to search and enable the following:

    • Kserve : version v0.19.0 or higher with the following Values configuration 

      kserve:
  controller:
    deploymentMode: RawDeployment
    gateway:
      disableIngressCreation: true

  4. Check if Kserve and Kserve LLMInferenceService resources are ready either in the GUI by watching for Running state or in the commandline as follows:

    kubens kserve
kubectl get pods    

    $ kubectl get pods 
#
NAME                                         READY   STATUS    RESTARTS   AGE
kserve-controller-manager-857dcfb7d8-fqmpw   2/2     Running   0          4m
llmisvc-controller-manager-cf84cf6db-mwftz   1/1     Running   0          4m

CloudNativePG

Note

NKP will have CloudNativePG pre-installed as a part of regular install. Check in the Applications Catalog of the NKP cluster for its presence.

Check for CloudNativePG v0.28.0, if present, skip this section.

  1. In the NKP GUI, Go to Clusters
  2. Click on Management Cluster Workspace

  3. Go to Applications to search and enable the following:

    • CloudNativePG : version 0.28.0

LeaderWorkerSet

LeaderWorkerSet (LWS) is an open-source, custom Kubernetes API designed to deploy and manage multi-node AI/ML workloads—such as large language model (LLM) distributed inference and training—as a single, cohesive unit. It automatically groups a collection of pods into a specific topology consisting of one leader pod and multiple worker pods, managing their entire lifecycle simultaneously so that if a single pod fails, the entire group restarts together to prevent data inconsistency. Furthermore, LWS simplifies network communication across these nodes by automatically injecting group environment variables and optimizing pod placement within the same network topology to guarantee high-throughput, low-latency data transfers between GPUs.

  1. In the NKP GUI, Go to Clusters
  2. Click on Management Cluster Workspace

  3. Go to Applications to search and enable the following:

    • LeaderWorkerSet : version 0.8.0

Opentelemetry

  1. In the NKP GUI, Go to Clusters
  2. Click on Management Cluster Workspace

  3. Go to Applications to search and enable the following:

    • Opentelemetry Operator : version v0.114.1

Deploy NAI

We will use the Docker login credentials we created in the previous section to download the NAI Docker images.

Deploy NAI Profiles using Helm

NAI v2.8.0 onwards has support for profiles for different capacity of NAI use cases.

Use the Helm method here to install these profiles depending on your requirements.

Name Capacity
Default 300 concurrent requests and 100 API Keys
c1k_k200 1000 concurrent requests and 200 API Keys
c5k_k1k 5000 concurrent requests and 1000 API Keys

Change the Docker login credentials

The following Docker based environment variable values need to be changed from your own Docker environment variables to the credentials downloaded from Nutanix Portal.

  • $DOCKER_NAI_USERNAME
  • $DOCKER_NAI_ASSWORD
  • $DOCKER_NAI_EMAIL

  1. Open $HOME/.env file in VSCode

  2. Add (append) the following environment variables and save it

    export NAI_USER=_your_desired_nai_ui_username
export NAI_TEMP_PASS=_your_desired_nai_ui_password # At least 8 characters
export REGISTRY_SECRET_NAME=_k8s_secret_for_nai
export DOCKER_SERVER=https://index.docker.io/v1/
export DOCKER_NAI_USERNAME=_GA_release_docker_username
export DOCKER_NAI_PASSWORD=_GA_release_docker_password
export DOCKER_NAI_EMAIL=_GA_release_docker_email
export NAI_CORE_VERSION=_GA_release_nai_core_version
export NAI_API_RWX_STORAGECLASS=_nkp_rwx_storage_class
export NAI_DEFAULT_RWO_STORAGECLASS=_nkp_rwo_storage_class
export NKP_WORKSPACE_NAMESPACE=_nkp_workspace name # (1)!
export CLUSTER_ISSUER=_cluster_issuer_name
export NAI_PUBLIC_DOMAIN_NAME=_nai_domain_name

    1. To get the workspace namespace, run the following commmand. Note the WORKSPACE NAMESPACE column in the output

      kubectl get workspaces

      $ kubectl get workspaces
#
NAME                  DISPLAY NAME                   WORKSPACE NAMESPACE           AGE
default-workspace     Default Workspace              kommander-default-workspace   3d
kommander-workspace   Management Cluster Workspace   kommander                     3d

    export NAI_USER=admin
export NAI_TEMP_PASS=_Xxxxxxxxxx
export REGISTRY_SECRET_NAME=nai-regcred
export DOCKER_SERVER=https://index.docker.io/v1/
export DOCKER_NAI_USERNAME=ntnxsvcgpt
export DOCKER_NAI_PASSWORD=dckr_pat_XXXXXXXXXXXXXXXXXXXXXXXXX
export DOCKER_NAI_EMAIL=ntnxsvcgpt
export NAI_CORE_VERSION=2.8.0
export NAI_API_RWX_STORAGECLASS=nai-nfs-storage
export NAI_DEFAULT_RWO_STORAGECLASS=nutanix-volume
export NKP_WORKSPACE_NAMESPACE=kommander
export CLUSTER_ISSUER=letsencrypt-cloudflare
export NAI_PUBLIC_DOMAIN_NAME=nai.domain.com

  3. Source the environment variables

    source $HOME/.env

  4. Create the nai-system namespace to install Nutanix Enterprise AI

    kubectl create namespace nai-system --dry-run=client -o yaml | kubectl apply -f -

    kubectl create namespace nai-system --dry-run=client -o yaml | kubectl apply -f -
#
namespace/nai-system created

  5. Create docker registry Secrets in both nai-systemand envoy-gateway-system namespaces.

    kubectl -n nai-system create secret docker-registry ${REGISTRY_SECRET_NAME} \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_NAI_USERNAME} \
--docker-password=${DOCKER_NAI_PASSWORD} \
--docker-email=${DOCKER_NAI_EMAIL} \
--dry-run=client -o yaml | kubectl apply -f -

    kubectl -n envoy-gateway-system create secret docker-registry ${REGISTRY_SECRET_NAME} \
 --docker-server=${DOCKER_SERVER} \
 --docker-username=${DOCKER_NAI_USERNAME} \
 --docker-password=${DOCKER_NAI_PASSWORD} \
 --docker-email=${DOCKER_NAI_EMAIL} \
 --dry-run=client -o yaml | kubectl apply -f -


    secret/nai-regcred configured

    secret/nai-regcred configured

  6. In the NKP GUI, Go to Clusters

  7. Click on Management Cluster Workspace

  8. Create a template file with Values configuration

    cat << EOF > nai-core-values.yaml
global:
  imagePullSecrets:
    - name: ${REGISTRY_SECRET_NAME}
  storage:
    storageClassNameRWX: ${NAI_API_RWX_STORAGECLASS}
    storageClassName: ${NAI_DEFAULT_RWO_STORAGECLASS}

naiMonitoring:
  nodeExporter:
    serviceMonitor:
      namespaceSelector:
        matchNames:
          - ${NKP_WORKSPACE_NAMESPACE}
  dcgmExporter:
    serviceMonitor:
      namespaceSelector:
        matchNames:
          - ${NKP_WORKSPACE_NAMESPACE}
naiApi:
  superAdmin:
    username: ${NAI_UI_USER}
    password: ${NAI_TEMP_PASS}      # At least 8 characters
    # email: admin@nutanix.com
    # firstName: admin
gateway:
  tlsSecretName: "nai-cert"         # secret name written by cert-manager
  certManager:
    selfSigned: true                # enables self-signed issuer + certificate

# Optional - use if you are using cert-manager and ClusterIssuer with your own domain 
# gateway:
#  certManager:
#    issuerRef:
#      name: letsencrypt-cloudflare # ClusterIssuer must be existing
#      kind: ClusterIssuer
#    dnsNames:
#      - nai.domain.com
EOF

    global:
  imagePullSecrets:
    - name: nai-regcred
  storage:
    storageClassNameRWX: nai-nfs-storage
    storageClassName: nutanix-volume

naiMonitoring:
  nodeExporter:
    serviceMonitor:
      namespaceSelector:
        matchNames:
          - kommander
  dcgmExporter:
    serviceMonitor:
      namespaceSelector:
        matchNames:
          - kommander

naiApi:
  superAdmin:
    username: admin
    password: _XXXXXXXXX # At least 8 characters
    # email: admin@nutanix.com
    # firstName: admin

gateway:
  tlsSecretName: "nai-cert"         # secret name written by cert-manager
  certManager:
    selfSigned: true                # enables self-signed issuer + certificate

# Optional - use if you are using cert-manager and ClusterIssuer with your own domain
# gateway:
#  certManager:
#    issuerRef:
#      name: letsencrypt-cloudflare
#      kind: ClusterIssuer
#    dnsNames:
#      - nai.domain.com

  9. Go to Applications to search and enable the following:

    • Nutanix Enterprise AI : version v2.8.0 or higher with contents of nai-core-values.yaml file from previous step.

  10. Check if NAI resources are ready either in the GUI by watching for Deployed state or in the commandline as follows:

    Note

    This operation will take at least 5 - 8 minutes depending on the resources available.

    The NKP Catalog application will deploy nai-operators first before proceeding to install NAI resources.

    kubens nai-system
kubectl get pods    

    Active namespace is "nai-system".

NAME                                                    READY   STATUS      RESTARTS   AGE
ai-gateway-controller-6fff98cbd6-lv8dd                   1/1     Running     0          67m
chi-nai-clickhouse-server-chcluster1-0-0-0               1/1     Running     0          64m
chk-nai-clickhouse-keeper-chkeeper-0-0-0                 1/1     Running     0          63m
iam-database-bootstrap-pdzxh-7j9m7                       0/1     Completed   0          64m
iam-proxy-5bd954bb85-qmdwx                               1/1     Running     0          64m
iam-proxy-control-plane-5ff697cdfb-5pnlz                 1/1     Running     0          64m
iam-themis-849478f448-7c6bx                              2/2     Running     0          60m
iam-themis-bootstrap-e5pyk-pl96p                         0/1     Completed   0          64m
iam-ui-7476458cb-q8rns                                   1/1     Running     0          64m
iam-user-authn-57d8b4dd67-nsdt5                          2/2     Running     0          60m
nai-agent-5d7b86946d-8b8jn                               1/1     Running     0          64m
nai-api-86d58c7b6f-5fg4h                                 1/1     Running     0          64m
nai-api-db-migrate-vk5y2-2dfr8                           0/1     Completed   5          64m
nai-clickhouse-schema-job-1788408841-hvwl8               0/1     Completed   0          64m
nai-db-iep-1                                             1/1     Running     0          66m
nai-oauth2-proxy-595d65db9c-lt6bs                        1/1     Running     0          64m
nai-operators-nai-clickhouse-operator-687479c97b-m9hn7   2/2     Running     0          67m
nai-otel-collector-collector-2b5xm                       1/1     Running     0          64m
nai-otel-collector-collector-5xj4k                       1/1     Running     0          64m
nai-otel-collector-collector-7jdln                       1/1     Running     0          64m
nai-otel-collector-collector-7tlz8                       1/1     Running     0          64m
nai-otel-collector-collector-b6vpl                       1/1     Running     0          64m
nai-otel-collector-collector-k52pv                       1/1     Running     0          64m
nai-otel-collector-collector-qnk9d                       1/1     Running     0          64m
nai-otel-collector-collector-w4zlf                       1/1     Running     0          64m
nai-otel-collector-targetallocator-7fcccfc477-ndgkr      1/1     Running     0          64m
nai-securityscan-manager-77679b5554-94p4z                1/1     Running     0          64m
nai-ui-7b5b9b88b4-c9wzz                                  1/1     Running     0          64m
nai-valkey-0                                             1/1     Running     0          67m
nai-valkey-sentinel-0                                    1/1     Running     0          67m

Install SSL Certificate and Gateway Elements

In this section we will install SSL Certificate to access the NAI UI. This is required as the endpoint will only work with a ssl endpoint with a valid certificate.

NAI UI is accessible using the Envoy Ingress Gateway.

Optional and manual - using Public Certificate Authority (CA)

If an organization generates certificates using a different mechanism then obtain the certificate + key and create a kubernetes secret manually using the following command:

  1. Create the certificate from the files (generated using certbot or provided to you)

    kubectl -n nai-system create secret tls nai-cert \
--cert=path/to/nai.crt \
--key=path/to/nai.key

  2. Combine the certificates to get the certificate bundle

    cat server.crt intermediate.crt > tls-bundle.crt

  3. Create a kubernetes secret in the nai-system namespace to use during NAI install

      kubectl secret tls nai-cert \
  --cert=tls-bundle.crt \
  --key=server.key \
  -n nai-system

  4. Patch the Envoy gateway with the nai-cert certificate details

    kubectl patch gateway nai-ingress-gateway -n nai-system --type='json' -p='[{"op": "replace", "path": "/spec/listeners/1/tls/certificateRefs/0/name", "value": "nai-cert"}]'
Optional to automate - using Public Certificate Authority (CA) and Cert Manager

Using Cert Manager to manage the Public Certificate Authority (CA) for NAI SSL Certificate is also a possiblity.

At a high level (Cloudflare Example):

  1. Get a API key from DNS provider woth Edit Zone rights

  2. Create a Kubernetes Secret from the API key

    apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: harbor
type: Opaque
stringData:
  api-token: _YOUR_CLOUDFLARE_API_TOKEN_HERE

    apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: null
  name: route53-api-token-secret
  namespace: cert-manager
data:
  access-key-id: "_YOUR_AWS_ACCESS_KEY_ID"
  secret-access-key: "_YOUR_AWS_SECRET_KEY_ID"

  3. Create a ClusterIssuer with Cert Mangager/Let's Encrypt - Configure cert-manager to use DNS-01 challenge with Cloudflare for automatic certificate issuance.

    apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare
  namespace: cert-manager
spec:
  acme:
    email: _YOUR_DOMAIN_OWNER_EMAIL_ADDRESS
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-cloudflare-account-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token

    apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare
spec:
  acme:
    email: _YOUR_DOMAIN_OWNER_EMAIL_ADDRESS
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: nai-letsencrypt-cluster
    solvers:
      - dns01:
          route53:
            region: us-east-1
            accessKeyIDSecretRef:
              name: route53-api-token-secret
              key: access-key-id
            secretAccessKeySecretRef:
              name: route53-api-token-secret
              key: secret-access-key
            hostedZoneID: _HOSTED_ZONE_ID

  4. Create the ingress resource certificate using the following command:

    cat << EOF | k apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nai-cert
  namespace: nai-system
spec:
  issuerRef:
    name: letsencrypt-cloudflare
    kind: ClusterIssuer
  secretName: nai-cert
  commonName: nai.domain.com
  dnsNames:
  - nai.domain.com
EOF

  5. Patch the Envoy gateway with the nai-cert certificate details

    kubectl patch gateway nai-ingress-gateway -n nai-system --type='json' -p='[{"op": "replace", "path": "/spec/listeners/1/tls/certificateRefs/0/name", "value": "nai-cert"}]'

The following steps show how cert-manager can be used to generate a self signed certificate using the default selfsigned-issuer present in the cluster for the purposes of the lab.

  1. Get the NAI UI ingress gateway host using the following command:

    NAI_UI_ENDPOINT=$(kubectl get svc -n envoy-gateway-system -l "gateway.envoyproxy.io/owning-gateway-name=nai-ingress-gateway,gateway.envoyproxy.io/owning-gateway-namespace=nai-system" -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

  2. Get the value of NAI_UI_ENDPOINT environment variable

    echo $NAI_UI_ENDPOINT

    10.x.x.216

  3. We will use the command output e.g: 10.x.x.216 as the IP address for NAI as reserved in this section

  4. Construct the FQDN of NAI UI using nip.io and we will use this FQDN as the certificate's Common Name (CN).

    nai.${NAI_UI_ENDPOINT}.nip.io

    nai.10.x.x.216.nip.io

  5. Create the ingress resource certificate using the following command:

    cat << EOF | k apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nai-cert
  namespace: nai-system
spec:
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
  secretName: nai-cert
  commonName: nai.${NAI_UI_ENDPOINT}.nip.io
  dnsNames:
  - nai.${NAI_UI_ENDPOINT}.nip.io
  ipAddresses:
  - ${NAI_UI_ENDPOINT}
EOF

  6. Patch the Envoy gateway with the nai-cert certificate details

    kubectl patch gateway nai-ingress-gateway -n nai-system --type='json' -p='[{"op": "replace", "path": "/spec/listeners/1/tls/certificateRefs/0/name", "value": "nai-cert"}]'

  7. Create EnvoyProxy

    k apply -f -<<EOF
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-service-config
  namespace: nai-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: LoadBalancer
EOF

  8. Patch the nai-ingress-gateway resource with the new EnvoyProxy details

    kubectl patch gateway nai-ingress-gateway -n nai-system --type=merge \
-p '{
    "spec": {
        "infrastructure": {
            "parametersRef": {
                "group": "gateway.envoyproxy.io",
                "kind": "EnvoyProxy",
                "name": "envoy-service-config"
            }
        }
    }
}'

Accessing the UI

  1. In a browser, open the following URL to connect to the NAI UI

    https://nai.10.x.x.216.nip.io

  2. Change the password for the admin user

  3. Login using admin user and password that was set in the install workflow

Test NAI

  1. Follow instructions here to test NAI inferencing endpoints with an LLM
  2. Follow instruction here to deploy NAI Unified Endpoints with LLM(s).