Pre-requisites for Deploying NAI
In this part of the lab we will prepare pre-requisites for LLM application on GPU nodes.
The following is the flow of the applications lab:
stateDiagram-v2
direction LR
state PreRequisites {
[*] --> CreateFilesShare
CreateFilesShare --> PrepareHuggingFace
PrepareHuggingFace --> [*]
}
state CreateOfflineHelmContainers {
[*] --> PrepareNAIHelmCharts
PrepareNAIHelmCharts --> PrepareNAIContainerImages
PrepareNAIContainerImages --> [*]
}
[*] --> PreRequisites
PreRequisites --> CreateOfflineHelmContainers
CreateOfflineHelmContainers --> DeployNAI : next section
DeployNAI --> TestNAI
TestNAI --> [*]
Prepare the following pre-requisites needed to deploy NAI on target kubernetes cluster.
Create Nutanix Files Storage Class
We will create Nutanix Files storage class which will be used to create a pvc that will store the LLama-3-8B
model files.
- In Prism Central, choose Files from the menu
- Choose the file server (e.g. labFS)
- Click on Shares & Exports
- Click on +New Share or Export
-
Fill the details of the Share
- Name - model_share
- Description - for NAI model store
- Share path - leave blank
- Max Size - 10 GiB (adjust to the model file size)
- Primary Protocol Access - NFS
-
Click Next and make sure Enable compression in checked
- Click Next
-
In NFS Protocol Access, choose the following:
- Authentication - System
- Default Access (for all clients) - Read-Write
- Squash - Root Squash
Note
Consider changing access options for Production environment
-
Click Next
- Confirm the share details and click on Create
Create the Files Storage Class
-
Run the following command to check K8S status of the
nkpdev
cluster -
In VSC Explorer, click on New File and create a config file with the following name:
Add the following content and replace the
nfsServerName
with the name of the Nutanix Files server name . -
Create the storage class
-
Check storage classes in the cluster for the Nutanix Files storage class
kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE dkp-object-store kommander.ceph.rook.io/bucket Delete Immediate false 28h nai-nfs-storage csi.nutanix.com Delete Immediate true 24h nutanix-volume (default) csi.nutanix.com Delete WaitForFirstConsumer false 28h
Request Access to Model on Hugging Face
Follow these steps to request access to the meta-llama/Meta-Llama-3.1-8B-Instruct
model:
LLM Recommendation
From testing google/gemma-2-2b-it
model is quicker to download and obtain download rights, than meta-llama/Meta-Llama-3.1-8B-Instruct
model.
Feel free to use the google/gemma-2-2b-it model if necessary. The procedure to request access to the model is the same.
-
Sign in to your Hugging Face account:
- Visit Hugging Face and log in to your account.
-
Navigate to the model page:
- Go to the Meta-Llama-3.1-8B-Instruct model page.
-
Request access:
- On the model page, you will see a section or button labeled Request Access (this is usually near the top of the page or near the "Files and versions" section).
- Click Request Access.
-
Complete the form:
- You may be prompted to fill out a form or provide additional details about your intended use of the model.
- Complete the required fields and submit the request.
-
Wait for approval:
- After submitting your request, you will receive a notification or email once your access is granted.
- This process can take some time depending on the approval workflow.
Once access is granted, there will be an email notification.
Note
Email from Hugging Face can take a few minutes or hours before it arrives.
Create a Hugging Face Token with Read Permissions
Follow these steps to create a Hugging Face token with read permissions:
-
Sign in to your Hugging Face account:
- Visit Hugging Face and log in to your account.
-
Access your account settings:
- Click on your profile picture in the top-right corner.
- From the dropdown, select Settings.
-
Navigate to the "Access Tokens" section:
- In the sidebar, click on Access Tokens.
- You will see a page where you can create and manage tokens.
-
Create a new token:
- Click the New token button.
- Enter a name for your token (i.e.,
read-only-token
).
-
Set token permissions:
- Under the permissions dropdown, select Read. For Example:
- Under the permissions dropdown, select Read. For Example:
-
Create and copy the token:
- After selecting the permissions, click Create.
- Your token will be generated and displayed only once, so make sure to copy it and store it securely.
Use this token for accessing Hugging Face resources with read-only permissions.
Prepare Helm Charts
In this section we will prepare the helm charts necessary for NAI and pre-requisite applications install
- NAI
- Envoy Gateway
- Kserve
The procedure will be done on the jumphost VM.
-
Login to Nutanix Portal using your credentials
-
Go to Downloads > NAI Airgapped Bundle
-
Download and extract the NAI air-gap
helm
bundle from the link you copied earlier- Open new
VSCode
window on your jumphost VM
- Open new
-
In
VSCode
Explorer pane, click on existing$HOME
folder -
Click on New Folder name it:
airgap-nai
-
On
VSCode
Explorer plane, click the$HOME/airgap-nai
folder -
On
VSCode
menu, selectTerminal
>New Terminal
-
Browse to
airgap-nai
directory -
In
VSC
, under the newly createdairgap-nai
folder, click on New File and create file with the following name: -
Add (append) the following environment variables and save it
Where to get associated container versions for NAI
v2.4.0
?Extract the downloaded
nai-core-2.4.0.tgz
file.Under the extracted folder path
$PWD/nai-core/values.yaml
file, we will be able to see all the associated container images and their tags for NAIv2.4.0
export KSERVE_VERSION=_your_kserve_version export ENVOY_GATEWAY_VERSION=_your_envoy_gateway_version export NAI_CORE_VERSION=_your_nai_core_version export NAI_API_VERSION=_your_nai_api_version export NAI_TGI_RUNTIME_VERSION=_your_tgi_version export NAI_PROMETHEUS_VERSION=_your_prometheus_version export NAI_POSTGRESQL_VERSION=_your_postgres_version export REGISTRY_HOST=harbor.10.x.x.111.nip.io/nkp export REGISTRY_USERNAME=admin export REGISTRY_CACERT=_path_to_ca_cert_of_registry # (1)!
- File must contain CA server and Harbor server's public certificate in one file
export KSERVE_VERSION=v0.15.0 export ENVOY_GATEWAY_VERSION=v1.5.0 export NAI_CORE_VERSION=v2.4.0 export NAI_API_VERSION=v2.4.0 export NAI_TGI_RUNTIME_VERSION="3.3.4-b2485c9" export NAI_PROMETHEUS_VERSION=v2.54.0 export NAI_POSTGRESQL_VERSION=16.1-alpine export REGISTRY_USERNAME=admin export REGISTRY_PASSWORD=xxxxxxx export REGISTRY_CACERT=$HOME/harbor/certs/full_chain.pem # (1)!
- File must contain CA server and Harbor server's public certificate in one file
-
Source the
.env
file to import environment variables -
Pull the Kserve and Envoy Gateway helm charts using the following commands
Pulled: docker.io/envoyproxy/gateway-helm:v1.5.0 Digest: sha256:a3dddd41ec3c58eae1b77dabe1f298bf92123fda6cac6f4940c23a11fc43c583 Pulled: ghcr.io/kserve/charts/kserve-crd:v0.14.0 Digest: sha256:6ae5af970d9a9400e8456ad1dbc86360d03f4b6bb00be4f16c48bc0542283d42 Pulled: ghcr.io/kserve/charts/kserve:v0.14.0 Digest: sha256:25129d39a4aa85f96159db6933729ea9f35e9d0f7f7cac7918c0a8013672eccb
-
Login to Harbor registry on the command line (if not done so)
-
Upload the downloaded and prepared helm charts to Harbor
-
Download the NAI air-gap binaries (NAI container images) from the link you copied earlier
Prepare Container Images
The Jumphost VM will be used as a medium to download the NAI container images and upload them to the internal Harbor container registry.
stateDiagram-v2
direction LR
state LoginToNutanixPortal {
[*] --> CreateDockerIDandAccessToken
CreateDockerIDandAccessToken --> LoginToDockerCLI
LoginToDockerCLI --> [*]
}
state PrepareNAIDockerImages {
[*] --> DownloadUploadImagesToHarbor
DownloadUploadImagesToHarbor --> [*]
}
[*] --> LoginToNutanixPortal
LoginToNutanixPortal --> PrepareNAIDockerImages
PrepareNAIDockerImages --> [*]
Upload NAI Docker Images to Harbor
Info
The download and upload of the container images will be done in one docker push
command which will use the internal Harbor container registry details.
This will be a two-step process.
- Upload the container images from the downloaded
nai-2.x.x.tar
to the jumphost VM local docker images store - Upload it to the internal Harbor container registry
-
Since we will be using the same internal Harbor container registry to upload container images, make sure the following environment variables are set (these were already set during air-gap NKP preparation)
-
(Optional) - To view the container images loaded in your local docker container registry, run the following command:
nutanix/nai-api:v2.4.0 nutanix/nai-inference-ui:v2.4.0 nutanix/nai-model-processor:v2.4.0 nutanix/nai-iep-operator:v2.4.0 nutanix/nai-tgi:3.3.4-b2485c9 nutanix/nai-kserve-huggingfaceserver:v0.15.2 nutanix/nai-kserve-huggingfaceserver:v0.15.2 nutanix/nai-kserve-huggingfaceserver:v0.15.2-gpu nutanix/nai-kserve-controller:v0.15.0 nutanix/nai-postgres:16.1-alpine
-
Push the images to the jumphost VM local docker images store
-
Login to the internal Harbor registy if the harbor project needs authentication
-
Tag and push all the NAI images to refer to the internal harbor registry
-
Download and push the Envoy Gateway, Kserve and Prometheus container images from the jumphost VM to harbor container registry.
for image in docker.io/envoyproxy/gateway-controller:${ENVOY_GATEWAY_VERSION} \ kserve/kserve-controller:${KSERVE_VERSION} \ quay.io/prometheus/prometheus:${NAI_PROMETHEUS_VERSION}; do docker tag $image $${REGISTRY_HOST}/$(echo $image | sed 's|.*/||') docker push $${REGISTRY_HOST}/$(echo $image | sed 's|.*/||') done
Now we are ready to deploy our NAI workloads.