Replicating and Recovering Application to a Different K8S Cluster
In this section we wil snapshot the Application components, replicate it to a second K8S cluster and recover.
Design
In this section there is one PC and two individual PE connected.
Each PE has a NKP cluster. Each PE has a Files Server.
We will be replicating an application (workload) from one NKP cluster in one PC to another NKP cluster in a different PC. The workload stores data in both Volumes and Dynamic Files CSI.
| # | PC | PE | NKP Cluster | Files Server |
|---|---|---|---|---|
| Source | PC-1 | PE-1 | nkpprimary |
filesprimary |
| Destination | PE-2 | nkpsecondary |
filessecondary |
The following is the flow of the application recovery.
stateDiagram-v2
direction LR
state PC1 {
[*] --> PE1
PE1 --> nkpprimary
nkpprimary --> SourceApp
SourceApp --> VolumesRWO1
VolumesRWO1 --> FilesRWM1
}
state PC2 {
[*] --> PE2
PE2 --> nkpsecondary
nkpsecondary --> DestinationApp
DestinationApp --> VolumesRWO2
VolumesRWO2 --> FilesRWM2
}
[*] --> PC1
PC1 --> PC2
PC2 --> [*]
Setup Destination NKP in Primary PC/PE/K8s
Make sure to name your NKP cluster appropriately so it is easy to identify
For the purposes of this lab, we will call the source NKP cluster as nkpsecondary
Follow instructions in NKP Deployment to setup destination/secondary NKP K8s cluster.
Replication Custom Resources
The lab includes configuring the following NDK custom resources:
| Custom Resource | Purpose |
|---|---|
StorageCluster |
Defines the Nutanix storage fabric and UUIDs for secondary NKP cluster |
Remote |
Defines a target Kubernetes cluster for replication on the target NKP cluster |
ReplicationTarget |
Specifies where to replicate an application snapshot. |
FileServerReplicationRelationships |
Specfies where to replicate a Files share |
ApplicationSnapshot |
Snapshot of the application and chosen resources |
ApplicationSnapshotContent |
Stores content of the application snapshot (location in Nutanix HCI) |
ApplicationSnapshotReplication |
Triggers snapshot replication to another cluster. |
ApplicationSnapshotRestore |
Restores an application snapshot. |
Configure Availability Zones on PCs
To enable replication between two PC and underlying PE, we will need to configure Availability Zone bi-directionally.
- Logon to Primary PC and go to Administration > Availability Zones
- Click on Connect to Availability Zone
-
Choose Physical Location and enter the Secondary PC details
- IP Address for Remote PC - 10.x.x.x
- Username - admin
- Password - xxxxxxxxxxx
-
Click on Connect
- Confirm addition of remote PC
- Repeat steps 1 - 5 on the remote PC to configure access to Primary PC
Install NDK on Secondary NKP Cluster
- Login to VSCode Terminal
-
Set you NKP cluster KUBECONFIG
-
Test connection to
nkpsecondarycluster~ ❯ kubectl get nodes NAME STATUS ROLES AGE VERSION nkpsec-md-0-fdrzg-clvf9-2gnqc Ready <none> 24h v1.32.3 nkpsec-md-0-fdrzg-clvf9-9msmd Ready <none> 24h v1.32.3 nkpsec-md-0-fdrzg-clvf9-hnjlm Ready <none> 24h v1.32.3 nkpsec-md-0-fdrzg-clvf9-t8t4l Ready <none> 24h v1.32.3 nkpsec-rhdh8-2xs7z Ready control-plane 24h v1.32.3 nkpsec-rhdh8-srm6h Ready control-plane 24h v1.32.3 nkpsec-rhdh8-wxbd9 Ready control-plane 24h v1.32.3 -
Install NDK
-
Check if all NDK custom resources are running (4 of 4 containers should be running inside the
ndk-controller-mangerpod)Active namespace is "ntnx-system". ~ ❯ kubectl get all -l app.kubernetes.io/name=ndk NAME READY STATUS RESTARTS AGE pod/ndk-controller-manager-57fd7fc56b-gg5nl 4/4 Running 0 19m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ndk-controller-manager-metrics-service ClusterIP 10.109.134.126 <none> 8443/TCP 19m service/ndk-intercom-service LoadBalancer 10.99.216.62 10.122.7.212 2021:30258/TCP 19m service/ndk-scheduler-webhook-service ClusterIP 10.96.174.148 <none> 9444/TCP 19m service/ndk-webhook-service ClusterIP 10.107.189.171 <none> 443/TCP 19m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ndk-controller-manager 1/1 1 1 19m NAME DESIRED CURRENT READY AGE replicaset.apps/ndk-controller-manager-57fd7fc56b 1 1 1 19m -
Get
uuidof secondary PC and PE using the following command -
Add (append) the following environment variables and save it
-
Note and export the external IP assigned to the NDK intercom service on the Primary Cluster
-
Add (append) the following environment variables file
$HOME/ndk/.envand save it -
Source the
.envfile -
Create the StorageCluster custom resource
-
Find and configure secondary NDK IP and port number
Now we are ready to create local cluster snapshots and snapshot restores using the following NDK custom resources:
- 1
ApplicationSnapshot(created in the previous section) - 2
ApplicationSnapshotReplicationand - 3
ApplicationSnapshotRestore
NDK Recover to the Secondary NKP Cluster
Since we have a sample workload configured on the primary NKP cluster, we will:
- Configure remote NKP cluster on the primary NKP cluster (using
RemoteReplicationTargetcustom resources) - Replicate the snapshot of the sample workload from the primary NKP to secondary NKP (using
ApplicationSnapshotReplicationcustom resource) - Restore the replicated snapshot on the secondary NKP to get the workloads (using
ApplicationSnapshotRestorecustom resource)
Create Remote Cluster on Primary NKP Cluster
-
Switch context to primary NKP cluster
nkpprimary -
Create the Remote resource on the primary NKP cluster
-
Make sure the
Remotecluster is healthy -
Create the replication target on the primary NKP cluster
-
Make sure the
ReplicationTargetis healthy -
Replicate the Snapshot to the Replication Cluster
k apply -f -<<EOF apiVersion: dataservices.nutanix.com/v1alpha1 kind: ApplicationSnapshotReplication metadata: name: wordpress-snap-replication namespace: default spec: applicationSnapshotName: wordpress-app-snapshot # (1)! replicationTargetName: ${NDK_REPLICATION_CLUSTER_NAME} EOF- Use the snapshot object's name from the previous section
wordpress-app-snapshot
- Use the snapshot object's name from the previous section
-
Monitor the progress of the replication and make sure to complete it
Recover Application in Remote NKP Cluster
The following NDK objects will be used in this section
- 1
ApplicationSnapshot(created in the previous lab) - 2
ApplicationSnapshotReplication(created in the previous section) - 3
ApplicationSnapshotRestore
Recover from Replicated Snapshot
-
Switch context to secondary NKP cluster
nkpsecondary -
Confirm if the
ApplicationSnapshothas been replicated in thedefaultnamespace- We are specifically using
defaultnamespace as this is where NDK will replicate the snapshot to
- We are specifically using
-
Restore the replicated
ApplicationSnapshot -
Monitor the restore
-
Monitor the restore steps to understand the flow
NAME SNAPSHOT-NAME COMPLETED status: completed: true conditions: - lastTransitionTime: "2025-07-16T21:57:53Z" message: All prechecks passed and finalizers on dependent resources set observedGeneration: 1 reason: PrechecksPassed status: "True" type: PrechecksPassed - lastTransitionTime: "2025-07-16T21:59:00Z" message: All eligible application configs restored observedGeneration: 1 reason: ApplicationConfigRestored status: "True" type: ApplicationConfigRestored - lastTransitionTime: "2025-07-16T21:59:15Z" message: All eligible volumes restored observedGeneration: 1 reason: VolumesRestored status: "True" type: VolumesRestored - lastTransitionTime: "2025-07-16T21:59:15Z" message: Application restore successfully finalised observedGeneration: 1 reason: ApplicationRestoreFinalised status: "True" type: ApplicationRestoreFinalised finishTime: "2025-07-16 21:59:15" startTime: "2025-07-16 21:57:52" -
Verify if Wordpress app resources (pod,pvc, etc) are restored to the
defaultnamespace~ ❯ kubectl get all NAME READY STATUS RESTARTS AGE pod/wordpress-6bc48cbf79-862wm 1/1 Running 0 6m pod/wordpress-mysql-7bd9d456c5-hxjr7 1/1 Running 0 6m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/wordpress ClusterIP 10.109.161.32 <none> 80/TCP 6m service/wordpress-mysql ClusterIP None <none> 3306/TCP 6m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/wordpress 1/1 1 1 6m deployment.apps/wordpress-mysql 1/1 1 1 6m NAME DESIRED CURRENT READY AGE replicaset.apps/wordpress-6bc48cbf79 1 1 1 6m replicaset.apps/wordpress-mysql-7bd9d456c5 1 1 1 6m -
Login to Wordpress GUI to check if the deleted user is now present

We have successfully replicated application data to a secondary NKP cluster and recovered it using NDK.
We have used the following NDK objects to achieve our cross-namespace application recovery.
- 1
ApplicationSnapshot(created in the previous lab) - 2
ApplicationSnapshotReplication(created in the previous section) - 4
ApplicationSnapshotRestore
Cross Namespace Recovery
Since the applicationSnapshot got replicated to default namespace, we might ideally want to restore it to the same wordpress namespace.
The following NDK objects will be used in this section
- 1
ApplicationSnapshot(created in the previous lab) - 2
ApplicationSnapshotReplication(created in the previous section) - 3
ReferenceGrant - 4
ApplicationSnapshotRestore
Info
NDK offers cross-namespace recovery capabilites. With this NKP or any supported kubernetes platform administrator can recover applicationSnapshot custom resource to a different namespace.
We will go through the process of cross-namespace restore in this section.
-
Create a referenceGrant resouce to grant permission to restore specific application snapshots from one namespace to another.
kubectl apply -f - <<EOF apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: _reference_grant_name namespace: _source_namespace spec: from: - group: dataservices.nutanix.com kind: ApplicationSnapshotRestore namespace: _target_namespace to: - group: dataservices.nutanix.com kind: ApplicationSnapshot name: _appplication_snapshot_name EOFkubectl apply -f - <<EOF apiVersion: gateway.networking.k8s.io/v1beta1 kind: ReferenceGrant metadata: name: wordpress-cross-ns-rg namespace: default spec: from: - group: dataservices.nutanix.com kind: ApplicationSnapshotRestore namespace: wordpress to: - group: dataservices.nutanix.com kind: ApplicationSnapshot name: wordpress-app-snapshot EOF -
Create a target namespace
-
Create and
applicationSnapshotRestorecustom resource to restore to targetwordpressnamespace -
Monitor the progress of
ApplicationSnapshotRestorecustom resourceWait until the status of ApplicationSnapshotRestore custom resource is trueName: wordpress-cross-ns-asr Namespace: restore API Version: dataservices.nutanix.com/v1alpha1 Kind: ApplicationSnapshotRestore Metadata: Creation Timestamp: 2025-07-08T05:40:31Z Finalizers: dataservices.nutanix.com/application-restore Generation: 1 Resource Version: 7484636 Spec: Application Snapshot Name: app1-snap Application Snapshot Namespace: default Status: Completed: true Conditions: Last Transition Time: 2025-07-08T05:40:31Z Message: Observed Generation: 1 Reason: RequestCompleted Status: False Type: Progressing Last Transition Time: 2025-07-08T05:40:31Z Message: All prechecks passed and finalizers on dependent resources set Observed Generation: 1 Reason: PrechecksPassed Status: True Type: PrechecksPassed Last Transition Time: 2025-07-08T05:40:31Z Message: Restore requests for all eligible volumes submitted Observed Generation: 1 Reason: VolumeRestoreRequestsSubmitted Status: True Type: VolumeRestoreRequestsSubmitted Last Transition Time: 2025-07-08T05:41:32Z Message: All eligible application configs restored Observed Generation: 1 Reason: ApplicationConfigRestored Status: True Type: ApplicationConfigRestored Last Transition Time: 2025-07-08T05:41:47Z Message: All eligible volumes restored Observed Generation: 1 Reason: VolumesRestored Status: True Type: VolumesRestored Last Transition Time: 2025-07-08T05:41:47Z Message: Application restore successfully finalised Observed Generation: 1 Reason: ApplicationRestoreFinalised Status: True Type: ApplicationRestoreFinalised Finish Time: 2025-07-08 05:41:47 Start Time: 2025-07-08 05:40:31 -
Verify if Wordpress app resources (pod,pvc, etc) are restored to the
defaultnamespace~ ❯ kubectl get all NAME READY STATUS RESTARTS AGE pod/wordpress-6bc48cbf79-862wm 1/1 Running 0 6m pod/wordpress-mysql-7bd9d456c5-hxjr7 1/1 Running 0 6m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/wordpress ClusterIP 10.109.161.32 <none> 80/TCP 6m service/wordpress-mysql ClusterIP None <none> 3306/TCP 6m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/wordpress 1/1 1 1 6m deployment.apps/wordpress-mysql 1/1 1 1 6m NAME DESIRED CURRENT READY AGE replicaset.apps/wordpress-6bc48cbf79 1 1 1 6m replicaset.apps/wordpress-mysql-7bd9d456c5 1 1 1 6m -
Login to Wordpress GUI to check if the deleted user is now present

We have used the following NDK objects to achieve our cross-namespace application recovery.
- 1
ApplicationSnapshot(created in the previous lab) - 2
ApplicationSnapshotReplication(created in the previous section) - 3
ReferenceGrant - 4
ApplicationSnapshotRestore