Nfs-client-provisioner是一个开源的NFS 外部Provisioner,利用NFS Server为Kubernetes集群提供持久化存储。部署nfs-client-provisioner后,您可以在Kubernetes集群中动态创建PV。
Kubernetes 命令行客户端 kubectl可以让您从客户端计算机连接到 Kubernetes 集群,实现应用部署。详情参考使用Kubectl客户端连接到Kubernetes集群。
详情参考在集群中部署nfs-client-provisioner。
StorageClass为PVC提供动态发现、绑定PV的配置。因此创建一个使用nfs-client-provisioner的Storage Class,并在创建PVC时显式指定对应的StorageClassName,就可以基于CFS云文件存储在Kubernetes集群中动态创建NFS类型的PV。
wget https://kubernetes.s3.cn-north-1.jdcloud-oss.com/CFS/storageClass-With-jdcloud-cfs.yml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: auto-cfs-storage provisioner: jdcloud-cfs #nfs-client-provisioner deployment env中定义的PROVISIONER_NAME保持一致 mountOptions: - vers=3 - noresvport parameters: mountOptions: "vers=3,noresvport" archiveOnDelete: "false" #archiveOnDelete定义为false时,删除NFS Server中对应的目录,为true则保留;
备注:CFS目前暂不支持flock,如果应用需要flock支持,请开工单单独开通. 如果NFS需要支持子目录,请将vers=3变更为vers=4
kubectl create -f storageClass-With-jdcloud-cfs.yml
kubectl get storageClass auto-cfs-storage NAME PROVISIONER AGE auto-cfs-storage jdcloud-cfs 5m15s
wget https://kubernetes.s3.cn-north-1.jdcloud-oss.com/CFS/Create-PV-dynamically.yml
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: auto-pv-with-nfs-client-provisioner spec: storageClassName: auto-cfs-storage #使用nfs-client-provisioner作为NFS外部provisioner的Storage Class Name; accessModes: - ReadWriteMany resources: requests: storage: 1Mi #NFS Server中对应的挂载目录大小;目前CFS文件存储不限制挂载目录的容量;storage不超过文件存储最大容量限制即可
kubectl create -f Create-PV-dynamically.yml
kubectl get pvc auto-pv-with-nfs-client-provisioner NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE auto-pv-with-nfs-client-provisioner Bound pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 1Mi RWX auto-cfs-storage 10s kubectl describe pvc auto-pv-with-nfs-client-provisioner Name: auto-pv-with-nfs-client-provisioner Namespace: default StorageClass: auto-cfs-storage Status: Bound Volume: pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 #根据Storage Class配置动态创建的PV名称 Labels: <none> Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-provisioner: jdcloud-cfs Finalizers: [kubernetes.io/pvc-protection] Capacity: 1Mi Access Modes: RWX Events:
kubectl get pv pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 1Mi RWX Delete Bound default/auto-pv-with-nfs-client-provisioner auto-cfs-storage 15m kubectl describe pv pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 Name: pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 Labels: <none> Annotations: pv.kubernetes.io/provisioned-by: jdcloud-cfs Finalizers: [kubernetes.io/pv-protection] StorageClass: auto-cfs-storage Status: Bound Claim: default/auto-pv-with-nfs-client-provisioner Reclaim Policy: Delete Access Modes: RWX Capacity: 1Mi Node Affinity: <none> Message: Source: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 10.XX.XX.11 #CFS文件存储中对应的挂载目标IP地址 Path: /cfs/default-auto-pv-with-nfs-client-provisioner-pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 # NFS Server中由nfs-client-provisioner自动创建的子目录 ReadOnly: false Events: <none>
sudo yum install –y nfs-utils
mkdir /cfs #在云主机上创建一个新目录 mount -t nfs 172.XX.XX.10:/cfs /cfs #将nfs 172.XX.XX.10:/cfs挂载到云主机的cfs目录上,其中172.XX.XX.10请使用云文件存储的挂载目标IP替换
cd /cfs #进入云主机上的cfs目录 ls -a # 查看cfs目录中的内容 default-auto-pv-with-nfs-client-provisioner-pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 #ls输出内容,验证cfs目录下新增了子目录,子目录名称与PV Source.Path一致的目录
wget https://kubernetes.s3.cn-north-1.jdcloud-oss.com/CFS/pod-touch-cfs.yml
kind: Pod apiVersion: v1 metadata: name: pod-touch-cfs spec: containers: - name: test-pod image: busybox imagePullPolicy: IfNotPresent command: - "/bin/sh" args: - "-c" - "touch /mnt/SUCCESS && exit 0 || exit 1" #在/mnt目录下新建一个名称为SUCCESS的文件,并写入helloworld volumeMounts: - name: nfs-pvc mountPath: "/mnt" #将volume挂载到Pod的/mnt目录 restartPolicy: "Never" volumes: - name: nfs-pvc persistentVolumeClaim: claimName: auto-pv-with-nfs-client-provisioner #指定与云文件存储建立绑定关系的PVC 名称
kubectl create -f pod-touch-cfs.yml
kubectl get pod pod-touch-cfs NAME READY STATUS RESTARTS AGE pod-touch-cfs 1/1 Running 0 11s 执行exec进入pod,验证文件内容 kubectl exec -it pod-touch-cfs /bin/sh / # cat /mnt/SUCCESS helloworld
cd /cfs #进入云主机上的cfs目录 ls -a # 查看cfs目录中的内容 default-auto-pv-with-nfs-client-provisioner-pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 #ls输出内容,验证cfs目录下新增了子目录,子目录名称与PV Source.Path一致的目录 cat default-auto-pv-with-nfs-client-provisioner-pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7/SUCCESS helloworld #与PV Source.Path一致的子目录下,查看新增文件SUCCESS的内容
kubectl delete pod pod-touch-cfs pod "pod-touch-cfs" deleted
wget https://kubernetes.s3.cn-north-1.jdcloud-oss.com/CFS/verify-pv-cfs.yml
kind: Pod apiVersion: v1 metadata: name: verify-pv-cfs spec: containers: - name: c1 image: busybox imagePullPolicy: IfNotPresent command: - /bin/sh args: - -c - 'while true; do ls -l /mnt/cfs/; sleep 2; done' volumeMounts: - mountPath: "/mnt/cfs" name: cfs-pv001 volumes: - name: cfs-pv001 persistentVolumeClaim: claimName: auto-pv-with-nfs-client-provisioner #与云文件存储建立绑定关系的PVC 名称
kubectl create -f verify-pv-cfs.yml pod/verify-pv-cfs created
kubectl get pod verify-pv-cfs NAME READY STATUS RESTARTS AGE verify-pv-cfs 1/1 Running 0 15s 查看Pod verify-pv-cfs输出的日志: kubectl logs verify-pv-cfs total 1 -rw-r--r-- 1 root root 18 Aug 7 07:29 SUCCESS exec进入Pod verify-pv-cfs,查看SUCCESS文件的内容: kubectl exec -it verify-pv-cfs /bin/sh / # cat /mnt/cfs/SUCCESS helloworld
kubectl delete pod verify-pv-cfs pod "verify-pv-cfs" deleted
kubectl delete pvc auto-pv-with-nfs-client-provisioner persistentvolumeclaim "auto-pv-with-nfs-client-provisioner" deleted
kubectl get pv pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7 Error from server (NotFound): persistentvolumes "pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7" not found 输出以上结果,说明与PVC关联的PV pvc-c44da35f-b8bc-11e9-b6cc-fa163e229fe7已被自动删除
cd /cfs #进入云主机上的cfs目录 ls -a # 查看cfs目录中的内容 输出内容说明:输出内容为空,则表明PVC、PV被删除后,CFS文件存储中由nfs-client-provisioner自动创建的子目录及文件内容已被成功回收