Kubernetes入门实验:statefulset

环境# kubectl get nodeNAME STATUS ROLES AGE VERSIONedge-node Ready <none> 15m v1.17.0edge-node2 Ready <none> 16m v1.17.0ubuntu Ready master 67d v1.17.0statefulset技术总结

创建statefulset,验证pvc存储。
master节点开放3个nfs目录。可读写。

实验简单示例

1、pv.yaml

apiVersion: v1kind: PersistentVolumemetadata: name: nfs-pv1 labels: storage: nfsspec: accessModes: ["ReadWriteOnce", "ReadWriteMany", "ReadOnlyMany"] # 支持多模式,可能好些 #accessModes: # - ReadWriteMany capacity: storage: 200Mi volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain # 值 delete Recycle Retain nfs: server: 192.168.0.102 path: /nfs1---apiVersion: v1kind: PersistentVolumemetadata: name: nfs-pv2 labels: storage: nfsspec: capacity: storage: 100Mi # 5Gi accessModes: ["ReadWriteOnce", "ReadWriteMany", "ReadOnlyMany"] nfs: server: 192.168.0.102 path: /nfs2---apiVersion: v1kind: PersistentVolumemetadata: name: nfs-pv3 labels: storage: nfsspec: capacity: storage: 100Mi # 5Gi accessModes: ["ReadWriteOnce", "ReadWriteMany", "ReadOnlyMany"] nfs: server: 192.168.0.102 path: /nfs3

2、 nginx-service.yaml:

apiVersion: v1kind: Servicemetadata: name: nginx labels: app: nginxspec: ports: - port: 80 name: web clusterIP: None selector: app: nginx---apiVersion: apps/v1kind: StatefulSetmetadata: name: web # statefulset名称spec: serviceName: "nginx" replicas: 3 # by default is 1 selector: matchLabels: app: nginx # has to match .spec.template.metadata.labels template: metadata: labels: app: nginx # has to match .spec.selector.matchLabels spec: terminationGracePeriodSeconds: 10 containers: - name: nginx image: latelee/lidch imagePullPolicy: IfNotPresent ports: - containerPort: 80 name: web volumeMounts: - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] #storageClassName: "my-storage-class" resources: requests: storage: 10Mi

注:创建3个pv,因为StatefulSet有3个副本,理论上pv应该多于副本。

查看

创建:

kubectl apply -f pv.yaml nginx-service.yaml

查看:

kubectl get statefulset查看详情:kubectl describe statefulset web查看pod:kubectl describe pod web-0

查看pv和pvc:

# kubectl get pv // 先创建,随后被www-web使用NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGEnfs-pv1 200Mi RWO,ROX,RWX Retain Bound default/www-web-2 3m47snfs-pv2 100Mi RWO,ROX,RWX Retain Bound default/www-web-0 3m47snfs-pv3 100Mi RWO,ROX,RWX Retain Bound default/www-web-1 3m47s# kubectl get pvc // 注!是自动创建的NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEwww-web-0 Bound nfs-pv2 100Mi RWO,ROX,RWX 3m51swww-web-1 Bound nfs-pv3 100Mi RWO,ROX,RWX 3m44swww-web-2 Bound nfs-pv1 200Mi RWO,ROX,RWX 3m41s测试存储

一终端执行kubectl get pod -w -l app=nginx观察状态,另一终端执行kubectl delete pod -l app=nginx,pod被删除后,按顺序重启创建。
查看主机名:

# for i in 0 1; do kubectl exec web-$i -- sh -c 'hostname'; doneweb-0web-1

在k8s中创建名为dns-test的pod,进入容器,如退出,则此pod被删除。

kubectl run -it --image latelee/busybox dns-test --restart=Never --rm /bin/sh

执行nslookup命令:nslookup web-XX.nginx。命令及输出:

/ # nslookup web-0.nginxServer: 10.96.0.10Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localnslookup: can't resolve 'web-0.nginx'/ # / # nslookup web-1.nginxServer: 10.96.0.10Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localnslookup: can't resolve 'web-1.nginx'/ # nslookup web-2.nginxServer: 10.96.0.10Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localnslookup: can't resolve 'web-2.nginx'/ #

注:为何解决不了web-0.nginx,暂未知。

依次将各自主机名写到网页index.html(注:因此已经挂载了,最终会写到各自nfs目录):

for i in 0 1 2; do kubectl exec web-$i -- sh -c 'echo $(hostname) > /usr/share/nginx/html/index.html'; done

结果pod名称为web-0者,其主机名为web-0,其它类推。
此时pod分布:

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESweb-0 1/1 Running 0 6m8s 10.244.4.133 edge-node2 <none> <none>web-1 1/1 Running 0 6m7s 10.244.1.125 edge-node <none> <none>web-2 1/1 Running 0 6m5s 10.244.4.134 edge-node2 <none> <none>

此时master主机nfs目录情况:

# cat /nfs1/index.html web-2# cat /nfs2/index.html web-0# cat /nfs3/index.html web-1

删除pod:

kubectl delete pod -l app=nginx

等待pod重新启动后。此时分布:

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESweb-0 1/1 Running 0 31s 10.244.4.142 edge-node2 <none> <none>web-1 1/1 Running 0 21s 10.244.1.129 edge-node <none> <none>web-2 1/1 Running 0 19s 10.244.4.143 edge-node2 <none> <none>

注:调度的节点无变化,但IP变化了。(姑且认为已经调度了)
查看各个pod的页面:

for i in 0 1 2; do kubectl exec web-$i -- sh -c 'cat /usr/share/nginx/html/index.html'; done输出为:web-0web-1web-2

再查看nfs目录,没有变化。
结论:文件内容已经存储在磁盘中,pod重启,文件不丢失,而且不会随着调度而变化。注意,nfs目录并不是一一对应的(本例中web-1对应/nfs3目录),但一旦确定了,就不会再改变。保证了内容的一致性。

扩容

扩容至5个:

# kubectl scale sts web --replicas=5

此时会pod无法创建成功,挂起状态。因为pv不够。

web-3 0/1 Pending 0 42s

在前面pv.yaml基础上添加2个nfs目录,并重启nfs服务。再用kubectl apply -f pv.yaml更新,此时pv情况:

# kubectl get pvNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGEnfs-pv1 200Mi RWO,ROX,RWX Retain Bound default/www-web-2 19mnfs-pv2 100Mi RWO,ROX,RWX Retain Bound default/www-web-0 19mnfs-pv3 100Mi RWO,ROX,RWX Retain Bound default/www-web-1 19mnfs-pv4 100Mi RWO,ROX,RWX Retain Bound default/www-web-3 8snfs-pv5 100Mi RWO,ROX,RWX Retain Available 8s

注:因为是在创建web-3后才创建pv的,所以一旦pv可用,立刻绑定,此刻pv5还没有绑定(稍等片刻即绑定)。

缩容:

kubectl patch sts web -p '{"spec":{"replicas":3}}'

此时,web-3和web-4被删除。

升级kubectl set image sts/web nginx=latelee/lidch:1.1

释义:更新sts/web,其容器名为nginx,镜像为latelee/lidch:1.1。
查看:

# kubectl get sts -o wideNAME READY AGE CONTAINERS IMAGESweb 3/3 69m nginx latelee/lidch:1.1

删除:

kubectl delete -f nginx-service.yamlkubectl delete -f pv.yaml

删除statefulset,pvc不会被删除,要手动删除。

问题

1、

error while running "VolumeBinding" filter plugin for pod "web-0": pod has unbound immediate PersistentVolumeClaimsno persistent volumes available for this claim and no storage class is set

原因:pvc没有创建。如副本为3,则会有3个pvc(也有3个pv)。
查看:

kubectl get pvc复制代码

继续查:

kubectl describe pvc www-web-0输出:storageclass.storage.k8s.io "my-storage-class" not found

清除所有,查询pv、pvc无输出:

kubectl get pvckubectl get pv

由于使用模板volumeClaimTemplates,不需要创建pvc,因此创建多个pv即可(本文示例为3个),再创建statefulset,会自动创建pvc。

2、如果不同的pv使用相同的路径,则会相互影响。如pod1修改的文件会影响到pod2。

3、

mount.nfs: access denied by server while mounting 192.168.0.102:/nfs3

1)、目录不存在;2)、没有导出该目录。


作者:李迟
链接:https://juejin.cn/post/6987237608526987272
来源:掘金
著作权归作者所有。商业转载请联系作者获得授权,非商业转载请注明出处。