首先安装好GlusterFS集群

K8S 的存储卷使用稍有点古怪，Gluster FS 的使用，需要首先定义一个 Endpoint + Service 形式的代理，来定义 Gluster FS 集群，然后就可以通过持久卷或者用 Pod 直接加载了。

准备

（1）、在各节点安装客户端

  1. # yum install -y glusterfs glusterfs-fuse

（2）、配置endpoints

  1. # curl -O https://raw.githubusercontent.com/kubernetes/examples/master/volumes/glusterfs/glusterfs-endpoints.json

修改glusterfs-endpoints.json，配置GlusterFS集群信息

  1. {
  2.   "kind": "Endpoints",
  3.   "apiVersion": "v1",
  4.   "metadata": {
  5.     "name": "glusterfs-cluster"
  6.   },
  7.   "subsets": [
  8.     {
  9.       "addresses": [
  10.         {
  11.           "ip": "10.1.10.128"
  12.         }
  13.       ],
  14.       "ports": [
  15.         {
  16.           "port": 2020
  17.         }
  18.       ]
  19.     }
  20.   ]
  21. }

port可以随意写，ip为GlusterFS的IP地址

创建配置文件

  1. # kubectl apply -f glusterfs-endpoints.json
  2. # kubectl get ep
  3. NAME                ENDPOINTS          AGE
  4. glusterfs-cluster   10.1.10.128:2020   7m26s
  5. kubernetes          10.1.10.128:6443   27d

（3）、创建Serivce

  1. curl -O https://raw.githubusercontent.com/kubernetes/examples/master/volumes/glusterfs/glusterfs-service.json

修改配置文件，我这里仅修改了端口

  1. {
  2.   "kind": "Service",
  3.   "apiVersion": "v1",
  4.   "metadata": {
  5.     "name": "glusterfs-cluster"
  6.   },
  7.   "spec": {
  8.     "ports": [
  9.       {"port": 2020}
  10.     ]
  11.   }
  12. }

创建配置文件对象

  1. # kubectl apply -f glusterfs-service.json
  2. # kubectl get svc
  3. NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
  4. glusterfs-cluster   ClusterIP   10.254.44.189   <none>        2020/TCP   10m
  5. kubernetes          ClusterIP   10.254.0.1      <none>        443/TCP    27d

测试

创建测试Pod

（1）、下载测试pod文件

  1. curl -O https://raw.githubusercontent.com/kubernetes/examples/master/volumes/glusterfs/glusterfs-pod.json

（2）、修改配置文件，修改volumes下的path为我们上面创建的volume名

  1. {
  2.     "apiVersion": "v1",
  3.     "kind": "Pod",
  4.     "metadata": {
  5.         "name": "glusterfs"
  6.     },
  7.     "spec": {
  8.         "containers": [
  9.             {
  10.                 "name": "glusterfs",
  11.                 "image": "nginx",
  12.                 "volumeMounts": [
  13.                     {
  14.                         "mountPath": "/mnt/glusterfs",
  15.                         "name": "glusterfsvol"
  16.                     }
  17.                 ]
  18.             }
  19.         ],
  20.         "volumes": [
  21.             {
  22.                 "name": "glusterfsvol",
  23.                 "glusterfs": {
  24.                     "endpoints": "glusterfs-cluster",
  25.                     "path": "models",
  26.                     "readOnly": true
  27.                 }
  28.             }
  29.         ]
  30.     }
  31. }

（3）、创建Pod

  1. # kubectl apply -f glusterfs-pod.yaml
  2. # kubectl get pod
  3. NAME        READY   STATUS    RESTARTS   AGE
  4. glusterfs   1/1     Running   0          51s
  5. pod-demo    1/1     Running   8          25h
  6. # kubectl exec -it glusterfs -- df -h
  7. Filesystem                  Size  Used Avail Use% Mounted on
  8. overlay                      17G  2.5G   15G  15% /
  9. tmpfs                        64M     0   64M   0% /dev
  10. tmpfs                       910M     0  910M   0% /sys/fs/cgroup
  11. /dev/mapper/centos-root      17G  2.5G   15G  15% /etc/hosts
  12. 10.1.10.128:gluster_volume   17G  5.3G   12G  31% /mnt/glusterfs
  13. shm                          64M     0   64M   0% /dev/shm
  14. tmpfs                       910M   12K  910M   1% /run/secrets/kubernetes.io/serviceaccount
  15. tmpfs                       910M     0  910M   0% /proc/acpi
  16. tmpfs                       910M     0  910M   0% /proc/scsi
  17. tmpfs                       910M     0  910M   0% /sys/firmware

我们可以看到挂载成功。

用PVC挂载

1、直接创建PV和PVC进行关联

（1）、创建PV

  1. apiVersion: v1
  2. kind: PersistentVolume
  3. metadata:
  4.   name: glusterfs-pv
  5. spec:
  6.   capacity:
  7.     storage: 5Mi
  8.   accessModes:
  9.   - ReadWriteMany
  10.   glusterfs:
  11.     endpoints: glusterfs-cluster
  12.     path: gluster_volume
  13. ---
  14. apiVersion: v1
  15. kind: PersistentVolumeClaim
  16. metadata:
  17.   name: glusterfs-pvc
  18. spec:
  19.   accessModes:
  20.   - ReadWriteMany
  21.   resources:
  22.     requests:
  23.       storage: 5Mi

（2）、创建存储

  1. # kubectl apply -f  glusterfs-pv.yaml
  2. # kubectl get pv
  3. NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE
  4. glusterfs-pv   5Mi        RWX            Retain           Bound    default/glusterfs-pvc                           15s
  5. # kubectl get pvc
  6. NAME            STATUS   VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  7. glusterfs-pvc   Bound    glusterfs-pv   5Mi        RWX                           18s

2、用SC动态创建PVC

2.1、安装Heketi

Heketi提供了一个RESTful管理界面，可以用来管理GlusterFS卷的生命周期。Heketi会动态在集群内选择bricks构建所需的volumes，从而确保数据的副本会分散到集群不同的故障域内。同时Heketi还支持任意数量的ClusterFS集群。

（1）、安装

  1. # yum -y install heketi heketi-client

（2）、配置Heketi
/etc/heketi/heketi.json

  1. {
  2.   "_port_comment": "Heketi Server Port Number",
  3.   "port": "48080",            # 请求端口，默认是8080
  5.   "_use_auth": "Enable JWT authorization. Please enable for deployment",
  6.   "use_auth": false,
  8.   "_jwt": "Private keys for access",
  9.   "jwt": {
  10.     "_admin": "Admin has access to all APIs",
  11.     "admin": {
  12.       "key": "admin@P@ssW0rd"            # 管理员密码
  13.     },
  14.     "_user": "User only has access to /volumes endpoint",
  15.     "user": {
  16.       "key": "user@P@ssW0rd"            # 普通用户密码
  17.     }
  18.   },
  20.   "_glusterfs_comment": "GlusterFS Configuration",
  21.   "glusterfs": {
  22.     "_executor_comment": [
  23.       "Execute plugin. Possible choices: mock, ssh",
  24.       "mock: This setting is used for testing and development.",
  25.       "      It will not send commands to any node.",
  26.       "ssh:  This setting will notify Heketi to ssh to the nodes.",
  27.       "      It will need the values in sshexec to be configured.",
  28.       "kubernetes: Communicate with GlusterFS containers over",
  29.       "            Kubernetes exec api."
  30.     ],
  31.     "executor": "ssh",
  33.     "_sshexec_comment": "SSH username and private key file information",
  34.     "sshexec": {
  35.       "keyfile": "/etc/hekeit/private_key",            # ssh私钥目录
  36.       "user": "root",                                                        # ssh用户
  37.       "port": "22",                                                            # ssh端口
  38.       "fstab": "/etc/fstab"
  39.     },
  41.     "_kubeexec_comment": "Kubernetes configuration",
  42.     "kubeexec": {
  43.       "host" :"https://kubernetes.host:8443",
  44.       "cert" : "/path/to/crt.file",
  45.       "insecure": false,
  46.       "user": "kubernetes username",
  47.       "password": "password for kubernetes user",
  48.       "namespace": "OpenShift project or Kubernetes namespace",
  49.       "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"
  50.     },
  52.     "_db_comment": "Database file name",
  53.     "db": "/var/lib/heketi/heketi.db",
  55.     "_loglevel_comment": [
  56.       "Set log level. Choices are:",
  57.       "  none, critical, error, warning, info, debug",
  58.       "Default is warning"
  59.     ],
  60.     "loglevel" : "debug"
  61.   }
  62. }

（3）、配置免密

  1. # ssh-keygen -t rsa -q -f /etc/heketi/private_key -N ""
  2. # ssh-copy-id -i /etc/heketi/private_key.pub root@10.1.10.128
  3. # ssh-copy-id -i /etc/heketi/private_key.pub root@10.1.10.129
  4. # ssh-copy-id -i /etc/heketi/private_key.pub root@10.1.10.130

（4）、启动Heketi

  1. # systemctl enable heketi.service && systemctl start heketi.service
  2. # 测试
  3. # curl http://10.1.10.128:48080/hello
  4. Hello from Heketi

（5）、配置Topology

  1. {
  2.     "clusters": [
  3.         {
  4.             "nodes": [
  5.                 {
  6.                     "node": {
  7.                         "hostnames": {
  8.                             "manage": [
  9.                                 "glusterfs-master"
  10.                             ],
  11.                             "storage": [
  12.                                 "10.1.10.128"
  13.                             ]
  14.                         },
  15.                         "zone": 1
  16.                     },
  17.                     "devices": [
  18.                         "/dev/sdb1"    # 必须是未创建文件系统的裸磁盘
  19.                     ]
  20.                 },
  21.                 {
  22.                     "node": {
  23.                         "hostnames": {
  24.                             "manage": [
  25.                                 "glusterfs-node01"
  26.                             ],
  27.                             "storage": [
  28.                                 "10.1.10.129"
  29.                             ]
  30.                         },
  31.                         "zone": 1
  32.                     },
  33.                     "devices": [
  34.                         "/dev/sdb1"
  35.                     ]
  36.                 },
  37.                 {
  38.                     "node": {
  39.                         "hostnames": {
  40.                             "manage": [
  41.                                 "glusterfs-node02"
  42.                             ],
  43.                             "storage": [
  44.                                 "10.1.10.130"
  45.                             ]
  46.                         },
  47.                         "zone": 1
  48.                     },
  49.                     "devices": [
  50.                         "/dev/sdb1"
  51.                     ]
  52.                 }
  53.             ]
  54.         }
  55.     ]
  56. }
  1. # echo "export HEKETI_CLI_SERVER=http://10.1.10.128:48080" >> /etc/profile.d/heketi.sh
  2. # echo "alias heketi-cli='heketi-cli --user admin --secret admin@P@ssW0rd'" >> ~/.bashrc
  3. # source /etc/profile.d/heketi.sh
  4. # source ~/.bashrc
  5. # echo $HEKETI_CLI_SERVER
  6. http://10.1.10.128:48080

添加Cluster

  1. # heketi-cli --server $HEKETI_CLI_SERVER --user admin --secret admin@P@ssW0rd topology load --json=/etc/heketi/topology.json
  2. Creating cluster ... ID: 48a31ff76104514d187f7e7ef8c2e056
  3.     Allowing file volumes on cluster.
  4.     Allowing block volumes on cluster.
  5.     Creating node glusterfs-master ... ID: e374f9d94f47e0fd3223732a383b7d03
  6.         Adding device /dev/sdb1 ... OK
  7.     Creating node glusterfs-node01 ... ID: 68f16b2d54acf1c18e354ec46aa736ad
  8.         Adding device /dev/sdb1 ... OK
  9.     Creating node glusterfs-node02 ... ID: 86f5a124f8c4c296042d6207a641cf81
  10.         Adding device /dev/sdb1 ... OK

查看集群信息

  1. # heketi-cli cluster list
  2. Clusters:
  3. Id:48a31ff76104514d187f7e7ef8c2e056 [file][block]
  5. # 查看详细信息
  6. # heketi-cli cluster info 48a31ff76104514d187f7e7ef8c2e056
  7. Cluster id: 48a31ff76104514d187f7e7ef8c2e056
  8. Nodes:
  9. 68f16b2d54acf1c18e354ec46aa736ad
  10. 86f5a124f8c4c296042d6207a641cf81
  11. e374f9d94f47e0fd3223732a383b7d03
  12. Volumes:
  14. Block: true
  16. File: true
  18. # 查看节点信息
  19. # heketi-cli node list
  20. Id:68f16b2d54acf1c18e354ec46aa736ad    Cluster:48a31ff76104514d187f7e7ef8c2e056
  21. Id:86f5a124f8c4c296042d6207a641cf81    Cluster:48a31ff76104514d187f7e7ef8c2e056
  22. Id:e374f9d94f47e0fd3223732a383b7d03    Cluster:48a31ff76104514d187f7e7ef8c2e056
  24. # 查看节点详细信息
  25. # heketi-cli node info 68f16b2d54acf1c18e354ec46aa736ad
  26. Node Id: 68f16b2d54acf1c18e354ec46aa736ad
  27. State: online
  28. Cluster Id: 48a31ff76104514d187f7e7ef8c2e056
  29. Zone: 1
  30. Management Hostname: glusterfs-node01
  31. Storage Hostname: 10.1.10.129
  32. Devices:
  33. Id:dacaffdbd8a2a00bb76acdc7b2420fbd   Name:/dev/sdb1           State:online    Size (GiB):500     Used (GiB):0       Free (GiB):500     Bricks:0

创建volume

  1. # heketi-cli volume create --size=2 --replica=2
  2. Name: vol_4f1a171ab06adf80460c84f2132e96e0
  3. Size: 2
  4. Volume Id: 4f1a171ab06adf80460c84f2132e96e0
  5. Cluster Id: 48a31ff76104514d187f7e7ef8c2e056
  6. Mount: 10.1.10.129:vol_4f1a171ab06adf80460c84f2132e96e0
  7. Mount Options: backup-volfile-servers=10.1.10.130,10.1.10.128
  8. Block: false
  9. Free Size: 0
  10. Reserved Size: 0
  11. Block Hosting Restriction: (none)
  12. Block Volumes: []
  13. Durability Type: replicate
  14. Distribute Count: 1
  15. Replica Count: 2
  17. # heketi-cli volume list
  18. Id:4f1a171ab06adf80460c84f2132e96e0    Cluster:48a31ff76104514d187f7e7ef8c2e056    Name:vol_4f1a171ab06adf80460c84f2132e96e0
  20. # heketi-cli volume info 4f1a171ab06adf80460c84f2132e96e0
  21. Name: vol_4f1a171ab06adf80460c84f2132e96e0
  22. Size: 2
  23. Volume Id: 4f1a171ab06adf80460c84f2132e96e0
  24. Cluster Id: 48a31ff76104514d187f7e7ef8c2e056
  25. Mount: 10.1.10.129:vol_4f1a171ab06adf80460c84f2132e96e0
  26. Mount Options: backup-volfile-servers=10.1.10.130,10.1.10.128
  27. Block: false
  28. Free Size: 0
  29. Reserved Size: 0
  30. Block Hosting Restriction: (none)
  31. Block Volumes: []
  32. Durability Type: replicate
  33. Distribute Count: 1
  34. Replica Count: 2
  36. # 挂载
  37. # mount -t glusterfs 10.1.10.129:vol_4f1a171ab06adf80460c84f2132e96e0 /mnt
  39. # 删除
  40. # heketi-cli volume delete 4f1a171ab06adf80460c84f2132e96e0

2.2、在k8s中测试

（1）、创建连接使用的secret（heketi-secret.yaml）

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: heketi-secret
  5. data:
  6.   key: YWRtaW5AUEBzc1cwcmQ=
  7. type: kubernetes.io/glusterfs

（2）、创建sc（heketi-storageclass.yaml）

  1. apiVersion: storage.k8s.io/v1
  2. kind: StorageClass
  3. metadata:
  4.   name: heketi-storageclass
  5. parameters:
  6.   resturl: "http://10.1.10.128:48080"
  7.   clusterid: "cca360f44db482f03297a151886eea19"
  8.   restauthenabled: "true"                                       #若heketi开启认证此处也必须开启auth认证
  9.   restuser: "admin"
  10.   secretName: "heketi-secret"                           #name/namespace与secret资源中定义一致
  11.   secretNamespace: "default"
  12.   volumetype: "replicate:3"
  13. provisioner: kubernetes.io/glusterfs
  14. reclaimPolicy: Delete

（3）、创建pvc（heketi-pvc.yaml）

  1. apiVersion: v1
  2. kind: PersistentVolumeClaim
  3. metadata:
  4.   name: heketi-pvc
  5.   annotations:
  6.     volume.beta.kubernetes.io/storage-class: heketi-storageclass
  7. spec:
  8.   accessModes:
  9.   - ReadWriteOnce
  10.   resources:
  11.     requests:
  12.       storage: 1Gi

（4）、查看PVC情况

  1. # kubectl get sc
  2. NAME                  PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
  3. heketi-storageclass   kubernetes.io/glusterfs   Delete          Immediate           false                  6m53s
  4. # kubectl get pvc
  5. NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
  6. glusterfs-pvc   Bound    glusterfs-pv                               5Mi        RWX                                  26h
  7. heketi-pvc      Bound    pvc-0feb8666-6e7f-451d-ae6f-7f205206b225   1Gi        RWO            heketi-storageclass   82s

（5）、创建Pod测试挂载
heketi-pod.yaml

  1. kind: Pod
  2. apiVersion: v1
  3. metadata:
  4.   name: heketi-pod
  5. spec:
  6.   containers:
  7.   - name: heketi-container
  8.     image: busybox
  9.     command:
  10.     - sleep
  11.     - "3600"
  12.     volumeMounts:
  13.     - name: heketi-volume
  14.       mountPath: "/pv-data"
  15.       readOnly: false
  16.   volumes:
  17.   - name: heketi-volume
  18.     persistentVolumeClaim:
  19.       claimName: heketi-pvc

创建Pod并查看结果

  1. # kubectl get pod
  2. NAME         READY   STATUS    RESTARTS   AGE
  3. glusterfs    1/1     Running   0          26h
  4. heketi-pod   1/1     Running   0          2m55s

创建文件测试

  1. # kubectl exec -it heketi-pod -- /bin/sh
  2. / # cd /pv-data/
  3. /pv-data # echo "text" > 1111.txt
  4. /pv-data # ls
  5. 1111.txt

然后在本地查看

  1. # cd /var/lib/heketi/mounts/vg_bffb11849513dded78f671f64e76750c/brick_6ff640a2d45a7f146a296473e7145ee7
  2. [root@k8s-master brick_6ff640a2d45a7f146a296473e7145ee7]# ll
  3. total 0
  4. drwxrwsr-x 3 root 2000 40 Feb  7 14:27 brick
  5. [root@k8s-master brick_6ff640a2d45a7f146a296473e7145ee7]# cd brick/
  6. [root@k8s-master brick]# ll
  7. total 4
  8. -rw-r--r-- 2 root 2000 5 Feb  7 14:27 1111.txt