哈尔滨门户网站是什么关键词林俊杰mp3在线听
操作系统 | k8s | mysql | minio | juicefs | 内核 |
centos8.2 | 1.19.1 | 8.0.39 | RELEASE.2023-12-20T01-00-02Z | v0.19.0 | 4.18.0-193.el8.x86_64 |
本文k8s较老采用老版本的juicefs,中间件也都是部署在k8s上。测试是否能成功创建动态pvc挂在到测试pod当中并查看到数据信息。一些偏理论知识就不多说了,直接开始操作
官方演示视频:JuiceFS CSI Driver 简介 - Office Hours 20220629_哔哩哔哩_bilibili
官方文档:安装 | JuiceFS Document Center
k8s部署mysql-CSDN博客
k8s部署minio集群-CSDN博客
参考以上两篇文章将中间件依赖环境先搭建起来
准备juicefs部署文件
k8s.yaml(juicefs)
# DO NOT EDIT: generated by 'kustomize build'
apiVersion: v1
kind: ServiceAccount
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-controller-sanamespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-node-sanamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-external-node-service-role
rules:
- apiGroups:- ""resources:- podsverbs:- get- list- create- update- delete- patch- watch
- apiGroups:- ""resources:- pods/logverbs:- get
- apiGroups:- ""resources:- secretsverbs:- get- create- update- delete- patch
- apiGroups:- batchresources:- jobsverbs:- get- create- update- delete- patch
- apiGroups:- ""resources:- nodes/proxyverbs:- '*'
- apiGroups:- ""resources:- persistentvolumesverbs:- get- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-external-provisioner-role
rules:
- apiGroups:- ""resources:- persistentvolumesverbs:- get- list- watch- create- delete
- apiGroups:- ""resources:- persistentvolumeclaimsverbs:- get- list- watch- update
- apiGroups:- storage.k8s.ioresources:- storageclassesverbs:- get- list- watch
- apiGroups:- ""resources:- eventsverbs:- list- watch- create- update- patch
- apiGroups:- storage.k8s.ioresources:- csinodesverbs:- get- list- watch
- apiGroups:- ""resources:- nodesverbs:- get- list- watch
- apiGroups:- ""resources:- secretsverbs:- get- create- update- patch- delete
- apiGroups:- ""resources:- pods- pods/logverbs:- get- list- watch- create- update- patch- delete
- apiGroups:- batchresources:- jobsverbs:- get- create- update- patch- delete
- apiGroups:- ""resources:- endpointsverbs:- get- list- watch- create- update- patch
- apiGroups:- appsresources:- daemonsetsverbs:- get- list
- apiGroups:- coordination.k8s.ioresources:- leasesverbs:- get- watch- list- delete- update- create
- apiGroups:- ""resources:- configmapsverbs:- get- watch- list- delete- update- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-node-service-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: juicefs-csi-external-node-service-role
subjects:
- kind: ServiceAccountname: juicefs-csi-node-sanamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-provisioner-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: juicefs-external-provisioner-role
subjects:
- kind: ServiceAccountname: juicefs-csi-controller-sanamespace: kube-system
---
apiVersion: apps/v1
kind: StatefulSet
metadata:labels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-controllernamespace: kube-system
spec:replicas: 1selector:matchLabels:app: juicefs-csi-controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterserviceName: juicefs-csi-controllertemplate:metadata:labels:app: juicefs-csi-controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterspec:containers:- args:- --endpoint=$(CSI_ENDPOINT)- --logtostderr- --nodeid=$(NODE_NAME)- --leader-election- --v=5env:- name: CSI_ENDPOINTvalue: unix:///var/lib/csi/sockets/pluginproxy/csi.sock- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: JUICEFS_MOUNT_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: JUICEFS_MOUNT_PATHvalue: /var/lib/juicefs/volume- name: JUICEFS_CONFIG_PATHvalue: /var/lib/juicefs/configimage: juicedata/juicefs-csi-driver:v0.19.0livenessProbe:failureThreshold: 5httpGet:path: /healthzport: healthzinitialDelaySeconds: 10periodSeconds: 10timeoutSeconds: 3name: juicefs-pluginports:- containerPort: 9909name: healthzprotocol: TCPresources:limits:cpu: 1000mmemory: 1Girequests:cpu: 100mmemory: 512MisecurityContext:capabilities:add:- SYS_ADMINprivileged: truevolumeMounts:- mountPath: /var/lib/csi/sockets/pluginproxy/name: socket-dir- mountPath: /jfsmountPropagation: Bidirectionalname: jfs-dir- mountPath: /root/.juicefsmountPropagation: Bidirectionalname: jfs-root-dir- args:- --csi-address=$(ADDRESS)- --timeout=60s- --enable-leader-election- --v=5env:- name: ADDRESSvalue: /var/lib/csi/sockets/pluginproxy/csi.sockimage: quay.io/k8scsi/csi-provisioner:v1.6.0name: csi-provisionervolumeMounts:- mountPath: /var/lib/csi/sockets/pluginproxy/name: socket-dir- args:- --csi-address=$(ADDRESS)- --health-port=$(HEALTH_PORT)env:- name: ADDRESSvalue: /csi/csi.sock- name: HEALTH_PORTvalue: "9909"image: quay.io/k8scsi/livenessprobe:v1.1.0name: liveness-probevolumeMounts:- mountPath: /csiname: socket-dirpriorityClassName: system-cluster-criticalserviceAccount: juicefs-csi-controller-satolerations:- key: CriticalAddonsOnlyoperator: Existsvolumes:- emptyDir: {}name: socket-dir- hostPath:path: /var/lib/juicefs/volumetype: DirectoryOrCreatename: jfs-dir- hostPath:path: /var/lib/juicefs/configtype: DirectoryOrCreatename: jfs-root-dirvolumeClaimTemplates: []
---
apiVersion: apps/v1
kind: DaemonSet
metadata:labels:app.kubernetes.io/component: nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-nodenamespace: kube-system
spec:selector:matchLabels:app: juicefs-csi-nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastertemplate:metadata:labels:app: juicefs-csi-nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterspec:containers:- args:- --endpoint=$(CSI_ENDPOINT)- --logtostderr- --nodeid=$(NODE_NAME)- --v=5- --enable-manager=trueenv:- name: CSI_ENDPOINTvalue: unix:/csi/csi.sock- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: JUICEFS_MOUNT_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: HOST_IPvalueFrom:fieldRef:fieldPath: status.hostIP- name: KUBELET_PORTvalue: "10250"- name: JUICEFS_MOUNT_PATHvalue: /var/lib/juicefs/volume- name: JUICEFS_CONFIG_PATHvalue: /var/lib/juicefs/configimage: juicedata/juicefs-csi-driver:v0.19.0lifecycle:preStop:exec:command:- /bin/sh- -c- rm /csi/csi.socklivenessProbe:failureThreshold: 5httpGet:path: /healthzport: healthzinitialDelaySeconds: 10periodSeconds: 10timeoutSeconds: 3name: juicefs-pluginports:- containerPort: 9909name: healthzprotocol: TCPresources:limits:cpu: 1000mmemory: 1Girequests:cpu: 100mmemory: 512MisecurityContext:privileged: truevolumeMounts:- mountPath: /var/lib/kubeletmountPropagation: Bidirectionalname: kubelet-dir- mountPath: /csiname: plugin-dir- mountPath: /devname: device-dir- mountPath: /jfsmountPropagation: Bidirectionalname: jfs-dir- mountPath: /root/.juicefsmountPropagation: Bidirectionalname: jfs-root-dir- args:- --csi-address=$(ADDRESS)- --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)- --v=5env:- name: ADDRESSvalue: /csi/csi.sock- name: DRIVER_REG_SOCK_PATHvalue: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sockimage: quay.io/k8scsi/csi-node-driver-registrar:v2.1.0name: node-driver-registrarvolumeMounts:- mountPath: /csiname: plugin-dir- mountPath: /registrationname: registration-dir- args:- --csi-address=$(ADDRESS)- --health-port=$(HEALTH_PORT)env:- name: ADDRESSvalue: /csi/csi.sock- name: HEALTH_PORTvalue: "9909"image: quay.io/k8scsi/livenessprobe:v1.1.0name: liveness-probevolumeMounts:- mountPath: /csiname: plugin-dirdnsPolicy: ClusterFirstWithHostNetpriorityClassName: system-node-criticalserviceAccount: juicefs-csi-node-satolerations:- key: CriticalAddonsOnlyoperator: Existsvolumes:- hostPath:path: /var/lib/kubelettype: Directoryname: kubelet-dir- hostPath:path: /var/lib/kubelet/csi-plugins/csi.juicefs.com/type: DirectoryOrCreatename: plugin-dir- hostPath:path: /var/lib/kubelet/plugins_registry/type: Directoryname: registration-dir- hostPath:path: /devtype: Directoryname: device-dir- hostPath:path: /var/lib/juicefs/volumetype: DirectoryOrCreatename: jfs-dir- hostPath:path: /var/lib/juicefs/configtype: DirectoryOrCreatename: jfs-root-dir
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: csi.juicefs.com
spec:attachRequired: falsepodInfoOnMount: false
secret.yaml(存储数据秘钥信息)
apiVersion: v1
kind: Secret
metadata:name: juicefs-secretnamespace: defaultlabels:# 增加该标签以启用认证信息校验juicefs.com/validate-secret: "true"
type: Opaque
stringData:name: data metaurl: mysql://root:PR38Ke2sP@(mysql.juicefs.svc.cluster.local:3306)/juicefs storage: minio bucket: http://minio-service.juicefs.svc.cluster.local:9000/juicefsaccess-key: adminsecret-key: 0oO1iIlLq9g#MINIO_ACCESS_KEY: admin #MINIO_SECRET_KEY: 0oO1iIlLq9g
#这里minio用来做数据存储,需要注意的是9000/juicefs这里是你的桶实现创建好,name: data会自行拼接到juicefs后面。也需要提前创建data出来
storageclasses.yaml(根据秘钥信息创建的sc)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: juicefs-sc
provisioner: csi.juicefs.com
parameters:csi.storage.k8s.io/provisioner-secret-name: juicefs-secretcsi.storage.k8s.io/provisioner-secret-namespace: defaultcsi.storage.k8s.io/node-publish-secret-name: juicefs-secretcsi.storage.k8s.io/node-publish-secret-namespace: default
reclaimPolicy: Retain
pvc-test.yaml(测试pvc挂载pod)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: juicefs-pvc#namespace: juicefs
spec:accessModes:- ReadWriteManyresources:requests:# 从 StorageClass 中申请 10GiB 存储容量storage: 3GistorageClassName: juicefs-sc
---
apiVersion: v1
kind: Pod
metadata:name: busybox#namespace: juicefs
spec:containers:- name: busyboximage: gcr.io/google-containers/busybox:latestimagePullPolicy: IfNotPresentcommand: ["sh", "-c", "tail -f /dev/null"]tty: true # 启用TTY以便可以连接到容器volumeMounts: # 确保此行与 'containers' 下的其他字段对齐- mountPath: /optname: juicefs-pvmountPropagation: HostToContainerrestartPolicy: Never # 设置为 Never 以便测试完成后不会自动重启volumes: # 确保此行与 'containers' 对齐- name: juicefs-pvpersistentVolumeClaim:claimName: juicefs-pvc
测试步骤
apply k8s.yaml 后会得到以下pod
当juicefs-csi相关pod成功拉起以后需要生成基于mysql与minio的secret。这个secret是用来给storageclasses使用的,让它能够调用csi去mount文件创建挂载点。
以上配置创建好以后apply pvc-test来测试动态创建pvc效果,发现会启用一个这种pod。是一个挂载点pod。
发现测试pod也成功创建了pvc也创建了并且绑定了busyboxpod
感兴趣可以看一下minio跟mysql数据存储的状态。这里就不截图演示了。
官方文档给的样例模板:创建和使用 PV | JuiceFS Document Center
#juicefs相关镜像
juicedata/juicefs-csi-driver:v0.19.0
quay.io/k8scsi/csi-node-driver-registrar:v2.1.0
quay.io/k8scsi/csi-provisioner:v1.6.0
quay.io/k8scsi/livenessprobe:v1.1.0
juicedata/mount:ce-v1.0.4