参考链接：

官方：https://goharbor.io/docs/2.7.0/install-config/harbor-ha-helm/

https://blog.51cto.com/lidabai/5195706

1、先决条件

  • Kubernetes 集群 1.10+
  • 头盔 2.8.0+
  • 高可用入口控制器（Harbor 不管理外部端点）
  • 高可用 PostgreSQL 9.6+（Harbor 不处理数据库 HA 的部署）
  • 高可用Redis（Harbor不处理Redis的HA部署）
  • 可以跨节点或外部对象存储共享的 PVC

2、下载Chart并部署

  1. # 添加仓库（helm repo add repoName repoURL）
  2. helm repo add harbor https://helm.goharbor.io
  4. # 查看添加的仓库
  5. helm repo list
  7. # 删除已经添加的仓库（helm repo remove repoName）
  8. # 将本地所添加的chart仓库的最新信息缓存到本地（helm repo update）
  10. # helm列出所有版本:
  11. helm search repo harbor -l
  13. # helm 下载指定版本：
  14. # helm fetch harbor/harbor --version 1.7.2
  15. helm fetch harbor/harbor --untar  # 拉取最新并解压

或者使用一下方式下载chart到不本地解压

  1. $ helm search repo harbor         # 搜索chart包
  2. NAME          CHART VERSION APP VERSION DESCRIPTION                                       
  3. harbor/harbor 1.8.2         2.4.2       An open source trusted cloud native registry th...
  4. $ helm pull harbor/harbor   # 下载Chart包
  5. $ tar zxvf harbor-1.8.2.tgz   # 解压包
  • 这里可以使用使用默认配置直接部署
  1. helm install my-harbor harbor/
  • 这种方式不建议
  • 建议使用单独的名称空间和外部储存进行部署

3、创建harbor名称空间

  1. kubectl create namespace harbor

4、修改values.yaml配置

  • chart目录结构
  1. [root@k8s-node1 harbor]# ll
  2. 总用量 256
  3. drwxr-xr-x  2 root root     36 12月 30 17:28 cert
  4. -rw-r--r--  1 root root    569 12月 30 17:28 Chart.yaml
  5. drwxr-xr-x  2 root root     58 12月 30 17:28 conf
  6. -rw-r--r--  1 root root  11357 12月 30 17:28 LICENSE
  7. -rw-r--r--  1 root root 204387 12月 30 17:28 README.md
  8. drwxr-xr-x 16 root root   4096 12月 30 17:28 templates
  9. -rw-r--r--  1 root root  34194 12月 30 17:28 values.yaml
  • values.yaml配置
  1. # 修改前建议先备份配置文件
  2. cp values.yaml valies.yaml-bak
  4. cat values.yaml 
  5. expose:
  6.   type: nodePort  # 对外暴露方式（默认使用ingress），这里没有ingress环境使用NodePort
  7.   tls:
  8.     enabled: false        # 关闭tls安全加密认证（如果开启需要配置证书）
  9.     certSource: auto
  10.     auto:
  11.       commonName: ""
  12.     secret:
  13.       secretName: ""
  14.       notarySecretName: ""
  15.   ingress:
  16.     hosts:
  17.       core: core.harbor.domain
  18.       notary: notary.harbor.domain
  19.     controller: default
  20.     kubeVersionOverride: ""
  21.     className: ""
  22.     annotations:
  23.       ingress.kubernetes.io/ssl-redirect: "true"
  24.       ingress.kubernetes.io/proxy-body-size: "0"
  25.       nginx.ingress.kubernetes.io/ssl-redirect: "true"
  26.       nginx.ingress.kubernetes.io/proxy-body-size: "0"
  27.     notary:
  28.       annotations: {}
  29.       labels: {}
  30.     harbor:
  31.       annotations: {}
  32.       labels: {}
  33.   clusterIP:
  34.     name: harbor
  35.     annotations: {}
  36.     ports:
  37.       httpPort: 80
  38.       httpsPort: 443
  39.       notaryPort: 4443
  40.   nodePort:
  41.     name: harbor
  42.     ports:
  43.       http:
  44.         port: 80
  45.         nodePort: 30002
  46.       https:
  47.         port: 443
  48.         nodePort: 30003
  49.       notary:
  50.         port: 4443
  51.         nodePort: 30004
  52.   loadBalancer:
  53.     name: harbor
  54.     IP: ""
  55.     ports:
  56.       httpPort: 80
  57.       httpsPort: 443
  58.       notaryPort: 4443
  59.     annotations: {}
  60.     sourceRanges: []
  61. externalURL: https://core.harbor.domain     # 使用nodePort且关闭tls认证，则此处需要修改为http协议和expose.nodePort.ports.http.nodePort指定的端口号，IP即为kubernetes的节点IP地址
  62. externalURL: http://192.168.6.31:30002
  63. internalTLS:
  64.   enabled: false
  65.   certSource: "auto"
  66.   trustCa: ""
  67.   core:
  68.     secretName: ""
  69.     crt: ""
  70.     key: ""
  71.   jobservice:
  72.     secretName: ""
  73.     crt: ""
  74.     key: ""
  75.   registry:
  76.     secretName: ""
  77.     crt: ""
  78.     key: ""
  79.   portal:
  80.     secretName: ""
  81.     crt: ""
  82.     key: ""
  83.   chartmuseum:
  84.     secretName: ""
  85.     crt: ""
  86.     key: ""
  87.   trivy:
  88.     secretName: ""
  89.     crt: ""
  90.     key: ""
  91. ipFamily:
  92.   ipv6:
  93.     enabled: true
  94.   ipv4:
  95.     enabled: true
  97. # 持久化存储配置部分
  98. persistence:
  99.   enabled: true     # 开启持久化存储
  100.   resourcePolicy: "keep"
  101.   persistentVolumeClaim:        # 定义Harbor各个组件的PVC持久卷部分
  102.     registry:        # registry组件（持久卷）配置部分
  103.       existingClaim: ""
  104.       storageClass: "managed-nfs-storage"        # StorageClass，其它组件同样配置
  105.       subPath: ""
  106.       accessMode: ReadWriteMany         # 卷的访问模式，需要修改为ReadWriteMany，允许多个组件读写，否则有的组件无法读取其它组件的数据
  107.       size: 5Gi
  108.       annotations: {}
  109.     chartmuseum:        # chartmuseum组件（持久卷）配置部分
  110.       existingClaim: ""
  111.       storageClass: "managed-nfs-storage"    #修改，同上
  112.       subPath: ""
  113.       accessMode: ReadWriteOnce
  114.       size: 5Gi
  115.       annotations: {}
  116.     jobservice:
  117.       jobLog:
  118.         existingClaim: ""
  119.         storageClass: ""
  120.         subPath: ""
  121.         accessMode: ReadWriteOnce
  122.         size: 1Gi
  123.         annotations: {}
  124.       scanDataExports:
  125.         existingClaim: ""
  126.         storageClass: ""
  127.         subPath: ""
  128.         accessMode: ReadWriteOnce
  129.         size: 1Gi
  130.         annotations: {}
  131.     database:            # PostgreSQl数据库组件（持久卷）配置部分
  132.       existingClaim: ""
  133.       storageClass: "managed-nfs-storage"            # 同上
  134.       subPath: ""
  135.       accessMode: ReadWriteMany  # 同上
  136.       size: 1Gi
  137.       annotations: {}
  138.     redis:            # Redis缓存组件（持久卷）配置部分
  139.       existingClaim: ""
  140.       storageClass: "managed-nfs-storage"            # 同上
  141.       subPath: ""
  142.       accessMode: ReadWriteMany        # 同上
  143.       size: 1Gi
  144.       annotations: {}
  145.     trivy:    # Trity漏洞扫描插件（持久卷）配置部分
  146.       existingClaim: ""
  147.       storageClass: "managed-nfs-storage"
  148.       subPath: ""
  149.       accessMode: ReadWriteMany
  150.       size: 5Gi
  151.       annotations: {}
  152.   imageChartStorage:
  153.     disableredirect: false
  154.     type: filesystem
  155.     filesystem:
  156.       rootdirectory: /storage
  157.     azure:
  158.       accountname: accountname
  159.       accountkey: base64encodedaccountkey
  160.       container: containername
  161.       existingSecret: ""
  162.     gcs:
  163.       bucket: bucketname
  164.       encodedkey: base64-encoded-json-key-file
  165.       existingSecret: ""
  166.       useWorkloadIdentity: false
  167.     s3:
  168.       region: us-west-1
  169.       bucket: bucketname
  170.     swift:
  171.       authurl: https://storage.myprovider.com/v3/auth
  172.       username: username
  173.       password: password
  174.       container: containername
  175.     oss:
  176.       accesskeyid: accesskeyid
  177.       accesskeysecret: accesskeysecret
  178.       region: regionname
  179.       bucket: bucketname
  180. imagePullPolicy: IfNotPresent
  181. imagePullSecrets:
  182. updateStrategy:
  183.   type: RollingUpdate
  184. logLevel: info
  185. harborAdminPassword: "Harbor12345"        # admin初始密码，不需要修改（生产环境建议修改）
  186. caSecretName: ""
  187. secretKey: "not-a-secure-key"
  188. existingSecretSecretKey: ""
  189. proxy:
  190.   httpProxy:
  191.   httpsProxy:
  192.   noProxy: 127.0.0.1,localhost,.local,.internal
  193.   components:
  194.     - core
  195.     - jobservice
  196.     - trivy
  197. enableMigrateHelmHook: false
  198. nginx:
  199.   image:
  200.     repository: goharbor/nginx-photon
  201.     tag: v2.7.0
  202.   serviceAccountName: ""
  203.   automountServiceAccountToken: false
  204.   replicas: 1
  205.   revisionHistoryLimit: 10
  206.   nodeSelector: {}
  207.   tolerations: []
  208.   affinity: {}
  209.   podAnnotations: {}
  210.   priorityClassName:
  211. portal:
  212.   image:
  213.     repository: goharbor/harbor-portal
  214.     tag: v2.7.0
  215.   serviceAccountName: ""
  216.   automountServiceAccountToken: false
  217.   replicas: 1
  218.   revisionHistoryLimit: 10
  219.   nodeSelector: {}
  220.   tolerations: []
  221.   affinity: {}
  222.   podAnnotations: {}
  223.   priorityClassName:
  224. core:
  225.   image:
  226.     repository: goharbor/harbor-core
  227.     tag: v2.7.0
  228.   serviceAccountName: ""
  229.   automountServiceAccountToken: false
  230.   replicas: 1
  231.   revisionHistoryLimit: 10
  232.   startupProbe:
  233.     enabled: true
  234.     initialDelaySeconds: 10
  235.   nodeSelector: {}
  236.   tolerations: []
  237.   affinity: {}
  238.   podAnnotations: {}
  239.   serviceAnnotations: {}
  240.   secret: ""
  241.   secretName: ""
  242.   xsrfKey: ""
  243.   priorityClassName:
  244.   artifactPullAsyncFlushDuration:
  245.   gdpr:
  246.     deleteUser: false
  247. jobservice:
  248.   image:
  249.     repository: goharbor/harbor-jobservice
  250.     tag: v2.7.0
  251.   replicas: 1
  252.   revisionHistoryLimit: 10
  253.   serviceAccountName: ""
  254.   automountServiceAccountToken: false
  255.   maxJobWorkers: 10
  256.   jobLoggers:
  257.     - file
  258.   nodeSelector: {}
  259.   tolerations: []
  260.   affinity: {}
  261.   podAnnotations: {}
  262.   secret: ""
  263.   priorityClassName:
  264. registry:
  265.   serviceAccountName: ""
  266.   automountServiceAccountToken: false
  267.   registry:
  268.     image:
  269.       repository: goharbor/registry-photon
  270.       tag: v2.7.0
  271.   controller:
  272.     image:
  273.       repository: goharbor/harbor-registryctl
  274.       tag: v2.7.0
  275.   replicas: 1
  276.   revisionHistoryLimit: 10
  277.   nodeSelector: {}
  278.   tolerations: []
  279.   affinity: {}
  280.   podAnnotations: {}
  281.   priorityClassName:
  282.   secret: ""
  283.   relativeurls: false
  284.   credentials:
  285.     username: "harbor_registry_user"
  286.     password: "harbor_registry_password"
  287.     existingSecret: ""
  288.   middleware:
  289.     enabled: false
  290.     type: cloudFront
  291.     cloudFront:
  292.       baseurl: example.cloudfront.net
  293.       keypairid: KEYPAIRID
  294.       duration: 3000s
  295.       ipfilteredby: none
  296.       privateKeySecret: "my-secret"
  297.   upload_purging:
  298.     enabled: true
  299.     age: 168h
  300.     interval: 24h
  301.     dryrun: false
  302. chartmuseum:
  303.   enabled: true
  304.   serviceAccountName: ""
  305.   automountServiceAccountToken: false
  306.   absoluteUrl: false
  307.   image:
  308.     repository: goharbor/chartmuseum-photon
  309.     tag: v2.7.0
  310.   replicas: 1
  311.   revisionHistoryLimit: 10
  312.   nodeSelector: {}
  313.   tolerations: []
  314.   affinity: {}
  315.   podAnnotations: {}
  316.   priorityClassName:
  317.   indexLimit: 0
  318. trivy:
  319.   enabled: true
  320.   image:
  321.     repository: goharbor/trivy-adapter-photon
  322.     tag: v2.7.0
  323.   serviceAccountName: ""
  324.   automountServiceAccountToken: false
  325.   replicas: 1
  326.   debugMode: false
  327.   vulnType: "os,library"
  328.   severity: "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL"
  329.   ignoreUnfixed: false
  330.   insecure: false
  331.   gitHubToken: ""
  332.   skipUpdate: false
  333.   offlineScan: false
  334.   securityCheck: "vuln"
  335.   timeout: 5m0s
  336.   resources:
  337.     requests:
  338.       cpu: 200m
  339.       memory: 512Mi
  340.     limits:
  341.       cpu: 1
  342.       memory: 1Gi
  343.   nodeSelector: {}
  344.   tolerations: []
  345.   affinity: {}
  346.   podAnnotations: {}
  347.   priorityClassName:
  348. notary:
  349.   enabled: true
  350.   server:
  351.     serviceAccountName: ""
  352.     automountServiceAccountToken: false
  353.     image:
  354.       repository: goharbor/notary-server-photon
  355.       tag: v2.7.0
  356.     replicas: 1
  357.     nodeSelector: {}
  358.     tolerations: []
  359.     affinity: {}
  360.     podAnnotations: {}
  361.     priorityClassName:
  362.   serviceAnnotations: {}
  363.   signer:
  364.     serviceAccountName: ""
  365.     automountServiceAccountToken: false
  366.     image:
  367.       repository: goharbor/notary-signer-photon
  368.       tag: v2.7.0
  369.     replicas: 1
  370.     nodeSelector: {}
  371.     tolerations: []
  372.     affinity: {}
  373.     podAnnotations: {}
  374.     priorityClassName:
  375.   secretName: ""
  376. database:
  377.   type: internal
  378.   internal:
  379.     serviceAccountName: ""
  380.     automountServiceAccountToken: false
  381.     image:
  382.       repository: goharbor/harbor-db
  383.       tag: v2.7.0
  384.     password: "changeit"
  385.     shmSizeLimit: 512Mi
  386.     livenessProbe:
  387.       timeoutSeconds: 1
  388.     readinessProbe:
  389.       timeoutSeconds: 1
  390.     nodeSelector: {}
  391.     tolerations: []
  392.     affinity: {}
  393.     priorityClassName:
  394.     initContainer:
  395.       migrator: {}
  396.       permissions: {}
  397.   external:
  398.     host: "192.168.0.1"
  399.     port: "5432"
  400.     username: "user"
  401.     password: "password"
  402.     coreDatabase: "registry"
  403.     notaryServerDatabase: "notary_server"
  404.     notarySignerDatabase: "notary_signer"
  405.     existingSecret: ""
  406.     sslmode: "disable"
  407.   maxIdleConns: 100
  408.   maxOpenConns: 900
  409.   podAnnotations: {}
  410. redis:
  411.   type: internal
  412.   internal:
  413.     serviceAccountName: ""
  414.     automountServiceAccountToken: false
  415.     image:
  416.       repository: goharbor/redis-photon
  417.       tag: v2.7.0
  418.     nodeSelector: {}
  419.     tolerations: []
  420.     affinity: {}
  421.     priorityClassName:
  422.   external:
  423.     addr: "192.168.0.2:6379"
  424.     sentinelMasterSet: ""
  425.     coreDatabaseIndex: "0"
  426.     jobserviceDatabaseIndex: "1"
  427.     registryDatabaseIndex: "2"
  428.     chartmuseumDatabaseIndex: "3"
  429.     trivyAdapterIndex: "5"
  430.     password: ""
  431.     existingSecret: ""
  432.   podAnnotations: {}
  433. exporter:
  434.   replicas: 1
  435.   revisionHistoryLimit: 10
  436.   podAnnotations: {}
  437.   serviceAccountName: ""
  438.   automountServiceAccountToken: false
  439.   image:
  440.     repository: goharbor/harbor-exporter
  441.     tag: v2.7.0
  442.   nodeSelector: {}
  443.   tolerations: []
  444.   affinity: {}
  445.   cacheDuration: 23
  446.   cacheCleanInterval: 14400
  447.   priorityClassName:
  448. metrics:  # 是否启用监控组件（可以使用Prometheus监控Harbor指标），非必须操作
  449.   enabled: true  # 默认关闭
  450.   core:
  451.     path: /metrics
  452.     port: 8001
  453.   registry:
  454.     path: /metrics
  455.     port: 8001
  456.   jobservice:
  457.     path: /metrics
  458.     port: 8001
  459.   exporter:
  460.     path: /metrics
  461.     port: 8001
  462.   serviceMonitor:
  463.     enabled: false
  464.     additionalLabels: {}
  465.     interval: ""
  466.     metricRelabelings:
  467.       []
  468.     relabelings:
  469.       []
  470. trace:
  471.   enabled: false
  472.   provider: jaeger
  473.   sample_rate: 1
  474.   jaeger:
  475.     endpoint: http://hostname:14268/api/traces
  476.   otel:
  477.     endpoint: hostname:4318
  478.     url_path: /v1/traces
  479.     compression: false
  480.     insecure: true
  481.     timeout: 10
  482. cache:
  483.   enabled: false
  484.   expireHours: 24

StorageClass部署手册：

https://www.yuque.com/geray-alxoc/bapt5y/ephx5o?singleDoc# 《7、k8s存储》

https://github.com/kubernetes-retired/external-storage/tree/master/nfs

  • 修改完成配置之后可以进行安装
  1. helm install  harbor ./harbor  -n harbor        # 将安装资源部署到harbor命名空间

helm部署harbor高可用 - 图1

  1. # 查看chart状态
  2. helm status harbor -n harbor
  4. # 卸载chart
  5. helm uninstall harbor -nharbor
  • 默认情况下helm3只显示默认名称空间（default）下的发布

5、服务验证

1. 查看所有服务启动状态

  1. kubectl get pod -n harbor 
  2. NAME                                   READY   STATUS    RESTARTS      AGE
  3. harbor-chartmuseum-5798db66df-kg7hm    1/1     Running   0             2m20s
  4. harbor-core-789d6c44f9-mgdlq           1/1     Running   0             2m20s
  5. harbor-database-0                      1/1     Running   0             2m18s
  6. harbor-exporter-7948fc6669-lw58b       1/1     Running   0             2m20s
  7. harbor-jobservice-66bf498cc4-l8cfc     1/1     Running   3 (57s ago)   2m20s
  8. harbor-nginx-fc68874d5-hbd59           1/1     Running   0             2m20s
  9. harbor-notary-server-f95bc8f6d-kblng   1/1     Running   1 (65s ago)   2m20s
  10. harbor-notary-signer-98cc8c75f-gfmr6   1/1     Running   0             2m20s
  11. harbor-portal-b7d5d9558-hl9jx          1/1     Running   0             2m20s
  12. harbor-redis-0                         1/1     Running   0             2m18s
  13. harbor-registry-5896d877fc-dbv7b       2/2     Running   0             2m20s
  14. harbor-trivy-0                         1/1     Running   0             2m18s

2. 查看service

  1. kubectl get svc -nharbor
  2. NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
  3. harbor                 NodePort    10.233.25.27    <none>        80:30002/TCP,4443:30004/TCP   15m
  4. harbor-chartmuseum     ClusterIP   10.233.19.145   <none>        80/TCP                        15m
  5. harbor-core            ClusterIP   10.233.60.97    <none>        80/TCP,8001/TCP               15m
  6. harbor-database        ClusterIP   10.233.48.217   <none>        5432/TCP                      15m
  7. harbor-exporter        ClusterIP   10.233.28.154   <none>        8001/TCP                      15m
  8. harbor-jobservice      ClusterIP   10.233.17.179   <none>        80/TCP,8001/TCP               15m
  9. harbor-notary-server   ClusterIP   10.233.43.35    <none>        4443/TCP                      15m
  10. harbor-notary-signer   ClusterIP   10.233.38.126   <none>        7899/TCP                      15m
  11. harbor-portal          ClusterIP   10.233.46.136   <none>        80/TCP                        15m
  12. harbor-redis           ClusterIP   10.233.34.145   <none>        6379/TCP                      15m
  13. harbor-registry        ClusterIP   10.233.22.128   <none>        5000/TCP,8080/TCP,8001/TCP    15m
  14. harbor-trivy           ClusterIP   10.233.62.176   <none>        8080/TCP                      15m

3. 登陆访问测试

  1. # 查看状态，寻找访问方式
  2. helm status harbor -n harbor
  3. NAME: harbor
  4. LAST DEPLOYED: Wed Feb  8 09:30:07 2023
  5. NAMESPACE: harbor
  6. STATUS: deployed
  7. REVISION: 1
  8. TEST SUITE: None
  9. NOTES:
  10. Please wait for several minutes for Harbor deployment to complete.
  11. Then you should be able to visit the Harbor portal at http://192.168.6.31:30002
  12. For more details, please visit https://github.com/goharbor/harbor
  • 拉取镜像打标签推送测试
  1. # 添加私服地址
  2. vim /etc/docker/daemon.json
  3. "insecure-registries": ["192.168.6.31:30002"]
  1. docker pull nginx
  2. docker tag nginx:latest 192.168.6.31:30002/mid/nginx:latest
  3. docker push 192.168.6.31:30002/mid/nginx:latest

helm部署harbor高可用 - 图2

6、错误排查

  1. kubectl describe pod harbor-notary-server-54f99bd444-shxl6 -n harbor
  2. ...
  3.   Normal   Started      12m (x4 over 15m)       kubelet            Started container notary-server
  4.   Normal   Pulled       12m (x3 over 14m)       kubelet            Container image "goharbor/notary-server-photon:v2.7.0" already present on machine
  5.   Warning  Unhealthy    7m36s (x12 over 9m14s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503
  6.   Warning  Unhealthy    104s (x6 over 14m)      kubelet            Readiness probe failed: Get "http://10.233.111.122:4443/_notary_server/health": dial tcp 10.233.111.122:4443: connect: connection refused

helm部署harbor高可用 - 图3

  1. kubectl describe pod harbor-notary-signer-5db9f5ff9-5rdtb -n harbor
  2. ...
  3.   Normal   Pulled       9m27s (x6 over 17m)  kubelet            Container image "goharbor/notary-signer-photon:v2.7.0" already present on machine
  4.   Warning  Unhealthy    3m31s (x7 over 17m)  kubelet            Readiness probe failed: Get "https://10.233.111.120:7899/": dial tcp 10.233.111.120:7899: connect: connection refused
  • 本以为是禁用了tls导致的，结果等了一会发现好了（自愈了）