小趙同學
0%

Kubernetes Ingress升级

发表于 更新于

Kubernetes Ingress升级

升级内容

  1. 部署方式调整：DaemonSet改为Deployment
  2. ingress-nginx 版本升级
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
➜  k -n nginx-ingress get all
NAME                      READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-ffjld   1/1     Running   5          58d
pod/nginx-ingress-fsdx5   1/1     Running   2          9d
pod/nginx-ingress-fv5wq   1/1     Running   9          16d
pod/nginx-ingress-grf25   1/1     Running   12         58d
pod/nginx-ingress-hd2gl   1/1     Running   2          9d
pod/nginx-ingress-hntl4   1/1     Running   2          9d
pod/nginx-ingress-rjgf5   1/1     Running   3          16d
pod/nginx-ingress-t5r5h   1/1     Running   4          9d

NAME              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                                      AGE
service/ingress   NodePort   10.96.229.176   <none>        10254:31527/TCP,80:32049/TCP,443:32040/TCP   182d

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress   8         8         8       8            8           <none>          182d

操作概览

  1. 备份数据
  2. 获取新版本Manifests
  3. 部署deployment
  4. 删除deamonset

准备工作

备份数据

1
2
3
4
5
6
7
8
9
mkdir -p back

cd back

# 备份列表
k get ing -A > all_list.log

# 备份yaml文件
k get ing -A -o yaml >> all_ingress_bak_0305.yaml

获取新版本nginx

https://github.com/nginxinc/kubernetes-ingress

https://codeload.github.com/nginxinc/kubernetes-ingress/tar.gz/refs/tags/v2.1.1

1
2
3
git clone https://github.com/nginxinc/kubernetes-ingress/
cd kubernetes-ingress/deployments
git checkout v2.1.2

安装

https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/

命名空间和服务账号

1
kubectl apply -f common/ns-and-sa.yaml
1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-ingress 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress 
  namespace: nginx-ingress

RBAC

1
kubectl apply -f rbac/rbac.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nginx-ingress
rules:
- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
  - update
  - create
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
  - list
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - list
  - watch
  - get
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - k8s.nginx.org
  resources:
  - virtualservers
  - virtualserverroutes
  - globalconfigurations
  - transportservers
  - policies
  verbs:
  - list
  - watch
  - get
- apiGroups:
  - k8s.nginx.org
  resources:
  - virtualservers/status
  - virtualserverroutes/status
  - policies/status
  - transportservers/status
  verbs:
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
- apiGroups:
    - cis.f5.com
  resources:
    - ingresslinks
  verbs:
    - list
    - watch
    - get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nginx-ingress
subjects:
- kind: ServiceAccount
  name: nginx-ingress
  namespace: nginx-ingress
roleRef:
  kind: ClusterRole
  name: nginx-ingress
  apiGroup: rbac.authorization.k8s.io

cecret

官网文档写的kubectl apply -f common/default-server-secret.yaml 

1
2
报错：
The Secret "default-server-secret" is invalid: type: Invalid value: "kubernetes.io/tls": field is immutable

这个TLS证书是用于404页面的证书，决定用自己公司的证书

1
kubectl -n nginx-ingress create secret tls izuche-2022-https --cert=./2022-zhaohongye.com.pem --key=./2022-zhaohongye.com.key

config

1
kubectl apply -f common/nginx-config.yaml

IngressClass

1
kubectl apply -f common/ingress-class.yaml

Deployment

1
kubectl apply -f deployment/nginx-ingress.yaml

需要修改deployment/nginx-ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
     #annotations:
       #prometheus.io/scrape: "true"
       #prometheus.io/port: "9113"
       #prometheus.io/scheme: http
    spec:
      serviceAccountName: nginx-ingress
      containers:
      # - image: nginx/nginx-ingress:2.1.2
      - image: registry.cn-beijing.aliyuncs.com/zhaohongye/nginx-ingress:2.1.2   #更新国内源
        imagePullPolicy: IfNotPresent
        name: nginx-ingress
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: readiness-port
          containerPort: 8081
        - name: prometheus
          containerPort: 9113
        readinessProbe:
          httpGet:
            path: /nginx-ready
            port: readiness-port
          periodSeconds: 1
        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        args:
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          # - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
          - -default-server-tls-secret=$(POD_NAMESPACE)/izuche-2022-https    #证书配置
          - -enable-custom-resources=false
         #- -v=3 # Enables extensive logging. Useful for troubleshooting.
         #- -report-ingress-status
         #- -external-service=nginx-ingress
         #- -enable-prometheus-metrics
         #- -global-configuration=$(POD_NAMESPACE)/nginx-configuration

Service

1
kubectl create -f service/nodeport.yaml

验证

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
k -n nginx-ingress get all

NAME                                 READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-5468c9789d-f4drp   1/1     Running   0          2d9h
pod/nginx-ingress-5468c9789d-ld8g2   1/1     Running   0          2d9h
pod/nginx-ingress-5468c9789d-v7rrq   1/1     Running   0          2d10h

NAME              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                                      AGE
service/ingress   NodePort   10.96.229.176   <none>        10254:31527/TCP,80:32049/TCP,443:32040/TCP   214d

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-ingress   3/3     3            3           31d

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-ingress-5468c9789d   3         3         3       2d10h
replicaset.apps/nginx-ingress-7589457d9b   0         0         0       2d10h
replicaset.apps/nginx-ingress-75f88d7497   0         0         0       2d10h
replicaset.apps/nginx-ingress-7df68f8846   0         0         0       31d
replicaset.apps/nginx-ingress-f4d7b8948    0         0         0       2d11h