K8S Helm 安装ingress-nginx/ingress-nginx

• 安装ingress-nginx/ingress-nginx

• • 参考：
  • https://www.cnblogs.com/syushin/p/15271304.html

• 添加helm仓库

  - [root@k8s-master ~]# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
  - [root@k8s-master ~]# helm repo update    

• 下载安装包

  • helm pull ingress-nginx/ingress-nginx
    • 解压
      • tar -zxvf ingress-nginx-4.0.1.tgz

• 备份并修改 values.yaml 文件

  • 修改controller的镜像地址

controller:
  name: /
  enableAnnotationValidations: false
  image:
    ## Keep false as default for now!
    chroot: false
    registry: docker.io
    image: willdockerhub/ingress-nginx-controller
    ## for backwards compatibility consider setting the full image url via the repository value below
    ## use *either* current default registry/image or repository format or installing chart by providing the values.yaml will fail
    ## repository:
    tag: "v1.0.0"
    digest: sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
    pullPolicy: IfNotPresent
    # www-data -> uid 101
    runAsUser: 101
    allowPrivilegeEscalation: true

• • 修改 hostNetwork 的值为 true：

# is merged hostNetwork: true ## Use host ports 80 and 443

• • dnsPolicy的值改为: ClusterFirstWithHostNet

# to keep resolving names inside the k8s network, use ClusterFirstWithHostNet. dnsPolicy: ClusterFirstWithHostNet

• • nodeSelector 添加标签: ingress: "true"，用于部署 ingress-controller 到指定节点

 # -- Node labels for controller pod assignment
  ## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
  ##
  nodeSelector:
    kubernetes.io/os: linux
    ingress: "true"

• • kind类型更改为：DaemonSet

 # -- Use a `DaemonSet` or `Deployment`
  kind: DaemonSet
  # -- Annotations to be added to the controller Deployment or DaemonSet

• • kube-webhook-certgen的镜像地址改为国内仓库地址

patch:
      enabled: true
      image:
        registry: registry.cn-hangzhou.aliyuncs.com
        image: google_containers/kube-webhook-certgen
        ## for backwards compatibility consider setting the full image url via the repository value below
        ## use *either* current default registry/image or repository format or installing chart by providing the values.yaml will fail
        ## repository:
        tag: "v20231011-8b53cabe0"
        #digest: sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
        digest: sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9
        pullPolicy: IfNotPresent

• 安装

  • 使用当前目录中的values.yaml文件
    • helm install ingress-nginx -n ingress-nginx .
  • 指定values.yaml文件，并使用debug参数，方便调试异常
    • helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --values values.yaml --debug
      • upgrade --install 会导致重新下载包
        • 如果不想重新下载，直接使用install
  • 安装成功，并给了一个创建ingress的例子

NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

• 处于pending状态的ingress-nginx-controller

[root@k8s-master01 ingress-nginx]# kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.99.163.44    <pending>     80:31268/TCP,443:31052/TCP   4m54s
ingress-nginx-controller-admission   ClusterIP      10.100.131.12   <none>        443/TCP                      4m54s


[root@k8s-master01 ingress-nginx]# kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
ingress-nginx-controller   LoadBalancer   10.99.163.44   <pending>     80:31268/TCP,443:31052/TCP   94s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

• 给节点打标签ingress=true
  • 需要给节点打上刚刚设置的标签ingress=true，让 Pod 调度到指定的节点，比如调度到 master 节点

# 给master节点打上标签 ingress=ture
[root@k8s-master ingress-nginx]# kubectl label node master1 ingress=true
node/master1 labeled

# k8s默认集群中，出于安全考虑，默认配置下Kubernetes不会将Pod调度到Master节点。测试环境无所谓，所以执行下面命令去除master的污点：
[root@k8s-master ingress-nginx]# kubectl taint node master1 node-role.kubernetes.io/master-

参考：

How do I set up a pod to allow scheduling on the master · Issue #1814 · k3s-io/k3s · GitHub How do I set up a pod to allow scheduling on the master

K8S 将 pod 调度到指定 nodes 上运行 - 知乎 K8S 将 pod 调度到指定 nodes 上运行

[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl taint nodes --all node-role.kubernetes.io/control-plane-
node/master01 untainted
taint "node-role.kubernetes.io/control-plane" not found
taint "node-role.kubernetes.io/control-plane" not found
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
master01   Ready    control-plane   44d   v1.28.2
node02     Ready    <none>          44d   v1.28.2
node03     Ready    <none>          44d   v1.28.2
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl get pods -A -owide |grep ingress
ingress-nginx            ingress-nginx-controller-bc96v                           1/1     Running            0                  41s     yourip     master01   <none>           <none>

k8s命令（pod相关（驱逐、强制删除）、让Master当Node用、修改nodeport端口范围、修改k8sDNS、运行nslookup容器）_kubectl drain-CSDN博客

#将 Master 也当作 Node 使用
kubectl taint node nodename node-role.kubernetes.io/master-

[root@app01 home]# kubectl taint node app01 node-role.kubernetes.io/master-
node/app01 untainted
[root@app01 home]# 

#将 Master 恢复成 Master Only 状态
kubectl taint node nodename node-role.kubernetes.io/master="":NoSchedule
  
[root@app01 rabbitmq]# kubectl taint node app01  node-role.kubernetes.io/master="":NoSchedule
node/app01 tainted
[root@app01 rabbitmq]# 

• • 本次直接将标签打到了node02上
    • 所以配置的域名要指向node02节点的ip地址
    • ingress-nginx-controller对应的pod会自动部署到node02节点上

• 配置访问hosts

  • 192.168.221.132 harbor.david.org
  • 测试
    • 用浏览器访问没问，curl缺少header中的一些参数，所以报308错误

[root@k8s-master01 harbor]# curl harbor.david.org
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

• configmap权限异常处理

• 查看ingress-nginx-controller日志

• • [root@k8s-master01 harbor]# kubectl logs -n ingress-nginx ingress-nginx-controller-4nl8l

• • • E1122 05:49:57.933496 7 leaderelection.go:329] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot create resource "configmaps" in API group "" in the namespace "ingress-nginx"
      • 需要create和update权限，修改role即可
        • [root@k8s-master01 nginx]# kubectl edit clusterrole -n ingress-nginx ingress-nginx
          • 添加
            • - create
            • - update
          • 保存并退出，再查看ingress-nginx-controller日志
            • [root@k8s-master01 harbor]# kubectl logs -n ingress-nginx ingress-nginx-controller-4nl8l
        • configmaps （用于配置文件注入）

rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - nodes
  - pods
  - secrets
  - namespaces
  verbs:
  - list
  - watch
  - create
  - update

• 异常调试

  • 安装加上debug参数 展示详细安装过程，方便排查错误
  • client.go:779: [debug] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
  • 提示job有异常，先找到这个job

[root@k8s-master01 ingress-nginx]# kubectl get job -A
NAMESPACE       NAME                             COMPLETIONS   DURATION   AGE
ingress-nginx   ingress-nginx-admission-create   0/1           8s         8s
ingress-nginx   ingress-nginx-admission-patch    0/1           69m        69m


[root@k8s-master01 ingress-nginx]# kubectl get pods -A
ingress-nginx          ingress-nginx-admission-create-r7bps    

• 有个job对应的pod有异常

[root@k8s-master01 ingress-nginx]# kubectl describe pods -n ingress-nginx          ingress-nginx-admission-create-r7bps

 Normal   BackOff    5s (x2 over 28s)   kubelet            Back-off pulling image "registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.5.1@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"

提示镜像有异常

Failed to pull image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": rpc error: code = NotFound desc = failed to pull and unpack image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": failed to resolve reference "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80: not found

修改了镜像还不管用，还需要修改sha256

使用ctr下载该镜像

[root@k8s-master01 ingress-nginx]# ctr -n=k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0

找到该镜像

[root@k8s-master01 ingress-nginx]# ctr -n=k8s.io images list

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0 application/vnd.docker.distribution.manifest.list.v2+json sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9 22.1 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/s390x io.cri-containerd.image=managed

把sha256字符串加到values.yaml里

digest: sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9

再次安装即可