Jenkins 使用中要注意的小问题

2025/10/20 jenkins 共 4565 字,约 14 分钟

Jenkins + Docker + Harbor + Kubernetes CI/CD 实践记录

一步步构建可复现、可自动化的持续集成与持续部署流水线。


一、背景与目标

在现代微服务架构中,持续集成(CI)与持续部署(CD)是必不可少的基础设施。本次记录内容为:

通过 Jenkins 实现自动化构建、测试、打包、推送镜像到 Harbor,并自动部署到 Kubernetes 集群。

最终实现效果:

  • Jenkins 自动拉取 GitLab 代码
  • 自动运行测试
  • 构建并推送 Docker 镜像到 Harbor
  • 自动部署更新到 K8s 集群
  • Flask 应用自动启动并可访问

二、系统环境概览

组件版本说明
OSUbuntu 22.04所有节点一致
Kubernetesv1.29三节点集群(1 Master + 2 Node)
Docker24.0.7已启用 /var/run/docker.sock
Jenkins2.462.1jenkins-admin ServiceAccount 部署在 K8s
Harbor2.9私有镜像仓库
GitLab16.x源代码托管
Python3.11Flask 应用运行环境

三、流水线总体结构

整个流程按阶段划分如下:

Start
  ↓
Checkout
  ↓
Test
  ↓
Build Docker
  ↓
Push Docker (Harbor)
  ↓
Deploy to K8s
  ↓
End

每个阶段都运行在 Jenkins 的 Kubernetes 动态 Agent Pod 中,通过不同容器完成对应任务。


四、Kubernetes 及 Jenkins 基础配置

1. Jenkins 使用的 ServiceAccount 与权限

创建 Jenkins 的集群角色与绑定:

# jenkins-01-serviceAccount.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-admin
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets"]
    verbs: ["*"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: devops-tools
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-admin
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: devops-tools

验证权限是否生效:

# first configured
kubectl apply -f jenkins-01-serviceAccount.yml
# and then ...
kubectl describe clusterrole jenkins-admin

输出中应包含:

Resources: *   Verbs: *

2. 创建应用命名空间

kubectl create namespace dev
kubectl get ns

确认输出中有:

dev   Active

五、自定义 kubectl-agent 镜像

为了在 Jenkins Agent 中执行 kubectl 命令,我们制作一个轻量级工具镜像:

# Dockerfile
FROM alpine:3.18
RUN apk add --no-cache bash curl wget ca-certificates
RUN wget https://dl.k8s.io/release/v1.34.1/bin/linux/amd64/kubectl -O /usr/local/bin/kubectl && \
    chmod +x /usr/local/bin/kubectl
WORKDIR /home/jenkins
CMD ["/bin/bash"]

构建与推送:

docker build -t kubernetes-registry.moon.com/moon/kubectl-agent:1.0 .
docker push kubernetes-registry.moon.com/moon/kubectl-agent:1.0

验证:

docker run --rm -it kubernetes-registry.moon.com/moon/kubectl-agent:1.0 bash
kubectl version --client

六、完整 Jenkinsfile

在撰写jenkinsfile之前可以参考:

https://www.cnblogs.com/fsdstudy/p/18263444

https://cloud.tencent.com/developer/article/2411170

pipeline {
    agent {
        kubernetes {
            cloud 'k8s-cloud'
            yaml '''
apiVersion: v1
kind: Pod
spec:
  serviceAccountName: jenkins-admin
  containers:
  - name: jnlp
    image: kubernetes-registry.moon.com/moon/inbound-agent:3345.v03dee9b_f88fc
    args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
  - name: docker
    image: kubernetes-registry.moon.com/moon/docker:24.0.7-cli
    command: ['cat']
    tty: true
    volumeMounts:
    - name: docker-sock
      mountPath: /var/run/docker.sock
  - name: python
    image: kubernetes-registry.moon.com/moon/python:3.11-slim
    command: ['cat']
    tty: true
  - name: kubectl-agent
    image: kubernetes-registry.moon.com/moon/kubectl-agent:1.0
    command: ['cat']
    tty: true
  volumes:
  - name: docker-sock
    hostPath:
      path: /var/run/docker.sock
      type: Socket
  dnsPolicy: ClusterFirst
  restartPolicy: Never
'''
        }
    }

    environment {
        REGISTRY = "kubernetes-registry.moon.com/moon"
        IMAGE = "my-flask-app"
        TAG = "v${env.BUILD_NUMBER}"
        HARBOR_CREDS = credentials('harbor-credentials')
    }

    stages {
        stage('Checkout') {
            steps { checkout scm }
        }

        stage('Test') {
            steps {
                container('python') {
                    sh '''
                    echo "Running pytest..."
                    pip install -r requirements.txt -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
                    pytest tests/
                    '''
                }
            }
        }

        stage('Build Docker') {
            steps {
                container('docker') {
                    sh '''
                    echo "Building image..."
                    docker build -t ${REGISTRY}/${IMAGE}:${TAG} .
                    '''
                }
            }
        }

        stage('Push Docker') {
            steps {
                container('docker') {
                    sh '''
                    echo "Pushing image to Harbor..."
                    echo "$HARBOR_CREDS_PSW" | docker login ${REGISTRY} -u "$HARBOR_CREDS_USR" --password-stdin
                    docker push ${REGISTRY}/${IMAGE}:${TAG}
                    '''
                }
            }
        }

        stage('Deploy to K8s') {
            steps {
                container('kubectl-agent') {
                    sh '''
                    echo "Deploying new image..."
                    kubectl set image deployment/my-flask-app my-flask-app=${REGISTRY}/${IMAGE}:${TAG} -n dev || \
                    kubectl create deployment my-flask-app --image=${REGISTRY}/${IMAGE}:${TAG} -n dev
                    '''
                }
            }
        }
    }

    post {
        success { echo "🎉 Deployment successful! ${REGISTRY}/${IMAGE}:${TAG}" }
        failure { echo "❌ Pipeline failed. Check the logs for details." }
    }
}

七、部署验证

查看 Deployment 和 Pod:

kubectl get deployments -n dev
kubectl get pods -n dev -o wide

确认 Pod 状态为:

my-flask-app   1/1     Running

测试应用:

curl <Pod_IP>:5000

输出示例:

{"env":"dev","message":"Hello from Flask"}

验证成功


八、常见问题与排错总结

问题原因解决方式
Cannot connect to the Docker daemonJenkins Pod 内无法访问宿主机 Docker在 Pod YAML 中挂载 /var/run/docker.sock
kubectl: not found镜像无 kubectl 命令使用自制 kubectl-agent
CrashLoopBackOff基础镜像无 bash改为 alpine:3.18 + bash
User ... cannot create deploymentsRBAC 权限不足创建 jenkins-admin ClusterRoleBinding
namespaces "dev" not found未创建命名空间kubectl create namespace dev
Address already in usePod 已启动 Flask 服务说明容器正常运行,无需手动启动

九、经验总结

  • Pipeline 分层清晰,每个容器负责单一任务(测试、构建、部署)。
  • 权限优先检查,RBAC 是部署阶段失败的高频原因。
  • 自定义工具镜像 能极大提升可复用性。
  • 每一步都要验证,特别是 RBAC、docker.sock、命名空间是否正确。
  • 最终闭环验证:能从 Jenkins 一键触发,到 K8s Pod 成功响应 HTTP 请求。

文档信息

Search

    Table of Contents