Jenkins + Docker + Harbor + Kubernetes CI/CD 实践记录
一步步构建可复现、可自动化的持续集成与持续部署流水线。
一、背景与目标
在现代微服务架构中,持续集成(CI)与持续部署(CD)是必不可少的基础设施。本次记录内容为:
通过 Jenkins 实现自动化构建、测试、打包、推送镜像到 Harbor,并自动部署到 Kubernetes 集群。
最终实现效果:
- Jenkins 自动拉取 GitLab 代码
- 自动运行测试
- 构建并推送 Docker 镜像到 Harbor
- 自动部署更新到 K8s 集群
- Flask 应用自动启动并可访问
二、系统环境概览
组件 | 版本 | 说明 |
---|---|---|
OS | Ubuntu 22.04 | 所有节点一致 |
Kubernetes | v1.29 | 三节点集群(1 Master + 2 Node) |
Docker | 24.0.7 | 已启用 /var/run/docker.sock |
Jenkins | 2.462.1 | 以 jenkins-admin ServiceAccount 部署在 K8s |
Harbor | 2.9 | 私有镜像仓库 |
GitLab | 16.x | 源代码托管 |
Python | 3.11 | Flask 应用运行环境 |
三、流水线总体结构
整个流程按阶段划分如下:
Start
↓
Checkout
↓
Test
↓
Build Docker
↓
Push Docker (Harbor)
↓
Deploy to K8s
↓
End
每个阶段都运行在 Jenkins 的 Kubernetes 动态 Agent Pod 中,通过不同容器完成对应任务。
四、Kubernetes 及 Jenkins 基础配置
1. Jenkins 使用的 ServiceAccount 与权限
创建 Jenkins 的集群角色与绑定:
# jenkins-01-serviceAccount.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jenkins-admin
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets", "daemonsets"]
verbs: ["*"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins-admin
namespace: devops-tools
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jenkins-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins-admin
subjects:
- kind: ServiceAccount
name: jenkins-admin
namespace: devops-tools
验证权限是否生效:
# first configured
kubectl apply -f jenkins-01-serviceAccount.yml
# and then ...
kubectl describe clusterrole jenkins-admin
输出中应包含:
Resources: * Verbs: *
2. 创建应用命名空间
kubectl create namespace dev
kubectl get ns
确认输出中有:
dev Active
五、自定义 kubectl-agent 镜像
为了在 Jenkins Agent 中执行 kubectl
命令,我们制作一个轻量级工具镜像:
# Dockerfile
FROM alpine:3.18
RUN apk add --no-cache bash curl wget ca-certificates
RUN wget https://dl.k8s.io/release/v1.34.1/bin/linux/amd64/kubectl -O /usr/local/bin/kubectl && \
chmod +x /usr/local/bin/kubectl
WORKDIR /home/jenkins
CMD ["/bin/bash"]
构建与推送:
docker build -t kubernetes-registry.moon.com/moon/kubectl-agent:1.0 .
docker push kubernetes-registry.moon.com/moon/kubectl-agent:1.0
验证:
docker run --rm -it kubernetes-registry.moon.com/moon/kubectl-agent:1.0 bash
kubectl version --client
六、完整 Jenkinsfile
在撰写jenkinsfile之前可以参考:
https://www.cnblogs.com/fsdstudy/p/18263444
https://cloud.tencent.com/developer/article/2411170
pipeline {
agent {
kubernetes {
cloud 'k8s-cloud'
yaml '''
apiVersion: v1
kind: Pod
spec:
serviceAccountName: jenkins-admin
containers:
- name: jnlp
image: kubernetes-registry.moon.com/moon/inbound-agent:3345.v03dee9b_f88fc
args: ['$(JENKINS_SECRET)', '$(JENKINS_NAME)']
- name: docker
image: kubernetes-registry.moon.com/moon/docker:24.0.7-cli
command: ['cat']
tty: true
volumeMounts:
- name: docker-sock
mountPath: /var/run/docker.sock
- name: python
image: kubernetes-registry.moon.com/moon/python:3.11-slim
command: ['cat']
tty: true
- name: kubectl-agent
image: kubernetes-registry.moon.com/moon/kubectl-agent:1.0
command: ['cat']
tty: true
volumes:
- name: docker-sock
hostPath:
path: /var/run/docker.sock
type: Socket
dnsPolicy: ClusterFirst
restartPolicy: Never
'''
}
}
environment {
REGISTRY = "kubernetes-registry.moon.com/moon"
IMAGE = "my-flask-app"
TAG = "v${env.BUILD_NUMBER}"
HARBOR_CREDS = credentials('harbor-credentials')
}
stages {
stage('Checkout') {
steps { checkout scm }
}
stage('Test') {
steps {
container('python') {
sh '''
echo "Running pytest..."
pip install -r requirements.txt -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pytest tests/
'''
}
}
}
stage('Build Docker') {
steps {
container('docker') {
sh '''
echo "Building image..."
docker build -t ${REGISTRY}/${IMAGE}:${TAG} .
'''
}
}
}
stage('Push Docker') {
steps {
container('docker') {
sh '''
echo "Pushing image to Harbor..."
echo "$HARBOR_CREDS_PSW" | docker login ${REGISTRY} -u "$HARBOR_CREDS_USR" --password-stdin
docker push ${REGISTRY}/${IMAGE}:${TAG}
'''
}
}
}
stage('Deploy to K8s') {
steps {
container('kubectl-agent') {
sh '''
echo "Deploying new image..."
kubectl set image deployment/my-flask-app my-flask-app=${REGISTRY}/${IMAGE}:${TAG} -n dev || \
kubectl create deployment my-flask-app --image=${REGISTRY}/${IMAGE}:${TAG} -n dev
'''
}
}
}
}
post {
success { echo "🎉 Deployment successful! ${REGISTRY}/${IMAGE}:${TAG}" }
failure { echo "❌ Pipeline failed. Check the logs for details." }
}
}
七、部署验证
查看 Deployment 和 Pod:
kubectl get deployments -n dev
kubectl get pods -n dev -o wide
确认 Pod 状态为:
my-flask-app 1/1 Running
测试应用:
curl <Pod_IP>:5000
输出示例:
{"env":"dev","message":"Hello from Flask"}
验证成功
八、常见问题与排错总结
问题 | 原因 | 解决方式 |
---|---|---|
Cannot connect to the Docker daemon | Jenkins Pod 内无法访问宿主机 Docker | 在 Pod YAML 中挂载 /var/run/docker.sock |
kubectl: not found | 镜像无 kubectl 命令 | 使用自制 kubectl-agent |
CrashLoopBackOff | 基础镜像无 bash | 改为 alpine:3.18 + bash |
User ... cannot create deployments | RBAC 权限不足 | 创建 jenkins-admin ClusterRoleBinding |
namespaces "dev" not found | 未创建命名空间 | kubectl create namespace dev |
Address already in use | Pod 已启动 Flask 服务 | 说明容器正常运行,无需手动启动 |
九、经验总结
- Pipeline 分层清晰,每个容器负责单一任务(测试、构建、部署)。
- 权限优先检查,RBAC 是部署阶段失败的高频原因。
- 自定义工具镜像 能极大提升可复用性。
- 每一步都要验证,特别是 RBAC、docker.sock、命名空间是否正确。
- 最终闭环验证:能从 Jenkins 一键触发,到 K8s Pod 成功响应 HTTP 请求。
文档信息
- 本文作者:Chaojie Men
- 本文链接:https://menchaojie.github.io/2025/10/20/jenkins/
- 版权声明:自由转载-非商用-非衍生-保持署名(创意共享3.0许可证)