本课讲得是基于配置中心数据库实现
主讲:k8s configmap,apollo
configmap和secret
在K8S中,有两种方式管理资源的配置,分别是configmap和secret,他们的最大区别是:
0.1.2 怎么使用configmap
0.1.3 更新configmap配置的方法
单配置更新法:
多配置更换法:
先将前面部署的3个dubbo服务的POD个数全部调整(scale)为0个,避免在应用configmap过程中可能的报错,也为了节省资源
直接在dashboard上操作即可,
App命名空间--à生产者和消费者都伸缩成0
Dubbo-monitor也设置为0
分测试和生产环境了,将原来3台zk集群连接断开,将zk1,和zk2做成单独的zk使用,zk3关掉
主机名
角色
ip
10.4.7.11
10.4.7.12
先关闭zk集群
#删除11,12节点上zookeeper上的数据和日志
[root@hdss7-12 ~]# rm /data/zookeeper/data/* -rf
[root@hdss7-12 ~]# rm /data/zookeeper/logs/* -rf
#修改配置文件.删除与其他主机通信(11,12主机上)
保留如下这些
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
重新开启(11,12)
老规矩,资源清单在7.200运维机上统一操作
cd /data/k8s-yaml/dubbo-monitor
创建comfigmap清单
apiVersion: v1
kind: ConfigMap
metadata:
name: dubbo-monitor-cm
namespace: infra
data:
EOF
其实就是把dubbo-monitor配置文件中的内容用configmap语法展示出来了
当然最前面加上了相应的元数据信息
Configmap作为一个卷挂载到了容器里
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
labels:
name: dubbo-monitor
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-monitor
template:
metadata:
labels:
app: dubbo-monitor
name: dubbo-monitor
spec:
containers:
- name: dubbo-monitor
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
imagePullPolicy: IfNotPresent
#----------------start---------------------------
volumeMounts:
- name: configmap-volume
mountPath: /dubbo-monitor-simple/conf
volumes:
- name: configmap-volume
configMap: #挂载到了configmap-volume目录上,容器中目录
name: dubbo-monitor-cm
#----------------end-----------------------------
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
EOF
Umask使挂载只读
应用资源配置清单
dashboard检查创建结果
在dashboard中查看infra名称空间中的configmap资源
然后检查容器中的配置
kubectl -n infra exec -it dubbo-monitor-5b7cdddbc5-xpft6 bash
# 容器内
....
检查dubbo-monitor页面的注册信息
多配置更新法
# 把资源名字改成dubbo-monitor-cm-pro
dashboard上修改zook地址
更新资源
# 应用新configmap
# 更新deploy
检查配置是否更新
新的pod已经起来了
~]# kubectl -n infra get pod
NAME READY STATUS RESTARTS AGE
dubbo-monitor-c7fbf68b9-7nffj 1/1 Running 0 52s
进去看看是不是应用的新的configmap配置:
kubectl -n infra exec -it dubbo-monitor-5cb756cc6c-xtnrt bash
# 容器内
看下dubbo-monitor的页面:已经是zk2了。
monutPath挂载的问题
我们使用的是mountPath,这个是挂载整个目录,会使容器内的被挂载目录中原有的文件不可见,可以看见我们。
查看我们pod容器启动的命令可以看见原来脚本中的命令已经无法对挂载的目录操作了
如何单独挂载一个配置文件:
只挂载单独一个文件而不是整个目录,需要添加subPath方法
更新配置
#----------------start---------------------------
volumeMounts:#卷挂载动作
- name: configmap-volume
mountPath: /dubbo-monitor-simple/conf
volumes:#声明卷
- name: configmap-volume
configMap:
name: dubbo-monitor-cm
#----------------end-----------------------------
# 调整为
#----------------start---------------------------
volumeMounts:
- name: configmap-volume
mountPath: /dubbo-monitor-simple/conf
- name: configmap-volume
volumes:
- name: configmap-volume
configMap:
name: dubbo-monitor-cm
#----------------end-----------------------------
kubectl -n infra exec -it dubbo-monitor-5cb756cc6c-xtnrt bash
# 容器内操作
bash-4.3# ls -l /var/
total 4
drwxr-xr-x 1 root root 29 Apr 13 2016 cache
-rw-r--r-- 1 root root 459 May
drwxr-xr-x 2 root root 6 Apr 1 2016 empty
.....
Client: dubbo-demo-custom,dubbo-demo-service
Apoll3兄弟:config service,admin service,portal
将dubbo的配置,写到config DB
Portal:web界面,可以在页面修改configdb中的配置数据
Apollo的注册中心
在运维主机200上执行
[root@mfyxw50 ~]# cd /opt/src
[root@mfyxw50 src]# mkdir -p /data/dockerfile/apollo-configservice
注意:MySQL版本应为5.6或以上!
导入GPG-KEY
[root@mfyxw10 ~]# yum makecache
[root@mfyxw10 ~]# yumlistMariaDB-server--show-duplicates
Loadedplugins: fastestmirror
Loadingmirrorspeedsfromcachedhostfile
AvailablePackages
[root@mfyxw10 ~]# yum -y install MariaDB-server
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
# this is read by the standalone daemon and embedded servers
[server]
# this is only for the mysqld standalone daemon
[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"
# * Galera-related settings
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
# Allow server to accept connections on all interfaces.
#bind-address=0.0.0.0
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
# this is only for embedded server
[embedded]
# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
# This group is only read by MariaDB-10.1 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.1]
EOF
# These groups are read by MariaDB command-line tools
# Use it for options that affect only one utility
[mysql]
default-character-set = utf8mb4
[mysql_upgrade]
[mysqladmin]
[mysqlbinlog]
[mysqlcheck]
[mysqldump]
[mysqlimport]
[mysqlshow]
[mysqlslap]
EOF
[root@mfyxw10 ~]# systemctl enable --now mariadb
[root@hdss7-11 ~]# systemctl start mysql
[root@hdss7-11 ~]# systemctl status mysql
[root@mfyxw10 ~]# netstat -tanlp | grep mysql #查看MariaDB的启动端口
[root@mfyxw10 ~]# mysqladmin -uroot password #设置密码:回车
[root@mfyxw10 ~]# mysql -uroot -p #输入数据库密码登录
MariaDB [(none)]> \s #检查字符集
MariaDB [(none)]> drop database test;
MariaDB [(none)]> use mysql;
MariaDB [mysql]> delete from user where user='';
或都使用如下命令对MariaDB数据库初始化设置
[root@mfyxw10 ~]# mysql_secure_installation
[root@mfyxw10 ~]# mysql-uroot-p
MariaDB[(none)]> \s
[root@mfyxw50 ~]# mkdir /data/dockerfile/apollo-portal
# DataSource
EOF
EOF
#!/bin/bash
SERVICE_NAME=apollo-portal
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_PORTAL_SERVICE_NAME=\$(hostname -i)
## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"
## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="\$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"
########### The following is the same for configservice, adminservice, portal ###########
# Find Java
if [[ -n "\$JAVA_HOME" ]] && [[ -x "\$JAVA_HOME/bin/java" ]]; then
javaexe="\$JAVA_HOME/bin/java"
eliftype -p java > /dev/null 2>&1; then
javaexe=\$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo"Unable to find Java"
exit 1
fi
if [[ "\$javaexe" ]]; then
version=\$("\$javaexe" -version 2>&1 | awk -F '"''/version/ {print \$2}')
version=\$(echo"\$version" | awk -F. '{printf("%03d%03d",\$1,\$2);}')
# now version is of format 009003 (9.3.x)
if [ \$version -ge 011000 ]; then
elif [ \$version -ge 010000 ]; then
elif [ \$version -ge 009000 ]; then
else
JAVA_OPTS="\$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="\$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi
printf"\$(date) ==== Starting ==== \n"
cd \`dirname \$0\`/..
chmod 755 \$SERVICE_NAME".jar"
./\$SERVICE_NAME".jar" start
rc=\$?;
if [[ \$rc != 0 ]];
then
exit \$rc;
fi
tail -f /dev/null
EOF
[root@mfyxw50 ~]# cat > /data/dockerfile/apollo-portal/Dockerfile << EOF
ENV VERSION 1.5.1
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo"Asia/Shanghai" > /etc/timezone
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts
EOF
[root@mfyxw50 ~]# cd /data/dockerfile/apollo-portal
\$TTL 600 ; 10 minutes
;序号请加1,表示比之前版本要新
2020031312 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
\$TTL 60 ; 1 minute
dns A 192.168.80.10
harbor A 192.168.80.50 ;添加harbor记录
k8s-yaml A 192.168.80.50
traefik A 192.168.80.100
dashboard A 192.168.80.100
zk1 A 192.168.80.10
zk2 A 192.168.80.20
zk3 A 192.168.80.30
jenkins A 192.168.80.100
dubbo-monitor A 192.168.80.100
demo A 192.168.80.100
mysql A 192.168.80.10
config A 192.168.80.100
portal A 192.168.80.100
EOF
[root@mfyxw10 ~]# systemctl restart named
192.168.80.100
[root@mfyxw50 ~]# mkdir -p /data/k8s-yaml/apollo-portal
kind:Deployment
apiVersion:extensions/v1beta1
metadata:
name:apollo-portal
namespace:infra
labels:
name:apollo-portal
spec:
replicas:1
selector:
matchLabels:
name:apollo-portal
template:
metadata:
labels:
app:apollo-portal
name:apollo-portal
spec:
volumes:
- name:configmap-volume
configMap:
name:apollo-portal-cm
containers:
- name:apollo-portal
ports:
- containerPort:8080
protocol:TCP
volumeMounts:
- name:configmap-volume
mountPath:/apollo-portal/config
terminationMessagePath:/dev/termination-log
terminationMessagePolicy:File
imagePullPolicy:IfNotPresent
imagePullSecrets:
- name:harbor
restartPolicy:Always
terminationGracePeriodSeconds:30
securityContext:
runAsUser:0
schedulerName:default-scheduler
strategy:
type:RollingUpdate
rollingUpdate:
maxUnavailable:1
maxSurge:1
revisionHistoryLimit:7
progressDeadlineSeconds:600
EOF
kind:Service
apiVersion:v1
metadata:
name:apollo-portal
namespace:infra
spec:
ports:
- protocol:TCP
port:8080
targetPort:8080
selector:
app:apollo-portal
clusterIP:None
type:ClusterIP
sessionAffinity:None
EOF
kind:Ingress
apiVersion:extensions/v1beta1
metadata:
name:apollo-portal
namespace:infra
spec:
rules:
paths:
- path:/
backend:
serviceName:apollo-portal
servicePort:8080
EOF
apiVersion:v1
kind:ConfigMap
metadata:
name:apollo-portal-cm
namespace:infra
data:
# DataSource
appId=100003173
EOF
[root@mfyxw30 ~]#
service/apollo-portal created
[root@mfyxw30 ~]#
[root@mfyxw30 ~]#
configmap/apollo-portal-cm created
[root@mfyxw30 ~]# kubectl get pod -n infra
NAME READY STATUS RESTARTS AGE
apollo-adminservice-5cccf97c64-bhqzb 1/1 Running 16h22m
apollo-configservice-5f6555448-7wxsp 1/1 Running 16h16m
apollo-portal-57bc86966d-jz5vg 1/1 Running 0112s
dubbo-monitor-6676dd74cc-9hghb 1/1 Running 1317d
dubbo-monitor-6676dd74cc-rd86g 1/1 Running 1217d
jenkins-b99776c69-p6skp 1/1 Running 1739d
对应修改了数据库键值如下
添加配置
#zk1是测试环境
#apollo分支编写的参数变量是对应apollo注册的参数
Jenkins-编写拉取参数
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-service
namespace: app
labels:
name: dubbo-demo-service
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-service
template:
metadata:
labels:
app: dubbo-demo-service
name: dubbo-demo-service
spec:
containers:
- name: dubbo-demo-service
ports:
- containerPort: 20880
protocol: TCP
env:
- name: JAR_BALL
- name: C_OPTS #添加的新环境变量
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
对应了dubbo-service服务提供者程序中如下2个变量
Register地址代表,服务注册的zk数据库地址,
Port代表,monitor探查dubbo-service服务状态暴露的端口,如果这里改了(dubbo-monitor应当也该改地址)
#相当于以环境变量的方式,在程序启动时中调用.
在Portal的web界面创建项目,写入键值,相当于在dashboard修改了configmap中的配置
Portal会把数据传递给adminservice,adminservice又把数据存到configdb中
注: 可以在程序中实现apollo监听器的方法,实现实时监听apollo配置中心键值变化,实时reload生效,达到pod更新配置的效果
不需要,人为删除pod重启
和dubbo服务提供者一样.重新制作dubbo-web的harbor镜像(将配置环境变量化)
提交,再点一下发布
#开始构建
再次应用.apollo的配置就在pod里的项目生效了
在实际企业环境,应该将测试环境和生产环境放到2个不同的k8s集群,而不是现在这样以名称空间区分
环境
命名空间
应用
测试环境(TEST)
test
apollo-config,apollo-admin,zk1,apolloconfigdb
测试环境(TEST)
test
dubbo-demo-service,dubbo-demo-web
生产环境(PROD)
prod
apollo-config,apollo-admin, apolloconfigdb
生产环境(PROD)
prod
dubbo-demo-service,dubbo-demo-web,zk2
ops环境(infra)--共用
infra
jenkins,dubbo-monitor,apollo-portal
1)首先停掉apollo3兄弟
创建test名称空间
[root@hdss7-21 ~]# kubectl create ns test
namespace/test created
#在test名称空间创建secret资源
查看创建的apolloconfig测试库
#修改数据库连接eruke信息,
#测试库使用的地址改一下
MariaDB [ApolloConfigTestDB]> select * from ServerConfig\G;
MariaDB [(none)]> grant SELECT,DELETE,UPDATE,INSERT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
Query OK, 0 rows affected (0.00 sec)
[root@hdss7-200 k8s-yaml]# mkdir -pv test/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
[root@hdss7-200 apollo-configservice]# pwd
/data/k8s-yaml/test/apollo-configservice
[root@hdss7-200 apollo-configservice]# cp /data/k8s-yaml/apollo-configservice/* .
[root@hdss7-200 apollo-configservice]# ls
修改资源清单中(所有)
1.名称空间改为test
修改资源清单(configmap,dp)
名称空间改成test
Configmap改:
应用资源配置清单:
[root@hdss7-200 dubbo-demo-service]# pwd
/data/k8s-yaml/test/dubbo-demo-service
namespace: test
[root@hdss7-200 dubbo-demo-consumer]# cp /data/k8s-yaml/dubbo-consumer/*.yaml .
namespace: test (ingress,dp,svc)
demo-test A 10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
1)首先停掉apollo3兄弟
创建prod名称空间
[root@hdss7-21 ~]# kubectl create ns prod
namespace/test created
#在prod名称空间创建secret资源
创建生产环境的configdb
#修改数据库连接eruke信息,
#生产库使用的地址改一下
MariaDB [ApolloConfigTestDB]> select * from ServerConfig\G;
MariaDB [(none)]> grant SELECT,DELETE,UPDATE,INSERT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
Query OK, 0 rows affected (0.00 sec)
[root@hdss7-200 k8s-yaml]# mkdir -pv prod/{apollo-configservice,apollo-adminservice,dubbo-demo-service,dubbo-demo-consumer}
[root@hdss7-200 apollo-configservice]# pwd
/data/k8s-yaml/prod/apollo-configservice
[root@hdss7-200 apollo-configservice]# cp /data/k8s-yaml/test/apollo-configservice/* .
[root@hdss7-200 apollo-configservice]# sed -i 's/test/prod/g' ./* #替换成prod
再将连接的数据库地址改掉
名称空间改成prod
Configmap改:
[root@hdss7-200 apollo-adminservice]# cp /data/k8s-yaml/test/apollo-adminservice/* .
[root@hdss7-200 apollo-adminservice]# sed -i 's/test/prod/g' ./* #不要这样改,会把harbor地址改错
手动修改数据库地址,域名
demo-prod A 10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
[root@hdss7-200 dubbo-demo-service]# cp /data/k8s-yaml/test/dubbo-demo-service/*.yaml .
namespace: prod
[root@hdss7-200 dubbo-demo-consumer]# cp /data/k8s-yaml/test/dubbo-demo-consumer/*.yaml .
名称空间
namespace: prod (ingress,dp,svc)
11主机上,mysql
drop database ApolloConfigDB;
MariaDB [ApolloPortalDB]> use ApolloPortalDB;
MariaDB [ApolloPortalDB]> update ServerConfig set value='fat,pro' where Id=1; #可支持的环境列表修改
修改标黄2行
应用cm资源清单
MariaDB [ApolloPortalDB]> use ApolloPortalDB;
MariaDB [ApolloPortalDB]> truncate table AppNamespace;
MariaDB [ApolloPortalDB]> truncate table App;
#检查admin service资源清单,发现harbor地址被替换错了
#如果apollo开启之前,项目创建有问题,需要删除重建
#添加对应环境的参数,添加完后点发布再编辑生产环境
#生产环境参数发布
创建项目
测试环境测试新版镜像,如果正常可以访问
再将生产环境消费者dp资源配置文件使用新版镜像(用测试环境做的镜像包)
1. dubbo的3个组件,提供者,消费者,发现者,他们通信的端口,zk地址需要保证一致性
2. dubbo的3个组件的启动顺序需要注意,monitor需要在后启动,不然在web界面容易发现不了其他2个组件
4. dubbo-monitor网站打不开,一定是zookeeper宕了.
5. 所有的问题都排查过了,发现dashboard上删除掉的dp,在k8s上居然没被删掉
6.启动顺序,zk,portal,数据库,apolloservice,apolloadmin,dubboservice,dubboconsumer