实验文档3:在kubernetes集群里集成Apollo配置中心


欢迎加入王导的VIP学习qq群:==>932194668<==


使用ConfigMap管理应用配置

拆分环境

主机名 角色 ip
HDSS7-11.host.com zk1.od.com(Test环境) 10.4.7.11
HDSS7-12.host.com zk2.od.com(Prod环境) 10.4.7.12

重配zookeeper

HDSS7-11.host.com上:

/opt/zookeeper/conf/zoo.cfg
1
2
3
4
5
6
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181

HDSS7-12.host.com上:

/opt/zookeeper/conf/zoo.cfg
1
2
3
4
5
6
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181

重启zk(删除数据文件)

1
2
3
[root@hdss7-11 ~]# /opt/zookeeper/bin/zkServer.sh restart && /opt/zookeeper/bin/zkServer.sh status
[root@hdss7-12 ~]# /opt/zookeeper/bin/zkServer.sh restart && /opt/zookeeper/bin/zkServer.sh status
[root@hdss7-21 ~]# /opt/zookeeper/bin/zkServer.sh stop

准备资源配置清单(dubbo-monitor)

在运维主机HDSS7-200.host.com上:

vi /data/k8s-yaml/dubbo-monitor/configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: v1
kind: ConfigMap
metadata:
name: dubbo-monitor-cm
namespace: infra
data:
dubbo.properties: |
dubbo.container=log4j,spring,registry,jetty
dubbo.application.name=simple-monitor
dubbo.application.owner=
dubbo.registry.address=zookeeper://zk1.od.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
dubbo.log4j.level=WARN

vi /data/k8s-yaml/dubbo-monitor/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-monitor
namespace: infra
labels:
name: dubbo-monitor
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-monitor
template:
metadata:
labels:
app: dubbo-monitor
name: dubbo-monitor
spec:
containers:
- name: dubbo-monitor
image: harbor.od.com/infra/dubbo-monitor:latest
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 20880
protocol: TCP
imagePullPolicy: IfNotPresent
volumeMounts:
- name: configmap-volume
mountPath: /dubbo-monitor-simple/conf
volumes:
- name: configmap-volume
configMap:
name: dubbo-monitor-cm
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600

应用资源配置清单

在任意一台k8s运算节点执行:

1
2
3
4
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/configmap.yaml
configmap/dubbo-monitor-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/deployment.yaml
deployment.extensions/dubbo-monitor configured

重新发版,修改dubbo项目的配置文件

修改项目源代码

  • duboo-demo-service

    dubbo-server/src/main/java/config.properties
    1
    2
    dubbo.registry=zookeeper://zk1.od.com:2181
    dubbo.port=28080
  • dubbo-demo-web

    dubbo-client/src/main/java/config.properties
    1
    dubbo.registry=zookeeper://zk1.od.com:2181

使用Jenkins进行CI

修改/应用资源配置清单

k8s的dashboard上,修改deployment使用的容器版本,提交应用

验证configmap的配置

在K8S的dashboard上,修改dubbo-monitor的configmap配置为不同的zk,重启POD,浏览器打开http://dubbo-monitor.od.com 观察效果

交付Apollo至Kubernetes集群

Apollo简介

Apollo(阿波罗)是携程框架部门研发的分布式配置中心,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景。

官方GitHub地址

Apollo官方地址
官方release包

基础架构

apollo基础架构

简化模型

apollo简化架构

交付apollo-configservice

准备软件包

在运维主机HDSS7-200.host.com上:
下载官方release包

/opt/src
1
2
3
4
5
6
7
8
9
10
11
12
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 23:29 apollo-configservice-1.3.0-github.zip
[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-configservice && unzip -o apollo-configservice-1.3.0-github.zip -d /data/dockerfile/apollo-configservice
Archive: apollo-configservice-1.3.0-github.zip
creating: /data/dockerfile/apollo-configservice/scripts/
inflating: /data/dockerfile/apollo-configservice/config/application-github.properties
inflating: /data/dockerfile/apollo-configservice/scripts/shutdown.sh
inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.3.0-sources.jar
inflating: /data/dockerfile/apollo-configservice/scripts/startup.sh
inflating: /data/dockerfile/apollo-configservice/config/app.properties
inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.3.0.jar
inflating: /data/dockerfile/apollo-configservice/apollo-configservice.conf

执行数据库脚本

在数据库主机HDSS7-11.host.com上:
注意:MySQL版本应为5.6或以上!

  • 更新yum源
/etc/yum.repos.d/MariaDB.repo
1
2
3
4
5
[mariadb]
name = MariaDB
baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64/
gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1
  • 导入GPG-KEY
1
[root@hdss7-11 ~]# rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
  • 更新数据库版本
1
[root@hdss7-11 ~]# yum update MariaDB-server -y
  • 配置my.cnf
/etc/my.cnf
1
2
3
4
5
6
[mysql]
default-character-set = utf8mb4
[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"

数据库脚本地址

1
2
3
[root@hdss7-11 ~]# mysql -uroot -p
mysql> create database ApolloConfigDB;
mysql> source ./apolloconfig.sql

数据库用户授权

1
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

修改初始数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
mysql> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

mysql> select * from ServerConfig\G
*************************** 1. row ***************************
Id: 1
Key: eureka.service.url
Cluster: default
Value: http://config.od.com/eureka
Comment: Eureka服务Url,多个service以英文逗号分隔
IsDeleted:
DataChange_CreatedBy: default
DataChange_CreatedTime: 2019-04-10 15:07:34
DataChange_LastModifiedBy:
DataChange_LastTime: 2019-04-11 16:28:57

制作Docker镜像

在运维主机HDSS7-200.host.com上:

  • 配置数据库连接串
/data/dockerfile/apollo-configservice
1
[root@hdss7-200 apollo-configservice]# cat config/application-github.properties
  • 更新startup.sh
/data/dockerfile/apollo-configservice/scripts/startup.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
#!/bin/bash
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms6144m -Xmx6144m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=4096m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]]; then
javaexe="/usr/bin/java"
else
echo "Unable to find Java"
exit 1
fi

if [[ "$javaexe" ]]; then
version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
# now version is of format 009003 (9.3.x)
if [ $version -ge 011000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 010000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
elif [ $version -ge 009000 ]; then
JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
else
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
exit $rc;
fi

tail -f /dev/null
  • 写Dockerfile
/data/dockerfile/apollo-configservice/Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
FROM stanleyws/jre8:8u112

ENV VERSION 1.3.0

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo "Asia/Shanghai" > /etc/timezone

ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
ADD config/ /apollo-configservice/config
ADD scripts/ /apollo-configservice/scripts

CMD ["/apollo-configservice/scripts/startup.sh"]
  • 制作镜像并推送
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
[root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.3.0
Sending build context to Docker daemon 61.91 MB
Step 1 : FROM stanleyws/jre8:8u112
---> fa3a085d6ef1
Step 2 : ENV VERSION 1.3.0
---> [Warning] IPv4 forwarding is disabled. Networking will not work.
---> Running in 685d51b5adb4
---> feb4c0289f04
Removing intermediate container 685d51b5adb4
Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
---> [Warning] IPv4 forwarding is disabled. Networking will not work.
---> Running in eaa05073feeb
---> a3e3fd61ae35
Removing intermediate container eaa05073feeb
Step 4 : ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
---> be09a59b83a2
Removing intermediate container ac6b8af3979b
Step 5 : ADD config/ /apollo-configservice/config
---> fb64fc0f3194
Removing intermediate container b73c5315ad20
Step 6 : ADD scripts/ /apollo-configservice/scripts
---> 96ff3d9b9456
Removing intermediate container 67ba203b3101
Step 7 : CMD /apollo-configservice/scripts/startup.sh
---> [Warning] IPv4 forwarding is disabled. Networking will not work.
---> Running in 80bd3f53fefc
---> 551ea2ba8de3
Removing intermediate container 80bd3f53fefc
Successfully built 551ea2ba8de3

[root@hdss7-200 apollo-configservice]# docker push harbor.od.com/infra/apollo-configservice:v1.3.0
The push refers to a repository [harbor.od.com/infra/apollo-configservice]
25efb9a44683: Pushed
b3572bb46247: Pushed
e7994b936025: Pushed
0ff1d078cbc4: Pushed
ebfb473df5c2: Pushed
aae5c057d1b6: Pushed
dee6aef5c2b6: Pushed
a464c54f93a9: Pushed
v1.3.0: digest: sha256:6a8e4fdda58de0dfba9985ebbf91c4d6f46f5274983d2efa8853b03f4e45fa06 size: 1992

解析域名

DNS主机HDSS7-11.host.com上:

/var/named/od.com.zone
1
2
mysql   60 IN A 10.4.7.11
config 60 IN A 10.4.7.10

准备资源配置清单

在运维主机HDSS7-200.host.com

/data/k8s-yaml
1
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-configservice && cd /data/k8s-yaml/apollo-configservice

vi deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
labels:
name: apollo-configservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-configservice
template:
metadata:
labels:
app: apollo-configservice
name: apollo-configservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-configservice-cm
containers:
- name: apollo-configservice
image: harbor.od.com/infra/apollo-configservice:v1.3.0
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-configservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600

vi svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: Service
apiVersion: v1
metadata:
name: apollo-configservice
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-configservice
clusterIP: None
type: ClusterIP
sessionAffinity: None

vi ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-configservice
namespace: infra
spec:
rules:
- host: config.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-configservice
servicePort: 8080

vi configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-configservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config.od.com/eureka
app.properties: |
appId=100003171

应用资源配置清单

在任意一台k8s运算节点执行:

1
2
3
4
5
6
7
8
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/configmap.yaml
configmap/apollo-configservice-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/deployment.yaml
deployment.extensions/apollo-configservice created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml
service/apollo-configservice created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml
ingress.extensions/apollo-configservice created

浏览器访问

http://config.od.com

交付apollo-adminservice

准备软件包

在运维主机HDSS7-200.host.com上:
下载官方release包

1
2
3
4
5
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 08:47 apollo-configservice-1.3.0-github.zip
-rw-r--r-- 1 root root 49418246 Feb 16 09:54 apollo-adminservice-1.3.0-github.zip

[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-adminservice && unzip -o apollo-adminservice-1.3.0-github.zip -d /data/dockerfile/apollo-adminservice

制作Docker镜像

在运维主机HDSS7-200.host.com上:

  • 配置数据库连接串

    /data/dockerfile/apollo-adminservice
    1
    [root@hdss7-200 apollo-adminservice]# cat config/application-github.properties
  • 更新starup.sh

    /data/dockerfile/apollo-adminservice/scripts/startup.sh
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    #!/bin/bash
    SERVICE_NAME=apollo-adminservice
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-adminservice
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:${SERVER_PORT}"
    SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"

    ## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]]; then
    javaexe="/usr/bin/java"
    else
    echo "Unable to find Java"
    exit 1
    fi

    if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
    JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
    JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
    JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
    JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
    JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
    JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
    fi

    printf "$(date) ==== Starting ==== \n"

    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start

    rc=$?;

    if [[ $rc != 0 ]];
    then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
    fi

    tail -f /dev/null
  • 写Dockerfile

    /data/dockerfile/apollo-adminservice/Dockerfile
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    FROM stanleyws/jre8:8u112

    ENV VERSION 1.3.0

    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone

    ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
    ADD config/ /apollo-adminservice/config
    ADD scripts/ /apollo-adminservice/scripts

    CMD ["/apollo-adminservice/scripts/startup.sh"]
  • 制作镜像并推送

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    [root@hdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.3.0
    Sending build context to Docker daemon 58.31 MB
    Step 1 : FROM stanleyws/jre8:8u112
    ---> fa3a085d6ef1
    Step 2 : ENV VERSION 1.3.0
    ---> Using cache
    ---> feb4c0289f04
    Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
    ---> Using cache
    ---> a3e3fd61ae35
    Step 4 : ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
    ---> 6a1eb9565777
    Removing intermediate container 7196df9af6af
    Step 5 : ADD config/ /apollo-adminservice/config
    ---> 9f364b732d46
    Removing intermediate container 9b24669c6c78
    Step 6 : ADD scripts/ /apollo-adminservice/scripts
    ---> b7bc5517b0fc
    Removing intermediate container f3e34e759148
    Step 7 : CMD /apollo-adminservice/scripts/startup.sh
    ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
    ---> Running in 18c6597914b4
    ---> 82145db3ee88
    Removing intermediate container 18c6597914b4
    Successfully built 82145db3ee88

    [root@hdss7-200 apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.3.0
    docker push harbor.od.com/infra/apollo-adminservice:v1.3.0
    The push refers to a repository [harbor.od.com/infra/apollo-adminservice]
    19b1ca6c066d: Pushed
    8fa6cde49908: Pushed
    0b2c9b9226cc: Pushed
    ebfb473df5c2: Mounted from infra/apollo-configservice
    aae5c057d1b6: Mounted from infra/apollo-configservice
    dee6aef5c2b6: Mounted from infra/apollo-configservice
    a464c54f93a9: Mounted from infra/apollo-configservice
    v1.3.0: digest: sha256:75367caab9bad3d0d281eb3324451a0734e84b6aa3ee860e38ad758d7166a7d1 size: 1785

准备资源配置清单

在运维主机HDSS7-200.host.com

/data/k8s-yaml
1
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-adminservice && cd /data/k8s-yaml/apollo-adminservice

vi deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-adminservice
namespace: infra
labels:
name: apollo-adminservice
spec:
replicas: 1
selector:
matchLabels:
name: apollo-adminservice
template:
metadata:
labels:
app: apollo-adminservice
name: apollo-adminservice
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-adminservice-cm
containers:
- name: apollo-adminservice
image: harbor.od.com/infra/apollo-adminservice:v1.3.0
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-adminservice/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600

vi configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-adminservice-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config.od.com/eureka
app.properties: |
appId=100003172

应用资源配置清单

在任意一台k8s运算节点执行:

1
2
3
4
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/configmap.yaml
configmap/apollo-adminservice-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/deployment.yaml
deployment.extensions/apollo-adminservice created

浏览器访问

http://config.od.com
apollo注册中心

交付apollo-portal

准备软件包

在运维主机HDSS7-200.host.com上:
下载官方release包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 08:37 apollo-configservice-1.3.0-github.zip
-rw-r--r-- 1 root root 49418246 Feb 16 09:54 apollo-adminservice-1.3.0-github.zip
-rw-r--r-- 1 root root 36459359 Feb 16 10:00 apollo-portal-1.3.0-github.zip

[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-portal && unzip -o apollo-portal-1.3.0-github.zip -d /data/dockerfile/apollo-portal
Archive: apollo-portal-1.3.0-github.zip
inflating: /data/dockerfile/apollo-portal/scripts/shutdown.sh
inflating: /data/dockerfile/apollo-portal/apollo-portal.conf
inflating: /data/dockerfile/apollo-portal/apollo-portal-1.3.0-sources.jar
creating: /data/dockerfile/apollo-portal/config/
inflating: /data/dockerfile/apollo-portal/config/application-github.properties
inflating: /data/dockerfile/apollo-portal/scripts/startup.sh
inflating: /data/dockerfile/apollo-portal/config/apollo-env.properties
inflating: /data/dockerfile/apollo-portal/config/app.properties
inflating: /data/dockerfile/apollo-portal/apollo-portal-1.3.0.jar

执行数据库脚本

在数据库主机HDSS7-11.host.com上:
数据库脚本地址

1
2
3
[root@hdss7-11 ~]# mysql -uroot -p
mysql> create database ApolloPortalDB;
mysql> source ./apolloportal.sql

数据库用户授权

1
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"172.7.%" identified by "123456";

制作Docker镜像

在运维主机HDSS7-200.host.com上:

  • 配置数据库连接串

    /data/dockerfile/apollo-portal
    1
    [root@hdss7-200 apollo-portal]# cat config/application-github.properties
  • 配置Portal的meta service

    /data/dockerfile/apollo-portal/config/apollo-env.properties
    1
    dev.meta=http://config.od.com
  • 更新starup.sh

    /data/dockerfile/apollo-portal/scripts/startup.sh
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    #!/bin/bash
    SERVICE_NAME=apollo-portal
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-portal-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:$SERVER_PORT"
    SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

    ## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]]; then
    javaexe="/usr/bin/java"
    else
    echo "Unable to find Java"
    exit 1
    fi

    if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
    JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
    JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
    JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
    JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
    JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
    JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
    fi

    printf "$(date) ==== Starting ==== \n"

    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start

    rc=$?;

    if [[ $rc != 0 ]];
    then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
    fi

    tail -f /dev/null
  • 写Dockerfile

/data/dockerfile/apollo-portal/Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
FROM stanleyws/jre8:8u112

ENV VERSION 1.3.0

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo "Asia/Shanghai" > /etc/timezone

ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts

CMD ["/apollo-portal/scripts/startup.sh"]
  • 制作镜像并推送
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@hdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.3.0
Sending build context to Docker daemon 43.35 MB
Step 1 : FROM stanleyws/jre8:8u112
---> fa3a085d6ef1
Step 2 : ENV VERSION 1.3.0
---> Using cache
---> feb4c0289f04
Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
---> Using cache
---> a3e3fd61ae35
Step 4 : ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
---> cfcf63e8eedc
Removing intermediate container 860b55bd3fc5
Step 5 : ADD config/ /apollo-portal/config
---> 3ee780369431
Removing intermediate container 6b67ee4224b5
Step 6 : ADD scripts/ /apollo-portal/scripts
---> 42c9aea2e9e3
Removing intermediate container 2dcf8d1bf4cf
Step 7 : CMD /apollo-portal/scripts/startup.sh
---> [Warning] IPv4 forwarding is disabled. Networking will not work.
---> Running in 9162dab8b63a
---> 0c020b79c36f
Removing intermediate container 9162dab8b63a
Successfully built 0c020b79c36f
[root@hdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.3.0
docker push harbor.od.com/infra/apollo-portal:v1.3.0
The push refers to a repository [harbor.od.com/infra/apollo-portal]
e7c0e96ded4e: Pushed
0076c5344476: Pushed
3851a45d7440: Pushed
ebfb473df5c2: Mounted from infra/apollo-adminservice
aae5c057d1b6: Mounted from infra/apollo-adminservice
dee6aef5c2b6: Mounted from infra/apollo-adminservice
a464c54f93a9: Mounted from infra/apollo-adminservice
v1.3.0: digest: sha256:1aa30aac8642cceb97c053b7d74632240af08f64c49b65d8729021fef65628a4 size: 1785

解析域名

DNS主机HDSS7-11.host.com上:

/var/named/od.com.zone
1
portal	60 IN A 10.4.7.10

准备资源配置清单

在运维主机HDSS7-200.host.com

/data/k8s-yaml
1
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-portal && cd /data/k8s-yaml/apollo-portal

vi deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: apollo-portal
namespace: infra
labels:
name: apollo-portal
spec:
replicas: 1
selector:
matchLabels:
name: apollo-portal
template:
metadata:
labels:
app: apollo-portal
name: apollo-portal
spec:
volumes:
- name: configmap-volume
configMap:
name: apollo-portal-cm
containers:
- name: apollo-portal
image: harbor.od.com/infra/apollo-portal:v1.3.0
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: configmap-volume
mountPath: /apollo-portal/config
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600

vi svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: Service
apiVersion: v1
metadata:
name: apollo-portal
namespace: infra
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: apollo-portal
clusterIP: None
type: ClusterIP
sessionAffinity: None

vi ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: apollo-portal
namespace: infra
spec:
rules:
- host: portal.od.com
http:
paths:
- path: /
backend:
serviceName: apollo-portal
servicePort: 8080

vi configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: ConfigMap
metadata:
name: apollo-portal-cm
namespace: infra
data:
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = apolloportal
spring.datasource.password = 123456
app.properties: |
appId=100003173
apollo-env.properties: |
dev.meta=http://config.od.com

应用资源配置清单

在任意一台k8s运算节点执行:

1
2
3
4
5
6
7
8
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/configmap.yaml
configmap/apollo-portal-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/deployment.yaml
deployment.extensions/apollo-portal created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml
service/apollo-portal created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml
ingress.extensions/apollo-portal created

浏览器访问

http://portal.od.com

  • 用户名:apollo
  • 密码: admin

apollo-portal

实战dubbo微服务接入Apollo配置中心

改造dubbo-demo-service项目

使用IDE拉取项目(这里使用git bash作为范例)

1
$ git clone git@gitee.com/stanleywang/dubbo-demo-service.git

切到apollo分支

1
$ git checkout -b apollo

修改pom.xml

  • 加入apollo客户端jar包的依赖
dubbo-server/pom.xml
1
2
3
4
5
<dependency>
<groupId>com.ctrip.framework.apollo</groupId>
<artifactId>apollo-client</artifactId>
<version>1.1.0</version>
</dependency>
  • 修改resource段
dubbo-server/pom.xml
1
2
3
4
5
6
7
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*</include>
</includes>
<filtering>false</filtering>
</resource>

增加resources目录

/d/workspace/dubbo-demo-service/dubbo-server/src/main
1
2
3
$ mkdir -pv resources/META-INF
mkdir: created directory 'resources'
mkdir: created directory 'resources/META-INF'

修改config.properties文件

/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/config.properties
1
2
dubbo.registry=${dubbo.registry}
dubbo.port=${dubbo.port}

修改srping-config.xml文件

  • beans段新增属性
/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml
1
xmlns:apollo="http://www.ctrip.com/schema/apollo"
  • xsi:schemaLocation段内新增属性
/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml
1
http://www.ctrip.com/schema/apollo http://www.ctrip.com/schema/apollo.xsd
  • 新增配置项
/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml
1
<apollo:config/>
  • 删除配置项(注释)
/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml
1
<!-- <context:property-placeholder location="classpath:config.properties"/> -->

增加app.properties文件

/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/META-INF/app.properties
1
app.id=dubbo-demo-service

提交git中心仓库(gitee)

1
$ git push origin apollo

配置apollo-portal

创建项目

  • 部门

    样例部门1(TEST1)

  • 应用id

    dubbo-demo-service

  • 应用名称

    dubbo服务提供者

  • 应用负责人

    apollo|apollo

  • 项目管理员

    apollo|apollo

提交

进入配置页面

新增配置项1

  • Key

    dubbo.registry

  • Value

    zookeeper://zk1.od.com:2181

  • 选择集群

    DEV

提交

新增配置项2

  • Key

    dubbo.port

  • Value

    20880

  • 选择集群

    DEV

提交

发布配置

点击发布,配置生效
apollo-release

使用jenkins进行CI

略(注意记录镜像的tag)

上线新构建的项目

准备资源配置清单

运维主机HDSS7-200.host.com上:

/data/k8s-yaml/dubbo-demo-service/deployment.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-service
namespace: app
labels:
name: dubbo-demo-service
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-service
template:
metadata:
labels:
app: dubbo-demo-service
name: dubbo-demo-service
spec:
containers:
- name: dubbo-demo-service
image: harbor.od.com/app/dubbo-demo-service:apollo_190119_1815
ports:
- containerPort: 20880
protocol: TCP
env:
- name: C_OPTS
value: -Denv=dev -Dapollo.meta=http://config.od.com
- name: JAR_BALL
value: dubbo-server.jar
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600

注意:增加了env段配置
注意:docker镜像新版的tag

应用资源配置清单

在任意一台k8s运算节点上执行:

1
2
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/deployment.yaml
deployment.extensions/dubbo-demo-service configured

观察项目运行情况

http://dubbo-monitor.od.com

改造dubbo-demo-web

配置apollo-portal

创建项目

  • 部门

    样例部门1(TEST1)

  • 应用id

    dubbo-demo-web

  • 应用名称

    dubbo服务消费者

  • 应用负责人

    apollo|apollo

  • 项目管理员

    apollo|apollo

提交

进入配置页面

新增配置项1

  • Key

    dubbo.registry

  • Value

    zookeeper://zk1.od.com:2181

  • 选择集群

    DEV

提交

发布配置

点击发布,配置生效

使用jenkins进行CI

略(注意记录镜像的tag)

上线新构建的项目

准备资源配置清单

运维主机HDSS7-200.host.com上:

/data/k8s-yaml/dubbo-demo-consumer/deployment.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: dubbo-demo-consumer
namespace: app
labels:
name: dubbo-demo-consumer
spec:
replicas: 1
selector:
matchLabels:
name: dubbo-demo-consumer
template:
metadata:
labels:
app: dubbo-demo-consumer
name: dubbo-demo-consumer
spec:
containers:
- name: dubbo-demo-consumer
image: harbor.od.com/app/dubbo-demo-consumer:apllo_190120_1815
ports:
- containerPort: 20880
protocol: TCP
- containerPort: 8080
protocol: TCP
env:
- name: C_OPTS
value: -Denv=dev -Dapollo.meta=http://config.od.com
- name: JAR_BALL
value: dubbo-client.jar
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: harbor
restartPolicy: Always
terminationGracePeriodSeconds: 30
securityContext:
runAsUser: 0
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600

注意:增加了env段配置
注意:docker镜像新版的tag

应用资源配置清单

在任意一台k8s运算节点上执行:

1
2
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-web/deployment.yaml
deployment.extensions/dubbo-demo-consumer configured

通过Apollo配置中心动态维护项目的配置

以dubbo-demo-service项目为例,不用修改代码

实战维护多套dubbo微服务环境

生产实践

  1. 迭代新需求/修复BUG(编码->提GIT)
  2. 测试环境发版,测试(应用通过编译打包发布至TEST命名空间)
  3. 测试通过,上线(应用镜像直接发布至PROD命名空间)

系统架构

  • 物理架构
主机名 角色 ip
HDSS7-11.host.com zk-test(测试环境Test) 10.4.7.11
HDSS7-12.host.com zk-prod(生产环境Prod) 10.4.7.12
HDSS7-21.host.com kubernetes运算节点 10.4.7.21
HDSS7-22.host.com kubernetes运算节点 10.4.7.22
HDSS7-200.host.com 运维主机,harbor仓库 10.4.7.200
  • K8S内系统架构
环境 命名空间 应用
测试环境(TEST) test apollo-config,apollo-admin
测试环境(TEST) test dubbo-demo-service,dubbo-demo-web
生产环境(PROD) prod apollo-config,apollo-admin
生产环境(PROD) prod dubbo-demo-service,dubbo-demo-web
ops环境(infra) infra jenkins,dubbo-monitor,apollo-portal

修改/添加域名解析

DNS主机HDSS7-11.host.com上:

/var/named/od.com.zone
1
2
3
4
5
6
zk-test 60 IN A 10.4.7.11
zk-prod 60 IN A 10.4.7.12
config-test 60 IN A 10.4.7.10
config-prod 60 IN A 10.4.7.10
demo-test 60 IN A 10.4.7.10
demo-prod 60 IN A 10.4.7.10

Apollo的k8s应用配置

  • 删除app命名空间内应用,创建test命名空间,创建prod命名空间
  • 删除infra命名空间内apollo-configservice,apollo-adminservice应用
  • 数据库内删除ApolloConfigDB,创建ApolloConfigTestDB,创建ApolloConfigProdDB
1
2
3
4
5
6
7
8
9
10
11
12
13
mysql> drop database ApolloConfigDB;

mysql> create database ApolloConfigTestDB;
mysql> use ApolloConfigTestDB;
mysql> source ./apolloconfig.sql
mysql> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

mysql> create database ApolloConfigProdDB;
mysql> use ApolloConfigProdDB;
mysql> source ./apolloconfig.sql
mysql> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
  • 准备apollo-config,apollo-admin的资源配置清单(各2套)

注:apollo-config/apollo-admin的configmap配置要点

  • Test环境
1
2
3
4
5
6
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config-test.od.com/eureka
  • Prod环境
1
2
3
4
5
6
application-github.properties: |
# DataSource
spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
spring.datasource.username = apolloconfig
spring.datasource.password = 123456
eureka.service.url = http://config-prod.od.com/eureka
  • 依次应用,分别发布在test和prod命名空间
  • 修改apollo-portal的configmap并重启portal
1
2
3
apollo-env.properties: |
TEST.meta=http://config-test.od.com
PROD.meta=http://config-prod.od.com

Apollo的portal配置

管理员工具

删除应用、集群、AppNamespace,将已配置应用删除

系统参数

  • Key

    apollo.portal.envs

  • Value

    TEST,PROD

查询

  • Value

    TEST,PROD

保存

新建dubbo-demo-service和dubbo-demo-web项目

在TEST/PROD环境分别增加配置项并发布

发布dubbo微服务

  • 准备dubbo-demo-service和dubbo-demo-web的资源配置清单(各2套)
  • 依次应用,分别发布至app-test和app-prod命名空间
  • 使用dubbo-monitor查验

互联网公司技术部的日常

  • 产品经理整理需求,需求评审,出产品原型
  • 开发同学夜以继日的开发,提测
  • 测试同学使用Jenkins持续集成,并发布至测试环境
  • 验证功能,通过->待上线or打回->修改代码
  • 提交发版申请,运维同学将测试后的包发往生产环境
  • 无尽的BUG修复(笑cry)
坚持原创技术分享,您的支持将鼓励我继续创作!