发表于 | 更新于 | 分类于
本文字数： 4.3k | 阅读时长 ≈ 4 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

OSI七层模型

OSI模型（Open System Interconnection Model）是一个由ISO提出得到概念模型，试图提供一个使各种不同的的计算机和网络在世界范围内实现互联的标准框架。

分层结构

OSI参考模型采用分层结构，如图所示。 不得不说，这张图真的超经典呀。一张图搞定你你不懂的一切。
OSI七层模型
主要分为以下七层（从下至上）：物理层、数据链路层、网络层、传输层、会话层、表示层、应用层。

各层功能

  • 物理层
    简单的说，物理层（Physical Layer）确保原始的数据可在各种物理媒体上传输。在这一层上面规定了激活、维持、关闭通信端点之间的机械特性、电气特性、功能特性以及过程特性，为上层协议提供了一个传输数据的物理媒体。这一层传输的是bit流。
  • 数据链路层
    数据链路层（Data Link Layer）在不可靠的物理介质上提供可靠的传输。该层的作用包括：物理地址寻址、数据的成帧、流量控制、数据的检错、重发等。这一层中将bit流封装成frame帧。
  • 网络层
    网络层（Network Layer）负责对子网间的数据包进行路由选择。此外，网络层还可以实现拥塞控制、网际互连等功能。在这一层，数据的单位称为数据包（packet）。
  • 传输层
    传输层是第一个端到端，即主机到主机的层次。传输层负责将上层数据分段并提供端到端的、可靠的或不可靠的传输。此外，传输层还要处理端到端的差错控制和流量控制问题。在这一层，数据的单位称为数据段（segment）。
  • 会话层
    这一层管理主机之间的会话进程，即负责建立、管理、终止进程之间的会话。会话层还利用在数据中插入校验点来实现数据的同步，访问验证和会话管理在内的建立和维护应用之间通信的机制。如服务器验证用户登录便是由会话层完成的。使通信会话在通信失效时从校验点继续恢复通信。
  • 表示层
    这一层主要解决用户信息的语法表示问题。它将欲交换的数据从适合于某一用户的抽象语法，转换为适合于OSI系统内部使用的传送语法。即提供格式化的表示和转换数据服务。数据的压缩和解压缩， 加密和解密等工作都由表示层负责。
  • 应用层
    这一层为操作系统或网络应用程序提供访问网络服务的接口。

各层传输协议、传输单元、主要功能性设备比较

名称 传输协议 主要功能设备/接口 主要功能设备/接口
物理层 IEEE 802.1A、IEEE 802.2 bit-flow 比特流 光纤、双绞线、中继器和集线器 & RJ-45(网线接口)
数据链路层 ARP、MAC、 FDDI、Ethernet、Arpanet、PPP、PDN frame 帧 网桥、二层交换机
网络层 IP、ICMP、ARP、RARP 数据包（packet） 路由器、三层交换机
传输层 TCP、UDP Segment/Datagram 四层交换机
会话层 SMTP、DNS 报文 QoS
表示层 Telnet、SNMP 报文 -
应用层 FTP、TFTP、Telnet、HTTP、DNS 报文 -

关于协议你应该知道这些

以上通过图表、文字向大家阐述了七层模型每一层的具体功能及其相关协议，但知道了这些还不够，你还要知道以下这些。

TCP/UDP

  • TCP/UDP是什么？
    TCP — Transmission Control Protocol，传输控制协议。
    UDP — User Data Protocol，用户数据报协议。
  • TCP/UDP的区别（优缺点）？
    (1)、TCP是面向连接的，UDP是面向无连接的。TCP在通信之前必须通过三次握手机制与对方建立连接，而UDP通信不必与对方建立连接，不管对方的状态就直接把数据发送给对方
    (2)、TCP连接过程耗时，UDP不耗时
    (3)、TCP连接过程中出现的延时增加了被攻击的可能，安全性不高，而UDP不需要连接，安全性较高
    (4)、TCP是可靠的，保证数据传输的正确性，不易丢包;UDP是不可靠的，易丢包
    (5)、tcp传输速率较慢，实时性差，udp传输速率较快。tcp建立连接需要耗时，并且tcp首部信息太多，每次传输的有用信息较少，实时性差。
    (6)、tcp是流模式，udp是数据包模式。tcp只要不超过缓冲区的大小就可以连续发送数据到缓冲区上，接收端只要缓冲区上有数据就可以读取，可以一次读取多个数据包，而udp一次只能读取一个数据包，数据包之间独立
  • TCP三次握手过程

STEP 1： 主机A通过向主机B发送一个含有同步序列号的标志位的数据段给主机B，向主机B请求建立连接,通过这个数据段，主机A告诉主机B两件事:我想要和你通信；你可以用哪个序列号作为起始数据段来回应我。
STEP 2： 主机B收到主机A的请求后,用一个带有确认应答(ACK)和同步序列号(SYN)标志位的数据段响应主机A，也告诉主机A两件事：我已经收到你的请求了，你可以传输数据了；你要用哪佧序列号作为起始数据段来回应我。
STEP 3： 主机A收到这个数据段后，再发送一个确认应答，确认已收到主机B的数据段：”我已收到回复，我现在要开始传输实际数据了。这样3次握手就完成了，主机A和主机B就可以传输数据了。

  • 注意
    此时需要注意的是，TCP建立连接要进行3次握手，而断开连接要进行4次。
  • 名词解释

ACK： TCP报头的控制位之一，对数据进行确认，确认由目的端发出，用它来告诉发送端这个序列号之前的数据段都收到了。比如，确认号为X，则表示前X-1个数据段都收到了，只有当ACK=1时，确认号才有效，当ACK=0时，确认号无效，这时会要求重传数据，保证数据的完整性。
SYN： 同步序列号，TCP建立连接时将这个位置1。
FIN ： 发送端完成发送任务位，当TCP完成数据传输需要断开时，提出断开连接的一方将这位置1。

  • TCP可靠性的四大手段
    (1)、顺序编号： tcp在传输文件的时候，会将文件拆分为多个tcp数据包，每个装满的数据包大小大约在1k左右，tcp协议为保证可靠传输，会将这些数据包顺序编号
    (2)、确认机制： 当数据包成功的被发送方发送给接收方，接收方会根据tcp协议反馈给发送方一个成功接收的ACK信号，信号中包含了当前包的序号
    (3)、超时重传： 当发送方发送数据包给接收方时，会为每一个数据包设置一个定时器，当在设定的时间内，发送方仍没有收到接收方的ACK信号，会再次发送该数据包，直到收到接收方的ACK信号或者连接已断开
    (4)、校验信息： tcp首部校验信息较多，udp首部校验信息较少。

    上文部分协议简单讲

  • IEEE 802.1A、IEEE 802.2
    IEEE是英文Institute of Electrical and Electronics Engineers的简称，其中文译名是电气和电子工程师协会。IEEE 802规范定义了网卡如何访问传输介质（如光缆、双绞线、无线等），以及如何在传输介质上传输数据的方法，还定义了传输信息的网络设备之间连接建立、维护和拆除的途径。遵循IEEE 802标准的产品包括网卡、桥接器、路由器以及其他一些用来建立局域网络的组件。
    IEEE802.1A —— 局域网体系结构
    IEEE802.2 ——- 逻辑链路控制(LLC)
  • FDDI
    光纤分布式数据接口（Fiber Distributed Data Interface）
  • PPP
    点对点协议（Point to Point Protocol），为在点对点连接上传输多协议数据包提供了一个标准方法。
  • IP
    互联网协议（Internet Protocol），为计算机网络相互连接进行通信而设计的协议。任何厂家生产的计算机系统，只要遵守IP协议就可以与因特网互连互通。IP地址具有唯一性，根据用户性质的不同，可以分为5类。
  • ICMP
    控制报文协议（Internet Control Message Protocol）。TCP/IP设计了ICMP协议，当某个网关发现传输错误时，立即向信源主机发送ICMP报文，报告出错信息，让信源主机采取相应处理措施，它是一种差错和控制报文协议，不仅用于传输差错报文，还传输控制报文。
  • ARP/RARP
    ARP (Address Resolution Protocol) 地址解析协议
    RARP (Reverse Address Resolution Protocol) 反向地址解析协议
  • SMTP
    简单邮件传输协议（Simple Mail Transfer Protocol），它是一组用于由源地址到目的地址传送邮件的规则，由它来控制信件的中转方式。SMTP协议属于TCP/IP协议簇，它帮助每台计算机在发送或中转信件时找到下一个目的地。通过SMTP协议所指定的服务器,就可以把E-mail寄到收信人的服务器上了。
  • SNMP
    简单网络管理协议（Simple Network Management Protocol ），该协议能够支持网络管理系统，用以监测连接到网络上的设备是否有任何引起管理上关注的情况。
  • DNS
    域名系统（Domain Name System），因特网上作为域名和IP地址相互映射的一个分布式数据库，能够使用户更方便的访问互联网，而不用去记住能够被机器直接读取的IP数串。通过主机名，最终得到该主机名对应的IP地址的过程叫做域名解析(或主机名解析)。DNS协议运行在UDP协议之上，使用端口号53。
  • FTP
    文本传输协议（File Transfer Protocol），用于Internet上的控制文件的双向传输。同时，它也是一个应用程序Application）。基于不同的操作系统有不同的FTP应用程序，而所有这些应用程序都遵守该协议以传输文件。在FTP的使用当中，用户经常“下载”（Download）和“上载”（Upload）。“下载”文件就是从远程主机拷贝文件至自己的计算机上；“上载”文件就是将文件从自己的计算机中拷贝至远程主机上。
  • HTTP
    超文本传输协议（HyperText Transfer Protocol），是互联网上应用最为广泛的一种网络协议。所有的WWW文件都必须遵守这个标准。它可以使浏览器更加高效，使网络传输减少。它不仅保证计算机正确快速地传输超文本文档，还确定传输文档中的哪一部分，以及哪部分内容首先显示(如文本先于图形)等。HTTP是一个应用层协议，由请求和响应构成，是一个标准的客户端服务器模型，是一个无状态的协议。

发表于 | 更新于 | 分类于
本文字数： 1.7k | 阅读时长 ≈ 2 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

运维安全的四个层次

网络安全

网络设备的安全

  • 思科、华为等网络设备定期升级，修复bug和曝出的漏洞
  • 公网防火墙，核心交换机等核心网络设备的管理

外网安全策略

  • IDC，防火墙策略，严把上行端口开放
  • 公网上下行流量监控
  • 对DDos攻击提高警惕，提前准备应急预案
    • 临时提高流量，硬抗
    • 启动流量清洗，将攻击流量引入黑洞，有可能误杀正常用户

专线安全策略

  • 对涉及金融、支付等项目设立专线

VPN安全策略

  • IPsec VPN：site to site
  • OpenVPN： peer to site
  • 摒弃PPTP等不含加密算法的vpn服务
  • 端口全禁止，需要通信的申请审批后，再由管理员开放

数据安全

数据库用户权限

  • 管理员权限限定，不允许远程root
  • 定期更换管理员密码
  • 应用权限最小化，专人管理
  • 手动查询权限可审计

数据库审计设备

  • 数据库主库不能开一般查询日志（为了性能）
  • 交换机上镜像流量，接入审计设备，实现实时审计
  • 不要设计串行在系统里，形成单点和瓶颈

数据库脱敏

  • 姓名、身份证、手机号、银行卡号等敏感信息应脱敏处理
  • 对程序脱敏协同系统架构部共同出规范
  • 对手动查询权限脱敏，按列授权，录屏

备份策略

  • 每周全备，每天增备
  • 备份文件要每天利用内网流量低谷时间，推送到远程主机，有条件的应跨机房备份
  • 一定要规划定期恢复测试，保证备份的可用性

应用安全

操作系统安全

  • 系统基础优化（内核优化，优化工具）
  • 日期，时区同步
  • root密码复杂度足够高，需要在操作系统里做定时过期策略
  • 每三个月使用脚本更新服务器的root密码和iDrac密码，并将更换后的密码加密打包发送给指定管理员邮箱，同时提交gitlab
  • 对系统关键文件进行md5监控，例如/etc/passwd，~/.ssh/authorized_keys文件等，如有变更，触发报警
  • 定期查毒，漏扫，定期安排更新操作系统
  • /etc/ssh/sshd_config里配置：
    • PasswordAuthentication no
    • PermitRootLogin without-password
  • 使用saltstack等批量管理软件进行特权命令执行和备份脚本执行（避开ssh协议）

应用系统安全

WEB应用防火墙（WAF）

  • 防SQL注入
  • 防CC攻击
  • 防XSS跨站脚本

应用系统漏洞

  • 关注0day漏洞新闻
  • 及时整改并上线投产
  • 组织技术力量测试，复现

日志收集和分析

  • 完善日志收集方案，集中转储
  • 通过应用系统日志分析，进行安全预警

DNS劫持

  • 全站https，购买泛域名证书
  • 有条件的可以自己维护公网DNS，上dnssec数字签名
  • 采购基调、听云等第三方拨测服务，分布式监控网站质量
  • 向ISP投诉，工信部举报

Basic Auth

  • 在nginx上做，非常简单
  • 对防脚本攻击有奇效

企业邮箱服务器安全

推荐使用微软的Exchange

功能强大，维护相对简单

投产反垃圾邮件网关

投产梭子鱼反垃圾邮件网关，防伪造发信人

群发审核管控

用好邮件组

接入AD域控

域名安全管理

做好ICP备案

  • 域名证书
  • 域名实名认证（公司模板）
  • 接入商处蓝色幕布拍照
  • 法人身份证、管理员身份证
  • 网站真实性核验单

公网解析

  • 专人管理，邮件申请，审批
  • 将业务解析至不同公网IP出口，双活机房
  • 智能解析，解析至不同线路
  • 如有条件，可购买公网解析套餐服务，安全服务等

内网安全

80%以上的企业IT安全问题出自内网安全

堡垒机

  • 一定要强制使用堡垒机登录服务器
  • ssh私钥通行短语机制，避免密钥失窃
  • 定期审计堡垒机操作日志
  • 如果有必要，可以上2FA（双因子验证）

AD域控

有条件一定要接入windows域控，要求密码复杂度和定期过期

  • 邮箱
  • wifi
  • vpn账号密码
  • 内网系统账号
  • 业务系统账号
  • 网络设备等

办公网安全

  • 专业的HelpDesk团队
  • 企业级杀毒软件
  • 办公电脑接入域控
  • 上网行为管理
  • 流量监控，mac地址绑定
  • 有条件的可以在办公环境上一个小型的业务机房
  • wifi管控，单做guest接入点，不能访问业务核心网络

发表于 | 更新于 | 分类于
本文字数： 1k | 阅读时长 ≈ 1 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

IT管理中的PPT

  • 人，流程，技术

服务是什么？

  • 服务是向客户提供价值的一种手段，使客户不用承担特定的成本和风险就可以获得所期望的结果。

服务管理

  • 服务管理是一套特定的组织能力，以服务的形式为客户提供价值。

ITIL简介

是什么？

  • Information Technology Infrastructure Library

    IT基础架构库，一个可以直接使用的标准，已于2005年12月15日被ISO接受为国际标准 – ISO20000

与ISO20000的区别

ITIL ISO20000
提供最佳实践指导 提供衡量ITSM的指标
没有固定的能力衡量指标 全球统一
对人员 对机构

目标

  • 将IT管理工作标准化、模式化，减少人为误操作带来的隐患
  • 通过服务目录，服务报告，告诉业务部门，我们可以做什么，做了什么
  • 通过系列流程，知识库减轻对英雄式工程师的依赖。把经验积累下来
  • 通过对流程的管控，减少成本，降低风险，提供客户满意度

IT Service CMM

初始级

个人英雄式工程师

可重复级

潜规则

定义级

  • 已将IT服务过程文档话，标准化，并综合成标准服务过程
  • 根据客户需求调整服务产品和服务战略
  • 适当的工具和信息报告

管理级

  • 受监督、可测量的IT服务体系
  • 根据业务战略调整服务体系

优化级

  • 持续改进的IT服务体系
  • IT与业务指标建立关系
  • IT与业务协作改进流程

ITIL v3

服务战略

从组织能力和战略资产两个角度出发，为组织进行服务战略方面的决策和战略设计提供了一套结构化的方法

  • 我们的业务是什么？
  • 我们的客户是谁？
  • 客户重视什么？
  • 谁依赖我们的服务？
  • 他们怎样使用我们的服务？
  • 服务为什么对他们有价值？

4P

观念

面向其目标客户的业务定位或服务提供方式

定位

描述了采纳和中立场的决策

计划

描述了将蓝图转化为现实的手段

模式

描述了一系列的稳定的决策和行动

服务设计

对服务及服务管理流程设计和开发的指导

服务目录管理

服务级别管理

容量管理

商业容量管理：吞吐量
服务容量管理：响应时间
资源容量管理：CPU

可用性管理

正常运行时间、宕机时间

IT服务持续性管理

灾备

信息安全管理

服务转换

服务运营

持续服务改进

RACI模型

谁负责，谁批准，咨询谁，通知谁

角色

  • 服务所有者
  • 流程所有者

流程

  • 简单问题复杂化，多元化
  • 效率、成本、质量、风险、稳定性、可持续性、用户体验

项目

临时性

运营

持续性

发表于 | 更新于 | 分类于
本文字数： 89k | 阅读时长 ≈ 1:21

欢迎加入王导的VIP学习qq群：==>932194668<==

实验环境

基础架构

主机名 角色 ip
HDSS7-11.host.com k8s代理节点1 10.4.7.11
HDSS7-12.host.com k8s代理节点2 10.4.7.12
HDSS7-21.host.com k8s运算节点1 10.4.7.21
HDSS7-22.host.com k8s运算节点2 10.4.7.22
HDSS7-200.host.com k8s运维节点(docker仓库) 10.4.7.200

硬件环境

  • 5台vm，每台至少2c2g

软件环境

前置准备工作

DNS服务安装部署

  • 创建主机域host.com
  • 创建业务域od.com
  • 主辅同步(10.4.7.11主、10.4.7.12辅)
  • 客户端配置指向自建DNS

略

准备签发证书环境

运维主机HDSS7-200.host.com上：

安装CFSSL

1
2
3
4
[root@hdss7-200 ~]# curl -s -L -o /usr/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 
[root@hdss7-200 ~]# curl -s -L -o /usr/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 
[root@hdss7-200 ~]# curl -s -L -o /usr/bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 
[root@hdss7-200 ~]# chmod +x /usr/bin/cfssl*

创建生成CA证书的JSON配置文件

/opt/certs/ca-config.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
{
    "signing": {
        "default": {
            "expiry": "175200h"
        },
        "profiles": {
            "server": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

证书类型
client certificate： 客户端使用，用于服务端认证客户端,例如etcdctl、etcd proxy、fleetctl、docker客户端
server certificate: 服务端使用，客户端以此验证服务端身份,例如docker服务端、kube-apiserver
peer certificate: 双向证书，用于etcd集群成员间通信

创建生成CA证书签名请求（csr）的JSON配置文件

/opt/certs/ca-csr.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
    "CN": "kubernetes-ca",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ],
    "ca": {
        "expiry": "175200h"
    }
}

CN: Common Name，浏览器使用该字段验证网站是否合法，一般写的是域名。非常重要。浏览器使用该字段验证网站是否合法
C: Country， 国家
ST: State，州，省
L: Locality，地区，城市
O: Organization Name，组织名称，公司名称
OU: Organization Unit Name，组织单位名称，公司部门

生成CA证书和私钥

/opt/certs
1
2
3
4
5
6
7
[root@hdss7-200 certs]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - 
2019/01/18 09:31:19 [INFO] generating a new CA key and certificate from CSR
2019/01/18 09:31:19 [INFO] generate received request
2019/01/18 09:31:19 [INFO] received CSR
2019/01/18 09:31:19 [INFO] generating key: rsa-2048
2019/01/18 09:31:19 [INFO] encoded CSR
2019/01/18 09:31:19 [INFO] signed certificate with serial number 345276964513449660162382535043012874724976422200

生成ca.pem、ca.csr、ca-key.pem(CA私钥,需妥善保管)

/opt/certs
1
2
3
4
5
6
[root@hdss7-200 certs]# ls -l
-rw-r--r-- 1 root root  836 Jan 16 11:04 ca-config.json
-rw-r--r-- 1 root root  332 Jan 16 11:10 ca-csr.json
-rw------- 1 root root 1675 Jan 16 11:17 ca-key.pem
-rw-r--r-- 1 root root 1001 Jan 16 11:17 ca.csr
-rw-r--r-- 1 root root 1354 Jan 16 11:17 ca.pem

部署docker环境

HDSS7-200.host.com,HDSS7-21.host.com,HDSS7-22.host.com上：

安装

1
2
3
4
# ls -l|grep docker-engine
-rw-r--r--  1 root root  20013304 Jan 16 18:16 docker-engine-1.12.6-1.el7.centos.x86_64.rpm
-rw-r--r--  1 root root     29112 Jan 16 18:15 docker-engine-selinux-1.12.6-1.el7.centos.noarch.rpm
# yum localinstall *.rpm

配置

/etc/docker/daemon.json 
1
2
3
4
5
6
7
8
9
# vi /etc/docker/daemon.json 
{
  "graph": "/data/docker",
  "storage-driver": "overlay",
  "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
  "bip": "172.7.21.1/24",
  "exec-opts": ["native.cgroupdriver=systemd"],
  "live-restore": true
}

注意：这里bip要根据宿主机ip变化

启动脚本

/usr/lib/systemd/system/docker.service
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

启动

1
2
# systemctl enable docker.service
# systemctl start docker.service

部署docker镜像私有仓库harbor

HDSS7-200.host.com上：

下载软件二进制包并解压

harbor下载地址

/opt/harbor
1
2
3
4
5
6
[root@hdss7-200 harbor]# tar xf harbor-offline-installer-v1.7.1.tgz -C /opt

[root@hdss7-200 harbor]# ll
total 583848
drwxr-xr-x 3 root root       242 Jan 23 15:28 harbor
-rw-r--r-- 1 root root 597857483 Jan 17 14:58 harbor-offline-installer-v1.7.1.tgz

配置

/opt/harbor/harbor.cfg
1
hostname = harbor.od.com
/opt/harbor/docker-compose.yml
1
2
3
4
ports:
  - 180:80
  - 1443:443
  - 4443:4443

安装docker-compose

1
2
3
[root@hdss7-200 harbor]# yum install docker-compose -y
[root@hdss7-200 harbor]# rpm -qa docker-compose
docker-compose-1.18.0-2.el7.noarch

安装harbor

/opt/harbor
1
[root@hdss7-200 harbor]# ./install.sh

检查harbor启动情况

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@hdss7-200 harbor]# docker-compose ps
       Name                     Command               State                                 Ports                               
--------------------------------------------------------------------------------------------------------------------------------
harbor-adminserver   /harbor/start.sh                 Up                                                                        
harbor-core          /harbor/start.sh                 Up                                                                        
harbor-db            /entrypoint.sh postgres          Up      5432/tcp                                                          
harbor-jobservice    /harbor/start.sh                 Up                                                                        
harbor-log           /bin/sh -c /usr/local/bin/ ...   Up      127.0.0.1:1514->10514/tcp                                         
harbor-portal        nginx -g daemon off;             Up      80/tcp                                                            
nginx                nginx -g daemon off;             Up      0.0.0.0:1443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:180->80/tcp
redis                docker-entrypoint.sh redis ...   Up      6379/tcp                                                          
registry             /entrypoint.sh /etc/regist ...   Up      5000/tcp                                                          
registryctl          /harbor/start.sh                 Up

配置harbor的dns内网解析

/var/named/od.com.zone
1
harbor	60 IN A 10.4.7.200

检查

1
2
[root@hdss7-200 harbor]# dig -t A harbor.od.com @10.4.7.11 +short
10.4.7.200

安装nginx并配置

安装

1
2
3
[root@hdss7-200 harbor]# yum install nginx -y
[root@hdss7-200 harbor]# rpm -qa nginx
nginx-1.12.2-2.el7.x86_64

配置

/etc/nginx/conf.d/harbor.od.com.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
server {
    listen       80;
    server_name  harbor.od.com;

    client_max_body_size 1000m;

    location / {
        proxy_pass http://127.0.0.1:180;
    }
}
server {
    listen       443 ssl;
    server_name  harbor.od.com;

    ssl_certificate "certs/harbor.od.com.pem";
    ssl_certificate_key "certs/harbor.od.com-key.pem";
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  10m;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
    client_max_body_size 1000m;

    location / {
        proxy_pass http://127.0.0.1:180;
    }
}

注意：这里需要自签ssl证书，自签过程略

(umask 077; openssl genrsa -out od.key 2048)
openssl req -new -key od.key -out od.csr -subj “/CN=*.od.com/ST=Beijing/L=beijing/O=od/OU=ops”
openssl x509 -req -in od.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out od.crt -days 365

启动

1
2
3
4
5
[root@hdss7-200 harbor]# nginx

[root@hdss7-200 harbor]# netstat -luntp|grep nginx
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      6590/nginx: master  
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      6590/nginx: master

浏览器打开http://harbor.od.com

  • 用户名：admin
  • 密码： Harbor12345

部署Master节点服务

部署etcd集群

集群规划

主机名 角色 ip
HDSS7-12.host.com etcd lead 10.4.7.12
HDSS7-21.host.com etcd follow 10.4.7.21
HDSS7-22.host.com etcd follow 10.4.7.22

注意：这里部署文档以HDSS7-12.host.com主机为例，另外两台主机安装部署方法类似

创建生成证书签名请求（csr）的JSON配置文件

运维主机HDSS7-200.host.com上：

/opt/certs/etcd-peer-csr.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
    "CN": "etcd-peer",
    "hosts": [
        "10.4.7.11",
        "10.4.7.12",
        "10.4.7.21",
        "10.4.7.22"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

生成etcd证书和私钥

/opt/certs
1
2
3
4
5
6
7
8
9
10
11
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json | cfssljson -bare etcd-peer
2019/01/18 09:35:09 [INFO] generate received request
2019/01/18 09:35:09 [INFO] received CSR
2019/01/18 09:35:09 [INFO] generating key: rsa-2048

2019/01/18 09:35:09 [INFO] encoded CSR
2019/01/18 09:35:10 [INFO] signed certificate with serial number 324191491384928915605254764031096067872154649010
2019/01/18 09:35:10 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

检查生成的证书、私钥

/opt/certs
1
2
3
4
5
[root@hdss7-200 certs]# ls -l|grep etcd
-rw-r--r-- 1 root root  387 Jan 18 12:32 etcd-peer-csr.json
-rw------- 1 root root 1679 Jan 18 12:32 etcd-peer-key.pem
-rw-r--r-- 1 root root 1074 Jan 18 12:32 etcd-peer.csr
-rw-r--r-- 1 root root 1432 Jan 18 12:32 etcd-peer.pem

创建etcd用户

HDSS7-12.host.com上：

1
[root@hdss7-12 ~]# useradd -s /sbin/nologin -M etcd

下载软件，解压，做软连接

etcd下载地址
HDSS7-12.host.com上：

/opt/src
1
2
3
4
5
6
7
8
9
10
[root@hdss7-12 src]# ls -l
total 9604
-rw-r--r-- 1 root root 9831476 Jan 18 10:45 etcd-v3.1.18-linux-amd64.tar.gz
[root@hdss7-12 src]# tar xf etcd-v3.1.18-linux-amd64.tar.gz -C /opt
[root@hdss7-12 src]# ln -s /opt/etcd-v3.1.18-linux-amd64 /opt/etcd
[root@hdss7-12 src]# ls -l /opt
total 0
lrwxrwxrwx 1 root   root   24 Jan 18 14:21 etcd -> etcd-v3.1.18-linux-amd64
drwxr-xr-x 4 478493 89939 166 Jun 16  2018 etcd-v3.1.18-linux-amd64
drwxr-xr-x 2 root   root   45 Jan 18 14:21 src

创建目录，拷贝证书、私钥

HDSS7-12.host.com上：

1
2
3
[root@hdss7-12 src]# mkdir -p /data/etcd /data/logs/etcd-server 
[root@hdss7-12 src]# chown -R etcd.etcd /data/etcd /data/logs/etcd-server/
[root@hdss7-12 src]# mkdir -p /opt/etcd/certs

将运维主机上生成的ca.pem、etcd-peer-key.pem、etcd-peer.pem拷贝到/opt/etcd/certs目录中，注意私钥文件权限600

/opt/etcd/certs
1
2
3
4
5
6
7
[root@hdss7-12 certs]# chmod 600 etcd-peer-key.pem
[root@hdss7-12 certs]# chown -R etcd.etcd /opt/etcd/certs/
[root@hdss7-12 certs]# ls -l
total 12
-rw-r--r-- 1 etcd etcd 1354 Jan 18 14:45 ca.pem
-rw------- 1 etcd etcd 1679 Jan 18 17:00 etcd-peer-key.pem
-rw-r--r-- 1 etcd etcd 1444 Jan 18 17:02 etcd-peer.pem

创建etcd服务启动脚本

HDSS7-12.host.com上：

/opt/etcd/etcd-server-startup.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/sh
./etcd --name etcd-server-7-12 \
       --data-dir /data/etcd/etcd-server \
       --listen-peer-urls https://10.4.7.12:2380 \
       --listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
       --quota-backend-bytes 8000000000 \
       --initial-advertise-peer-urls https://10.4.7.12:2380 \
       --advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
       --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
       --ca-file ./certs/ca.pem \
       --cert-file ./certs/etcd-peer.pem \
       --key-file ./certs/etcd-peer-key.pem \
       --client-cert-auth  \
       --trusted-ca-file ./certs/ca.pem \
       --peer-ca-file ./certs/ca.pem \
       --peer-cert-file ./certs/etcd-peer.pem \
       --peer-key-file ./certs/etcd-peer-key.pem \
       --peer-client-cert-auth \
       --peer-trusted-ca-file ./certs/ca.pem \
       --log-output stdout

注意：etcd集群各主机的启动脚本略有不同，部署其他节点时注意修改。

调整权限和目录

HDSS7-12.host.com上：

1
2
[root@hdss7-12 certs]# chmod +x /opt/etcd/etcd-server-startup.sh
[root@hdss7-12 certs]# mkdir -p /data/logs/etcd-server

安装supervisor软件

HDSS7-12.host.com上：

1
2
3
[root@hdss7-12 certs]# yum install supervisor -y
[root@hdss7-12 certs]# systemctl start supervisord
[root@hdss7-12 certs]# systemctl enable supervisord

创建etcd-server的启动配置

HDSS7-12.host.com上：

/etc/supervisord.d/etcd-server.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[program:etcd-server-7-12]
command=/opt/etcd/etcd-server-startup.sh                        ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/etcd                                             ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=22                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd                                                       ; setuid to this UNIX account to run the program
redirect_stderr=false                                           ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log           ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/etcd-server/etcd.stderr.log           ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                        ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                     ; emit events on stderr writes (default false)

注意：etcd集群各主机启动配置略有不同，配置其他节点时注意修改。

启动etcd服务并检查

HDSS7-12.host.com上：

1
2
3
4
[root@hdss7-12 certs]# supervisorctl start all
etcd-server-7-12: started
[root@hdss7-12 certs]# supervisorctl status   
etcd-server-7-12                 RUNNING   pid 6692, uptime 0:00:05

安装部署启动检查所有集群规划主机上的etcd服务

略

检查集群状态

3台均启动后，检查集群状态

1
2
3
4
5
6
7
8
9
10
[root@hdss7-12 ~]# /opt/etcd/etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy

[root@hdss7-12 ~]# /opt/etcd/etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true

部署kube-apiserver集群

集群规划

主机名 角色 ip
HDSS7-21.host.com kube-apiserver 10.4.7.21
HDSS7-22.host.com kube-apiserver 10.4.7.22
HDSS7-11.host.com 4层负载均衡 10.4.7.11
HDSS7-12.host.com 4层负载均衡 10.4.7.12
注意：这里10.4.7.11和10.4.7.12使用nginx做4层负载均衡器，用keepalived跑一个vip：10.4.7.10，代理两个kube-apiserver，实现高可用

这里部署文档以HDSS7-21.host.com主机为例，另外一台运算节点安装部署方法类似

下载软件，解压，做软连接

HDSS7-21.host.com上：
kubernetes下载地址

/opt/src
1
2
3
4
5
6
7
8
9
[root@hdss7-21 src]# ls -l|grep kubernetes
-rw-r--r-- 1 root root 417761204 Jan 17 16:46 kubernetes-server-linux-amd64.tar.gz
[root@hdss7-21 src]# tar xf kubernetes-server-linux-amd64.tar.gz -C /opt
[root@hdss7-21 src]# mv /opt/kubernetes /opt/kubernetes-v1.13.2-linux-amd64
[root@hdss7-21 src]# ln -s /opt/kubernetes-v1.13.2-linux-amd64 /opt/kubernetes
[root@hdss7-21 src]# mkdir /opt/kubernetes/server/bin/{cert,conf}
[root@hdss7-21 src]# ls -l /opt|grep kubernetes
lrwxrwxrwx 1 root   root         31 Jan 18 10:49 kubernetes -> kubernetes-v1.13.2-linux-amd64/
drwxr-xr-x 4 root   root         50 Jan 17 17:40 kubernetes-v1.13.2-linux-amd64

签发client证书

运维主机HDSS7-200.host.com上：

创建生成证书签名请求（csr）的JSON配置文件

/opt/certs/client-csr.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
    "CN": "k8s-node",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

生成client证书和私钥

1
2
3
4
5
6
7
8
9
10
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json | cfssljson -bare client
2019/01/18 14:02:50 [INFO] generate received request
2019/01/18 14:02:50 [INFO] received CSR
2019/01/18 14:02:50 [INFO] generating key: rsa-2048
2019/01/18 14:02:51 [INFO] encoded CSR
2019/01/18 14:02:51 [INFO] signed certificate with serial number 423108651040279300242366884100637974155370861448
2019/01/18 14:02:51 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

检查生成的证书、私钥

1
2
3
4
[root@hdss7-200 certs]# ls -l|grep client
-rw------- 1 root root 1679 Jan 21 11:13 client-key.pem
-rw-r--r-- 1 root root  989 Jan 21 11:13 client.csr
-rw-r--r-- 1 root root 1367 Jan 21 11:13 client.pem

签发kube-apiserver证书

运维主机HDSS7-200.host.com上：

创建生成证书签名请求（csr）的JSON配置文件

/opt/certs/apiserver-csr.json 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
    "CN": "apiserver",
    "hosts": [
        "127.0.0.1",
        "192.168.0.1",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "10.4.7.10",
        "10.4.7.21",
        "10.4.7.22",
        "10.4.7.23"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

生成kube-apiserver证书和私钥

1
2
3
4
5
6
7
8
9
10
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json | cfssljson -bare apiserver 
2019/01/18 14:05:44 [INFO] generate received request
2019/01/18 14:05:44 [INFO] received CSR
2019/01/18 14:05:44 [INFO] generating key: rsa-2048
2019/01/18 14:05:46 [INFO] encoded CSR
2019/01/18 14:05:46 [INFO] signed certificate with serial number 633406650960616624590510576685608580490218676227
2019/01/18 14:05:46 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

检查生成的证书、私钥

1
2
3
4
5
6
[root@hdss7-200 certs]# ls -l|grep apiserver
total 72
-rw-r--r-- 1 root root  406 Jan 21 14:10 apiserver-csr.json
-rw------- 1 root root 1675 Jan 21 14:11 apiserver-key.pem
-rw-r--r-- 1 root root 1082 Jan 21 14:11 apiserver.csr
-rw-r--r-- 1 root root 1599 Jan 21 14:11 apiserver.pem

拷贝证书至各运算节点，并创建配置

HDSS7-21.host.com上：

拷贝证书、私钥，注意私钥文件属性600

/opt/kubernetes/server/bin/cert
1
2
3
4
5
6
7
8
[root@hdss7-21 cert]# ls -l /opt/kubernetes/server/bin/cert
total 40
-rw------- 1 root root 1676 Jan 21 16:39 apiserver-key.pem
-rw-r--r-- 1 root root 1599 Jan 21 16:36 apiserver.pem
-rw------- 1 root root 1675 Jan 21 13:55 ca-key.pem
-rw-r--r-- 1 root root 1354 Jan 21 13:50 ca.pem
-rw------- 1 root root 1679 Jan 21 13:53 client-key.pem
-rw-r--r-- 1 root root 1368 Jan 21 13:53 client.pem

创建配置

/opt/kubernetes/server/bin/conf/audit.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

创建启动脚本

HDSS7-21.host.com上：

/opt/kubernetes/server/bin/kube-apiserver.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/bin/bash
./kube-apiserver \
  --apiserver-count 2 \
  --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \
  --audit-policy-file ./conf/audit.yaml \
  --authorization-mode RBAC \
  --client-ca-file ./cert/ca.pem \
  --requestheader-client-ca-file ./cert/ca.pem \
  --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
  --etcd-cafile ./cert/ca.pem \
  --etcd-certfile ./cert/client.pem \
  --etcd-keyfile ./cert/client-key.pem \
  --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
  --service-account-key-file ./cert/ca-key.pem \
  --service-cluster-ip-range 192.168.0.0/16 \
  --service-node-port-range 3000-29999 \
  --target-ram-mb=1024 \
  --kubelet-client-certificate ./cert/client.pem \
  --kubelet-client-key ./cert/client-key.pem \
  --log-dir  /data/logs/kubernetes/kube-apiserver \
  --tls-cert-file ./cert/apiserver.pem \
  --tls-private-key-file ./cert/apiserver-key.pem \
  --v 2

调整权限和目录

HDSS7-21.host.com上：

/opt/kubernetes/server/bin
1
2
[root@hdss7-21 bin]# chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh
[root@hdss7-21 bin]# mkdir -p /data/logs/kubernetes/kube-apiserver

创建supervisor配置

HDSS7-21.host.com上：

/etc/supervisord.d/kube-apiserver.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[program:kube-apiserver]
command=/opt/kubernetes/server/bin/kube-apiserver.sh            ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                            ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=22                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                       ; setuid to this UNIX account to run the program
redirect_stderr=false                                           ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log        ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stderr.log        ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                        ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                     ; emit events on stderr writes (default false)

启动服务并检查

HDSS7-21.host.com上：

1
2
3
4
5
[root@hdss7-21 bin]# supervisorctl update
kube-apiserverr: added process group
[root@hdss7-21 bin]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 6661, uptime 1 day, 8:41:13
kube-apiserver                   RUNNING   pid 43765, uptime 2:09:41

安装部署启动检查所有集群规划主机上的kube-apiserver

略

配4层反向代理

HDSS7-11.host.com,HDSS7-12.host.com上：

nginx配置

/etc/nginx/nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
stream {
    upstream kube-apiserver {
        server 10.4.7.21:6443     max_fails=3 fail_timeout=30s;
        server 10.4.7.22:6443     max_fails=3 fail_timeout=30s;
    }
    server {
        listen 7443;
        proxy_connect_timeout 2s;
        proxy_timeout 900s;
        proxy_pass kube-apiserver;
    }
}

keepalived配置

check_port.sh
/etc/keepalived/check_port.sh 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/bin/bash
#keepalived 监控端口脚本
#使用方法：
#在keepalived的配置文件中
#vrrp_script check_port {#创建一个vrrp_script脚本,检查配置
#    script "/etc/keepalived/check_port.sh 6379" #配置监听的端口
#    interval 2 #检查脚本的频率,单位（秒）
#}
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
        PORT_PROCESS=`ss -lt|grep $CHK_PORT|wc -l`
        if [ $PORT_PROCESS -eq 0 ];then
                echo "Port $CHK_PORT Is Not Used,End."
                exit 1
        fi
else
        echo "Check Port Cant Be Empty!"
fi
keepalived主

HDSS7-11.host.com上

1
2
[root@hdss7-11 ~]# rpm -qa keepalived
keepalived-1.3.5-6.el7.x86_64
/etc/keepalived/keepalived.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
! Configuration File for keepalived

global_defs {
   router_id 10.4.7.11

}

vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 7443"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 10.4.7.11
    nopreempt

    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        10.4.7.10
    }
}
keepalived备

HDSS7-12.host.com上

1
2
[root@hdss7-12 ~]# rpm -qa keepalived
keepalived-1.3.5-6.el7.x86_64
/etc/keepalived/keepalived.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
! Configuration File for keepalived
global_defs {
	router_id 10.4.7.12
}
vrrp_script chk_nginx {
	script "/etc/keepalived/check_port.sh 7443"
	interval 2
	weight -20
}
vrrp_instance VI_1 {
	state BACKUP
	interface eth0
	virtual_router_id 251
	mcast_src_ip 10.4.7.12
	priority 90
	advert_int 1
	authentication {
		auth_type PASS
		auth_pass 11111111
	}
	track_script {
		chk_nginx
	}
	virtual_ipaddress {
		10.4.7.10
	}
}

启动代理并检查

HDSS7-11.host.com,HDSS7-12.host.com上：

  • 启动

    1
    2
    3
    4
    5
    6
    7
    [root@hdss7-11 ~]# systemctl start keepalived
    [root@hdss7-11 ~]# systemctl enable keepalived
    [root@hdss7-11 ~]# nginx -s reload

    [root@hdss7-12 ~]# systemctl start keepalived
    [root@hdss7-12 ~]# systemctl enable keepalived
    [root@hdss7-12 ~]# nginx -s reload

  • 检查

    1
    2
    3
    4
    5
    6
    7
    8
    [root@hdss7-11 ~]## netstat -luntp|grep 7443
    tcp        0      0 0.0.0.0:7443            0.0.0.0:*               LISTEN      17970/nginx: master
    [root@hdss7-12 ~]## netstat -luntp|grep 7443
    tcp        0      0 0.0.0.0:7443            0.0.0.0:*               LISTEN      17970/nginx: master
    [root@hdss7-11 ~]# ip add|grep 10.4.9.10
        inet 10.9.7.10/32 scope global vir0
    [root@hdss7-11 ~]# ip add|grep 10.4.9.10
        （空）

部署controller-manager

集群规划

主机名 角色 ip
HDSS7-21.host.com controller-manager 10.4.7.21
HDSS7-22.host.com controller-manager 10.4.7.22

注意：这里部署文档以HDSS7-21.host.com主机为例，另外一台运算节点安装部署方法类似

创建启动脚本

HDSS7-21.host.com上：

/opt/kubernetes/server/bin/kube-controller-manager.sh
1
2
3
4
5
6
7
8
9
10
#!/bin/sh
./kube-controller-manager \
  --cluster-cidr 172.7.0.0/16 \
  --leader-elect true \
  --log-dir /data/logs/kubernetes/kube-controller-manager \
  --master http://127.0.0.1:8080 \
  --service-account-private-key-file ./cert/ca-key.pem \
  --service-cluster-ip-range 192.168.0.0/16 \
  --root-ca-file ./cert/ca.pem \
  --v 2

调整文件权限，创建目录

HDSS7-21.host.com上：

/opt/kubernetes/server/bin
1
2
[root@hdss7-21 bin]# chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh
[root@hdss7-21 bin]# mkdir -p /data/logs/kubernetes/kube-controller-manager

创建supervisor配置

HDSS7-21.host.com上：

/etc/supervisord.d/kube-conntroller-manager.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[program:kube-controller-manager]
command=/opt/kubernetes/server/bin/kube-controller-manager.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                              ; directory to cwd to before exec (def no cwd)
autostart=true                                                                    ; start at supervisord start (default: true)
autorestart=true                                                                  ; retstart at unexpected quit (default: true)
startsecs=22                                                                      ; number of secs prog must stay running (def. 1)
startretries=3                                                                    ; max # of serial start failures (default 3)
exitcodes=0,2                                                                     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                                   ; signal used to kill process (default TERM)
stopwaitsecs=10                                                                   ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                         ; setuid to this UNIX account to run the program
redirect_stderr=false                                                             ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controll.stdout.log  ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                                       ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-controller-manager/controll.stderr.log  ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                                      ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                                          ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                                       ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                                       ; emit events on stderr writes (default false)

启动服务并检查

HDSS7-21.host.com上：

1
2
3
4
5
6
[root@hdss7-21 bin]# supervisorctl update
kube-controller-manager: added process group
[root@hdss7-21 bin]# supervisorctl status   
etcd-server-7-21                 RUNNING   pid 6661, uptime 1 day, 8:41:13
kube-apiserver                   RUNNING   pid 43765, uptime 2:09:41
kube-controller-manager          RUNNING   pid 44230, uptime 2:05:01

安装部署启动检查所有集群规划主机上的kube-controller-manager服务

略

部署kube-scheduler

集群规划

主机名 角色 ip
HDSS7-21.host.com kube-scheduler 10.4.7.21
HDSS7-22.host.com kube-scheduler 10.4.7.22

注意：这里部署文档以HDSS7-21.host.com主机为例，另外一台运算节点安装部署方法类似

创建启动脚本

HDSS7-21.host.com上：

/opt/kubernetes/server/bin/kube-scheduler.sh
1
2
3
4
5
6
#!/bin/sh
./kube-scheduler \
  --leader-elect  \
  --log-dir /data/logs/kubernetes/kube-scheduler \
  --master http://127.0.0.1:8080 \
  --v 2

调整文件权限，创建目录

HDSS7-21.host.com上：

/opt/kubernetes/server/bin
1
2
[root@hdss7-21 bin]# chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh
[root@hdss7-21 bin]# mkdir -p /data/logs/kubernetes/kube-scheduler

创建supervisor配置

HDSS7-21.host.com上：

/etc/supervisord.d/kube-scheduler.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[program:kube-scheduler]
command=/opt/kubernetes/server/bin/kube-scheduler.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                               ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                     ; directory to cwd to before exec (def no cwd)
autostart=true                                                           ; start at supervisord start (default: true)
autorestart=true                                                         ; retstart at unexpected quit (default: true)
startsecs=22                                                             ; number of secs prog must stay running (def. 1)
startretries=3                                                           ; max # of serial start failures (default 3)
exitcodes=0,2                                                            ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                          ; signal used to kill process (default TERM)
stopwaitsecs=10                                                          ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                ; setuid to this UNIX account to run the program
redirect_stderr=false                                                    ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                              ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stderr.log ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                                 ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                              ; emit events on stderr writes (default false)

启动服务并检查

HDSS7-21.host.com上：

1
2
3
4
5
6
7
[root@hdss7-21 bin]# supervisorctl update
kube-scheduler: added process group
[root@hdss7-21 bin]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 6661, uptime 1 day, 8:41:13
kube-apiserver                   RUNNING   pid 43765, uptime 2:09:41
kube-controller-manager          RUNNING   pid 44230, uptime 2:05:01
kube-scheduler                   RUNNING   pid 44779, uptime 2:02:27

安装部署启动检查所有集群规划主机上的kube-scheduler服务

略

部署Node节点服务

部署kubelet

集群规划

主机名 角色 ip
HDSS7-21.host.com kubelet 10.4.7.21
HDSS7-22.host.com kubelet 10.4.7.22

注意：这里部署文档以HDSS7-21.host.com主机为例，另外一台运算节点安装部署方法类似

签发kubelet证书

运维主机HDSS7-200.host.com上：

创建生成证书签名请求（csr）的JSON配置文件

kubelet-csr.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
    "CN": "kubelet-node",
    "hosts": [
    "127.0.0.1",
    "10.4.7.10",
    "10.4.7.21",
    "10.4.7.22",
    "10.4.7.23",
    "10.4.7.24",
    "10.4.7.25",
    "10.4.7.26",
    "10.4.7.27",
    "10.4.7.28"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

生成kubelet证书和私钥

/opt/certs
1
2
3
4
5
6
7
8
9
10
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssljson -bare kubelet
2019/01/18 17:51:16 [INFO] generate received request
2019/01/18 17:51:16 [INFO] received CSR
2019/01/18 17:51:16 [INFO] generating key: rsa-2048
2019/01/18 17:51:17 [INFO] encoded CSR
2019/01/18 17:51:17 [INFO] signed certificate with serial number 48870268157415133698067712395152321546974943470
2019/01/18 17:51:17 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

检查生成的证书、私钥

/opt/certs
1
2
3
4
5
6
[root@hdss7-200 certs]# ls -l|grep kubelet
total 88
-rw-r--r-- 1 root root  415 Jan 22 16:58 kubelet-csr.json
-rw------- 1 root root 1679 Jan 22 17:00 kubelet-key.pem
-rw-r--r-- 1 root root 1086 Jan 22 17:00 kubelet.csr
-rw-r--r-- 1 root root 1456 Jan 22 17:00 kubelet.pem

拷贝证书至各运算节点，并创建配置

HDSS7-21.host.com上：

拷贝证书、私钥，注意私钥文件属性600

/opt/kubernetes/server/bin/cert
1
2
3
4
5
6
7
8
9
10
[root@hdss7-21 cert]# ls -l /opt/kubernetes/server/bin/cert
total 40
-rw------- 1 root root 1676 Jan 21 16:39 apiserver-key.pem
-rw-r--r-- 1 root root 1599 Jan 21 16:36 apiserver.pem
-rw------- 1 root root 1675 Jan 21 13:55 ca-key.pem
-rw-r--r-- 1 root root 1354 Jan 21 13:50 ca.pem
-rw------- 1 root root 1679 Jan 21 13:53 client-key.pem
-rw-r--r-- 1 root root 1368 Jan 21 13:53 client.pem
-rw------- 1 root root 1679 Jan 22 17:00 kubelet-key.pem
-rw-r--r-- 1 root root 1456 Jan 22 17:00 kubelet.pem

创建配置

HDSS7-21.host.com上：

给kubectl创建软连接
/opt/kubernetes/server/bin
1
2
3
[root@hdss7-21 bin]# ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl
[root@hdss7-21 bin]# which kubectl
/usr/bin/kubectl
set-cluster

注意：在conf目录下

/opt/kubernetes/server/conf
1
2
3
4
5
6
7
[root@hdss7-21 conf]# kubectl config set-cluster myk8s \
  --certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem \
  --embed-certs=true \
  --server=https://10.4.7.10:7443 \
  --kubeconfig=kubelet.kubeconfig

Cluster "myk8s" set.
set-credentials

注意：在conf目录下

/opt/kubernetes/server/conf
1
2
3
[root@hdss7-21 conf]# kubectl config set-credentials k8s-node --client-certificate=/opt/kubernetes/server/bin/cert/client.pem --client-key=/opt/kubernetes/server/bin/cert/client-key.pem --embed-certs=true --kubeconfig=kubelet.kubeconfig 

User "k8s-node" set.
set-context

注意：在conf目录下

/opt/kubernetes/server/conf
1
2
3
4
5
6
[root@hdss7-21 conf]# kubectl config set-context myk8s-context \
  --cluster=myk8s \
  --user=k8s-node \
  --kubeconfig=kubelet.kubeconfig

Context "myk8s-context" created.
use-context

注意：在conf目录下

/opt/kubernetes/server/conf
1
2
3
[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=kubelet.kubeconfig

Switched to context "myk8s-context".
k8s-node.yaml
  • 创建资源配置文件
/opt/kubernetes/server/bin/conf/k8s-node.yaml
1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: k8s-node
  • 应用资源配置文件
/opt/kubernetes/server/conf
1
2
3
[root@hdss7-21 conf]# kubectl create -f k8s-node.yaml

clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
  • 检查
/opt/kubernetes/server/conf
1
2
3
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node
NAME           AGE
k8s-node       3m

准备infra_pod基础镜像

运维主机HDSS7-200.host.com上：

下载

1
2
3
4
5
6
7
8
[root@hdss7-200 ~]# docker pull xplenty/rhel7-pod-infrastructure:v3.4
Trying to pull repository docker.io/xplenty/rhel7-pod-infrastructure ... 
sha256:9314554780673b821cb7113d8c048a90d15077c6e7bfeebddb92a054a1f84843: Pulling from docker.io/xplenty/rhel7-pod-infrastructure
615bc035f9f8: Pull complete 
1c5fd9dfeaa8: Pull complete 
7653a8c7f937: Pull complete 
Digest: sha256:9314554780673b821cb7113d8c048a90d15077c6e7bfeebddb92a054a1f84843
Status: Downloaded newer image for docker.io/xplenty/rhel7-pod-infrastructure:v3.4

提交至私有仓库（harbor）中

  • 配置主机登录私有仓库
/root/.docker/config.json
1
2
3
4
5
6
7
{
	"auths": {
		"harbor.od.com": {
			"auth": "YWRtaW46SGFyYm9yMTIzNDU="
		}
	}
}

这里代表：用户名admin，密码Harbor12345
[root@hdss7-200 ~]# echo YWRtaW46SGFyYm9yMTIzNDU=|base64 -d
admin:Harbor12345

注意：也可以在各运算节点使用docker login harbor.od.com，输入用户名，密码

  • 给镜像打tag
1
2
3
[root@hdss7-200 ~]# docker images|grep v3.4
xplenty/rhel7-pod-infrastructure   v3.4                34d3450d733b        2 years ago         205 MB
[root@hdss7-200 ~]# docker tag 34d3450d733b harbor.od.com/k8s/pod:v3.4
  • push到harbor
1
2
3
4
5
6
[root@hdss7-200 ~]# docker push harbor.od.com/k8s/pod:v3.4
The push refers to a repository [harbor.od.com/k8s/pod]
ba3d4cbbb261: Pushed 
0a081b45cb84: Pushed 
df9d2808b9a9: Pushed 
v3.4: digest: sha256:73cc48728e707b74f99d17b4e802d836e22d373aee901fdcaa781b056cdabf5c size: 948

创建kubelet启动脚本

HDSS7-21.host.com上：

/opt/kubernetes/server/bin/kubelet-721.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/bin/sh
./kubelet \
  --anonymous-auth=false \
  --cgroup-driver systemd \
  --cluster-dns 192.168.0.2 \
  --cluster-domain cluster.local \
  --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice \
  --fail-swap-on="false" \
  --client-ca-file ./cert/ca.pem \
  --tls-cert-file ./cert/kubelet.pem \
  --tls-private-key-file ./cert/kubelet-key.pem \
  --hostname-override 10.4.7.21 \
  --image-gc-high-threshold 20 \
  --image-gc-low-threshold 10 \
  --kubeconfig ./conf/kubelet.kubeconfig \
  --log-dir /data/logs/kubernetes/kube-kubelet \
  --pod-infra-container-image harbor.od.com/k8s/pod:v3.4 \
  --root-dir /data/kubelet

注意：kubelet集群各主机的启动脚本略有不同，部署其他节点时注意修改。

检查配置，权限，创建日志目录

HDSS7-21.host.com上：

/opt/kubernetes/server/conf
1
2
3
4
5
[root@hdss7-21 conf]# ls -l|grep kubelet.kubeconfig 
-rw------- 1 root root 6471 Jan 22 17:33 kubelet.kubeconfig

[root@hdss7-21 conf]# chmod +x /opt/kubernetes/server/bin/kubelet-721.sh
[root@hdss7-21 conf]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet

创建supervisor配置

HDSS7-21.host.com上：

/etc/supervisord.d/kube-kubelet.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[program:kube-kubelet]
command=/opt/kubernetes/server/bin/kubelet-721.sh                 ; the program (relative uses PATH, can take args)
numprocs=1                                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                              ; directory to cwd to before exec (def no cwd)
autostart=true                                                    ; start at supervisord start (default: true)
autorestart=true              									  ; retstart at unexpected quit (default: true)
startsecs=22                  									  ; number of secs prog must stay running (def. 1)
startretries=3                									  ; max # of serial start failures (default 3)
exitcodes=0,2                 									  ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT               									  ; signal used to kill process (default TERM)
stopwaitsecs=10               									  ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                         ; setuid to this UNIX account to run the program
redirect_stderr=false                                             ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log   ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                       ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stderr.log   ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                      ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                          ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB   									  ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false   									  ; emit events on stderr writes (default false)

启动服务并检查

HDSS7-21.host.com上：

1
2
3
4
5
6
7
8
[root@hdss7-21 bin]# supervisorctl update
kube-kubelet: added process group
[root@hdss7-21 bin]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 9507, uptime 22:44:48
kube-apiserver                   RUNNING   pid 9770, uptime 21:10:49
kube-controller-manager          RUNNING   pid 10048, uptime 18:22:10
kube-kubelet                     STARTING  
kube-scheduler                   RUNNING   pid 10041, uptime 18:22:13

检查运算节点

HDSS7-21.host.com上：

1
2
3
[root@hdss7-21 bin]# kubectl get node
NAME        STATUS   ROLES    AGE   VERSION
10.4.7.21   Ready    <none>   3m   v1.13.2

非常重要！

安装部署启动检查所有集群规划主机上的kubelet服务

略

部署kube-proxy

集群规划

主机名 角色 ip
HDSS7-21.host.com kube-proxy 10.4.7.21
HDSS7-22.host.com kube-proxy 10.4.7.22

注意：这里部署文档以HDSS7-21.host.com主机为例，另外一台运算节点安装部署方法类似

签发kube-proxy证书

运维主机HDSS7-200.host.com上：

创建生成证书签名请求（csr）的JSON配置文件

/opt/certs/kube-proxy-csr.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
    "CN": "system:kube-proxy",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}

生成kube-proxy证书和私钥

/opt/certs
1
2
3
4
5
6
7
8
9
10
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json | cfssljson -bare kube-proxy-client
2019/01/18 18:14:23 [INFO] generate received request
2019/01/18 18:14:23 [INFO] received CSR
2019/01/18 18:14:23 [INFO] generating key: rsa-2048
2019/01/18 18:14:23 [INFO] encoded CSR
2019/01/18 18:14:23 [INFO] signed certificate with serial number 375797145588654714099258750873820528127028390681
2019/01/18 18:14:23 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

检查生成的证书、私钥

/opt/certs
1
2
3
4
5
[root@hdss7-200 certs]# ls -l|grep kube-proxy
-rw------- 1 root root 1679 Jan 22 17:31 kube-proxy-client-key.pem
-rw-r--r-- 1 root root 1005 Jan 22 17:31 kube-proxy-client.csr
-rw-r--r-- 1 root root 1383 Jan 22 17:31 kube-proxy-client.pem
-rw-r--r-- 1 root root  268 Jan 22 17:23 kube-proxy-csr.json

拷贝证书至各运算节点，并创建配置

HDSS7-21.host.com上：

拷贝证书、私钥，注意私钥文件属性600

/opt/kubernetes/server/bin/cert
1
2
3
4
5
6
7
8
9
10
11
12
[root@hdss7-21 cert]# ls -l /opt/kubernetes/server/bin/cert
total 40
-rw------- 1 root root 1676 Jan 21 16:39 apiserver-key.pem
-rw-r--r-- 1 root root 1599 Jan 21 16:36 apiserver.pem
-rw------- 1 root root 1675 Jan 21 13:55 ca-key.pem
-rw-r--r-- 1 root root 1354 Jan 21 13:50 ca.pem
-rw------- 1 root root 1679 Jan 21 13:53 client-key.pem
-rw-r--r-- 1 root root 1368 Jan 21 13:53 client.pem
-rw------- 1 root root 1679 Jan 22 17:00 kubelet-key.pem
-rw-r--r-- 1 root root 1456 Jan 22 17:00 kubelet.pem
-rw------- 1 root root 1679 Jan 22 17:31 kube-proxy-client-key.pem
-rw-r--r-- 1 root root 1383 Jan 22 17:31 kube-proxy-client.pem

创建配置

set-cluster

注意：在conf目录下

/opt/kubernetes/server/bin/conf
1
2
3
4
5
6
7
[root@hdss7-21 conf]# kubectl config set-cluster myk8s \
  --certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem \
  --embed-certs=true \
  --server=https://10.4.7.10:7443 \
  --kubeconfig=kube-proxy.kubeconfig

Cluster "myk8s" set.
set-credentials

注意：在conf目录下

/opt/kubernetes/server/bin/conf
1
2
3
4
5
6
7
[root@hdss7-21 conf]# kubectl config set-credentials kube-proxy \
  --client-certificate=/opt/kubernetes/server/bin/cert/kube-proxy-client.pem \
  --client-key=/opt/kubernetes/server/bin/cert/kube-proxy-client-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

User "kube-proxy" set.
set-context

注意：在conf目录下

/opt/kubernetes/server/bin/conf
1
2
3
4
5
6
[root@hdss7-21 conf]# kubectl config set-context myk8s-context \
  --cluster=myk8s \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

Context "myk8s-context" created.
use-context

注意：在conf目录下

/opt/kubernetes/server/bin/conf
1
2
3
[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=kube-proxy.kubeconfig

Switched to context "myk8s-context".

创建kube-proxy启动脚本

HDSS7-21.host.com上：

/opt/kubernetes/server/bin/kube-proxy-721.sh
1
2
3
4
5
#!/bin/sh
./kube-proxy \
  --cluster-cidr 172.7.0.0/16 \
  --hostname-override 10.4.7.21 \
  --kubeconfig ./conf/kube-proxy.kubeconfig

注意：kube-proxy集群各主机的启动脚本略有不同，部署其他节点时注意修改。

检查配置，权限，创建日志目录

HDSS7-21.host.com上：

/opt/kubernetes/server/conf
1
2
3
4
5
[root@hdss7-21 conf]# ls -l|grep kube-proxy.kubeconfig 
-rw------- 1 root root 6471 Jan 22 17:33 kube-proxy.kubeconfig

[root@hdss7-21 conf]# chmod +x /opt/kubernetes/server/bin/kube-proxy-721.sh
[root@hdss7-21 conf]# mkdir -p /data/logs/kubernetes/kube-proxy

创建supervisor配置

HDSS7-21.host.com上：

/etc/supervisord.d/kube-proxy.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[program:kube-proxy]
command=/opt/kubernetes/server/bin/kube-proxy-721.sh                 ; the program (relative uses PATH, can take args)
numprocs=1                                                           ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                 ; directory to cwd to before exec (def no cwd)
autostart=true                                                       ; start at supervisord start (default: true)
autorestart=true                                                     ; retstart at unexpected quit (default: true)
startsecs=22                                                         ; number of secs prog must stay running (def. 1)
startretries=3                                                       ; max # of serial start failures (default 3)
exitcodes=0,2                                                        ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                      ; signal used to kill process (default TERM)
stopwaitsecs=10                                                      ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                		         ; setuid to this UNIX account to run the program
redirect_stderr=false                                           		 ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log     ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    		 ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        		 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     		 ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     		 ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/kubernetes/kube-proxy/proxy.stderr.log     ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                    		 ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                        		 ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB   						                           ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false   						                           ; emit events on stderr writes (default false)

启动服务并检查

HDSS7-21.host.com上：

1
2
3
4
5
6
7
8
9
[root@hdss7-21 bin]# supervisorctl update
kube-proxy: added process group
[root@hdss7-21 bin]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 9507, uptime 22:44:48
kube-apiserver                   RUNNING   pid 9770, uptime 21:10:49
kube-controller-manager          RUNNING   pid 10048, uptime 18:22:10
kube-kubelet                     RUNNING   pid 14597, uptime 0:32:59
kube-proxy                       STARTING  
kube-scheduler                   RUNNING   pid 10041, uptime 18:22:13

安装部署启动检查所有集群规划主机上的kube-proxy服务

略

部署addons插件

验证kubernetes集群

在任意一个运算节点，创建一个资源配置清单

这里我们选择HDSS7-21.host.com主机

/root/nginx-ds.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: v1
kind: Service
metadata:
  name: nginx-ds
  labels:
    app: nginx-ds
spec:
  type: NodePort
  selector:
    app: nginx-ds
  ports:
  - name: http
    port: 80
    targetPort: 80

---

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ds
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

应用资源配置，并检查

/root
1
2
3
4
5
[root@hdss7-21 ~]# kubectl create -f nginx-ds.yaml
[root@hdss7-21 ~]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
nginx-ds-6hnc7   1/1     Running   0          99m
nginx-ds-m5q6j   1/1     Running   0          18h

验证

补

部署flannel

集群规划

主机名 角色 ip
HDSS7-21.host.com flannel 10.4.7.21
HDSS7-22.host.com flannel 10.4.7.22

注意：这里部署文档以HDSS7-21.host.com主机为例，另外一台运算节点安装部署方法类似

在各运算节点上增加iptables规则

注意：iptables规则各主机的略有不同，其他运算节点上执行时注意修改。

  • 优化SNAT规则，各运算节点之间的各POD之间的网络通信不再出网
1
2
# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE

10.4.7.21主机上的，来源是172.7.21.0/24段的docker的ip，目标ip不是172.7.0.0/16段，网络发包不从docker0桥设备出站的，才进行SNAT转换

各运算节点保存iptables规则

1
[root@hdss7-21 ~]# iptables-save > /etc/sysconfig/iptables

下载软件，解压，做软连接

HDSS7-21.host.com上：

/opt/src
1
2
3
4
5
6
7
8
[root@hdss7-21 src]# ls -l|grep flannel
-rw-r--r-- 1 root root 417761204 Jan 17 18:46 flannel-v0.10.0-linux-amd64.tar.gz
[root@hdss7-21 src]# mkdir -p /opt/flannel-v0.10.0-linux-amd64/cert
[root@hdss7-21 src]# tar xf flannel-v0.10.0-linux-amd64.tar.gz -C /opt/flannel-v0.10.0-linux-amd64
[root@hdss7-21 src]# ln -s /opt/flannel-v0.10.0-linux-amd64 /opt/flannel
[root@hdss7-21 src]# ls -l /opt|grep flannel
lrwxrwxrwx 1 root   root         31 Jan 17 18:49 flannel -> flannel-v0.10.0-linux-amd64/
drwxr-xr-x 4 root   root         50 Jan 17 18:47 flannel-v0.10.0-linux-amd64

最终目录结构

/opt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[root@hdss7-21 opt]# tree -L 2
.
|-- etcd -> etcd-v3.1.18-linux-amd64
|-- etcd-v3.1.18-linux-amd64
|   |-- Documentation
|   |-- README-etcdctl.md
|   |-- README.md
|   |-- READMEv2-etcdctl.md
|   |-- certs
|   |-- etcd
|   |-- etcd-server-startup.sh
|   `-- etcdctl
|-- flannel -> flannel-v0.10.0/
|-- flannel-v0.10.0
|   |-- README.md
|   |-- cert
|   |-- flanneld
|   `-- mk-docker-opts.sh
|-- kubernetes -> kubernetes-v1.13.2-linux-amd64/
|-- kubernetes-v1.13.2-linux-amd64
|   |-- LICENSES
|   |-- addons
|   `-- server
`-- src
    |-- etcd-v3.1.18-linux-amd64.tar.gz
    |-- flannel-v0.10.0-linux-amd64.tar.gz
    `-- kubernetes-server-linux-amd64.tar.gz

操作etcd，增加host-gw

HDSS7-21.host.com上：

/opt/etcd
1
2
[root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}

创建配置

HDSS7-21.host.com上：

/opt/flannel/subnet.env
1
2
3
4
FLANNEL_NETWORK=172.7.0.0/16
FLANNEL_SUBNET=172.7.21.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false

注意：flannel集群各主机的配置略有不同，部署其他节点时注意修改。

创建启动脚本

HDSS7-21.host.com上：

/opt/flannel/flanneld.sh
1
2
3
4
5
6
7
8
9
10
#!/bin/sh
./flanneld \
  --public-ip=10.4.7.21 \
  --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
  --etcd-keyfile=./cert/client-key.pem \
  --etcd-certfile=./cert/client.pem \
  --etcd-cafile=./cert/ca.pem \
  --iface=eth0 \
  --subnet-file=./subnet.env \
  --healthz-port=2401

注意：flannel集群各主机的启动脚本略有不同，部署其他节点时注意修改。

检查配置，权限，创建日志目录

HDSS7-21.host.com上：

/opt/flannel
1
2
3
[root@hdss7-21 flannel]# chmod +x /opt/flannel/flanneld.sh 

[root@hdss7-21 flannel]# mkdir -p /data/logs/flanneld

创建supervisor配置

HDSS7-21.host.com上：

/etc/supervisord.d/flanneld.ini
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[program:flanneld]
command=/opt/flannel/flanneld.sh                             ; the program (relative uses PATH, can take args)
numprocs=1                                                   ; number of processes copies to start (def 1)
directory=/opt/flannel                                       ; directory to cwd to before exec (def no cwd)
autostart=true                                               ; start at supervisord start (default: true)
autorestart=true                                             ; retstart at unexpected quit (default: true)
startsecs=22                   ; number of secs prog must stay running (def. 1)
startretries=3     				     ; max # of serial start failures (default 3)
exitcodes=0,2      				     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT    				     ; signal used to kill process (default TERM)
stopwaitsecs=10    				     ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                    ; setuid to this UNIX account to run the program
redirect_stderr=false                                        ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log       ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                 ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                     ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                  ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                  ; emit events on stdout writes (default false)
stderr_logfile=/data/logs/flanneld/flanneld.stderr.log       ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=64MB                                 ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=4                                     ; # of stderr logfile backups (default 10)
stderr_capture_maxbytes=1MB                                  ; number of bytes in 'capturemode' (default 0)
stderr_events_enabled=false                                  ; emit events on stderr writes (default false)

启动服务并检查

HDSS7-21.host.com上：

1
2
3
4
5
6
7
8
9
10
[root@hdss7-21 flanneld]# supervisorctl update
flanneld: added process group
[root@hdss7-21 flanneld]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 9507, uptime 1 day, 20:35:42
flanneld                         STARTING  
kube-apiserver                   RUNNING   pid 9770, uptime 1 day, 19:01:43
kube-controller-manager          RUNNING   pid 37646, uptime 0:58:48
kube-kubelet                     RUNNING   pid 32640, uptime 17:16:36
kube-proxy                       RUNNING   pid 15097, uptime 17:55:36
kube-scheduler                   RUNNING   pid 37803, uptime 0:55:47

安装部署启动检查所有集群规划主机上的flannel服务

略

再次验证集群

部署k8s资源配置清单的内网http服务

在运维主机HDSS7-200.host.com上，配置一个nginx虚拟主机，用以提供k8s统一的资源配置清单访问入口

/etc/nginx/conf.d/k8s-yaml.od.com.conf
1
2
3
4
5
6
7
8
9
10
server {
    listen       80;
    server_name  k8s-yaml.od.com;

    location / {
        autoindex on;
        default_type text/plain;
        root /data/k8s-yaml;
    }
}

配置内网DNS解析

HDSS7-11.host.com上

/var/named/od.com.zone
1
k8s-yaml	60 IN A 10.4.7.200

以后所有的资源配置清单统一放置在运维主机的/data/k8s-yaml目录下即可

1
[root@hdss7-200 ~]# nginx -s reload

部署kube-dns(coredns)

准备coredns-v1.3.1镜像

运维主机HDSS7-200.host.com上：

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@hdss7-200 ~]# docker pull coredns/coredns:1.3.1
1.3.1: Pulling from coredns/coredns
e0daa8927b68: Pull complete 
3928e47de029: Pull complete 
Digest: sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4
Status: Downloaded newer image for coredns/coredns:1.3.1
[root@hdss7-200 ~]# docker tag eb516548c180 harbor.od.com/k8s/coredns:v1.3.1
[root@hdss7-200 ~]# docker push harbor.od.com/k8s/coredns:v1.3.1
docker push harbor.od.com/k8s/coredns:v1.3.1
The push refers to a repository [harbor.od.com/k8s/coredns]
c6a5fc8a3f01: Pushed 
fb61a074724d: Pushed 
v1.3.1: digest: sha256:e077b9680c32be06fc9652d57f64aa54770dd6554eb87e7a00b97cf8e9431fda size: 739

任意一台运算节点上：

1
[root@hdss7-21 ~]# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 --docker-email=stanley.wang.m@qq.com -n kube-system

准备资源配置清单

运维主机HDSS7-200.host.com上：

1
[root@hdss7-200 ~]# mkdir -p /data/k8s-yaml/coredns && cd /data/k8s-yaml/coredns

vi /data/k8s-yaml/coredns/rbac.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
  labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
\--\-
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: Reconcile
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
\--\-
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: EnsureExists
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system

vi /data/k8s-yaml/coredns/configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        log
        health
        kubernetes cluster.local 192.168.0.0/16
        proxy . /etc/resolv.conf
        cache 30
       }

vi /data/k8s-yaml/coredns/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: coredns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: coredns
  template:
    metadata:
      labels:
        k8s-app: coredns
    spec:
      serviceAccountName: coredns
      containers:
      - name: coredns
        image: harbor.od.com/k8s/coredns:v1.3.1
        args:
        - -conf
        - /etc/coredns/Corefile
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
      dnsPolicy: Default
      imagePullSecrets:
      - name: harbor
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile

vi /data/k8s-yaml/coredns/svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: v1
kind: Service
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: coredns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: coredns
  clusterIP: 192.168.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53

依次执行创建

浏览器打开：http://k8s-yaml.od.com/coredns 检查资源配置清单文件是否正确创建
在任意运算节点上应用资源配置清单

1
2
3
4
5
6
7
8
9
10
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/coredns/rbac.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/coredns/configmap.yaml
configmap/coredns created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/coredns/deployment.yaml
deployment.extensions/coredns created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/coredns/svc.yaml
service/coredns created

检查

1
2
3
4
5
6
7
8
9
10
[root@hdss7-21 ~]# kubectl get pods -n kube-system -o wide
NAME                       READY   STATUS    RESTARTS   AGE
coredns-7ccccdf57c-5b9ch   1/1     Running   0          3m4s

[root@hdss7-21 coredns]# kubectl get svc -n kube-system
NAME      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
coredns   ClusterIP   192.168.0.2   <none>        53/UDP,53/TCP   29s

[root@hdss7-21 ~]# dig -t A nginx-ds.default.svc.cluster.local. @192.168.0.2 +short
192.168.0.3

部署traefik（ingress）

准备traefik镜像

运维主机HDSS7-200.host.com上：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@hdss7-200 ~]# docker pull traefik:v1.7-alpine
v1.7-alpine: Pulling from library/traefik
bdf0201b3a05: Pull complete 
9dfd896cc066: Pull complete 
de06b5685128: Pull complete 
c4d82a21fa27: Pull complete 
Digest: sha256:0531581bde9da0670fc2c7a4e419e1cc38abff74e7ba06410bf2b1b55c70ef15
Status: Downloaded newer image for traefik:v1.7-alpine
[root@hdss7-200 ~]# docker tag 1930b7508541 harbor.od.com/k8s/traefik:v1.7       
[root@hdss7-200 ~]# docker push harbor.od.com/k8s/traefik:v1.7
The push refers to a repository [harbor.od.com/k8s/traefik]
a3e3d574f6ae: Pushed 
a7c355c1a104: Pushed 
e89059911fc9: Pushed 
a464c54f93a9: Mounted from infra/apollo-portal 
v1.7: digest: sha256:8f92899f5feb08db600c89d3016145e838fa7ff0d316ee21ecd63d9623643410 size: 1157

准备资源配置清单

运维主机HDSS7-200.host.com上：

1
[root@hdss7-200 ~]# mkdir -p /data/k8s-yaml/traefik && cd /data/k8s-yaml/traefik

vi /data/k8s-yaml/traefik/rbac.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
\--\-
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
\--\-
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

vi /data/k8s-yaml/traefik/daemonset.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: harbor.od.com/k8s/traefik:v1.7
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
          hostPort: 81
        - name: admin
          containerPort: 8080
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        args:
        - -\-api
        - -\-kubernetes
        - -\-logLevel=INFO
        - -\-insecureskipverify=true
        - -\-kubernetes.endpoint=https://10.4.7.10:7443
        - -\-accesslog
        - -\-accesslog.filepath=/var/log/traefik_access.log
        - -\-traefiklog
        - -\-traefiklog.filepath=/var/log/traefik.log
        - -\-metrics.prometheus
      imagePullSecrets:
      - name: harbor

vi /data/k8s-yaml/traefik/svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin

vi /data/k8s-yaml/traefik/ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: traefik.od.com
    http:
      paths:
      - backend:
          serviceName: traefik-ingress-service
          servicePort: 8080

解析域名

HDSS7-11.host.com上

/var/named/od.com.zone
1
traefik	60 IN A 10.4.7.10

依次执行创建

浏览器打开：http://k8s-yaml.od.com/traefik 检查资源配置清单文件是否正确创建
在任意运算节点应用资源配置清单

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/rbac.yaml 
serviceaccount/traefik-ingress-controller created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/daemonset.yaml 
daemonset.extensions/traefik-ingress-controller created

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/svc.yaml 
service/traefik-ingress-service created

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/ingress.yaml 
ingress.extensions/traefik-web-ui created

配置反代

HDSS7-11.host.com和HDSS7-12.host.com两台主机上的nginx均需要配置，这里可以考虑使用saltstack或者ansible进行统一配置管理

/etc/nginx/conf.d/od.com.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
upstream default_backend_traefik {
    server 10.4.7.21:81    max_fails=3 fail_timeout=10s;
    server 10.4.7.22:81    max_fails=3 fail_timeout=10s;
}
server {
    server_name *.od.com;
  
    location / {
        proxy_pass http://default_backend_traefik;
        proxy_set_header Host       $http_host;
        proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
    }
}

浏览器访问

http://traefik.od.com

部署dashboard

准备dashboard镜像

运维主机HDSS7-200.host.com上：

1
2
3
4
5
6
7
8
9
10
11
[root@hdss7-200 ~]# docker pull k8scn/kubernetes-dashboard-amd64:v1.8.3
v1.8.3: Pulling from k8scn/kubernetes-dashboard-amd64
a4026007c47e: Pull complete 
Digest: sha256:ebc993303f8a42c301592639770bd1944d80c88be8036e2d4d0aa116148264ff
Status: Downloaded newer image for k8scn/kubernetes-dashboard-amd64:v1.8.3
[root@hdss7-200 ~]# docker tag 0c60bcf89900 harbor.od.com/k8s/dashboard:v1.8.3
[root@hdss7-200 ~]# docker push harbor.od.com/k8s/dashboard:v1.8.3
docker push harbor.od.com/k8s/dashboard:v1.8.3
The push refers to a repository [harbor.od.com/k8s/dashboard]
23ddb8cbb75a: Pushed 
v1.8.3: digest: sha256:e76c5fe6886c99873898e4c8c0945261709024c4bea773fc477629455631e472 size: 529

准备资源配置清单

运维主机HDSS7-200.host.com上：

1
[root@hdss7-200 ~]# mkdir -p /data/k8s-yaml/dashboard && cd /data/k8s-yaml/dashboard

vi /data/k8s-yaml/dashboard/rbac.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
    addonmanager.kubernetes.io/mode: Reconcile
  name: kubernetes-dashboard-admin
  namespace: kube-system
\--\-
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard-admin
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard-admin
  namespace: kube-system

vi /data/k8s-yaml/dashboard/secret.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
    # Allows editing resource and makes sure it is created first.
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kubernetes-dashboard-certs
  namespace: kube-system
type: Opaque
\--\-
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
    # Allows editing resource and makes sure it is created first.
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kubernetes-dashboard-key-holder
  namespace: kube-system
type: Opaque

vi /data/k8s-yaml/dashboard/configmap.yaml

1
2
3
4
5
6
7
8
9
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    k8s-app: kubernetes-dashboard
    # Allows editing resource and makes sure it is created first.
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kubernetes-dashboard-settings
  namespace: kube-system

vi /data/k8s-yaml/dashboard/svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: Service
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    k8s-app: kubernetes-dashboard
  ports:
  - port: 443
    targetPort: 8443

vi /data/k8s-yaml/dashboard/ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: dashboard.od.com
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 443

vi /data/k8s-yaml/dashboard/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      priorityClassName: system-cluster-critical
      containers:
      - name: kubernetes-dashboard
        image: harbor.od.com/k8s/dashboard:v1.8.3
        resources:
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 50m
            memory: 100Mi
        ports:
        - containerPort: 8443
          protocol: TCP
        args:
          # PLATFORM-SPECIFIC ARGS HERE
          - -\-auto-generate-certificates
        volumeMounts:
        - name: kubernetes-dashboard-certs
          mountPath: /certs
        - name: tmp-volume
          mountPath: /tmp
        livenessProbe:
          httpGet:
            scheme: HTTPS
            path: /
            port: 8443
          initialDelaySeconds: 30
          timeoutSeconds: 30
      volumes:
      - name: kubernetes-dashboard-certs
        secret:
          secretName: kubernetes-dashboard-certs
      - name: tmp-volume
        emptyDir: {}
      serviceAccountName: kubernetes-dashboard-admin
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      imagePullSecrets:
      - name: harbor

解析域名

HDSS7-11.host.com上

/var/named/od.com.zone
1
dashboard	60 IN A 10.4.7.10

依次执行创建

浏览器打开：http://k8s-yaml.od.com/dashboard 检查资源配置清单文件是否正确创建
在任意运算节点应用资源配置清单

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/rbac.yaml 
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-admin created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-admin created

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/secret.yaml 
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-key-holder created

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/configmap.yaml 
configmap/kubernetes-dashboard-settings created

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/svc.yaml 
service/kubernetes-dashboard created

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/ingress.yaml 
ingress.extensions/kubernetes-dashboard created

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/deployment.yaml 
deployment.apps/kubernetes-dashboard created

浏览器访问

http://dashboard.od.com

配置认证

  • 下载新版dashboard
1
2
3
[root@hdss7-200 ~]# docker pull hexun/kubernetes-dashboard-amd64:v1.10.1
[root@hdss7-200 ~]# docker tag f9aed6605b81 harbor.od.com/k8s/dashboard:v1.10.1
[root@hdss7-200 ~]# docker push harbor.od.com/k8s/dashboard:v1.10.1

  • 应用新版dashboard

  • 修改nginx配置，走https

/etc/nginx/conf.d/dashboard.od.com.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
server {
    listen       80;
    server_name  dashboard.od.com;

    rewrite ^(.*)$ https://${server_name}$1 permanent;
}
server {
    listen       443 ssl;
    server_name  dashboard.od.com;

    ssl_certificate "certs/dashboard.od.com.crt";
    ssl_certificate_key "certs/dashboard.od.com.key";
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  10m;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://default_backend_traefik;
	      proxy_set_header Host       $http_host;
        proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
    }
}
  • 获取token
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@hdsss7-21 ~]# kubectl describe secret kubernetes-dashboard-admin-token-rhr62 -n kube-system
Name:         kubernetes-dashboard-admin-token-rhr62
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard-admin
              kubernetes.io/service-account.uid: cdd3c552-856d-11e9-ae34-782bcb321c07

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1354 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1yaHI2MiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImNkZDNjNTUyLTg1NmQtMTFlOS1hZTM0LTc4MmJjYjMyMWMwNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.72OcJZCm_3I-7QZcEJTRPyIJSxQwSwZfVsB6Bx_RAZRJLOv3-BXy88PclYgxRy2dDqeX6cpjvFPBrmNOGQoxT9oD8_H49pvBnqdCdNuoJbXK7aBIZdkZxATzXd-63zmhHhUBsM3Ybgwy5XxD3vj8VUYfux5c5Mr4TzU_rnGLCj1H5mq_JJ3hNabv0rwil-ZAV-3HLikOMiIRhEK7RdMs1bfXF2yvse4VOabe9xv47TvbEYns97S4OlZvsurmOk0B8dD85OSaREEtqa8n_ND9GrHeeL4CcALqWYJHLrr7vLfndXi1QHDVrUzFKvgkAeYpDVAzGwIWL7rgHwp3sQguGA

部署heapster

heapster官方github地址

准备heapster镜像

运维主机HDSS7-200.host.com上

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@hdss7-200 ~]# docker pull quay.io/bitnami/heapster:1.5.4
1.5.4: Pulling from bitnami/heapster
4018396ca1ba: Pull complete 
0e4723f815c4: Pull complete 
d8569f30adeb: Pull complete 
Digest: sha256:6d891479611ca06a5502bc36e280802cbf9e0426ce4c008dd2919c2294ce0324
Status: Downloaded newer image for quay.io/bitnami/heapster:1.5.4
[root@hdss7--200 ~]# docker tag c359b95ad38b harbor.od.com/k8s/heapster:v1.5.4
[root@hdss7--200 ~]# docker push !$
docker push harbor.od.com/k8s/heapster:v1.5.4
The push refers to a repository [harbor.od.com/k8s/heapster]
20d37d828804: Pushed 
b9b192015e25: Pushed 
b76dba5a0109: Pushed 
v1.5.4: digest: sha256:1203b49f2b2b07e02e77263bce8bb30563a91e1d7ee7c6742e9d125abcb3abe6 size: 952

准备资源配置清单

vi /data/k8s-yaml/dashboard/heapster/rbac.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system
\--\-
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: heapster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:heapster
subjects:
- kind: ServiceAccount
  name: heapster
  namespace: kube-system

vi /data/k8s-yaml/dashboard/heapster/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: harbor.od.com/k8s/heapster:v1.5.4
        imagePullPolicy: IfNotPresent
        command:
        - /opt/bitnami/heapster/bin/heapster
        - \--source=kubernetes:https://kubernetes.default

vi /data/k8s-yaml/dashboard/heapster/svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster

应用资源配置清单

任意运算节点上：

1
2
3
4
5
6
7
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/rbac.yaml 
serviceaccount/heapster created
clusterrolebinding.rbac.authorization.k8s.io/heapster created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/deployment.yaml 
deployment.extensions/heapster created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dashboard/heapster/svc.yaml 
service/heapster created

重启dashboard

浏览器访问：http://dashboard.od.com
加入heapster插件的dashboard

排错专用命令

1
for j in `kubectl get ns|sed '1d'|awk '{print $1}'`;do for i in `kubectl get pods -n $j|grep -iv running|sed '1d'|awk '{print $1}'`;do kubectl delete pods $i -n $j --force --grace-period=0;done;done

发表于 | 更新于 | 分类于
本文字数： 49k | 阅读时长 ≈ 45 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

基础架构

主机名 角色 ip
HDSS7-11.host.com k8s代理节点1，zk1 10.4.7.11
HDSS7-12.host.com k8s代理节点2，zk2 10.4.7.12
HDSS7-21.host.com k8s运算节点1，zk3 10.4.7.21
HDSS7-22.host.com k8s运算节点2，jenkins 10.4.7.22
HDSS7-200.host.com k8s运维节点(docker仓库) 10.4.7.200

部署zookeeper

安装jdk1.8（3台zk角色主机）

jdk下载地址
jdk1.6
jdk1.7
jdk1.8

/opt/src
1
2
3
4
5
6
7
8
9
[root@hdss7-11 src]# ls -l|grep jdk
-rw-r--r-- 1 root root 153530841 Jan 17 17:49 jdk-8u201-linux-x64.tar.gz
[root@hdss7-11 src]# mkdir /usr/java
[root@hdss7-11 src]# tar xf jdk-8u201-linux-x64.tar.gz -C /usr/java
[root@hdss7-11 src]# ln -s /usr/java/jdk1.8.0_201 /usr/java/jdk
[root@hdss7-11 src]# vi /etc/profile
export JAVA_HOME=/usr/java/jdk
export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$PATH
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

安装zookeeper（3台zk角色主机）

zk下载地址
zookeeper

解压、配置

/opt/src
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@hdss7-11 src]# ls -l|grep zoo
-rw-r--r-- 1 root root 153530841 Jan 17 18:10 zookeeper-3.4.14.tar.gz
[root@hdss7-11 src]# tar xf /opt/src/zookeeper-3.4.14.tar.gz -C /opt
[root@hdss7-11 opt]# ln -s /opt/zookeeper-3.4.14/ /opt/zookeeper
[root@hdss7-11 opt]# mkdir -pv /data/zookeeper/data /data/zookeeper/logs
[root@hdss7-11 opt]# vi /opt/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
server.1=zk1.od.com:2888:3888
server.2=zk2.od.com:2888:3888
server.3=zk3.od.com:2888:3888

注意：各节点zk配置相同。

myid

HDSS7-11.host.com上：

/data/zookeeper/data/myid
1
1

HDSS7-12.host.com上：

/data/zookeeper/data/myid
1
2

HDSS7-21.host.com上：

/data/zookeeper/data/myid
1
3

做dns解析

HDSS7-11.host.com上

/var/named/od.com.zone
1
2
3
zk1	60 IN A 10.4.7.11
zk2	60 IN A 10.4.7.12
zk3	60 IN A 10.4.7.21

依次启动

1
2
3
4
[root@hdss7-11 opt]# /opt/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

部署jenkins

准备镜像

jenkins官网
jenkins镜像

在运维主机下载官网上的稳定版（这里下载2.164.1）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[root@hdss7-200 ~]#  docker pull jenkins/jenkins:2.164.1
2.164.1: Pulling from jenkins/jenkins
22dbe790f715: Pull complete 
0250231711a0: Pull complete 
6fba9447437b: Pull complete 
c2b4d327b352: Pull complete 
cddb9bb0d37c: Pull complete 
b535486c968f: Pull complete 
f3e976e6210c: Pull complete 
b2c11b10291d: Pull complete 
f4c0181e1976: Pull complete 
924c8e712392: Pull complete 
d13006b7c9dd: Pull complete 
fc80aeb92627: Pull complete 
36a6e96ba1b5: Pull complete 
f50f33dc1d0a: Pull complete 
b10642432117: Pull complete 
850c260511d8: Pull complete 
47f95e65a629: Pull complete 
3b33ce546dc6: Pull complete 
051c7665e760: Pull complete 
fe379aecc538: Pull complete 
Digest: sha256:12fd14965de7274b5201653b2bffa62700c5f5f336ec75c945321e2cb70d7af0
Status: Downloaded newer image for jenkins/jenkins:2.164.1

[root@hdss7-200 ~]# docker tag 256cb12e72d6 harbor.od.com/public/jenkins:v2.164.1
[root@hdss7-200 ~]# docker push harbor.od.com/public/jenkins:v2.164.1

自定义Dockerfile

在运维主机HDSS7-200.host.com上编辑自定义dockerfile

/data/dockerfile/jenkins/Dockerfile
1
2
3
4
5
6
7
8
9
FROM harbor.od.com/public/jenkins:v2.164.1
USER root
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ 
    echo 'Asia/Shanghai' >/etc/timezone
ADD id_rsa /root/.ssh/id_rsa
ADD config.json /root/.docker/config.json
ADD get-docker.sh /get-docker.sh
RUN echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config &&\
    /get-docker.sh

这个Dockerfile里我们主要做了以下几件事

  • 设置容器用户为root
  • 设置容器内的时区
  • 将ssh私钥加入（使用git拉代码时要用到，配对的公钥应配置在gitlab中）
  • 加入了登录自建harbor仓库的config文件
  • 修改了ssh客户端的
  • 安装一个docker的客户端

生成ssh密钥对：

1
[root@hdss7-200 ~]# ssh-keygen -t rsa -b 2048 -C "stanley.wang.m@qq.com" -N "" -f /root/.ssh/id_rsa
1
2
3
4
5
6
7
{
    "auths": {
        "harbor.od.com": {
            "auth": "YWRtaW46SGFyYm9yMTIzNDU="
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
#!/bin/sh
set -e

# This script is meant for quick & easy install via:
#   $ curl -fsSL get.docker.com -o get-docker.sh
#   $ sh get-docker.sh
#
# For test builds (ie. release candidates):
#   $ curl -fsSL test.docker.com -o test-docker.sh
#   $ sh test-docker.sh
#
# NOTE: Make sure to verify the contents of the script
#       you downloaded matches the contents of install.sh
#       located at https://github.com/docker/docker-install
#       before executing.
#
# Git commit from https://github.com/docker/docker-install when
# the script was uploaded (Should only be modified by upload job):
SCRIPT_COMMIT_SHA=36b78b2


# This value will automatically get changed for:
#   * edge
#   * test
#   * experimental
DEFAULT_CHANNEL_VALUE="edge"
if [ -z "$CHANNEL" ]; then
    CHANNEL=$DEFAULT_CHANNEL_VALUE
fi

DEFAULT_DOWNLOAD_URL="https://download.docker.com"
if [ -z "$DOWNLOAD_URL" ]; then
    DOWNLOAD_URL=$DEFAULT_DOWNLOAD_URL
fi

DEFAULT_REPO_FILE="docker-ce.repo"
if [ -z "$REPO_FILE" ]; then
    REPO_FILE="$DEFAULT_REPO_FILE"
fi

SUPPORT_MAP="
x86_64-centos-7
x86_64-fedora-26
x86_64-fedora-27
x86_64-fedora-28
x86_64-debian-wheezy
x86_64-debian-jessie
x86_64-debian-stretch
x86_64-debian-buster
x86_64-ubuntu-trusty
x86_64-ubuntu-xenial
x86_64-ubuntu-bionic
x86_64-ubuntu-artful
s390x-ubuntu-xenial
s390x-ubuntu-bionic
s390x-ubuntu-artful
ppc64le-ubuntu-xenial
ppc64le-ubuntu-bionic
ppc64le-ubuntu-artful
aarch64-ubuntu-xenial
aarch64-ubuntu-bionic
aarch64-debian-jessie
aarch64-debian-stretch
aarch64-debian-buster
aarch64-fedora-26
aarch64-fedora-27
aarch64-fedora-28
aarch64-centos-7
armv6l-raspbian-jessie
armv7l-raspbian-jessie
armv6l-raspbian-stretch
armv7l-raspbian-stretch
armv7l-debian-jessie
armv7l-debian-stretch
armv7l-debian-buster
armv7l-ubuntu-trusty
armv7l-ubuntu-xenial
armv7l-ubuntu-bionic
armv7l-ubuntu-artful
"

mirror=''
DRY_RUN=${DRY_RUN:-}
while [ $# -gt 0 ]; do
    case "$1" in
        --mirror)
            mirror="$2"
            shift
            ;;
        --dry-run)
            DRY_RUN=1
            ;;
        --*)
            echo "Illegal option $1"
            ;;
    esac
    shift $(( $# > 0 ? 1 : 0 ))
done

case "$mirror" in
    Aliyun)
        DOWNLOAD_URL="https://mirrors.aliyun.com/docker-ce"
        ;;
    AzureChinaCloud)
        DOWNLOAD_URL="https://mirror.azure.cn/docker-ce"
        ;;
esac

command_exists() {
    command -v "$@" > /dev/null 2>&1
}

is_dry_run() {
    if [ -z "$DRY_RUN" ]; then
        return 1
    else
        return 0
    fi
}

deprecation_notice() {
    distro=$1
    date=$2
    echo
    echo "DEPRECATION WARNING:"
    echo "    The distribution, $distro, will no longer be supported in this script as of $date."
    echo "    If you feel this is a mistake please submit an issue at https://github.com/docker/docker-install/issues/new"
    echo
    sleep 10
}

get_distribution() {
    lsb_dist=""
    # Every system that we officially support has /etc/os-release
    if [ -r /etc/os-release ]; then
        lsb_dist="$(. /etc/os-release && echo "$ID")"
    fi
    # Returning an empty string here should be alright since the
    # case statements don't act unless you provide an actual value
    echo "$lsb_dist"
}

add_debian_backport_repo() {
    debian_version="$1"
    backports="deb http://ftp.debian.org/debian $debian_version-backports main"
    if ! grep -Fxq "$backports" /etc/apt/sources.list; then
        (set -x; $sh_c "echo \"$backports\" >> /etc/apt/sources.list")
    fi
}

echo_docker_as_nonroot() {
    if is_dry_run; then
        return
    fi
    if command_exists docker && [ -e /var/run/docker.sock ]; then
        (
            set -x
            $sh_c 'docker version'
        ) || true
    fi
    your_user=your-user
    [ "$user" != 'root' ] && your_user="$user"
    # intentionally mixed spaces and tabs here -- tabs are stripped by "<<-EOF", spaces are kept in the output
    echo "If you would like to use Docker as a non-root user, you should now consider"
    echo "adding your user to the \"docker\" group with something like:"
    echo
    echo "  sudo usermod -aG docker $your_user"
    echo
    echo "Remember that you will have to log out and back in for this to take effect!"
    echo
    echo "WARNING: Adding a user to the \"docker\" group will grant the ability to run"
    echo "         containers which can be used to obtain root privileges on the"
    echo "         docker host."
    echo "         Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface"
    echo "         for more information."

}

# Check if this is a forked Linux distro
check_forked() {

    # Check for lsb_release command existence, it usually exists in forked distros
    if command_exists lsb_release; then
        # Check if the `-u` option is supported
        set +e
        lsb_release -a -u > /dev/null 2>&1
        lsb_release_exit_code=$?
        set -e

        # Check if the command has exited successfully, it means we're in a forked distro
        if [ "$lsb_release_exit_code" = "0" ]; then
            # Print info about current distro
            cat <<-EOF
            You're using '$lsb_dist' version '$dist_version'.
            EOF

            # Get the upstream release info
            lsb_dist=$(lsb_release -a -u 2>&1 | tr '[:upper:]' '[:lower:]' | grep -E 'id' | cut -d ':' -f 2 | tr -d '[:space:]')
            dist_version=$(lsb_release -a -u 2>&1 | tr '[:upper:]' '[:lower:]' | grep -E 'codename' | cut -d ':' -f 2 | tr -d '[:space:]')

            # Print info about upstream distro
            cat <<-EOF
            Upstream release is '$lsb_dist' version '$dist_version'.
            EOF
        else
            if [ -r /etc/debian_version ] && [ "$lsb_dist" != "ubuntu" ] && [ "$lsb_dist" != "raspbian" ]; then
                if [ "$lsb_dist" = "osmc" ]; then
                    # OSMC runs Raspbian
                    lsb_dist=raspbian
                else
                    # We're Debian and don't even know it!
                    lsb_dist=debian
                fi
                dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')"
                case "$dist_version" in
                    9)
                        dist_version="stretch"
                    ;;
                    8|'Kali Linux 2')
                        dist_version="jessie"
                    ;;
                    7)
                        dist_version="wheezy"
                    ;;
                esac
            fi
        fi
    fi
}

semverParse() {
    major="${1%%.*}"
    minor="${1#$major.}"
    minor="${minor%%.*}"
    patch="${1#$major.$minor.}"
    patch="${patch%%[-.]*}"
}

ee_notice() {
    echo
    echo
    echo "  WARNING: $1 is now only supported by Docker EE"
    echo "           Check https://store.docker.com for information on Docker EE"
    echo
    echo
}

do_install() {
    echo "# Executing docker install script, commit: $SCRIPT_COMMIT_SHA"

    if command_exists docker; then
        docker_version="$(docker -v | cut -d ' ' -f3 | cut -d ',' -f1)"
        MAJOR_W=1
        MINOR_W=10

        semverParse "$docker_version"

        shouldWarn=0
        if [ "$major" -lt "$MAJOR_W" ]; then
            shouldWarn=1
        fi

        if [ "$major" -le "$MAJOR_W" ] && [ "$minor" -lt "$MINOR_W" ]; then
            shouldWarn=1
        fi

        cat >&2 <<-'EOF'
            Warning: the "docker" command appears to already exist on this system.

            If you already have Docker installed, this script can cause trouble, which is
            why we're displaying this warning and provide the opportunity to cancel the
            installation.

            If you installed the current Docker package using this script and are using it
        EOF

        if [ $shouldWarn -eq 1 ]; then
            cat >&2 <<-'EOF'
            again to update Docker, we urge you to migrate your image store before upgrading
            to v1.10+.

            You can find instructions for this here:
            https://github.com/docker/docker/wiki/Engine-v1.10.0-content-addressability-migration
            EOF
        else
            cat >&2 <<-'EOF'
            again to update Docker, you can safely ignore this message.
            EOF
        fi

        cat >&2 <<-'EOF'

            You may press Ctrl+C now to abort this script.
        EOF
        ( set -x; sleep 20 )
    fi

    user="$(id -un 2>/dev/null || true)"

    sh_c='sh -c'
    if [ "$user" != 'root' ]; then
        if command_exists sudo; then
            sh_c='sudo -E sh -c'
        elif command_exists su; then
            sh_c='su -c'
        else
            cat >&2 <<-'EOF'
            Error: this installer needs the ability to run commands as root.
            We are unable to find either "sudo" or "su" available to make this happen.
            EOF
            exit 1
        fi
    fi

    if is_dry_run; then
        sh_c="echo"
    fi

    # perform some very rudimentary platform detection
    lsb_dist=$( get_distribution )
    lsb_dist="$(echo "$lsb_dist" | tr '[:upper:]' '[:lower:]')"

    case "$lsb_dist" in

        ubuntu)
            if command_exists lsb_release; then
                dist_version="$(lsb_release --codename | cut -f2)"
            fi
            if [ -z "$dist_version" ] && [ -r /etc/lsb-release ]; then
                dist_version="$(. /etc/lsb-release && echo "$DISTRIB_CODENAME")"
            fi
        ;;

        debian|raspbian)
            dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')"
            case "$dist_version" in
                9)
                    dist_version="stretch"
                ;;
                8)
                    dist_version="jessie"
                ;;
                7)
                    dist_version="wheezy"
                ;;
            esac
        ;;

        centos)
            if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
                dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
            fi
        ;;

        rhel|ol|sles)
            ee_notice "$lsb_dist"
            exit 1
            ;;

        *)
            if command_exists lsb_release; then
                dist_version="$(lsb_release --release | cut -f2)"
            fi
            if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
                dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
            fi
        ;;

    esac

    # Check if this is a forked Linux distro
    check_forked

    # Check if we actually support this configuration
    if ! echo "$SUPPORT_MAP" | grep "$(uname -m)-$lsb_dist-$dist_version" >/dev/null; then
        cat >&2 <<-'EOF'

        Either your platform is not easily detectable or is not supported by this
        installer script.
        Please visit the following URL for more detailed installation instructions:

        https://docs.docker.com/engine/installation/

        EOF
        exit 1
    fi

    # Run setup for each distro accordingly
    case "$lsb_dist" in
        ubuntu|debian|raspbian)
            pre_reqs="apt-transport-https ca-certificates curl"
            if [ "$lsb_dist" = "debian" ]; then
                if [ "$dist_version" = "wheezy" ]; then
                    add_debian_backport_repo "$dist_version"
                fi
                # libseccomp2 does not exist for debian jessie main repos for aarch64
                if [ "$(uname -m)" = "aarch64" ] && [ "$dist_version" = "jessie" ]; then
                    add_debian_backport_repo "$dist_version"
                fi
            fi

            # TODO: August 31, 2018 delete from here,
            if [ "$lsb_dist" =  "ubuntu" ] && [ "$dist_version" = "artful" ]; then
                deprecation_notice "$lsb_dist $dist_version" "August 31, 2018"
            fi
            # TODO: August 31, 2018 delete to here,

            if ! command -v gpg > /dev/null; then
                pre_reqs="$pre_reqs gnupg"
            fi
            apt_repo="deb [arch=$(dpkg --print-architecture)] $DOWNLOAD_URL/linux/$lsb_dist $dist_version $CHANNEL"
            (
                if ! is_dry_run; then
                    set -x
                fi
                $sh_c 'apt-get update -qq >/dev/null'
                $sh_c "apt-get install -y -qq $pre_reqs >/dev/null"
                $sh_c "curl -fsSL \"$DOWNLOAD_URL/linux/$lsb_dist/gpg\" | apt-key add -qq - >/dev/null"
                $sh_c "echo \"$apt_repo\" > /etc/apt/sources.list.d/docker.list"
                if [ "$lsb_dist" = "debian" ] && [ "$dist_version" = "wheezy" ]; then
                    $sh_c 'sed -i "/deb-src.*download\.docker/d" /etc/apt/sources.list.d/docker.list'
                fi
                $sh_c 'apt-get update -qq >/dev/null'
            )
            pkg_version=""
            if [ ! -z "$VERSION" ]; then
                if is_dry_run; then
                    echo "# WARNING: VERSION pinning is not supported in DRY_RUN"
                else
                    # Will work for incomplete versions IE (17.12), but may not actually grab the "latest" if in the test channel
                    pkg_pattern="$(echo "$VERSION" | sed "s/-ce-/~ce~.*/g" | sed "s/-/.*/g").*-0~$lsb_dist"
                    search_command="apt-cache madison 'docker-ce' | grep '$pkg_pattern' | head -1 | cut -d' ' -f 4"
                    pkg_version="$($sh_c "$search_command")"
                    echo "INFO: Searching repository for VERSION '$VERSION'"
                    echo "INFO: $search_command"
                    if [ -z "$pkg_version" ]; then
                        echo
                        echo "ERROR: '$VERSION' not found amongst apt-cache madison results"
                        echo
                        exit 1
                    fi
                    pkg_version="=$pkg_version"
                fi
            fi
            (
                if ! is_dry_run; then
                    set -x
                fi
                $sh_c "apt-get install -y -qq --no-install-recommends docker-ce$pkg_version >/dev/null"
            )
            echo_docker_as_nonroot
            exit 0
            ;;
        centos|fedora)
            yum_repo="$DOWNLOAD_URL/linux/$lsb_dist/$REPO_FILE"
            if ! curl -Ifs "$yum_repo" > /dev/null; then
                echo "Error: Unable to curl repository file $yum_repo, is it valid?"
                exit 1
            fi
            if [ "$lsb_dist" = "fedora" ]; then
                if [ "$dist_version" -lt "26" ]; then
                    echo "Error: Only Fedora >=26 are supported"
                    exit 1
                fi

                pkg_manager="dnf"
                config_manager="dnf config-manager"
                enable_channel_flag="--set-enabled"
                pre_reqs="dnf-plugins-core"
                pkg_suffix="fc$dist_version"
            else
                pkg_manager="yum"
                config_manager="yum-config-manager"
                enable_channel_flag="--enable"
                pre_reqs="yum-utils"
                pkg_suffix="el"
            fi
            (
                if ! is_dry_run; then
                    set -x
                fi
                $sh_c "$pkg_manager install -y -q $pre_reqs"
                $sh_c "$config_manager --add-repo $yum_repo"

                if [ "$CHANNEL" != "stable" ]; then
                    $sh_c "$config_manager $enable_channel_flag docker-ce-$CHANNEL"
                fi
                $sh_c "$pkg_manager makecache"
            )
            pkg_version=""
            if [ ! -z "$VERSION" ]; then
                if is_dry_run; then
                    echo "# WARNING: VERSION pinning is not supported in DRY_RUN"
                else
                    pkg_pattern="$(echo "$VERSION" | sed "s/-ce-/\\\\.ce.*/g" | sed "s/-/.*/g").*$pkg_suffix"
                    search_command="$pkg_manager list --showduplicates 'docker-ce' | grep '$pkg_pattern' | tail -1 | awk '{print \$2}'"
                    pkg_version="$($sh_c "$search_command")"
                    echo "INFO: Searching repository for VERSION '$VERSION'"
                    echo "INFO: $search_command"
                    if [ -z "$pkg_version" ]; then
                        echo
                        echo "ERROR: '$VERSION' not found amongst $pkg_manager list results"
                        echo
                        exit 1
                    fi
                    # Cut out the epoch and prefix with a '-'
                    pkg_version="-$(echo "$pkg_version" | cut -d':' -f 2)"
                fi
            fi
            (
                if ! is_dry_run; then
                    set -x
                fi
                $sh_c "$pkg_manager install -y -q docker-ce$pkg_version"
            )
            echo_docker_as_nonroot
            exit 0
            ;;
    esac
    exit 1
}

# wrapped up in a function so that we have some protection against only getting
# half the file during "curl | sh"
do_install

制作自定义镜像

/data/dockerfile/jenkins
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
[root@hdss7-200 jenkins]# ls -l
total 24
-rw------- 1 root root    98 Jan  17 15:58 config.json
-rw-r--r-- 1 root root   158 Jan  17 15:59 Dockerfile
-rwxr-xr-x 1 root root 13847 Jan  17 15:37 get-docker.sh
-rw------- 1 root root  1679 Jan  17 15:39 id_rsa
[root@hdss7-200 jenkins]# docker build . -t harbor.od.com/infra/jenkins:v2.164.1
Sending build context to Docker daemon 19.46 kB
Step 1 : FROM harbor.od.com/public/jenkins:v2.164.1
 ---> 256cb12e72d6
Step 2 : USER root
 ---> Running in d600e9db8305
 ---> 03687cf21cb3
Removing intermediate container d600e9db8305
Step 3 : RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo 'Asia/Shanghai' >/etc/timezone
 ---> Running in 3d79b4025e97
 ---> e4790b3bb6d9
Removing intermediate container 3d79b4025e97
Step 4 : ADD id_rsa /root/.ssh/id_rsa
 ---> 39d80713d43c
Removing intermediate container 7b4e66e726dd
Step 5 : ADD config.json /root/.docker/config.json
 ---> a44402fd4bc1
Removing intermediate container f1ae1871d035
Step 6 : ADD get-docker.sh /get-docker.sh
 ---> 189ccca429e4
Removing intermediate container a0ff59237fe5
Step 7 : RUN /get-docker.sh
 ---> Running in 5a7d69c1af45
# Executing docker install script, commit: cfba462
+ sh -c apt-get update -qq >/dev/null
+ sh -c apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
debconf: delaying package configuration, since apt-utils is not installed
+ sh -c curl -fsSL "https://download.docker.com/linux/debian/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sh -c echo "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
debconf: delaying package configuration, since apt-utils is not installed
If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:

  sudo usermod -aG docker your-user

Remember that you will have to log out and back in for this to take effect!

WARNING: Adding a user to the "docker" group will grant the ability to run
         containers which can be used to obtain root privileges on the
         docker host.
         Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
         for more information.

** DOCKER ENGINE - ENTERPRISE **

If you’re ready for production workloads, Docker Engine - Enterprise also includes:

  * SLA-backed technical support
  * Extended lifecycle maintenance policy for patches and hotfixes
  * Access to certified ecosystem content

** Learn more at https://dockr.ly/engine2 **

ACTIVATE your own engine to Docker Engine - Enterprise using:

  sudo docker engine activate

 ---> 64c74242ee28
Removing intermediate container 5a7d69c1af45
Successfully built 64c74242ee28
[root@hdss7-200 jenkins]# docker push harbor.od.com/infra/jenkins:v2.164.1

准备共享存储

运维主机，以及所有运算节点上：

1
# yum install nfs-utils -y
  • 配置NFS服务

运维主机HDSS7-200.host.com上：

/etc/exports
1
/data/nfs-volume 10.4.7.0/24(rw,no_root_squash)
  • 启动NFS服务

运维主机HDSS7-200.host.com上：

1
2
3
[root@hdss7-200 ~]# mkdir -p /data/nfs-volume
[root@hdss7-200 ~]# systemctl start nfs
[root@hdss7-200 ~]# systemctl enable nfs

准备资源配置清单

运维主机HDSS7-200.host.com上：

/data/k8s-yaml
1
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/jenkins && mkdir /data/nfs-volume/jenkins_home && cd /data/k8s-yaml/jenkins

vi deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: jenkins
  namespace: infra
  labels: 
    name: jenkins
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: jenkins
  template:
    metadata:
      labels: 
        app: jenkins 
        name: jenkins
    spec:
      volumes:
      - name: data
        nfs: 
          server: hdss7-200
          path: /data/nfs-volume/jenkins_home
      - name: docker
        hostPath: 
          path: /run/docker.sock
          type: ''
      containers:
      - name: jenkins
        image: harbor.od.com/infra/jenkins:v2.164.1
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: JAVA_OPTS
          value: -Xmx512m -Xms512m
        resources:
          limits: 
            cpu: 500m
            memory: 1Gi
          requests: 
            cpu: 500m
            memory: 1Gi
        volumeMounts:
        - name: data
          mountPath: /var/jenkins_home
        - name: docker
          mountPath: /run/docker.sock
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: Service
apiVersion: v1
metadata: 
  name: jenkins
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector:
    app: jenkins
  type: ClusterIP
  sessionAffinity: None

vi ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: jenkins
  namespace: infra
spec:
  rules:
  - host: jenkins.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: jenkins
          servicePort: 80

应用资源配置清单

任意一个k8s运算节点上

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@hdss7-21 ~]# kubectl create namespace infra
[root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/jenkins/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/jenkins/svc.yaml
[root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/jenkins/ingress.yaml

[root@hdss7-21 ~]# kubectl get pods -n infra|grep jenkins
NAME                                READY     STATUS    RESTARTS   AGE
jenkins-84455f9675-jpkr8            1/1       Running   0          0d

[root@hdss7-21 ~]# kubectl get svc -n infra|grep jenkins
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
jenkins        ClusterIP   None             <none>        8080/TCP   0d

[root@hdss7-21 ~]# kubectl get ingress -n infra|grep jenkins
NAME           HOSTS                          ADDRESS   PORTS     AGE
jenkins        jenkins.od.com                           80        0d

解析域名

HDSS7-11.host.com上

/var/named/od.com.zone
1
jenkins	60 IN A 10.4.7.10

浏览器访问

http://jenkins.od.com

页面配置jenkins

jenkins初始化页面

初始化密码

/data/nfs-volume/jenkins_home/secrets/initialAdminPassword
1
2
[root@hdss7-200 secrets]# cat initialAdminPassword 
08d17edc125444a28ad6141ffdfd5c69

安装插件

jenkins安装页面

设置用户

jenkins设置用户

完成安装

jenkins完成安装1
jenkins完成安装2

使用admin登录

jenkins登录

安装Blue Ocean插件

  • Manage Jenkins
  • Manage Plugins
  • Available
  • Blue Ocean

调整安全选项

  • Manage Jenkins
  • Configure Global Security
  • Allow anonymous read access

配置New job

  • create new jobs

  • Enter an item name

    dubbo-demo

  • Pipeline -> OK

  • Discard old builds

    Days to keep builds : 3
    Max # of builds to keep : 30

  • This project is parameterized

  1. Add Parameter -> String Parameter

    Name : app_name
    Default Value :
    Description : project name. e.g: dubbo-demo-service

  2. Add Parameter -> String Parameter

    Name : image_name
    Default Value :
    Description : project docker image name. e.g: app/dubbo-demo-service

  3. Add Parameter -> String Parameter

    Name : git_repo
    Default Value :
    Description : project git repository. e.g: https://gitee.com/stanleywang/dubbo-demo-service.git

  4. Add Parameter -> String Parameter

    Name : git_ver
    Default Value :
    Description : git commit id of the project.

  5. Add Parameter -> String Parameter

    Name : add_tag
    Default Value :
    Description : project docker image tag, date_timestamp recommended. e.g: 190117_1920

  6. Add Parameter -> String Parameter

    Name : mvn_dir
    Default Value : ./
    Description : project maven directory. e.g: ./

  7. Add Parameter -> String Parameter

    Name : target_dir
    Default Value : ./target
    Description : the relative path of target file such as .jar or .war package. e.g: ./dubbo-server/target

  8. Add Parameter -> String Parameter

    Name : mvn_cmd
    Default Value : mvn clean package -Dmaven.test.skip=true
    Description : maven command. e.g: mvn clean package -e -q -Dmaven.test.skip=true

  9. Add Parameter -> Choice Parameter

    Name : base_image
    Default Value :

    • base/jre7:7u80
    • base/jre8:8u112
      Description : project base image list in harbor.od.com.

  10. Add Parameter -> Choice Parameter

    Name : maven
    Default Value :

    • 3.6.0-8u181
    • 3.2.5-6u025
    • 2.2.1-6u025
      Description : different maven edition.

Pipeline Script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
pipeline {
  agent any 
    stages {
      stage('pull') { //get project code from repo 
        steps {
          sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
        }
      }
      stage('build') { //exec mvn cmd
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
        }
      }
      stage('package') { //move jar file into project_dir
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
        }
      }
      stage('image') { //build image and push to registry
        steps {
          writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/project_dir"""
          sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
        }
      }
    }
}

最后的准备工作

检查jenkins容器里的docker客户端

进入jenkins的docker容器里，检查docker客户端是否可用。

1
2
[root@hdss7-22 ~]# docker exec -ti 52e250789b78 bash
root@52e250789b78:/# docker ps -a

检查jenkins容器里的SSH key

进入jenkins的docker容器里，检查ssh连接git仓库，确认是否能拉到代码。

1
2
3
4
5
[root@hdss7-22 ~]# docker exec -ti 52e250789b78 bash
root@52e250789b78:/# ssh -i /root/.ssh/id_rsa -T git@gitee.com                                                                                              
Hi Anonymous! You've successfully authenticated, but GITEE.COM does not provide shell access.
Note: Perhaps the current use is DeployKey.
Note: DeployKey only supports pull/fetch operations

部署maven软件

maven官方下载地址
在运维主机HDSS7-200.host.com上二进制部署，这里部署maven-3.6.0版

/opt/src
1
2
3
4
5
6
7
[root@hdss7-22 src]# ls -l
total 8852
-rw-r--r-- 1 root root 9063587 Jan  17 19:57 apache-maven-3.6.0-bin.tar.gz
[root@hdss7-200 src]# tar xf apache-maven-3.6.0-bin.tar.gz -C /data/nfs-volume/jenkins_home/maven-3.6.0-8u181
[root@hdss7-200 src]# mv /data/nfs-volume/jenkins_home/apache-maven-3.6.0/ /data/nfs-volume/jenkins_home/maven-3.6.0-8u181
[root@hdss7-200 src]# ls -ld /data/nfs-volume/jenkins_home/maven-3.6.0-8u181
drwxr-xr-x 6 root root 99 Jan  17 19:58 /data/nfs-volume/jenkins_home/maven-3.6.0-8u181

设置国内镜像源

/data/nfs-volume/jenkins_home/maven-3.6.0-8u181/conf/setting.xml
1
2
3
4
5
6
<mirror>
  <id>alimaven</id>
  <name>aliyun maven</name>
  <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
  <mirrorOf>central</mirrorOf>        
</mirror>

其他版本略

制作dubbo微服务的底包镜像

运维主机HDSS7-200.host.com上

  1. 自定义Dockerfile
/data/dockerfile/jre8/Dockerfile
1
2
3
4
5
6
7
8
FROM stanleyws/jre8:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo 'Asia/Shanghai' >/etc/timezone
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/
WORKDIR /opt/project_dir
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]

vi config.yml

1
2
3
\--\-
rules:
  - pattern: '.*'
1
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar

vi entrypoint.sh (不要忘了给执行权限)

1
2
3
4
5
#!/bin/sh
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"
C_OPTS=${C_OPTS}
JAR_BALL=${JAR_BALL}
exec java -jar ${M_OPTS} ${C_OPTS} ${JAR_BALL}
  1. 制作dubbo服务docker底包
/data/dockerfile/jre8
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
[root@hdss7-200 jre8]# ls -l
total 372
-rw-r--r-- 1 root root     29 Jan  17 19:09 config.yml
-rw-r--r-- 1 root root    287 Jan  17 19:06 Dockerfile
-rwxr--r-- 1 root root    250 Jan  17 19:11 entrypoint.sh
-rw-r--r-- 1 root root 367417 May 10  2018 jmx_javaagent-0.3.1.jar

[root@hdss7-200 jre8]# docker build . -t harbor.od.com/base/jre8:8u112
Sending build context to Docker daemon 372.2 kB
Step 1 : FROM stanleyws/jre8:8u112
 ---> fa3a085d6ef1
Step 2 : RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo 'Asia/Shanghai' >/etc/timezone
 ---> Using cache
 ---> 5da5ab0b1a48
Step 3 : ADD config.yml /opt/prom/config.yml
 ---> Using cache
 ---> 70d3ebfe88f5
Step 4 : ADD jmx_javaagent-0.3.1.jar /opt/prom/
 ---> Using cache
 ---> 08b38a0684a8
Step 5 : WORKDIR /opt/project_dir
 ---> Using cache
 ---> f06adf17fb69
Step 6 : ADD entrypoint.sh /entrypoint.sh
 ---> e34f185d5c52
Removing intermediate container ee213576ca0e
Step 7 : CMD /entrypoint.sh
 ---> Running in 655f594bcbe2
 ---> 47852bc0ade9
Removing intermediate container 655f594bcbe2
Successfully built 47852bc0ade9

[root@hdss7-200 jre8]# docker push harbor.od.com/base/jre8:8u112
The push refers to a repository [harbor.od.com/base/jre8]
0b2b753b122e: Pushed 
67e1b844d09c: Pushed 
ad4fa4673d87: Pushed 
0ef3a1b4ca9f: Pushed 
052016a734be: Pushed 
0690f10a63a5: Pushed 
c843b2cf4e12: Pushed 
fddd8887b725: Pushed 
42052a19230c: Pushed 
8d4d1ab5ff74: Pushed 
8u112: digest: sha256:252e3e869039ee6242c39bdfee0809242e83c8c3a06830f1224435935aeded28 size: 2405

注意：jre7底包制作类似，这里略

交付dubbo微服务至kubernetes集群

dubbo服务提供者（dubbo-demo-service）

通过jenkins进行一次CI

打开jenkins页面，使用admin登录，准备构建dubbo-demo项目

jenkins构建
点Build with Parameters

jenkins构建详情
依次填入/选择：

  • app_name

    dubbo-demo-service

  • image_name

    app/dubbo-demo-service

  • git_repo

    https://gitee.com/stanleywang/dubbo-demo-service.git

  • git_ver

    master

  • add_tag

    190117_1920

  • mvn_dir

    /

  • target_dir

    ./dubbo-server/target

  • mvn_cmd

    mvn clean package -Dmaven.test.skip=true

  • base_image

    base/jre8:8u112

  • maven

    3.6.0-8u181

点击Build进行构建，等待构建完成。

test $? -eq 0 && 成功，进行下一步 || 失败，排错直到成功

检查harbor仓库内镜像

harbor仓库内镜像

准备k8s资源配置清单

运维主机HDSS7-200.host.com上，准备资源配置清单：

/data/k8s-yaml/dubbo-demo-service/deployment.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-service
  namespace: app
  labels: 
    name: dubbo-demo-service
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:master_190117_1920
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

应用资源配置清单

在任意一台k8s运算节点执行：

1
2
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/deployment.yaml
deployment.extensions/dubbo-demo-service created

检查docker运行情况及zk里的信息

/opt/zookeeper/bin/zkCli.sh
1
2
3
[root@hdss7-11 ~]# /opt/zookeeper/bin/zkCli.sh -server localhost
[zk: localhost(CONNECTED) 0] ls /dubbo
[com.od.dubbotest.api.HelloService]

dubbo-monitor工具

dubbo-monitor源码包

准备docker镜像

下载源码

下载到运维主机HDSS7-200.host.com上

/opt/src
1
2
[root@hdss7-200 src]# ls -l|grep dubbo-monitor
drwxr-xr-x 4 root root      81 Jan  17 13:58 dubbo-monitor

修改配置

/opt/src/dubbo-monitor/dubbo-monitor-simple/conf/dubbo_origin.properties
1
2
3
4
5
6
dubbo.registry.address=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.statistics.directory=/dubbo-monitor-simple/statistics
dubbo.log4j.file=logs/dubbo-monitor.log

制作镜像

  1. 准备环境

    1
    2
    3
    4
    [root@hdss7-200 src]# mkdir /data/dockerfile/dubbo-monitor
    [root@hdss7-200 src]# cp -a dubbo-monitor/* /data/dockerfile/dubbo-monitor/
    [root@hdss7-200 src]# cd /data/dockerfile/dubbo-monitor/
    [root@hdss7-200 dubbo-monitor]# sed -r -i -e '/^nohup/{p;:a;N;$!ba;d}'  ./dubbo-monitor-simple/bin/start.sh && sed  -r -i -e "s%^nohup(.*)%exec \1%"  ./dubbo-monitor-simple/bin/start.sh

  2. 准备Dockerfile

    /data/dockerfile/dubbo-monitor/Dockerfile
    1
    2
    3
    4
    FROM jeromefromcn/docker-alpine-java-bash
    MAINTAINER Jerome Jiang
    COPY dubbo-monitor-simple/ /dubbo-monitor-simple/
    CMD /dubbo-monitor-simple/bin/start.sh

  3. build镜像

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    [root@hdss7-200 dubbo-monitor]# docker build . -t harbor.od.com/infra/dubbo-monitor:latest
    Sending build context to Docker daemon 26.21 MB
    Step 1 : FROM harbor.od.com/base/jre7:7u80
     ---> dbba4641da57
    Step 2 : MAINTAINER Stanley Wang
     ---> Running in 8851a3c55d4b
     ---> 6266a6f15dc5
    Removing intermediate container 8851a3c55d4b
    Step 3 : COPY dubbo-monitor-simple/ /opt/dubbo-monitor/
     ---> f4e0a9067c5c
    Removing intermediate container f1038ecb1055
    Step 4 : WORKDIR /opt/dubbo-monitor
     ---> Running in 4056339d1b5a
     ---> e496e2d3079e
    Removing intermediate container 4056339d1b5a
    Step 5 : CMD /opt/dubbo-monitor/bin/start.sh
     ---> Running in c33b8fb98326
     ---> 97e40c179bbe
    Removing intermediate container c33b8fb98326
    Successfully built 97e40c179bbe

    [root@hdss7-200 dubbo-monitor]# docker push harbor.od.com/infra/dubbo-monitor:latest
    The push refers to a repository [harbor.od.com/infra/dubbo-monitor]
    750135a87545: Pushed 
    0b2b753b122e: Pushed 
    5b1f1b5295ff: Pushed 
    d54f1d9d76d3: Pushed 
    8d51c20d6553: Pushed 
    106b765202e9: Pushed 
    c6698ca565d0: Pushed 
    50ecb880731d: Pushed 
    fddd8887b725: Pushed 
    42052a19230c: Pushed 
    8d4d1ab5ff74: Pushed 
    190107_1930: digest: sha256:73007a37a55ecd5fd72bc5b36d2ab0bb639c96b32b7879984d5cdbc759778790 size: 2617

解析域名

在DNS主机HDSS7-11.host.com上：

/var/named/od.com.zone
1
dubbo-monitor IN A 60 10.9.7.10

准备k8s资源配置清单

运维主机HDSS7-200.host.com上

vi /data/k8s-yaml/dubbo-monitor/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels: 
    name: dubbo-monitor
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-monitor
  template:
    metadata:
      labels: 
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi /data/k8s-yaml/dubbo-monitor/svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: Service
apiVersion: v1
metadata: 
  name: dubbo-monitor
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: dubbo-monitor
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None

vi /data/k8s-yaml/dubbo-monitor/ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-monitor
  namespace: infra
spec:
  rules:
  - host: dubbo-monitor.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-monitor
          servicePort: 8080

应用资源配置清单

在任意一台k8s运算节点执行：

1
2
3
4
5
6
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/deployment.yaml
deployment.extensions/dubbo-monitor created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/svc.yaml
service/dubbo-monitor created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/ingress.yaml
ingress.extensions/dubbo-monitor created

浏览器访问

http://dubbo-monitor.od.com

dubbo服务消费者（dubbo-demo-consumer）

通过jenkins进行一次CI

打开jenkins页面，使用admin登录，准备构建dubbo-demo项目

jenkins构建
点Build with Parameters

jenkins构建详情
依次填入/选择：

  • app_name

    dubbo-demo-consumer

  • image_name

    app/dubbo-demo-consumer

  • git_repo

    git@gitee.com:stanleywang/dubbo-demo-web.git

  • git_ver

    master

  • add_tag

    190117_1950

  • mvn_dir

    /

  • target_dir

    ./dubbo-client/target

  • mvn_cmd

    mvn clean package -Dmaven.test.skip=true

  • base_image

    base/jre8:8u112

  • maven

    3.6.0-8u181

点击Build进行构建，等待构建完成。

test $? -eq 0 && 成功，进行下一步 || 失败，排错直到成功

检查harbor仓库内镜像

harbor仓库内镜像

解析域名

在DNS主机HDSS7-11.host.com上：

/var/named/od.com.zone
1
demo IN A 60 10.9.7.10

准备k8s资源配置清单

运维主机HDSS7-200.host.com上，准备资源配置清单

vi /data/k8s-yaml/dubbo-demo-consumer/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: app
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:master_190119_2015
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi /data/k8s-yaml/dubbo-demo-consumer/svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: Service
apiVersion: v1
metadata: 
  name: dubbo-demo-consumer
  namespace: app
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: dubbo-demo-consumer
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None

vi /data/k8s-yaml/dubbo-demo-consumer/ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-demo-consumer
  namespace: app
spec:
  rules:
  - host: demo.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-demo-consumer
          servicePort: 8080

应用资源配置清单

在任意一台k8s运算节点执行：

1
2
3
4
5
6
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/deployment.yaml
deployment.extensions/dubbo-demo-consumer created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/svc.yaml
service/dubbo-demo-consumer created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-consumer/ingress.yaml
ingress.extensions/dubbo-demo-consumer created

检查docker运行情况及dubbo-monitor

http://dubbo-monitor.od.com

浏览器访问

http://demo.od.com/hello?name=wangdao

实战维护dubbo微服务集群

更新（rolling update）

  • 修改代码提git（发版）
  • 使用jenkins进行CI
  • 修改并应用k8s资源配置清单

    或者在k8s的dashboard上直接操作

扩容（scaling）

  • k8s的dashboard上直接操作

发表于 | 更新于 | 分类于
本文字数： 44k | 阅读时长 ≈ 40 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

使用ConfigMap管理应用配置

拆分环境

主机名 角色 ip
HDSS7-11.host.com zk1.od.com(Test环境) 10.4.7.11
HDSS7-12.host.com zk2.od.com(Prod环境) 10.4.7.12

重配zookeeper

HDSS7-11.host.com上：

/opt/zookeeper/conf/zoo.cfg
1
2
3
4
5
6
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181

HDSS7-12.host.com上：

/opt/zookeeper/conf/zoo.cfg
1
2
3
4
5
6
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181

重启zk(删除数据文件)

1
2
3
[root@hdss7-11 ~]# /opt/zookeeper/bin/zkServer.sh restart && /opt/zookeeper/bin/zkServer.sh status
[root@hdss7-12 ~]# /opt/zookeeper/bin/zkServer.sh restart && /opt/zookeeper/bin/zkServer.sh status
[root@hdss7-21 ~]# /opt/zookeeper/bin/zkServer.sh stop

准备资源配置清单(dubbo-monitor)

在运维主机HDSS7-200.host.com上：

vi /data/k8s-yaml/dubbo-monitor/configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: v1
kind: ConfigMap
metadata:
  name: dubbo-monitor-cm
  namespace: infra
data:
  dubbo.properties: |
    dubbo.container=log4j,spring,registry,jetty
    dubbo.application.name=simple-monitor
    dubbo.application.owner=
    dubbo.registry.address=zookeeper://zk1.od.com:2181
    dubbo.protocol.port=20880
    dubbo.jetty.port=8080
    dubbo.jetty.directory=/dubbo-monitor-simple/monitor
    dubbo.charts.directory=/dubbo-monitor-simple/charts
    dubbo.statistics.directory=/dubbo-monitor-simple/statistics
    dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
    dubbo.log4j.level=WARN

vi /data/k8s-yaml/dubbo-monitor/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels: 
    name: dubbo-monitor
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-monitor
  template:
    metadata:
      labels: 
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: configmap-volume
            mountPath: /dubbo-monitor-simple/conf
      volumes:
        - name: configmap-volume
          configMap:
            name: dubbo-monitor-cm
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

应用资源配置清单

在任意一台k8s运算节点执行：

1
2
3
4
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/configmap.yaml
configmap/dubbo-monitor-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/deployment.yaml
deployment.extensions/dubbo-monitor configured

重新发版，修改dubbo项目的配置文件

修改项目源代码

  • duboo-demo-service

    dubbo-server/src/main/java/config.properties
    1
    2
    dubbo.registry=zookeeper://zk1.od.com:2181
    dubbo.port=28080

  • dubbo-demo-web

    dubbo-client/src/main/java/config.properties
    1
    dubbo.registry=zookeeper://zk1.od.com:2181

使用Jenkins进行CI

略

修改/应用资源配置清单

k8s的dashboard上，修改deployment使用的容器版本，提交应用

验证configmap的配置

在K8S的dashboard上，修改dubbo-monitor的configmap配置为不同的zk，重启POD，浏览器打开http://dubbo-monitor.od.com 观察效果

交付Apollo至Kubernetes集群

Apollo简介

Apollo（阿波罗）是携程框架部门研发的分布式配置中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性，适用于微服务配置管理场景。

官方GitHub地址

Apollo官方地址
官方release包

基础架构

apollo基础架构

简化模型

apollo简化架构

交付apollo-configservice

准备软件包

在运维主机HDSS7-200.host.com上：
下载官方release包

/opt/src
1
2
3
4
5
6
7
8
9
10
11
12
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 23:29 apollo-configservice-1.3.0-github.zip
[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-configservice && unzip -o apollo-configservice-1.3.0-github.zip -d /data/dockerfile/apollo-configservice
Archive:  apollo-configservice-1.3.0-github.zip
   creating: /data/dockerfile/apollo-configservice/scripts/
  inflating: /data/dockerfile/apollo-configservice/config/application-github.properties  
  inflating: /data/dockerfile/apollo-configservice/scripts/shutdown.sh  
  inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.3.0-sources.jar  
  inflating: /data/dockerfile/apollo-configservice/scripts/startup.sh  
  inflating: /data/dockerfile/apollo-configservice/config/app.properties  
  inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.3.0.jar  
  inflating: /data/dockerfile/apollo-configservice/apollo-configservice.conf

执行数据库脚本

在数据库主机HDSS7-11.host.com上：
注意：MySQL版本应为5.6或以上！

  • 更新yum源
/etc/yum.repos.d/MariaDB.repo
1
2
3
4
5
[mariadb]
name = MariaDB
baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64/
gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1
  • 导入GPG-KEY
1
[root@hdss7-11 ~]# rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
  • 更新数据库版本
1
[root@hdss7-11 ~]# yum update MariaDB-server -y
  • 配置my.cnf
/etc/my.cnf
1
2
3
4
5
6
[mysql]
default-character-set = utf8mb4
[mysqld]
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
init_connect = "SET NAMES 'utf8mb4'"

数据库脚本地址

1
2
3
[root@hdss7-11 ~]# mysql -uroot -p
mysql> create database ApolloConfigDB;
mysql> source ./apolloconfig.sql

数据库用户授权

1
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

修改初始数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
mysql> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from ServerConfig\G
*************************** 1. row ***************************
                       Id: 1
                      Key: eureka.service.url
                  Cluster: default
                    Value: http://config.od.com/eureka
                  Comment: Eureka服务Url，多个service以英文逗号分隔
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2019-04-10 15:07:34
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2019-04-11 16:28:57

制作Docker镜像

在运维主机HDSS7-200.host.com上：

  • 配置数据库连接串
/data/dockerfile/apollo-configservice
1
[root@hdss7-200 apollo-configservice]# cat config/application-github.properties
  • 更新startup.sh
/data/dockerfile/apollo-configservice/scripts/startup.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
#!/bin/bash
SERVICE_NAME=apollo-configservice
## Adjust log dir if necessary
LOG_DIR=/opt/logs/apollo-config-server
## Adjust server port if necessary
SERVER_PORT=8080
APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
#export JAVA_OPTS="-Xms6144m -Xmx6144m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=4096m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
#export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
    javaexe="$JAVA_HOME/bin/java"
elif type -p java > /dev/null 2>&1; then
    javaexe=$(type -p java)
elif [[ -x "/usr/bin/java" ]];  then
    javaexe="/usr/bin/java"
else
    echo "Unable to find Java"
    exit 1
fi

if [[ "$javaexe" ]]; then
    version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
    version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
    # now version is of format 009003 (9.3.x)
    if [ $version -ge 011000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 010000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    elif [ $version -ge 009000 ]; then
        JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
    else
        JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
        JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
        JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
    fi
fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
chmod 755 $SERVICE_NAME".jar"
./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
then
    echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
    exit $rc;
fi

tail -f /dev/null
  • 写Dockerfile
/data/dockerfile/apollo-configservice/Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
FROM stanleyws/jre8:8u112

ENV VERSION 1.3.0

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
ADD config/ /apollo-configservice/config
ADD scripts/ /apollo-configservice/scripts

CMD ["/apollo-configservice/scripts/startup.sh"]
  • 制作镜像并推送
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
[root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.3.0
Sending build context to Docker daemon 61.91 MB
Step 1 : FROM stanleyws/jre8:8u112
 ---> fa3a085d6ef1
Step 2 : ENV VERSION 1.3.0
 ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
 ---> Running in 685d51b5adb4
 ---> feb4c0289f04
Removing intermediate container 685d51b5adb4
Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
 ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
 ---> Running in eaa05073feeb
 ---> a3e3fd61ae35
Removing intermediate container eaa05073feeb
Step 4 : ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
 ---> be09a59b83a2
Removing intermediate container ac6b8af3979b
Step 5 : ADD config/ /apollo-configservice/config
 ---> fb64fc0f3194
Removing intermediate container b73c5315ad20
Step 6 : ADD scripts/ /apollo-configservice/scripts
 ---> 96ff3d9b9456
Removing intermediate container 67ba203b3101
Step 7 : CMD /apollo-configservice/scripts/startup.sh
 ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
 ---> Running in 80bd3f53fefc
 ---> 551ea2ba8de3
Removing intermediate container 80bd3f53fefc
Successfully built 551ea2ba8de3

[root@hdss7-200 apollo-configservice]# docker push harbor.od.com/infra/apollo-configservice:v1.3.0
The push refers to a repository [harbor.od.com/infra/apollo-configservice]
25efb9a44683: Pushed 
b3572bb46247: Pushed 
e7994b936025: Pushed 
0ff1d078cbc4: Pushed 
ebfb473df5c2: Pushed 
aae5c057d1b6: Pushed 
dee6aef5c2b6: Pushed 
a464c54f93a9: Pushed 
v1.3.0: digest: sha256:6a8e4fdda58de0dfba9985ebbf91c4d6f46f5274983d2efa8853b03f4e45fa06 size: 1992

解析域名

DNS主机HDSS7-11.host.com上：

/var/named/od.com.zone
1
2
mysql   60 IN A 10.4.7.11
config	60 IN A 10.4.7.10

准备资源配置清单

在运维主机HDSS7-200.host.com上

/data/k8s-yaml
1
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-configservice && cd /data/k8s-yaml/apollo-configservice

vi deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-configservice
  namespace: infra
  labels: 
    name: apollo-configservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-configservice
  template:
    metadata:
      labels: 
        app: apollo-configservice 
        name: apollo-configservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-configservice-cm
      containers:
      - name: apollo-configservice
        image: harbor.od.com/infra/apollo-configservice:v1.3.0
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-configservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: Service
apiVersion: v1
metadata: 
  name: apollo-configservice
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-configservice
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None

vi ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-configservice
  namespace: infra
spec:
  rules:
  - host: config.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-configservice
          servicePort: 8080

vi configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-configservice-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003171

应用资源配置清单

在任意一台k8s运算节点执行：

1
2
3
4
5
6
7
8
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/configmap.yaml
configmap/apollo-configservice-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/deployment.yaml
deployment.extensions/apollo-configservice created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml
service/apollo-configservice created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml
ingress.extensions/apollo-configservice created

浏览器访问

http://config.od.com

交付apollo-adminservice

准备软件包

在运维主机HDSS7-200.host.com上：
下载官方release包

1
2
3
4
5
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 08:47 apollo-configservice-1.3.0-github.zip
-rw-r--r-- 1 root root 49418246 Feb 16 09:54 apollo-adminservice-1.3.0-github.zip

[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-adminservice && unzip -o apollo-adminservice-1.3.0-github.zip -d /data/dockerfile/apollo-adminservice

制作Docker镜像

在运维主机HDSS7-200.host.com上：

  • 配置数据库连接串

    /data/dockerfile/apollo-adminservice
    1
    [root@hdss7-200 apollo-adminservice]# cat config/application-github.properties

  • 更新starup.sh

    /data/dockerfile/apollo-adminservice/scripts/startup.sh
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    #!/bin/bash
    SERVICE_NAME=apollo-adminservice
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-adminservice
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:${SERVER_PORT}"
    SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"

    ## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi

    if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi

    printf "$(date) ==== Starting ==== \n"

    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start

    rc=$?;

    if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi

    tail -f /dev/null

  • 写Dockerfile

    /data/dockerfile/apollo-adminservice/Dockerfile
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    FROM stanleyws/jre8:8u112

    ENV VERSION 1.3.0

    RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
        echo "Asia/Shanghai" > /etc/timezone

    ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
    ADD config/ /apollo-adminservice/config
    ADD scripts/ /apollo-adminservice/scripts

    CMD ["/apollo-adminservice/scripts/startup.sh"]

  • 制作镜像并推送

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    [root@hdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.3.0
    Sending build context to Docker daemon 58.31 MB
    Step 1 : FROM stanleyws/jre8:8u112
     ---> fa3a085d6ef1
    Step 2 : ENV VERSION 1.3.0
     ---> Using cache
     ---> feb4c0289f04
    Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
     ---> Using cache
     ---> a3e3fd61ae35
    Step 4 : ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
     ---> 6a1eb9565777
    Removing intermediate container 7196df9af6af
    Step 5 : ADD config/ /apollo-adminservice/config
     ---> 9f364b732d46
    Removing intermediate container 9b24669c6c78
    Step 6 : ADD scripts/ /apollo-adminservice/scripts
     ---> b7bc5517b0fc
    Removing intermediate container f3e34e759148
    Step 7 : CMD /apollo-adminservice/scripts/startup.sh
     ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
     ---> Running in 18c6597914b4
     ---> 82145db3ee88
    Removing intermediate container 18c6597914b4
    Successfully built 82145db3ee88

    [root@hdss7-200 apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.3.0
    docker push harbor.od.com/infra/apollo-adminservice:v1.3.0
    The push refers to a repository [harbor.od.com/infra/apollo-adminservice]
    19b1ca6c066d: Pushed 
    8fa6cde49908: Pushed 
    0b2c9b9226cc: Pushed 
    ebfb473df5c2: Mounted from infra/apollo-configservice 
    aae5c057d1b6: Mounted from infra/apollo-configservice 
    dee6aef5c2b6: Mounted from infra/apollo-configservice 
    a464c54f93a9: Mounted from infra/apollo-configservice 
    v1.3.0: digest: sha256:75367caab9bad3d0d281eb3324451a0734e84b6aa3ee860e38ad758d7166a7d1 size: 1785

准备资源配置清单

在运维主机HDSS7-200.host.com上

/data/k8s-yaml
1
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-adminservice && cd /data/k8s-yaml/apollo-adminservice

vi deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-adminservice
  namespace: infra
  labels: 
    name: apollo-adminservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-adminservice
  template:
    metadata:
      labels: 
        app: apollo-adminservice 
        name: apollo-adminservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-adminservice-cm
      containers:
      - name: apollo-adminservice
        image: harbor.od.com/infra/apollo-adminservice:v1.3.0
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-adminservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-adminservice-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003172

应用资源配置清单

在任意一台k8s运算节点执行：

1
2
3
4
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/configmap.yaml
configmap/apollo-adminservice-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/deployment.yaml
deployment.extensions/apollo-adminservice created

浏览器访问

http://config.od.com
apollo注册中心

交付apollo-portal

准备软件包

在运维主机HDSS7-200.host.com上：
下载官方release包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 08:37 apollo-configservice-1.3.0-github.zip
-rw-r--r-- 1 root root 49418246 Feb 16 09:54 apollo-adminservice-1.3.0-github.zip
-rw-r--r-- 1 root root 36459359 Feb 16 10:00 apollo-portal-1.3.0-github.zip

[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-portal && unzip -o apollo-portal-1.3.0-github.zip -d /data/dockerfile/apollo-portal
Archive:  apollo-portal-1.3.0-github.zip
  inflating: /data/dockerfile/apollo-portal/scripts/shutdown.sh  
  inflating: /data/dockerfile/apollo-portal/apollo-portal.conf  
  inflating: /data/dockerfile/apollo-portal/apollo-portal-1.3.0-sources.jar  
   creating: /data/dockerfile/apollo-portal/config/
  inflating: /data/dockerfile/apollo-portal/config/application-github.properties  
  inflating: /data/dockerfile/apollo-portal/scripts/startup.sh  
  inflating: /data/dockerfile/apollo-portal/config/apollo-env.properties  
  inflating: /data/dockerfile/apollo-portal/config/app.properties  
  inflating: /data/dockerfile/apollo-portal/apollo-portal-1.3.0.jar

执行数据库脚本

在数据库主机HDSS7-11.host.com上：
数据库脚本地址

1
2
3
[root@hdss7-11 ~]# mysql -uroot -p
mysql> create database ApolloPortalDB;
mysql> source ./apolloportal.sql

数据库用户授权

1
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"172.7.%" identified by "123456";

制作Docker镜像

在运维主机HDSS7-200.host.com上：

  • 配置数据库连接串

    /data/dockerfile/apollo-portal
    1
    [root@hdss7-200 apollo-portal]# cat config/application-github.properties

  • 配置Portal的meta service

    /data/dockerfile/apollo-portal/config/apollo-env.properties
    1
    dev.meta=http://config.od.com

  • 更新starup.sh

    /data/dockerfile/apollo-portal/scripts/startup.sh
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    #!/bin/bash
    SERVICE_NAME=apollo-portal
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-portal-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:$SERVER_PORT"
    SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

    ## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

    ## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

    ########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

    # Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi

    if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi

    printf "$(date) ==== Starting ==== \n"

    cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start

    rc=$?;

    if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi

    tail -f /dev/null

  • 写Dockerfile

/data/dockerfile/apollo-portal/Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
FROM stanleyws/jre8:8u112

ENV VERSION 1.3.0

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts

CMD ["/apollo-portal/scripts/startup.sh"]
  • 制作镜像并推送
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@hdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.3.0
Sending build context to Docker daemon 43.35 MB
Step 1 : FROM stanleyws/jre8:8u112
 ---> fa3a085d6ef1
Step 2 : ENV VERSION 1.3.0
 ---> Using cache
 ---> feb4c0289f04
Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
 ---> Using cache
 ---> a3e3fd61ae35
Step 4 : ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
 ---> cfcf63e8eedc
Removing intermediate container 860b55bd3fc5
Step 5 : ADD config/ /apollo-portal/config
 ---> 3ee780369431
Removing intermediate container 6b67ee4224b5
Step 6 : ADD scripts/ /apollo-portal/scripts
 ---> 42c9aea2e9e3
Removing intermediate container 2dcf8d1bf4cf
Step 7 : CMD /apollo-portal/scripts/startup.sh
 ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
 ---> Running in 9162dab8b63a
 ---> 0c020b79c36f
Removing intermediate container 9162dab8b63a
Successfully built 0c020b79c36f
[root@hdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.3.0
docker push harbor.od.com/infra/apollo-portal:v1.3.0
The push refers to a repository [harbor.od.com/infra/apollo-portal]
e7c0e96ded4e: Pushed 
0076c5344476: Pushed 
3851a45d7440: Pushed 
ebfb473df5c2: Mounted from infra/apollo-adminservice 
aae5c057d1b6: Mounted from infra/apollo-adminservice 
dee6aef5c2b6: Mounted from infra/apollo-adminservice 
a464c54f93a9: Mounted from infra/apollo-adminservice 
v1.3.0: digest: sha256:1aa30aac8642cceb97c053b7d74632240af08f64c49b65d8729021fef65628a4 size: 1785

解析域名

DNS主机HDSS7-11.host.com上：

/var/named/od.com.zone
1
portal	60 IN A 10.4.7.10

准备资源配置清单

在运维主机HDSS7-200.host.com上

/data/k8s-yaml
1
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-portal && cd /data/k8s-yaml/apollo-portal

vi deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-portal
  namespace: infra
  labels: 
    name: apollo-portal
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-portal
  template:
    metadata:
      labels: 
        app: apollo-portal 
        name: apollo-portal
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-portal-cm
      containers:
      - name: apollo-portal
        image: harbor.od.com/infra/apollo-portal:v1.3.0
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-portal/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi svc.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: Service
apiVersion: v1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-portal
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None

vi ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  rules:
  - host: portal.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-portal
          servicePort: 8080

vi configmap.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-portal-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
    spring.datasource.username = apolloportal
    spring.datasource.password = 123456
  app.properties: |
    appId=100003173
  apollo-env.properties: |
    dev.meta=http://config.od.com

应用资源配置清单

在任意一台k8s运算节点执行：

1
2
3
4
5
6
7
8
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/configmap.yaml
configmap/apollo-portal-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/deployment.yaml
deployment.extensions/apollo-portal created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml
service/apollo-portal created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml
ingress.extensions/apollo-portal created

浏览器访问

http://portal.od.com

  • 用户名：apollo
  • 密码： admin

apollo-portal

实战dubbo微服务接入Apollo配置中心

改造dubbo-demo-service项目

使用IDE拉取项目（这里使用git bash作为范例）

1
$ git clone git@gitee.com/stanleywang/dubbo-demo-service.git

切到apollo分支

1
$ git checkout -b apollo

修改pom.xml

  • 加入apollo客户端jar包的依赖
dubbo-server/pom.xml
1
2
3
4
5
<dependency>
  <groupId>com.ctrip.framework.apollo</groupId>
  <artifactId>apollo-client</artifactId>
  <version>1.1.0</version>
</dependency>
  • 修改resource段
dubbo-server/pom.xml
1
2
3
4
5
6
7
<resource>
  <directory>src/main/resources</directory>
  <includes>
  <include>**/*</include>
  </includes>
  <filtering>false</filtering>
</resource>

增加resources目录

/d/workspace/dubbo-demo-service/dubbo-server/src/main
1
2
3
$ mkdir -pv resources/META-INF
mkdir: created directory 'resources'
mkdir: created directory 'resources/META-INF'

修改config.properties文件

/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/config.properties
1
2
dubbo.registry=${dubbo.registry}
dubbo.port=${dubbo.port}

修改srping-config.xml文件

  • beans段新增属性
/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml
1
xmlns:apollo="http://www.ctrip.com/schema/apollo"
  • xsi:schemaLocation段内新增属性
/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml
1
http://www.ctrip.com/schema/apollo http://www.ctrip.com/schema/apollo.xsd
  • 新增配置项
/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml
1
<apollo:config/>
  • 删除配置项（注释）
/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml
1
<!-- <context:property-placeholder location="classpath:config.properties"/> -->

增加app.properties文件

/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/META-INF/app.properties
1
app.id=dubbo-demo-service

提交git中心仓库（gitee）

1
$ git push origin apollo

配置apollo-portal

创建项目

  • 部门

    样例部门1（TEST1）

  • 应用id

    dubbo-demo-service

  • 应用名称

    dubbo服务提供者

  • 应用负责人

    apollo|apollo

  • 项目管理员

    apollo|apollo

提交

进入配置页面

新增配置项1

  • Key

    dubbo.registry

  • Value

    zookeeper://zk1.od.com:2181

  • 选择集群

    DEV

提交

新增配置项2

  • Key

    dubbo.port

  • Value

    20880

  • 选择集群

    DEV

提交

发布配置

点击发布，配置生效
apollo-release

使用jenkins进行CI

略（注意记录镜像的tag）

上线新构建的项目

准备资源配置清单

运维主机HDSS7-200.host.com上：

/data/k8s-yaml/dubbo-demo-service/deployment.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-service
  namespace: app
  labels: 
    name: dubbo-demo-service
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:apollo_190119_1815
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com
        - name: JAR_BALL
          value: dubbo-server.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

注意：增加了env段配置
注意：docker镜像新版的tag

应用资源配置清单

在任意一台k8s运算节点上执行：

1
2
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/deployment.yaml
deployment.extensions/dubbo-demo-service configured

观察项目运行情况

http://dubbo-monitor.od.com

改造dubbo-demo-web

略

配置apollo-portal

创建项目

  • 部门

    样例部门1（TEST1）

  • 应用id

    dubbo-demo-web

  • 应用名称

    dubbo服务消费者

  • 应用负责人

    apollo|apollo

  • 项目管理员

    apollo|apollo

提交

进入配置页面

新增配置项1

  • Key

    dubbo.registry

  • Value

    zookeeper://zk1.od.com:2181

  • 选择集群

    DEV

提交

发布配置

点击发布，配置生效

使用jenkins进行CI

略（注意记录镜像的tag）

上线新构建的项目

准备资源配置清单

运维主机HDSS7-200.host.com上：

/data/k8s-yaml/dubbo-demo-consumer/deployment.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: app
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:apllo_190120_1815
        ports:
        - containerPort: 20880
          protocol: TCP
        - containerPort: 8080
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

注意：增加了env段配置
注意：docker镜像新版的tag

应用资源配置清单

在任意一台k8s运算节点上执行：

1
2
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-web/deployment.yaml
deployment.extensions/dubbo-demo-consumer configured

通过Apollo配置中心动态维护项目的配置

以dubbo-demo-service项目为例，不用修改代码

实战维护多套dubbo微服务环境

生产实践

  1. 迭代新需求/修复BUG（编码->提GIT）
  2. 测试环境发版，测试（应用通过编译打包发布至TEST命名空间）
  3. 测试通过，上线（应用镜像直接发布至PROD命名空间）

系统架构

  • 物理架构
主机名 角色 ip
HDSS7-11.host.com zk-test(测试环境Test) 10.4.7.11
HDSS7-12.host.com zk-prod(生产环境Prod) 10.4.7.12
HDSS7-21.host.com kubernetes运算节点 10.4.7.21
HDSS7-22.host.com kubernetes运算节点 10.4.7.22
HDSS7-200.host.com 运维主机，harbor仓库 10.4.7.200
  • K8S内系统架构
环境 命名空间 应用
测试环境（TEST） test apollo-config，apollo-admin
测试环境（TEST） test dubbo-demo-service，dubbo-demo-web
生产环境（PROD） prod apollo-config，apollo-admin
生产环境（PROD） prod dubbo-demo-service，dubbo-demo-web
ops环境（infra） infra jenkins，dubbo-monitor，apollo-portal

修改/添加域名解析

DNS主机HDSS7-11.host.com上：

/var/named/od.com.zone
1
2
3
4
5
6
zk-test 60 IN A 10.4.7.11
zk-prod 60 IN A 10.4.7.12
config-test	60 IN A 10.4.7.10
config-prod	60 IN A 10.4.7.10
demo-test 60 IN A 10.4.7.10
demo-prod 60 IN A 10.4.7.10

Apollo的k8s应用配置

  • 删除app命名空间内应用，创建test命名空间，创建prod命名空间
  • 删除infra命名空间内apollo-configservice，apollo-adminservice应用
  • 数据库内删除ApolloConfigDB，创建ApolloConfigTestDB，创建ApolloConfigProdDB
1
2
3
4
5
6
7
8
9
10
11
12
13
mysql> drop database ApolloConfigDB;

mysql> create database ApolloConfigTestDB;
mysql> use ApolloConfigTestDB;
mysql> source ./apolloconfig.sql
mysql> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

mysql> create database ApolloConfigProdDB;
mysql> use ApolloConfigProdDB;
mysql> source ./apolloconfig.sql
mysql> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
  • 准备apollo-config，apollo-admin的资源配置清单（各2套）

注：apollo-config/apollo-admin的configmap配置要点

  • Test环境
1
2
3
4
5
6
application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config-test.od.com/eureka
  • Prod环境
1
2
3
4
5
6
application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config-prod.od.com/eureka
  • 依次应用，分别发布在test和prod命名空间
  • 修改apollo-portal的configmap并重启portal
1
2
3
apollo-env.properties: |
    TEST.meta=http://config-test.od.com
    PROD.meta=http://config-prod.od.com

Apollo的portal配置

管理员工具

删除应用、集群、AppNamespace，将已配置应用删除

系统参数

  • Key

    apollo.portal.envs

  • Value

    TEST,PROD

查询

  • Value

    TEST,PROD

保存

新建dubbo-demo-service和dubbo-demo-web项目

在TEST/PROD环境分别增加配置项并发布

发布dubbo微服务

  • 准备dubbo-demo-service和dubbo-demo-web的资源配置清单（各2套）
  • 依次应用，分别发布至app-test和app-prod命名空间
  • 使用dubbo-monitor查验

互联网公司技术部的日常

  • 产品经理整理需求，需求评审，出产品原型
  • 开发同学夜以继日的开发，提测
  • 测试同学使用Jenkins持续集成，并发布至测试环境
  • 验证功能，通过->待上线or打回->修改代码
  • 提交发版申请，运维同学将测试后的包发往生产环境
  • 无尽的BUG修复（笑cry）

发表于 | 更新于 | 分类于
本文字数： 3.1k | 阅读时长 ≈ 3 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

安装部署BIND9

操作系统版本和内核版本

1
2
3
4
5
#cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

#uname -a
Linux node 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

使用yum安装BIND9

1
2
3
4
5
6
#yum install bind
=============================================================================================================================================================
 Package                                       Arch                          Version                                    Repository                      Size
=============================================================================================================================================================
Installing:
 bind                                          x86_64                        32:9.9.4-73.el7_6                          updates                        1.8 M

安装的版本为9.9.4

BIND9主配置文件/etc/named.conf

  1. 主配置文件的格式

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    options{
         //全局选项
    }
    zone　"zone name" {
       //定于区域
    }
    logging{
        //日志文件
    }
    include：加载别的文件

  2. 主配置文件的配置注意事项

    • 语法严格，分号，空格
    • 文件的权限，属主：root，属组：named，640

  3. 主配置文件范例

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    options {
    	listen-on port 53 { 10.4.7.11; };
    	directory 	"/var/named";
    	dump-file 	"/var/named/data/cache_dump.db";
    	statistics-file "/var/named/data/named_stats.txt";
    	memstatistics-file "/var/named/data/named_mem_stats.txt";
    	allow-query     { any; };

    	/* 
    	 - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
    	 - If you are building a RECURSIVE (caching) DNS server, you need to enable 
    	   recursion. 
    	 - If your recursive DNS server has a public IP address, you MUST enable access 
    	   control to limit queries to your legitimate users. Failing to do so will
    	   cause your server to become part of large scale DNS amplification 
    	   attacks. Implementing BCP38 within your network would greatly
    	   reduce such attack surface 
    	*/
    	recursion yes;
    	
    		dnssec-enable no;
    		dnssec-validation no;

    	/* Path to ISC DLV key */
    	bindkeys-file "/etc/named.iscdlv.key";

    	managed-keys-directory "/var/named/dynamic";

    	pid-file "/run/named/named.pid";
    	session-keyfile "/run/named/session.key";
    };


    logging {
            channel default_debug {
                    file "data/named.run";
                    severity dynamic;
            };
    };

    zone "." IN {
    	type hint;
    	file "named.ca";
    };

    include "/etc/named.rfc1912.zones";
    include "/etc/named.root.key";

BIND9服务的启动

检查配置文件

1
# named-checkconf

没有报错就是正常的

启动BIND9服务

1
# systemctl start named

检查BIND9服务状态

1
# systemctl status named

这样就完成了一个最基本的转发DNS的部署，它可以为我们的内网客户端提供DNS递归查询，例如查询并返回www.baidu.com的解析结果。

验证解析

配置DNS服务器指向

在/etc/resole里配置DNS服务器的ip地址为我们部署的主机ip

1
2
3
# cat /etc/resolv.conf    
# Generated by NetworkManager
nameserver 10.4.7.11

验证解析

1
2
# ping baidu.com
PING baidu.com (220.181.57.216) 56(84) bytes of data.

发表于 | 更新于 | 分类于
本文字数： 4.9k | 阅读时长 ≈ 4 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

自定义区域配置文件

自定义区域的配置范例如下：

1
2
3
4
5
zone "host.com" IN {
        type  master;
        file  "host.com.zone";
        allow-update { 10.4.7.11;10.4.7.12; };
};

这里自定义了一个host.com的主机域，可以放在/etc/named.rfc1912.zones文件中，也可以放置在自定义的文件中，在/etc/named.conf里include进来

主机域

  • 主机域和业务域无关，且建议分开
  • 主机域其实是一个假域，也就是说，主机域其实是不能解析到互联网上的，它只对局域网（内网）提供服务

自定义区域数据库文件

  • 一般而言是文本文件，且只包含资源记录、宏定义和注释
  • 需在自定义区域配置文件中指定存放路径，可以绝对路径或相对路径（相对于/var/named目录）
  • 注意文件的属性（属主、属组及权限）

配置范例

1
2
3
4
5
6
7
8
9
10
11
12
13
$ORIGIN .
$TTL 600	; 10 minutes
host.com			IN SOA	ns1.host.com. dnsadmin.host.com. (
				2018121601 ; serial
				10800      ; refresh (3 hours)
				900        ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
				NS   ns1.host.com.
$ORIGIN host.com.
$TTL 60	; 1 minute
ns1                             A    10.4.7.11

资源记录（Resource Record）

资源记录格式

1
name [ttl(缓存时间)] IN 资源记录类型（RRtype）  Value

常用资源记录类型（RR-type）

SOA记录

SOA: 起始授权，只能有一条

  • name:只能是区域名称，通常可以简写为@，例如：od.com.
  • value:有n个数值，最主要的是主DNS服务器的FQDN，点不可省略

注意：SOA必须是区域数据库文件第一条记录

例子：

1
2
3
4
5
6
7
@ 600 IN SOA  dns.host.com. 管理员邮箱（dnsadmin.host.com.）（
     序列号(serial number) ;注释内容，十进制数据，不能超过10位，通常使用日期时间戳，例如2018121601
     刷新时间(refresh time) ;即每隔多久到主服务器检查一次
     重试时间(retry time) ;应该小于refresh time
     过期时间(expire time);当辅助DNS服务器无法联系上主DNS服务器时，辅助DNS服务器可以在多长时间内认为其缓存是有效的，并供用户查询。
     netgative answer ttl ;非权威应答的ttl，缓存DNS服务器可以缓存记录多长时间
 ）

NS记录

NS：可以有多条，每一个NS记录，必须对应一个A记录

  • name:区域名称，通常可以简写为@
  • value:DNS服务器的FQDN(可以使用相对名称)

    例子：

    1
    @ 600 IN NS ns1

A记录

A：只能定义在正向区域数据库文件中（ipv4->FQDN）

  • name:FQDN（可以使用相对名称)
  • value:IP

    例子：

    1
    2
    www  600(单位s) IN A 10.4.7.11
    www  600(单位s) IN A 10.4.7.12

注 可以做轮询

MX记录

MX:邮件交换记录，可以有多个（用的不多）

  • name:区域名称，用于标识smtp服务器
  • value:包含优先级和FQDN
  • 优先级:0-99，数字越小，级别越高，

    例子：

    1
    2
    @ 600 IN MX 10 mail
    @ 600 IN MX 20 smtp

CNAME记录

CNAME：canonical name，别名（FQDN->FQDN）

  • name ：FQDN
  • value :FQDN

    例子：

    1
    eshop IN CNAME www

宏定义

  • $ORIGIN .
  • $TTL 60

注释

区域数据库文件中使用;（分号）来进行注释

实战正解主机域配置

在/etc/named.rfc1912.zones文件内最下，添加以下内容

1
2
3
4
5
zone "host.com" IN {
        type  master;
        file  "host.com.zone";
        allow-update { 10.4.7.11;10.4.7.12; };
};

在/var/named下创建host.com.zone文件，写入以下内容

/var/named/host.com.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
$TTL 600	; 10 minutes
@       		IN SOA	dns.host.com. 87527941.qq.com. (
				2018121601 ; serial
				10800      ; refresh (3 hours)
				900        ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			NS   dns.host.com.
$ORIGIN host.com.
$TTL 60	; 1 minute
HDSS7-11 	        A    10.4.7.11
dns      	        A    10.4.7.11

三种配置方式：

  • 用宏定义$ORIGIN . 下面用host.com
  • 不用宏定义，下面用@
  • 不用宏定义，下面用host.com.

检查配置并生效

检查自定义区域配置

1
2
3
#named-checkzone host.com. /var/named/host.com.zone
zone host.com/IN: loaded serial 2018121601
OK

检查主配置文件

1
#named-checkconf

重启named服务

1
#systemctl restart named

检查该正解域是否生效

配置主机名

1
2
# hostnamectl set-hostname hdss7-11.host.com
# logout

开启第二台虚机，配置好DNS后验证解析

维护正解域（增、删、改、查）

增加一条资源记录

/var/named/host.com.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
14
$TTL 600	; 10 minutes
@       		IN SOA	dns.host.com. 87527941.qq.com. (
				2018121602 ; serial
				10800      ; refresh (3 hours)
				900        ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			NS   dns.host.com.
$ORIGIN host.com.
$TTL 60	; 1 minute
HDSS7-11 	        A    10.4.7.11
HDSS7-12 	        A    10.4.7.12
dns      	        A    10.4.7.11

增加一个HDSS7-12.host.com的A记录解析，并验证

修改一条资源记录

给10.4.7.12这台主机增加一个辅助ip

1
# ip addr add 10.4.7.13/24 dev eth0

修改DNS服务器上的区域数据库文件

/var/named/host.com.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
14
$TTL 600	; 10 minutes
@       		IN SOA	dns.host.com. 87527941.qq.com. (
				2018121603 ; serial
				10800      ; refresh (3 hours)
				900        ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			NS   dns.host.com.
$ORIGIN host.com.
$TTL 60	; 1 minute
HDSS7-11 	        A    10.4.7.11
HDSS7-12 	        A    10.4.7.13
dns      	        A    10.4.7.11

修改HDSS7-12.host.com的A记录解析，指向新增的辅助ip10.4.7.13
检查：

1
2
3
4
5
6
7
8
9
ping HDSS7-12.host.com
ING hdss7-12.host.com (10.4.7.13) 56(84) bytes of data.
64 bytes from 10.4.7.13 (10.4.7.13): icmp_seq=1 ttl=64 time=0.318 ms
64 bytes from 10.4.7.13 (10.4.7.13): icmp_seq=2 ttl=64 time=0.206 ms
64 bytes from 10.4.7.13 (10.4.7.13): icmp_seq=3 ttl=64 time=0.391 ms
^C
--- hdss7-12.host.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.206/0.305/0.391/0.076 ms

删除一条资源记录

/var/named/host.com.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
$TTL 600	; 10 minutes
@       		IN SOA	ns1.host.com. dnsadmin.host.com. (
				2018121604 ; serial
				10800      ; refresh (3 hours)
				900        ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			NS   ns1.host.com.
$ORIGIN host.com.
$TTL 60	; 1 minute
ns1      	        A    10.4.7.11
HDSS7-11 	        A    10.4.7.11

删除HDSS7-12.host.com的A记录解析，并验证

查询记录

略

发表于 | 更新于 | 分类于
本文字数： 2.4k | 阅读时长 ≈ 2 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

添加反解域的自定义区域配置

/etc/named.rfc1912.zones
1
2
3
4
5
zone "7.4.10.in-addr.arpa" IN {
        type master;
        file "7.4.10.in-addr.arpa.zone";
        allow-update { 10.4.7.11;10.4.7.12; };
};

添加反解域的区域数据库文件

/var/named/7.4.10.in-addr.arpa.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
$TTL 600	; 10 minutes
@	     		IN SOA	dns.host.com. dnsadmin.host.com. (
				2018121603 ; serial
				10800      ; refresh (3 hours)
				900       ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			    NS   ns1.host.com.
$ORIGIN 7.4.10.in-addr.arpa.
$TTL 60	; 1 minute
11			PTR		 HDSS7-11.host.com.
12			PTR		 HDSS7-12.host.com.

注意：一个IP只能对应唯一的FQDN反解PTR记录，且应该与正解A记录对应

检查反解域的配置

1
2
3
[root@hdss7-11 ~]# named-checkzone 7.4.10.in-addr.arpa /var/named/7.4.10.in-addr.arpa.zone
zone 7.4.10.in-addr.arpa/IN: loaded serial 2018121603
OK

重启BIND9服务

1
[root@hdss7-11 ~]# systemctl restart named.service

检查解析是否生效

  • 方法1：

    1
    2
    [root@hdss7-11 ~]# dig -t PTR 11.7.4.10.in-addr.arpa. @10.4.7.11 +short
    HDSS7-11.host.com.

  • 方法2：

    1
    2
    [root@hdss7-11 ~]# dig -x 10.4.7.11 @10.4.7.11 +short
    HDSS7-11.host.com.

增删改

增加一条反解记录

/var/named/7.4.10.in-addr.arpa.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
14
$TTL 600	; 10 minutes
@	     		IN SOA	dns.host.com. dnsadmin.host.com. (
				2018121604 ; serial
				10800      ; refresh (3 hours)
				900       ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			    NS   ns1.host.com.
$ORIGIN 7.4.10.in-addr.arpa.
$TTL 60	; 1 minute
11			PTR		 HDSS7-11.host.com.
12			PTR		 HDSS7-12.host.com.
13			PTR		 HDSS7-13.host.com.

删除一条反解记录

/var/named/7.4.10.in-addr.arpa.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
$TTL 600	; 10 minutes
@	     		IN SOA	dns.host.com. dnsadmin.host.com. (
				2018121605 ; serial
				10800      ; refresh (3 hours)
				900       ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			    NS   ns1.host.com.
$ORIGIN 7.4.10.in-addr.arpa.
$TTL 60	; 1 minute
11			PTR		 HDSS7-11.host.com.
12			PTR		 HDSS7-12.host.com.

修改一条反解记录

/var/named/7.4.10.in-addr.arpa.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
$TTL 600	; 10 minutes
@	     		IN SOA	dns.host.com. dnsadmin.host.com. (
				2018121606 ; serial
				10800      ; refresh (3 hours)
				900       ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			    NS   ns1.host.com.
$ORIGIN 7.4.10.in-addr.arpa.
$TTL 60	; 1 minute
11			PTR		 HDSS7-11.host.com.
12			PTR		 HDSS7-13.host.com.

发表于 | 更新于 | 分类于
本文字数： 4.8k | 阅读时长 ≈ 4 分钟

欢迎加入王导的VIP学习qq群：==>932194668<==

DNS主辅同步架构

IP 主机名 功能
10.4.7.11 HDSS7-11.host.com DNS主
10.4.7.12 HDSS7-12.host.com DNS辅
注意：所有资源记录的增、删、改的操作，均在主DNS上进行，辅助DNS仅提供查询功能

辅助DNS主机上安装部署BIND9

安装BIND9软件

1
2
3
4
5
6
7

#yum install bind
=============================================================================================================================================================
 Package                                       Arch                          Version                                    Repository                      Size
=============================================================================================================================================================
Installing:
 bind                                          x86_64                        32:9.9.4-73.el7_6                          updates                        1.8 M

注意：辅助DNS的BIND9软件版本应小于等于主DNS的BIND9软件版本

修改辅助DNS主配置文件

修改主配置文件，并加入masterfile-format text;

/etc/named.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
options {
  listen-on port 53 { 10.4.7.12; };
  directory 	"/var/named";
  dump-file 	"/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  memstatistics-file "/var/named/data/named_mem_stats.txt";
  allow-query     { any; };
  masterfile-format text;
  /* 
   - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
   - If you are building a RECURSIVE (caching) DNS server, you need to enable 
     recursion. 
   - If your recursive DNS server has a public IP address, you MUST enable access 
     control to limit queries to your legitimate users. Failing to do so will
     cause your server to become part of large scale DNS amplification 
     attacks. Implementing BCP38 within your network would greatly
     reduce such attack surface 
  */
  recursion yes;

  dnssec-enable no;
  dnssec-validation no;

  /* Path to ISC DLV key */
  bindkeys-file "/etc/named.iscdlv.key";

  managed-keys-directory "/var/named/dynamic";

  pid-file "/run/named/named.pid";
  session-keyfile "/run/named/session.key";
};


logging {
  channel default_debug {
    file "data/named.run";
    severity dynamic;
  };
};

zone "." IN {
  type hint;
  file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

修改主DNS主配置文件

加入以下配置

/etc/named.conf
1
2
allow-transfer { 10.4.7.12; };
also-notify { 10.4.7.12; };

检查配置并重启主DNS

1
2
# named-checkconf
# systemctl restart named

检查完全区域数据传送

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@hdss7-12 ~]# dig -t axfr host.com @10.4.7.11

; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> -t axfr host.com @10.4.7.11
;; global options: +cmd
host.com.		600	IN	SOA	dns.host.com. dnsadmin.host.com. 2018121601 10800 900 604800 86400
host.com.		600	IN	NS	ns1.host.com.
HDSS7-11.host.com.	60	IN	A	10.4.7.11
HDSS7-12.host.com.	60	IN	A	10.4.7.12
ns1.host.com.		60	IN	A	10.4.7.11
host.com.		600	IN	SOA	dns.host.com. dnsadmin.host.com. 2018121601 10800 900 604800 86400
;; Query time: 0 msec
;; SERVER: 10.4.7.11#53(10.4.7.11)
;; WHEN: Sun Dec 16 14:16:01 CST 2018
;; XFR size: 6 records (messages 1, bytes 220)

辅助DNS上创建自定义正解区域配置

/etc/named.rfc1912.zones
1
2
3
4
5
zone "host.com" IN {
  type  slave;
  masters { 10.4.7.11; };
  file  "slaves/host.com.zone";
};

检查配置并启动辅助DNS

1
2
# named-checkconf
# systemctl start named

检查同步过来的区域数据库文件

/var/named/slaves/host.com.zone
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@hdss7-12 slaves]# ll
-rw-r--r-- 1 named named 392 Feb 10 21:08 host.com.zone
[root@hdss7-12 slaves]# cat host.com.zone 
$ORIGIN .
$TTL 600	; 10 minutes
host.com		IN SOA	dns.host.com. dnsadmin.host.com. (
				2018121601 ; serial
				10800      ; refresh (3 hours)
				900        ; retry (15 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			NS	ns1.host.com.
$ORIGIN host.com.
$TTL 60	; 1 minute
HDSS7-11		A	10.4.7.11
HDSS7-12		A	10.4.7.12
ns1			A	10.4.7.11

检查解析是否正确

使用主DNS查询一个A记录

1
2
# dig -t A HDSS7-11.host.com @10.4.7.11 +short
10.4.7.11

使用辅助DNS查询一个A记录

1
2
# dig -t A HDSS7-11.host.com @10.4.7.12 +short
10.4.7.11

辅助DNS上创建自定义反解区域配置

略

增加、删除、修改记录，并验证同步

注意：一定要手动修改主DNS上SOA记录里的serial值！

增加记录

删除记录

修改记录

再增加一个od.com的业务域，并验证主辅同步（复习）

主DNS上增加自定义区域

主DNS上增加自定义区域数据库文件

主DNS上增加自定义区域资源记录

检查配置并重启主DNS服务

辅助DNS上增加自定义区域

检查完全区域数据传送

检查配置并重启辅助DNS服务

验证主辅同步

分别使用主DNS和辅助DNS查询新业务域的A记录

在主DNS上新增一条A记录，并验证主辅同步

在主DNS上修改一条A记录，并验证主辅同步

在主DNS上删除一条A记录，并验证主辅同步

客户端配置DNS解析高可用

在客户端主机（以Linux主机为例，Windows和Mac操作系统略）配置主、辅DNS

/etc/resolv.conf
1
2
3
4
5
#cat /etc/resolv.conf 
# Generated by NetworkManager
search host.com od.com
nameserver 10.4.7.11
nameserver 10.4.7.12

这样客户端高可用就配置好了，任意一个DNS服务器宕机也不会影响正常解析