Kolla-ansible 源码分析，kolla-ansible源码
Kolla-ansible 源码分析，kolla-ansible源码
Kolla-ansible 源码分析
简介
Kolla-ansible项目提供一个完整的Ansible Playbook，来部署Docker的镜像，再完成openstack组件的自动化部署。并提供all-in-one和multihost的环境。
源码地址：https://github.com/openstack/kolla-ansible.git
源码目录概要
一级目录
Paste_Image.png
- Ansible: ansible的整个playbook代码,包括部署docker容器和openstack组件。源码主要集中在这个目录下。
- Contrib:包括用heat和magnum的部署环境和vagrant的部署环境。
- Deploy-guide: 部署指南，主要包括all-in-one和mulihosts两种部署方式的指南。
- Doc:文档。
- Etc: 一些配置文件，安装完了引用到了/etc目录下，all-in-one只要修改很少量的配置。
- Kolla-ansible: 记录版本信息，cmd子目录下有生成密码和合并密码的两个脚本。pbr打包的时候被封装成为可执行命令。
- Releasenodes: 发布特性说明。
- Specs: 包含有Kolla社区关键参数代码库的变化。
- Tests: 包括一些功能测试工具，这里还包括两个自定义的ansible plugin（merge_config）和module（kolla_docker）的测试。
- Tools: 一些和kolla交换的脚本工具，大部分是可手动调用，主要完成一些安装前后的一些操作。有些会被ansible目录下的task调用到。
二级目录
- Ansible/action_plugins: 自定义ansible插件，两个脚本，用于合并yml和conifg的配置文件。
- Ansible/group_vars: ansible脚本的全局变量定义。
- Ansible/inventory: 包含all-in-one和mulitnode的样板hosts清单。
- Ansible/library: 包括一些自定义的ansible模块，bslurp.py和kolla_docker.py用到比较多。
- Ansible/role: 所有的openstack的组件，几乎包含了说有开源项目，当前版本有60个组件。
- Ansible:除了文件夹之外的ansible脚本，主要用户安装前后的环境准备和清理，数据库恢复等特殊系统级的操作。
关键代码解读
Setup.cfg安装配置入口文件， 见中文注释
[metadata]
name = kolla-ansible // 项目名称
summary = Ansible Deployment of Kolla containers
description-file = README.rst
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = http://docs.openstack.org/developer/kolla-ansible/
license = Apache License, Version 2.0
classifier =
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.5
[files]
packages = kolla_ansible //包名
data_files = //pbr方式打包对应的文件映射
share/kolla-ansible/ansible = ansible/*
share/kolla-ansible/tools = tools/validate-docker-execute.sh
share/kolla-ansible/tools = tools/cleanup-containers
share/kolla-ansible/tools = tools/cleanup-host
share/kolla-ansible/tools = tools/cleanup-images
share/kolla-ansible/tools = tools/stop-containers
share/kolla-ansible/doc = doc/*
share/kolla-ansible/etc_examples = etc/*
share/kolla-ansible = tools/init-runonce
share/kolla-ansible = tools/init-vpn
share/kolla-ansible = tools/openrc-example
share/kolla-ansible = setup.cfg
scripts = //可执行脚本
tools/kolla-ansible
[entry_points]
console_scripts = //控制台可执行脚本，执行两个Python文件的main函数
kolla-genpwd = kolla_ansible.cmd.genpwd:main
kolla-mergepwd = kolla_ansible.cmd.mergepwd:main
[global]
setup-hooks =
pbr.hooks.setup_hook
[pbr] //打包方式
[build_sphinx]
all_files = 1
build-dir = doc/build
source-dir = doc
[build_releasenotes]
all_files = 1
build-dir = releasenotes/build
source-dir = releasenotes/source
Setup.py
安装执行脚本，通过pbr打包，执行过程会读取setup.cfg配置，还会安装同父目录下requirements.txt中的依赖。更多参考https://julien.danjou.info/blog/2017/packaging-python-with-pbr
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr>=2.0.0'],
pbr=True)
tools\kolla-ansible
该脚本是封装了ansible-playbook，对kolla进行了ansible的定制。主要根据action的类型，传递不同的配置文件。
中间基础变量定义：
find_base_dir
INVENTORY="${BASEDIR}/ansible/inventory/all-in-one"
PLAYBOOK="${BASEDIR}/ansible/site.yml"
VERBOSITY=
EXTRA_OPTS=${EXTRA_OPTS}
CONFIG_DIR="/etc/kolla"
PASSWORDS_FILE="${CONFIG_DIR}/passwords.yml"
DANGER_CONFIRM=
INCLUDE_IMAGES=
Find_base_dir是一个脚本开始时候的一个函数（不展开解释），用于找到kolla-ansible脚本所在的路径。
脚本传参解释：
while [ "$#" -gt 0 ]; do
case "$1" in
(--inventory|-i)
INVENTORY="$2"
shift 2
;;
(--playbook|-p)
PLAYBOOK="$2"
shift 2
;;
(--tags|-t)
EXTRA_OPTS="$EXTRA_OPTS --tags $2"
shift 2
;;
(--verbose|-v)
VERBOSITY="$VERBOSITY --verbose"
shift 1
;;
(--configdir)
CONFIG_DIR="$2"
shift 2
;;
(--yes-i-really-really-mean-it)
DANGER_CONFIRM="$1"
shift 1
;;
(--include-images)
INCLUDE_IMAGES="$1"
shift 1
;;
(--key|-k)
VAULT_PASS_FILE="$2"
EXTRA_OPTS="$EXTRA_OPTS --vault-password-file=$VAULT_PASS_FILE"
shift 2
;;
(--extra|-e)
EXTRA_OPTS="$EXTRA_OPTS -e $2"
shift 2
;;
(--passwords)
PASSWORDS_FILE="$2"
shift 2
;;
(--help|-h)
usage
shift
exit 0
;;
(--)
shift
break
;;
(*)
echo "error"
exit 3
;;
esac
done
case "$1" in
(prechecks)
ACTION="Pre-deployment checking"
EXTRA_OPTS="$EXTRA_OPTS -e action=precheck"
;;
(check)
ACTION="Post-deployment checking"
EXTRA_OPTS="$EXTRA_OPTS -e action=check"
;;
(mariadb_recovery)
ACTION="Attempting to restart mariadb cluster"
EXTRA_OPTS="$EXTRA_OPTS -e action=deploy -e common_run=true"
PLAYBOOK="${BASEDIR}/ansible/mariadb_recovery.yml"
;;
(destroy)
ACTION="Destroy Kolla containers, volumes and host configuration"
PLAYBOOK="${BASEDIR}/ansible/destroy.yml"
if [[ "${INCLUDE_IMAGES}" == "--include-images" ]]; then
EXTRA_OPTS="$EXTRA_OPTS -e destroy_include_images=yes"
fi
if [[ "${DANGER_CONFIRM}" != "--yes-i-really-really-mean-it" ]]; then
cat << EOF
WARNING:
This will PERMANENTLY DESTROY all deployed kolla containers, volumes and host configuration.
There is no way to recover from this action. To confirm, please add the following option:
--yes-i-really-really-mean-it
EOF
exit 1
fi
;;
(bootstrap-servers)
ACTION="Bootstraping servers"
PLAYBOOK="${BASEDIR}/ansible/kolla-host.yml"
EXTRA_OPTS="$EXTRA_OPTS -e action=bootstrap-servers"
;;
(deploy)
ACTION="Deploying Playbooks"
EXTRA_OPTS="$EXTRA_OPTS -e action=deploy"
;;
(deploy-bifrost)
ACTION="Deploying Bifrost"
PLAYBOOK="${BASEDIR}/ansible/bifrost.yml"
EXTRA_OPTS="$EXTRA_OPTS -e action=deploy"
;;
(deploy-servers)
ACTION="Deploying servers with bifrost"
PLAYBOOK="${BASEDIR}/ansible/bifrost.yml"
EXTRA_OPTS="$EXTRA_OPTS -e action=deploy-servers"
;;
(post-deploy)
ACTION="Post-Deploying Playbooks"
PLAYBOOK="${BASEDIR}/ansible/post-deploy.yml"
;;
(pull)
ACTION="Pulling Docker images"
EXTRA_OPTS="$EXTRA_OPTS -e action=pull"
;;
(upgrade)
ACTION="Upgrading OpenStack Environment"
EXTRA_OPTS="$EXTRA_OPTS -e action=upgrade -e serial=${ANSIBLE_SERIAL}"
;;
(reconfigure)
ACTION="Reconfigure OpenStack service"
EXTRA_OPTS="$EXTRA_OPTS -e action=reconfigure -e serial=${ANSIBLE_SERIAL}"
;;
(stop)
ACTION="Stop Kolla containers"
PLAYBOOK="${BASEDIR}/ansible/stop.yml"
;;
(certificates)
ACTION="Generate TLS Certificates"
PLAYBOOK="${BASEDIR}/ansible/certificates.yml"
;;
(genconfig)
ACTION="Generate configuration files for enabled OpenStack services"
EXTRA_OPTS="$EXTRA_OPTS -e action=config"
;;
(*) usage
exit 0
;;
esac
这段是根据传递的参数，不同的参数针对不同的配置文件等额外属性。这里第一个参数有好多action，如deploy，post-deploy，stop等等。
最后三行，组合命令，执行：
CONFIG_OPTS="-e @${CONFIG_DIR}/globals.yml -e @${PASSWORDS_FILE} -e CONFIG_DIR=${CONFIG_DIR}"
CMD="ansible-playbook -i $INVENTORY $CONFIG_OPTS $EXTRA_OPTS $PLAYBOOK $VERBOSITY"
process_cmd
传递进来的参数组合成ansible-playbook的CMD命令后，调用process_cmd函数执行。
- 例子1：初始命令
kolla-ansible deploy -i /home/all-in-one
封装后命令
ansible-playbook -i /home/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /usr/share/kolla-ansible/ansible/site.yml - 例子2：初始命令
kolla-ansible post-deploy
封装后命令
ansible-playbook -i /usr/share/kolla-ansible/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla /usr/share/kolla-ansible/ansible/post-deploy.yml
ansible剧本代码解
由于openstack的组件比较多，且大多数处于并列关系。这里不再一一展开，以neutron为例进行解读。其他没有涉及的有重要的点，会进行内容穿插。
Ansible/library/
该目录下是一些自定义的一些模块，这些module在目标节点上运行，包括bslurp.py，kolla_container_facts.py，kolla_docker.py，kolla-toolbox.py，merge_configs.py，merge_yaml.py，前四个我会一一介绍，后两个是空文件，代码其实在action_plugins目录中（那两个空文件是映射同名的action,我们可以向module一样使用action）。
Bslurp.py，好像做了文件分发的事情，通过copy_from_host（我没查到调用它的地方）和copy_to_host（在ceph角色部署的时候，下发keyring到osd节点是通过这个模块函数下发的）两个函数实现，。判断依据是模块参数dest。见代码中文注释。
def copy_from_host(module):
#此处省略不少代码
module.exit_json(content=base Ａ64.b64encode(data), sha1=sha1, mode=mode,
source=src)
def copy_to_host(module):
compress = module.params.get('compress')
dest = module.params.get('dest')
mode = int(module.params.get('mode'), 0)
sha1 = module.params.get('sha1')
# src是加密后的数据
src = module.params.get('src')
# decode已经加密的数据
data = base64.b64decode(src)
#解压数据
raw_data = zlib.decompress(data) if compress else data
#sha1安全算法数据校验
if sha1:
if os.path.exists(dest):
if os.access(dest, os.R_OK):
with open(dest, 'rb') as f:
if hashlib.sha1(f.read()).hexdigest() == sha1:
module.exit_json(changed=False)
else:
module.exit_json(failed=True, changed=False,
msg='file is not accessible: {}'.format(dest))
if sha1 != hashlib.sha1(raw_data).hexdigest():
module.exit_json(failed=True, changed=False,
msg='sha1 sum does not match data')
# 保存数据到dest值。这段代码有健壮性问题，没有考虑到磁盘写满的场景，保险的做法创建一个tmp文件，把数据拷贝到tmp文件，再把tmp文件重命名为dest值。否则容易把文件写空。
with os.fdopen(os.open(dest, os.O_WRONLY | os.O_CREAT, mode), 'wb') as f:
f.write(raw_data)
#调用module要求的exit_json接口退出。
module.exit_json(changed=True)
def main():
# 定义dict类型的参数，ansible module的接口要求
argument_spec = dict(
compress=dict(default=True, type='bool'),
dest=dict(type='str'),
mode=dict(default='0644', type='str'),
sha1=dict(default=None, type='str'),
src=dict(required=True, type='str')
)
# 创建ansible模块对象
module = AnsibleModule(argument_spec)
# 获取模块dest参数值
dest = module.params.get('dest')
try:
if dest:
# 如果dest参数存在，则推送操作，push下发到相应的host
copy_to_host(module)
else:
# 如果dest参数不存在，则进行pull操作。
copy_from_host(module)
except Exception:
# 异常场景下退出，ansible自定义模块语法规范。
module.exit_json(failed=True, changed=True,
msg=repr(traceback.format_exc()))
# import module snippets
from ansible.module_utils.basic import * # noqa
if __name__ == '__main__':
main()
kolla_docker.py，容器相关的操作，openstack的组件都通过容器部署，每个组件role的部署都会用到，非常重要。
...
#创建docker的client函数
def get_docker_client():
try:
return docker.Client
except AttributeError:
return docker.APIClient
A
class DockerWorker(object):
def __init__(self, module):
# 构造函数，传入参数是AnsibleModule类型的对象
self.module = module
# params参数续传
self.params = self.module.params
self.changed = False
# TLS not fully implemented
# tls_config = self.generate_tls()
# 创建一个docker.client对象
options = {
'version': self.params.get('api_version')
}
self.dc = get_docker_client()(**options)
# ....
# ....
# 启动容器的函数，是AnsibleModule的其中一种action
def start_container(self):
#检查镜像是否存在，不存在pull
if not self.check_image():
self.pull_image()
#检查容器
container = self.check_container()
#容器异样，则删除，再回调
if container and self.check_container_differs():
self.stop_container()
self.remove_container()
container = self.check_container()
#容器不存在，创建，再回调
if not container:
self.create_container()
container = self.check_container()
#容器状态非启动，则启动
if not container['Status'].startswith('Up '):
self.changed = True
self.dc.start(container=self.params.get('name'))
# We do not want to detach so we wait around for container to exit
#如果container没有detach断开，那么进入wait状态,调用fail_json方法，传递fail的参数
if not self.params.get('detach'):
rc = self.dc.wait(self.params.get('name'))
if rc != 0:
self.module.fail_json(
failed=True,
changed=True,
msg="Container exited with non-zero return code"
)
#如果返回参数remove_on_exit，那么删除该container
if self.params.get('remove_on_exit'):
self.stop_container()
self.remove_container()
def generate_module():
# NOTE(jeffrey4l): add empty string '' to choices let us use
# pid_mode: "{{ service.pid_mode | default ('') }}" in yaml
#定义参数字典，ansible module的api规范
argument_spec = dict(
common_options=dict(required=False, type='dict', default=dict()),
#action参数，必须传递，类型为str,value值必须在choices的列表
action=dict(required=True, type='str',
choices=['compare_container', 'compare_image',
'create_volume', 'get_container_env',
'get_container_state', 'pull_image',
'recreate_or_restart_container',
'remove_container', 'remove_volume',
'restart_container', 'start_container',
'stop_container']),
api_version=dict(required=False, type='str', default='auto'),
auth_email=dict(required=False, type='str'),
auth_password=dict(required=False, type='str'),
auth_registry=dict(required=False, type='str'),
auth_username=dict(required=False, type='str'),
detach=dict(required=False, type='bool', default=True),
labels=dict(required=False, type='dict', default=dict()),
name=dict(required=False, type='str'),
environment=dict(required=False, type='dict'),
image=dict(required=False, type='str'),
ipc_mode=dict(required=False, type='str', choices=['host', '']),
cap_add=dict(required=False, type='list', default=list()),
security_opt=dict(required=False, type='list', default=list()),
pid_mode=dict(required=False, type='str', choices=['host', '']),
privileged=dict(required=False, type='bool', default=False),
graceful_timeout=dict(required=False, type='int', default=10),
remove_on_exit=dict(required=False, type='bool', default=True),
restart_policy=dict(required=False, type='str', choices=[
'no',
'never',
'on-failure',
'always',
'unless-stopped']),
restart_retries=dict(required=False, type='int', default=10),
tls_verify=dict(required=False, type='bool', default=False),
tls_cert=dict(required=False, type='str'),
tls_key=dict(required=False, type='str'),
tls_cacert=dict(required=False, type='str'),
volumes=dict(required=False, type='list'),
volumes_from=dict(required=False, type='list')
)
# 属性依赖ansible module的api规范，如start_container这个action, #必须要image和name这个两个属性。
required_if = [
['action', 'pull_image', ['image']],
['action', 'start_container', ['image', 'name']],
['action', 'compare_container', ['name']],
['action', 'compare_image', ['name']],
['action', 'create_volume', ['name']],
['action', 'get_container_env', ['name']],
['action', 'get_container_state', ['name']],
['action', 'recreate_or_restart_container', ['name']],
['action', 'remove_container', ['name']],
['action', 'remove_volume', ['name']],
['action', 'restart_container', ['name']],
['action', 'stop_container', ['name']]
]
#实例化
module = AnsibleModule(
argument_spec=argument_spec,
required_if=required_if,
bypass_checks=False
)
#以下部分主要做环境变量和通用参数以及特殊参数的更新。
new_args = module.params.pop('common_options', dict())
# NOTE(jeffrey4l): merge the environment
env = module.params.pop('environment', dict())
if env:
new_args['environment'].update(env)
for key, value in module.params.items():
if key in new_args and value is None:
continue
new_args[key] = value
# if pid_mode = ""/None/False, remove it
if not new_args.get('pid_mode', False):
new_args.pop('pid_mode', None)
# if ipc_mode = ""/None/False, remove it
if not new_args.get('ipc_mode', False):
new_args.pop('ipc_mode', None)
module.params = new_args
# 返回为AnsibleModule实例
return module
def main():
module = generate_module()
try:
dw = DockerWorker(module)
# TODO(inc0): We keep it bool to have ansible deal with consistent
# types. If we ever add method that will have to return some
# meaningful data, we need to refactor all methods to return dicts.
#返回值 result是action传递的函数名的运行成功与否的结果，意义#不大
result = bool(getattr(dw, module.params.get('action'))())
module.exit_json(changed=dw.changed, result=result)
except Exception:
module.exit_json(failed=True, changed=True,
msg=repr(traceback.format_exc()))
# import module snippets
from ansible.module_utils.basic import * # noqa
if __name__ == '__main__':
main()
Kolla_toolbox.py，在toolbox容器中运行ansible命令。这个我就不展开了，只贴关键代码。
#生成commandline函数，只包含ansible的命令
def gen_commandline(params):
command = ['ansible', 'localhost']
....
....
return command
#主函数
def main():
....
....
client = get_docker_client()(
version=module.params.get('api_version'))
#调用函数，获取命令（dict类型）
command_line = gen_commandline(module.params)
#过滤名称为kolla_toolbox的容器列表
kolla_toolbox = client.containers(filters=dict(name='kolla_toolbox',
status='running'))
if not kolla_toolbox:
module.fail_json(msg='kolla_toolbox container is not running.')
#默认只有一个，所有选了数组的第一个。kolla_toolbox变量名不建议重复使用，开源代码就是坑。
kolla_toolbox = kolla_toolbox[0]
#在容器中执行命令
job = client.exec_create(kolla_toolbox, command_line)
output = client.exec_start(job)
....
....
module.exit_json(**ret)
****Kolla_container_facts.py**** 调用dockerclient的python接口获取指定容器的facts信息，只传递一个name值即可。result类型是dict(changed=xxx, _containers=[]),代码不展开了。
Ansible/action_plugins/
该目录下记录了自定义的action的plugins，这些plugins在master上运行。但也可以在library目录下定义同名空文件，可以当作module使用。这里有两个代码文件，merge_configs.py和merge_yaml.py，用于conf和yml配置文件的合并。这里就分下下merge_yaml.py这个action plugin.
Merge_yaml.py，在task的参数sources中传递多个yml文件，合并之后输出到目标节点的dest中。期间在合并的同时，进行了参数模拟变量的渲染工作，最后调用copy模块把渲染后的数据文件复制过去。分析代码如下。
from ansible.plugins import action
#继承父类action.ActionBase
class ActionModule(action.ActionBase):
TRANSFERS_FILES = True
def read_config(self, source):
result = None
# Only use config if present
if os.access(source, os.R_OK):
with open(source, 'r') as f:
template_data = f.read()
# 渲染template模板数据，因为最终执行copy模块的时候，
# 变量被重新还原了，所以这里要先template变量先渲染，
# 因为有些变量对可能在copy模块中会消失
template_data = self._templar.template(template_data)
#把YAML数据，转化为dict对象
result = safe_load(template_data)
return result or {}
# 自定义action plugin必须实现的方法
def run(self, tmp=None, task_vars=None):
#task_vars是这个task的一些外传入的变量,
# 如host vars, group vars, config vars,etc
if task_vars is None:
task_vars = dict()
#自定义action plugin必须调用父类的run方法
result = super(ActionModule, self).run(tmp, task_vars)
# NOTE(jeffrey4l): Ansible 2.1 add a remote_user param to the
# _make_tmp_path function. inspect the number of the args here. In
# this way, ansible 2.0 and ansible 2.1 are both supported
#创建tmp临时目录，兼容2.0以后的版本
make_tmp_path_args = inspect.getargspec(self._make_tmp_path)[0]
if not tmp and len(make_tmp_path_args) == 1:
tmp = self._make_tmp_path()
if not tmp and len(make_tmp_path_args) == 2:
remote_user = (task_vars.get('ansible_user')
or self._play_context.remote_user)
tmp = self._make_tmp_path(remote_user)
# save template args.
# _task.args是这个task的参数，这里把参数中key为vars的对应值
# 保存到extra_vars变量中
extra_vars = self._task.args.get('vars', list())
# 备份template的可用变量
old_vars = self._templar._available_variables
# 将task_vars和extra_vars的所有变量merge到一起，赋值到temp_vars
temp_vars = task_vars.copy()
temp_vars.update(extra_vars)
#把最新的变量数据设置到templar对象（模板对象）
self._templar.set_available_variables(temp_vars)
output = {}
# 获取task的参数为sources的values值,可能是单个文件，
# 也有可能是多个文件组成的list
sources = self._task.args.get('sources', None)
#非数组，转化为只有一个item的数组
if not isinstance(sources, list):
sources = [sources]
#便历sources数组，读取文件中内容，并合并更新
#dict.update方式有去重效果，相当于merge
for source in sources:
output.update(self.read_config(source))
# restore original vars
#还原templar对象的变量
self._templar.set_available_variables(old_vars)
#把最新的合并好的数据output传递到远端的target host。复制给xfered变量
remote_path = self._connection._shell.join_path(tmp, 'src')
xfered = self._transfer_data(remote_path,
dump(output,
default_flow_style=False))
#把本task的参数拷贝，作为新模块的参数new_module_args
new_module_args = self._task.args.copy()
#更新new_module_args的src的值,后面copy模块的参数要求
new_module_args.update(
dict(
src=xfered
)
)
#删除new_module_args的sources参数，后面copy模块的参数要求
del new_module_args['sources']
#传入最新的参数new_module_args，task_vars执行copy的module
result.update(self._execute_module(module_name='copy',
module_args=new_module_args,
task_vars=task_vars,
tmp=tmp))
#返回result， action plugin 接口的要求
return result
Ansible/inventory/all-in-one
#control主机组包含本地localhost节点，连接方式为local
[control]
localhost ansible_connection=local
[network]
localhost ansible_connection=local
#neutron主机组包含network组下的所有节点
[neutron:children]
network
# Neutron
#neutron-server主机组包含control组下的所有节点
[neutron-server:children]
control
#neutron-dhcp-agent主机组包含neutron组下的所有节点
[neutron-dhcp-agent:children]
neutron
[neutron-l3-agent:children]
neutron
[neutron-lbaas-agent:children]
neutron
[neutron-metadata-agent:children]
neutron
[neutron-vpnaas-agent:children]
neutron
[neutron-bgp-dragent:children]
neutron
Ansible/site.yml
#调用ansible的setup获取节点的facts。gather_facts被设置为false是为了避免ansible再次去gathering facts.
- name: Gather facts for all hosts
hosts: all
serial: '{{ serial|default("0") }}'
gather_facts: false
tasks:
- setup:
tags: always
# NOTE(pbourke): This case covers deploying subsets of hosts using --limit. The
# limit arg will cause the first play to gather facts only about that node,
# meaning facts such as IP addresses for rabbitmq nodes etc. will be undefined
# in the case of adding a single compute node.
# We don't want to add the delegate parameters to the above play as it will
# result in ((num_nodes-1)^2) number of SSHs when running for all nodes
# which can be very inefficient.
- name: Gather facts for all hosts (if using --limit)
hosts: all
serial: '{{ serial|default("0") }}'
gather_facts: false
tasks:
- setup:
delegate_facts: True
delegate_to: "{{ item }}"
with_items: "{{ groups['all'] }}"
when:
- (play_hosts | length) != (groups['all'] | length)
#检测openstack_release全局变量信息，默认在globals.yml是不配置的，而
#在ansible/group_vars/all.yml中配置的默认值是auto。这里的两个tasks
#就是如果在auto的场景下，通过python的pbr包去检测安装好的#kolla-ansible版本，再将该版本号赋值给openstack_release变量。
#这里用到了ansible自带的local_action模块和register中间信息存储模块。
- name: Detect openstack_release variable
hosts: all
gather_facts: false
tasks:
- name: Get current kolla-ansible version number
local_action: command python -c "import pbr.version; print(pbr.version.VersionInfo('kolla-ansible'))"
register: kolla_ansible_version
changed_when: false
when: openstack_release == "auto"
- name: Set openstack_release variable
set_fact:
openstack_release: "{{ kolla_ansible_version.stdout }}"
when: openstack_release == "auto"
tags: always
#对所有节点进行recheck检查，前提条件是ansible-playbook命令传递的#action的值是precheck。
- name: Apply role prechecks
gather_facts: false
hosts:
- all
roles:
- role: prechecks
when: action == "precheck"
# 基于ntp时间同步角色的部署，hosts组为chrony-server和chrony
#前提条件是enable_chrony变量是否是yes，该值可在etc/kolla/globals.yml
#中配置，默认是no。
- name: Apply role chrony
gather_facts: false
hosts:
- chrony-server
- chrony
serial: '{{ serial|default("0") }}'
roles:
- { role: chrony,
tags: chrony,
when: enable_chrony | bool }
#部署neutron角色，这里部署的节点除了neutron相关的host组之外，还包括#compute和manila-share（openstack的一个文件共享组件）组。
- name: Apply role neutron
gather_facts: false
hosts:
- neutron-server
- neutron-dhcp-agent
- neutron-l3-agent
- neutron-lbaas-agent
- neutron-metadata-agent
- neutron-vpnaas-agent
- compute
- manila-share
serial: '{{ serial|default("0") }}'
roles:
- { role: neutron,
tags: neutron,
when: enable_neutron | bool }
Ansible/role/neutron/task 检查场景
该场景的action是precheck。由tasks/main.yml引用precheck.yml。
---
# kolla_container_facts是自定义的library，上文已经分析过代码，
# 用于获取容器名为neutron_server的一些容器属性数据，注册到中间变量container_facts
- name: Get container facts
kolla_container_facts:
name:
- neutron_server
register: container_facts
# 中间变量container_facts没有找到neutron_server关键字且该主机在neutron-server主机组中，
# 判断neutron_server_port 端口是否已经stopped
- name: Checking free port for Neutron Server
wait_for:
host: "{{ hostvars[inventory_hostname]['ansible_' + api_interface]['ipv4']['address'] }}"
port: "{{ neutron_server_port }}"
connect_timeout: 1
timeout: 1
state: stopped
when:
- container_facts['neutron_server'] is not defined
- inventory_hostname in groups['neutron-server']
# enable_neutron_agent_ha是true,且只规划了多个一个dhcp和l3服务节点，给出fail提示
- name: Checking number of network agents
local_action: fail msg="Number of network agents are less than two when enabling agent ha"
changed_when: false
when:
- enable_neutron_agent_ha | bool
- groups['neutron-dhcp-agent'] | length < 2
or groups['neutron-l3-agent'] | length < 2
# When MountFlags is set to shared, a signal bit configured on 20th bit of a number
# We need to check the 20th bit. 2^20 = 1048576. So we are validating against it.
# 检查docker服务的MountFlags是否设置为了shared
- name: Checking if 'MountFlags' for docker service is set to 'shared'
command: systemctl show docker
register: result
changed_when: false
failed_when: result.stdout.find('MountFlags=1048576') == -1
when:
- (inventory_hostname in groups['neutron-dhcp-agent']
or inventory_hostname in groups['neutron-l3-agent']
or inventory_hostname in groups['neutron-metadata-agent'])
- ansible_os_family == 'RedHat' or ansible_distribution == 'Ubuntu'
Ansible/role/neutron/task 部署场景
该场景的action是deploy。由tasks/main.yml引用deploy.yml
# enforce ironic usage only with openvswitch
# 裸机部署检查，检查ironic服务必须启动，neutron的plugin必须使用OpenvSwitch
- include: ironic-check.yml
#在neutron-server的节点执行注册
- include: register.yml
when: inventory_hostname in groups['neutron-server']
#执行配置，拷贝配置文件，启动组件容器主要都在这里实现
- include: config.yml
#在nova fake driver模拟场景下，计算节点执行config-neutron-fake.yml，不详细分析
#nova fake driver可以在单个计算节点中创建多个docker容器运行novc-compute，
#Nova fake driver can not work with all-in-one deployment. This is because the fake
#neutron-openvswitch-agent for the fake nova-compute container conflicts with
#neutron-openvswitch-agent on the compute nodes. Therefore, in the inventory
#the network node must be different than the compute node.
- include: config-neutron-fake.yml
when:
- enable_nova_fake | bool
- inventory_hostname in groups['compute']
#在neutron-server的节点执行创建数据库，创建容器
#bootstrap.yml会去创建数据库相关信息，结束后会去调用#bootstrap_servcie.yml该文件是用于在server节点上创建容器。
- include: bootstrap.yml
when: inventory_hostname in groups['neutron-server']
#执行handlers目录下的task任务
- name: Flush Handlers
meta: flush_handlers
Register.yml, 往keystone中注册neutron服务的鉴权相关信息。
---
# 在keystone创建neutron的service 和endpoint
# kolla_toolbox见library分析，用于在toolbox容器中执行ansible命令
# kolla_keystone_service模块是kolla-ansible的父项目kolla中的代码，已经是一个可调用的ansible模块
#service名称为neutron,对应的endpoint分为内部，管理员，公共三个。
# 变量主要在ansible/role/neutron/defauts/main.yml和ansible/group_vars/all.yml中
- name: Creating the Neutron service and endpoint
kolla_toolbox:
module_name: "kolla_keystone_service"
module_args:
service_name: "neutron"
service_type: "network"
description: "Openstack Networking"
endpoint_region: "{{ openstack_region_name }}"
url: "{{ item.url }}"
interface: "{{ item.interface }}"
region_name: "{{ openstack_region_name }}"
auth: "{{ '{{ openstack_neutron_auth }}' }}"
module_extra_vars:
openstack_neutron_auth: "{{ openstack_neutron_auth }}"
run_once: True
with_items:
- {'interface': 'admin', 'url': '{{ neutron_admin_endpoint }}'}
- {'interface': 'internal', 'url': '{{ neutron_internal_endpoint }}'}
- {'interface': 'public', 'url': '{{ neutron_public_endpoint }}'}
# 同上，创建项目，用户，角色。经分析openstack_neutron_auth变量实际为#openstack的admin的auth。
- name: Creating the Neutron project, user, and role
kolla_toolbox:
module_name: "kolla_keystone_user"
module_args:
project: "service"
user: "{{ neutron_keystone_user }}"
password: "{{ neutron_keystone_password }}"
role: "admin"
region_name: "{{ openstack_region_name }}"
auth: "{{ '{{ openstack_neutron_auth }}' }}"
module_extra_vars:
openstack_neutron_auth: "{{ openstack_neutron_auth }}"
run_once: True
Config.yml，配置文件合并下发，创建或重启容器。
#调用sysctl模块，配置ip转发相关配置
- name: Setting sysctl values
vars:
neutron_l3_agent: "{{ neutron_services['neutron-l3-agent'] }}"
neutron_vpnaas_agent: "{{ neutron_services['neutron-vpnaas-agent'] }}"
sysctl: name={{ item.name }} value={{ item.value }} sysctl_set=yes
with_items:
- { name: "net.ipv4.ip_forward", value: 1}
- { name: "net.ipv4.conf.all.rp_filter", value: 0}
- { name: "net.ipv4.conf.default.rp_filter", value: 0}
when:
- set_sysctl | bool
- (neutron_l3_agent.enabled | bool and neutron_l3_agent.host_in_groups | bool)
or (neutron_vpnaas_agent.enabled | bool and neutron_vpnaas_agent.host_in_groups | bool)
# 创建neutron各服务的配置文件目录，前提条件主要看host_in_groups变量，这个在
# ansible/role/neutron/defauts/main.yml文件中进行了详细的定义
- name: Ensuring config directories exist
file:
path: "{{ node_config_directory }}/{{ item.key }}"
state: "directory"
recurse: yes
when:
- item.value.enabled | bool
- item.value.host_in_groups | bool
with_dict: "{{ neutron_services }}"
....
....
#下发配置文件到指定目录，三份陪配置文件合一，merge_conifgs模块在action plugin中已经分析过了
#文件下发完了之后，通知相应组件的容器重启。在handlers目录下。
# 重启容器这个操作会调用recreate_or_restart_container这个action，第一次会创建容器。
- name: Copying over neutron_lbaas.conf
vars:
service_name: "{{ item.key }}"
services_need_neutron_lbaas_conf:
- "neutron-server"
- "neutron-lbaas-agent"
merge_configs:
sources:
- "{{ role_path }}/templates/neutron_lbaas.conf.j2"
- "{{ node_custom_config }}/neutron/neutron_lbaas.conf"
- "{{ node_custom_config }}/neutron/{{ inventory_hostname }}/neutron_lbaas.conf"
dest: "{{ node_config_directory }}/{{ item.key }}/neutron_lbaas.conf"
register: neutron_lbaas_confs
when:
- item.value.enabled | bool
- item.value.host_in_groups | bool
- item.key in services_need_neutron_lbaas_conf
with_dict: "{{ neutron_services }}"
notify:
- "Restart {{ item.key }} container"
....
....
#kolla_docker是自定义的模块，通过调用compare_container查找该节点的所有neutron service容器
- name: Check neutron containers
kolla_docker:
action: "compare_container"
common_options: "{{ docker_common_options }}"
name: "{{ item.value.container_name }}"
image: "{{ item.value.image }}"
privileged: "{{ item.value.privileged | default(False) }}"
volumes: "{{ item.value.volumes }}"
register: check_neutron_containers
when:
- action != "config"
- item.value.enabled | bool
- item.value.host_in_groups | bool
with_dict: "{{ neutron_services }}"
notify:
- "Restart {{ item.key }} container"
Bootstrap.yml，创建neutron数据库对象。
---
# kolla_toolbox自定义模块，在toolbox容器中调用mysql_db的ansible模块，创建db
# delegate_to指定在第一个neutron-server上执行，run_onece只运行一次
- name: Creating Neutron database
kolla_toolbox:
module_name: mysql_db
module_args:
login_host: "{{ database_address }}"
login_port: "{{ database_port }}"
login_user: "{{ database_user }}"
login_password: "{{ database_password }}"
name: "{{ neutron_database_name }}"
register: database
run_once: True
delegate_to: "{{ groups['neutron-server'][0] }}"
# 创建neuron数据库的用户并设置权限
- name: Creating Neutron database user and setting permissions
kolla_toolbox:
module_name: mysql_user
module_args:
login_host: "{{ database_address }}"
login_port: "{{ database_port }}"
login_user: "{{ database_user }}"
login_password: "{{ database_password }}"
name: "{{ neutron_database_name }}"
password: "{{ neutron_database_password }}"
host: "%"
priv: "{{ neutron_database_name }}.*:ALL"
append_privs: "yes"
run_once: True
delegate_to: "{{ groups['neutron-server'][0] }}"
#数据库改变之后，调用bootstrap_service.yml
- include: bootstrap_service.yml
when: database.changed
Bootstrap_service.yml，创建bootrsap_neutron，bootrsap_neutron_lbassd-agent，bootrsap_neutron_vpnaas_agent容器。
Ansible/role/neutron/handlers/main.yml，重建或重启neutron相关容器。列出一个分析下。
# 举例分析：neutron_lbaas_confs变量是之前执行的时候注册的变量，这里的task变量
# neutron_lbaas_conf从neutron_lbaas_confs的结果中取值分析作为when的条件判断
# kolla_docker模块的volumes参数是从role/neutron/defaults/main.yml中获取，如下，
#volumes:
# - "{{ node_config_directory }}/neutron-lbaas-agent/:{{
# container_config_directory }}/:ro"
# - "/etc/localtime:/etc/localtime:ro"
# - "/run:/run:shared"
# - "kolla_logs:/var/log/kolla/"
#且在config.yml已经把neutron_lbaas的配置文件下发{{ node_config_directory }}/
# neutron-lbaas-agent/这个目录系了。 容器中的路径container_config_directory变量
# 可以在groups_var中找到,值为 /var/lib/kolla/config_files。
- name: Restart neutron-server container
vars:
service_name: "neutron-server"
service: "{{ neutron_services[service_name] }}"
config_json: "{{ neutron_config_jsons.results|selectattr('item.key', 'equalto', service_name)|first }}"
neutron_conf: "{{ neutron_confs.results|selectattr('item.key', 'equalto', service_name)|first }}"
neutron_lbaas_conf: "{{ neutron_lbaas_confs.results|selectattr('item.key', 'equalto', service_name)|first }}"
neutron_ml2_conf: "{{ neutron_ml2_confs.results|selectattr('item.key', 'equalto', service_name)|first }}"
policy_json: "{{ policy_jsons.results|selectattr('item.key', 'equalto', service_name)|first }}"
neutron_server_container: "{{ check_neutron_containers.results|selectattr('item.key', 'equalto', service_name)|first }}"
kolla_docker:
action: "recreate_or_restart_container"
common_options: "{{ docker_common_options }}"
name: "{{ service.container_name }}"
image: "{{ service.image }}"
volumes: "{{ service.volumes }}"
privileged: "{{ service.privileged | default(False) }}"
when:
- action != "config"
- service.enabled | bool
- service.host_in_groups | bool
- config_json | changed
or neutron_conf | changed
or neutron_lbaas_conf | changed
or neutron_vpnaas_conf | changed
or neutron_ml2_conf | changed
or policy_json | changed
or neutron_server_container | changed
Ansible/role/neutron/task 下拉镜像
Pull.yml 下拉镜像
#调用kolla_docker的pull_image 下拉镜像
- name: Pulling neutron images
kolla_docker:
action: "pull_image"
common_options: "{{ docker_common_options }}"
image: "{{ item.value.image }}"
when:
- item.value.enabled | bool
- item.value.host_in_groups | bool
with_dict: "{{ neutron_services }}"
流程介绍
镜像路径
kolla-ansible\ansible\roles\chrony\defaults\main.yml文件中的一段，如下:
#docker_registry是本地注册的registry地址，在global.yml中会配置。Registry虚拟机为会从定向5000端口带宿主机的4000端口。
docker_namespace也在global.yml定义，如果从官网下载源码镜像的话，配置成lokolla。
kolla_base_distro默认centos，也可以配置成ubuntu
kolla_install_type 默认为binary, 我们配置在global.yml配置成了source
例子：
docker_registry：192.168.102.15:4000
docker_namespace：lokolla
kolla_base_distro：“centos”
kolla_install_type: source
openstack_release： auto (自发现，前文有讲到)
所以最后的chrony_image_full为192.168.102.15:4000/lokolla/centos-source-chrony:4.0.2
chrony_image: "{{ docker_registry ~ '/' if docker_registry else '' }}{{ docker_namespace }}/{{ kolla_base_distro }}-{{ kolla_install_type }}-chrony"
chrony_tag: "{{ openstack_release }}"
chrony_image_full: "{{ chrony_image }}:{{ chrony_tag }}"链接：https://www.jianshu.com/p/d66d0f61ad34
來源：简书
简书著作权归作者所有，任何形式的转载都请联系作者获得授权并注明出处。
评论暂时关闭