OpenStack 动态迁移流程分析

OpenStack 动态迁移流程分析

Live Migration RPC Calls

Let's consider the two hosts called:

• src (where inst runs)
• dest

The user call maps to:

• scheduler rpc: live_migration(block_migration, disk_over_commit, instance_id, dest)

Scheduler driver does the following:

• check instance exists
• check source alive
• check destination alive, and has enough memory
• check source + destination hypervisor type match and dest is same or newer version than src
• compute: check_can_live_migrate_on_dest (on dest) *TODO*
• updates instance db entry to "migrating"
• compute: live_migration (on src)

Compute:check_can_live_migrate_on_destination does the following (on dest): *TODO*

• calls compute_driver check_can_live_migration_on_dest
• which can call the delegate that calls check_can_live_migrate_on_src

Compute:check_can_live_migrate_on_src does the following (on src): *TODO*

• takes a dictionary from dest
• possibly throws an exception if there is an issue

Compute: live_migration does the following (on src):

• check_for_export with volume
• calls pre_live_migration on dest
• does rollback on exception
• calls driver

Compute_driver: live_migration (on src)

• on success calls manager's post_live_migration
• on failure calls manager's rollback_live_migration

Compute: post_live_migration (on src):

• updates floating ips
• deletes old traces of image
• calls post_live_migration_at_destination on dest

Compute: rollback_live_migration (on src)

• updates DB
• sorts out volumes and networks
• on block migration, calls rollback_live_migration_at_destination on dest

Compute: post_live_migration_at_destination (on dest)

• sets up networking
• sets task as complete

Compute: rollback_live_migration_at_destination (on dest)

• does some clean up
  

A. nova/api/openstack/compute/contrib/admin_actions.py:定义了若干管理员权限运行的管理虚拟机的操作

_migrate_live()函数:虚拟机动态迁移在Opentack Nova中的

    @wsgi.action('os-migrateLive')
    def _migrate_live(self, req, id, body):
        """Permit admins to (live) migrate a server to a new host."""
        context = req.environ["nova.context"]
        authorize(context, 'migrateLive')

        try:
            block_migration = body["os-migrateLive"]["block_migration"]
            disk_over_commit = body["os-migrateLive"]["disk_over_commit"]
            host = body["os-migrateLive"]["host"]
        except (TypeError, KeyError):
            msg = _("host, block_migration and disk_over_commit must "
                    "be specified for live migration.")
            raise exc.HTTPBadRequest(explanation=msg)

        try:
            instance = self.compute_api.get(context, id, want_objects=True)
            self.compute_api.live_migrate(context, instance, block_migration,
                                          disk_over_commit, host) #转B
        except (exception.ComputeServiceUnavailable,
                exception.InvalidHypervisorType,
                exception.UnableToMigrateToSelf,
                exception.DestinationHypervisorTooOld) as ex:
            raise exc.HTTPBadRequest(explanation=ex.format_message())
        except Exception:
            if host is None:
                msg = _("Live migration of instance %s to another host "
                        "failed") % id
            else:
                msg = _("Live migration of instance %(id)s to host %(host)s "
                        "failed") % {'id': id, 'host': host}
            LOG.exception(msg)
            # Return messages from scheduler
            raise exc.HTTPBadRequest(explanation=msg)

        return webob.Response(status_int=202)

B. nova/compute/api.py: 注意，此处在live_migrate返回之后，还调用check_instance_state对迁移后的vm进行状态检查

    @check_instance_cell
    @check_instance_state(vm_state=[vm_states.ACTIVE])
    def live_migrate(self, context, instance, block_migration,
                     disk_over_commit, host_name):
        """Migrate a server lively to a new host."""
        LOG.debug(_("Going to try to live migrate instance to %s"),
                  host_name or "another host", instance=instance)

        instance.task_state = task_states.MIGRATING
        instance.save(expected_task_state=None)

        self.compute_task_api.live_migrate_instance(context, instance, #转C
                host_name, block_migration=block_migration,
                disk_over_commit=disk_over_commit)

C.nova/conductor/api.py: class ComputeTaskAPI()

    def live_migrate_instance(self, context, instance, host_name,
                              block_migration, disk_over_commit):
        scheduler_hint = {'host': host_name}
        self.conductor_compute_rpcapi.migrate_server(  #转D
            context, instance, scheduler_hint, True, False, None,
            block_migration, disk_over_commit, None)

D. nova/conductor/rpcapi.py: migrate_server()

    def migrate_server(self, context, instance, scheduler_hint, live, rebuild,
                  flavor, block_migration, disk_over_commit,
                  reservations=None):
        if self.client.can_send_version('1.6'):
            version = '1.6'
        else:
            instance = jsonutils.to_primitive(
                    objects_base.obj_to_primitive(instance))
            version = '1.4'
        flavor_p = jsonutils.to_primitive(flavor)
        cctxt = self.client.prepare(version=version)
        return cctxt.call(context, 'migrate_server',  #转E
                          instance=instance, scheduler_hint=scheduler_hint,
                          live=live, rebuild=rebuild, flavor=flavor_p,
                          block_migration=block_migration,
                          disk_over_commit=disk_over_commit,
                          reservations=reservations)

E. nova/conductor/manager.py: migrate_server()

    def migrate_server(self, context, instance, scheduler_hint, live, rebuild,
            flavor, block_migration, disk_over_commit, reservations=None):
        if instance and not isinstance(instance, instance_obj.Instance):
            # NOTE(danms): Until v2 of the RPC API, we need to tolerate
            # old-world instance objects here
            attrs = ['metadata', 'system_metadata', 'info_cache',
                     'security_groups']
            instance = instance_obj.Instance._from_db_object(
                context, instance_obj.Instance(), instance,
                expected_attrs=attrs)
        if live and not rebuild and not flavor:
            self._live_migrate(context, instance, scheduler_hint, #转F
                               block_migration, disk_over_commit)
        elif not live and not rebuild and flavor:
            instance_uuid = instance['uuid']
            with compute_utils.EventReporter(context, ConductorManager(),
                                         'cold_migrate', instance_uuid):
                self._cold_migrate(context, instance, flavor,
                                   scheduler_hint['filter_properties'],
                                   reservations)
        else:
            raise NotImplementedError()

F.nova/conductor/manager.py : _live_migrate()

    def _live_migrate(self, context, instance, scheduler_hint,
                      block_migration, disk_over_commit):
        destination = scheduler_hint.get("host")
        try:
            live_migrate.execute(context, instance, destination, #转3
                             block_migration, disk_over_commit)
        except (exception.NoValidHost,
                exception.ComputeServiceUnavailable,
                exception.InvalidHypervisorType,
                exception.UnableToMigrateToSelf,
                exception.DestinationHypervisorTooOld,
                exception.InvalidLocalStorage,
                exception.InvalidSharedStorage,
                exception.MigrationPreCheckError) as ex:
            with excutils.save_and_reraise_exception():
                #TODO(johngarbutt) - eventually need instance actions here
                request_spec = {'instance_properties': {
                    'uuid': instance['uuid'], },
                }
                scheduler_utils.set_vm_state_and_notify(context,
                        'compute_task', 'migrate_server',
                        dict(vm_state=instance['vm_state'],
                             task_state=None,
                             expected_task_state=task_states.MIGRATING,),
                        ex, request_spec, self.db)
        except Exception as ex:
            with excutils.save_and_reraise_exception():
                request_spec = {'instance_properties': {
                    'uuid': instance['uuid'], },
                }
                scheduler_utils.set_vm_state_and_notify(context,
                        'compute_task', 'migrate_server',
                        {'vm_state': vm_states.ERROR},
                        ex, request_spec, self.db)

1.动态迁移调用

nova/scheduler/manager.py:SchedulerManager()

    def live_migration(self, context, instance, dest,
                       block_migration, disk_over_commit):
        try:
            self._schedule_live_migration(context, instance, dest,
                    block_migration, disk_over_commit) #调用2
        except (exception.NoValidHost,
                exception.ComputeServiceUnavailable,
                exception.InvalidHypervisorType,
                exception.UnableToMigrateToSelf,
                exception.DestinationHypervisorTooOld,
                exception.InvalidLocalStorage,
                exception.InvalidSharedStorage,
                exception.MigrationPreCheckError) as ex:
            request_spec = {'instance_properties': {
                'uuid': instance['uuid'], },
            }
            with excutils.save_and_reraise_exception():
                self._set_vm_state_and_notify('live_migration',
                            dict(vm_state=instance['vm_state'],
                                 task_state=None,
                                 expected_task_state=task_states.MIGRATING,),
                                              context, ex, request_spec)
        except Exception as ex:
            request_spec = {'instance_properties': {
                'uuid': instance['uuid'], },
            }
            with excutils.save_and_reraise_exception():
                self._set_vm_state_and_notify('live_migration',
                                             {'vm_state': vm_states.ERROR},
                                             context, ex, request_spec)

2.用户调用scheduler rpc准备动态迁移

nova/scheduler/manager.py

    def _schedule_live_migration(self, context, instance, dest,
            block_migration, disk_over_commit): 
        task = live_migrate.LiveMigrationTask(context, instance,
                    dest, block_migration, disk_over_commit) #调用3中的__init__()函数进行初始化
        return task.execute() #返回3中的execute()函数

3.nova/conductor/tasks/live_migration.py

execute()函数中，主要进行了三部分的内容，第一，如果目前主机不存在，则由调度算法选取一个目标主机，并且进行相关的检测，确保能够进行实时迁移操作；第二，如果目标主机存在，则直接进行相关的检测操作，确保能够进行实时迁移操作；第三，执行迁移操作。

class LiveMigrationTask(object):
    def __init__(self, context, instance, destination,
                 block_migration, disk_over_commit):
        self.context = context
        self.instance = instance
        self.destination = destination
        self.block_migration = block_migration
        self.disk_over_commit = disk_over_commit
        self.source = instance.host
        self.migrate_data = None
        self.compute_rpcapi = compute_rpcapi.ComputeAPI()
        self.servicegroup_api = servicegroup.API()
        self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
        self.image_service = glance.get_default_image_service()

    def execute(self):
        self._check_instance_is_running() #检查实例是否存在(check instance exists)
        self._check_host_is_up(self.source) #检测源虚机状态(check source alive)

        if not self.destination:
            self.destination = self._find_destination() #若目的虚机不存在,则找到并转3.1
        else:
            self._check_requested_destination() #转到3.2检测目的虚机状态

        #TODO(johngarbutt) need to move complexity out of compute manager
        #跳向nova/compute/rpcapi.py:live_migration()函数然后使用cast形式调用4
        return self.compute_rpcapi.live_migration(self.context, 
                host=self.source,
                instance=self.instance,
                dest=self.destination,
                block_migration=self.block_migration,
                migrate_data=self.migrate_data)
                #TODO(johngarbutt) disk_over_commit?
            

     

nova/conductor/tasks/live_migration.py

    def _find_destination(self):
        #TODO(johngarbutt) this retry loop should be shared
        attempted_hosts = [self.source]
        image = None
        if self.instance.image_ref:
            image = compute_utils.get_image_metadata(self.context,
                                                     self.image_service,
                                                     self.instance.image_ref,
                                                     self.instance)
        instance_type = flavors.extract_flavor(self.instance)

        host = None
        while host is None:
            self._check_not_over_max_retries(attempted_hosts)

            host = self._get_candidate_destination(image,
                    instance_type, attempted_hosts)
            try:
                self._check_compatible_with_source_hypervisor(host) #检测源和目的虚机hypervisor版本是否匹配
                self._call_livem_checks_on_host(host) #检测能否在目的机上进行迁移 转3.1.1
            except exception.Invalid as e:
                LOG.debug(_("Skipping host: %(host)s because: %(e)s") %
                    {"host": host, "e": e})
                attempted_hosts.append(host)
                host = None
        return host

3.1.1 nova/computer/rpcapi.py：check_can_live_migrate_destination() 这个方法实现的是在目标主机上检测是否可以进行实时迁移，并将检测结果返回给源主机

    def check_can_live_migrate_destination(self, ctxt, instance, destination,
                                           block_migration, disk_over_commit):
        if self.client.can_send_version('2.38'):
            version = '2.38'
        else:
            version = '2.0'
            instance = jsonutils.to_primitive(
                    objects_base.obj_to_primitive(instance))
        cctxt = self.client.prepare(server=destination, version=version)
        #这里调用call方法实现在一个主题topic上发送一条远程消息，实现在目标主机上进行检测是否可以进行实时迁移，并等待响应。调用nova/compute/manager.py:check_can_live_migrate_destination()执行具体的检测
        return cctxt.call(ctxt, 'check_can_live_migrate_destination',
                          instance=instance,
                          block_migration=block_migration,
                          disk_over_commit=disk_over_commit)

3.2 nova/conductor/tasks/live_migration.py :_check_requested_destination()函数进行目的虚机的状态检测

    def _check_requested_destination(self):
        self._check_destination_is_not_source() #目的虚机与源虚机是否一致
        self._check_host_is_up(self.destination) #目的虚机是否已启动
        self._check_destination_has_enough_memory() #是否有足够内存
        self._check_compatible_with_source_hypervisor(self.destination) #源与目的虚机hypervisor是否匹配
        self._call_livem_checks_on_host(self.destination) #目的虚机能否进行迁移

4.nova/compute/manager.py:live_migration()函数执行动态迁移

目的主机上发起migrate过程，调用了两个方法pre_live_migration和live_migration

    def live_migration(self, context, dest, instance,
                       block_migration=False, migrate_data=None):
        """Executing live migration.

        :param context: security context
        :param instance: instance dict
        :param dest: destination host
        :param block_migration: if true, prepare for block migration
        :param migrate_data: implementation specific params

        """
        # Create a local copy since we'll be modifying the dictionary
        migrate_data = dict(migrate_data or {})
        try:
            if block_migration:
                disk = self.driver.get_instance_disk_info(instance['name']) 
            else:
                disk = None

            pre_migration_data = self.compute_rpcapi.pre_live_migration(
                context, instance,
                block_migration, disk, dest, migrate_data) #通过compute_rpc发送一个pre_live_migration的请求给源主机的nova-compute进行动态迁移的预处理
            migrate_data['pre_live_migration_result'] = pre_migration_data

        except Exception:
            with excutils.save_and_reraise_exception():
                LOG.exception(_('Pre live migration failed at %s'),
                              dest, instance=instance)
                self._rollback_live_migration(context, instance, dest,
                                              block_migration, migrate_data) #如遇异常，则rollback

        # Executing live migration
        # live_migration might raises exceptions, but
        # nothing must be recovered in this version.
        #根据本地的配置(compute_driver=libvirt.LibvirtDriver)，所以这里的driver是nova.virt.libvirt.driver.LibvirtDriver
        self.driver.live_migration(context, instance, dest, #调用目的主机上的libvirt驱动进行实际的迁移过程
                                   self._post_live_migration,  #迁移成功,转4.1 调用nova/compute/manager.py _post_live_migration()方法
                                   self._rollback_live_migration, #迁移异常,转4.2调用nova/compute/manager.py _rollback_live_migration()
                                   block_migration, migrate_data) 

    @wrap_exception()
    def _post_live_migration(self, ctxt, instance_ref,
                            dest, block_migration=False, migrate_data=None):
        """Post operations for live migration.

        This method is called from live_migration
        and mainly updating database record.

        :param ctxt: security context
        :param instance_ref: nova.db.sqlalchemy.models.Instance
        :param dest: destination host
        :param block_migration: if true, prepare for block migration
        :param migrate_data: if not None, it is a dict which has data
        required for live migration without shared storage

        """
        LOG.info(_('_post_live_migration() is started..'),
                 instance=instance_ref)

        # Cleanup source host post live-migration
        block_device_info = self._get_instance_volume_block_device_info(
                            ctxt, instance_ref)
        self.driver.post_live_migration(ctxt, instance_ref, block_device_info)

        # Detaching volumes.
        connector = self.driver.get_volume_connector(instance_ref)
        for bdm in self._get_instance_volume_bdms(ctxt, instance_ref):
            # NOTE(vish): We don't want to actually mark the volume
            #             detached, or delete the bdm, just remove the
            #             connection from this host.

            # remove the volume connection without detaching from hypervisor
            # because the instance is not running anymore on the current host
            self.volume_api.terminate_connection(ctxt, bdm['volume_id'],
                                                 connector)

        # Releasing vlan.
        # (not necessary in current implementation?)

        network_info = self._get_instance_nw_info(ctxt, instance_ref)

        self._notify_about_instance_usage(ctxt, instance_ref,
                                          "live_migration._post.start",
                                          network_info=network_info)
        # Releasing security group ingress rule.
        self.driver.unfilter_instance(instance_ref,
                                      network_info)

        migration = {'source_compute': self.host,
                     'dest_compute': dest, }
        self.conductor_api.network_migrate_instance_start(ctxt,
                                                          instance_ref,
                                                          migration)

        # Define domain at destination host, without doing it,
        # pause/suspend/terminate do not work.
        self.compute_rpcapi.post_live_migration_at_destination(ctxt,
                instance_ref, block_migration, dest) #转到4.1.1

        # No instance booting at source host, but instance dir
        # must be deleted for preparing next block migration
        # must be deleted for preparing next live migration w/o shared storage
        is_shared_storage = True
        if migrate_data:
            is_shared_storage = migrate_data.get('is_shared_storage', True)
        if block_migration or not is_shared_storage:
            self.driver.destroy(instance_ref, network_info)
        else:
            # self.driver.destroy() usually performs  vif unplugging
            # but we must do it explicitly here when block_migration
            # is false, as the network devices at the source must be
            # torn down
            self.driver.unplug_vifs(instance_ref, network_info)

        # NOTE(tr3buchet): tear down networks on source host
        self.network_api.setup_networks_on_host(ctxt, instance_ref,
                                                self.host, teardown=True)

        self._notify_about_instance_usage(ctxt, instance_ref,
                                          "live_migration._post.end",
                                          network_info=network_info)
        LOG.info(_('Migrating instance to %s finished successfully.'),
                 dest, instance=instance_ref)
        LOG.info(_("You may see the error \"libvirt: QEMU error: "
                   "Domain not found: no domain with matching name.\" "
                   "This error can be safely ignored."),
                 instance=instance_ref)

        if CONF.vnc_enabled or CONF.spice.enabled:
            if CONF.cells.enable:
                self.cells_rpcapi.consoleauth_delete_tokens(ctxt,
                        instance_ref['uuid'])
            else:
                self.consoleauth_rpcapi.delete_tokens_for_instance(ctxt,
                        instance_ref['uuid'])

    def post_live_migration_at_destination(self, context, instance,
                                           block_migration=False):
        """Post operations for live migration .

        :param context: security context
        :param instance: Instance dict
        :param block_migration: if true, prepare for block migration

        """
        LOG.info(_('Post operation of migration started'),
                 instance=instance)

        # NOTE(tr3buchet): setup networks on destination host
        #                  this is called a second time because
        #                  multi_host does not create the bridge in
        #                  plug_vifs
        self.network_api.setup_networks_on_host(context, instance,
                                                         self.host) #sets up networking
        migration = {'source_compute': instance['host'],
                     'dest_compute': self.host, }
        self.conductor_api.network_migrate_instance_finish(context,
                                                           instance,
                                                           migration) #sets task as complete

        network_info = self._get_instance_nw_info(context, instance)
        self._notify_about_instance_usage(
                     context, instance, "live_migration.post.dest.start",
                     network_info=network_info)
        block_device_info = self._get_instance_volume_block_device_info(
                            context, instance)

        self.driver.post_live_migration_at_destination(context, instance,
                                            network_info,
                                            block_migration, block_device_info)
        # Restore instance state
        current_power_state = self._get_power_state(context, instance)
        node_name = None
        try:
            compute_node = self._get_compute_info(context, self.host)
            node_name = compute_node['hypervisor_hostname']
        except exception.NotFound:
            LOG.exception(_('Failed to get compute_info for %s') % self.host)
        finally:
            instance = self._instance_update(context, instance['uuid'],
                    host=self.host, power_state=current_power_state,
                    vm_state=vm_states.ACTIVE, task_state=None,
                    expected_task_state=task_states.MIGRATING,
                    node=node_name)

        # NOTE(vish): this is necessary to update dhcp
        self.network_api.setup_networks_on_host(context, instance, self.host)
        self._notify_about_instance_usage(
                     context, instance, "live_migration.post.dest.end",
                     network_info=network_info)

    @wrap_exception()
    def _rollback_live_migration(self, context, instance,
                                 dest, block_migration, migrate_data=None):
        """Recovers Instance/volume state from migrating -> running.

        :param context: security context
        :param instance: nova.db.sqlalchemy.models.Instance
        :param dest:
            This method is called from live migration src host.
            This param specifies destination host.
        :param block_migration: if true, prepare for block migration
        :param migrate_data:
            if not none, contains implementation specific data.

        """
        host = instance['host']
        instance = self._instance_update(context, instance['uuid'],
                host=host, vm_state=vm_states.ACTIVE,
                task_state=None, expected_task_state=task_states.MIGRATING) #updates DB

        # NOTE(tr3buchet): setup networks on source host (really it's re-setup)
        self.network_api.setup_networks_on_host(context, instance, self.host) #sorts out volumes

        for bdm in self._get_instance_volume_bdms(context, instance):  #sorts out  networks
            volume_id = bdm['volume_id']
            self.compute_rpcapi.remove_volume_connection(context, instance,
                    volume_id, dest)

        self._notify_about_instance_usage(context, instance,
                                          "live_migration._rollback.start")

        # Block migration needs empty image at destination host
        # before migration starts, so if any failure occurs,
        # any empty images has to be deleted.
        # Also Volume backed live migration w/o shared storage needs to delete
        # newly created instance-xxx dir on the destination as a part of its
        # rollback process
        is_volume_backed = False
        is_shared_storage = True
        if migrate_data:
            is_volume_backed = migrate_data.get('is_volume_backed', False)
            is_shared_storage = migrate_data.get('is_shared_storage', True)
        if block_migration or (is_volume_backed and not is_shared_storage):
            self.compute_rpcapi.rollback_live_migration_at_destination(context,
                    instance, dest)  #调用nova/compute/rpcapi.py:rollback_live_migration_at_destination()做一些清理工作

        self._notify_about_instance_usage(context, instance,
                                          "live_migration._rollback.end")

    @wrap_exception()
    def pre_live_migration(self, context, instance,
                           block_migration=False, disk=None,
                           migrate_data=None):
        """Preparations for live migration at dest host.

        :param context: security context
        :param instance: dict of instance data
        :param block_migration: if true, prepare for block migration
        :param migrate_data : if not None, it is a dict which holds data
        required for live migration without shared storage.

        """
        #取出虚拟机实例的块设备信息
        block_device_info = self._get_instance_volume_block_device_info(
                            context, instance, refresh_conn_info=True)
        # 取出虚拟机实例的网络配置信息(实际调用的是network_api向源主机请求数据)
        network_info = self._get_instance_nw_info(context, instance)
        self._notify_about_instance_usage(
                     context, instance, "live_migration.pre.start",
                     network_info=network_info)

        # TODO(tr3buchet): figure out how on the earth this is necessary
        fixed_ips = network_info.fixed_ips()
        if not fixed_ips:
            raise exception.FixedIpNotFoundForInstance(
                                       instance_uuid=instance['uuid'])

        pre_live_migration_data = self.driver.pre_live_migration(context,
                                       instance,
                                       block_device_info,
                                       network_info,
                                       disk,
                                       migrate_data)
        # NOTE(tr3buchet): setup networks on destination host
        self.network_api.setup_networks_on_host(context, instance,
                                                         self.host)

        # Creating filters to hypervisors and firewalls.
        # An example is that nova-instance-instance-xxx,
        # which is written to libvirt.xml(Check "virsh nwfilter-list")
        # This nwfilter is necessary on the destination host.
        # In addition, this method is creating filtering rule
        # onto destination host.
        # 创建hypervisors和firewalls相关的过滤器
        self.driver.ensure_filtering_rules_for_instance(instance,
                                            network_info)

        self._notify_about_instance_usage(
                     context, instance, "live_migration.pre.end",
                     network_info=network_info)

        return pre_live_migration_data