Cinder中的iscsitarget和tgt的使用
Cinder中的iscsitarget和tgt的使用
OpenStack中的Cinder-volume组件的iSCSI target volume可以通过iscsitarget或者
环境说明：Ubuntu 12.04+OpenStack Grizzly
组件说明：tgt和iscsitarget使用不同的管理工具，tgt使用的是tgtadm，iscsitarget使用的是ietadm，体现在步骤5中的iscsi_helper属性。
ietadm： iSCSI Enterprise Target Administration Utility ---------- iscsitarget
tgtadm： Linux SCSI Target Administration Utility ---------- tgt
apt-get install -y cinder-api cinder-scheduler cinder-volume
apt-get install -y cinder-api cinder-scheduler cinder-volume iscsitarget open-iscsi iscsitarget-dkms
2、创建数据库
CREATE DATABASE cinder; GRANT ALL ON cinder.* TO 'cinderUser'@'%' IDENTIFIED BY 'cinderPass';
3、tgt和iscsitarget都是使用3260端口，两个服务不能共存，启动的时候tgt在iscsitarget之前启动，抢先占用3260端口，所以如果使用iscsitarget前需要停止tgt服务
service tgt stop为了防止操作系统重启后tgt占用iscsitarget端口，可以将tgt的启动文件删除或者修改名称
mv /etc/init/tgt.conf /etc/init/tgt.conf.disable配置iscsitarget可用
sed -i 's/false/true/g' /etc/default/iscsitarget
重启相关iSCSI服务
service iscsitarget restart service open-iscsi restart
4、修改/etc/cinder/api-paste.ini
[filter:authtoken] paste.filter_factory = keystone.middleware.auth_token:filter_factory service_protocol = http service_host = 10.68.19.61 service_port = 5000 auth_host = 10.68.20.61 auth_port = 35357 auth_protocol = http admin_tenant_name = service admin_user = cinder admin_password = service_pass signing_dir = /var/lib/cinder5、修改/etc/cinder/cinder.conf
[DEFAULT] rootwrap_config=/etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:cinderPass@10.68.20.61/cinder api_paste_config = /etc/cinder/api-paste.ini iscsi_helper=ietadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone iscsi_ip_address=10.68.20.61若是使用tgt，内容为：
[DEFAULT] rootwrap_config = /etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:cinderPass@10.68.19.61/cinder api_paste_confg = /etc/cinder/api-paste.ini iscsi_helper = tgtadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone state_path = /var/lib/cinder lock_path = /var/lock/cinder volumes_dir = /var/lib/cinder/volumes
6、数据库同步
cinder-manage db sync7、创建VG
pvcreate -ff /dev/sdb vgcreate cinder-volumes /dev/sdb8、重启所有的cinder服务
cd /etc/init.d/; for i in $( ls cinder-* ); do sudo service $i restart; done
常见问题
问题一、操作系统重启或者iscsitarget、tgt服务重启之后，cinder-volume服务无法启动，什么原因造成的，如何解决？原因分析：出现该问题的是由于iscsitarget和tgt发生冲突，通常是启动的服务与cinder-volume中配置使用的管理工具不一致造成，两个组件使用同一个端口，系统重启后tgt服务会先于iscsitarget启动，占用了3260端口导致iscsitarget启动失败，而此时若/etc/cinder/cinder.conf文件中的iscsi_helper属性为ietadm时，cinder-volume服务就无法启动，同样，若/etc/cinder/cinder.conf文件中的iscsi_helper属性为tgtadm，而启动的服务是iscsitarget，cinder-volume服务也无法启动。
解决办法：
1、使用lsof命令查看侦听3260端口的服务到底是iscsitarget还是tgt：
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ietd 13025 root 7u IPv4 18939 0t0 TCP *:3260 (LISTEN) ietd 13025 root 8u IPv6 18940 0t0 TCP *:3260 (LISTEN)ietd进程表示当前服务是iscsitarget，对应的iscsi_helper属性应该为ietadm。
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME tgtd 14564 root 4u IPv4 21457 0t0 TCP *:3260 (LISTEN) tgtd 14564 root 5u IPv6 21458 0t0 TCP *:3260 (LISTEN) tgtd 14565 root 4u IPv4 21457 0t0 TCP *:3260 (LISTEN) tgtd 14565 root 5u IPv6 21458 0t0 TCP *:3260 (LISTEN)tgtd进程表示当前服务是tgt，对应的iscsi_helper属性应该为tgtadm。
2、修改/etc/cinder/cinder.conf文件配置
ietd--iscsitarget--ietadm方案配置内容：
[DEFAULT] rootwrap_config=/etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:cinderPass@10.68.20.61/cinder api_paste_config = /etc/cinder/api-paste.ini iscsi_helper=ietadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone iscsi_ip_address=10.68.20.61tgtd--tgt--tgtadm方案配置内容：
rootwrap_config = /etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:cinderPass@10.68.19.61/cinder api_paste_confg = /etc/cinder/api-paste.ini iscsi_helper = tgtadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone state_path = /var/lib/cinder lock_path = /var/lock/cinder volumes_dir = /var/lib/cinder/volumes3、重启cinder-volume服务
service cinder-volume restart
问题二、系统重启或者iscsitarget服务重启的之后，创建第X个volume的时候失败，继续创建却没有问题，错误日志如下，是什么原因造成，如何解决？
2014-01-02 16:33:19 ERROR [cinder.volume.manager] volume volume-08cf713a-0d62-44c9-8bc0-8367783102f2: create failed
2014-01-02 16:33:19 ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data
rval = self.proxy.dispatch(ctxt, version, method, **args)
File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch
return getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 288, in create_volume
LOG.error(_("volume %s: create failed"), volume_ref['name'])
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 281, in create_volume
model_update = self.driver.create_export(context, volume_ref)
File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/lvm.py", line 496, in create_export
chap_auth)
File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 229, in create_iscsi_target
self._new_target(name, tid, **kwargs)
File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 288, in _new_target
**kwargs)
File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 73, in _run
self._execute(self._cmd, *args, run_as_root=True, **kwargs)
File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 190, in execute
cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf ietadm --op new --tid=5 --params Name=iqn.2010-10.org.openstack:volume-08cf713a-0d62-44c9-8bc0-8367783102f2
Exit code: 239
Stdout: ''
Stderr: 'File exists.\n'原因分析：iscsitarget服务重启的时候，会将target挂载重新进行排序并且会恢复部分已经删除的挂载信息，服务重启之前的挂载信息为：root@controller:/var/log/cinder# cat /proc/net/iet/volume
tid:4 name:iqn.2010-10.org.openstack:volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
lun:0 state:0 iotype:fileio iomode:wt blocks:8388608 blocksize:512 path:/dev/cinder-volumes/volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
tid:2 name:iqn.2010-10.org.openstack:volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
tid:1 name:iqn.2010-10.org.openstack:volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2root@controller:/var/log/cinder# cat /proc/net/iet/session
tid:4 name:iqn.2010-10.org.openstack:volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
sid:1125899977949696 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none
tid:2 name:iqn.2010-10.org.openstack:volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
sid:844424950907392 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none
tid:1 name:iqn.2010-10.org.openstack:volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
sid:281475031040512 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:nonetid=3的挂载盘已经删除，但是此时重新启动iscsitarget服务之后，会发现，ietadm会将原来删除掉的tid=3的挂载信息恢复，只不过不是活动状态，信息如下：root@controller:/var/log/cinder# cat /proc/net/iet/volume
tid:4 name:iqn.2010-10.org.openstack:volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
lun:0 state:0 iotype:fileio iomode:wt blocks:8388608 blocksize:512 path:/dev/cinder-volumes/volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
tid:3 name:iqn.2010-10.org.openstack:volume-42da15af-f951-4dec-bf54-3fe6a6240aac
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-42da15af-f951-4dec-bf54-3fe6a6240aac
tid:2 name:iqn.2010-10.org.openstack:volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
tid:1 name:iqn.2010-10.org.openstack:volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
lun:0 state:0 iotype:fileio iomode:wt blocks:33554432 blocksize:512 path:/dev/cinder-volumes/volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2root@controller:/var/log/cinder# cat /proc/net/iet/session
tid:4 name:iqn.2010-10.org.openstack:volume-0ae29ff3-e54f-4b54-be99-bd5fe28b44f8
sid:844425001239040 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none
tid:3 name:iqn.2010-10.org.openstack:volume-847be27e-7ed7-4fd2-bc5e-716c20051965
tid:2 name:iqn.2010-10.org.openstack:volume-c8e6b7d4-ce72-427a-aa4f-ed40e1d44104
sid:1125899927618048 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none
tid:1 name:iqn.2010-10.org.openstack:volume-12ca34c1-600f-421c-b300-6d1df8a1c4e2
sid:281475031040512 initiator:iqn.1993-08.org.debian:01:8cae6e47cc16
cid:0 ip:10.68.20.62 state:active hd:none dd:none此时，cinder数据库的iscsi_targets表中的id3的target的volume_id已经更新为null，再执行创建volume的时候，会执行命令：sudo cinder-rootwrap /etc/cinder/rootwrap.conf ietadm --op new --tid=3 --params Name=iqn.2010-10.org.openstack:volume-08cf713a-0d62-44c9-8bc0-8367783102f2
从而发生错误提示“File exists.”，这个tid=3的值是取自iscsi_targets表volume_id为null的主键最小值。
解决办法：
方法一、删除没有活动的挂载
ietadm --op delete --tid=3
说明：有的时候还会出现tid挂载错位，即ietadm中显示挂载了volume-xxx的tid为5，但是在cinder数据库的iscsi_targets表中却是6，这个时候需要手动进行调整
方法二、使用tgt替换iscsitarget
1、停止iscsitarget服务
service iscsitarget stop2、启动tgt服务
service tgt start
Rather than invoking init scripts through /etc/init.d, use the service(8) utility, e.g. service tgt stop initctl: Unknown job: tgt Since the script you are attempting to invoke has been converted to an Upstart job, you may also use the stop(8) utility, e.g. stop tgt检查tgt的启动文件是否正常/etc/init/tgt.conf，本文的安装过程中就曾将该文件改名防止其启动。
3、修改/etc/cinder/cinder.conf内容
rootwrap_config = /etc/cinder/rootwrap.conf sql_connection = mysql://cinderUser:cinderPass@10.68.19.61/cinder api_paste_confg = /etc/cinder/api-paste.ini iscsi_helper = tgtadm volume_name_template = volume-%s volume_group = cinder-volumes verbose = True auth_strategy = keystone state_path = /var/lib/cinder lock_path = /var/lock/cinder volumes_dir = /var/lib/cinder/volumes
评论暂时关闭