ceph-ansible部署L版ceph 及 通过iscsi 共享rbd 对接xencenter
仅供参考： 这里要注意python的版本 软件源的匹配
https://blog.csdn.net/cojn52/article/details/85793902
服务器端
1 准备镜像
镜像centos7.5 1804
2 关闭防火墙 selinux
systemctl status firewalld
ystemctl stop firewalld
getenforce #查看selinux 状态
setenforce 0 #临时关闭seLinux
也可以永久关闭selinux 但是需要重启
3 设置主机名
4 安装 Ansible git vim
5 从git 上下载ceph-ansible
6 进入ceph-ansible 目录
git tag
git checkout v3.2.0
git branch
7 在ceph-ansible 目录
vim hosts
8 给免密操作
ssh-keygen
8在ceph-ansible 目录 进入group_vars 编辑 all.yml
9 添加网络源
在/etc/yum.repo.d/在原有基础上添加网络源
9.1、 这个是ceph的yum源 前提是必须要有epel源解决依赖问题
[root@ceph-1 yum.repos.d]#vim ceph_stable.repo
[ceph_stable]
baseurl = http://mirrors.163.com/ceph/rpm-luminous/el7/$basearch
gpgcheck = 1
gpgkey = http://mirrors.163.com/ceph/keys/release.asc
name = Ceph Stable repo
9.2、[root@ceph-1 yum.repos.d]# vim elrepo.repo
### Name: ELRepo.org Community Enterprise Linux Repository for el7
### URL: http://elrepo.org/
[elrepo]
name=ELRepo.org Community Enterprise Linux Repository - el7
baseurl=http://elrepo.org/linux/elrepo/el7/$basearch/
http://mirrors.coreix.net/elrepo/elrepo/el7/$basearch/
http://mirror.rackspace.com/elrepo/elrepo/el7/$basearch/
http://repos.lax-noc.com/elrepo/elrepo/el7/$basearch/
http://mirror.ventraip.net.au/elrepo/elrepo/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo.el7
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0
[elrepo-testing]
name=ELRepo.org Community Enterprise Linux Testing Repository - el7
baseurl=http://elrepo.org/linux/testing/el7/$basearch/
http://mirrors.coreix.net/elrepo/testing/el7/$basearch/
http://mirror.rackspace.com/elrepo/testing/el7/$basearch/
http://repos.lax-noc.com/elrepo/testing/el7/$basearch/
http://mirror.ventraip.net.au/elrepo/testing/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-testing.el7
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0
[elrepo-kernel]
name=ELRepo.org Community Enterprise Linux Kernel Repository - el7
baseurl=http://elrepo.org/linux/kernel/el7/$basearch/
http://mirrors.coreix.net/elrepo/kernel/el7/$basearch/
http://mirror.rackspace.com/elrepo/kernel/el7/$basearch/
http://repos.lax-noc.com/elrepo/kernel/el7/$basearch/
http://mirror.ventraip.net.au/elrepo/kernel/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-kernel.el7
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0
[elrepo-extras]
name=ELRepo.org Community Enterprise Linux Extras Repository - el7
baseurl=http://elrepo.org/linux/extras/el7/$basearch/
http://mirrors.coreix.net/elrepo/extras/el7/$basearch/
http://mirror.rackspace.com/elrepo/extras/el7/$basearch/
http://repos.lax-noc.com/elrepo/extras/el7/$basearch/
http://mirror.ventraip.net.au/elrepo/extras/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-extras.el7
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0
9.3、[root@ceph-1 yum.repos.d]# vim epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
9.4、 [root@ceph-1 yum.repos.d]# vim epel-testing.repo
[epel-testing]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-testing-debuginfo]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-debug-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
[epel-testing-source]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-source-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
10安装依赖包 不安装会报错丢失notario
yum -y install epel-release
yum -y install python-pip
pip install notario
11 部署ceph
在ceph-ansible目录下执行
ansible-playbook site.yml.sample -i hosts
跑完之后
12 部署osd
12.1 安装ceph-osd安装包
yum install ceph-osd –y
12.2 查看磁盘
12.3 格式化磁盘
ceph-disk prepare --bluestore /dev/sdb
ceph-disk prepare --bluestore /dev/sdc
ceph-disk prepare --bluestore /dev/sdd
12.4 给ceph添加osd盘
ceph-disk prepare --bluestore /dev/sdb
ceph-disk prepare --bluestore /dev/sdc
ceph-disk prepare --bluestore /dev/sd
13 ceph部署完成
14 创建rbd 块
14.1 ceph osd pool create rbd 512 创建rbd
14.2 rbd create volume -p rbd --size 1T 指定大小
15 TCMU实现ISCSI接口
注：大部分对接失败的原因就是没有通过tcmu 实现iscsi的借口
15.1 在tmp目录下载tcmp代码
cd /tmp/
git clone https://github.com/open-iscsi/tcmu-runner.git
15.2 安装tcmu
(1) cd tcmu-runner/
(2) ./extra/install_dep.sh
(3) cmake -Dwith-glfs=false -Dwith-qcow=false -DSUPPORT_SYSTEMD=ON -DCMAKE_INSTALL_PREFIX=/usr .
(4) make
(5) make install
(6) 启动服务tcmu
systemctl status tcmu-runner
systemctl start tcmu-runner
systemctl status tcmu-runner
systemctl enable tcmu-runner
16 安装ISCSI targetcli
yum install targetcli –y
17 配置iscsi 服务器端
[root@ceph-1 ~]# targetcli
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> cd backstores/user:rbd/
/backstores/user:rbd> create cfgstring=rbd/volume name=disk size=1T
/backstores/user:rbd> cd /
/> cd iscsi
/iscsi> create iqn.2019-01.redhat.ceph
/iscsi> cd iqn.2019-01.redhat.ceph/tpg1/luns
/iscsi/iqn.20...eph/tpg1/luns> create /backstores/user:rbd/disk
/iscsi/iqn.20...eph/tpg1/luns> cd /
/> cd iscsi/iqn.2019-01.redhat.ceph/tpg1/
/iscsi/iqn.20...hat.ceph/tpg1> set attribute authentication=0 demo_mode_write_protect=0
/iscsi/iqn.20...hat.ceph/tpg1> cd /
/> cd iscsi/iqn.2019-01.redhat.ceph/tpg1/portals/ # 保持不变
/iscsi/iqn.20.../tpg1/portals> exit
18、安装iscsi 软件包
yum install iscsi-initiator-utils
19、查看是否共享成功
iscsiadm -m discovery -t sendtargets -p 10.22.20.25
客户端
- 通过ssh或者界面console进入平台
- 检查是否安装iscsi安装包如果没有进行安装
yum install iscsi-initiator-utils -y
3、查看是否能查看到共享成功
iscsiadm -m discovery -t sendtargets -p 10.22.20.25
- 修改vim /etc/iscsi/initiatorname.iscsi配置文件并重启服务
iscsiadm -m discovery -t sendtargets -p 10.22.20.25的输出
InitiatorName=10.22.20.25:3260,1 iqn.2019-01.redhat.ceph
InitiatorAlias=xenserver
systemctl restart iscsid
- 通过xencenter界面挂载iscsi
至此客户端挂载成功下面测试是否可用