CentOS 7
Ads

Pacemaker : CLVM + GFS2
2015/07/08
 
Configure Storage cluster with CLVM + GFS2.
This example is based on the environment like follows.
                        +--------------------+
                        | [ Shared Storage ] |
                        |   ISCSI Target     |
                        +---------+----------+
                         10.0.0.30|
                                  |
+----------------------+          |          +----------------------+
| [       Node01     ] |10.0.0.51 | 10.0.0.52| [       Node02     ] |
|   node01.srv.world   +----------+----------+   node02.srv.world   |
|         CLVM         |                     |          CLVM        |
+----------------------+                     +----------------------+

[1]
[2]
Create iscsi shared satorages, refer to here.
It needs 2 shared storage devices, for data and for fence device.
This example uses "iqn.2015-07.world.server:storage.target01" for data, and "iqn.2015-07.world.server:fence.target00" for fence device.
[3]
Configure ISCSI initiater on all Nodes, refer to here.
It's OK not to create partitions on them yet.
[4] Install required packages on all Nodes.
[root@node01 ~]#
yum -y install fence-agents-all lvm2-cluster gfs2-utils
[root@node01 ~]#
lvmconf --enable-cluster

[root@node01 ~]#
reboot

[5] Configure fence device. It's OK to set on a node. The example "/dev/sda" below is just the device of shared storage.
# confirm fence device disk (it is set on sda on this example)

[root@node01 ~]#
cat /proc/partitions

major minor  #blocks  name
.....
.....
 253        2    1048576 dm-2
   8        0    1048576 sda
   8       16   20971520 sdb

# confirm disk's ID

[root@node01 ~]#
ll /dev/disk/by-id | grep sda

lrwxrwxrwx 1 root root  9 Jul 10 11:44 scsi-36001405189b893893594dffb3a2cb3e9 -> ../../sda
lrwxrwxrwx 1 root root  9 Jul 10 11:44 wwn-0x6001405189b893893594dffb3a2cb3e9 -> ../../sda

[root@node01 ~]#
pcs stonith create scsi-shooter fence_scsi devices=/dev/disk/by-id/wwn-0x6001405189b893893594dffb3a2cb3e9 meta provides=unfencing
[root@node01 ~]#
pcs property set no-quorum-policy=freeze

[root@node01 ~]#
pcs stonith show scsi-shooter

 Resource: scsi-shooter (class=stonith type=fence_scsi)
  Attributes: devices=/dev/disk/by-id/wwn-0x6001405189b893893594dffb3a2cb3e9
  Meta Attrs: provides=unfencing
  Operations: monitor interval=60s (scsi-shooter-monitor-interval-60s)
[6] Add required resources. It's OK to set on a node.
[root@node01 ~]#
pcs resource create dlm ocf:pacemaker:controld op monitor interval=30s on-fail=fence clone interleave=true ordered=true

[root@node01 ~]#
pcs resource create clvmd ocf:heartbeat:clvm op monitor interval=30s on-fail=fence clone interleave=true ordered=true

[root@node01 ~]#
pcs constraint order start dlm-clone then clvmd-clone

Adding dlm-clone clvmd-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@node01 ~]#
pcs constraint colocation add clvmd-clone with dlm-clone

[root@node01 ~]#
pcs status resources

 Clone Set: dlm-clone [dlm]
     Started: [ node01.srv.world node02.srv.world ]
 Clone Set: clvmd-clone [clvmd]
     Started: [ node01.srv.world node02.srv.world ]
[7] Create volumes on shared storage and format with GFS2. It's OK to set on a node. On this example, it is set on sdb and create partitions on it and set LVM type with fdisk.
[root@node01 ~]#
pvcreate /dev/sdb1

Physical volume "/dev/sdb1" successfully created
# create cluster volume group

[root@node01 ~]#
vgcreate -cy vg_cluster /dev/sdb1

Clustered volume group "vg_cluster" successfully created
[root@node01 ~]#
lvcreate -l100%FREE -n lv_cluster vg_cluster

Logical volume "lv_cluster" created.
[root@node01 ~]#
mkfs.gfs2 -p lock_dlm -t ha_cluster:gfs2 -j 2 /dev/vg_cluster/lv_cluster

/dev/vg_cluster/lv_cluster is a symbolic link to /dev/dm-3
This will destroy any data on /dev/dm-3
Are you sure you want to proceed? [y/n]
y

Device:                    /dev/vg_cluster/lv_cluster
Block size:                4096
Device size:               0.99 GB (260096 blocks)
Filesystem size:           0.99 GB (260092 blocks)
Journals:                  2
Resource groups:           5
Locking protocol:          "lock_dlm"
Lock table:                "ha_cluster:gfs2"
UUID:                      cdda1b15-8c57-67a1-481f-4ad3bbeb1b2f
[8] Add shared storage to cluster resource. It's OK to set on a node.
[root@node01 ~]#
pcs resource create fs_gfs2 Filesystem \
device="/dev/vg_cluster/lv_cluster" directory="/mnt" fstype="gfs2" \
options="noatime,nodiratime" op monitor interval=10s on-fail=fence clone interleave=true
[root@node01 ~]#
pcs resource show

 Clone Set: dlm-clone [dlm]
     Started: [ node01.srv.world ]
     Stopped: [ node02.srv.world ]
 Clone Set: clvmd-clone [clvmd]
     Started: [ node01.srv.world ]
     Stopped: [ node02.srv.world ]
 Clone Set: fs_gfs2-clone [fs_gfs2]
     Started: [ node01.srv.world ]

[root@node01 ~]#
pcs constraint order start clvmd-clone then fs_gfs2-clone

Adding clvmd-clone fs_gfs2-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@node01 ~]#
pcs constraint colocation add fs_gfs2-clone with clvmd-clone

[root@node01 ~]#
pcs constraint show

Location Constraints:
Ordering Constraints:
  start dlm-clone then start clvmd-clone (kind:Mandatory)
  start clvmd-clone then start fs_gfs2-clone (kind:Mandatory)
Colocation Constraints:
  clvmd-clone with dlm-clone (score:INFINITY)
  fs_gfs2-clone with clvmd-clone (score:INFINITY)
[9] It's OK all. Make sure GFS2 filesystem is mounted on an active node and also make sure GFS2 mounts will move to another node if current active node will be down.
[root@node01 ~]#
df -hT

Filesystem                        Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root           xfs        27G  1.1G   26G   4% /
devtmpfs                          devtmpfs  2.0G     0  2.0G   0% /dev
tmpfs                             tmpfs     2.0G   76M  1.9G   4% /dev/shm
tmpfs                             tmpfs     2.0G  8.4M  2.0G   1% /run
tmpfs                             tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1                         xfs       497M  126M  371M  26% /boot
/dev/mapper/vg_cluster-lv_cluster gfs2     1016M  259M  758M  26% /mnt
 
Tweet