创建LVM条带

LVM的概念很简单,先要将物理卷加入LVM,并创建一个vg来管理物理卷,然后在vg上分逻辑分区叫lv,然后在逻辑分区上创建文件系统。

现状

一般是先将创建,再讲缩减,由于实际项目原因,我反过来了,先将缩减,再讲创建,接上篇,我们的磁盘信息如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
host ~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 119.2G 0 disk
|-sda1 8:1 0 500M 0 part /boot
|-sda2 8:2 0 50G 0 part /home/myvg
|-sda3 8:3 0 25G 0 part /
|-sda4 8:4 0 1K 0 part
|-sda5 8:5 0 20G 0 part /home/myvg/var/logs
|-sda6 8:6 0 10G 0 part /home/myvg/var/tmp
|-sda7 8:7 0 9.9G 0 part /ac
`-sda8 8:8 0 3.9G 0 part
sdb 8:16 0 931.5G 0 disk
sdc 8:32 0 1.8T 0 disk
sdd 8:48 0 3.7T 0 disk
`-myvg-data 253:0 0 3T 0 lvm /data
sde 8:64 0 1.8T 0 disk

sda是系统盘,sdb sdc sde三块磁盘空着,没有使用,而sdd是做成了一个LVM分区。

现在,我们准备将sdb, sdc, sde三块盘做成一个新的LVM分区,并做成条带化(striped)。

创建

pvcreate创建物理卷

1
2
3
4
host ~ # pvcreate /dev/sdb /dev/sdc /dev/sde
Physical volume "/dev/sdb" successfully created
Physical volume "/dev/sdc" successfully created
Physical volume "/dev/sde" successfully created

创建完后,用pvdisplay就可以看到这几块盘的信息。

vgcreate创建卷组

现在将几个PV一起组建一个新的VG,VG名字叫做data2:

1
2
host ~ # vgcreate data2 /dev/sdb /dev/sdc /dev/sde
Volume group "data2" successfully created

lvcreate创建逻辑分区

接下来,创建逻辑分区,逻辑分区采用条带,底层使用3个PV,创建3T的卷。

type参数,目前支持很多种类型,包括raid,thin等,我们测试条带使用striped参数,参数解释:

1
2
3
4
5
6
7
8
9
--type SegmentType
Creates a logical volume with the specified segment type. Supported types are: cache, cache-pool, error, linear, mirror, raid1, raid4, raid5_la, raid5_ls (= raid5), raid5_ra, raid5_rs, raid6_nc, raid6_nr, raid6_zr (= raid6), raid10, snapshot,
striped, thin, thin-pool or zero.
-I, --stripesize StripeSize
Gives the number of kilobytes for the granularity of the stripes.
StripeSize must be 2^n (n = 2 to 9) for metadata in LVM1 format. For metadata in LVM2 format, the stripe size may be a larger power of 2 but must not exceed the physical extent size.
-i, --stripes Stripes
Gives the number of stripes. This is equal to the number of physical volumes to scatter the logical volume. When creating a RAID 4/5/6 logical volume, the extra devices which are necessary for parity are internally accounted for. Specifying -i3
would use 3 devices for striped logical volumes, 4 devices for RAID 4/5, and 5 devices for RAID 6. Alternatively, RAID 4/5/6 will stripe across all PVs in the volume group or all of the PVs specified if the -i argument is omitted.

因为sdb的大小只有931.5G,不到1T,平均分布不下3个1T,所以创建失败:

1
2
host ~ # lvcreate -L 3T -i 3 -I 64 --type striped data2
Insufficient suitable allocatable extents for logical volume lvol0: 71031 more required

缩小一下,创建2.7T的卷:

1
2
3
4
host ~ # lvcreate -L 2.7T -i 3 -I 64 --type striped data2
Rounding up size to full physical extent 2.70 TiB
Rounding size (707789 extents) up to stripe boundary size (707790 extents).
Logical volume "lvol0" created.

查看一下创建好的lv,这里显示了两个,第一个是之前创建的,非条带类型,我们只需要看 /dev/data2/lvol0 接口,第一个/dev/myvg/data可以用来对比看下有什么不同:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
host ~ # lvdisplay -m
--- Logical volume ---
LV Path /dev/myvg/data
LV Name data
VG Name myvg
LV UUID 4YcW3O-NbZD-3zrR-olkO-CFQ9-xcJz-0vzYEZ
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2017-08-09 14:32:48 +0800
LV Status available
# open 1
LV Size 3.00 TiB
Current LE 786432
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
--- Segments ---
Logical extents 0 to 786431:
Type linear
Physical volume /dev/sdd
Physical extents 0 to 786431
--- Logical volume ---
LV Path /dev/data2/lvol0
LV Name lvol0
VG Name data2
LV UUID dobaVd-v9Fo-CS1l-t3Zf-YkJy-HCiz-7K8Oi1
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2017-11-16 12:07:01 +0800
LV Status available
# open 0
LV Size 2.70 TiB
Current LE 707790
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 768
Block device 253:1
--- Segments ---
Logical extents 0 to 707789:
Type striped
Stripes 3
Stripe size 64.00 KiB
Stripe 0:
Physical volume /dev/sdb
Physical extents 0 to 235929
Stripe 1:
Physical volume /dev/sdc
Physical extents 0 to 235929
Stripe 2:
Physical volume /dev/sde
Physical extents 0 to 235929

mke2fs创建文件系统

创建文件系统,关闭惰性初始化:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
host ~ # mke2fs -t ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/data2/lvol0
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=16 blocks, Stripe width=48 blocks
181198848 inodes, 724776960 blocks
36238848 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2873098240
22119 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

挂载分区

1
2
3
4
5
6
host ~ # mkdir /data2
host ~ # mount -t ext4 -o rw,noatime,nodiratime,data=ordered,journal_async_commit,inode_readahead_blks=512 /dev/data2/lvol0 /data2
host ~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/myvg-data 3.0T 784G 2.1T 28% /data
/dev/mapper/data2-lvol0 2.7T 89M 2.6T 1% /data2

LVM条带测试

上面我们创建的LVM,为3块磁盘组成条带,简单对比下(dd测试是不专业的,但是大致能看出问题)和非条带的方式,其性能有和差异:

8k

上图为bs=8k的情况。

  • 1,2对比,3,4对比,发现不管几个并发进程,读IO在条带方式下,都比非条带快。同理,5,6对比,7,8对比,写IO也类似。
  • 1,3对比,2,4对比,发现并发读,性能下降非常明显,5,7对比,6,8对比,写IO也类似。

1M

上图为bs=1M的情况。

  • 和8k的情况对比,IO利用充分,吞吐量随着bs增大而增大,例如8k read为110MB/s, 1M read为187MB/s。
  • 和8k的情况对比,并发导致的性能下降没有8k时那么明显。
  • bs足够大,磁盘的读写性能和条带底层磁盘个数成正比,但非线性叠加。

测试方法:

1
2
3
4
5
6
7
8
9
# sync; dd if=/dev/zero of=test.dd oflag=direct,nonblock bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 26.158 s, 401 MB/s
# sync; dd if=/dev/mapper/data2-lvol0 iflag=direct,nonblock of=/dev/null bs=8k count=1250000
1250000+0 records in
1250000+0 records out
10240000000 bytes (10 GB) copied, 70.0835 s, 146 MB/s

LVM条带优劣势分析

  • LVM条带的优势:

    1. 可以极大提高磁盘的IO性能。
    2. 磁盘空间相比liner方式无减少。
  • LVM条带劣势:

    1. 一块磁盘坏,所有数据全坏。
      为什么不做RAID5 OR RAID10?
      由于没有RAID卡,软RAID方案或者LVM目前已经支持的RAID方案,软RAID5的计算放到了CPU上,会加大CPU负担,而RAID10的方案太浪费空间,都不是很好的方案,加之同等磁盘数量下冗余RAID的性能根本就不如条带。
    2. 扩容难度加大(需要一次加入之前PV数量的磁盘或分区)。
    3. 依靠软件做负载,性能不如RAID卡,性能非线性增长。

磁盘IO分布相对均匀,如果某块磁盘性能差一些,那IO UTIL会高,但速度维持一致。