bigjocker's den

EBS backed LVM volumes on Amazon EC2

We have a very large monolithic database that we’ve recently migrated into Amazon EC2. This DB has been growing quite fast these past weeks, and we were about to hit the 1TB cap for EBS volumes on Amazon EC2. I’m not quite sure why do they have this limit, given the size of Amazon’s infrastructure, or why doesn’t Amazon offer an official alternative (like RAID1 via the Web Interface) to overcome this limitation. I’m quite sure this is a problem that is affecting a lot of customers, and attacking it would seem a priority to me.

Anyways, we need a logical volume that can hold over 1TB of data. Our first approach was to create several 1TB EBS disks, and use symlinks for some directories. But this does not scale, and we would have had to look for another solution in the short term: one of our directories has over 600.000 files, almost 900GB and growing about 20GB a week, so the 1TB limit would have been reached very shortly.

We had two options: RAID or LVM. Although RAID is the more standard way of approaching these cases, I read a lot of posts on Amazon’s forums about issues with snapshotting and recovering the data. Also, growing a RAID array on the fly is not a trivial task. So we decided to use LVM, and it’s been really easy so far.

The first step is to create the EBS disks, in our case, we created two 1TB disks, and attached them to our instance as /dev/sdf and /dev/sdh. Then we used the pvcreate utility to initialize these disks to be used by LVM:

root@backend:/mnt# pvcreate /dev/sdf /dev/sdh

Then we create a volume group called vgebs. This volume group includes our two disks:

root@backend:/mnt# vgcreate vgebs /dev/sdf /dev/sdh

We then create the logical volume, and tell the LVM subsystem to use all the free space (in our case we just want one volume that spans the whole array of disks):

root@backend:/mnt# lvcreate -i 2 -I 2M -l 100%FREE -n lvebs vgebs

This will create a new device located at /dev/mapper/vgebs-lvebs. This is our LVM volume, and we can now create a filesystem and mount it:

root@backend:/mnt# mkfs.ext3 /dev/mapper/vgebs-lvebs
root@backend:/mnt# mount /dev/mapper/vgebs-lvebs /mnt/disk3

And that’s it, we now have a LVM volume with a size of 2.0TB:

root@backend:/mnt# df -h /mnt/disk3
/dev/mapper/vgebs-lvebs
                   2.0T   48M  2.0T   0% /mnt/disk3
root@backend:/mnt#

There are a couple of things to note: when snapshotting the EBS volumes we must make sure that all members of the LVM array are in a consistent state, that is, there can not be writes being performed on any of the disks while we snapshot any of them. To accomplish this we must suspend all operations on the volume, snapshot, and resume operations:

root@backend:/mnt# dmsetup suspend vgebs-lvebs
root@backend:/mnt# python /opt/bigjocker/manage_snapshots.py vol-9e7a4886 3 'LVM disk 1'
root@backend:/mnt# python /opt/bigjocker/manage_snapshots.py vol-8454fc1c 3 'LVM disk 2'
root@backend:/mnt# dmsetup resume vgebs-lvebs

The snapshots will then be consistent with each other, and the LVM array can be rebuilt with lvscan mounted on another instance (never the same instance! unless you’ve detached the original LVM array, and even then, you should use another instance) using two EBS volumes created from the snapshots.

8 comments for “EBS backed LVM volumes on Amazon EC2

  1. Hlopezvg
    June 23, 2012 at 3:21 pm

    Maybe another combination would be LVM + XFS, for the “growing” part you mentioned on the post.

  2. Hlopezvg
    June 23, 2012 at 7:51 pm

    Maybe another combination would be LVM + XFS, for the “growing” part you mentioned on the post.

  3. james
    December 21, 2014 at 7:06 am

    This worked great. I am having a hell of a time extending a volume after this has been finished. I run into errors such as

    Insufficient suitable allocatable extents for logical volume stripe1: 126720 more required

    have you been able to add additional volumes past this?

  4. james
    December 21, 2014 at 7:55 am

    To be more clear.

    I was able to create a VG with TB successfully. My question is after you fill up the 2TBS, have you been able to add another volume and extend past it?

  5. james
    December 21, 2014 at 8:16 am

    I figured it out! Looks like you have to add EBS in pairs of two.

    I added two new volumes without an issue. Then ran resize2fs with success.

  6. james
    January 8, 2015 at 6:35 pm

    Could you explain a bit more on managing snapshots?

    root@backend:/mnt# python /opt/bigjocker/manage_snapshots.py vol-9e7a4886 3 ‘LVM disk 1’
    root@backend:/mnt# python /opt/bigjocker/manage_snapshots.py vol-8454fc1c 3 ‘LVM disk 2’

    I am to the point where I would like to snapshot my LVM array and not sure what you have going on here. Would you be willing to share you manage_snapshots.py?

    Or will I be able to use AWS Dashboard to snapshot it? I am using four ebs volumes after following your instructions

  7. james
    January 16, 2015 at 6:43 pm

    Hi there,
    Looks like my last comment did not make it. I was hoping to learn more about the snapshot scripts you are using. Any direction would be great.

    Using your process I have mounted 4TB of EBS storage to my instance. I am now ready to begin the backup phase.

Leave a Reply

Your email address will not be published. Required fields are marked *