How to Combine AWS Ephemeral Storage by Using RAID 0

As we know, ephemeral storage is SSD storage which is provided by AWS freely with specific higher configuration of instances. It can be used to get the better disk I/O, as this storage may lose data if instances stop. So we never use this ephemeral storage for important data.

Below are the use cases where this combination can be used.

1. While uploading static content, we generally, we use CDN to serve the static content. First the Static content needs to be uploaded to the server, then it has to be synced to S3.  Here we can use ephemeral storage as upload location.
2. Can also be used only to store temporary data or SWAP (virtual memory) files.
3. Can be used for application logs, cache.
4. In case you need to format with different file-system types (ext3, ext4) to test something.

Some of the instances have two volumes of ephemeral storage. We can combine them with RAID technology (RAID 0 – striped volume). Refer below steps for the same. This is scripted as it would be better to use in case we have to use both volumes of ephemeral storage as a single disk.

1. Create File system (xvdb & xvdc): In this blog, we have two Volumes attached with an instance. (each 1 GB)

1.1. Check the volumes which are available (xvdb & xvdc)

[root@ip-172-31-35-168 ~]# lsblk
 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
 xvda 202:0 0 8G 0 disk
 └─xvda1 202:1 0 8G 0 part /
 xvdb 202:16 0 1G 0 disk 
 xvdc 202:32 0 1G 0 disk

1.2.  As per above output, there is no file system for xvdb and xvdc,  First create files system here for xvdb

1.3. Do above steps for  xvdc

1.4. Now we can see file system that has been created for both Volumes (xvdb1 & xvdc1)

[root@ip-172-31-35-168 ~]# lsblk
 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
 xvda 202:0 0 8G 0 disk
 └─xvda1 202:1 0 8G 0 part /
 xvdb 202:16 0 1G 0 disk
 └─xvdb1 202:17 0 1023M 0 part
 xvdc 202:32 0 1G 0 disk
 └─xvdc1 202:33 0 1023M 0 part 

2. Create RAID 0 (striped): – Combine both of them as a single Volume.

2.1.  Install the mdadm packages

// yum install mdadm
[root@ip-172-31-35-168 ~]# mdadm --create --verbose --auto=yes /dev/md0 --level=0 --raid-devices=2 /dev/xvdb1 /dev/xvdc1
 mdadm: chunk size defaults to 512K
 mdadm: Defaulting to version 1.2 metadata
 mdadm: array /dev/md0 started.

2.2. Check the details of Volumes

[root@ip-172-31-35-168 ~]# mdadm --details /dev/md0
mdadm: unrecognized option '--details'
Usage: mdadm --help
 for help
[root@ip-172-31-35-168 ~]# mdadm --detail /dev/md0
/dev/md0:
 Version : 1.2
 Creation Time : Mon Dec 19 06:15:53 2016
 Raid Level : raid0
 Array Size : 2094080 (2045.34 MiB 2144.34 MB)
 Raid Devices : 2
 Total Devices : 2
 Persistence : Superblock is persistent

 Update Time : Mon Dec 19 06:15:53 2016
 State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
 Spare Devices : 0

 Chunk Size : 512K

 Name : ip-172-31-35-168:0 (local to host ip-172-31-35-168)
 UUID : 446d819b:7bedab09:68580914:5d433e75
 Events : 0

 Number Major Minor RaidDevice State
 0 202 17 0 active sync /dev/sdb1
 1 202 33 1 active sync /dev/sdc1

2.3. Verify again – We can see each filesystem belongs to md0 and showing 2 GB of space

[root@ip-172-31-35-168 ~]# lsblk
 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
 xvda 202:0 0 8G 0 disk
 └─xvda1 202:1 0 8G 0 part /
 xvdb 202:16 0 1G 0 disk
 └─xvdb1 202:17 0 1023M 0 part
 └─md0 9:0 0 2G 0 raid0
 xvdc 202:32 0 1G 0 disk
 └─xvdc1 202:33 0 1023M 0 part
 └─md0 9:0 0 2G 0 raid0

3. Format and mount the filesystem (/dev/md0)

// To Format
[root@ip-172-31-35-168 ~]# mkfs.ext4  /dev/md0

// To Mount temporary
[root@ip-172-31-35-168 ~]# mount /dev/md0 /mnt

// To Mount Permanently
echo /dev/md0    /mnt        ext4    defaults        0   0 >> /etc/fstab

 4. To automate above steps in case of autoscaling

 4.1. Script to create partition and RAID.

$ vim create_ephemeral_raid_md0.sh
#!/bin/sh
# Script to create software RAID0

DISK1="/dev/xvdb"
DISK2="/dev/xvdc"

/sbin/fdisk $DISK1 < /root/admin_scripts/partition_create_options

/sbin/fdisk $DISK2 < /root/admin_scripts/partition_create_options

/sbin/mdadm --create --verbose --auto=yes /dev/md0 --level=0 --raid-devices=2 /dev/xvdb1 /dev/xvdc1

/sbin/mkfs.ext4  /dev/md0

 “partition_create_options” is a file that contains all options which are needed to create a file system.

 4.2. Make an entry to auto start this script at boot

// Add a cron at system boot @reboot /bin/sh <path of script>/create_ephemeral_raid_md0.sh

I hope the below blog helps the AWS DevOps engineers to use ephemeral storage.  I will explain the use cases in a more descriptive way in my next blog.