Structured Thought . ORG

EBS Volumes

EBS (Elastic Block Storage) is a way to attach disk volumes to your running EC2 instances. The main difference between EBS volumes and the default ephemeral storage that is attached to EC2 instances by default is that EBS volumes persist beyond reboots and can also be moved between instances without losing the data contained within.

EBS Volumes are billed out in 1GB increments at $.10/GB/month. This is not per usage, this is per allocated GB. For example, if you have a 10GB volume allocated and only 1GB of it is in use, you’re still paying $1/month.

For the purposes of this example, we’ll start with 2 1GB volumes ($.20/month for allocation) which will be mirrored for redundancy at the OS level. This will ensure that a failure of one of the volumes will not affect the operation.

Allocating an EBS Volume

NOTE: For the purposes of this example, the architecture of the running instance is irrelevant. Therefore, to save myself some cost while putting together this part of the demo, I’ll be using a 32-bit ’small’ instance. This will not affect the operation, and will work identically to the 64-bit ‘large’ instance used in the previous examples.

Before you create a volume, you need to determine which availability zone your instance is running within. Think of an availability zone as a server room. To check this, click on the row in your Instances section of the AWS console, and look for the ‘Zone’ entry in the panel below. It should read something like ‘us-east-1d’. Take note of this zone, as it will be required for assigning your EBS volumes (Volumes can only be attached to instances within the same availability zone).

To start the process, click the ‘Volumes’ shortcut under the ‘Elastic Block Store’ header in your AWS console. The default view will show no volumes.

To create a volume, click on the ‘Create Volume’ button in the upper left. You will be presented with a menu asking for the size and availability zone. The size can be anywhere from 1GiB to a theoretical unlimited TiB. In this case, choose 1GiB, and pick the availability zone that matches your running instance. Do this twice, so you end up with 2 1GiB volumes.

NOTE: You may not be familiar with the GiB and TiB nomenclature. For our purposes they are the same as GB and TB, but they do have a significant difference. A TB refers to 1,000 GB, which is 1,000MB, which is 1,000KB, which is 1,000 bytes. A TiB is the binary-counting equivalent, meaning: a TiB is 1,024GiB, which is 1,024MiB, which is 1,024KiB, which is 1,024 bytes. So while a TB is 1,000,000,000,000 bytes, a TiB is 1,099,511,627,776, an eventual difference of 99.5 Billion bytes.

Once the volumes have been created, it’s time to attach them to the system. If you haven’t already, launch an ssh session to your running instance and issue the command ‘tail -f /var/log/messages’. This will allow you to see when and how the system detects that the volumes have been attached.

To perform the attachment, select the checkbox next to one of the volumes and click the ‘Attach Volume’ button that becomes available. You will be provided with the option of which instance to attach the volume. There should be only one. Choose the next available device name (/dev/sd* in the list). If you don’t see your running instance, be sure you created the volumes in the proper availability zone. Do this for both volumes, making sure that you have different device names for each.

Configuring the Volumes

When the EBS volumes are created, they are just like brand new disks out of the box, completely empty, and without any information whatsoever. Since we’re going to be using them as PVs (Physical Volumes) within LVM, this is ok, as they’ll be initialized properly when we start to use them. The first of these steps is to tell LVM that we have 2 new PVs we’d like to use. To do so, issue the following commands (assuming /dev/sdf and /dev/sdg for our new EBS volumes):

• sudo pvcreate /dev/sdf
• sudo pvcreate /dev/sdg

Next, we’ll want to create a VG (Volume Group) out of these new PVs, which will allow us to then start creating usable file space. To create a VG, issue the following command

• sudo vgcreate datavg1 /dev/sdf /dev/sdg

Next, we need to create an LV (Logical Volume) within the VG, which will house our filesystem. In this case, we want to create a roughly 1GB LV that is mirrored (2 copies, one on each physical volume). This will take up both of the EBS volumes we attached to the server earlier. To create this mirrored LV, issue the following command

• sudo lvcreate -L 1000M -n datalv1 -m 1 datavg1 –corelog

This creates a 1000MB volume named ‘datalv1′ with 1 mirrored device (2 total copies) within the datavg1 VG. The ‘–corelog’ option tells LVM to manage logging (the system keeps track of changes made to the LV) internally to the LV, otherwise, a third device would need to be used for logging. We can’t use a size of 1GB (-L 1G), because of the ‘–corelog’ option, which requires some space within the LV. This leaves a little bit of unused space within the VG, but it’s negligible.

Using the Volume

Finally, we need to create a filesystem within the LV that will allow us to actually use the space. To do so, issue the following command:

• sudo mke2fs -j /dev/datavg1/datalv1

This will create an ext3 filesystem (standard within the Linux world, but you may update the command to use whichever filesystem type you desire) that fills up the entire LV called ‘datalv1′. As a test, you can create a mount point, which is simply an empty directory, and mount the file system, this will show you your available space in the volume.

If you’re not planning on utilizing these volumes right away, you can essentially reverse the process to unmount the filesystem, delete the LV, remove the VG, detach the EBS Volumes, and finally remove the EBS volumes. I recommend you do this until you have the process down and can get things running in more-or-less one sitting.

Elastic IPs

Fortunately, this section is rather straightforward, and will just give an overview of how Elastic IPs work, and how to attach them to a running instance. Later, I will show you how to create a nameserver, and then begin running resources through your Elastic IP into your running instance.

To allocate an Elastic IP for your use, click the ‘Elastic IPs’ option under the ‘Networking & Security’ heading in your AWS console.

Since you won’t already have any Elastic IPs allocated, you will be asked to create one right away. Click the button you’re presented with, and you will see an IP appear almost instantly.

NOTE: Elastic IP usage is billed differently than other AWS resources. Since IPs themselves are relatively inexpensive, Amazon doesn’t charge you for using them. But, to keep people from hoarding IPs and not using them, Amazon charges $.01/hour for any Elastic IPs allocated that are not being actively used. While this isn’t much in the short-term, keep aware, as this can add up if you’re not watching.

To attach this IP to your running instance, simply click the checkbox next to the new IP, then click the ‘Associate’ button that becomes available. Again, since you are only running one instance at this time, you should be presented with only one option. Click ‘Associate’ in the menu, and then return to your instances display in your AWS console. After a few seconds of clicking ‘Refresh’ (this could take up to 10 minutes, according to Amazon, but I’ve never seen it take more than 2 minutes), you should see the ‘Public DNS’ item change to match your new IP.

At this point, you should be able to access your instance via SSH to the new IP address, rather than the old one.

That’s it for dealing with Elastic IPs. In the next installment, we’ll work with setting up our hosting control systems, including configuring a domain and DNS.

Now that you have all the prerequisites together from the previous article, it’s time to set up the target instance that will be used for the rest of these articles.

Starting the base AMI

For managing our server instances, EBS volumes, and Elastic IPs, we will be using the Amazon AWS Management console, which provides a nice web-based front-end to all of the necessary functions.

The first step is to choose the AMIs menu item on the left of the console window. An AMI is a base image that can be booted into an EC2 instance. Think of it as a backup of a running system that can be restored to another system in the event of a failure. This link will bring up a selection list of available AMIs, which will be used to create our default instance.

The initial list of available AMIs you are shown covers all public AMI images. Any one of these can be booted into an instance. We are going to filter this list so we see only the types of images we want. To do so, click the dropdown arrow next to the ‘All Platforms’ item, and choose ‘Ubuntu’. This will filter the list for only Ubuntu-based AMIs. Then, in the filter box, type ‘intrepid’. This will further filter the list so that we only see the most recent fully-released version of Ubuntu, titled ‘Intrepid Ibex’. Check the box next to the AMI with the ID ‘ami-255bbc4c’.

To start an instance based off of this AMI, check the box and click the ‘Launch’ button at the top of the display window.

NOTE: Using Amazon EC2 is NOT FREE. When you launch an instance, you will start the clock, which will run at $.40/hour in the US (other zones vary) in the case of this configuration. While not initially expensive, if you forget to stop an instance, the cost can add up quickly.

When you launch the instance based off of the chosen AMI, you will be presented with a menu. Since we’re launching an x86_64 instance, your choice is a ‘Large’ or ‘Extra Large’ configuration. In rough terms, the ‘Extra Large’ instance is twice as powerful as the ‘Large’ instance. It also carries twice the cost, so beware. If you feel you’ll be able to utilize the additional horsepower and memory, feel free to use the ‘Extra Large’ instance, as it does not affect the configuration steps.

You will also be asked to choose or create a keypair, which will allow you to do the initial connections via SSH. In the example configuration, we will not be using (and will actually be removing the requirement) a keypair, but we need it initially to access the instance after it has been started. Once you create a keypair, you need to download your personal certificate, which will allow you to access your running instance. If you lose this file, there is no way to retrieve it, and you will need to create a new pair.

You also have the option of creating and using Security Groups, which allow you to segment your EC2 virtual network for security and operational customization. For the purpose of this tutorial, we are only using one server, and therefore do not have a need for custom Security Groups.

After launching your instance, you will be presented with your Instance status window. The rotating icon on the left indicates that the status window will update automatically. If you change to a different view, and then come back to the instance status window, that icon will switch to a checkbox, and the status will not update automatically. At this point, you can update manually by clicking the ‘refresh’ button in the upper right of the instances window.

When your instance is started, you will see a status of ‘Running’, and an entry in the ‘Public DNS’ field.

Customizing the instance – Pass 1

To access your newly started instance, use your favorite SSH program (I personally use Putty). You will need to utilize your downloaded key file. Upon login, use the id ‘ubuntu’. You will not be prompted for a password, as the key file acts as the pass.

The first thing we’re going to do is remove the passkey portion from the SSH configuration. Since we’re going to be configuring this server for multiple client use, we want them to use passwords. To change this, edit the file /etc/ssh/sshd_config file using the command ’sudo vi /etc/ssh/sshd_config’. This will open the file in the vi editor with root authority, allowing you to make changes. To enable password authentication, change the line that reads ‘PasswordAuthentication no’ to ‘PasswordAuthentication yes’.

Before restarting the ssh daemon, which implements the change, we want to set a password for the Ubuntu user so we don’t lose our ability to connect. To do so, use the command ’sudo passwd ubuntu’. This will allow you to set a new password for the ‘ubuntu’ user.

Once the password is changed, restart the sshd daemon using the following command ’sudo /etc/init.d/ssh restart’. Before closing the existing connection, be sure to connect via a new session, using your new password for the ‘ubuntu’ user. Once you are confident you can connect reliably, feel free to close the initial window that used the keypair for authentication.

The next thing we want to do is update the system to the latest patch levels. Since the AMI was built when Intrepid Ibex was released, we want to make sure any existing security issues have been repaired. To check for new updates, use the command ’sudo apt-get update’. When the command completes (should only take a few seconds at the most), use the command ’sudo apt-get upgrade’, which will start the upgrade process. The system will check against the updated package list, and build a list of packages to be updated. When it is ready, it will ask you if you wish to update. Type ‘Y’ to confirm.

NOTE: EC2 instances are not static. If you shut down your running instance without following the next steps to save your configuration, you will need to go through the beginning of this article again.

Finally, to prep the system for the LVM (Logical Volume Management) setup we’ll be using for user data, LVM packages must be installed. To install LVM, simply execute the command ’sudo apt-get install lvm2′

Archiving the instance – Pass 1

Once the updates are complete, it is time to do the first ‘backup’ of the running instance into a usable AMI in your personal S3 storage location.

If you do not already have one, you will need to create an x.509 certificate keypair, which will allow you to upload your image to S3 and keep it for storage.

EC2 creates what is referred to as an ‘ephemeral’ storage location which allows you to store temporary data, which is ideal for creating our AMI. The storage is mounted by default at /mnt. We will be using this storage as our default location for the next several steps. To access it, simply type ‘cd /mnt’.

Next, we want to create an image of the running instance in an archive format that is understood by EC2 and can be registered as an AMI that can further be launched as a new instance.

To enable the bundling of an image and registering as an AMI, we need to make sure the appropriate x.509 keypair exists on the system. Upload the two files you should have downloaded when you created your certificate to the /mnt folder on the instance. If you bundle the image with incorrect keypair information, you will be unable to unbundle and register an AMI.

The default location for bundling is in the /tmp directory, which unfortunately exists under / in the default EC2 configuration. This is not a big problem, necessarily, but it necessitates us choosing a different default location as more information is added to the running system. Since the ephemeral storage mounted in /mnt is about 400GB, that’s plenty of space for our needs. To keep things separated during the bundling process, create a folder called ‘bundle’ under the /mnt directory using the command ’sudo mkdir /mnt/bundle’.

To bundle the running image for upload to S3, you will issue the command ’sudo ec2-bundle-vol -k /mnt/pk-<your private key string>.pem -c /mnt/cert-<your cert string>.pem -d /mnt/bundle -s 1000 -u <your account number>’. The account number is your Amazon AWS account number without the dashes. When you hit enter, confirm that you are creating an x86_64 instance, and the program will chug along, creating an image of instance, and placing the files in /mnt/bundle.

With this nice powerful instance, the bundling process takes about two minutes. When it completes, you will see a list of image.part.x files in the /mnt/bundle folder, along with a file called ‘image.manifest.xml’. This is the file you will use for uploading your image to S3 for storage.

To upload the files to an S3 bucket, issue the command ’sudo ec2-upload-bundle -b my-bucket -m image.manifest -a my-aws-access-key-id -s my-secret-key-id’ from the /mnt/bundle folder. The bucket must be an existing bucket within your S3 store. The access key and secret key for your S3 account can be found on your AWS account page.

When complete, you should be able to see the image.part files and your image.manifest.xml in your previously-mentioned S3 bucket.

To register your newly bundled AMI, return to your AWS EC2 console, and click the AMIs entry again in the left. At the top of the AMI display window, you will see a button labeled ‘Register New AMI’. The popup starts the URL for your manifest file. All you need to do is fill in the name of the bucket and the filename. The resulting URL should look like ‘http://s3.amazonaws.com:80/my-bucket/image.manifest.xml’. The registration process should be almost instantaneous. If you receive any errors, go back over the previous steps and make sure your keypairs and other identifying information match appropriately.

Before shutting down the previous instance, we need to boot the new instance and make sure that our configuration and updates have been carried over. In the AMI window of your console, you should already be set to view ‘Owned By Me’, which should show you your new AMI. If not, click the dropdown and select ‘Owned By Me’.

Testing the new custom instance

Launch the AMI into a new instance using no keypair and wait until you see a public DNS entry in your console window.

If you can access your new instance appropriately (using ‘ubuntu’ and the password you created), and you see that the instance has it’s updates (’sudo apt-get update’ shows no available updates), you can safely shut down both instances by checking their boxes and selecting the ‘Terminate Instance’ option in the instance display window.

Next time, we’ll look at configuring EBS volumes and Elastic IPs.

Amazon’s Elastic Compute Cloud (EC2) has proven itself to be a fast, stable, inexpensive way to create and manage servers on the Internet. With the availability of both Linux and Windows platforms, the options available to developers and hosts are virtually unlimited.

This series of articles will look at the viability of using EC2 as a platform for reselling web hosting services. Listed below are the base items used in the development of the series. Your choice of platforms and options may vary. While I may explore other options, don’t take the examples I give here as the only way to approach things.

Requirements:

These items are required for creating an environment that matches the example given in the series.

1. Amazon Web Services account. Free to obtain, and attaches to your existing Amazon account if you choose. See this link to register.
2. A fully or demo licensed copy of CPanel/WHM control panel software.
3. A fully or demo licensed copy of WHMCS (WHM Complete Solution)
4. An available primary domain name (Optional: one or more secondary domain names for testing client configurations).
5. Potential clientele
6. A willingness to learn and experiment potentially outside your comfort zone.

Our Test Environment:

The environment we’ll be using for these articles will consist of 1 Amazon EC2 Large (x64) instance (Roughly equivalent to a 3GHz Quad Core Xeon processor, with 7.5GB of RAM), Multiple Amazon Elastic Block Store (EBS) volumes of varying sizes, Multiple Amazon Elastic IP addresses to be used for both server and client configuration.

The next article will deal with the initial base setup of the EC2 image and archiving it for use in the next steps.