Mount AWS S3 Bucket on CentOS EC2 Instance

From Bonus Bits
Jump to: navigation, search

Purpose

This article gives the steps to mount an Amazon Web Service Simple Storage Service on CentOS or RHEL 6 over a VPC Endpoint.


Create IAM User

Create an IAM user that will be used to access S3 from your EC2 instances.

  1. Login to AWS web console
  2. Open IAM | Users
  3. Select Create New User
  4. Enter a user name
    1. i.e. s3user
  5. Copy generated access keys
  6. Set a complicated/long password for the user

OR You can use an IAM Role then include the role in the s3fs mount command

iam_role (default is no role)


Create S3 Bucket

  1. Login to AWS web console
  2. Select Services | S3
  3. Create a S3 bucket
    1. mywebapp-uploads
  4. Add bucket policy to allow IAM User and VPC Endpoint access to all files inside the bucket.
{
	"Version": "2012-10-17",
	"Id": "wiki-uploads",
	"Statement": [
		{
			"Sid": "allow-iam-user-readwrite",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::1234567890:user/s3user"
			},
			"Action": [
				"s3:PutObject",
				"s3:GetObject",
				"s3:DeleteObject",
				"s3:DeleteObjectVersion",
				"s3:GetObjectVersion"
			],
			"Resource": "arn:aws:s3:::wiki-uploads/*"
		},
		{
			"Sid": "allow-vpce-read-bucket",
			"Effect": "Allow",
			"Principal": "*",
			"Action": [
				"s3:ListBucket",
				"s3:ListBucketVersions"
			],
			"Resource": "arn:aws:s3:::wiki-uploads",
			"Condition": {
				"StringEquals": {
					"aws:sourceVpce": "vpce-12345678"
				}
			}
		},
		{
			"Sid": "allow-vpce-read-content",
			"Effect": "Allow",
			"Principal": "*",
			"Action": [
				"s3:GetObject",
				"s3:GetObjectVersion"
			],
			"Resource": "arn:aws:s3:::wiki-uploads/*",
			"Condition": {
				"StringEquals": {
					"aws:sourceVpce": "vpce-12345678"
				}
			}
		}
	]
}


Install Dependencies

CentOS/RHEL/Amazon

yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap automake fuse-devel git libcurl-devel libxml2-devel make

Ubuntu

sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config


Install Latest Fuse

s3fs requires fuse >= 2.8.4. Default yum repo on CentOS 6 at the time of writing this pulls down fuse-2.8.3-4. So you'll want to uninstall the repo fuse version and install the latest fuse from source. Amazon Linux default is high enough version

  1. Uninstall fuse
    yum erase fuse
    
  2. Download latest version
    1. http://sourceforge.net/projects/fuse/files/fuse-2.X/
    cd /usr/src/
    
    wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.9.4/fuse-2.9.4.tar.gz
    
  3. Extract tar
    tar xzf fuse-2.9.4.tar.gz
    
  4. Change to the extracted directory
    cd fuse-2.9.4
    
  5. Configure package
    ./configure --prefix=/usr/local
    
  6. Compile and Install package
    make && make install
    
    export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
    
    ldconfig
    
    modprobe fuse
    
  7. Symlink fusermount to /usr/bin (Optional)
    ln -s /usr/local/bin/fusermount /usr/bin/fusermount
    


Install s3fs (Github)

  1. Download s3fs
    1. git clone https://github.com/s3fs-fuse/s3fs-fuse.git
  2. Change to the extracted directory
    cd s3fs-fuse
    
  3. Run Auto Generate Shell Script
    ./autogen.sh
    
  4. Configure package
    ./configure
    
  5. Compile and Install package
    make && make install
    
  6. Symlink s3fs to /usr/bin (Optional)
    ln -s /usr/local/bin/s3fs /usr/bin/s3fs
    

Install s3fs (Older Method)

  1. Download s3fs
    1. http://tecadmin.net/mount-s3-bucket-centosrhel-ubuntu-using-s3fs/#
  2. Extract tar
    tar -zxvf s3fs-1.74.tar.gz
    
  3. Change to the extracted directory
    cd s3fs-1.74
    
  4. Configure package
    ./configure --prefix=/usr/local
    
  5. Compile and Install package
    make && make install
    
  6. Symlink s3fs to /usr/bin (Optional)
    ln -s /usr/local/bin/s3fs /usr/bin/s3fs
    


Install s3fs Libraries

  1. Edit /etc/ld.so.conf
    vim /etc/ld.so.conf
    
  2. Add this line to the top:
    /usr/local/lib/
  3. Rerun ldconfig
    ldconfig
    


Create Access Key File

Create a text file with the access keys.

  1. echo AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY > /etc/passwd-s3fs
    
    chmod 600 /etc/passwd-s3fs
    


Setup Auto Mount (fstab)

  1. mkdir /tmp/cache
    
    mkdir /mnt/s3mnt
    
    chmod 777 /tmp/cache /mnt/s3mnt
    
  2. Edit /etc/fstab
    vim /etc/fstab
    
  3. Add the following
    s3fs#<bucket> /mnt/s3mnt fuse allow_other,use_cache=/tmp/cache,uid=userid,gid=groupid 0 0
  4. Remount all in fstab
    mount -a
    


Related Articles


Sources