aws

Mount Amazon S3 Buckets on Ubuntu

I wanted the ability to mount a S3 bucket straight onto my Linux server so I wouldn’t have to rely on local storage, I have a write up on how to do this:

You can do this without installing FUSE just by installing s3fs but that only allows the user who mounted it to access the bucket and it is also not possible to mount using /etc/fstab

Before you run any commands, make sure your system is up to date by running:

Ubuntu Dependancies

Check Fuse Package

Install s3fs

Download the latest verion of s3fs from the Google Code here

Download the latest version of S3fs from Github

Change directory into the newly extracted s3fs folder

Configure time

This is important and missing from most how-to guides! If the time of the local server isn’t within 15 minutes of the Amazon server time, it won’t connect.

Install NTP

Edit ntp.conf

Comment out the existing time servers and add the following

Restart ntpd

Check the time of the Amazon server

It should give you some output like this example:

Change the timezone on your server to GMT time (as this is the timezone of the Amazon server)

Select “None of the above”

Select “GMT”

Set the date and time on your server

Change the values to the same as the Amazon server time

Configuring s3fs

Create a file called “passwd-s3fs under /etc

Put your access key and secret access key seperated by a “:”

Example: ACCCESS_KEY_ID:SECRET_ACCESS_KEY

Change the permissions for security reasons

Create a tmp folder for caching

Create a mount point

Change permissions on cache and mount point folders

Mounting s3 Bucket

Mount bucket

Allow a few minutes to sync

Check to see if it is mounted:

Example:

aws

Automounting

To mount automatically at boot, your /etc/fstab should look like this:

A brief description of the mount arguments:

  • Allow_other = allow all users to access the mount point
  • _netdev = The filesystem resides on a device that requires network access
  • nosuid = Do not allow set-user-identifier or set-group-identifierbits to take effect
  • nodev = Do not interpret character or block special devices on the file system
  • url = Use HTTPS instead of HTTP when configure as above

Should you have any questions, comments or suggestions, please don’t hesitate to comment below. If you like what you have read, please share it on your favourite social media medium.

Scripts to mount bucket

Mounting

Replace “bucketname” with the bucket you wish to mount and change the mount location (/mnt/aws)

Unmounting

Errors

RequestTimeTooSkewed

The difference between the request time and the current time is too large

This is caused because the Amazon server time is not within 15 minutes of your local server time, check to make sure you have set the correct timezone, date and time in the instructions above.

Troubleshooting

1. The best way to find out if there are any problems is to run s3fs with the -f switch as it will give you full output of the what it is trying to do.

2. Check /var/log/syslog for any messages

9 Responses to “Mount Amazon S3 Bucket on Ubuntu”

  1. Jim Maas

    This didn’t work for me on Ubuntu 14.04. Not an expert at this stuff but I don’t install or “make install” anything except as sudo. So because of file permissions this installation would only work for root and not for user. Is there a way to do an install that will work for user as well?

    Thanks

    J

    Reply
    • cr_admin

      Hi,

      There is no reason you can’t “sudo make” and “sudo make install” to install this.

      Thanks,
      Chris

      Reply
  2. Kevin

    I appreciate the effort you have put in for this, however, I too am getting an error on mounting.

    I am getting
    df: '/mnt/MyBucket': Transport endpoint is not connected after I do the df -h

    And upon adding to my fstab, and doing a mount -a, I am getting:
    s3fs: unable to access MOUNTPOINT /media/MyBucket: Transport endpoint is not connected

    Any thoughts?

    Reply
  3. Brandon W

    I just want to say, I have been searching the internet for ages trying to figure out how to mount my amazon s3 bucket to my server. This is by far the easiest thing to follow. I had to play around with my own settings but it works. I have 256TB of free space.

    Reply
  4. Phil

    How do you clear the local cache? I mounted my backup dir to an s2 bucket to save on local storage, and my /tmp/cache is now at 47G.

    Reply
    • Chris

      Hi Phil,
      You should just be able to delete the cache so long as it’s uploaded and not sync’ing. I actually changed to riofs which seems to work a lot better than S3FS, check that out as an alternative. Thanks, Chris

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.