S3cmd Bucket Backup Script Highlights:
- Bash script to be installed via SSH on any Linux Server
- Backups for DigitalOcean Spaces, Amazon S3
- Backup in second account/region
- Uses S3cmd library
- Daily, Weekly, and Monthly Full Backups
- Manages Backup Rotation/Lifecycle
- Test and Production Modus
- Restore to bucket on new account/region possible
- User-friendly menu
The following video explains how this backup script works:
Our S3cmd Bucket backup script uses the Secure Shell Protocol (SSH) and the S3cmd library to manage the daily backups of a Bucket from one account in DigitalOcean Spaces or Amazon S3 to a second separate account. The script uses a unique sequence of calling S3cmd commands to sync two buckets from different Amazon S3 or DigitalOcean Spaces accounts.
What Problems does the S3cmd Bucket Backup Script solve?
Problem 1: The standard S3cmd library was made to access primarily one account in DigitalOcean Spaces or Amazon S3 but does not come with an easy solution to create a backup from a bucket in one account to a second account in DigitalOcean Spaces or Amazon S3.
Problem 2: The standard way to take backups of Cloud Storage Buckets is by syncing a bucket to another. The problem is that you need to spend days developing your own script if you are looking to create daily backups as per a specific point in time for certain restore scenarios.
The Solution offered by S3cmd script: The S3cmd bucket backup script manages daily, weekly and monthly backups of Production Bucket to Backup bucket, as well as restore scenarios to Production, Restore or Test Buckets.
The bucket backup solution will create Daily, Weekly, and Monthly folders in the backup bucket. The backup script will fill the folders with exact replicas of the Production Bucket as per the applicable backup dates.
Each backup will have its folder, which is labeled in YYYYMMDD_HHMMSS format.
- Linux Server (e.g., a Droplet in DigitalOcean with Ubuntu)
- Account 1 in DigitalOcean Spaces or Amazon S3 containing a Production bucket
- Account 2 in DigitalOcean Spaces or Amazon S3 (containing a Backup bucket)
- S3cmd library
- Account 3 in DigitalOcean or Amazon S3 (for Test Bucket)
- Account 4 in DigitalOcean or Amazon S3 (for Restore Bucket in case Production Bucket would be compromised)
The S3cmd bucket backup script needs to be installed on the Linux server via SSH.
Script Main Menu:
Once the bucket backup script is installed, the user can access a menu via (by entering the sh menu.sh command in the respective folder). This will open a menu via the command to manage the whole configuration to manage backups and restores for DigitalOcean Spaces bucket or an Amazon S3 bucket. The offered menus are the following:
(01) Create Backup (productionbucket => backupbucket)
(02) Restore to PRODUCTION Bucket (backupbucket => productionbucket)
(03) Restore to RESTORE Bucket (backupbucket => restorebucket)
(04) Restore to TEST Bucket (backupbucket => testbucket)
(05) List Buckets in Backup Account
(06) List Backups in Backup Bucket backupbucket
Configuration and Setup:
(07) Change Modus: Test/Live (Current Modus: production)
(08) Set Backup Period (Lifecycle Policy)
(09) Set API Settings
(10) Check API Connections
(11) Make Restore New Production Bucket
(12) Update s3cmd Library
(13) Set Folders Daily/Weekly/Monthly
(14) Set Cronjobs
(15) Set Timezone
(16) Closing Ports
(17) Quick Setup
(18) View READ.md
Menus 1-4 offer quick actions to do an ad-hoc backup or restore any backups to the Production, Restore, or Test bucket.
Menus 5-6 offers access to view the contents of the backup account, its buckets, and the available backups in the Backup folder.
Menus 7-17 allow making configuration changes, setting cronjobs, the server’s time zone, changing from Production to Test Modus, setting the archiving period for Daily, Weekly, and Monthly backups.
Menu 18 contains a README.md documentation file which will open in SSH.
Why do you need a bucket-backup script?
To summarize, here are the main reasons why the S3cmd backup script is a good choice:
- Keeps exact complete copies of your buckets
- Benefit from easy access to Daily, Weekly, Monthly Backups
- Backup possible to bucket on second account/region
- Script installed on your server, no dependencies on anybody else
- Uses S3cmd library, one of the most established libraries for managing cloud storage buckets
- Supports DigitalOcean Spaces to Amazon S3 and vice versa
- User-friendly menu via SSH, which can trigger restore actions quickly
- Quick setup and configuration
- Test and Production Modus
- Restore possible on fresh accounts (by defining the Restore Bucket)
Frequently Asked Questions
We recommend installing the backup script on an independent server droplet, e.g., a Server Droplet with Linux Ubuntu installed from DigitalOcean. However, you can install the script on any Linux server in principle as it is working in Bash via SSH.
- Do the two cloud storage buckets, Production and Backup bucket, need to be located in the same region (datacenter)?
No, they can be at a completely different location; however, it’s best to choose the same region if you like to have a fast backup (datacenter).
Server Droplet: The server droplet will host the script and temporary files when doing a backup/restore. The script takes next to no space but the server Droplet needs to have a minimum storage size as the Production Bucket. We recommend some safety margin, so depending on your growth factor 2.0x – 10.0x the size of your Production bucket.
Backup Bucket: The required disk space depends heavily on how many backups you keep in the backup bucket. If, e.g., you keep daily backups for 30 days, weekly backups for eight weeks, and monthly backups for six months, this means for every production, 44 backups exist. If your production bucket has 20 GB, you will need 1 TB of storage space in the bucket Backup folder. Therefore, it is essential to choose an archiving strategy that optimizes your objectives and the storage space.
Not at the moment. Although the script uses sync commands to synchronize two buckets, full backups are taken every time a new folder is created. This backup offers a premium solution for users who need accurate and complete backups per a selected point in time. We might think about adding an incremental backup solution later in time, depending on customer feedback received.
The price does not include installation support. If you want, we can set up the script for you, but we will charge an extra fee. Contact us if you are interested.
The goal was to create a high-quality backup solution using multiple Cloud Storage accounts. For this reason, we wanted to offer backup folders which you can easily access and find out quickly if a file is there or not. If you have to unzip the folder first, it takes additional time to find out what the backup exactly contains. The additional time you usually won’t have in a restore scenario, which is an emergency.
The script will work with any cloud storage provider using the S3cmd library such as Amazon S3 and DigitalOcean Spaces. We believe it will work with other cloud storage providers as well.
No. We use the sync command in the s3cmd library to sync two buckets. No files are deleted. Only the missing files are being transferred.
Yes, you can. In this case, you would need to duplicate the daily cronjob placed in the crontab and trigger additional backups at specified hours. However, please keep in mind, this will multiply the required storage space significantly.
Yes, the script will still work but you will not receive further updates or product support.
Get the Bucket Backup Script Today!
Note: All paid subscriptions below includes the latest script version (in Bash) as well as product support and updates for one year. In case you want us to install the script on your Linux server, contact us.