Use Rclone with the Koyeb Serverless Platform to Manage Data Across Cloud Storage Providers

Last updated 24 days ago

Introduction

This guide demonstrates how to use Rclone on the Koyeb serverless platform and perform diverse operations such as data synchronization, copying, and moving data between two cloud storage providers. Rclone is a command-line program offering a large set of features to manage your data stored across different cloud storage providers. You can imagine Rclone as the Unix rsync command for the cloud.

As Koyeb provides a unified interface to manage your data and preview the result of each operation we will do in this guide from a single interface, we will create two Koyeb Stores connected to each of the providers used in this guide: AWS S3 and Scaleway Object Storage. However, you can use Rclone with whichever cloud service providers you want! Find the full list of providers that Rclone supports here.

Below is a non-exhaustive list of operations you can perform using Rclone:

  • Back up data stored with your cloud storage providers,
  • Synchronize files and directories from different cloud storage providers,
  • Copy files and directories from a cloud storage provider to another,
  • Move files and directories from one cloud storage provider to the next,
  • Compare the content of different repositories and buckets across different cloud storage providers,
  • And much more all between any cloud storage providers.

Using Rclone with Koyeb lets you manage and move your data across different cloud storage providers through a reliable and easy-to-use serverless platform, Koyeb. Since Koyeb allows long-running functions, you can synchronize your data or copy data from a cloud storage provider to another without having to care about the execution time the operation took to perform. Combining Rclone with Koyeb allows you to run Rclone in batch mode or at a regular interval using Koyeb's Cron functions. This tutorial will cover how to use Rclone to:

Requirements

Steps

To use Rclone with Koyeb to manage and move your data across different cloud storage providers, follow these steps:

  1. Create a Stack and deploy a new revision to sync your data
  2. Copy files from source to destination
  3. Check Directories for Missing or Extra Files

Create a Stack and deploy a new revision to sync your data

Stacks are processing environments you deploy functions and Docker containers. There are different ways to invoke your Stack functions using:

  • Cron job: To trigger a function on a time-based period
  • Store events: To trigger a function when an event occurs on your Stores
  • Manual invocation: To invoke your function manually

All Stacks you create benefit from native versioning to track changes and rollback to previous versions easily.

To create a new Stack:

  1. In the Koyeb Console, click the Create button in the top navigation bar and select Stack.
  2. You land on the Stack creation page, Select Koyeb Simple Versioning, and give your Stack a name.

On your Stack page, you are asked to deploy your first Stack revision containing your Stack configuration. A Stack configuration is defined by a YAML file. Each time you deploy a new Stack revision, Koyeb reads your YAML instructions to build and deploy your functions.

In this guide, we use the Rclone Docker image to showcase how to sync, copy, or check data across two different cloud storage providers.

Below is the Stack configuration YAML used to perform a sync operation.

1functions:
2  - name: rclone
3    image: rclone/rclone
4    command: ["/bin/sh", "-c"]
5    args: [
6      RCLONE_CONFIG_AWS_TYPE=s3
7      RCLONE_CONFIG_AWS_ACCESS_KEY_ID=$KOYEB_STORE_aws_ACCESS_KEY
8      RCLONE_CONFIG_AWS_SECRET_ACCESS_KEY=$KOYEB_STORE_aws_SECRET_KEY
9      RCLONE_CONFIG_AWS_ENDPOINT=$KOYEB_STORE_aws_ENDPOINT
10      RCLONE_CONFIG_AWS_REGION=$KOYEB_STORE_aws_REGION
11      RCLONE_CONFIG_SCALEWAY_TYPE=s3
12      RCLONE_CONFIG_SCALEWAY_ACCESS_KEY_ID=$KOYEB_STORE_scaleway_ACCESS_KEY
13      RCLONE_CONFIG_SCALEWAY_SECRET_ACCESS_KEY=$KOYEB_STORE_scaleway_SECRET_KEY
14      RCLONE_CONFIG_SCALEWAY_ENDPOINT=$KOYEB_STORE_scaleway_ENDPOINT
15      RCLONE_CONFIG_SCALEWAY_REGION=$KOYEB_STORE_scaleway_REGION
16      rclone sync aws:store-connected-to-aws-s3 scaleway:store-connected-2-scaleway
17    ]
18    volumes:
19      - name: scaleway
20        store: store-connected-2-scaleway
21      - name: aws
22        store: store-connected-to-aws-s3
23

This operation syncs the content of aws:store-connected-to-aws-s3 to scaleway:store-connected-2-scaleway, modifying only the destination directory. Since the destination directory will be modified to match the source directory, files will either be removed or added to your destination directory.

Click Deploy Revision and wait for your Stack to build.

Once your Stack has been built, click Invoke to manually invoke this function.

Koyeb Stack Invoke

You can test if your Stack has successfully synchronized your data by:

  • Manually comparing if the Koyeb Store connected to your Scaleway matches the content of your Koyeb Store connected to AWS S3 without leaving the Koyeb control panel
  • Using Rclone's check operation to verify the bucket contents match

Copy files from source to destination

You can use Rclone to copy files from one directory to another, only adding files to the destination directory and skipping files that are already in both directories. Copy is different than the sync operation because copy will not remove files from the destination directory, it will only add non-duplicate files.

In your Stack, click Manage Stack Configuration and edit the Rclone command in the YAML:

Enter the following Stack configuration:

1functions:
2  - name: rclone
3    image: rclone/rclone
4    command: ["/bin/sh", "-c"]
5    args: [
6      RCLONE_CONFIG_AWS_TYPE=s3
7      RCLONE_CONFIG_AWS_ACCESS_KEY_ID=$KOYEB_STORE_aws_ACCESS_KEY
8      RCLONE_CONFIG_AWS_SECRET_ACCESS_KEY=$KOYEB_STORE_aws_SECRET_KEY
9      RCLONE_CONFIG_AWS_ENDPOINT=$KOYEB_STORE_aws_ENDPOINT
10      RCLONE_CONFIG_AWS_REGION=$KOYEB_STORE_aws_REGION
11      RCLONE_CONFIG_SCALEWAY_TYPE=s3
12      RCLONE_CONFIG_SCALEWAY_ACCESS_KEY_ID=$KOYEB_STORE_scaleway_ACCESS_KEY
13      RCLONE_CONFIG_SCALEWAY_SECRET_ACCESS_KEY=$KOYEB_STORE_scaleway_SECRET_KEY
14      RCLONE_CONFIG_SCALEWAY_ENDPOINT=$KOYEB_STORE_scaleway_ENDPOINT
15      RCLONE_CONFIG_SCALEWAY_REGION=$KOYEB_STORE_scaleway_REGION
16      rclone copy aws:store-connected-to-aws-s3 scaleway:store-connected-2-scaleway
17    ]
18    volumes:
19      - name: scaleway
20        store: store-connected-2-scaleway
21      - name: aws
22        store: store-connected-to-aws-s3
23

Click Deploy Revision and wait for your Stack to build.

Once your Stack has been built, click Invoke to manually invoke this function.

Koyeb Stack Invoke

To see if your files were successfully copied from your source bucket to your destination bucket, you can:

  • Manually comparing if the Koyeb Store connected to your Scaleway match the content of your Koyeb Store connected to AWS S3 without leaving the Koyeb control panel
  • Using Rclone check operation to verify the bucket contents match

Check Directories for Missing or Extra Files

You can use Rclone to compare one directory to another, checking if there are missing or extra files between the source and destination directories.

In your Stack, click Manage Stack Configuration and edit the Rclone command in the yaml:

Enter the following Stack configuration:

1functions:
2  - name: rclone
3    image: rclone/rclone
4    command: ["/bin/sh", "-c"]
5    args: [
6      RCLONE_CONFIG_AWS_TYPE=s3
7      RCLONE_CONFIG_AWS_ACCESS_KEY_ID=$KOYEB_STORE_aws_ACCESS_KEY
8      RCLONE_CONFIG_AWS_SECRET_ACCESS_KEY=$KOYEB_STORE_aws_SECRET_KEY
9      RCLONE_CONFIG_AWS_ENDPOINT=$KOYEB_STORE_aws_ENDPOINT
10      RCLONE_CONFIG_AWS_REGION=$KOYEB_STORE_aws_REGION
11      RCLONE_CONFIG_SCALEWAY_TYPE=s3
12      RCLONE_CONFIG_SCALEWAY_ACCESS_KEY_ID=$KOYEB_STORE_scaleway_ACCESS_KEY
13      RCLONE_CONFIG_SCALEWAY_SECRET_ACCESS_KEY=$KOYEB_STORE_scaleway_SECRET_KEY
14      RCLONE_CONFIG_SCALEWAY_ENDPOINT=$KOYEB_STORE_scaleway_ENDPOINT
15      RCLONE_CONFIG_SCALEWAY_REGION=$KOYEB_STORE_scaleway_REGION
16      rclone check aws:store-connected-to-aws-s3 scaleway:store-connected-2-scaleway
17    ]
18    volumes:
19      - name: scaleway
20        store: store-connected-2-scaleway
21      - name: aws
22        store: store-connected-to-aws-s3
23

Click Deploy Revision and wait for your Stack to build.

Once your Stack has been built, click Invoke to manually invoke this function.

Koyeb Stack Invoke

Click Show Logs to see the results.

Koyeb Stack Show Logs

The example above shows 2 files in the destination directory are missing compared to the source directory.

Conclusion

In this guide, you learned how to manage and move your data across different cloud storage providers with Rclone using Koyeb's serverless platform. Running Rclone on Koyeb's serverless platform ensures you only use the computing time your Rclone command requires, nothing more.

This tutorial covered how to use Rclone to synchronize, copy, move, and check your data across your different cloud service providers. You can check out all other Rclone supported operations here.

Want to follow this tutorial and use different third-party cloud service providers than AWS and Scaleway? Learn how to connect your cloud storage provider to Koyeb.

If you need any assistance or have additional questions, do not hesitate to reach out to us via Koyeb's in-app support or over Slack. We're happy to help!


Ready to get started?

Get in touch or create an account and deploy your serverless stack in minutes.

footer-frame