Move your S3 files from AWS to Brewless

A Laravel app that comes from Vapor, or from anywhere else on AWS, keeps its uploads in an S3 bucket. When it moves to Scaleway with Brewless, the environment gets a new, private bucket on Scaleway Object Storage, and the files have to get there. This post shows how brewless storage:import copies them, which values go where, and what happens to your keys.

How do you move an S3 bucket to Brewless?#

Run brewless storage:import with the environment and the bucket the files are in now. It copies every file into the environment's bucket with rclone, in a serverless job of your own Scaleway project, and removes nothing. Run it again just before you move your domain and it only copies what changed.

export SOURCE_AWS_ACCESS_KEY_ID=AKIA…
export SOURCE_AWS_SECRET_ACCESS_KEY=…
brewless storage:import production s3://my-vapor-bucket --source-region=eu-west-1

The command says where it copies from and to, asks once, and follows the copy until it is done.

Which value goes where#

The names are the ones Laravel's s3 disk reads from config/filesystems.php. For the source you give them; for the destination Brewless already set them in the environment's variables when it made the bucket.

The AWS_* variables of your own shell are not read on purpose: they often belong to another AWS account than the one your files are in.

Three ways to give the source#

From the .env of the old application, which reads only its AWS_* lines:

brewless storage:import production --from-env=../old-app/.env

With variables, the same names as Laravel's with SOURCE_ in front:

export SOURCE_AWS_ACCESS_KEY_ID=AKIA…
export SOURCE_AWS_SECRET_ACCESS_KEY=…
brewless storage:import production s3://my-vapor-bucket/uploads --source-region=eu-west-1

With a profile of the AWS CLI, from ~/.aws/credentials and ~/.aws/config:

brewless storage:import production s3://my-vapor-bucket --source-profile=migration

A key is never an option on the command line, where it would end up in your shell's history. Without a key in any of these places the command asks for it in the terminal, and run from a script it stops.

Note

An app on Vapor runs with an IAM role, so its .env usually has no AWS_ACCESS_KEY_ID. Vapor sets AWS_BUCKET itself. Make a key at AWS that may only read the bucket, as below.

A key at AWS that may only read#

Do not hand out the key of your AWS account. Create an IAM user for the move, with only this inline policy, and an access key for an application outside AWS:

{
    "Version": "2012-10-17",
    "Statement": [
        { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-vapor-bucket" },
        { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-vapor-bucket/*" }
    ]
}

It can list and read the bucket and nothing else. Remove the user once you have moved.

In your project or from your laptop#

In the default mode Brewless runs rclone in a serverless job in your project. Both keys, the one for AWS and the environment's own key for its bucket, reach the job in one secret of your Secret Manager. When the copy ends, however it ends, that version of the secret is deleted. The operation Brewless records only knows which version it was, never a key. That is the same rule as for your other secrets: they stay in your account.

Follow it, or come back later#

While the copy runs, the command shows how many files of how many and how much of how much. Ctrl+C stops following, not the copy:

brewless storage:status production

shows the last copy into the environment's bucket, while it runs and after it ended. It is also on the environment's page in the console, where you can cancel it; what was copied stays.

--dry-run copies nothing and counts what the source holds first, which is handy to know how much AWS will bill for the data that leaves it.

After the move#

Warning

brewless env:push replaces the environment's variables as a whole. Do not push the .env of your old app with its own AWS_BUCKET and keys: that points the app back at the old bucket. Start from brewless env:pull instead.

Then change or remove AWS_URL, which pointed at the old bucket or CloudFront, run the import once more just before you move your domain, and remove the read key at AWS. Your app reads its files through the same s3 disk as before, now from Scaleway; see Laravel uploads on Scaleway Object Storage for how that bucket is set up.

The full reference, with other S3 services and every option, is in the documentation: https://docs.brewless.eu/applications/moving-files