Skip to main content

Upload Buckets on AWS

Introduction​

Tator uses S3-compatible storage for upload locations for all media. Tator deployments can be configured with a single default upload bucket that can be used for all projects; however, another bucket may be specified at the organization level and used on a per-project basis. This tutorial will walk you through how to configure an S3 bucket and IAM account for upload use with Tator on Amazon Web Services (AWS), how to register the bucket with your organization in Tator, and how to use the bucket in a specific Tator project.

This tutorial depends on the Project Specific Buckets on AWS tutorial. You will need no additional administrative privileges other than the ones specified there.

Create and configure an S3 bucket on AWS​

Follow the steps from the same section of the Project Specific Buckets on AWS tutorial, with changes to the following steps:

2. Use tator-upload-bucket as the bucket name.

3. Do not enable bucket versioning.

8. Only select the action Transition current versions of objects between storage classes.

9. Transition current versions to Glacier Deep Archive after 7 days.

11. After this step is complete, the bucket is now ready to use as a upload bucket for your project. Do not complete steps 12 - 14.

Create and configure an IAM account on AWS​

If you have set up an IAM account on AWS by following the steps from the same section of the Project Specific Buckets on AWS tutorial, you may use the same access and secret keys for the next step. If you have not, or would like a separate IAM account for accessing the upload bucket, you may follow those steps with no changes now.

Create a Bucket and Project in Tator​

We will use tator-py to register the bucket to our organization, then create a project that will use that bucket. You can also change the bucket property of an existing project.

  1. Use tator-py to create a Bucket object, replacing the host, token, bucket name, access key, secret key, region, and organization ID as appropriate:

    import tator
    api = tator.get_api(host="https://cloud.tator.io", token="YOUR_TOKEN")
    spec = {
        "name": "tator-upload-bucket",
        "config": {
            "aws_access_key_id": "YOUR_IAM_ACCESS_KEY",
            "aws_secret_access_key": "YOUR_IAM_SECRET_KEY",
            "endpoint_url": "https://tator-upload-bucket.s3-us-east-2.amazonaws.com",
            "region_name": "us-east-2",
        },
        "store_type": "AWS",
    }
    response = api.create_bucket(organization=YOUR_ORGANIZATION_ID, bucket_spec=spec)
    bucket_id = response.id

  2. Use the bucket ID to create a project:

    spec = {
        "name": "My Project with Upload",
        "summary": "Project demonstrating an S3 upload bucket.",
        "organization": YOUR_ORGANIZATION_ID,
        "upload_bucket": bucket_id,
    }
    response = api.create_project(project_spec=spec)

The project is now ready to use.