External Storage - AWS S3

External Storage / AWS S3

In order to use AWS S3 as your External Storage of choice, you must set the provider attribute to aws within the store hash and, at a minimum, provide your AWS credentials (key and secret) as well as a bucket name and the AWS region where your bucket is located. The full list of store hash attibutes for AWS S3 storage can be found below.

Alternatively, you can use our Secure Storage Connectors. Simply save your credentials in your Optidash Account and reference themy by ID. This mechanism will greatly increase the security of your cloud credentials. When Secure Storage Connectors are in use, you only have to provide Optidash API with your Connector id instead of provider, key, and secret properties. You can add new Connector in your Optidash Account.

Authentication

When passing AWS credentials in your request JSON, you have to set the following authentication properties:

Attribute Type Description
provider String provider must be set to aws
key String AWS Access Key Id
secret String AWS Secret Access Key 
{
    "store": {
        "provider": "aws",
        "key": "your-aws-key",
        "secret": "your-aws-secret"
    }
}

When using Secure Storage Connectors, you only need to provide your Connector ID:

Attribute Type Description
id String Secure Storage Connector ID 
{
    "store": {
        "id": "your-connector-id"
    }
}

AWS S3 properties and settings

Attribute Type Required Description
bucket String Yes Name of a destination bucket in your Amazon S3 account.
region String Yes Name of the AWS Region in which your S3 bucket is located.
path String No Destination path in your S3 bucket (without leading slash). Defaults to root.
acl String No The Access Control List of the destination object. Defaults to "public-read".
class String No Custom Storage Class you would like to set on your object. Valid values are standard, reduced-redundancy, standard-ia, onezone-ia, intelligent-tiering, glacier and deep-archive. Defaults to standard.
metadata Hash No Custom S3 Metadata you would like to set on your object.
headers Hash No Custom HTTP headers you would like to set on your object.
tags Hash No Custom S3 Tags you would like to set on your object.

The Optidash API allows you to set the following custom headers on your objects: Expires, Cache-Control, Content-Type, Content-Encoding, Content-Language and Content-Disposition.

{
    "store": {
        "provider": "aws",
        "key": "your-aws-key",
        "secret": "your-aws-secret",
        "bucket": "images",
        "region": "eu-central-1",
        "path": "assets/image.jpg",
        "acl": "public-read",
        "metadata": {
            "key": "value"
        },
        "headers": {
            "Cache-Control": "max-age=2592000000"
        },
        "tags": {
            "key": "value"
        }
    }
}

An example cURL request of using AWS S3 as External Storage provider will look like the following:

curl https://api.optidash.ai/1.0/fetch -X POST -u your-api-key: \
-H "Content-Type: application/json" \
-d '{
    "url": "https://www.website.com/image.jpg",
    "resize": {
        "width": 100,
        "height": 75
    },
    "store": {
        "provider": "aws",
        "key": "your-aws-key",
        "secret": "your-aws-secret",
        "bucket": "bucket-name",
        "region": "eu-central-1",
        "path": "assets/image.jpg",
        "headers": {
            "Cache-Control": "max-age=2592000000"
        }
    }
}'

When using AWS as your External Storage, the url property within the JSON response will point to the object's location within the S3 bucket and you can safely use that URL in production, for example:

HTTP/1.1 200 OK

Date: 
Status: 200 OK
Content-Type: application/json

{
    "success": true,
    "code": 200,
    "id": "9fccf4b5-cfab-4e92-9276-5d2028fcb6a0",
    "input": {
        "name": "image.jpg",
        ..
    },
    "output": {
        "url": "https://bucket-name.s3.eu-central-1.amazonaws.com/assets/image.jpg",
        ..
    }
}

Custom Bucket Policies

If you would like to create an AWS User dedicated only for the Optidash API or are using a custom bucket policy, please make sure to include s3:PutObject and s3:PutObjectAcl entries in the allowed actions section in your bucket policy file (and replace bucket-name in this example):

{
    "Statement": {
        "Effect": "Allow",
        "Action": [
            "s3:PutObject",
            "s3:PutObjectAcl"
        ],
        "Resource": [
            "arn:aws:s3:::bucket-name/*"
        ]
    }
}