FileMaker Container fields are easy to implement and great for a variety of use-cases, but when used to store large files or large numbers of files, they impose a heavy burden on your FileMaker infrastructure. It can be like a ball and chain attached to your FileMaker database, making it more cumbersome to move data between systems or to manage backups.

There is a better way!

Update: Jason presented this topic at SOFA in January, 2022. The recording is available on at: https://youtu.be/-OcZ_cC6wtc

Object Storage Basics

S3 and similar services are known as “object storage”. Objects are files. Object storage is unlike a drive volume in the sense that your files aren’t guaranteed to be physically stored on a single device, together on the same file system. In fact they’re almost guaranteed to be spread across many devices, perhaps hundreds of metres apart!

All objects are organized into “buckets”, which exist in a “region” (physical location in the world). You can use one or many buckets to organize your objects. You might decide to make one bucket for each distinct FileMaker app you build. Or you might decide to segment your objects further by adding separate buckets for distinct modules within your app, or perhaps a “public” and “private” bucket with different privileges.

Objects have paths that can look like directory structures, but they aren’t real directory structures – the path is really just an extended object name, or “key”. Note: the S3 console will allow you to navigate the hierarchy as if it were a real directory structure.

The Costs

In AWS S3 Standard, you can store files for $0.023US per GB per month in US East. The same amount of storage on a General Purpose EBS volume costs $0.08US (3.5 times more!). More importantly, you never pay for space you don’t use on S3, whereas in EC2, you provision storage in advance and pay for all the space you don’t use, too!

There are additional costs for requests & retrievals, data transfer, etc, and many ways to reduce your costs for infrequently accessed data, but unless you’re storing hundreds of gigabytes of data, you’re probably not going to need to think about this for a while.

Other Advantages

Durability

Any object stored in S3 is automatically replicated across at least 3 “Availability Zones” by the time each transaction completes. This is one reason Amazon can promise “eleven nines durability” (durability of 99.999999999% per year).

If you store ten million objects in S3, you should expect to lose one file every ten thousand years, on average.

Availability Zones are separated by several km (source), but if you’re concerned about large scale disasters, you can configure redundancy across geographic regions also.

Caveat – this type of redundancy doesn’t protect you against accidental deletion. More on that later…

Take a load off

Your FileMaker Server, that is… leave the business of large file transfer, storage, and backups to an external service so your FileMaker Server can focus on the things it does best.

Web Publishing

Need to display an asset in a web interface? No problem! Your FileMaker app can generate a secure “presigned” URL designed to work for seconds or days, so the user’s browser can request the file directly from your S3 bucket. It’s similar to the streaming URL generated in the FileMaker Data API, but more flexible.

Getting Started with AWS and Configuring S3

Sign up for AWS

If you don’t already have an AWS account, go to console.aws.amazon.com and “Create a new AWS account”. It is strongly recommended to add two-factor authentication to this account and protect your credentials carefully.

You can quickly navigate to AWS Services by typing the name in the search bar at the top of the screen and selecting the service from the result.

Configure S3

Go to the S3 service

Click “Create bucket”, choose a bucket name, and select an AWS Region near you.

As you scroll down, you will see that by default, “Block all public access” is enabled. Leave this on. With this option enabled, you ensure that no matter what you do on individual objects, you can’t accidentally make anything in your bucket public. Only an AWS identity with privileges to read the bucket can see files in your bucket. While S3 has a highly granular and complex permission structure, most use cases are possible while applying this strict policy.

On this screen you could enable Bucket Versioning if you wish. When Bucket Versioning is on, if you attempt to upload an object to a path (or “key”) that already exists, the existing object will be retained as a “version”. The same happens if you “delete” an object. To really delete an object, you would need to explicitly delete the version or use a lifecycle rule (for example, to automatically delete versions after a certain number of days). Bucket versioning can also be enabled or disabled later.

There are other options on this page that are beyond the scope of this tutorial.

When you’re ready, click “Create Bucket”. You should now see your bucket in your “Buckets” list:

Click on the bucket name, then open the “Permissions” tab. Scroll down to the section: “Cross-origin resource sharing (CORS)”. If you plan to interact with your bucket through a web browser (including a web viewer), click “Edit” and paste the following CORS configuration:

                    

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD",
            "PUT"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": []
    }
]


                

Save your changes.

Configure IAM Policy & User

Now you need to configure a User and Policy so that you can authenticate to the bucket and interact with it through an API.

In AWS, users are managed through the Identity and Access Management (IAM) service. Use the search bar to navigate to the IAM service.

First, create a policy with the appropriate permissions. Click “Policies” in the menu at the left, then click “Create Policy”.

Select the JSON tab and replace the existing contents with the following. This policy grants 4 privileges. s3:PutObject (upload), s3:GetObject (download), s3:ListBucket (list bucket contents), and s3:DeleteObject. These privileges apply to the specific resource listed, in this case the bucket and its contents.

                    

{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "VisualEditor0",
             "Effect": "Allow",
             "Action": [
                 "s3:PutObject",
                 "s3:GetObject",
                 "s3:ListBucket",
                 "s3:DeleteObject"
             ],
             "Resource": [
                 "arn:aws:s3:::YOUR-BUCKET-NAME/*",
                 "arn:aws:s3:::YOUR-BUCKET-NAME"
             ]
         }
     ]
 }


                

Be sure to replace YOUR-BUCKET-NAME with the actual name of your bucket.

You may not need all of these privileges or you may need to add additional privileges for your use case, for example if you want the API user to be able to view and/or delete versions.

Click “Next: tags” then “Next: review”. Give your policy a name like “s3FileMakerApiUser” and add a description if you wish. Click “Create policy”.

Next, create a User. Click “Users” in the menu at the left, then click “Add users”.

Input a user name like “s3FileMakerApiUser” and choose “Programmatic access” as the “Access type”. Click “Next: Permissions”

Click “Attach existing policies directly”. Find the policy you created earlier and select the box, then click “Next: Tags”, then “Next: Review”, and finally “Create user”.

Click “Show” to expose the Secret access key. The Access key ID and Secret access key will be required to connect to S3 from FileMaker. You will not be able to retrieve the secret key again, but you can generate a new one at any time.

You would generally store this information in a preferences table in your FileMaker app. If you do not have a preferences table, you could temporarily paste this information directly into “Set Variable” script steps, but be aware of the security risks. Most importantly, your API credentials will be exposed in plain text in any DDR or XML output of your FileMaker file.

Congratulations! Configuration on the AWS side is done.

The Presigned URL

This little guy does all the heavy lifting.

An AWS presigned URL is a URL for an AWS API call that includes a signature derived from the request details including the URL, method, headers, and credentials. The URL can only be used to run the specific request for which it was intended – if you try to change the URL, method, or headers, the signature will be invalid.

The presigned URL is especially useful when you want to allow someone to download an object, such as a user who you have authenticated on your website. You can generate a presigned URL to retrieve an object in S3, and you can make the URL valid for a window spanning anywhere from 1 second to 7 days. You can then incorporate this URL into a webpage, and the image will load just like any other resource URL (to an image file, for example).

You can also use a presigned URL for other actions, like uploading files to S3, deleting files, or getting object metadata.

Without the presigned URL, you would not be able to publish objects to a website without making the objects public or passing them through another web service, and you would not be able to create a service that allowed end-users to upload files directly to S3 without going through another web service.

The s3presignedUrl() FileMaker custom function

s3presignedUrl ( method ; bucket ; region ; theFilePath ; expireSeconds ; accessKey ; secretKey ; optionsObj )

Get the function and docs: quarfie / s3presignedUrl_FMFunction on GitHub

FileMaker Usage

Here are some example scripts for working with S3 in FileMaker:

Uploading a file from a container to S3

                    

Set Variable [ $path ; "folder/thefilename.jpg" ]
Set Variable [ $url ; s3presignedUrl ( "PUT" ; $bucket ; $region ; $path ; 60 ; $access_key ; $secret_key ; "" ]
Set Variable [ $container ; Table::Container ]
Set Variable [ $contentType ; "image/jpeg" ]
Set Variable [ $curl ; "-X PUT -H " & Quote ( "Content-Type: " & $contentType) & " --data-binary @$container -D $responseHeaders" ]
Insert From URL [ Select ; With dialog: Off ; Target: $result ; $url ; Verify SSL Certificates ; cURL options: $curl ; Do not automatically encode URL ]
If [ PatternCount ( $responseHeaders ; "200 OK" ) ]
 //File was successfully uploaded
Else
 //Error
End If


                

NOTE: The above example includes a hard-coded Content-Type header (“image/jpeg”). You can automatically determine the content type (or “MIME type”) using a custom function like GetMIMEType ( fileNameOrExtension ). The Content-Type is important if you wish to render the file in a browser (including a web viewer), as it will be sent back as a header that the browser requires to display it. Without this information, the browser will simply download the file.

Download a file from S3 to a container field

                    

Set Variable [ $path ; "folder/thefilename.jpg" ]
Set Variable [ $url ; s3presignedUrl ( "GET" ; $bucket ; $region ; $path ; 60 ; $access_key ; $secret_key ; "" ]
Set Variable [ $curl ; "--FM-return-container-variable -D $responseHeaders" ]
Insert From URL [ Select ; With dialog: Off ; Target: $result ; $url ; Verify SSL Certificates ; cURL options: $curl ; Do not automatically encode URL ]
If [ PatternCount ( $responseHeaders ; "200 OK" ) ]
 //File is in $result
 Set Field [ Table::Container ; $result ]
Else
 //Error
End If


                

View a file in a web viewer

This example assumes the file is an image file that can load into an image tag.

                    

Let ([
   ~path = Table::path ;
   ~url = s3presignedUrl ( "GET" ; $bucket ; $region ; ~path ; 60*5 ; $accessKey ; $secretKey ; "" )
];
"data:text/html,<!DOCTYPE html><html><head></head><body style=\"margin:0px;\"><div style=\"width:100wv;height:100vh;\"><img style=\"max-width:100%;max-height:100%;margin-left:auto;margin-right:auto;display:block;\" src=\"" & ~url & "\"></div></body></html>"
)


                

Tip: Rather than using a calculation in a web viewer object directly, use the Set Web Viewer script step. This can be called OnRecordLoad or whenever the web viewer is exposed. This should solve refresh issues that you may encounter when resizing windows or modifying the underlying record.

Delete File Example

                    

Set Variable [ $path ; "folder/thefilename.jpg" ]
Set Variable [ $url ; s3presignedUrl ( "DELETE" ; $bucket ; $region ; $path ; 60 ; $access_key ; $secret_key ; "" ]
Set Variable [ $curl ; "-X DELETE -D $responseHeaders" ]
Insert From URL [ Select ; With dialog: Off ; Target: $result ; $url ; Verify SSL Certificates ; cURL options: $curl ; Do not automatically encode URL ]
If [ PatternCount ( $responseHeaders ; "204 No Content" ) ]
 //File was successfully deleted or never existed to begin with
Else
 //Error
End If


                

Get Metadata about the object in S3 (Content-Type and File Size)

                    

Set Variable [ $path ; "folder/thefilename.jpg" ]
Set Variable [ $url ; s3presignedUrl ( "HEAD" ; $bucket ; $region ; $path ; 60 ; $access_key ; $secret_key ; "" ]
Set Variable [ $curl ; "--head" ]
Insert From URL [ Select ; With dialog: Off ; Target: $result ; $url ; Verify SSL Certificates ; cURL options: $curl ; Do not automatically encode URL ]
Set Variable [ $responseCode ; Value: MiddleWords ( $result ; 3 ; 1 ) ]
If [ $responseCode = 200 ]
 //File exists and you can now parse out the headers Content-Type and Content-Length
Else
 //Error
End If


                

Taking it to the next level: eliminating container fields and building a great UI

By adding a JavaScript-based uploader like Uppy, and taking advantage of FileMaker 19’s JavaScript integration features, you can completely eliminate the need for FileMaker containers!

The attached file demonstrates this capability and works in FileMaker Pro, Go, and WebDirect.

Similarly, when it comes time to display the file, conditional logic generates the appropriate html/css to display the file based on its type. If it cannot render the file type or the file type is unfamiliar, it will present an html page with a download link. When clicked, the file is downloaded to the user’s download folder.

Uppy can also be used on your website! When the user clicks “Upload”, use the Data API to run a script in FileMaker to create the corresponding record and generate the presigned URL, then send that back down to Uppy to complete the upload directly to S3.

The file is intended as a demonstration and not a starter file. I encourage you to build your own file in a way that is consistent with your own coding style.

NOTE: The demo file might not have the latest available version of the s3PresignedUrl() custom function. Be sure to check for the latest version from GitHub.

Closing Thoughts

I’m so glad you made it all the way to the bottom of this very long post! I am looking forward to hearing from you, especially if there is anything I could have done better! If your comment is not posted, please reach out to me, as I employ a very aggressive spam filter!

Updates

2022-02-03 – This article previously stated that AWS Availability Zones may exist under one roof. In fact, they are always separated by a “meaningful distance”. Various AWS resources describe the minimum distance as “1 mile” (1.6km) or “many kilometres”, and the maximum distance is consistently stated as 100 km (60 miles). eg: https://aws.amazon.com/about-aws/global-infrastructure/regions_az/

2024-09-21 – Added the note under the S3 Demo download button to check for the latest version of s3PresignedURL from GitHub. The older versions of the custom function does not work with newer AWS regions that only support S3 dot region endpoint structure and not the older dash region endpoint structure.