Skip navigation links

Package software.amazon.awscdk.services.s3.deployment

AWS S3 Deployment Construct Library

See: Description

Package software.amazon.awscdk.services.s3.deployment Description

AWS S3 Deployment Construct Library

This library allows populating an S3 bucket with the contents of .zip files from other S3 buckets or from local disk.

The following example defines a publicly accessible S3 bucket with web hosting enabled and populates it from a local directory on disk. 

 Bucket websiteBucket = Bucket.Builder.create(this, "WebsiteBucket")
         .websiteIndexDocument("index.html")
         .publicReadAccess(true)
         .build();
 
 BucketDeployment.Builder.create(this, "DeployWebsite")
         .sources(List.of(Source.asset("./website-dist")))
         .destinationBucket(websiteBucket)
         .destinationKeyPrefix("web/static")
         .build();

This is what happens under the hood:

  1. When this stack is deployed (either via cdk deploy or via CI/CD), the contents of the local website-dist directory will be archived and uploaded to an intermediary assets bucket. If there is more than one source, they will be individually uploaded.
  2. The BucketDeployment construct synthesizes a custom CloudFormation resource of type Custom::CDKBucketDeployment into the template. The source bucket/key is set to point to the assets bucket.
  3. The custom resource downloads the .zip archive, extracts it and issues aws s3 sync --delete against the destination bucket (in this case websiteBucket). If there is more than one source, the sources will be downloaded and merged pre-deployment at this step.

If you are referencing the filled bucket in another construct that depends on the files already be there, be sure to use deployment.deployedBucket. This will ensure the bucket deployment has finished before the resource that uses the bucket is created: 

 Bucket websiteBucket;
 
 
 BucketDeployment deployment = BucketDeployment.Builder.create(this, "DeployWebsite")
         .sources(List.of(Source.asset(join(__dirname, "my-website"))))
         .destinationBucket(websiteBucket)
         .build();
 
 new ConstructThatReadsFromTheBucket(this, "Consumer", Map.of(
         // Use 'deployment.deployedBucket' instead of 'websiteBucket' here
         "bucket", deployment.getDeployedBucket()));

Supported sources

The following source types are supported for bucket deployments:

To create a source from a single file, you can pass AssetOptions to exclude all but a single file:

IMPORTANT The aws-s3-deployment module is only intended to be used with zip files from trusted sources. Directories bundled by the CDK CLI (by using Source.asset() on a directory) are safe. If you are using Source.asset() or Source.bucket() to reference an existing zip file, make sure you trust the file you are referencing. Zips from untrusted sources might be able to execute arbitrary code in the Lambda Function used by this module, and use its permissions to read or write unexpected files in the S3 bucket.

Retain on Delete

By default, the contents of the destination bucket will not be deleted when the BucketDeployment resource is removed from the stack or when the destination is changed. You can use the option retainOnDelete: false to disable this behavior, in which case the contents will be deleted.

Configuring this has a few implications you should be aware of:

General Recommendations

Shared Bucket

If the destination bucket is not dedicated to the specific BucketDeployment construct (i.e shared by other entities), we recommend to always configure the destinationKeyPrefix property. This will prevent the deployment from accidentally deleting data that wasn't uploaded by it.

Dedicated Bucket

If the destination bucket is dedicated, it might be reasonable to skip the prefix configuration, in which case, we recommend to remove retainOnDelete: false, and instead, configure the autoDeleteObjects property on the destination bucket. This will avoid the logical ID problem mentioned above.

Prune

By default, files in the destination bucket that don't exist in the source will be deleted when the BucketDeployment resource is created or updated. You can use the option prune: false to disable this behavior, in which case the files will not be deleted. 

 Bucket destinationBucket;
 
 BucketDeployment.Builder.create(this, "DeployMeWithoutDeletingFilesOnDestination")
         .sources(List.of(Source.asset(join(__dirname, "my-website"))))
         .destinationBucket(destinationBucket)
         .prune(false)
         .build();

This option also enables you to multiple bucket deployments for the same destination bucket & prefix, each with its own characteristics. For example, you can set different cache-control headers based on file extensions: 

 Bucket destinationBucket;
 
 BucketDeployment.Builder.create(this, "BucketDeployment")
         .sources(List.of(Source.asset("./website", AssetOptions.builder().exclude(List.of("index.html")).build())))
         .destinationBucket(destinationBucket)
         .cacheControl(List.of(CacheControl.fromString("max-age=31536000,public,immutable")))
         .prune(false)
         .build();
 
 BucketDeployment.Builder.create(this, "HTMLBucketDeployment")
         .sources(List.of(Source.asset("./website", AssetOptions.builder().exclude(List.of("*", "!index.html")).build())))
         .destinationBucket(destinationBucket)
         .cacheControl(List.of(CacheControl.fromString("max-age=0,no-cache,no-store,must-revalidate")))
         .prune(false)
         .build();

Exclude and Include Filters

There are two points at which filters are evaluated in a deployment: asset bundling and the actual deployment. If you simply want to exclude files in the asset bundling process, you should leverage the exclude property of AssetOptions when defining your source: 

 Bucket destinationBucket;
 
 BucketDeployment.Builder.create(this, "HTMLBucketDeployment")
         .sources(List.of(Source.asset("./website", AssetOptions.builder().exclude(List.of("*", "!index.html")).build())))
         .destinationBucket(destinationBucket)
         .build();

If you want to specify filters to be used in the deployment process, you can use the exclude and include filters on BucketDeployment. If excluded, these files will not be deployed to the destination bucket. In addition, if the file already exists in the destination bucket, it will not be deleted if you are using the prune option: 

 Bucket destinationBucket;
 
 BucketDeployment.Builder.create(this, "DeployButExcludeSpecificFiles")
         .sources(List.of(Source.asset(join(__dirname, "my-website"))))
         .destinationBucket(destinationBucket)
         .exclude(List.of("*.txt"))
         .build();

These filters follow the same format that is used for the AWS CLI. See the CLI documentation for information on Using Include and Exclude Filters.

Objects metadata

You can specify metadata to be set on all the objects in your deployment. There are 2 types of metadata in S3: system-defined metadata and user-defined metadata. System-defined metadata have a special purpose, for example cache-control defines how long to keep an object cached. User-defined metadata are not used by S3 and keys always begin with x-amz-meta- (this prefix is added automatically).

System defined metadata keys include the following:

You can find more information about system defined metadata keys in S3 PutObject documentation and aws s3 sync documentation. 

 Bucket websiteBucket = Bucket.Builder.create(this, "WebsiteBucket")
         .websiteIndexDocument("index.html")
         .publicReadAccess(true)
         .build();
 
 BucketDeployment.Builder.create(this, "DeployWebsite")
         .sources(List.of(Source.asset("./website-dist")))
         .destinationBucket(websiteBucket)
         .destinationKeyPrefix("web/static") // optional prefix in destination bucket
         .metadata(UserDefinedObjectMetadata.builder().A("1").b("2").build()) // user-defined metadata
 
         // system-defined metadata
         .contentType("text/html")
         .contentLanguage("en")
         .storageClass(StorageClass.INTELLIGENT_TIERING)
         .serverSideEncryption(ServerSideEncryption.AES_256)
         .cacheControl(List.of(CacheControl.setPublic(), CacheControl.maxAge(Duration.hours(1))))
         .accessControl(BucketAccessControl.BUCKET_OWNER_FULL_CONTROL)
         .build();

CloudFront Invalidation

You can provide a CloudFront distribution and optional paths to invalidate after the bucket deployment finishes. 

 import software.amazon.awscdk.services.cloudfront.*;
 import software.amazon.awscdk.services.cloudfront.origins.*;
 
 
 Bucket bucket = new Bucket(this, "Destination");
 
 // Handles buckets whether or not they are configured for website hosting.
 Distribution distribution = Distribution.Builder.create(this, "Distribution")
         .defaultBehavior(BehaviorOptions.builder().origin(new S3Origin(bucket)).build())
         .build();
 
 BucketDeployment.Builder.create(this, "DeployWithInvalidation")
         .sources(List.of(Source.asset("./website-dist")))
         .destinationBucket(bucket)
         .distribution(distribution)
         .distributionPaths(List.of("/images/*.png"))
         .build();

Size Limits

The default memory limit for the deployment resource is 128MiB. If you need to copy larger files, you can use the memoryLimit configuration to increase the size of the AWS Lambda resource handler.

The default ephemeral storage size for the deployment resource is 512MiB. If you need to upload larger files, you may hit this limit. You can use the ephemeralStorageSize configuration to increase the storage size of the AWS Lambda resource handler.

NOTE: a new AWS Lambda handler will be created in your stack for each combination of memory and storage size.

EFS Support

If your workflow needs more disk space than default (512 MB) disk space, you may attach an EFS storage to underlying lambda function. To Enable EFS support set efs and vpc props for BucketDeployment.

Check sample usage below. Please note that creating VPC inline may cause stack deletion failures. It is shown as below for simplicity. To avoid such condition, keep your network infra (VPC) in a separate stack and pass as props. 

 Bucket destinationBucket;
 Vpc vpc;
 
 
 BucketDeployment.Builder.create(this, "DeployMeWithEfsStorage")
         .sources(List.of(Source.asset(join(__dirname, "my-website"))))
         .destinationBucket(destinationBucket)
         .destinationKeyPrefix("efs/")
         .useEfs(true)
         .vpc(vpc)
         .retainOnDelete(false)
         .build();

Data with deploy-time values

The content passed to Source.data() or Source.jsonData() can include references that will get resolved only during deployment.

For example: 

 import software.amazon.awscdk.services.sns.*;
 
 Bucket destinationBucket;
 Topic topic;
 
 
 Map<String, Object> appConfig = Map.of(
         "topic_arn", topic.getTopicArn(),
         "base_url", "https://my-endpoint");
 
 BucketDeployment.Builder.create(this, "BucketDeployment")
         .sources(List.of(Source.jsonData("config.json", appConfig)))
         .destinationBucket(destinationBucket)
         .build();

The value in topic.topicArn is a deploy-time value. It only gets resolved during deployment by placing a marker in the generated source file and substituting it when its deployed to the destination with the actual value.

Notes

Development

The custom resource is implemented in Python 3.7 in order to be able to leverage the AWS CLI for "aws s3 sync". The code is under lib/lambda and unit tests are under test/lambda.

This package requires Python 3.7 during build time in order to create the custom resource Lambda bundle and test it. It also relies on a few bash scripts, so might be tricky to build on Windows.

Roadmap

Skip navigation links

Copyright © 2022. All rights reserved.