365 lines
13 KiB
Markdown
365 lines
13 KiB
Markdown
# cloudformation-helpers
|
|
A collection of AWS Lambda funtions that fill in the gaps that existing CloudFormation resources do not cover.
|
|
|
|
AWS CloudFormation supports Custom Resources (http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html),
|
|
which can be used to call AWS Lambda functions. CloudFormation covers much of the AWS API landscape, but
|
|
does leave some gaps unsupported. AWS Lambda can contain any sort of logic, including interacting with the
|
|
AWS API in ways not covered by CloudFormation. By combining the two, CloudFormation deploys should be able
|
|
to approach the full resource support given by the AWS API.
|
|
|
|
Warning: most of these functions require fairly wide permissions, since they need access to resources in a
|
|
general manner - much the same way CloudFormation itself has permission to do almost anything.
|
|
|
|
|
|
## Usage
|
|
1. Use https://s3.amazonaws.com/com.gilt.public.backoffice.{your-region-here}/cloudformation_templates/create_cloudformation_helper_functions.template
|
|
to deploy a stack that creates the Lambda functions for you. Remember the stack name.
|
|
2. Include the following resources in your CloudFormation template. These will create a) a nested stack that
|
|
looks up the ARNs from the previous step and b) a custom resource that allows your template to read those ARNs.
|
|
|
|
```
|
|
"CFHelperStack": {
|
|
"Type": "AWS::CloudFormation::Stack",
|
|
"Properties": {
|
|
"TemplateURL": { "Fn:Join": [ "", ["https://s3.amazonaws.com/com.gilt.public.backoffice", { "Ref": "AWS:Region" }, "/cloudformation_templates/lookup_stack_outputs.template" ] ] }
|
|
}
|
|
},
|
|
"CFHelper": {
|
|
"Type": "Custom::CFHelper",
|
|
"Properties": {
|
|
"ServiceToken": { "Fn::GetAtt" : ["CFHelperStack", "Outputs.LookupStackOutputsArn"] },
|
|
"StackName": "your-helper-stack-name-here"
|
|
},
|
|
"DependsOn": [
|
|
"CFHelperStack"
|
|
]
|
|
}
|
|
```
|
|
|
|
You can either hardcode the stack name of your helper functions, or request it as a parameter.
|
|
3. Use the ARNs from the previous step in a custom resource, to call those Lambda functions:
|
|
|
|
```
|
|
"PopulateTable": {
|
|
"Type": "Custom::PopulateTable",
|
|
"Properties": {
|
|
"ServiceToken": { "Fn::GetAtt" : ["CFHelper", "DynamoDBPutItemsFunctionArn"] },
|
|
"TableName": "your-table-name",
|
|
"Items": [
|
|
{
|
|
"key": "foo1",
|
|
"value": {
|
|
"bar": 1.5,
|
|
"baz": "qwerty"
|
|
}
|
|
},
|
|
{
|
|
"key": "foo2",
|
|
"value": false
|
|
}
|
|
]
|
|
},
|
|
"DependsOn": [
|
|
"CFHelper"
|
|
]
|
|
}
|
|
```
|
|
|
|
|
|
## Warning
|
|
If you update your helper stack, all client stacks (i.e. other stacks that use the helpers) will be okay
|
|
after the new version is deployed. But some helpers rely on stored data to do the delete when the client
|
|
stack is deleted. This is stored in a DynamoDB table and will get blown away when you delete-then-recreate
|
|
the helper stack. You can get around this by updating the stack instead of tearing it down and recreating
|
|
it. But it's probably a good idea to export the helper's DynamoDB table so you have a backup. If you lose
|
|
the data, when you tear down client stacks, you may have to manually delete some resources created by the
|
|
helpers.
|
|
|
|
|
|
## A note on AWS Regions
|
|
At times, AWS can be touchy when it comes to referencing resources across regions. With that in mind, this
|
|
project is replicated to all regions - so you simply need to reference the version that lives in the S3
|
|
bucket that corresponds to the region you're working in. If there is a new region that is not part of the
|
|
ring of replications, please create an [issue](https://github.com/gilt/cloudformation-helpers/issues).
|
|
|
|
|
|
## Steps for adding a new included function
|
|
|
|
1. Find the file to edit.
|
|
* If a file already exists for the AWS service in the [aws directory](aws), add the code there.
|
|
* If not, create a new file for the AWS service.
|
|
2. Implement the function
|
|
* Add a new class that implements base.Handler for the new functionality. In most cases, it will be sufficient
|
|
to pass all parameters (other than ServiceToken through to the AWS client).
|
|
* If the 'delete' can't be implemented using the 'create' parameters, make sure the 'create' returns all
|
|
necessary data needed for the 'delete' (such as unknowable ids); these will be stored in DynamoDB for future
|
|
reference in the 'delete'.
|
|
3. Add a new export method that calls the new class.
|
|
4. Add two new Resources to the [create_cloudformation_helper_functions.template](create_cloudformation_helper_functions.template):
|
|
* The Role for the new function.
|
|
* The function itself (referencing the new method from #3).
|
|
5. Add an output for the function ARN to the template.
|
|
6. Add an example template to the [test directory](test/aws).
|
|
7. Update the [README](README.md) to explain the new helper function.
|
|
|
|
|
|
## Included functions
|
|
|
|
### Create a full API in API Gateway
|
|
|
|
Pass in all of the components of the API endpoints, and the API will be created in API Gateway.
|
|
|
|
This will delete the entire API when the corresponding stack is deleted.
|
|
|
|
#### Parameters
|
|
|
|
##### name
|
|
The name of the API, seen in the list of APIs in AWS Console.
|
|
|
|
##### description
|
|
The description of what the API does.
|
|
|
|
##### endpoints
|
|
A JSON array of (see the example template below for a working example):
|
|
|
|
```javascript
|
|
{
|
|
"resource": {
|
|
"sub-resource": {
|
|
"GET": {
|
|
"authorizationType": "NONE",
|
|
"integration": {
|
|
"type": "MOCK",
|
|
"contentType": "text/html"
|
|
}
|
|
}
|
|
},
|
|
"POST": {
|
|
"authorizationType": "NONE",
|
|
"integration": {
|
|
"type": "AWS",
|
|
"integrationHttpMethod": "POST",
|
|
"uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:*:*:function:your-function-name/invocations",
|
|
"contentType": "application/json"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
1. The properties of the JSON document are either a) the HTTP method or b) a sub-resource.
|
|
2. The resources cannot include a '/' - any resources nested in the path should also be nested
|
|
in the JSON document.
|
|
3. The parameters of each http method object are the same as putMethod callh ere:
|
|
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/APIGateway.html#putMethod-property -
|
|
except that httpMethod, resourceId, and restApiId will be added for you.
|
|
4. Each http method object can also optionally specify an "integration" property, which follows
|
|
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/APIGateway.html#putIntegration-property with
|
|
the same properties automatically filled in for you.
|
|
5. The "integration" property must include a "contentType" property that specifies the response
|
|
Content-Type of the endpoint.
|
|
|
|
#### Output
|
|
|
|
##### baseUrl
|
|
The base url of the API endpoints. Combine this with the relative paths defined in the config to
|
|
put together the full url for the API call.
|
|
|
|
##### restApiId
|
|
The id of the API that is created.
|
|
|
|
|
|
#### Reference Output Name
|
|
ApiGatewayCreateApiFunction
|
|
|
|
#### Example/Test Template
|
|
[apiGateway.createApi.template](test/aws/apiGateway.createApi.template)
|
|
|
|
|
|
### Insert items into DynamoDB
|
|
|
|
Pass in a list of items to be inserted into a DynamoDB table. This is useful to provide a template for the
|
|
content of the table, or to populate a config table. There is no data-checking, so it is up to the client
|
|
to ensure that the format of the data is correct.
|
|
|
|
Warning: it is a PUT, so it will overwrite any items that already exist for the table's primary key.
|
|
|
|
This will delete the items when the corresponding stack is deleted.
|
|
|
|
#### Parameters
|
|
|
|
##### TableName
|
|
The name of the DynamoDB table to insert into. Must exist at the time of the insert, i.e. will not create if
|
|
it does not already exist.
|
|
|
|
##### Items
|
|
A JSON array of items to be inserted, in JSON format (not DynamoDB format).
|
|
|
|
#### Output
|
|
The list of TableName/Key pairs of items created.
|
|
|
|
#### Reference Output Name
|
|
DynamoDBPutItemsFunctionArn
|
|
|
|
#### Example/Test Template
|
|
[dynamo.putItems.template](test/aws/dynamo.putItems.template)
|
|
|
|
|
|
### Create a Kinesis stream
|
|
|
|
Mirrors the [Kinesis.createStream API method](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Kinesis.html#createStream-property).
|
|
Yes, CloudFormation does support creating a Kinesis stream, but it does not allow you to specify the steam name; this
|
|
helper does.
|
|
|
|
This will delete the stream when the corresponding stack is deleted.
|
|
|
|
#### Parameters
|
|
|
|
##### ShardCount
|
|
The number of shards for the stream. Required; no default.
|
|
|
|
##### StreamName
|
|
The name of the stream. Required.
|
|
|
|
#### Output
|
|
StreamName and Arn (of the stream) - matches the output of the existing CloudFormation task.
|
|
|
|
#### Reference Output Name
|
|
KinesisCreateStreamFunctionArn
|
|
|
|
#### Example/Test Template
|
|
[kinesis.createStream.template](test/aws/kinesis.createStream.template)
|
|
|
|
|
|
### Put CloudWatch Logs Metric Filter
|
|
|
|
Mirrors the [CloudWatchLogs.putMetricFilter API method](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchLogs.html#putMetricFilter-property).
|
|
|
|
This will delete the metric filter when the corresponding stack is deleted.
|
|
|
|
#### Parameters
|
|
Exactly the same as the documentation above.
|
|
|
|
#### Reference Output Name
|
|
CloudWatchLogsPutMetricFilterFunctionArn
|
|
|
|
#### Example/Test Template
|
|
[cloudWatchLogs.putMetricFilter.template](test/aws/cloudWatchLogs.putMetricFilter.template)
|
|
|
|
|
|
### Put S3 Objects
|
|
|
|
Mirrors the [S3.putObject API method](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property).
|
|
|
|
This will delete the objects and any added sub-objects (for "folders") when the corresponding stack is deleted.
|
|
|
|
#### Parameters
|
|
|
|
##### Bucket
|
|
The S3 bucket to put the object into
|
|
|
|
##### Key
|
|
The name of the object. Can include a "path" for organization in S3.
|
|
|
|
##### Body
|
|
The content of the object being put into S3 (a string).
|
|
|
|
##### Other
|
|
Please see the reference above for further parameters - these are only the most commonly-used ones.
|
|
|
|
#### Output
|
|
The result of the S3.PutObject API method.
|
|
|
|
#### Reference Output Name
|
|
S3PutObjectFunctionArn
|
|
|
|
#### Example/Test Template
|
|
[s3.putObject.template](test/aws/s3.putObject.template)
|
|
|
|
|
|
### Put S3 Bucket Policy
|
|
|
|
Mirrors the [S3.putBucketPolicy API method](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putBucketPolicy-property).
|
|
|
|
This will replace the existing policy if any is already configured.
|
|
|
|
#### Parameters
|
|
|
|
##### Bucket
|
|
The S3 bucket to put the policy
|
|
|
|
##### Policy
|
|
The policy to put (it is a string containing a JSON description of the policy. All quotes in the policy must hence be escaped)
|
|
|
|
#### Reference Output Name
|
|
S3PutBucketPolicyFunctionArn
|
|
|
|
#### Example/Test Template
|
|
[s3.putBucketPolicy.template](test/aws/s3.putBucketPolicy.template)
|
|
|
|
### Subscribe to SNS topics
|
|
|
|
Mirrors the [SNS.Subscribe API method](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SNS.html#subscribe-property).
|
|
To be used when the SNS topic already exists (since CloudFormation allows subscriptions to be created when creating SNS
|
|
topics only).
|
|
|
|
This will delete the subscription when the corresponding stack is deleted.
|
|
|
|
#### Parameters
|
|
|
|
##### Endpoint
|
|
The endpoint that receives the SNS messages.
|
|
|
|
##### Protocol
|
|
The type of endpoint. Can be one of the following values: application, email, email-json, http, https, lambda, sms, sqs.
|
|
|
|
##### TopicArn
|
|
The SNS topic to subscribe to.
|
|
|
|
#### Output
|
|
The result of the SNS.Subscribe API method, i.e. SubscriptionArn
|
|
|
|
#### Reference Output Name
|
|
SnsSubscribeFunctionArn
|
|
|
|
#### Example/Test Template
|
|
[sns.subscribe.template](test/aws/sns.subscribe.template)
|
|
|
|
|
|
### Create a SES Receipt Rule
|
|
|
|
Allows to create an SES Receipt Rule inside an existing SES Rule set (active or not).
|
|
Mirrors the [SES.CreateReceipRule API method](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SES.html#createReceiptRule-property).
|
|
This will delete the rule when the corresponding stack is deleted.
|
|
|
|
#### Paramters
|
|
|
|
See the reference above or the example below for full list of parameters. All parameters are directly passed 'as is' except boolean which are converted.
|
|
|
|
#### Reference Output Name
|
|
SesCreateReceiptRuleFunctionArn
|
|
|
|
#### Example/Test Template
|
|
[ses.createReceiptRule.template](test/aws/ses.createReceiptRule.template)
|
|
|
|
|
|
|
|
## Deployment (contributors)
|
|
After making changes (i.e. adding a new helper function), please do the following:
|
|
|
|
1. Upload this zipped repo to the com.gilt.public.backoffice.us-east-1/lambda_functions bucket. To produce the .zip file:
|
|
|
|
```
|
|
zip -r cloudformation-helpers.zip . -x *.git* -x *cloudformation-helpers.zip*
|
|
```
|
|
|
|
Unfortunately we can't use the Github .zip file directly, because it zips the code into a subdirectory named after
|
|
the repo; AWS Lambda then can't find the .js file containing the helper functions because it is not on the top-level.
|
|
|
|
2. Upload the edited create_cloudformation_helper_functions.template to com.gilt.public.backoffice.us-east-1/cloudformation_templates
|
|
|
|
|
|
## License
|
|
Copyright 2016 Gilt Groupe, Inc.
|
|
|
|
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
|