Amazon s3 + Lambda + API Gateway

3 min readJun 13, 2023

Challenge is to create an endpoint to validate the file content prior to sending it to the s3 bucket.

We have used API Gateway to expose an endpoint to upload files, Amazon Lambda to validate the file content, s3 to store the validated files & SAM templates to automate the whole architecture resources.

Cloud Architecture

Implementation

So let’s go step by step on how to achieve this.

SAM Template to create RestApi with multipart enabled.

Step 1

Create a REST API Gateway with required resource & method.
Make sure to enable its configuration to accept multipart form file data.
Attach Lambda (that will be created in Step 2) as Integration proxy to the method of the Rest API.

Step 2

Create lambda with policies as AWSLambdaBasicExecutionRole & S3CrudPolicy defined in SAM template. This will allow Lambda to access s3 bucket.

Before writing the python code, first understand how the gateway sends file to lambda?

API Gateway sends the file content to lambda in the “event” parameter of the lambda handler in the form of JSON. Sample event json will look like the following.

{
  "resource":"/file",
  "path":"/report",
  "httpMethod":"POST",
  "headers":{},
  "multiValueHeaders":{},
  "queryStringParameters":"None",
  "multiValueQueryStringParameters":"None",
  "pathParameters":"None",
  "stageVariables":"None",
  "requestContext":{},
  "body":"---------------------------048403183655214093472440\r\nContent-Disposition: form-data; name=\"file\"; filename=\"sample.csv\"\r\nContent-Type: text/csv\r\n\r\nName,Roll No.,Date"
}

“event” json includes all information regarding that file in “body” field as follows.

Content-Disposition, name, filename, Content-Type & actual File Content

Sample Validation Function

Step 3

Parse the “event” json & extract the contents of the file. We’re using requests_toolbelt multipart python library to decode the file contents. Now that you have the file content, write the required logic to validate the content of the file.

Now after validation, we need to send the file to s3 after restoring back all its content. Use the MultipartDecoder to decode the file content and then convert the file content using bytes with ‘utf-8’ format as shown in the code.