Stream Your Data to Kinesis in a Snap with API Gateway and the AWS CDK was initially published on Wednesday January 04 2023 on the Tech Dev Blog. For the latest up-to-date content, fresh out of the oven, visit https://techdevblog.io and subscribe to our newsletter!

Welcome to this tutorial on integrating AWS API Gateway and Kinesis using API Gateway VTL Mapping Templates and the AWS CDK! In this tutorial, we will be setting up an API Gateway endpoint that directly writes to a Kinesis stream, without using an AWS Lambda function as an intermediary.

Get Started

To get started, you'll need to have the AWS CDK installed and configured on your local machine. If you don't have it set up yet, you can follow the instructions here to get started.

Once you have the AWS CDK set up, you'll also need to create a new CDK project. You can do this by running the following command in your terminal:

cdk init sample-app --language=typescript

This will create a new directory called sample-app, which will contain the boilerplate code for your CDK project.

Next, you'll need to install the @aws-cdk/aws-apigateway and @aws-cdk/aws-kinesis modules by running the following command in your terminal:

npm install @aws-cdk/aws-apigateway @aws-cdk/aws-kinesis

Once these modules are installed, you can start building your API Gateway and Kinesis integration.

Integrate API Gateway & Kinesis

Kinesis Stream

First, let's create a new Kinesis stream by adding the following code to your CDK project:

import * as kinesis from '@aws-cdk/aws-kinesis';

const stream = new kinesis.Stream(this, 'MyStream', {
  shardCount: 1
});

This code creates a new Kinesis stream called MyStream, with a single shard.

API Gateway Endpoint & Integration

Next, let's create an API Gateway endpoint that writes to this stream. Add the following code to your CDK project:

import * as apigateway from '@aws-cdk/aws-apigateway';
import * as iam from '@aws-cdk/aws-iam';

const api = new apigateway.RestApi(this, 'MyApi', {
  restApiName: 'My API'
});

const role = new iam.Role(this, 'MyRole', {
  assumedBy: new iam.ServicePrincipal('apigateway.amazonaws.com'),
  inlinePolicies: {
    myPolicy: new iam.PolicyDocument({
      statements: [
        new iam.PolicyStatement({
          actions: ['kinesis:PutRecord'],
          resources: [stream.streamArn]
        })
      ]
    })
  }
});

const postIntegration = new apigateway.AwsIntegration({
  service: 'kinesis',
  action: 'PutRecord',
  options: {
    credentialsRole: role,
    integrationHttpMethod: 'POST',
    passthroughBehavior: apigateway.PassthroughBehavior.NEVER,
    requestParameters: {
      'integration.request.header.Content-Type': "'application/x-amz-json-1.1'",
      'integration.request.header.X-Amz-Target': "'Kinesis_20131202.PutRecord'"
    },
    requestTemplates: {
      'application/json': `{
        "StreamName": "${stream.streamName}",
        "Data": $input.json('$')
      }`
    }
  }
});

const postMethod = new apigateway.HttpMethod(this, 'PostMethod', {
  httpMethod: 'POST',
  integration: postIntegration,
  requestParameters:{
      'integration.request.header.Content-Type': true
  }
});

const recordResource = api.root.addResource('record');
recordResource.addMethod('POST', postMethod);

This code creates a new IAM role called MyRole, which is assumed by the apigateway.amazonaws.com service principal. The role also has an inline policy attached to it, called myPolicy, which allows the PutRecord action on the MyStream Kinesis stream.

The role variable is then passed as the credentialsRole option to the AwsIntegration object in the previous example, giving the API Gateway the necessary permissions to write to the Kinesis stream.

We then create a new REST API called MyAPI, with a single POST method that writes to the MyStream Kinesis stream we created earlier.

The AwsIntegration object is used to integrate the API Gateway with the Kinesis service, using the PutRecord action. The requestTemplates property is a mapping of content-types to VTL (Velocity Template Language) templates, which are used to transform the incoming request data into the format required by the Kinesis PutRecord action. In this case, we are specifying a template that takes the incoming JSON payload and puts it into the Data field of the PutRecord request.

Finally, the HttpMethod object is used to define the actual POST method for the record resource, and associate it with the postIntegration integration we created earlier.

Conclusion

And that's it! You now have an API Gateway endpoint that directly writes to a Kinesis stream, without using an AWS Lambda function.

I hope this tutorial was helpful and got you up and running with integrating AWS API Gateway and Kinesis. If you have any questions or run into any issues, don't hesitate to ask!