ffmpeg streaming to KVS
Amazon Kinesis Video Streams can be used to get access to other AWS services with streaming video. In this demo, we will see how to use Node.js and ffmpeg to convert a file into a stream that can be sent to KVS.
- An object is created in the Amazon Simple Storage Service (Amazon S3) bucket. When this happens, a notification is sent to the associated AWS Lambda function.
- This Lambda makes a request to the Application Load Balancer associated with the AWS Fargate task with the object information.
- The Fargate application downloads the object from S3 and begins processing (processing details below).
- The Fargate application streams the contents of the object to Amazon Kinesis Video Streams.
Processing with ffmpeg
Once the file has been downloaded from S3, the application will begin processing the file with ffmpeg.
To do this, we will create a
PassThrough stream that will be used as the output of the ffmpeg pipe. By using
.native() as an input option, the ffmpeg pipe will read the contents of the file at the native frame rate. The output options used will format the output stream so that it can be used by KVS.
Because Kinesis Video Streams PutMedia does not have a corresponding AWS SDK command, we must use a direct HTTP request. This involves several extra steps:
The first step to sending a stream to KVS is to get an endpoint for the stream we are going to use. We have already created the stream during the deployment of the CDK, so we will use that stream as the target.
In order to stream to the KVS, we will need to assume an IAM role that has permission to do this. More details about doing this using a local Docker are outlined below.
During the deployment of the CDK, an IAM Role is created. This is the role that we will assume and use to
PutMedia. This Role is passed to the application through environment variables.
Using the credentials from the assumed role, the Stream ARN, and the DataEndpoint, we will use aws4 to sign the request we will be making and get a URL to use in the
Next, we will combine all of the pieces together as we prepare to
PutMedia. Using the stream generated by
ffmpeg and the url generated by
aws4, we will create a
POST request using axios.
Finally, we will make the
POST request to the
PutMedia API and process the responses. For every fragment sent, KVS will acknowledge with a response:
In order to test this demo, you can upload a movie file to the S3 bucket that is created during the CDK deployment. This will start processing on the Fargate container and streaming to KVS. This stream can be seen in the KVS Console and finding the stream named
This stream can be viewed with
Developing Locally with Docker
Because development like this can require extensive trial and error testing, it is useful to test locally with a Docker image before deploying to Fargate. This will involve several steps:
- Build a local Docker container
- Pass credentials and environment variables to this container
- Run the local Docker container
- Trigger this container
Build the Docker container
To do this, we will use Docker Compose and a
Pass credentials and environment variables
This will use the
Dockerfile used in the deployment with environment variables and a mounted volume that references
~./aws. This will allow us to pass credentials from the
~.aws/credentials file and use the associated
Alternatively, you can export
AWS_SESSION_TOKEN as environment variables.
KVS_STREAM_ARN are created during the deployment of the CDK and passed to the
.env file through the outputs of the CDK.
Run the Docker container
To run the Docker container:
This will re-build and run the Docker container
Trigger the Docker container
Postman is a useful tool for making API request. This is similar to what we will be doing with the Lambda function and serves as a convenient way to test locally. The Docker container will be running on port 80, so we will make a
POST request to
localhost:80/processObject passing the
keyName of the object to be processed.
Using this demo
- yarn installed
- ARM processor
- Docker desktop running