
subaud is authored by Court Schuett (he/him). While Court is an 8090 Inc employee, opinions and statements on this blog are his own.


Court Schuett


# Deploying an NVIDIA NIM to EC2

This demo will show you how to deploy an [NVIDIA NIM](https://www.nvidia.com/en-us/ai/) to an Amazon EC2 instance and enable it for inference.

## GitHub Repository

https://github.com/schuettc/ec2-nim-deployment

## Requirements

- NGC API key
- Route 53 Hosted Zone/Domain
- On-Demand vCPU quota

### NGC API Key

In order to use this demo, you will need an NGC API key. This key can be generated here: https://org.ngc.nvidia.com/setup/personal-keys

We will upload this key to [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) and will be used within the EC2 instance to log in to NGC to download the container.

### Route 53 Hosted Zone/Domain

To enable [TLS security on the Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html), we need to generate a certificate using [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/). In order to generate this certificate, we need a domain and Hosted Zone in [Amazon Route 53](https://aws.amazon.com/route53/). We will configure this before launching the stack.

### On-Demand vCPU quota

This demo will launch a G5 EC2 instance. The number of vCPUs used by this instance will depend on the size. For example, a G5.12xlarge instance will use 48 vCPUs. Be sure to [check your utilization and quota value](https://us-west-2.console.aws.amazon.com/servicequotas/home/services/ec2/quotas/L-DB2E81BA) to make sure you can launch this instance. You can request an increase if necessary.

## Setup

### .env Configuration

Before launching, be sure to configure a `.env` file. Many of these fields can be left with their defaults.

```bash
# Stack Configuration
STACK_NAME=nim-stack
AWS_REGION=us-west-2
TEMPLATE_FILE=nims-stack.yaml

# Instance Configuration
INSTANCE_TYPE=g5.12xlarge

# AWS Resources
KEY_NAME=REPLACE_WITH_YOUR_KEY
VPC_ID=
SUBNET_IDS=

# Domain Configuration
DOMAIN_NAME=HOSTNAME.EXAMPLE.COM
HOSTED_ZONE_ID=REPLACE_WITH_YOUR_HOSTED_ZONE_ID

# NVIDIA Configuration
NGC_API_KEY_SECRET_NAME=NGCApiKeySecret
NGC_API_KEY=REPLACE_WITH_YOUR_NGC_API_KEY
REPOSITORY=nim/meta/llama3-8b-instruct
LATEST_TAG=1.0.0
```

You will need to update the following before launching:

```bash
# AWS Resources
KEY_NAME=REPLACE_WITH_YOUR_KEY

# Domain Configuration
DOMAIN_NAME=HOSTNAME.EXAMPLE.COM
HOSTED_ZONE_ID=REPLACE_WITH_YOUR_HOSTED_ZONE_ID

# NVIDIA Configuration
NGC_API_KEY=REPLACE_WITH_YOUR_NGC_API_KEY
```

### Secret Update

To upload the NGC API key to Secrets Manager, run the script:

```bash
./update_ngc_api_key.sh
```

This API Key will be retrieved by the EC2 instance during deployment.

### Launch Cloudformation Stack

You can either create the stack in the AWS Console, or through the included script. The script will populate the parameters required that were configured in the `.env` file. You will need to configure the required parameters in the console.

## Inference

Once the deployment succeeds, you can run inference requests against it using a variety of tools. Included is a script that uses `curl` to make a request to the endpoint using the configured domain name.

```bash
curl -s -X POST "https://$DOMAIN_NAME/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "'"$model_id"'",
        "messages": [
            {"role": "user", "content": "Hello! How are you?"},
            {"role": "assistant", "content": "Hi! I am quite well, how can I help you today?"},
            {"role": "user", "content": "Write a short limerick about the wonders of GPU computing."}
        ],
        "max_tokens": 100
    }' | jq '.choices[0].message.content'
```

To test the endpoint and inference:

```bash
./inference.sh
```

# How it Works

## Components

![Overview](/images/blog/deploying-an-nvidia-nim-to-ec2/Overview.png)

### EC2 Instance

- Uses a G5 instance type (default: g5.12xlarge) for GPU capabilities
- Runs Ubuntu with NVIDIA drivers, Docker, and NVIDIA Container Toolkit
- Automatically pulls and runs the specified NIM container

### Auto Scaling Group (ASG)

- Manages a single EC2 instance
- Ensures high availability and easy updates
- Uses a Launch Template for instance configuration

### Application Load Balancer (ALB)

- Provides HTTPS termination
- Routes traffic to the EC2 instance
- Uses an ACM certificate for HTTPS

### VPC and Networking

- Creates a new VPC with public subnets (or uses existing ones)
- Sets up necessary security groups and routing

### Route 53

- Creates a DNS record pointing to the ALB

## EC2 Configuration

1. The stack creates or uses existing VPC and subnets
2. An EC2 instance is launched with the necessary software and configurations
3. The NIM container is pulled and started on the EC2 instance
4. An ALB is set up to route traffic to the EC2 instance
5. A Route 53 record is created to point the specified domain to the ALB

The stack uses conditions to determine whether to create new VPC resources or use existing ones, making it flexible for different deployment scenarios.

## UserData Script Details

The UserData script in the Launch Template performs several key setup tasks:

1. **System Updates and Dependencies**

   - Updates the package list
   - Installs Python3-pip, AWS CLI, and CloudFormation helper scripts

2. **NVIDIA Driver and CUDA Installation**

   - Installs Linux headers
   - Adds NVIDIA CUDA repository
   - Installs CUDA drivers

3. **Docker Installation**

   - Adds Docker repository
   - Installs Docker CE

4. **NVIDIA Container Toolkit Installation**

   - Adds NVIDIA Docker repository
   - Installs nvidia-docker2
   - Restarts Docker service

5. **NGC API Key Retrieval**

   - Fetches the NGC API Key from AWS Secrets Manager

6. **NIM Container Setup and Run**

   - Sets up environment variables for the container
   - Creates a local cache directory
   - Logs in to the NGC container registry
   - Runs the NIM container with the following configuration:
     - Uses NVIDIA runtime
     - Exposes port 8000
     - Mounts a local cache volume
     - Sets the NGC API Key as an environment variable

7. **Signaling**
   - Signals the success or failure of the setup to CloudFormation

## Docker Container and ALB Interaction

1. **Container Exposure**

   - The NIM container exposes port 8000 inside the EC2 instance

2. **Security Group Configuration**

   - The EC2 security group allows inbound traffic on port 8000 from the ALB security group

3. **ALB Target Group**

   - An ALB Target Group is created with the following properties:
     - Protocol: HTTP
     - Port: 8000
     - Health check path: /health

4. **ALB Listener**

   - The ALB listener is configured for HTTPS (port 443)
   - It forwards traffic to the Target Group

5. **Request Flow**

   - Incoming HTTPS requests hit the ALB
   - The ALB terminates SSL and forwards the request as HTTP to the EC2 instance on port 8000
   - The Docker container receives the request on port 8000 and processes it

6. **Health Checks**
   - The ALB periodically sends health check requests to the `/health` endpoint
   - The NIM container responds to these health checks, allowing the ALB to determine if the instance is healthy

## Instance Types and Costs

This CloudFormation template supports the following instance types for NVIDIA AI Microservices (NIM):

### G5 Instances

- g5.xlarge: 1 GPU, 4 vCPUs, 16 GB RAM
- g5.2xlarge: 1 GPU, 8 vCPUs, 32 GB RAM
- g5.4xlarge: 1 GPU, 16 vCPUs, 64 GB RAM
- g5.8xlarge: 1 GPU, 32 vCPUs, 128 GB RAM
- g5.16xlarge: 1 GPU, 64 vCPUs, 256 GB RAM
- g5.12xlarge: 4 GPUs, 48 vCPUs, 192 GB RAM
- g5.24xlarge: 4 GPUs, 96 vCPUs, 384 GB RAM
- g5.48xlarge: 8 GPUs, 192 vCPUs, 768 GB RAM

⚠️ **Warning**: These instances can be very expensive to run. Please check the current AWS pricing for your region before deploying. For example, as of 2023, the costs for G5 instances in the US East (N. Virginia) region range from $1.006/hour for g5.xlarge to $16.288/hour for g5.48xlarge.

Always remember to stop or terminate your instances when not in use to avoid unnecessary charges.


In this demo we will see how to deploy an NVIDIA NIM to EC2 with Docker to do your own inference

Deploying an NVIDIA NIM to EC2


# Simple Dev EC2 Instance

This simple Cloudformation template will deploy a new EC2 instance in a new VPC that can be used for dev work. This instance uses [Ubuntu](https://ubuntu.com/) with either an [ARM64](https://aws.amazon.com/ec2/instance-types/m7g/) or [X86_64](https://aws.amazon.com/ec2/instance-types/m5/) CPU and can be deployed as a `LARGE`, `XLARGE`, or `2XLARGE` instance size. It comes pre-installed with several commonly used packages:

- NodeJS 18
- Python 3.10
- pip
- jq
- Docker

## Adding Keys (optional)

If you'd like to copy private or public keys to the instance during deployment, you can add them to SSM before deploying.

```bash
aws ssm put-parameter --name /devServer/publicKey --overwrite --type SecureString --value $(cat ~/.ssh/key.pub | base64)
aws ssm put-parameter --name /devServer/githubKey --overwrite --type SecureString --value $(cat ~/.ssh/private.pem | base64)
```

These commands will base64 encode the contents of these files and store in them in [SSM Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html).

During deployment, these parameters will be read and used to configure the instance.

```bash
GITHUB_KEY=$(aws ssm get-parameter --with-decryption --name /devServer/githubKey --query Parameter.Value --output text 2> /dev/null)
EXIT_CODE=$?
if [[ "$EXIT_CODE" -ne 0 ]]; then echo 'No GitHub Key found'; else echo 'Found GitHub Key' && echo $GITHUB_KEY | base64 -d > /home/ec2-user/.ssh/github.pem; fi
PUBLIC_KEY=$(aws ssm get-parameter --with-decryption --name /devServer/publicKey --query Parameter.Value --output text 2> /dev/null)
EXIT_CODE=$?
if [[ "$EXIT_CODE" -ne 0 ]]; then echo 'No Public Key found'; else echo 'Found Public Key' && echo $PUBLIC_KEY | base64 -d >> /home/ec2-user/.ssh/authorized_keys; fi
chmod 600 /home/ec2-user/.ssh/github.pem
chown ec2-user:ec2-user /home/ec2-user/.ssh/*
```

## Security Group

Several TCP ports will be opened in the Security Group.

```yaml
SecurityGroupIngress:
  - IpProtocol: tcp
    FromPort: '22'
    ToPort: '22'
    CidrIp: 0.0.0.0/0
  - IpProtocol: tcp
    FromPort: '80'
    ToPort: '80'
    CidrIp: 0.0.0.0/0
  - IpProtocol: tcp
    FromPort: '443'
    ToPort: '443'
    CidrIp: 0.0.0.0/0
  - IpProtocol: tcp
    FromPort: '3000'
    ToPort: '3000'
    CidrIp: 0.0.0.0/0
  - IpProtocol: tcp
    FromPort: '8080'
    ToPort: '8080'
    CidrIp: 0.0.0.0/0
```

This will open the following ports:

- 22
- 80
- 443
- 3000
- 8080

## Deployment Options

![DeploymentOptions](/images/blog/DeploymentOptions.png)

During deployment, you can choose the CPU Type, the Instance Size, and if you wish to add Administrator Access.

> Adding Administrator Access will allow you to use all AWS resources, but **must** be used with caution

[![Launch](https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://subaud-resources.s3.amazonaws.com/ec2-cloudformation/template.yaml&stackName=DevServer)


This Cloudformation template will deploy a simple EC2 instance to be used as a development server.  Several common packages are included in the deployment and keys can be installed during deployment.

Deploy a simple EC2 Instance via Cloudformation


### Overview

![EC2](/images/blog/Example-EC2.png)

Cloud-init is a powerful way of configuring an EC2 instance. CDK can create an EC2 instance and use cloud-init to pass dynamically created variables to this EC2 instance which can be used to configure the instance. When combined, these tools can be used to create fully configured and deployed dynamic EC2 instances.

[This GitHub repo](https://github.com/schuettc/cdk-ec2-example) will deploy an EC2 instance using CDK. This EC2 instance will download and install [Asterisk 18](https://wiki.asterisk.org/wiki/display/AST/Asterisk+18+Documentation) on the EC2 instance. Additionally, it will create a `pjsip.conf` file and configure that using the Elastic IP address created by the CDK.

Once the Elastic IP is created, by the CDK, it is passed to the EC2 Instance using cloud-init:

```ts
      init: ec2.CloudFormationInit.fromConfigSets({
        configSets: {
          default: ['install', 'config'],
        },
        configs: {
          install: new ec2.InitConfig([
            ec2.InitFile.fromObject('/etc/config.json', {
              IP: ec2Eip.ref,
            }),
```

This file will be stored on the newly created instance based on resources created by the CDK. In this example, the Elastic IP address is passed to the JSON object. This information can also be [queried with metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html), but is used in this example as information that can be dynamically created by the CDK and passed to the EC2 Instance.

Files can be loaded to the EC2 instance from [multiple sources](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.InitSource.html). In this example, we are loading a bash script from the local deployment of the CDK and executing it on the EC2 instance. This cloud-init configuration in the CDK will load the [`install.sh`](https://github.com/schuettc/cdk-ec2-example/blob/main/src/asteriskConfig/install.sh) file from the `./src/asteriskConfig/` directory into the `/etc/` directory of the Instance and then execute it.

```ts
            ec2.InitFile.fromFileInline(
              '/etc/install.sh',
              './src/asteriskConfig/install.sh',
            ),
            ec2.InitCommand.shellCommand('chmod +x /etc/install.sh'),
            ec2.InitCommand.shellCommand('cd /tmp'),
            ec2.InitCommand.shellCommand('/etc/install.sh'),
          ]),
```

The example script will download and install an Asterisk PBX on to the EC2 instance. Additional files are included in the GitHub repo that are used to more fully configure the Asterisk server using `jq` and `sed` with the information passed to the Instance from the CDK:

```bash
IP=$( jq -r '.IP' /etc/config.json )

sed -i "s/IP_ADDRESS/$IP/g" /etc/asterisk/pjsip.conf
```

[initOptions](https://docs.aws.amazon.com/cdk/api/v2//docs/aws-cdk-lib.aws_ec2.ApplyCloudFormationInitOptions.html) can also be used to ensure that the configuration of the EC2 instance succeeds.

```json
      initOptions: {
        timeout: Duration.minutes(15),
      },
```

If the timeout fails or the Instance returns a failure, the CDK will roll back to the previous state. When the cloud-init configuration of the Instance succeeds, it will pass this information back to the CDK. Until then, the CDK will remain in a Creating state and will not progress. This is useful when you want to ensure that the build and configuration of the EC2 instance succeeds before deploying additional resources with the CDK.

More details are provided in the [GitHub repo](https://github.com/schuettc/cdk-ec2-example).


Cloud-init is a powerful way of configuring an EC2 instance. CDK can create an EC2 instance and use cloud-init to pass dynamically created variables to this EC2 instance which can be used to configure the instance.  When combined, these tools can be used to create fully configured and deployed dynamic EC2 instances.

Ec2

Deploying an NVIDIA NIM to EC2

Deploy a simple EC2 Instance via Cloudformation

Building an EC2 Instance with CDKv2 and Cloud-Init