
subaud is authored by Court Schuett (he/him). While Court is an 8090 Inc employee, opinions and statements on this blog are his own.


Court Schuett

subaud.io


# AWS CDK Cognito Protected API Demo

This project demonstrates how to create a protected API using AWS CDK with Cognito authentication. It sets up a complete serverless infrastructure including API Gateway, Lambda, and Cognito User Pool. To test the API, we'll configure and use Postman with OAuth 2.0 to sign in and get an access token. This can be an easy way to test your API without setting up a frontend while still using JWTs for authentication.

## Architecture

The infrastructure consists of three main components:

**Cognito User Pool**: Handles user authentication

- Email-based sign-up and sign-in
- Password policies and email verification
- OAuth 2.0 flows support
- Hosted UI for sign-up/sign-in

**API Gateway**: Protected REST API

- Cognito authorizer integration
- Single GET endpoint `/hello`
- JWT token validation

**Lambda Function**: Backend logic

- Returns protected data
- Includes user information from Cognito claims

![Overview](/images/blog/cdk-cognito-api-postman/CogntioAPIPostman.drawio.png)

## Infrastructure Code Overview

### Cognito Setup (CognitoConstruct)

```typescript
this.userPool = new UserPool(this, "UserPool", {
  userPoolName: "postman-demo-user-pool",
  selfSignUpEnabled: true,
  signInAliases: { email: true },
  standardAttributes: {
    email: { required: true, mutable: true },
    givenName: { required: true, mutable: true },
  },
  passwordPolicy: {
    minLength: 8,
    requireLowercase: true,
    requireUppercase: true,
    requireDigits: true,
    requireSymbols: true,
  },
  signInCaseSensitive: false,
  userVerification: {
    emailStyle: VerificationEmailStyle.LINK,
    emailSubject: "Verify your email for our demo app!",
  },
});

this.userPoolClient = new UserPoolClient(this, "UserPoolClient", {
  userPool: this.userPool,
  oAuth: {
    flows: {
      authorizationCodeGrant: true,
      implicitCodeGrant: true,
    },
    scopes: [OAuthScope.EMAIL, OAuthScope.OPENID, OAuthScope.PROFILE],
    callbackUrls: [
      "http://localhost:3000/callback",
      "https://oauth.pstmn.io/v1/callback",
    ],
  },
  preventUserExistenceErrors: true,
  accessTokenValidity: Duration.minutes(60),
  idTokenValidity: Duration.minutes(60),
  refreshTokenValidity: Duration.days(30),
});
```

### API Gateway Setup (ApiGatewayConstruct)

```typescript
const api = new RestApi(this, "ProtectedApi", {
  restApiName: "Protected API Demo",
  description: "API Gateway with Cognito authorization",
});

const authorizer = new CognitoUserPoolsAuthorizer(this, "ApiAuthorizer", {
  cognitoUserPools: [props.userPool],
  authorizerName: "CognitoAuthorizer",
  identitySource: "method.request.header.Authorization",
  resultsCacheTtl: Duration.minutes(5),
});

const resource = api.root.addResource("hello");
resource.addMethod("GET", new LambdaIntegration(props.lambdaFunction), {
  authorizer: authorizer,
  authorizationType: AuthorizationType.COGNITO,
  authorizationScopes: ["email", "openid", "profile"],
});
```

## Understanding Cognito Tokens & API Security

### Token Flow

When a user authenticates through Cognito, three different tokens are issued:

1. **ID Token**: A JWT containing user identity claims (profile info, email, etc.)
2. **Access Token**: A JWT specifically for authorizing API requests
3. **Refresh Token**: Used to obtain new ID and Access tokens when they expire

### How API Gateway Authorization Works

1. Client obtains tokens through Cognito authentication
2. Client includes Access Token in API request (`Authorization: Bearer <token>`)
3. API Gateway:
   - Validates token signature using Cognito's public keys
   - Checks token expiration
   - Verifies audience (client_id) and issuer (Cognito User Pool)
   - Passes user claims to Lambda in `event.requestContext.authorizer`

### JWT Token Structure

Each token is a JSON Web Token (JWT) with three parts:

```bash
header.payload.signature
```

#### ID Token Example

https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-the-id-token.html

```json
{
  "sub": "user-uuid",
  "email_verified": true,
  "iss": "https://cognito-idp.{region}.amazonaws.com/{userPoolId}",
  "cognito:username": "john.doe",
  "given_name": "John",
  "aud": "{clientId}",
  "token_use": "id",
  "exp": 1234567890,
  "iat": 1234567890
}
```

#### Access Token Example

https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-the-access-token.html

```json
{
  "sub": "user-uuid",
  "device_key": "device-id",
  "scope": "email openid profile",
  "iss": "https://cognito-idp.{region}.amazonaws.com/{userPoolId}",
  "token_use": "access",
  "exp": 1234567890,
  "iat": 1234567890
}
```

### Token Validation Process

https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html

1. **Structure Verification**

   - Check if token is well-formed JWT
   - Verify token hasn't expired (`exp` claim)
   - Validate token use (`token_use` claim)

2. **Signature Validation**

   - Download JWKs from Cognito
   - Verify token signature using public key
   - Ensure issuer (`iss`) matches User Pool

3. **Claims Validation**
   - Verify audience (`aud`) matches client ID
   - Check required scopes are present
   - Validate any custom claims

### Security Best Practices

#### Token Configuration

- Access tokens expire in 60 minutes (configured in Cognito client)
- ID tokens expire in 60 minutes (configured in Cognito client)
- Refresh tokens expire in 30 days
- Authorization cache TTL set to 5 minutes in API Gateway
- Tokens must be sent in Authorization header

#### Cognito Security Settings

- Case-insensitive email sign-in enabled
- User existence errors prevented (security feature)
- Password requirements enforced:
  - Minimum length: 8 characters
  - Requires lowercase letters
  - Requires uppercase letters
  - Requires numbers
  - Requires special characters
- Email verification required
- OAuth flows:
  - Authorization Code Grant (for web apps)
  - Implicit Grant (for mobile apps)
- Strict callback URL validation

#### API Gateway Security

- Cognito authorizer validates tokens automatically
- Required scopes: email, openid, profile
- Authorization caching for performance (5-minute TTL)
- Token claims available to Lambda function

### Security Benefits

- **Token-based Authentication**: Stateless, scalable authentication
- **Short-lived Tokens**: Access tokens expire in 1 hour (configurable)
- **Signature Verification**: Prevents token tampering
- **Scope-based Access**: Can restrict API access based on token scopes
- **User Context**: Lambda receives verified user information
- **Standard OAuth2/OIDC**: Industry-standard security protocols

## Testing with Postman

### **Get the Hosted UI URL**

- After deploying, find the `HostedUISignUpUrl` in CloudFormation outputs
- Create an account through the Hosted UI
- Verify your email address

![SignUp](/images/blog/cdk-cognito-api-postman/SignUp.png)
_Sign up page_

![VerificationLink](/images/blog/cdk-cognito-api-postman/VerificationLink.png)
_Verification link_

![VerificationEmail](/images/blog/cdk-cognito-api-postman/VerificationEmail.png)
_Verification email_

###**Configure Postman**

- Create a new request
- Under Authorization tab:
  - Type: OAuth 2.0
  - Grant Type: Authorization Code
  - Callback URL: https://oauth.pstmn.io/v1/callback
  - Auth URL: [CognitoDomainUrl]/oauth2/authorize
  - Access Token URL: [CognitoDomainUrl]/oauth2/token
  - Client ID: [UserPoolClientId]
  - Scope: email openid profile

![UnauthorizedGET](/images/blog/cdk-cognito-api-postman/UnauthorizedGET.png)
_Example of unauthorized GET request_

![AuthorizationSetup1](/images/blog/cdk-cognito-api-postman/AuthorizationSetup1.png)
_Authorization tab with Auth Type set to OAuth 2.0_

![AuthorizationSetup2](/images/blog/cdk-cognito-api-postman/AuthorizationSetup2.png)
_Authorization tab Configuring new token_

### **Get Access Token**

- Click "Get New Access Token"
- Log in with your Cognito credentials
- Use the token for API requests

![AuthorizationSetup3](/images/blog/cdk-cognito-api-postman/AuthorizationSetup3.png)
_Click "Get New Access Token"_

![LoginPopup](/images/blog/cdk-cognito-api-postman/LoginPopup.png)
_Login popup_

![AuthenticationComplete](/images/blog/cdk-cognito-api-postman/AuthenticationComplete.png)
_Authentication complete_

![UseToken](/images/blog/cdk-cognito-api-postman/UseToken.png)
_Click "Use Token"_

### **Make API Requests**

- GET [ApiUrl]/hello

![AuthorizedGET](/images/blog/cdk-cognito-api-postman/AuthorizedGET.png)
_Example of authorized GET request_

## Deployment

```bash
yarn
yarn cdk deploy
```

The deployment will output:

- Cognito Domain URL
- User Pool ID
- Client ID
- API Gateway URL
- Hosted UI URL for sign-up
- Auth URL
- Token URL

## Cleanup

```bash
yarn cdk destroy
```


In this demo we will see how to deploy a Cognito protected API and test it with Postman using OAuth 2.0 for authentication and login

CDK Cognito Protected API Demo with Postman


# Next.js SSR App Runner Deployment with CloudFront

This project demonstrates how to deploy a Next.js SSR application using AWS App Runner and CloudFront. It uses an AWS CDK to define and deploy the infrastructure. The project is managed with projen and includes several custom tasks to support the development process.

Update to: https://subaud.io/blog/next-js-app-runner-with-deployment

GitHub: https://github.com/schuettc/next-js-app-runner-deployment

## Architecture

The architecture consists of the following components:

- Next.js application
- AWS App Runner to host and run the Next.js application
- Amazon CloudFront as a content delivery network (CDN) in front of App Runner
- Lambda@Edge to rewrite headers and support dynamic routing
- Route53 ARecords for DNS
- AWS CDK to define and deploy the infrastructure
- GitHub actions for CI/CD

![Overview](/images/blog/updated-nextjs-app-runner-deployment-with-srr/NextJSAppCloudfront.png)

## Prerequisites

- AWS Account
- Node.js (version 18 or later)
- AWS CDK CLI
- Docker (for local development and testing)

## Getting Started

Clone the repository:

```bash
git clone https://github.com/schuettc/next-js-app-runner-deployment.git
cd next-js-app-runner-deployment
```

Install dependencies:

```bash
yarn
```

Deploy the stack:

```bash
yarn launch
```

## Project Structure

- `src/`: Contains the CDK infrastructure code
  - `appRunner.ts`: Defines the App Runner service
  - `cloudfront.ts`: Sets up the CloudFront distribution
  - `lambda.ts`: Creates the Lambda@Edge function
  - `route53.ts`: Defines the DNS records
  - `next-js-app-runner-deployment.ts`: Main stack definition
- `src/resources/app/`: Contains the Next.js application code

## Custom Origin Request Policy

This project implements a custom origin request policy for CloudFront, which is required when using App Runner as the origin. The policy allows specific headers to be forwarded to the origin:

```typescript:src/cloudfront.ts
    const customOriginRequestPolicy = new OriginRequestPolicy(
      this,
      'UserAgentRefererHeadersPolicy',
      {
        headerBehavior: OriginRequestHeaderBehavior.allowList(
          'User-Agent',
          'Referer',
        ),
      },
    );
```

This custom policy ensures that the `User-Agent` and `Referer` headers are forwarded to the App Runner service, which solves the 404 error that occurs when using AppRunner as the origin for a CloudFront distribution.

https://docs.aws.amazon.com/apprunner/latest/dg/request-route-404-troubleshoot.html

## Lambda@Edge for Header Rewriting

This project uses a Lambda@Edge function to rewrite headers, which is crucial for supporting dynamic routing in Next.js when using App Runner behind CloudFront. The function is associated with both the Origin Request and Viewer Request events in CloudFront.

Here's the Lambda@Edge function code:

```typescript:src/resources/lambdaEdge/index.ts
import { CloudFrontRequestEvent, CloudFrontRequestResult } from 'aws-lambda';

export const handler = async (
  event: CloudFrontRequestEvent,
): Promise<CloudFrontRequestResult> => {
  const request = event.Records[0].cf.request;
  const origin = request.origin?.custom;

  if (origin && origin.domainName) {
    request.headers.host = [{ key: 'Host', value: origin.domainName }];
  }

  return request;
};

```

This function rewrites the `Host` header to match the origin's domain name, ensuring that App Runner receives the correct host information. This is essential for Next.js to properly handle dynamic routes.

The Lambda@Edge function is associated with CloudFront in the following way:

```typescript
        edgeLambdas: [
          {
            functionVersion: lambdaEdgeFunction.function.currentVersion,
            eventType: LambdaEdgeEventType.ORIGIN_REQUEST,
          },
          {
            functionVersion: lambdaEdgeFunction.function.currentVersion,
            eventType: LambdaEdgeEventType.VIEWER_REQUEST,
          },
        ],
      },
```

By attaching this function to both the Origin Request and Viewer Request events, we ensure that the headers are properly rewritten for all incoming requests, allowing Next.js to handle dynamic routing correctly when deployed on App Runner behind CloudFront.

## Deployment

The project includes a GitHub Actions workflow for automated deployments. When you push changes to the `main` branch, it will automatically deploy the updated stack to AWS.

To deploy manually:

1. Configure your AWS credentials
2. Run `yarn deploy`

### projen configuration

```typescript
const deploy = project.github?.addWorkflow("deploy");
deploy?.on({
  push: {
    branches: ["main"],
  },
  workflow_dispatch: {},
});

deploy?.addJobs({
  deploy: {
    runsOn: ["ubuntu-latest"],
    permissions: {
      contents: JobPermission.READ,
      id_token: JobPermission.WRITE,
    },
    steps: [
      { uses: "actions/checkout@v4" },
      {
        uses: "actions/setup-node@v4",
        with: {
          "node-version": "18",
        },
      },
      { run: "yarn install --frozen-lockfile" },
      {
        uses: "aws-actions/configure-aws-credentials@v4",
        with: {
          "aws-access-key-id": "${{ secrets.AWS_ACCESS_KEY_ID }}",
          "aws-secret-access-key": "${{ secrets.AWS_SECRET_ACCESS_KEY }}",
          "aws-region": "${{ secrets.AWS_REGION }}",
        },
      },
      { run: "npx cdk deploy --require-approval never" },
    ],
  },
});
```

## Server Side Rendering

This NextJS includes support for Server Side Rendering (SRR) through dynamic routes and API routes:

```typescript
export default function Post({ params }: { params: { id: string } }) {
  return (
    <div>
      <h1>Post {params.id}</h1>
      <p>This is a dynamic route for post with ID: {params.id}</p>
    </div>
  );
}
```

```typescript
import { NextResponse } from "next/server";

export async function GET() {
  return NextResponse.json({ message: "Hello from the API!" });
}

export async function POST(request: Request) {
  const body = await request.json();
  return NextResponse.json({ message: "Hello from the API!", data: body });
}
```

## Local Development

To run the Next.js application locally:

Navigate to the app directory:

```bash
cd src/resources/app
```

Install dependencies:

```bash
yarn
```

Start the development server:

```bash
yarn dev
```

Open [http://localhost:3000](http://localhost:3000) in your browser

## GitHub Updates with Projen

The project uses projen to manage the project and the Next.js application. We have replaced the default projen upgrade with a custom upgrade. This allows us to upgrade the dependencies for both the main project and the Next.js application in a single step.

```typescript
const upgradeWorkflow = project.github?.addWorkflow("upgrade");
upgradeWorkflow?.on({
  schedule: [{ cron: "0 0 * * 1" }],
  workflow_dispatch: {},
});

upgradeWorkflow?.addJobs({
  upgrade: {
    runsOn: ["ubuntu-latest"],
    permissions: {
      contents: JobPermission.WRITE,
      pullRequests: JobPermission.WRITE,
    },
    steps: [
      { uses: "actions/checkout@v4" },
      { uses: "actions/setup-node@v4", with: { "node-version": "18" } },
      { run: "yarn install" },
      { run: "npx projen upgrade:projen" },
      { run: "npx projen upgrade:nextjs" },
      {
        name: "Create Pull Request",
        uses: "peter-evans/create-pull-request@v6",
        with: {
          token: "${{ secrets.PROJEN_GITHUB_TOKEN }}",
          "commit-message": "chore: upgrade dependencies",
          branch: "projen-upgrade",
          title: "chore: upgrade dependencies",
          body: "This PR upgrades project dependencies, including projen and Next.js. Auto-generated by projen upgrade workflow.",
        },
      },
    ],
  },
});

project.addTask("upgrade:nextjs", {
  exec: ["cd src/resources/app", "yarn upgrade --latest", "cd ../../.."].join(
    " && "
  ),
});

project.addTask("upgrade:projen", {
  description: "Upgrade projen dependencies",
  steps: [
    { exec: "yarn upgrade projen @projen/awscdk-app-ts" },
    { exec: "npx projen" },
  ],
});
```

## Route 53 DNS Configuration

This project uses Amazon Route 53 to manage DNS records for the deployed application. The `Route53Resources` construct in `src/route53.ts` sets up the necessary DNS records to point your domain to the CloudFront distribution.

Two A records are created:

- One for the apex domain (e.g., example.com)
- One for the www subdomain (e.g., www.example.com)

Both A records are set up as alias records pointing to the CloudFront distribution:

```typescript
new ARecord(this, "AliasRecord", {
  zone: hostedZone,
  target: RecordTarget.fromAlias(new CloudFrontTarget(props.distribution)),
  recordName: `${props.domainName}.`,
});

new ARecord(this, "WWWAliasRecord", {
  zone: hostedZone,
  target: RecordTarget.fromAlias(new CloudFrontTarget(props.distribution)),
  recordName: `www.${props.domainName}.`,
});
```

This configuration ensures that both the apex domain and the www subdomain resolve to the CloudFront distribution, which in turn serves the Next.js application running on App Runner.

By using alias records, we benefit from Route 53's ability to automatically update the DNS records if the CloudFront distribution's domain name changes, without any manual intervention.

Make sure to update your domain's name servers to use the ones provided by your Route 53 hosted zone for this configuration to take effect.


Updated NextJS App deployed to App Runner with SSR and GitHub Actions


# Multi-Region Pipeline with AWS CDK

This project implements a multi-region deployment pipeline using AWS CDK. It demonstrates how to create a centralized pipeline that can deploy resources across multiple AWS regions, with a focus on passing parameters between stacks and regions.

## GitHub Repo

https://github.com/schuettc/multi-region-pipeline-with-parameters

## Architecture Overview

![Architecture Diagram](/images/blog/multi-region-deployment-pipeline/MultiRegionPipelineOverview.png)

The solution consists of the following main components:

- **Central Pipeline Stack**: Orchestrates the deployment across multiple regions.
- **Data Stack**: Deployed in the primary region (us-east-1) to manage shared resources.
- **Regional Stacks**: Deployed in each specified region, containing region-specific resources.
- **SNS Topic**: Used for cross-region communication.
- **Lambda Functions**: Deployed in each region to demonstrate the multi-region setup.

## Key Features

- **Multi-Region Deployment**: Automatically deploys resources to multiple AWS regions.
- **Cross-Region Parameter Sharing**: Uses SSM Parameter Store to share information across regions.
- **Centralized SNS Topic**: Facilitates communication between regional resources.
- **Dynamic Region Configuration**: Easily add or remove regions from the deployment.
- **Automated CI/CD**: Uses AWS CodePipeline for continuous integration and deployment.

## Prerequisites

- AWS Account
- AWS CLI configured with appropriate credentials
- Node.js and npm installed

## Project Structure

```bash
.
├── src/
│ ├── config/
│ │ └── regions.ts # Region configuration
│ ├── dataStack/ # Central data stack
│ ├── regionStacks/ # Regional stack definition
│ └── pipeline.ts # Main pipeline definition
├── bootstrapRegions.ts # Script to bootstrap CDK in all regions
├── invokeRegionalLambdas.ts # Script to test regional Lambdas
└── .projenrc.ts # Projen configuration
```

## Deployment

Fork the repository

Clone your forked repository:

```bash
git clone https://github.com/your-username/multi-region-pipeline.git
cd multi-region-pipeline
```

Configure [CodeStar Connection](https://docs.aws.amazon.com/codepipeline/latest/userguide/connections-github.html) with your AWS Account

Update the configuration in `src/multi-region-pipeline.ts` with your information:

```typescript
const pipelineName = "MultiRegionPipeline";
const repoOwner = "schuettc";
const repoName = "multi-region-pipeline-with-parameters";
const repoBranch = "main";
const connectionArn =
  "arn:aws:codeconnections:sa-east-1:104621577074:connection/xxxxxxxxxxxxx";
const accountId = "104621577074";
```

Install dependencies:

```bash
yarn
```

Bootstrap CDK in all required regions:

```bash
yarn bootstrapRegions
```

7. Build the project:

```bash
yarn build
```

Deploy the pipeline:

```bash
yarn launch
```

## How it Works

The regions used in this example are defined in `src/config/regions.ts` file:

```typescript
export const multiRegionConfig: RegionConfig[] = [
  {
    region: "us-east-1",
    services: ["service-a", "service-b", "service-c"],
  },
  {
    region: "us-west-2",
    services: ["service-a", "service-b", "service-c", "service-d"],
  },
];
```

## Pipeline

The pipeline is defined in `src/pipeline.ts` file:

```typescript
const pipeline = new CodePipeline(this, "Pipeline", {
  pipelineName: props.pipelineConfig.pipelineName,
  crossAccountKeys: true,
  synth: new ShellStep("Synth", {
    input: CodePipelineSource.connection(
      `${props.pipelineConfig.repoOwner}/${props.pipelineConfig.repoName}`,
      props.pipelineConfig.repoBranch,
      {
        connectionArn: props.pipelineConfig.connectionArn,
      }
    ),
    env: {
      PIPELINE_NAME: props.pipelineConfig.pipelineName,
      REPO_OWNER: props.pipelineConfig.repoOwner,
      REPO_NAME: props.pipelineConfig.repoName,
      REPO_BRANCH: props.pipelineConfig.repoBranch,
      CONNECTION_ARN: props.pipelineConfig.connectionArn,
      ACCOUNT_ID: this.account,
    },
    commands: ["yarn install --frozen-lockfile", "yarn build", "npx cdk synth"],
  }),
});
```

This provides us the pipeline that we will use to deploy our application.

## Data Stack

The data stack is deployed in the primary region (us-east-1) and contains resources that are shared across all regions. In this example it contains an SNS Topic that is used to receive events from the regional lambdas. We then save the ARN of this topic in the SSM Parameter Store, so that we can reference it in the regional stacks. We will add it to the pipeline so that it is deployed to the `us-east-1` region.

```typescript
const dataStage = new DataStage(this, "DataStage", {
  env: { account: this.account, region: "us-east-1" },
});
pipeline.addStage(dataStage);
```

The stack is defined in `src/dataStack/dataStack.ts` file:

```typescript
export class DataStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const snsResources = new SNSResources(this, "SNSResources");
    new LambdaResources(this, "LambdaResources", snsResources.topic);

    new CfnOutput(this, "TopicArn", { value: snsResources.topic.topicArn });

    new StringParameter(this, "TopicArnParam", {
      parameterName: "/mulit-region-pipeline/topic-arn",
      stringValue: snsResources.topic.topicArn,
    });
  }
}
```

## Regional Stacks

The regional stacks are deployed in each region that we have defined in the `multiRegionConfig` array. They contain the resources that are specific to that region. In this example they contain a Lambda function that sends a message to the SNS topic that is shared across regions. First, we'll add a wave to our pipeline for the regional deployments.

```typescript
const regionalWave = pipeline.addWave("RegionalDeployments");

getAllRegions().forEach((region) => {
  const regionalStage = new RegionalStage(this, `RegionalStage-${region}`, {
    env: { account: this.account, region },
  });
  regionalWave.addStage(regionalStage);
});
```

The Stack that is deployed in each region is defined in `src/regionStacks/regionStack.ts` file:

```typescript
export class RegionStack extends Stack {
  constructor(scope: Construct, id: string, _props: StackProps) {
    super(scope, id, _props);

    const ssmReaders = new SSMReaders(this, "SSMReaders");

    const topic = Topic.fromTopicArn(
      this,
      "ImportedErrorTopic",
      ssmReaders.topicArn
    );
    const lambdaResources = new LambdaResources(this, "LambdaResources", topic);

    new CfnOutput(this, "LambdaFunctionName", {
      value: lambdaResources.lambdaFunction.functionName,
      exportName: `${this.stackName}-LambdaFunctionName`,
    });
  }
}
```

In order to read the parameter from the SSM Parameter Store, we will use a Custom Resource that will read the parameter and return the value to the regional stack. This is defined in `src/regionStacks/ssmReader.ts` file:

```typescript
const ssmPolicy = new PolicyStatement({
  actions: ["ssm:GetParameter"],
  resources: [
    `arn:aws:ssm:us-east-1:${
      Stack.of(this).account
    }:parameter/mulit-region-pipeline/*`,
  ],
});

const snsTopicArnReader = new AwsCustomResource(this, "SNSTopicArnReader", {
  onCreate: {
    service: "SSM",
    action: "getParameter",
    parameters: {
      Name: "/mulit-region-pipeline/topic-arn",
    },
    region: "us-east-1",
    physicalResourceId: PhysicalResourceId.of("TopicArnParameter"),
  },
  onUpdate: {
    service: "SSM",
    action: "getParameter",
    parameters: {
      Name: "/mulit-region-pipeline/topic-arn",
    },
    region: "us-east-1",
    physicalResourceId: PhysicalResourceId.of("TopicArnParameter"),
  },
  policy: AwsCustomResourcePolicy.fromStatements([ssmPolicy]),
});

this.topicArn = snsTopicArnReader.getResponseField("Parameter.Value");
```

This will allow us to read the parameter from the SSM Parameter Store and use it in the regional stack as an environment variable in the Lambda function.

## Testing

After deployment, you can test the regional Lambda functions using the `invokeRegionalLambdas.ts` script:

```bash
yarn invokeRegionalLambdas
```

This will invoke the Lambda functions in each region and print the response to the console.

This script will invoke the Lambda function in each deployed region and display the results.

## Cross-Region Communication

The project demonstrates cross-region communication using a central SNS topic. Regional Lambdas can publish messages to this topic, which can be consumed by a central Lambda function.

## Cleanup

To remove all deployed resources:

```bash
yarn cdk destroy
```


This demo will explain how to deploy a multi-region pipeline with AWS CDK by passing parameters between stacks.

Deploying a multi-region pipeline with AWS CDK


# Next.js App Runner Deployment with CloudFront

This project demonstrates how to deploy a Next.js application using AWS App Runner and CloudFront. It uses the AWS CDK to define and deploy the infrastructure. The project is managed with projen and includes several custom tasks to support the development process.

Please see update: [https://subaud.io/blog/updated-nextjs-app-runner-deployment-with-ssr](https://subaud.io/blog/updated-nextjs-app-runner-deployment-with-ssr)

## Architecture

The architecture consists of the following components:

1. A Next.js application
2. AWS App Runner to host and run the Next.js application
3. Amazon CloudFront as a content delivery network (CDN) in front of App Runner
4. AWS CDK to define and deploy the infrastructure

## Key Features

- Serverless deployment of a Next.js application using AWS App Runner
- CloudFront distribution with custom origin request policy
- Automated CI/CD pipeline using GitHub Actions
- Infrastructure as Code using AWS CDK

## Prerequisites

- AWS Account
- Node.js (version 18 or later)
- AWS CDK CLI
- Docker (for local development and testing)

## Getting Started

Clone the repository:

```bash
git clone https://github.com/schuettc/next-js-app-runner-deployment.git
cd next-js-app-runner-deployment
```

Install dependencies:

```bash
yarn
```

Deploy the stack:

```bash
yarn launch
```

## Project Structure

- `src/`: Contains the CDK infrastructure code
  - `appRunner.ts`: Defines the App Runner service
  - `cloudfront.ts`: Sets up the CloudFront distribution
  - `next-js-app-runner-deployment.ts`: Main stack definition
- `src/resources/app/`: Contains the Next.js application code

## Custom Origin Request Policy

This project implements a custom origin request policy for CloudFront, which is required when using App Runner as the origin. The policy allows specific headers to be forwarded to the origin:

```typescript:src/cloudfront.ts
    const customOriginRequestPolicy = new OriginRequestPolicy(
      this,
      'UserAgentRefererHeadersPolicy',
      {
        headerBehavior: OriginRequestHeaderBehavior.allowList(
          'User-Agent',
          'Referer',
        ),
      },
    );
```

This custom policy ensures that the `User-Agent` and `Referer` headers are forwarded to the App Runner service, which solves the 404 error that occurs when using AppRunner as the origin for a CloudFront distribution.

https://docs.aws.amazon.com/apprunner/latest/dg/request-route-404-troubleshoot.html

## Deployment

The project includes a GitHub Actions workflow for automated deployments. When you push changes to the `main` branch, it will automatically deploy the updated stack to AWS.

To deploy manually:

1. Configure your AWS credentials
2. Run `yarn deploy`

### projen configuration

```typescript
const deploy = project.github?.addWorkflow('deploy');
deploy?.on({
  push: {
    branches: ['main'],
  },
  workflow_dispatch: {},
});

deploy?.addJobs({
  deploy: {
    runsOn: ['ubuntu-latest'],
    permissions: {
      contents: JobPermission.READ,
      id_token: JobPermission.WRITE,
    },
    steps: [
      { uses: 'actions/checkout@v4' },
      {
        uses: 'actions/setup-node@v4',
        with: {
          'node-version': '18',
        },
      },
      { run: 'yarn install --frozen-lockfile' },
      {
        uses: 'aws-actions/configure-aws-credentials@v4',
        with: {
          'aws-access-key-id': '${{ secrets.AWS_ACCESS_KEY_ID }}',
          'aws-secret-access-key': '${{ secrets.AWS_SECRET_ACCESS_KEY }}',
          'aws-region': '${{ secrets.AWS_REGION }}',
        },
      },
      { run: 'npx cdk deploy --require-approval never' },
    ],
  },
});
```

## Local Development

To run the Next.js application locally:

Navigate to the app directory:

```bash
cd src/resources/app
```

Install dependencies:

```bash
yarn
```

Start the development server:

```bash
yarn dev
```

Open [http://localhost:3000](http://localhost:3000) in your browser

## GitHub Updates with Projen

The project uses projen to manage the project and the Next.js application. We have replaced the default projen upgrade with a custom upgrade. This allows us to upgrade the dependencies for both the main project and the Next.js application in a single step.

```typescript
const upgradeWorkflow = project.github?.addWorkflow('upgrade');
upgradeWorkflow?.on({
  schedule: [{ cron: '0 0 * * 1' }],
  workflow_dispatch: {},
});

upgradeWorkflow?.addJobs({
  upgrade: {
    runsOn: ['ubuntu-latest'],
    permissions: {
      contents: JobPermission.WRITE,
      pullRequests: JobPermission.WRITE,
    },
    steps: [
      { uses: 'actions/checkout@v4' },
      { uses: 'actions/setup-node@v4', with: { 'node-version': '18' } },
      { run: 'yarn install' },
      { run: 'npx projen upgrade:projen' },
      { run: 'npx projen upgrade:nextjs' },
      {
        name: 'Create Pull Request',
        uses: 'peter-evans/create-pull-request@v6',
        with: {
          'token': '${{ secrets.PROJEN_GITHUB_TOKEN }}',
          'commit-message': 'chore: upgrade dependencies',
          'branch': 'projen-upgrade',
          'title': 'chore: upgrade dependencies',
          'body':
            'This PR upgrades project dependencies, including projen and Next.js. Auto-generated by projen upgrade workflow.',
        },
      },
    ],
  },
});

project.addTask('upgrade:nextjs', {
  exec: ['cd src/resources/app', 'yarn upgrade --latest', 'cd ../../..'].join(
    ' && ',
  ),
});

project.addTask('upgrade:projen', {
  description: 'Upgrade projen dependencies',
  steps: [
    { exec: 'yarn upgrade projen @projen/awscdk-app-ts' },
    { exec: 'npx projen' },
  ],
});
```


NextJS App deployed to App Runner with GitHub Actions


# Deploying an NVIDIA NIM to EC2

This demo will show you how to deploy an [NVIDIA NIM](https://www.nvidia.com/en-us/ai/) to an Amazon EC2 instance and enable it for inference.

## GitHub Repository

https://github.com/schuettc/ec2-nim-deployment

## Requirements

- NGC API key
- Route 53 Hosted Zone/Domain
- On-Demand vCPU quota

### NGC API Key

In order to use this demo, you will need an NGC API key. This key can be generated here: https://org.ngc.nvidia.com/setup/personal-keys

We will upload this key to [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) and will be used within the EC2 instance to log in to NGC to download the container.

### Route 53 Hosted Zone/Domain

To enable [TLS security on the Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html), we need to generate a certificate using [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/). In order to generate this certificate, we need a domain and Hosted Zone in [Amazon Route 53](https://aws.amazon.com/route53/). We will configure this before launching the stack.

### On-Demand vCPU quota

This demo will launch a G5 EC2 instance. The number of vCPUs used by this instance will depend on the size. For example, a G5.12xlarge instance will use 48 vCPUs. Be sure to [check your utilization and quota value](https://us-west-2.console.aws.amazon.com/servicequotas/home/services/ec2/quotas/L-DB2E81BA) to make sure you can launch this instance. You can request an increase if necessary.

## Setup

### .env Configuration

Before launching, be sure to configure a `.env` file. Many of these fields can be left with their defaults.

```bash
# Stack Configuration
STACK_NAME=nim-stack
AWS_REGION=us-west-2
TEMPLATE_FILE=nims-stack.yaml

# Instance Configuration
INSTANCE_TYPE=g5.12xlarge

# AWS Resources
KEY_NAME=REPLACE_WITH_YOUR_KEY
VPC_ID=
SUBNET_IDS=

# Domain Configuration
DOMAIN_NAME=HOSTNAME.EXAMPLE.COM
HOSTED_ZONE_ID=REPLACE_WITH_YOUR_HOSTED_ZONE_ID

# NVIDIA Configuration
NGC_API_KEY_SECRET_NAME=NGCApiKeySecret
NGC_API_KEY=REPLACE_WITH_YOUR_NGC_API_KEY
REPOSITORY=nim/meta/llama3-8b-instruct
LATEST_TAG=1.0.0
```

You will need to update the following before launching:

```bash
# AWS Resources
KEY_NAME=REPLACE_WITH_YOUR_KEY

# Domain Configuration
DOMAIN_NAME=HOSTNAME.EXAMPLE.COM
HOSTED_ZONE_ID=REPLACE_WITH_YOUR_HOSTED_ZONE_ID

# NVIDIA Configuration
NGC_API_KEY=REPLACE_WITH_YOUR_NGC_API_KEY
```

### Secret Update

To upload the NGC API key to Secrets Manager, run the script:

```bash
./update_ngc_api_key.sh
```

This API Key will be retrieved by the EC2 instance during deployment.

### Launch Cloudformation Stack

You can either create the stack in the AWS Console, or through the included script. The script will populate the parameters required that were configured in the `.env` file. You will need to configure the required parameters in the console.

## Inference

Once the deployment succeeds, you can run inference requests against it using a variety of tools. Included is a script that uses `curl` to make a request to the endpoint using the configured domain name.

```bash
curl -s -X POST "https://$DOMAIN_NAME/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "'"$model_id"'",
        "messages": [
            {"role": "user", "content": "Hello! How are you?"},
            {"role": "assistant", "content": "Hi! I am quite well, how can I help you today?"},
            {"role": "user", "content": "Write a short limerick about the wonders of GPU computing."}
        ],
        "max_tokens": 100
    }' | jq '.choices[0].message.content'
```

To test the endpoint and inference:

```bash
./inference.sh
```

# How it Works

## Components

![Overview](/images/blog/deploying-an-nvidia-nim-to-ec2/Overview.png)

### EC2 Instance

- Uses a G5 instance type (default: g5.12xlarge) for GPU capabilities
- Runs Ubuntu with NVIDIA drivers, Docker, and NVIDIA Container Toolkit
- Automatically pulls and runs the specified NIM container

### Auto Scaling Group (ASG)

- Manages a single EC2 instance
- Ensures high availability and easy updates
- Uses a Launch Template for instance configuration

### Application Load Balancer (ALB)

- Provides HTTPS termination
- Routes traffic to the EC2 instance
- Uses an ACM certificate for HTTPS

### VPC and Networking

- Creates a new VPC with public subnets (or uses existing ones)
- Sets up necessary security groups and routing

### Route 53

- Creates a DNS record pointing to the ALB

## EC2 Configuration

1. The stack creates or uses existing VPC and subnets
2. An EC2 instance is launched with the necessary software and configurations
3. The NIM container is pulled and started on the EC2 instance
4. An ALB is set up to route traffic to the EC2 instance
5. A Route 53 record is created to point the specified domain to the ALB

The stack uses conditions to determine whether to create new VPC resources or use existing ones, making it flexible for different deployment scenarios.

## UserData Script Details

The UserData script in the Launch Template performs several key setup tasks:

1. **System Updates and Dependencies**

   - Updates the package list
   - Installs Python3-pip, AWS CLI, and CloudFormation helper scripts

2. **NVIDIA Driver and CUDA Installation**

   - Installs Linux headers
   - Adds NVIDIA CUDA repository
   - Installs CUDA drivers

3. **Docker Installation**

   - Adds Docker repository
   - Installs Docker CE

4. **NVIDIA Container Toolkit Installation**

   - Adds NVIDIA Docker repository
   - Installs nvidia-docker2
   - Restarts Docker service

5. **NGC API Key Retrieval**

   - Fetches the NGC API Key from AWS Secrets Manager

6. **NIM Container Setup and Run**

   - Sets up environment variables for the container
   - Creates a local cache directory
   - Logs in to the NGC container registry
   - Runs the NIM container with the following configuration:
     - Uses NVIDIA runtime
     - Exposes port 8000
     - Mounts a local cache volume
     - Sets the NGC API Key as an environment variable

7. **Signaling**
   - Signals the success or failure of the setup to CloudFormation

## Docker Container and ALB Interaction

1. **Container Exposure**

   - The NIM container exposes port 8000 inside the EC2 instance

2. **Security Group Configuration**

   - The EC2 security group allows inbound traffic on port 8000 from the ALB security group

3. **ALB Target Group**

   - An ALB Target Group is created with the following properties:
     - Protocol: HTTP
     - Port: 8000
     - Health check path: /health

4. **ALB Listener**

   - The ALB listener is configured for HTTPS (port 443)
   - It forwards traffic to the Target Group

5. **Request Flow**

   - Incoming HTTPS requests hit the ALB
   - The ALB terminates SSL and forwards the request as HTTP to the EC2 instance on port 8000
   - The Docker container receives the request on port 8000 and processes it

6. **Health Checks**
   - The ALB periodically sends health check requests to the `/health` endpoint
   - The NIM container responds to these health checks, allowing the ALB to determine if the instance is healthy

## Instance Types and Costs

This CloudFormation template supports the following instance types for NVIDIA AI Microservices (NIM):

### G5 Instances

- g5.xlarge: 1 GPU, 4 vCPUs, 16 GB RAM
- g5.2xlarge: 1 GPU, 8 vCPUs, 32 GB RAM
- g5.4xlarge: 1 GPU, 16 vCPUs, 64 GB RAM
- g5.8xlarge: 1 GPU, 32 vCPUs, 128 GB RAM
- g5.16xlarge: 1 GPU, 64 vCPUs, 256 GB RAM
- g5.12xlarge: 4 GPUs, 48 vCPUs, 192 GB RAM
- g5.24xlarge: 4 GPUs, 96 vCPUs, 384 GB RAM
- g5.48xlarge: 8 GPUs, 192 vCPUs, 768 GB RAM

⚠️ **Warning**: These instances can be very expensive to run. Please check the current AWS pricing for your region before deploying. For example, as of 2023, the costs for G5 instances in the US East (N. Virginia) region range from $1.006/hour for g5.xlarge to $16.288/hour for g5.48xlarge.

Always remember to stop or terminate your instances when not in use to avoid unnecessary charges.


In this demo we will see how to deploy an NVIDIA NIM to EC2 with Docker to do your own inference

Deploying an NVIDIA NIM to EC2


# GitHub Repository Metrics: A Serverless AWS Implementation

## Overview

This project implements a serverless solution for automatically collecting and analyzing GitHub repository metrics. It uses AWS services including Lambda, S3, Athena, Glue, and EventBridge to gather data from GitHub, store it efficiently, and make it available for analysis.

![Overview](/images/blog/github-insights/GitHubInsights.png)

## GitHub

All code is available here: https://github.com/schuettc/github-insights

## System Architecture

The solution consists of five main components:

1. Lambda Function: Fetches data from GitHub API
2. S3 Bucket: Stores data in Parquet format
3. Glue Database and Table: Defines schema for the data
4. Athena Workgroup and Query: Enables SQL-based analysis
5. EventBridge Rule: Schedules daily data collection

## Implementation Details

### 1. Lambda Function (LambdaResources)

The Lambda function is responsible for querying the GitHub API and writing the results to S3. It's implemented using Node.js and the AWS SDK.

```typescript
export class LambdaResources extends Construct {
  public insightQueryLambda: Function;
  constructor(scope: Construct, id: string, props: LambdaResourcesProps) {
    super(scope, id);

    // ... role creation code omitted for brevity

    this.insightQueryLambda = new NodejsFunction(this, "InsightsQueryLambda", {
      entry: "./src/resources/insightQuery/index.ts",
      runtime: Runtime.NODEJS_LATEST,
      architecture: Architecture.ARM_64,
      memorySize: 1024,
      handler: "handler",
      timeout: Duration.minutes(5),
      role: insightQueryLambdaRole,
      environment: {
        LOG_LEVEL: props.logLevel,
        GITHUB_TOKEN_SECRET_NAME: props.githubTokenSecretName,
        INSIGHTS_BUCKET: props.insightsBucket.bucketName,
      },
    });
    props.insightsBucket.grantReadWrite(this.insightQueryLambda);
  }
}
```

#### Lambda Function Implementation

The core of the data collection is performed by the Lambda function. Here's a simplified version of the main handler:

```typescript
export const handler = async (): Promise<void> => {
  try {
    const githubToken = await getGitHubToken();
    const octokit = new Octokit({ auth: githubToken });

    const repositories = await getRepositoriesFromS3();
    const insights = await fetchRepositoryInsights(octokit, repositories);

    await uploadInsightsToS3(insights);
  } catch (error) {
    console.error("Error in Lambda execution:", error);
    throw error;
  }
};
```

The `fetchRepositoryInsights` function is where the actual GitHub API calls occur:

```typescript
async function fetchRepositoryInsights(
  octokit: Octokit,
  repositories: Repository[]
): Promise<RepoInsights[]> {
  return (
    await Promise.all(
      repositories.map(async ({ owner, repo }) => {
        try {
          const repoData = await octokit.rest.repos.get({ owner, repo });
          const [
            issuesData,
            pullRequestsData,
            contributorsData,
            releasesData,
            commitsData,
          ] = await Promise.all([
            octokit.rest.issues.listForRepo({ owner, repo, state: "all" }),
            octokit.rest.pulls.list({ owner, repo, state: "all" }),
            octokit.rest.repos.listContributors({ owner, repo }),
            octokit.rest.repos
              .getLatestRelease({ owner, repo })
              .catch(() => null),
            octokit.rest.repos.getCommitActivityStats({ owner, repo }),
          ]);

          // Process and aggregate data
          // ...

          return {
            repoName: `${owner}/${repo}`,
            description: repoData.data.description || "",
            stars: repoData.data.stargazers_count,
            forks: repoData.data.forks_count,
            openIssues: repoData.data.open_issues_count,
            // ... other fields
          };
        } catch (error) {
          console.error(`Error fetching data for ${owner}/${repo}:`, error);
          return null;
        }
      })
    )
  ).filter((insight): insight is RepoInsights => insight !== null);
}
```

The collected data is then written to S3 in Parquet format:

```typescript
async function uploadInsightsToS3(insights: RepoInsights[]): Promise<void> {
  const schema = new parquet.ParquetSchema({
    repoName: { type: "UTF8" },
    description: { type: "UTF8" },
    stars: { type: "INT64" },
    // ... other fields
  });

  const writer = await ParquetWriter.openFile(schema, "/tmp/insights.parquet");

  for (const insight of insights) {
    await writer.appendRow({
      ...insight,
      date: `${year}-${month}-${day}`,
    });
  }

  await writer.close();

  const fileContent = await fs.promises.readFile("/tmp/insights.parquet");

  await s3Client.send(
    new PutObjectCommand({
      Bucket: process.env.INSIGHTS_BUCKET,
      Key: `github-insights/year=${year}/month=${month}/day=${day}/insights_${timestamp}.parquet`,
      Body: fileContent,
      ContentType: "application/octet-stream",
    })
  );
}
```

### 2. S3 Bucket (S3Resources)

The S3 bucket is used to store the GitHub repository to query and then to store the GitHub data in Parquet format. The data is partitioned by year, month, and day for efficient querying.

### 3. Glue Database and Table (GlueResources)

AWS Glue is used to define the schema for the GitHub data and set up partition projection.

```typescript
export class GlueResources extends Construct {
  public insightsDatabase: glue.CfnDatabase;
  public insightsTable: CfnTable;
  constructor(scope: Construct, id: string, props: GlueResourcesProps) {
    super(scope, id);

    // ... database creation code omitted for brevity

    this.insightsTable = new glue.CfnTable(this, "GitHubInsightsTable", {
      catalogId: Stack.of(this).account,
      databaseName: this.insightsDatabase.ref,
      tableInput: {
        name: "github_insights",
        tableType: "EXTERNAL_TABLE",
        parameters: {
          EXTERNAL: "TRUE",
          "parquet.compression": "SNAPPY",
          "projection.enabled": "true",
          "projection.year.type": "integer",
          "projection.year.range": "2020,2030",
          "projection.year.digits": "4",
          "projection.month.type": "integer",
          "projection.month.range": "1,12",
          "projection.month.digits": "2",
          "projection.day.type": "integer",
          "projection.day.range": "1,31",
          "projection.day.digits": "2",
          "storage.location.template": `s3://${props.insightsBucket.bucketName}/github-insights/year=$\{year}/month=$\{month}/day=$\{day}`,
        },
        storageDescriptor: {
          // ... storage descriptor details omitted for brevity
        },
        partitionKeys: partitionKeys,
      },
    });
  }
}
```

### 4. Athena Workgroup and Query (AthenaResources)

Amazon Athena is configured with a dedicated workgroup and a sample query for analyzing the GitHub data.

```typescript
export class AthenaResources extends Construct {
  constructor(scope: Construct, id: string, props: AthenaResourceProps) {
    super(scope, id);

    // ... workgroup creation code omitted for brevity

    new athena.CfnNamedQuery(this, "GitHubInsightsSampleQuery", {
      database: props.insightsDatabase.ref,
      queryString: `
        SELECT
          reponame,
          AVG(stars) as avg_stars,
          AVG(forks) as avg_forks,
          AVG(openissues) as avg_open_issues,
          AVG(commitslastmonth) as avg_commits_last_month
        FROM
          github_insights
        WHERE
          year = '${new Date().getUTCFullYear()}'
          AND month = '${(new Date().getUTCMonth() + 1)
            .toString()
            .padStart(2, "0")}'
        GROUP BY
          reponame
        ORDER BY
          avg_stars DESC
      `,
      description:
        "Average stars, forks, open issues, and commits for each repository this month",
      name: "GitHubInsightsMonthlyAverage",
      workGroup: workgroup.ref,
    });
  }
}
```

### 5. EventBridge Rule (EventBridgeResources)

An EventBridge rule is set up to trigger the Lambda function daily.

```typescript
export class EventBridgeResources extends Construct {
  constructor(scope: Construct, id: string, props: EventBridgeResourcesProps) {
    super(scope, id);

    new Rule(this, "DailyLambdaTrigger", {
      schedule: Schedule.cron({ minute: "0", hour: "0" }),
      targets: [new LambdaFunction(props.insightQueryLambda)],
    });
  }
}
```

## Data Flow

1. The EventBridge rule triggers the Lambda function daily.
2. The Lambda function fetches data from GitHub and writes it to S3 in Parquet format.
3. The data is automatically partitioned in S3 based on the Glue table definition.
4. Users can query the data using Athena, with the sample query providing a starting point for analysis.

## Deployment

The entire infrastructure is defined using AWS CDK, allowing for version-controlled, reproducible deployments. The main stack is defined in the `GitHubInsights` class:

```typescript
export class GitHubInsights extends Stack {
  constructor(scope: Construct, id: string, props: GitHubInsightsProps) {
    super(scope, id, props);

    const s3Resources = new S3Resources(this, "S3Resources");
    const glueResources = new GlueResources(this, "GlueResources", {
      insightsBucket: s3Resources.insightsBucket,
    });
    new AthenaResources(this, "AthenaResources", {
      insightsBucket: s3Resources.insightsBucket,
      insightsDatabase: glueResources.insightsDatabase,
    });
    const lambdaResources = new LambdaResources(this, "LambdaResources", {
      logLevel: props.logLevel,
      insightsBucket: s3Resources.insightsBucket,
      githubTokenSecretName: props.githubTokenSecretName,
    });
    new EventBridgeResources(this, "EventBridgeResources", {
      insightQueryLambda: lambdaResources.insightQueryLambda,
    });
  }
}
```

## Querying Data with Athena

Once the data is stored in S3 and the Glue table is set up, users can query the data using Amazon Athena. Here are some example queries:

1. Monthly average metrics for each repository:

```sql
SELECT
  reponame,
  AVG(stars) as avg_stars,
  AVG(forks) as avg_forks,
  AVG(openissues) as avg_open_issues,
  AVG(commitslastmonth) as avg_commits_last_month
FROM
  github_insights
WHERE
  year = '2024'
  AND month = '08'
GROUP BY
  reponame
ORDER BY
  avg_stars DESC
```

2. Repositories with the most growth in stars over the last 30 days:

```sql
WITH current_stats AS (
  SELECT reponame, stars
  FROM github_insights
  WHERE year = '2024' AND month = '08' AND day = '15'
),
previous_stats AS (
  SELECT reponame, stars
  FROM github_insights
  WHERE year = '2024' AND month = '07' AND day = '16'
)
SELECT
  c.reponame,
  c.stars - p.stars AS star_growth,
  ((c.stars - p.stars) / p.stars) * 100 AS growth_percentage
FROM current_stats c
JOIN previous_stats p ON c.reponame = p.reponame
ORDER BY star_growth DESC
LIMIT 10
```

3. Repositories with the highest ratio of closed to open issues:

```sql
SELECT
  reponame,
  AVG(closedissues) as avg_closed_issues,
  AVG(openissues) as avg_open_issues,
  AVG(closedissues) / NULLIF(AVG(openissues), 0) as closed_to_open_ratio
FROM
  github_insights
WHERE
  year = '2024'
  AND month = '08'
GROUP BY
  reponame
HAVING
  AVG(openissues) > 0
ORDER BY
  closed_to_open_ratio DESC
LIMIT 10
```

To run these queries:

1. Open the Athena console in AWS.
2. Select the `github_insights_db` database.
3. Enter the SQL query in the query editor.
4. Click "Run query" to execute and view the results.

The partition projection set up in the Glue table definition allows Athena to efficiently query the data without the need for manual partition management, making it easy to analyze historical trends and current repository states.

![ExampleQuery](/images/blog/github-insights/ExampleQuery.png)

# Deploying

## Configuration Files

### 1. `.env`

This file contains environment variables used by the scripts and the CDK deployment.

**Example content:**

```bash
GITHUB_TOKEN_SECRET=gph_xxxxxxxxxxx
STACK_NAME=GitHubInsights
AWS_REGION=us-east-1
```

**Fields:**

- `GITHUB_TOKEN_SECRET`: Your GitHub personal access token. You will need to create one within GitHub.
- `STACK_NAME`: The name of your CDK stack (used by `upload_repo_list.sh`).
- `AWS_REGION`: The AWS region where your resources are deployed.

**Note:** Never commit the `.env` file to version control. It's included in `.gitignore` by default.

### 2. `repositories.json`

This file contains the list of GitHub repositories you want to monitor.

**Example content:**

```json
[
  {
    "owner": "schuettc",
    "repo": "github-insights"
  }
]
```

**Usage:**

- Add or remove repository objects as needed.
- Each repository object should have an "owner" and "repo" field.
- After modifying this file, run `upload_repo_list.sh` to update the list in your S3 bucket.

## Helper Scripts

### 1. `setup_github_token.sh`

This script is used to securely store your GitHub token in AWS Secrets Manager.

**Purpose:**

- Creates or updates a secret in AWS Secrets Manager to store your GitHub token.
- Allows secure access to the GitHub API without exposing the token in your code.

**Usage:**

1. Ensure you have the AWS CLI configured with appropriate permissions.
2. Create a `.env` file in the project root (see Configuration Files section below).
3. Run the script:

```bash
./setup_github_token.sh
```

**What it does:**

- Reads the `GITHUB_TOKEN_SECRET` from the `.env` file.
- Creates or updates a secret in AWS Secrets Manager with the provided token.
- Uses the `GITHUB_TOKEN_SECRET_NAME` (default: "/github_insights/github_token") as the secret name.
- Sets the AWS region based on the `AWS_REGION` in `.env` or defaults to "us-east-1".

### 2. `upload_repo_list.sh`

This script uploads the list of repositories to be monitored to your S3 bucket.

**Purpose:**

- Uploads the `repositories.json` file to the S3 bucket created by the CDK stack.
- Allows you to easily update the list of repositories to be monitored.

**Usage:**

1. Ensure you have the AWS CLI configured with appropriate permissions.
2. Update the `repositories.json` file with the repositories you want to monitor.
3. Run the script:

```bash
./upload_repo_list.sh
```

**What it does:**

- Retrieves the S3 bucket name from the CloudFormation stack outputs.
- Uploads the `repositories.json` file to the root of the S3 bucket.
- Uses the `STACK_NAME` from the `.env` file to identify the correct CloudFormation stack.
- Sets the AWS region based on the `AWS_REGION` in `.env` or defaults to "us-east-1".

## Workflow

1. Set up your `.env` file with your GitHub token and stack name.
2. Deploy your CDK stack by running `yarn launch`
3. Run `setup_github_token.sh` to securely store your GitHub token.
4. Update `repositories.json` with the list of repositories you want to monitor.
5. Run `upload_repo_list.sh` to upload the repository list to your S3 bucket.

By following this workflow, you ensure that your GitHub token is securely stored and your Lambda function has the latest list of repositories to monitor.

## Conclusion

This serverless architecture provides an efficient method for collecting and analyzing GitHub repository metrics. It leverages AWS services to automate data collection, storage, and query capabilities, enabling data-driven insights into repository activity and health.


In this demo we will see how to deploy an application on AWS that will capture insights from GitHub and store them in S3 to be easily queried with Amazon Athena