
subaud is authored by Court Schuett (he/him). While Court is an 8090 Inc employee, opinions and statements on this blog are his own.


Court Schuett


I use [projen](https://github.com/projen/projen) for almost all of my projects because it allows me to not worry about all of the very necessary but tedious parts of a project - `.gitignore`, `.eslintrc.json`, GitHub Actions workflows, etc. However, I often find myself having both a front end and back end in the same repository. While projen and [GitHub Actions](https://github.com/features/actions) will automatically take care of upgrading dependencies in my main folder, I also want it to take care of dependencies in other parts of my project.

## Schedule

For the main folder dependencies, once a day upgrades is very aggressive. We can adjust that in our `.projenrc.js` file to weekly:

```javascript
const { UpgradeDependenciesSchedule } = require('projen/lib/javascript');

const project = new awscdk.AwsCdkTypeScriptApp({
  ...projectConfig,
  depsUpgradeOptions: {
    ignoreProjen: false,
    workflowOptions: {
      labels: ['auto-approve', 'auto-merge'],
      schedule: UpgradeDependenciesSchedule.WEEKLY,
    },
  },
});
```

This pushes our `upgrade` workflow out to [once a week](https://crontab.guru/#0_0_*_*_1):

```yaml
name: upgrade
on:
  workflow_dispatch: {}
  schedule:
    - cron: 0 0 * * 1
```

## Additional Actions

To add an action, we will once again edit our `.projenrc.js` file. This time we will add a workflow.

```javascript
const { JobPermission } = require('projen/lib/github/workflows-model');

const upgradeSite = project.github.addWorkflow('upgrade-site');
upgradeSite.on({ schedule: [{ cron: '0 0 * * 1' }], workflowDispatch: {} });
```

This will create a new workflow on the same schedule as the existing `upgrade` workflow.

Next, we will add a job that will:

- Install dependencies via `yarn`
- Upgrade dependencies
- Create a Pull Request

```javascript
upgradeSite.addJobs({
  upgradeSite: {
    runsOn: ['ubuntu-latest'],
    name: 'upgrade-site',
    permissions: {
      actions: JobPermission.WRITE,
      contents: JobPermission.READ,
      idToken: JobPermission.WRITE,
    },
    steps: [
      { uses: 'actions/checkout@v3' },
      {
        name: 'Setup Node.js',
        uses: 'actions/setup-node@v3',
        with: {
          'node-version': '16',
        },
      },
      {
        run: 'yarn install --check-files --frozen-lockfile',
        workingDirectory: 'site',
      },
      { run: 'yarn upgrade', workingDirectory: 'site' },
      {
        name: 'Create Pull Request',
        uses: 'peter-evans/create-pull-request@v4',
        with: {
          'token': '${{ secrets.' + AUTOMATION_TOKEN + ' }}',
          'commit-message': 'chore: upgrade site',
          'branch': 'auto/projen-upgrade',
          'title': 'chore: upgrade site',
          'body': 'This PR upgrades site',
          'labels': 'auto-merge, auto-approve',
          'author': 'github-actions <github-actions@github.com>',
          'committer': 'github-actions <github-actions@github.com>',
          'signoff': true,
        },
      },
    ],
  },
});
```

Adjust the `workingDirectory` as needed for any directory that has dependencies that can be upgraded.

## Result

Now, we will have a weekly GitHub Actions workflow that will upgrade dependencies for both our main folder, as well as a subfolder of our choosing. This can be used to help keep [Dependabot](https://github.com/dependabot/dependabot-core) alerts to a minimum by keeping up with security updates.

```yaml
name: upgrade-site
on:
  schedule:
    - cron: 0 5 * * 1
  workflow_dispatch: {}
jobs:
  upgradeSite:
    name: upgrade-site
    runs-on: ubuntu-latest
    permissions:
      actions: write
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v3
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16'
      - run: yarn install --check-files --frozen-lockfile
        working-directory: site
      - run: yarn upgrade
        working-directory: site
      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v4
        with:
          token: ${{ secrets.PROJEN_GITHUB_TOKEN }}
          commit-message: 'chore: upgrade site'
          branch: auto/projen-upgrade
          title: 'chore: upgrade site'
          body: This PR upgrades site
          labels: auto-merge, auto-approve
          author: github-actions <github-actions@github.com>
          committer: github-actions <github-actions@github.com>
          signoff: true
```


This article will show you how to add custom GitHub Actions workflows with Projen

Custom GitHub Action workflows with projen


Because GitHub will only provide stats for the previous 14 days, if we want to maintain long-term stats, we need to pull stats and store them. [PyGitHub](https://pypi.org/project/PyGithub/) is a useful Python library for doing that using the [GitHub API v3](https://docs.github.com/en/rest). This Python script will pull stats from queried repositories and store them in a CSV.

```python
import os
import csv
from github import Github

GITHUB_KEY = os.environ.get("POETRY_GITHUB_KEY")
g = Github(GITHUB_KEY)

with open("repos.csv", "w", newline="", encoding="utf-8") as csvfile:
    repo_writer = csv.writer(csvfile, delimiter=",")
    repo_writer.writerow([
      "Repo Name",
      "Repo Link",
      "Repo Pushed At",
      "Forks", "StarGazers",
      "Open Issues",
      "Security Alerts",
      "Open Pull Requests"])
    for repo in g.search_repositories(query="user:schuettc cdk"):
        repo_writer.writerow(
            [
                repo.name,
                repo.html_url,
                repo.pushed_at,
                repo.forks_count,
                repo.stargazers_count,
                repo.open_issues_count,
                repo.get_vulnerability_alert(),
                repo.get_pulls().totalCount,
            ]
        )
```

This script will query [GitHub](https://github.com) for [`user:schuettc cdk`](https://github.com/search?q=user%3Aschuettc+cdk&type=repositories). This query will return all GitHub repositories created by user [`schuettc`](https://github.com/schuettc) (me) with `cdk` in the repository name. The result will be used to create a CSV file that contains these repositories and various GitHub stats.

Be sure to create a [GitHub personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) with an appropriate scope and save as `POETRY_GITHUB_KEY` in a local `.env` file to use.

The result will be something like this:

![CSV](/images/blog/PyGithubRepos.png)

- GitHub repository: [pygithub-query-csv](https://github.com/schuettc/pygithub-query-csv)


If you need to modify the keys of a Python dictionary, a recursive function can be a useful way to work through the entire dictionary.  This [gist](https://gist.github.com/schuettc/3875da7458e3573646599e70b7b17a61) is an example of how to do that.

Using PyGitHub to Manage and Track GitHub Repositories


# Next.js SSR App Runner Deployment with CloudFront

This project demonstrates how to deploy a Next.js SSR application using AWS App Runner and CloudFront. It uses an AWS CDK to define and deploy the infrastructure. The project is managed with projen and includes several custom tasks to support the development process.

Update to: https://subaud.io/blog/next-js-app-runner-with-deployment

GitHub: https://github.com/schuettc/next-js-app-runner-deployment

## Architecture

The architecture consists of the following components:

- Next.js application
- AWS App Runner to host and run the Next.js application
- Amazon CloudFront as a content delivery network (CDN) in front of App Runner
- Lambda@Edge to rewrite headers and support dynamic routing
- Route53 ARecords for DNS
- AWS CDK to define and deploy the infrastructure
- GitHub actions for CI/CD

![Overview](/images/blog/updated-nextjs-app-runner-deployment-with-srr/NextJSAppCloudfront.png)

## Prerequisites

- AWS Account
- Node.js (version 18 or later)
- AWS CDK CLI
- Docker (for local development and testing)

## Getting Started

Clone the repository:

```bash
git clone https://github.com/schuettc/next-js-app-runner-deployment.git
cd next-js-app-runner-deployment
```

Install dependencies:

```bash
yarn
```

Deploy the stack:

```bash
yarn launch
```

## Project Structure

- `src/`: Contains the CDK infrastructure code
  - `appRunner.ts`: Defines the App Runner service
  - `cloudfront.ts`: Sets up the CloudFront distribution
  - `lambda.ts`: Creates the Lambda@Edge function
  - `route53.ts`: Defines the DNS records
  - `next-js-app-runner-deployment.ts`: Main stack definition
- `src/resources/app/`: Contains the Next.js application code

## Custom Origin Request Policy

This project implements a custom origin request policy for CloudFront, which is required when using App Runner as the origin. The policy allows specific headers to be forwarded to the origin:

```typescript:src/cloudfront.ts
    const customOriginRequestPolicy = new OriginRequestPolicy(
      this,
      'UserAgentRefererHeadersPolicy',
      {
        headerBehavior: OriginRequestHeaderBehavior.allowList(
          'User-Agent',
          'Referer',
        ),
      },
    );
```

This custom policy ensures that the `User-Agent` and `Referer` headers are forwarded to the App Runner service, which solves the 404 error that occurs when using AppRunner as the origin for a CloudFront distribution.

https://docs.aws.amazon.com/apprunner/latest/dg/request-route-404-troubleshoot.html

## Lambda@Edge for Header Rewriting

This project uses a Lambda@Edge function to rewrite headers, which is crucial for supporting dynamic routing in Next.js when using App Runner behind CloudFront. The function is associated with both the Origin Request and Viewer Request events in CloudFront.

Here's the Lambda@Edge function code:

```typescript:src/resources/lambdaEdge/index.ts
import { CloudFrontRequestEvent, CloudFrontRequestResult } from 'aws-lambda';

export const handler = async (
  event: CloudFrontRequestEvent,
): Promise<CloudFrontRequestResult> => {
  const request = event.Records[0].cf.request;
  const origin = request.origin?.custom;

  if (origin && origin.domainName) {
    request.headers.host = [{ key: 'Host', value: origin.domainName }];
  }

  return request;
};

```

This function rewrites the `Host` header to match the origin's domain name, ensuring that App Runner receives the correct host information. This is essential for Next.js to properly handle dynamic routes.

The Lambda@Edge function is associated with CloudFront in the following way:

```typescript
        edgeLambdas: [
          {
            functionVersion: lambdaEdgeFunction.function.currentVersion,
            eventType: LambdaEdgeEventType.ORIGIN_REQUEST,
          },
          {
            functionVersion: lambdaEdgeFunction.function.currentVersion,
            eventType: LambdaEdgeEventType.VIEWER_REQUEST,
          },
        ],
      },
```

By attaching this function to both the Origin Request and Viewer Request events, we ensure that the headers are properly rewritten for all incoming requests, allowing Next.js to handle dynamic routing correctly when deployed on App Runner behind CloudFront.

## Deployment

The project includes a GitHub Actions workflow for automated deployments. When you push changes to the `main` branch, it will automatically deploy the updated stack to AWS.

To deploy manually:

1. Configure your AWS credentials
2. Run `yarn deploy`

### projen configuration

```typescript
const deploy = project.github?.addWorkflow("deploy");
deploy?.on({
  push: {
    branches: ["main"],
  },
  workflow_dispatch: {},
});

deploy?.addJobs({
  deploy: {
    runsOn: ["ubuntu-latest"],
    permissions: {
      contents: JobPermission.READ,
      id_token: JobPermission.WRITE,
    },
    steps: [
      { uses: "actions/checkout@v4" },
      {
        uses: "actions/setup-node@v4",
        with: {
          "node-version": "18",
        },
      },
      { run: "yarn install --frozen-lockfile" },
      {
        uses: "aws-actions/configure-aws-credentials@v4",
        with: {
          "aws-access-key-id": "${{ secrets.AWS_ACCESS_KEY_ID }}",
          "aws-secret-access-key": "${{ secrets.AWS_SECRET_ACCESS_KEY }}",
          "aws-region": "${{ secrets.AWS_REGION }}",
        },
      },
      { run: "npx cdk deploy --require-approval never" },
    ],
  },
});
```

## Server Side Rendering

This NextJS includes support for Server Side Rendering (SRR) through dynamic routes and API routes:

```typescript
export default function Post({ params }: { params: { id: string } }) {
  return (
    <div>
      <h1>Post {params.id}</h1>
      <p>This is a dynamic route for post with ID: {params.id}</p>
    </div>
  );
}
```

```typescript
import { NextResponse } from "next/server";

export async function GET() {
  return NextResponse.json({ message: "Hello from the API!" });
}

export async function POST(request: Request) {
  const body = await request.json();
  return NextResponse.json({ message: "Hello from the API!", data: body });
}
```

## Local Development

To run the Next.js application locally:

Navigate to the app directory:

```bash
cd src/resources/app
```

Install dependencies:

```bash
yarn
```

Start the development server:

```bash
yarn dev
```

Open [http://localhost:3000](http://localhost:3000) in your browser

## GitHub Updates with Projen

The project uses projen to manage the project and the Next.js application. We have replaced the default projen upgrade with a custom upgrade. This allows us to upgrade the dependencies for both the main project and the Next.js application in a single step.

```typescript
const upgradeWorkflow = project.github?.addWorkflow("upgrade");
upgradeWorkflow?.on({
  schedule: [{ cron: "0 0 * * 1" }],
  workflow_dispatch: {},
});

upgradeWorkflow?.addJobs({
  upgrade: {
    runsOn: ["ubuntu-latest"],
    permissions: {
      contents: JobPermission.WRITE,
      pullRequests: JobPermission.WRITE,
    },
    steps: [
      { uses: "actions/checkout@v4" },
      { uses: "actions/setup-node@v4", with: { "node-version": "18" } },
      { run: "yarn install" },
      { run: "npx projen upgrade:projen" },
      { run: "npx projen upgrade:nextjs" },
      {
        name: "Create Pull Request",
        uses: "peter-evans/create-pull-request@v6",
        with: {
          token: "${{ secrets.PROJEN_GITHUB_TOKEN }}",
          "commit-message": "chore: upgrade dependencies",
          branch: "projen-upgrade",
          title: "chore: upgrade dependencies",
          body: "This PR upgrades project dependencies, including projen and Next.js. Auto-generated by projen upgrade workflow.",
        },
      },
    ],
  },
});

project.addTask("upgrade:nextjs", {
  exec: ["cd src/resources/app", "yarn upgrade --latest", "cd ../../.."].join(
    " && "
  ),
});

project.addTask("upgrade:projen", {
  description: "Upgrade projen dependencies",
  steps: [
    { exec: "yarn upgrade projen @projen/awscdk-app-ts" },
    { exec: "npx projen" },
  ],
});
```

## Route 53 DNS Configuration

This project uses Amazon Route 53 to manage DNS records for the deployed application. The `Route53Resources` construct in `src/route53.ts` sets up the necessary DNS records to point your domain to the CloudFront distribution.

Two A records are created:

- One for the apex domain (e.g., example.com)
- One for the www subdomain (e.g., www.example.com)

Both A records are set up as alias records pointing to the CloudFront distribution:

```typescript
new ARecord(this, "AliasRecord", {
  zone: hostedZone,
  target: RecordTarget.fromAlias(new CloudFrontTarget(props.distribution)),
  recordName: `${props.domainName}.`,
});

new ARecord(this, "WWWAliasRecord", {
  zone: hostedZone,
  target: RecordTarget.fromAlias(new CloudFrontTarget(props.distribution)),
  recordName: `www.${props.domainName}.`,
});
```

This configuration ensures that both the apex domain and the www subdomain resolve to the CloudFront distribution, which in turn serves the Next.js application running on App Runner.

By using alias records, we benefit from Route 53's ability to automatically update the DNS records if the CloudFront distribution's domain name changes, without any manual intervention.

Make sure to update your domain's name servers to use the ones provided by your Route 53 hosted zone for this configuration to take effect.


Updated NextJS App deployed to App Runner with SSR and GitHub Actions


# Next.js App Runner Deployment with CloudFront

This project demonstrates how to deploy a Next.js application using AWS App Runner and CloudFront. It uses the AWS CDK to define and deploy the infrastructure. The project is managed with projen and includes several custom tasks to support the development process.

Please see update: [https://subaud.io/blog/updated-nextjs-app-runner-deployment-with-ssr](https://subaud.io/blog/updated-nextjs-app-runner-deployment-with-ssr)

## Architecture

The architecture consists of the following components:

1. A Next.js application
2. AWS App Runner to host and run the Next.js application
3. Amazon CloudFront as a content delivery network (CDN) in front of App Runner
4. AWS CDK to define and deploy the infrastructure

## Key Features

- Serverless deployment of a Next.js application using AWS App Runner
- CloudFront distribution with custom origin request policy
- Automated CI/CD pipeline using GitHub Actions
- Infrastructure as Code using AWS CDK

## Prerequisites

- AWS Account
- Node.js (version 18 or later)
- AWS CDK CLI
- Docker (for local development and testing)

## Getting Started

Clone the repository:

```bash
git clone https://github.com/schuettc/next-js-app-runner-deployment.git
cd next-js-app-runner-deployment
```

Install dependencies:

```bash
yarn
```

Deploy the stack:

```bash
yarn launch
```

## Project Structure

- `src/`: Contains the CDK infrastructure code
  - `appRunner.ts`: Defines the App Runner service
  - `cloudfront.ts`: Sets up the CloudFront distribution
  - `next-js-app-runner-deployment.ts`: Main stack definition
- `src/resources/app/`: Contains the Next.js application code

## Custom Origin Request Policy

This project implements a custom origin request policy for CloudFront, which is required when using App Runner as the origin. The policy allows specific headers to be forwarded to the origin:

```typescript:src/cloudfront.ts
    const customOriginRequestPolicy = new OriginRequestPolicy(
      this,
      'UserAgentRefererHeadersPolicy',
      {
        headerBehavior: OriginRequestHeaderBehavior.allowList(
          'User-Agent',
          'Referer',
        ),
      },
    );
```

This custom policy ensures that the `User-Agent` and `Referer` headers are forwarded to the App Runner service, which solves the 404 error that occurs when using AppRunner as the origin for a CloudFront distribution.

https://docs.aws.amazon.com/apprunner/latest/dg/request-route-404-troubleshoot.html

## Deployment

The project includes a GitHub Actions workflow for automated deployments. When you push changes to the `main` branch, it will automatically deploy the updated stack to AWS.

To deploy manually:

1. Configure your AWS credentials
2. Run `yarn deploy`

### projen configuration

```typescript
const deploy = project.github?.addWorkflow('deploy');
deploy?.on({
  push: {
    branches: ['main'],
  },
  workflow_dispatch: {},
});

deploy?.addJobs({
  deploy: {
    runsOn: ['ubuntu-latest'],
    permissions: {
      contents: JobPermission.READ,
      id_token: JobPermission.WRITE,
    },
    steps: [
      { uses: 'actions/checkout@v4' },
      {
        uses: 'actions/setup-node@v4',
        with: {
          'node-version': '18',
        },
      },
      { run: 'yarn install --frozen-lockfile' },
      {
        uses: 'aws-actions/configure-aws-credentials@v4',
        with: {
          'aws-access-key-id': '${{ secrets.AWS_ACCESS_KEY_ID }}',
          'aws-secret-access-key': '${{ secrets.AWS_SECRET_ACCESS_KEY }}',
          'aws-region': '${{ secrets.AWS_REGION }}',
        },
      },
      { run: 'npx cdk deploy --require-approval never' },
    ],
  },
});
```

## Local Development

To run the Next.js application locally:

Navigate to the app directory:

```bash
cd src/resources/app
```

Install dependencies:

```bash
yarn
```

Start the development server:

```bash
yarn dev
```

Open [http://localhost:3000](http://localhost:3000) in your browser

## GitHub Updates with Projen

The project uses projen to manage the project and the Next.js application. We have replaced the default projen upgrade with a custom upgrade. This allows us to upgrade the dependencies for both the main project and the Next.js application in a single step.

```typescript
const upgradeWorkflow = project.github?.addWorkflow('upgrade');
upgradeWorkflow?.on({
  schedule: [{ cron: '0 0 * * 1' }],
  workflow_dispatch: {},
});

upgradeWorkflow?.addJobs({
  upgrade: {
    runsOn: ['ubuntu-latest'],
    permissions: {
      contents: JobPermission.WRITE,
      pullRequests: JobPermission.WRITE,
    },
    steps: [
      { uses: 'actions/checkout@v4' },
      { uses: 'actions/setup-node@v4', with: { 'node-version': '18' } },
      { run: 'yarn install' },
      { run: 'npx projen upgrade:projen' },
      { run: 'npx projen upgrade:nextjs' },
      {
        name: 'Create Pull Request',
        uses: 'peter-evans/create-pull-request@v6',
        with: {
          'token': '${{ secrets.PROJEN_GITHUB_TOKEN }}',
          'commit-message': 'chore: upgrade dependencies',
          'branch': 'projen-upgrade',
          'title': 'chore: upgrade dependencies',
          'body':
            'This PR upgrades project dependencies, including projen and Next.js. Auto-generated by projen upgrade workflow.',
        },
      },
    ],
  },
});

project.addTask('upgrade:nextjs', {
  exec: ['cd src/resources/app', 'yarn upgrade --latest', 'cd ../../..'].join(
    ' && ',
  ),
});

project.addTask('upgrade:projen', {
  description: 'Upgrade projen dependencies',
  steps: [
    { exec: 'yarn upgrade projen @projen/awscdk-app-ts' },
    { exec: 'npx projen' },
  ],
});
```


NextJS App deployed to App Runner with GitHub Actions


# GitHub Repository Metrics: A Serverless AWS Implementation

## Overview

This project implements a serverless solution for automatically collecting and analyzing GitHub repository metrics. It uses AWS services including Lambda, S3, Athena, Glue, and EventBridge to gather data from GitHub, store it efficiently, and make it available for analysis.

![Overview](/images/blog/github-insights/GitHubInsights.png)

## GitHub

All code is available here: https://github.com/schuettc/github-insights

## System Architecture

The solution consists of five main components:

1. Lambda Function: Fetches data from GitHub API
2. S3 Bucket: Stores data in Parquet format
3. Glue Database and Table: Defines schema for the data
4. Athena Workgroup and Query: Enables SQL-based analysis
5. EventBridge Rule: Schedules daily data collection

## Implementation Details

### 1. Lambda Function (LambdaResources)

The Lambda function is responsible for querying the GitHub API and writing the results to S3. It's implemented using Node.js and the AWS SDK.

```typescript
export class LambdaResources extends Construct {
  public insightQueryLambda: Function;
  constructor(scope: Construct, id: string, props: LambdaResourcesProps) {
    super(scope, id);

    // ... role creation code omitted for brevity

    this.insightQueryLambda = new NodejsFunction(this, "InsightsQueryLambda", {
      entry: "./src/resources/insightQuery/index.ts",
      runtime: Runtime.NODEJS_LATEST,
      architecture: Architecture.ARM_64,
      memorySize: 1024,
      handler: "handler",
      timeout: Duration.minutes(5),
      role: insightQueryLambdaRole,
      environment: {
        LOG_LEVEL: props.logLevel,
        GITHUB_TOKEN_SECRET_NAME: props.githubTokenSecretName,
        INSIGHTS_BUCKET: props.insightsBucket.bucketName,
      },
    });
    props.insightsBucket.grantReadWrite(this.insightQueryLambda);
  }
}
```

#### Lambda Function Implementation

The core of the data collection is performed by the Lambda function. Here's a simplified version of the main handler:

```typescript
export const handler = async (): Promise<void> => {
  try {
    const githubToken = await getGitHubToken();
    const octokit = new Octokit({ auth: githubToken });

    const repositories = await getRepositoriesFromS3();
    const insights = await fetchRepositoryInsights(octokit, repositories);

    await uploadInsightsToS3(insights);
  } catch (error) {
    console.error("Error in Lambda execution:", error);
    throw error;
  }
};
```

The `fetchRepositoryInsights` function is where the actual GitHub API calls occur:

```typescript
async function fetchRepositoryInsights(
  octokit: Octokit,
  repositories: Repository[]
): Promise<RepoInsights[]> {
  return (
    await Promise.all(
      repositories.map(async ({ owner, repo }) => {
        try {
          const repoData = await octokit.rest.repos.get({ owner, repo });
          const [
            issuesData,
            pullRequestsData,
            contributorsData,
            releasesData,
            commitsData,
          ] = await Promise.all([
            octokit.rest.issues.listForRepo({ owner, repo, state: "all" }),
            octokit.rest.pulls.list({ owner, repo, state: "all" }),
            octokit.rest.repos.listContributors({ owner, repo }),
            octokit.rest.repos
              .getLatestRelease({ owner, repo })
              .catch(() => null),
            octokit.rest.repos.getCommitActivityStats({ owner, repo }),
          ]);

          // Process and aggregate data
          // ...

          return {
            repoName: `${owner}/${repo}`,
            description: repoData.data.description || "",
            stars: repoData.data.stargazers_count,
            forks: repoData.data.forks_count,
            openIssues: repoData.data.open_issues_count,
            // ... other fields
          };
        } catch (error) {
          console.error(`Error fetching data for ${owner}/${repo}:`, error);
          return null;
        }
      })
    )
  ).filter((insight): insight is RepoInsights => insight !== null);
}
```

The collected data is then written to S3 in Parquet format:

```typescript
async function uploadInsightsToS3(insights: RepoInsights[]): Promise<void> {
  const schema = new parquet.ParquetSchema({
    repoName: { type: "UTF8" },
    description: { type: "UTF8" },
    stars: { type: "INT64" },
    // ... other fields
  });

  const writer = await ParquetWriter.openFile(schema, "/tmp/insights.parquet");

  for (const insight of insights) {
    await writer.appendRow({
      ...insight,
      date: `${year}-${month}-${day}`,
    });
  }

  await writer.close();

  const fileContent = await fs.promises.readFile("/tmp/insights.parquet");

  await s3Client.send(
    new PutObjectCommand({
      Bucket: process.env.INSIGHTS_BUCKET,
      Key: `github-insights/year=${year}/month=${month}/day=${day}/insights_${timestamp}.parquet`,
      Body: fileContent,
      ContentType: "application/octet-stream",
    })
  );
}
```

### 2. S3 Bucket (S3Resources)

The S3 bucket is used to store the GitHub repository to query and then to store the GitHub data in Parquet format. The data is partitioned by year, month, and day for efficient querying.

### 3. Glue Database and Table (GlueResources)

AWS Glue is used to define the schema for the GitHub data and set up partition projection.

```typescript
export class GlueResources extends Construct {
  public insightsDatabase: glue.CfnDatabase;
  public insightsTable: CfnTable;
  constructor(scope: Construct, id: string, props: GlueResourcesProps) {
    super(scope, id);

    // ... database creation code omitted for brevity

    this.insightsTable = new glue.CfnTable(this, "GitHubInsightsTable", {
      catalogId: Stack.of(this).account,
      databaseName: this.insightsDatabase.ref,
      tableInput: {
        name: "github_insights",
        tableType: "EXTERNAL_TABLE",
        parameters: {
          EXTERNAL: "TRUE",
          "parquet.compression": "SNAPPY",
          "projection.enabled": "true",
          "projection.year.type": "integer",
          "projection.year.range": "2020,2030",
          "projection.year.digits": "4",
          "projection.month.type": "integer",
          "projection.month.range": "1,12",
          "projection.month.digits": "2",
          "projection.day.type": "integer",
          "projection.day.range": "1,31",
          "projection.day.digits": "2",
          "storage.location.template": `s3://${props.insightsBucket.bucketName}/github-insights/year=$\{year}/month=$\{month}/day=$\{day}`,
        },
        storageDescriptor: {
          // ... storage descriptor details omitted for brevity
        },
        partitionKeys: partitionKeys,
      },
    });
  }
}
```

### 4. Athena Workgroup and Query (AthenaResources)

Amazon Athena is configured with a dedicated workgroup and a sample query for analyzing the GitHub data.

```typescript
export class AthenaResources extends Construct {
  constructor(scope: Construct, id: string, props: AthenaResourceProps) {
    super(scope, id);

    // ... workgroup creation code omitted for brevity

    new athena.CfnNamedQuery(this, "GitHubInsightsSampleQuery", {
      database: props.insightsDatabase.ref,
      queryString: `
        SELECT
          reponame,
          AVG(stars) as avg_stars,
          AVG(forks) as avg_forks,
          AVG(openissues) as avg_open_issues,
          AVG(commitslastmonth) as avg_commits_last_month
        FROM
          github_insights
        WHERE
          year = '${new Date().getUTCFullYear()}'
          AND month = '${(new Date().getUTCMonth() + 1)
            .toString()
            .padStart(2, "0")}'
        GROUP BY
          reponame
        ORDER BY
          avg_stars DESC
      `,
      description:
        "Average stars, forks, open issues, and commits for each repository this month",
      name: "GitHubInsightsMonthlyAverage",
      workGroup: workgroup.ref,
    });
  }
}
```

### 5. EventBridge Rule (EventBridgeResources)

An EventBridge rule is set up to trigger the Lambda function daily.

```typescript
export class EventBridgeResources extends Construct {
  constructor(scope: Construct, id: string, props: EventBridgeResourcesProps) {
    super(scope, id);

    new Rule(this, "DailyLambdaTrigger", {
      schedule: Schedule.cron({ minute: "0", hour: "0" }),
      targets: [new LambdaFunction(props.insightQueryLambda)],
    });
  }
}
```

## Data Flow

1. The EventBridge rule triggers the Lambda function daily.
2. The Lambda function fetches data from GitHub and writes it to S3 in Parquet format.
3. The data is automatically partitioned in S3 based on the Glue table definition.
4. Users can query the data using Athena, with the sample query providing a starting point for analysis.

## Deployment

The entire infrastructure is defined using AWS CDK, allowing for version-controlled, reproducible deployments. The main stack is defined in the `GitHubInsights` class:

```typescript
export class GitHubInsights extends Stack {
  constructor(scope: Construct, id: string, props: GitHubInsightsProps) {
    super(scope, id, props);

    const s3Resources = new S3Resources(this, "S3Resources");
    const glueResources = new GlueResources(this, "GlueResources", {
      insightsBucket: s3Resources.insightsBucket,
    });
    new AthenaResources(this, "AthenaResources", {
      insightsBucket: s3Resources.insightsBucket,
      insightsDatabase: glueResources.insightsDatabase,
    });
    const lambdaResources = new LambdaResources(this, "LambdaResources", {
      logLevel: props.logLevel,
      insightsBucket: s3Resources.insightsBucket,
      githubTokenSecretName: props.githubTokenSecretName,
    });
    new EventBridgeResources(this, "EventBridgeResources", {
      insightQueryLambda: lambdaResources.insightQueryLambda,
    });
  }
}
```

## Querying Data with Athena

Once the data is stored in S3 and the Glue table is set up, users can query the data using Amazon Athena. Here are some example queries:

1. Monthly average metrics for each repository:

```sql
SELECT
  reponame,
  AVG(stars) as avg_stars,
  AVG(forks) as avg_forks,
  AVG(openissues) as avg_open_issues,
  AVG(commitslastmonth) as avg_commits_last_month
FROM
  github_insights
WHERE
  year = '2024'
  AND month = '08'
GROUP BY
  reponame
ORDER BY
  avg_stars DESC
```

2. Repositories with the most growth in stars over the last 30 days:

```sql
WITH current_stats AS (
  SELECT reponame, stars
  FROM github_insights
  WHERE year = '2024' AND month = '08' AND day = '15'
),
previous_stats AS (
  SELECT reponame, stars
  FROM github_insights
  WHERE year = '2024' AND month = '07' AND day = '16'
)
SELECT
  c.reponame,
  c.stars - p.stars AS star_growth,
  ((c.stars - p.stars) / p.stars) * 100 AS growth_percentage
FROM current_stats c
JOIN previous_stats p ON c.reponame = p.reponame
ORDER BY star_growth DESC
LIMIT 10
```

3. Repositories with the highest ratio of closed to open issues:

```sql
SELECT
  reponame,
  AVG(closedissues) as avg_closed_issues,
  AVG(openissues) as avg_open_issues,
  AVG(closedissues) / NULLIF(AVG(openissues), 0) as closed_to_open_ratio
FROM
  github_insights
WHERE
  year = '2024'
  AND month = '08'
GROUP BY
  reponame
HAVING
  AVG(openissues) > 0
ORDER BY
  closed_to_open_ratio DESC
LIMIT 10
```

To run these queries:

1. Open the Athena console in AWS.
2. Select the `github_insights_db` database.
3. Enter the SQL query in the query editor.
4. Click "Run query" to execute and view the results.

The partition projection set up in the Glue table definition allows Athena to efficiently query the data without the need for manual partition management, making it easy to analyze historical trends and current repository states.

![ExampleQuery](/images/blog/github-insights/ExampleQuery.png)

# Deploying

## Configuration Files

### 1. `.env`

This file contains environment variables used by the scripts and the CDK deployment.

**Example content:**

```bash
GITHUB_TOKEN_SECRET=gph_xxxxxxxxxxx
STACK_NAME=GitHubInsights
AWS_REGION=us-east-1
```

**Fields:**

- `GITHUB_TOKEN_SECRET`: Your GitHub personal access token. You will need to create one within GitHub.
- `STACK_NAME`: The name of your CDK stack (used by `upload_repo_list.sh`).
- `AWS_REGION`: The AWS region where your resources are deployed.

**Note:** Never commit the `.env` file to version control. It's included in `.gitignore` by default.

### 2. `repositories.json`

This file contains the list of GitHub repositories you want to monitor.

**Example content:**

```json
[
  {
    "owner": "schuettc",
    "repo": "github-insights"
  }
]
```

**Usage:**

- Add or remove repository objects as needed.
- Each repository object should have an "owner" and "repo" field.
- After modifying this file, run `upload_repo_list.sh` to update the list in your S3 bucket.

## Helper Scripts

### 1. `setup_github_token.sh`

This script is used to securely store your GitHub token in AWS Secrets Manager.

**Purpose:**

- Creates or updates a secret in AWS Secrets Manager to store your GitHub token.
- Allows secure access to the GitHub API without exposing the token in your code.

**Usage:**

1. Ensure you have the AWS CLI configured with appropriate permissions.
2. Create a `.env` file in the project root (see Configuration Files section below).
3. Run the script:

```bash
./setup_github_token.sh
```

**What it does:**

- Reads the `GITHUB_TOKEN_SECRET` from the `.env` file.
- Creates or updates a secret in AWS Secrets Manager with the provided token.
- Uses the `GITHUB_TOKEN_SECRET_NAME` (default: "/github_insights/github_token") as the secret name.
- Sets the AWS region based on the `AWS_REGION` in `.env` or defaults to "us-east-1".

### 2. `upload_repo_list.sh`

This script uploads the list of repositories to be monitored to your S3 bucket.

**Purpose:**

- Uploads the `repositories.json` file to the S3 bucket created by the CDK stack.
- Allows you to easily update the list of repositories to be monitored.

**Usage:**

1. Ensure you have the AWS CLI configured with appropriate permissions.
2. Update the `repositories.json` file with the repositories you want to monitor.
3. Run the script:

```bash
./upload_repo_list.sh
```

**What it does:**

- Retrieves the S3 bucket name from the CloudFormation stack outputs.
- Uploads the `repositories.json` file to the root of the S3 bucket.
- Uses the `STACK_NAME` from the `.env` file to identify the correct CloudFormation stack.
- Sets the AWS region based on the `AWS_REGION` in `.env` or defaults to "us-east-1".

## Workflow

1. Set up your `.env` file with your GitHub token and stack name.
2. Deploy your CDK stack by running `yarn launch`
3. Run `setup_github_token.sh` to securely store your GitHub token.
4. Update `repositories.json` with the list of repositories you want to monitor.
5. Run `upload_repo_list.sh` to upload the repository list to your S3 bucket.

By following this workflow, you ensure that your GitHub token is securely stored and your Lambda function has the latest list of repositories to monitor.

## Conclusion

This serverless architecture provides an efficient method for collecting and analyzing GitHub repository metrics. It leverages AWS services to automate data collection, storage, and query capabilities, enabling data-driven insights into repository activity and health.


In this demo we will see how to deploy an application on AWS that will capture insights from GitHub and store them in S3 to be easily queried with Amazon Athena

GitHub Insights - Capturing Repository Metrics


In this demo we'll see how to deploy a NextJS application to AWS App Runner via CDK using GitHub Actions. This demo will combine and showcase several useful tools to help with deployments. Much of this could also be done using a CDK Pipeline. For smaller and simpler use cases, this can be a lighter weight way of doing it, but is more fragile. This is a simpler and cheaper version the previous [CDK Pipeline for ECS hosted NextJS App](https://subaud.io/blog/cdk-pipeline-ecs-hosted-nextjs) while still supporting the main features in it.

GitHub: https://github.com/schuettc/cdk-nextjs-apprunner

## Overview

![Overview](/images/blog/nextjs-app-deployed-to-app-runner-with-cdk/Overview.png)

The core of this demo is the mechanism to take a local [NextJS](https://nextjs.org/) app, upload it to an S3 bucket, build it using Docker and deploy it to [AWS App Runner](https://aws.amazon.com/apprunner/). By using [AWS CodePipeline](https://aws.amazon.com/codepipeline/) and [AWS CodeBuild](https://aws.amazon.com/codebuild/) to build the [Docker](https://www.docker.com/) image, we eliminate the need to have Docker running locally for our deployment. This allows us to use GitHub actions to use CDK to deploy this application.

## Uploading to S3

```typescript
const dockerAsset = new Asset(this, 'DockerAsset', {
  path: './site',
  exclude: ['**/node_modules/**', '**/lib/**', '/**/.next/**'],
  ignoreMode: IgnoreMode.GIT,
});
```

By using [Asset](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3_assets.Asset.html) we are able to Zip and upload all of the relevant files in our NextJS application to an S3 bucket managed by CDK. We will use this S3 bucket as the trigger for our CodePipeline.

## CodePipeline

The actual pipeline for this demo is relatively simple. Using polling on the S3 bucket, it will download the Zip file from S3, extract the files, and execute a build on them.

```typescript
this.pipeline = new Pipeline(this, 'Pipeline', {
  stages: [
    {
      stageName: 'Source',
      actions: [
        new S3SourceAction({
          actionName: 'S3Source',
          bucket: dockerAsset.bucket,
          bucketKey: dockerAsset.s3ObjectKey,
          output: sourceOutput,
        }),
      ],
    },
    {
      stageName: 'Build',
      actions: [
        new CodeBuildAction({
          actionName: 'DockerBuild',
          project: codeBuildProject,
          input: sourceOutput,
          outputs: [buildOutput],
        }),
      ],
    },
  ],
});
```

The buildSpec for this involves three basic steps:

1. Log in to ECR
2. Execute `docker build`
3. Execute `docker push`

```typescript
buildSpec: BuildSpec.fromObject({
  version: '0.2',
  phases: {
    pre_build: {
      commands: [
        'echo Logging in to Amazon ECR...',
        'aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com',
      ],
    },
    build: {
      commands: [
        'echo Build started on `date`',
        'echo Building the Docker image...',
        'docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .',
        'docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG',
      ],
    },
    post_build: {
      commands: [
        'echo Build completed on `date`',
        'echo Pushing the Docker image...',
        'docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG',
      ],
    },
  },
}),
```

With these three steps, we are able to create a Docker image and push to our [Amazon ECR repository](https://docs.aws.amazon.com/ecr/). This is what will be consumed by our App Runner service.

## ImageChecker

In order to make sure the Docker image has been created and is available for the App Runner service, we will create a Lambda backed Custom Resource that will periodically check the pipeline to ensure it has completed successfully. When it has, the Customer Resource will complete and the CDK can continue. We can enforce this by adding a dependency to the Customer Resource to the App Runner service.

```typescript
appRunnerService.service.node.addDependency(imageChecker);
```

Now the CDK will not start deploying the App Runner service until the Customer Resource has completed. This will only run on the initial build. On updates, when a new image is pushed to ECR after the build process completes, App Runner will automatically deploy the updated image.

## App Runner

Finally, we will create the App Runner service and automatically deploy the image.

```typescript
this.service = new apprunnerAlpha.Service(this, 'Service', {
  source: apprunnerAlpha.Source.fromEcr({
    imageConfiguration: { port: 3000 },
    repository: ecrRepository.repository,
    tagOrDigest: 'latest',
  }),
  autoDeploymentsEnabled: true,
  accessRole: appRunnerRole,
});
```

To ensure this works correctly with NextJS we will add a `HOSTNAME` environment variable to the App Runner service.

```typescript
this.service.addEnvironmentVariable('HOSTNAME', '0.0.0.0');
```

## Projen

To help manage this project, we will be using [projen](https://projen.io/).

### Subproject

In addition to the main project, we are also using a subproject to manage the NextJS application with a `parent` of the main project.

```typescript
const site = new web.NextJsTypeScriptProject({
  parent: project,
  defaultReleaseBranch: 'main',
  name: 'cdk-nextjs-apprunner-site',
  outdir: 'site',
  projenrcTs: true,
  tailwind: false,
  tsconfig: {
    compilerOptions: {
      rootDir: '.',
      baseUrl: '.',
    },
    include: [
      'pages/**/*.ts',
      'pages/**/*.tsx',
      'components/**/*.ts',
      'components/**/*.tsx',
      '**/*.ts',
      '**/*.tsx',
      'next-env.d.ts',
    ],
    exclude: ['node_modules'],
  },
  deps: ['@cloudscape-design/components', '@cloudscape-design/global-styles'],
});

site.synth();
```

### CDK Deploy GitHub Action

Because we are using CodeBuild to build the Docker image, we can use GitHub actions to start the deploy process. To do this with projen, we will create a new workflow. This will be added to the existing actions and trigger on a merge or push to `main`.

```typescript
const cdkDeploy = project.github.addWorkflow('cdk-deploy');
cdkDeploy.on({
  push: { branches: ['main'] },
  pullRequest: { branches: ['main'], types: ['closed'] },
});

cdkDeploy.addJobs({
  deploy: {
    runsOn: ['ubuntu-latest'],
    name: 'Deploy CDK Stack',
    permissions: {
      actions: JobPermission.WRITE,
      contents: JobPermission.READ,
      idToken: JobPermission.WRITE,
    },
    if: "github.event.pull_request.merged == true || github.event_name == 'push'",
    steps: [
      { uses: 'actions/checkout@v3' },
      {
        name: 'Setup Node.js',
        uses: 'actions/setup-node@v3',
        with: {
          'node-version': '18',
        },
      },
      { run: 'yarn install --frozen-lockfile' },
      {
        name: 'Configure AWS Credentials',
        uses: 'aws-actions/configure-aws-credentials@v4',
        with: {
          'aws-access-key-id': '${{ secrets.AWS_ACCESS_KEY_ID }}',
          'aws-secret-access-key': '${{ secrets.AWS_SECRET_ACCESS_KEY }}',
          'aws-region': 'us-east-1',
          'role-session-name': 'GitHubActionsCDKDeploy',
        },
      },
      {
        name: 'CDK Diff',
        run: 'npx cdk diff',
      },
      {
        name: 'CDK Deploy',
        run: 'npx cdk deploy --all --require-approval never',
      },
    ],
  },
});
```

### AWS Credentials

In order to use this feature, you will need to create a user in your AWS account that has sufficient permissions to deploy the CDK. Using the ACCESS_KEY_ID and SECRET_ACCESS_KEY generated for this user, GitHub can deploy the CDK. To create a user with permissions you can run:

```bash
./setup_aws_cdk_deploy
```

The output will contain an ACCESS_KEY_ID and SECRET_ACCESS_KEY that can be added to your GitHub repo as secrets.

![GitHubSecrets](/images/blog/nextjs-app-deployed-to-app-runner-with-cdk/GitHubSecrets.png)

The policy attached to this user has extensive permissions so care must be taken with this ACCESS_KEY_ID.


Deploy a NextJS Application to AWS App Runner using CDK

Github

Custom GitHub Action workflows with projen

Using PyGitHub to Manage and Track GitHub Repositories

Updated NextJS App deployed to App Runner with SSR and GitHub Actions

NextJS App deployed to App Runner with GitHub Actions

GitHub Insights - Capturing Repository Metrics

NextJS App deployed to App Runner with CDK