Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version2 #4

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions _index.ee.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
---
title: "Overcoming Bottlenecks in RDS & Enhancing Development Workflows with Neon " # MODIFY THIS TO BE THE TITLE OF YOUR WORKSHOP
title: "Overcoming Bottlenecks in RDS & Accelerating Development/Testing Workflows with Neon " # MODIFY THIS TO BE THE TITLE OF YOUR WORKSHOP
chapter: true
weight: 1
---

# AWS Workshop: Overcoming RDS Bottlenecks & Enhancing Development Workflows with Neon <!-- CHANGE THIS TO BE THE TITLE OF YOUR WORKSHOP -->
# Overcoming RDS Bottlenecks & Enhancing Development Workflows with Neon <!-- CHANGE THIS TO BE THE TITLE OF YOUR WORKSHOP -->
<br>
![Partner Logo](/images/AWS_Logo.png) <!-- ADD YOUR PARTNER LOGO HERE USING THE INSTRUCTIONS BELOW -->
<br>

## Welcome to Neon!

In this workshop you will learn how to boost your development speed with Neon and AWS RDS
In this workshop you will learn how to accelerate development in RDS by creating ephemeral environments in Neon Postgres.

- We'll show you how to speed up your development process by integrating **Neon** with your existing **AWS RDS Postgres** setup. If you don't have an existing RDS Postgres setup, we'll walk you through how to set one up and migrate your staging data to Neon.
- While RDS is great for production, it has limitations when it comes to development. Neon, a serverless Postgres solution with features like **database branching**, provides a more flexible and developer-friendly environment for testing and development. With Neon, you can improve your workflows and collaboration without making disruptive changes to your production database.
Expand Down
14 changes: 6 additions & 8 deletions content/1_ModuleOne/12_CohostedWorkshopPrerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,9 @@ Neon is built on an innovative [branch-based architecture](https://neon.tech/bl
- Once everything is set up, you’ll be able to create additional branches from the main branch to [duplicate your development environment in a second](https://neon.tech/blog/how-to-copy-large-postgres-databases-in-seconds), without additional storage costs, as many times as you need.
- Every engineer on your team can have their own dev branch, facilitating parallel development without adding overheads in database management or costs.

<br>
🚨 Once you’ve set up your Neon account and your project, **create a database in the main branch, and make a note of the connection string.** You will need this in the next step.
<br>
{{% notice warning %}}
🚨 Once you’ve set up your Neon account and your project, **create a database in the main branch, and make a note of the connection string.** You will need this in the next steps.
{{% /notice %}}

#### Select Your Project -> Connection Details -> Connection String

Expand All @@ -63,11 +63,9 @@ Neon is built on an innovative [branch-based architecture](https://neon.tech/bl


### 3. GitHub repository access to Actions and Secrets
- **Directions for preparing the GitHub Repository** <br>
- **Create a GitHub repository** if you don't have one already, as this will store the automation scripts and workflows for the synchronization. <br>

[GitHub Directions For Creating a Repository Here]
First you will need to **make sure you have a GitHub Account**, if you do not have one already, you can sign up for one [here.](https://github.com/signup) <br>

- **Add necessary secrets** to GitHub Secrets: You’ll need the connection URLs for both your RDS production database and your Neon database (these can be added to GitHub Secrets under `PROD_DATABASE_URL` and `DEV_DATABASE_URL`).
**Create a GitHub repository** if you don't have one already, as this will store the automation scripts and workflows for the synchronization. <br>

[GitHub Directions For Adding Secrets]
- **Add necessary secrets** to GitHub Secrets: You’ll need the connection URLs for both your RDS production database and your Neon database (these can be added to GitHub Secrets under `PROD_DATABASE_URL` and `DEV_DATABASE_URL`). Directions on how to add secrets to GitHub can be found [here.](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions?tool=webui)
90 changes: 89 additions & 1 deletion content/2_ModuleTwo/21_CohostedNeonTwinDeploySetUp.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,96 @@ weight: 1 # MODIFY THIS VALUE TO REFLECT THE ORDERING OF THE MODULES

If you're already running RDS in production, migrating a live database can be a real challenge. The good news is, you don’t have to move everything over to enjoy Neon's advantages in development speed and cost efficiency for non-production databases.

![Neon Twin](/static/images/neontwin.jpg)

With a Neon Twin—a synchronized copy of your RDS data in Neon—there’s a simple, automated way to keep your development environment up-to-date. This copy updates automatically each night using ```pg_dump/restore ``` and ```GitHub Actions```, so your development data stays fresh without any heavy lifting.

## Submodule One Heading <!-- MODIFY THIS SUBHEADING -->


## Create a Neon Twin in your GitHub Actions workflow:
### 1. Create a workflow YAML file:
Add a new workflow file in your GitHub repository at ```.github/workflows/create-neon-twin.yml.``` This file will define how the synchronization happens.


```yaml
name: Create Neon Twin

on:
schedule:
- cron: '0 0 * * *' # Runs every day at midnight
workflow_dispatch: # Allows manual trigger

env:
PROD_DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }} # RDS connection string
DEV_DATABASE_URL: ${{ secrets.DEV_DATABASE_URL }} # Neon connection string
PG_VERSION: '16'

jobs:
dump-and-restore:
runs-on: ubuntu-latest

steps:
- name: Install PostgreSQL
run: |
sudo apt update
yes '' | sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo apt install -y postgresql-${{ env.PG_VERSION }}

- name: Dump from RDS and Restore to Neon
run: |
/usr/lib/postgresql/${{ env.PG_VERSION }}/bin/pg_dump "${{ env.PROD_DATABASE_URL }}" -Fc -f "${{ github.workspace }}/prod-dump-file.dump"
/usr/lib/postgresql/${{ env.PG_VERSION }}/bin/pg_restore -d "${{ env.DEV_DATABASE_URL }}" --clean --no-owner --no-acl --if-exists "${{ github.workspace }}/prod-dump-file.dump"

```

### 2. Understanding the Workflow Steps

#### Workflow Components

| Component | Description |
|-----------|-------------|
| **Cron** | Runs nightly at midnight (Eastern Time) using POSIX cron syntax |
| **workflow_dispatch** | Enables manual workflow triggering during development |

#### Environment Variables

| Variable | Description |
|----------|-------------|
| **PROD_DATABASE_URL** | Connection string for production database (stored in GitHub Secrets) |
| **DEV_DATABASE_URL** | Connection string for Neon Twin (stored in GitHub Secrets) |
| **PG_VERSION** | Specifies the Postgres version to install |

#### Database Operations

| Command | Flag | Description |
|---------|------|-------------|
| **pg_dump** | `-Fc` | Uses custom format for dump file |
| | `-f` | Specifies dump file path and name |
| **pg_restore** | `--clean` | Removes existing objects before restore |
| | `--if-exists` | Prevents errors for non-existent objects |
| | `--no-owner` | Skips original ownership restoration |
| | `--no-acl` | Skips original permissions restoration |
| | `-d` | Specifies target database using DEV_DATABASE_URL |

> **Note**: Environment variables mentioned above must be configured in your GitHub repository's secrets for the Action to work properly.



### 2. Configuring secrets in GitHub Actions

In the workshop prerequisites module you will have created a GitHub repository and added secrets to the repository.

- In your GitHub Actions workflow (e.g., in `.github/workflows/create-neon-twin.yml`), reference these secrets like this:

```yaml
yaml
Copy code
env:
PROD_DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }}
DEV_DATABASE_URL: ${{ secrets.DEV_DATABASE_URL }}

```
This syntax ensures that the sensitive values, such as database credentials or API keys, are securely injected into your workflow at runtime without hardcoding them into the file.



8 changes: 4 additions & 4 deletions content/2_ModuleTwo/_index.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: " Module One " # MODIFY THIS TITLE IF APPLICABLE
title: " Set Up Instructions " # MODIFY THIS TITLE IF APPLICABLE
chapter: true
weight: 2
---

# Addressing RDS Development Challenges and Unlocking Neon's Potential <!-- MODIFY THIS HEADING -->
## Overcoming Bottlenecks & Enhancing Workflows with Neon
!-- MODIFY THIS HEADING -->
# Overcoming Bottlenecks & Enhancing Dev/Test Workflows with Neon
AWS RDS for PostgreSQL is a solid, established solution that has been instrumental in bringing Postgres into modern workplaces. While Neon can handle mission-critical workloads, its real strength lies in optimizing the developer workflow so your developers can build faster and optimize costs.

## RDS in Development: Overview of AWS RDS Bottlenecks
Expand All @@ -31,4 +31,4 @@ Neon’s key features and benefits, focusing on how it improves development and
- **Instant Environment Provisioning:** Learn how Neon enables the quick setup of isolated environments for each team member, speeding up development and reducing bottlenecks.
- **Database Branching:** Explore how Neon’s database branching feature provides isolated, production-like environments for safe experimentation and testing.
- **CI/CD Integration with GitHub Actions:** Understand how Neon integrates seamlessly with CI/CD workflows, allowing for efficient automation and testing pipelines.
- Discover how Neon’s scale-to-zero functionality helps save on infrastructure costs by automatically scaling down environments when not in use.
- **Scale-to-zero:** Discover how Neon’s scale-to-zero functionality helps save on infrastructure costs by automatically scaling down environments when not in use.
Binary file added content/2_ModuleTwo/image-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added content/2_ModuleTwo/image.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/images/neontwin.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.