Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version2 #4

Merged
merged 4 commits into from
Dec 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 0 additions & 26 deletions _index.ee.md

This file was deleted.

14 changes: 6 additions & 8 deletions content/1_ModuleOne/12_CohostedWorkshopPrerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,9 @@ Neon is built on an innovative [branch-based architecture](https://neon.tech/bl
- Once everything is set up, you’ll be able to create additional branches from the main branch to [duplicate your development environment in a second](https://neon.tech/blog/how-to-copy-large-postgres-databases-in-seconds), without additional storage costs, as many times as you need.
- Every engineer on your team can have their own dev branch, facilitating parallel development without adding overheads in database management or costs.

<br>
🚨 Once you’ve set up your Neon account and your project, **create a database in the main branch, and make a note of the connection string.** You will need this in the next step.
<br>
{{% notice warning %}}
🚨 Once you’ve set up your Neon account and your project, **create a database in the main branch, and make a note of the connection string.** You will need this in the next steps.
{{% /notice %}}

#### Select Your Project -> Connection Details -> Connection String

Expand All @@ -63,11 +63,9 @@ Neon is built on an innovative [branch-based architecture](https://neon.tech/bl


### 3. GitHub repository access to Actions and Secrets
- **Directions for preparing the GitHub Repository** <br>
- **Create a GitHub repository** if you don't have one already, as this will store the automation scripts and workflows for the synchronization. <br>

[GitHub Directions For Creating a Repository Here]
First you will need to **make sure you have a GitHub Account**, if you do not have one already, you can sign up for one [here.](https://github.com/signup) <br>

- **Add necessary secrets** to GitHub Secrets: You’ll need the connection URLs for both your RDS production database and your Neon database (these can be added to GitHub Secrets under `PROD_DATABASE_URL` and `DEV_DATABASE_URL`).
**Create a GitHub repository** if you don't have one already, as this will store the automation scripts and workflows for the synchronization. <br>

[GitHub Directions For Adding Secrets]
- **Add necessary secrets** to GitHub Secrets: You’ll need the connection URLs for both your RDS production database and your Neon database (these can be added to GitHub Secrets under `PROD_DATABASE_URL` and `DEV_DATABASE_URL`). Directions on how to add secrets to GitHub can be found [here.](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions?tool=webui)
90 changes: 89 additions & 1 deletion content/2_ModuleTwo/21_CohostedNeonTwinDeploySetUp.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,96 @@ weight: 1 # MODIFY THIS VALUE TO REFLECT THE ORDERING OF THE MODULES

If you're already running RDS in production, migrating a live database can be a real challenge. The good news is, you don’t have to move everything over to enjoy Neon's advantages in development speed and cost efficiency for non-production databases.

![Neon Twin](/static/images/neontwin.jpg)

With a Neon Twin—a synchronized copy of your RDS data in Neon—there’s a simple, automated way to keep your development environment up-to-date. This copy updates automatically each night using ```pg_dump/restore ``` and ```GitHub Actions```, so your development data stays fresh without any heavy lifting.

## Submodule One Heading <!-- MODIFY THIS SUBHEADING -->


## Create a Neon Twin in your GitHub Actions workflow:
### 1. Create a workflow YAML file:
Add a new workflow file in your GitHub repository at ```.github/workflows/create-neon-twin.yml.``` This file will define how the synchronization happens.


```yaml
name: Create Neon Twin

on:
schedule:
- cron: '0 0 * * *' # Runs every day at midnight
workflow_dispatch: # Allows manual trigger

env:
PROD_DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }} # RDS connection string
DEV_DATABASE_URL: ${{ secrets.DEV_DATABASE_URL }} # Neon connection string
PG_VERSION: '16'

jobs:
dump-and-restore:
runs-on: ubuntu-latest

steps:
- name: Install PostgreSQL
run: |
sudo apt update
yes '' | sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo apt install -y postgresql-${{ env.PG_VERSION }}

- name: Dump from RDS and Restore to Neon
run: |
/usr/lib/postgresql/${{ env.PG_VERSION }}/bin/pg_dump "${{ env.PROD_DATABASE_URL }}" -Fc -f "${{ github.workspace }}/prod-dump-file.dump"
/usr/lib/postgresql/${{ env.PG_VERSION }}/bin/pg_restore -d "${{ env.DEV_DATABASE_URL }}" --clean --no-owner --no-acl --if-exists "${{ github.workspace }}/prod-dump-file.dump"

```

### 2. Understanding the Workflow Steps

#### Workflow Components

| Component | Description |
|-----------|-------------|
| **Cron** | Runs nightly at midnight (Eastern Time) using POSIX cron syntax |
| **workflow_dispatch** | Enables manual workflow triggering during development |

#### Environment Variables

| Variable | Description |
|----------|-------------|
| **PROD_DATABASE_URL** | Connection string for production database (stored in GitHub Secrets) |
| **DEV_DATABASE_URL** | Connection string for Neon Twin (stored in GitHub Secrets) |
| **PG_VERSION** | Specifies the Postgres version to install |

#### Database Operations

| Command | Flag | Description |
|---------|------|-------------|
| **pg_dump** | `-Fc` | Uses custom format for dump file |
| | `-f` | Specifies dump file path and name |
| **pg_restore** | `--clean` | Removes existing objects before restore |
| | `--if-exists` | Prevents errors for non-existent objects |
| | `--no-owner` | Skips original ownership restoration |
| | `--no-acl` | Skips original permissions restoration |
| | `-d` | Specifies target database using DEV_DATABASE_URL |

> **Note**: Environment variables mentioned above must be configured in your GitHub repository's secrets for the Action to work properly.



### 2. Configuring secrets in GitHub Actions

In the workshop prerequisites module you will have created a GitHub repository and added secrets to the repository.

- In your GitHub Actions workflow (e.g., in `.github/workflows/create-neon-twin.yml`), reference these secrets like this:

```yaml
yaml
Copy code
env:
PROD_DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }}
DEV_DATABASE_URL: ${{ secrets.DEV_DATABASE_URL }}

```
This syntax ensures that the sensitive values, such as database credentials or API keys, are securely injected into your workflow at runtime without hardcoding them into the file.



8 changes: 4 additions & 4 deletions content/2_ModuleTwo/_index.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: " Module One " # MODIFY THIS TITLE IF APPLICABLE
title: " Set Up Instructions " # MODIFY THIS TITLE IF APPLICABLE
chapter: true
weight: 2
---

# Addressing RDS Development Challenges and Unlocking Neon's Potential <!-- MODIFY THIS HEADING -->
## Overcoming Bottlenecks & Enhancing Workflows with Neon
!-- MODIFY THIS HEADING -->
# Overcoming Bottlenecks & Enhancing Dev/Test Workflows with Neon
AWS RDS for PostgreSQL is a solid, established solution that has been instrumental in bringing Postgres into modern workplaces. While Neon can handle mission-critical workloads, its real strength lies in optimizing the developer workflow so your developers can build faster and optimize costs.

## RDS in Development: Overview of AWS RDS Bottlenecks
Expand All @@ -31,4 +31,4 @@ Neon’s key features and benefits, focusing on how it improves development and
- **Instant Environment Provisioning:** Learn how Neon enables the quick setup of isolated environments for each team member, speeding up development and reducing bottlenecks.
- **Database Branching:** Explore how Neon’s database branching feature provides isolated, production-like environments for safe experimentation and testing.
- **CI/CD Integration with GitHub Actions:** Understand how Neon integrates seamlessly with CI/CD workflows, allowing for efficient automation and testing pipelines.
- Discover how Neon’s scale-to-zero functionality helps save on infrastructure costs by automatically scaling down environments when not in use.
- **Scale-to-zero:** Discover how Neon’s scale-to-zero functionality helps save on infrastructure costs by automatically scaling down environments when not in use.
Binary file added content/2_ModuleTwo/image-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added content/2_ModuleTwo/image.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/images/neontwin.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.