colem/todo-hetzner
1
0
Fork 0
mirror of https://github.com/cole-maxwell1/todo-hetzner.git synced 2026-10-27 19:02:49 -05:00
Code Issues Packages Projects Releases Wiki Activity

feat: add Hetzner-hosted todo application stack

Browse Source 
Implement the server-rendered Go todo application with PostgreSQL persistence, Tailwind styling, health checks, and test coverage.

Provision production and staging infrastructure with OpenTofu, including private networking, static IPs, Object Storage backups, and load balancing.

Configure Rocky Linux hosts with Ansible, rootless Podman, PostgreSQL, WAL-G backup automation, staging restores, and generated inventory.
This commit is contained in:
cole-maxwell1
2026-08-06 21:34:56 -05:00
commit 1382c42803
68 changed files with 2380 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+347
View File
@@ -0,0 +1,347 @@
# Todo on Hetzner
This repository deploys a server-rendered Go todo application and its supporting infrastructure to Hetzner Cloud. A shared OpenTofu state owns the private network, production and staging Rocky Linux 10 servers, static primary IPs, firewall, Object Storage bucket, and HTTP load balancer. Ansible configures host PostgreSQL, rootless Podman, WAL-G backups, and scheduled staging restores.
Staging contains an unchanged copy of production todo data. Treat staging hosts, logs, database access, and backups as production-sensitive.
## Architecture
- The load balancer accepts HTTP on port 80 and reaches production on `10.0.1.10:8080` over the private network.
- The Go application runs as the rootless `todo` user through Quadlet-generated `app-todo.service` with host networking.
- PostgreSQL listens only on `127.0.0.1:5432`; the cloud firewall does not expose it.
- Production continuously archives WAL and performs an hourly base backup to `s3://<bucket>/prod`.
- Staging restores `LATEST` at 06:00 and 12:00 in `America/Chicago`, then resets the application role to the staging-specific password.
- Staging PostgreSQL has archiving disabled, and its WAL-G wrapper rejects `wal-push`.
## Prerequisites
- Go 1.22 or newer
- OpenTofu 1.8 or newer
- Ansible Core 2.16 or newer
- Docker or Podman for image builds
- `curl` and `make`
- A Hetzner Cloud project, API token, SSH key, and Object Storage credentials
- A container registry if deploying a non-public image
The app intentionally uses `xorm.io/xorm` with `github.com/lib/pq`, which is the PostgreSQL driver supported by xorm in this stack.
## Local Development
Run a local PostgreSQL database and export a connection URL:
```sh
export DATABASE_URL='postgres://todo_app:local-password@127.0.0.1:5432/todo?sslmode=disable'
make css
make build-app
./app/bin/todo
```
The server listens on `:8080` by default. Override it with `APP_ADDR`. The app runs the additive `todos` table migration at startup.
Useful checks:
```sh
make test
make vet
make format
make validate
docker build -t todo:local app
```
`make css` downloads the pinned standalone Tailwind CLI to `app/tailwindcss`. The Docker build independently downloads the CLI before compiling CSS, so it does not depend on a pre-existing generated stylesheet.
## Hetzner Credentials And Access
The stack needs two independent Hetzner credentials:
- A Hetzner Cloud API token for servers, networks, firewalls, primary IPs, and the load balancer.
- An Object Storage access key and secret for the S3 bucket and WAL-G.
The Cloud API token does not grant Object Storage access.
### Cloud API Token
Open the Hetzner Console, select the Cloud project, go to **Security > API Tokens**, and create a **Read & Write** token. Export it as an OpenTofu input:
```sh
export TF_VAR_hcloud_token='replace-with-cloud-api-token'
```
Do not place the token in a tracked file or pass it directly on a command line that is retained in shell history.
### SSH Key And ID
Hetzner installs the selected public SSH key on each server. Only upload the public `.pub` file; never upload the corresponding private key.
Generate an Ed25519 key if one does not already exist:
```sh
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519
```
Install the Hetzner CLI on macOS, expose the Cloud token to it, and upload the public key:
```sh
brew install hcloud
export HCLOUD_TOKEN="$TF_VAR_hcloud_token"
hcloud ssh-key create \
--name home-mac \
--public-key-from-file ~/.ssh/id_ed25519.pub
```
List the project keys and copy the numeric ID from the `ID` column:
```sh
hcloud ssh-key list
```
Configure that value in `prod.tfvars`:
```hcl
ssh_key_ids = [123456]
```
The key can also be uploaded in the Hetzner Console under **Security > SSH Keys**. `hcloud ssh-key list` is the reliable way to obtain its numeric ID.
### SSH From A Dynamic Home IP
`ssh_source_cidrs` is required by the current firewall configuration. An empty list blocks SSH and prevents Ansible from connecting. A static address is not required: determine the current public IPv4 address and permit only that address with a `/32` suffix:
```sh
curl -4 https://icanhazip.com
```
For an address such as `198.51.100.25`, configure:
```hcl
ssh_source_cidrs = ["198.51.100.25/32"]
```
If the home address changes, update this value and apply the OpenTofu stack again. The Hetzner Console remains available for firewall recovery if the old address locks out SSH.
Allowing SSH from the entire internet is possible but not recommended:
```hcl
ssh_source_cidrs = ["0.0.0.0/0", "::/0"]
```
SSH keys still authenticate connections in that configuration, but the hosts receive unnecessary scanning and attack traffic. For stable private administration, use Tailscale or another VPN and remove public SSH after the VPN is operational.
### Object Storage Credentials
In the Hetzner Console, open **Object Storage > Access Keys** and generate an access key and secret. Save the secret immediately because it is displayed only once. Export both credentials for OpenTofu:
```sh
export TF_VAR_s3_access_key='replace-with-object-storage-access-key'
export TF_VAR_s3_secret_key='replace-with-object-storage-secret-key'
```
Object Storage endpoints use the selected location. For Nuremberg, configure:
```hcl
s3_endpoint = "https://nbg1.your-objectstorage.com"
s3_region = "nbg1"
```
The exact endpoint is also shown in the Object Storage access information in the Hetzner Console. The endpoint location and `s3_region` must match. The stack defaults its servers to `nbg1`, but the bucket may use another supported Object Storage location when both values are changed together.
OpenTofu creates the bucket; do not create it manually. Generate a hard-to-collide lowercase suffix:
```sh
openssl rand -hex 6
```
Use the result in a descriptive bucket name:
```hcl
bucket_name = "cole-todo-prod-backups-a83d109cf241"
```
Bucket names must be 3-63 characters, use lowercase letters, numbers, and hyphens, and start and end with a letter or number. If the provider reports that the name is taken, generate a new suffix. Never include credentials or other secrets in a bucket name.
Secret values are marked sensitive where the providers permit it, but OpenTofu state can still contain sensitive infrastructure data. Store production state in an encrypted, access-controlled backend.
## Infrastructure
The root stack uses one state. The environment directories split input examples for readability; they are not separate root modules or states.
```sh
cp infra/environments/prod/prod.tfvars.example infra/environments/prod/prod.tfvars
cp infra/environments/staging/staging.tfvars.example infra/environments/staging/staging.tfvars
```
When using the recommended `TF_VAR_*` environment variables, remove the `hcloud_token`, `s3_access_key`, and `s3_secret_key` placeholder assignments from `prod.tfvars`. Command-line variable files take precedence over environment variables, so leaving the placeholders would override the exported credentials.
Set the remaining values in `infra/environments/prod/prod.tfvars`:
```hcl
project_name = "todo"
network_zone = "eu-central"
server_location = "nbg1"
server_type = "cx22"
ssh_key_ids = [123456]
ssh_source_cidrs = ["198.51.100.25/32"]
prod_private_ip = "10.0.1.10"
s3_endpoint = "https://nbg1.your-objectstorage.com"
s3_region = "nbg1"
bucket_name = "cole-todo-prod-backups-a83d109cf241"
```
Keep the default staging private address in `infra/environments/staging/staging.tfvars` unless it conflicts with another resource:
```hcl
staging_private_ip = "10.0.1.20"
```
Initialize and plan:
```sh
tofu -chdir=infra init
tofu -chdir=infra plan \
-var-file=environments/prod/prod.tfvars \
-var-file=environments/staging/staging.tfvars
tofu -chdir=infra apply \
-var-file=environments/prod/prod.tfvars \
-var-file=environments/staging/staging.tfvars
```
The AWS provider is configured for the Hetzner S3-compatible endpoint with path-style requests. The `hcloud` provider does not create Object Storage buckets.
The load balancer is provisioned immediately and health-checks `/healthz`. HTTPS termination and DNS are intentionally external operational concerns; add a certificate service to the LB before exposing sensitive traffic on the public internet.
## Configuration And Deployment
After a successful OpenTofu apply, generate the inventory directly from the provisioned server names and addresses:
```sh
make generate-inventory
```
This writes `ansible/inventory/hosts.yml`, which is ignored by Git. The generated inventory contains only host names, public and private addresses, the SSH user, and environment labels. It does not contain credentials or database passwords. Inspect it with:
```sh
ANSIBLE_CONFIG=ansible/ansible.cfg \
ansible-inventory -i ansible/inventory/hosts.yml --graph
```
Create the shared non-secret Ansible configuration from its example. The destination is ignored by Git:
```sh
cp ansible/group_vars/all/main.yml.example ansible/group_vars/all/main.yml
```
Create one shared Vault for Object Storage credentials and separate environment Vaults for database passwords:
```sh
ansible-vault create ansible/group_vars/all/vault.yml
ansible-vault create ansible/group_vars/prod/vault.yml
ansible-vault create ansible/group_vars/staging/vault.yml
```
Store the Object Storage credentials needed by WAL-G in `ansible/group_vars/all/vault.yml`:
```yaml
wal_g_access_key: "replace-with-object-storage-access-key"
wal_g_secret_key: "replace-with-object-storage-secret-key"
```
Store only the production database password in `ansible/group_vars/prod/vault.yml`:
```yaml
app_db_password: "replace-with-production-database-password"
```
Store a distinct staging password in `ansible/group_vars/staging/vault.yml`:
```yaml
app_db_password: "replace-with-distinct-staging-database-password"
```
Configure the matching non-secret values in `ansible/group_vars/all/main.yml`:
```yaml
wal_g_s3_prefix: s3://cole-todo-prod-backups-a83d109cf241/prod
wal_g_s3_endpoint: https://nbg1.your-objectstorage.com
wal_g_s3_region: nbg1
```
The bucket name must match `bucket_name` from OpenTofu, and `wal_g_s3_prefix` must end in `/prod`. To run Ansible from the repository root, set `ANSIBLE_CONFIG` explicitly so Ansible can find the project roles:
```sh
ANSIBLE_CONFIG=ansible/ansible.cfg \
ansible-playbook \
-i ansible/inventory/hosts.yml \
ansible/playbooks/setup-node.yml \
--ask-vault-pass
```
Alternatively, run from the `ansible` directory:
```sh
cd ansible
ansible-playbook -i inventory/hosts.yml playbooks/setup-node.yml --ask-vault-pass
```
The Make deployment target loads `ansible/ansible.cfg` by changing into that directory and regenerates inventory before deployment. Configure `ANSIBLE_VAULT_PASSWORD_FILE` so both playbooks can unlock the Vault files non-interactively:
```sh
export ANSIBLE_VAULT_PASSWORD_FILE="$HOME/.config/ansible/todo-vault-password"
make deploy-ansible
```
Build and push the image, then run configuration in this order:
```sh
docker build -t registry.example.com/todo:2026-08-06 app
docker push registry.example.com/todo:2026-08-06
make deploy-ansible
```
`setup-node.yml` installs and configures the host. `deploy-app.yml` optionally authenticates to a registry, pulls the image, reloads the user systemd manager, restarts `app-todo.service`, and checks health. Omit registry variables for a public image. Run `make generate-inventory` again after OpenTofu replaces a server or changes a public address.
## Backup Operations
Inspect production backup status:
```sh
systemctl status wal-g-backup.timer
systemctl status wal-g-backup.service
journalctl -u wal-g-backup.service
sudo -u postgres /usr/local/sbin/wal-g-postgres backup-list
```
The hourly service runs `backup-push` and retains the newest configured number of full backups with their dependent WAL. Object Storage bucket versioning is enabled, and incomplete multipart uploads expire after seven days.
Trigger an immediate production backup:
```sh
systemctl start wal-g-backup.service
```
## Staging Restore Operations
Inspect or trigger staging synchronization:
```sh
systemctl list-timers db-sync.timer
systemctl start db-sync.service
journalctl -u db-sync.service
```
The restore script refuses to delete any directory except `/var/lib/pgsql/data`, requires an initialized PostgreSQL directory before cleanup, and requires a production WAL-G prefix. It stops the user application and PostgreSQL, clears `PGDATA`, fetches `LATEST`, configures WAL recovery, waits for recovery to complete, resets the application role password, and starts the app only after success.
If restore fails, the app remains stopped. Diagnose and rerun `db-sync.service`; do not manually point staging at a different data directory.
## Important Paths
- Application entrypoint: `app/cmd/server/main.go`
- HTTP handlers and tests: `app/internal/handler/`
- Embedded templates and CSS: `app/web/`
- Shared OpenTofu stack: `infra/main.tf`
- Generated inventory output: `infra/outputs.tf`
- Reusable modules: `infra/modules/compute/`, `infra/modules/storage/`
- Node setup: `ansible/playbooks/setup-node.yml`
- Rootless Quadlet: `ansible/roles/app/templates/app-todo.container.j2`
- Production backup units: `ansible/roles/wal_g/templates/wal-g-backup.*`
- Guarded staging restore: `ansible/roles/wal_g/templates/db-sync.sh.j2`