Todo on Hetzner

This repository deploys a server-rendered Go todo application and its supporting infrastructure to Hetzner Cloud. A shared OpenTofu state owns the private network, production and staging Rocky Linux 10 servers, static primary IPs, firewall, Object Storage bucket, and HTTP load balancer. Ansible configures host PostgreSQL, rootless Podman, WAL-G backups, and scheduled staging restores.

Staging contains an unchanged copy of production todo data. Treat staging hosts, logs, database access, and backups as production-sensitive.

Architecture

flowchart TB
    client([HTTP clients])

    subgraph cloud[Hetzner Cloud]
        lb[Hetzner Load Balancer]

        subgraph network[Private Network 10.0.0.0/16]
            subgraph prod[Production VM - Rocky Linux 10]
                prod_app[Go App<br/>Rootless Podman]
                prod_db[(PostgreSQL<br/>127.0.0.1:5432)]
                prod_app -->|Local connection| prod_db
            end

            subgraph staging[Staging VM - Rocky Linux 10]
                staging_app[Go App<br/>Rootless Podman]
                staging_db[(PostgreSQL<br/>127.0.0.1:5432)]
                staging_app -->|Local connection| staging_db
            end
        end

        storage[(Hetzner Object Storage<br/>S3-compatible bucket)]
    end

    client -->|Port 80| lb
    lb -->|10.0.1.10:8080| prod_app
    prod_db -->|WAL archive and hourly base backup| storage
    storage -->|Restore LATEST at 06:00 and 12:00| staging_db

    classDef production fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d,stroke-width:2px;
    classDef stagingEnv fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a,stroke-width:2px;
    class prod_app,prod_db production;
    class staging_app,staging_db stagingEnv;
    style prod fill:#fff1f2,stroke:#b91c1c,stroke-width:2px
    style staging fill:#eff6ff,stroke:#1d4ed8,stroke-width:2px

• The load balancer accepts HTTP on port 80 and reaches production on 10.0.1.10:8080 over the private network.
• The Go application runs as the rootless todo user through Quadlet-generated app-todo.service with host networking.
• PostgreSQL listens only on 127.0.0.1:5432; the cloud firewall does not expose it.
• Production continuously archives WAL and performs an hourly base backup to s3://<bucket>/prod.
• Staging restores LATEST at 06:00 and 12:00 in America/Chicago, then resets the application role to the staging-specific password.
• Staging PostgreSQL has archiving disabled, and its WAL-G wrapper rejects wal-push.

Prerequisites

• Go 1.22 or newer
• OpenTofu 1.8 or newer
• Ansible Core 2.16 or newer
• Docker or Podman for image builds
• curl and make
• A Hetzner Cloud project, API token, SSH key, and Object Storage credentials
• A container registry account only if replacing the public application image with a private image

The app intentionally uses xorm.io/xorm with github.com/lib/pq, which is the PostgreSQL driver supported by xorm in this stack.

Local Development

Run a local PostgreSQL database and export a connection URL:

export DATABASE_URL='postgres://todo_app:local-password@127.0.0.1:5432/todo?sslmode=disable'
make css
make build-app
./app/bin/todo

The server listens on :8080 by default. Override it with APP_ADDR. The app runs the additive todos table migration at startup.

Useful checks:

make test
make vet
make format
make validate
docker build -t todo:local app

make css downloads the pinned standalone Tailwind CLI to app/tailwindcss. The Docker build independently downloads the CLI before compiling CSS, so it does not depend on a pre-existing generated stylesheet.

Published Container Image

GitHub Releases publish the application image to GitHub Container Registry for Linux AMD64 and ARM64 hosts:

docker pull ghcr.io/cole-maxwell1/todo-hetzner:latest

The latest tag tracks the newest stable release and is the convenient default for this demonstration stack. Version tags are safer for reproducible deployments. Release v1.2.3, for example, publishes v1.2.3, 1.2.3, 1.2, and 1; prereleases publish only versioned tags. Pin the registry digest when the deployed bytes must be immutable:

app_image: ghcr.io/cole-maxwell1/todo-hetzner@sha256:replace-with-published-digest

After the initial visibility step below, the package is public and does not require registry credentials to pull. A fork can change app_image in ansible/inventory/group_vars/all/main.yml to use another public or private image.

Continuous Integration And Releases

The Go tests workflow runs for pull requests targeting main and pushes to main when files under app/ or the workflow itself change. It starts PostgreSQL, runs every Go test including TestStoreLifecycle, and runs go vet. Infrastructure-only changes do not trigger the workflow.

Configure a branch protection rule for main that requires pull requests. Do not make the path-filtered test job a globally required status check: GitHub leaves that check pending when an infrastructure-only pull request does not trigger the workflow. The repository cannot define branch protection in workflow YAML.

Published GitHub Releases build and publish the container. Release tags must use semantic versioning with a v prefix. Create a release after its commit is merged and CI succeeds:

git tag -a v0.1.0 -m "v0.1.0"
git push origin v0.1.0
gh release create v0.1.0 --verify-tag --generate-notes

The release workflow checks out the tagged commit, reruns tests and vet with PostgreSQL, builds the multi-architecture image, publishes its tags, and records a provenance attestation. Draft releases do not publish images. Stable releases update latest; prereleases do not.

GHCR creates the package as private on its first publication. After the first release workflow succeeds, open the package settings, change its visibility to public, and verify that an unauthenticated pull succeeds. This is a one-time repository-owner operation; package visibility cannot be declared by this repository's workflow files.

Hetzner Credentials And Access

The stack needs two independent Hetzner credentials:

• A Hetzner Cloud API token for servers, networks, firewalls, primary IPs, and the load balancer.
• An Object Storage access key and secret for the S3 bucket and WAL-G.

The Cloud API token does not grant Object Storage access.

Cloud API Token

Open the Hetzner Console, select the Cloud project, go to Security > API Tokens, and create a Read & Write token. Export it as an OpenTofu input:

export TF_VAR_hcloud_token='replace-with-cloud-api-token'

Do not place the token in a tracked file or pass it directly on a command line that is retained in shell history.

SSH Key And ID

Hetzner installs the selected public SSH key on each server. Only upload the public .pub file; never upload the corresponding private key.

Generate an Ed25519 key if one does not already exist:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519

Install the Hetzner CLI and expose the Cloud token to it, and upload the public key:

brew install hcloud
export HCLOUD_TOKEN="$TF_VAR_hcloud_token"
hcloud ssh-key create \
  --name yourkey \
  --public-key-from-file ~/.ssh/id_ed25519.pub

List the project keys and copy the numeric ID from the ID column:

hcloud ssh-key list

Ex:

❯ hcloud ssh-key list

ID          NAME       FINGERPRINT                                       AGE
116588791   yourkey   44:36:cf:03:cd:6c:bb:24:84:4c:38:d5:83:ca:99:74   12h

Configure that value in prod.tfvars:

ssh_key_ids = [116588791]

The key can also be uploaded in the Hetzner Console under Security > SSH Keys. hcloud ssh-key list is the reliable way to obtain its numeric ID.

SSH From A Dynamic Home IP

ssh_source_cidrs is required by the current firewall configuration. An empty list blocks SSH and prevents Ansible from connecting. A static address is not required: determine the current public IPv4 address and permit only that address with a /32 suffix:

curl -4 https://icanhazip.com

For an address such as 198.51.100.25, configure:

ssh_source_cidrs = ["198.51.100.25/32"]

If the home address changes, update this value and apply the OpenTofu stack again. The Hetzner Console remains available for firewall recovery if the old address locks out SSH.

Allowing SSH from the entire internet is possible but not recommended:

ssh_source_cidrs = ["0.0.0.0/0", "::/0"]

SSH keys still authenticate connections in that configuration, but the hosts receive unnecessary scanning and attack traffic. For stable private administration, use Tailscale or another VPN and remove public SSH after the VPN is operational.

Object Storage Credentials

In the Hetzner Console, open Object Storage > Access Keys and generate an access key and secret. Save the secret immediately because it is displayed only once. Export both credentials for OpenTofu:

export TF_VAR_s3_access_key='replace-with-object-storage-access-key'
export TF_VAR_s3_secret_key='replace-with-object-storage-secret-key'

Object Storage endpoints use the selected location. For Nuremberg, configure:

s3_endpoint = "https://nbg1.your-objectstorage.com"
s3_region   = "nbg1"

The exact endpoint is also shown in the Object Storage access information in the Hetzner Console. The endpoint location and s3_region must match. The stack defaults its servers to nbg1, but the bucket may use another supported Object Storage location when both values are changed together.

OpenTofu creates the bucket; do not create it manually. Generate a hard-to-collide lowercase suffix:

openssl rand -hex 6

Use the result in a descriptive bucket name:

bucket_name = "todo-test-prod-backups-a83d109cf241"

Bucket names must be 3-63 characters, use lowercase letters, numbers, and hyphens, and start and end with a letter or number. If the provider reports that the name is taken, generate a new suffix. Never include credentials or other secrets in a bucket name.

Secret values are marked sensitive where the providers permit it, but OpenTofu state can still contain sensitive infrastructure data. Store production state in an encrypted, access-controlled backend.

Infrastructure

The root stack uses one state. The environment directories split input examples for readability; they are not separate root modules or states.

cp infra/environments/prod/prod.tfvars.example infra/environments/prod/prod.tfvars
cp infra/environments/staging/staging.tfvars.example infra/environments/staging/staging.tfvars

When using the recommended TF_VAR_* environment variables, remove the hcloud_token, s3_access_key, and s3_secret_key placeholder assignments from prod.tfvars. Command-line variable files take precedence over environment variables, so leaving the placeholders would override the exported credentials.

Set the remaining values in infra/environments/prod/prod.tfvars:

project_name     = "todo"
network_zone     = "eu-central"
server_location  = "nbg1"
server_type      = "cx22"
ssh_key_ids      = [116588791]
ssh_source_cidrs = ["198.51.100.25/32"]
prod_private_ip  = "10.0.1.10"

s3_endpoint = "https://nbg1.your-objectstorage.com"
s3_region   = "nbg1"
bucket_name = "todo-test-prod-backups-a83d109cf241"

Keep the default staging private address in infra/environments/staging/staging.tfvars unless it conflicts with another resource:

staging_private_ip = "10.0.1.20"

Initialize and plan:

tofu -chdir=infra init
tofu -chdir=infra plan \
  -var-file=environments/prod/prod.tfvars \
  -var-file=environments/staging/staging.tfvars
tofu -chdir=infra apply \
  -var-file=environments/prod/prod.tfvars \
  -var-file=environments/staging/staging.tfvars

The AWS provider is configured for the Hetzner S3-compatible endpoint with path-style requests. The hcloud provider does not create Object Storage buckets.

The load balancer is provisioned immediately and health-checks /healthz. HTTPS termination and DNS are intentionally external operational concerns; add a certificate service to the LB before exposing sensitive traffic on the public internet.

Configuration And Deployment

After a successful OpenTofu apply, generate the inventory directly from the provisioned server names and addresses:

make generate-inventory

This writes ansible/inventory/hosts.yml, which is ignored by Git. The generated inventory contains only host names, public and private addresses, the SSH user, and environment labels. It does not contain credentials or database passwords. Inspect it with:

ANSIBLE_CONFIG=ansible/ansible.cfg \
ansible-inventory -i ansible/inventory/hosts.yml --graph

Create the shared non-secret Ansible configuration from its example. The destination is ignored by Git:

cp ansible/inventory/group_vars/all/main.yml.example ansible/inventory/group_vars/all/main.yml

Create one shared Vault for Object Storage credentials and separate environment Vaults for database passwords:

ansible-vault create ansible/inventory/group_vars/all/vault.yml
ansible-vault create ansible/inventory/group_vars/prod/vault.yml
ansible-vault create ansible/inventory/group_vars/staging/vault.yml

Store the Object Storage credentials needed by WAL-G in ansible/inventory/group_vars/all/vault.yml:

wal_g_access_key: "replace-with-object-storage-access-key"
wal_g_secret_key: "replace-with-object-storage-secret-key"

Store only the production database password in ansible/inventory/group_vars/prod/vault.yml:

app_db_password: "replace-with-production-database-password"

Store a distinct staging password in ansible/inventory/group_vars/staging/vault.yml:

app_db_password: "replace-with-distinct-staging-database-password"

Configure the matching non-secret values in ansible/inventory/group_vars/all/main.yml:

wal_g_s3_prefix: s3://cole-todo-prod-backups-a83d109cf241/prod
wal_g_s3_endpoint: https://nbg1.your-objectstorage.com
wal_g_s3_region: nbg1

The bucket name must match bucket_name from OpenTofu, and wal_g_s3_prefix must end in /prod. To run Ansible from the repository root, set ANSIBLE_CONFIG explicitly so Ansible can find the project roles:

ANSIBLE_CONFIG=ansible/ansible.cfg \
ansible-playbook \
  -i ansible/inventory/hosts.yml \
  ansible/playbooks/setup-node.yml \
  --ask-vault-pass

Alternatively, run from the ansible directory:

cd ansible
ansible-playbook -i inventory/hosts.yml playbooks/setup-node.yml --ask-vault-pass

For non-interactive deployments, Ansible needs a file containing the password used to encrypt all three Vault files. The project configuration defaults to ansible/.vault_pass.txt; create it from the example, replace the placeholder with the Vault password, and restrict its permissions:

cp ansible/.vault_pass.txt.example ansible/.vault_pass.txt
chmod 600 ansible/.vault_pass.txt

ansible/.vault_pass.txt is ignored by Git and must never be committed. All three Vault files must use the password stored in this file. To keep the password outside the repository instead, set ANSIBLE_VAULT_PASSWORD_FILE:

export ANSIBLE_VAULT_PASSWORD_FILE="$HOME/.config/ansible/todo-vault-password"

The Make deployment target loads ansible/ansible.cfg, regenerates inventory, and unlocks the Vault files non-interactively:

make deploy-ansible

The default Ansible configuration pulls the public latest image. Run configuration in this order:

docker pull ghcr.io/cole-maxwell1/todo-hetzner:latest
make deploy-ansible

The explicit local pull is optional and only verifies public access from the operator's machine. setup-node.yml installs and configures the host. deploy-app.yml optionally authenticates to a private registry, pulls app_image on each host, reloads the user systemd manager, restarts app-todo.service, and checks health. Leave the registry variables unset for the default public image. Run make generate-inventory again after OpenTofu replaces a server or changes a public address.

Backup Operations

Inspect production backup status:

systemctl status wal-g-backup.timer
systemctl status wal-g-backup.service
journalctl -u wal-g-backup.service
sudo -u postgres /usr/local/sbin/wal-g-postgres backup-list

The hourly service runs backup-push and retains the newest configured number of full backups with their dependent WAL. Object Storage bucket versioning is enabled, and incomplete multipart uploads expire after seven days.

Trigger an immediate production backup:

systemctl start wal-g-backup.service

Staging Restore Operations

Inspect or trigger staging synchronization:

systemctl list-timers db-sync.timer
systemctl start db-sync.service
journalctl -u db-sync.service

The restore script refuses to delete any directory except /var/lib/pgsql/data, requires an initialized PostgreSQL directory before cleanup, and requires a production WAL-G prefix. It stops the user application and PostgreSQL, clears PGDATA, fetches LATEST, configures WAL recovery, waits for recovery to complete, resets the application role password, and starts the app only after success.

If restore fails, the app remains stopped. Diagnose and rerun db-sync.service; do not manually point staging at a different data directory.

Important Paths

• Application entrypoint: app/cmd/server/main.go
• HTTP handlers and tests: app/internal/handler/
• Embedded templates and CSS: app/web/
• Shared OpenTofu stack: infra/main.tf
• Generated inventory output: infra/outputs.tf
• Reusable modules: infra/modules/compute/, infra/modules/storage/
• Node setup: ansible/playbooks/setup-node.yml
• Rootless Quadlet: ansible/roles/app/templates/app-todo.container.j2
• Production backup units: ansible/roles/wal_g/templates/wal-g-backup.*
• Guarded staging restore: ansible/roles/wal_g/templates/db-sync.sh.j2