OwnStack

Database auth: password authentication failed

Three things have to agree for a deploy to authenticate against Postgres: the app's DATABASE_URL, dokku's stored service DSN, and the actual password of the postgres role inside the container. When one drifts, deploys fail with password authentication failed for user "postgres".

First, run the diagnostic

$ ownstack db check --app=<app>

Look at the JSON it returns. The relevant fields:

FieldMeaning
password_matchWhether DATABASE_URL's password equals the dokku-stored service DSN's password.
auth_okWhether a real connection attempt succeeds (note: this is unreliable due to docker-socket permissions on some hosts; treat as advisory only).
app_linked_to_serviceWhether dokku has the app linked to the postgres service.
database_url_hostWhat hostname DATABASE_URL is pointing at.

Pick the right fix based on the symptom

Right after a database import (pg_dumpall or similar)

Fix: Repair after import. The dump's ALTER ROLE overrode the role's password; DATABASE_URL still has dokku's original. Pipe one ALTER ROLE via postgres:connect to align the role to DATABASE_URL. Don't run ownstack db repair — it aligns to the wrong target.

password_match: false reported by db check

DATABASE_URL has drifted from the service DSN. Two paths:

  1. Same password on both sides: run ownstack db repair. It re-links and refreshes DATABASE_URL.
  2. Different password on both sides: figure out which one's right (check the dashboard's audit log for the most recent change), align via ALTER ROLE if needed.

Multiple DOKKU_POSTGRES_*_URL entries on config:show

Dokku links from all linked services on each rebuild and may pick the wrong one to set DATABASE_URL. Unlink the wrong service:

$ ssh dokku@<stack-ip> config:show <app> | grep DOKKU_POSTGRES
DOKKU_POSTGRES_AQUA_URL=postgres://…@dokku-postgres-app-db:5432/...
DOKKU_POSTGRES_BLACK_URL=postgres://…@dokku-postgres-postgres-app:5432/...   # ← wrong service

$ ssh dokku@<stack-ip> postgres:unlink postgres-app <app>
$ ssh dokku@<stack-ip> config:set --no-restart <app> \
    DATABASE_URL='postgres://…@dokku-postgres-app-db:5432/...'
$ ssh dokku@<stack-ip> config:unset --no-restart <app> DOKKU_POSTGRES_BLACK_URL
$ ssh dokku@<stack-ip> ps:rebuild <app>

Suddenly broken after a control-plane upgrade

Roll back the control-plane deploy if it landed recently. Past incidents have rewritten DATABASE_URL incorrectly when control-plane code changed how the postgres link was managed. See control-plane#289.

How to never see this in the first place

  • Use pg_dump (per-database) rather than pg_dumpall (cluster-wide) when restoring. pg_dump output doesn't include role globals.
  • Don't manually postgres:reset-password on a service that's already linked to an app — the app won't see the new password.
  • If you have multiple postgres services on one stack, link only one to each app. Multiple links create rebuild-time ambiguity.