elastic-arm-0b1d6a93: building pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv initdb: warning: enabling "trust" authentication for local connections initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. waiting for server to start.... done server started 2026-10-08T03:13:34.349665Z INFO horae: Running migrations... 2026-10-08T03:13:34.709226Z INFO horae: Migrations complete. 2026-10-08T03:13:34.741640Z INFO horae::seed: Seeding demo data… 2026-10-08T03:13:34.830269Z INFO horae::seed: Seed complete. 2026-10-08T03:13:34.842200Z INFO horae::seed: time_entries: 10 rows 2026-10-08T03:13:34.842834Z INFO horae::seed: clients: 2 rows PASS: header actions, supported metrics and labelled native budget progress PASS: design heading, compact row typography and inline project type PASS: filtered empty state resets search, scope and client PASS: narrow/short layouts, keyboard row actions and export formats PASS: loading spend is unavailable until the read completes PASS: failed spend is unavailable; retry restores real amounts PASS: configured monthly consumption replaces lifetime spend only after loading PASS: keyboard-accessible scope breakdown exposes overruns without layout overflow PASS: failed configured budget never falls back to lifetime figures locator.ariaSnapshot: Timeout 30000ms exceeded. Call log: - waiting for locator('body') at /nix/store/zn2f4srw60ixzmxqx0dgar80gvbmzry4-browser/projects-design.cjs:321:90 { name: 'TimeoutError' } 2026-10-08T03:13:34.953174Z WARN horae: DEV_LOGIN=1: /auth/dev-login signs anyone in as an admin without a password. Use it for local development only. 2026-10-08T03:14:11.698437Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.03421695s elapsed_secs=1.03421695 slow_threshold=1s 2026-10-08T03:14:21.175626Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.095349979s elapsed_secs=1.095349979 slow_threshold=1s 2026-10-08T03:14:21.641646Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=3.106311946s elapsed_secs=3.106311946 slow_threshold=1s 2026-10-08T03:14:24.182821Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=2.004892815s elapsed_secs=2.004892815 slow_threshold=1s 2026-10-08T03:14:26.507257Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.507582934s elapsed_secs=1.507582934 slow_threshold=1s 2026-10-08T03:14:27.851323Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.560586096s elapsed_secs=1.560586096 slow_threshold=1s 2026-10-08T03:14:31.562088Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.320009263s elapsed_secs=1.320009263 slow_threshold=1s 2026-10-08T03:14:32.185474Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=2.668447913s elapsed_secs=2.668447913 slow_threshold=1s 2026-10-08T03:14:35.076457Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.67342364s elapsed_secs=1.67342364 slow_threshold=1s 2026-10-08T03:14:35.754819Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.639186822s elapsed_secs=1.639186822 slow_threshold=1s 2026-10-08T03:14:39.543512Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=1.12329726s elapsed_secs=1.12329726 slow_threshold=1s 2026-10-08T03:14:40.723281Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.787434467s elapsed_secs=1.787434467 slow_threshold=1s 2026-10-08T03:14:43.282986Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.427356845s elapsed_secs=1.427356845 slow_threshold=1s 2026-10-08T03:14:44.809936Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.550733606s elapsed_secs=1.550733606 slow_threshold=1s 2026-10-08T03:14:45.747916Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=10.028326278s elapsed_secs=10.028326278 slow_threshold=1s 2026-10-08T03:14:49.596880Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.156545854s elapsed_secs=2.156545854 slow_threshold=1s 2026-10-08T03:14:50.575286Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.663449826s elapsed_secs=1.663449826 slow_threshold=1s 2026-10-08T03:14:50.819215Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT ps.project_id, ps.org_id FROM …" db.statement="\n\nSELECT ps.project_id, ps.org_id FROM project_settings ps\n JOIN projects p ON p.id = ps.project_id AND p.org_id = ps.org_id\n WHERE ps.alert_enabled AND p.active AND ps.project_id > $1\n ORDER BY ps.project_id LIMIT 100\n" rows_affected=0 rows_returned=0 elapsed=4.559051529s elapsed_secs=4.5590515289999995 slow_threshold=1s 2026-10-08T03:14:53.477254Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.411615434s elapsed_secs=1.411615434 slow_threshold=1s 2026-10-08T03:14:56.854646Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.00133484s elapsed_secs=1.00133484 slow_threshold=1s 2026-10-08T03:15:02.968388Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=3.50546767s elapsed_secs=3.50546767 slow_threshold=1s 2026-10-08T03:15:09.757751Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=14.30244938s elapsed_secs=14.30244938 slow_threshold=1s 2026-10-08T03:15:20.284501Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=3.050003098s elapsed_secs=3.050003098 slow_threshold=1s 2026-10-08T03:15:20.346264Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=14.058526646s elapsed_secs=14.058526646 slow_threshold=1s 2026-10-08 03:13:34.021 UTC [18] LOG: starting PostgreSQL 17.10 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit 2026-10-08 03:13:34.024 UTC [18] LOG: listening on Unix socket "/tmp/horae-browser.2IqbMz/.s.PGSQL.5432" 2026-10-08 03:13:34.032 UTC [21] LOG: database system was shut down at 2026-10-08 03:13:33 UTC 2026-10-08 03:13:34.041 UTC [18] LOG: database system is ready to accept connections 2026-10-08 03:15:28.491 UTC [18] LOG: received fast shutdown request 2026-10-08 03:15:28.499 UTC [18] LOG: aborting any active transactions 2026-10-08 03:15:28.508 UTC [18] LOG: background worker "logical replication launcher" (PID 24) exited with exit code 1 2026-10-08 03:15:28.516 UTC [19] LOG: shutting down 2026-10-08 03:15:28.517 UTC [19] LOG: checkpoint starting: shutdown immediate 2026-10-08 03:15:28.625 UTC [19] LOG: checkpoint complete: wrote 1150 buffers (7.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.098 s, sync=0.001 s, total=0.110 s; sync files=0, longest=0.000 s, average=0.000 s; distance=5994 kB, estimate=5994 kB; lsn=0/1AC71C0, redo lsn=0/1AC71C0 2026-10-08 03:15:28.636 UTC [18] LOG: database system is shut down