this derivation will be built:
/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv
building '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv' on 'grpc://farm.numtide.com:50051?client-cert=/run/secrets/vars/per-machine/bld1/nix-farm-ci/crt&client-key=/run/secrets/vars/per-machine/bld1/nix-farm-ci/key&connect-timeout=20&system=aarch64-linux'
building '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv' on farm.numtide.com:50051
horae-browser-checks> elastic-arm-0b1d6a93: building pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv
horae-browser-checks> initdb: warning: enabling "trust" authentication for local connections
horae-browser-checks> initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
horae-browser-checks> waiting for server to start.... done
horae-browser-checks> server started
horae-browser-checks> 2026-10-08T03:02:35.828954Z INFO horae: Running migrations...
horae-browser-checks> 2026-10-08T03:02:36.158412Z INFO horae: Migrations complete.
horae-browser-checks> 2026-10-08T03:02:36.202525Z INFO horae::seed: Seeding demo data…
horae-browser-checks> 2026-10-08T03:02:36.322078Z INFO horae::seed: Seed complete.
horae-browser-checks> 2026-10-08T03:02:36.337579Z INFO horae::seed: time_entries: 10 rows
horae-browser-checks> 2026-10-08T03:02:36.340584Z INFO horae::seed: clients: 2 rows
horae-browser-checks> PASS: header actions, supported metrics and labelled native budget progress
horae-browser-checks> PASS: design heading, compact row typography and inline project type
horae-browser-checks> PASS: filtered empty state resets search, scope and client
horae-browser-checks> PASS: narrow/short layouts, keyboard row actions and export formats
horae-browser-checks> PASS: loading spend is unavailable until the read completes
horae-browser-checks> PASS: failed spend is unavailable; retry restores real amounts
horae-browser-checks> PASS: configured monthly consumption replaces lifetime spend only after loading
horae-browser-checks> PASS: keyboard-accessible scope breakdown exposes overruns without layout overflow
horae-browser-checks> PASS: failed configured budget never falls back to lifetime figures
horae-browser-checks> PASS: empty-state guidance and action visibility for admin
horae-browser-checks> PASS: empty-state guidance and action visibility for manager
horae-browser-checks> PASS: empty-state guidance and action visibility for member
horae-browser-checks> PASS: shared Menu, Combobox, empty-state and navigation-icon defaults are unchanged
horae-browser-checks> PASS: /clients header at 320px
horae-browser-checks> PASS: /clients header at 390px
horae-browser-checks> PASS: /clients header at 768px
horae-browser-checks> PASS: /clients header at 1280px
horae-browser-checks> PASS: /clients header at 1440px
horae-browser-checks> PASS: /projects header at 320px
horae-browser-checks> PASS: /projects header at 390px
horae-browser-checks> PASS: /projects header at 768px
horae-browser-checks> PASS: /projects header at 1280px
horae-browser-checks> PASS: /projects header at 1440px
horae-browser-checks> PASS: /invoices header at 320px
horae-browser-checks> PASS: /invoices header at 390px
horae-browser-checks> PASS: /invoices header at 768px
horae-browser-checks> PASS: /invoices header at 1280px
horae-browser-checks> PASS: /invoices header at 1440px
horae-browser-checks> PASS: /reports header at 320px
horae-browser-checks> PASS: /reports header at 390px
horae-browser-checks> PASS: /reports header at 768px
horae-browser-checks> PASS: /reports header at 1280px
horae-browser-checks> PASS: /reports header at 1440px
horae-browser-checks> PASS: /admin/users header at 320px
horae-browser-checks> PASS: /admin/users header at 390px
horae-browser-checks> PASS: /admin/users header at 768px
horae-browser-checks> PASS: /admin/users header at 1280px
horae-browser-checks> PASS: /admin/users header at 1440px
horae-browser-checks> PASS: /approvals header at 320px
horae-browser-checks> PASS: /approvals header at 390px
horae-browser-checks> PASS: /approvals header at 768px
horae-browser-checks> PASS: /approvals header at 1280px
horae-browser-checks> PASS: /approvals header at 1440px
horae-browser-checks> PASS: /settings header at 320px
horae-browser-checks> PASS: /settings header at 390px
horae-browser-checks> PASS: /settings header at 768px
horae-browser-checks> PASS: /settings header at 1280px
horae-browser-checks> PASS: /settings header at 1440px
horae-browser-checks> PASS: /admin/importers header at 320px
horae-browser-checks> PASS: /admin/importers header at 390px
horae-browser-checks> PASS: /admin/importers header at 768px
horae-browser-checks> PASS: /admin/importers header at 1280px
horae-browser-checks> PASS: /admin/importers header at 1440px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 320px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 390px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 768px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 1280px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 1440px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 320px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 390px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 768px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 1280px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 1440px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 320px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 390px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 768px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 1280px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 1440px
horae-browser-checks> PASS: project names and badges stay within aligned columns; scrolling to last-row actions opens editing
horae-browser-checks> PASS: small-screen timesheet pager, date picker and view controls remain usable
horae-browser-checks> PASS: admin subnavigation and its content remain reachable on mobile
horae-browser-checks> PASS: day strip scrolls to Sunday and long entry labels do not cover actions
horae-browser-checks> CHECK: late popover initialization restores missing focus without overriding deliberate focus
horae-browser-checks> PASS: late popover initialization restores missing focus without overriding deliberate focus
horae-browser-checks> CHECK: content reflow preserves an open menu while deliberate scrolling still dismisses it
horae-browser-checks> PASS: content reflow preserves an open menu while deliberate scrolling still dismisses it
horae-browser-checks> CHECK: first and last project-row actions are initially visible and clickable
horae-browser-checks> PASS: first and last project-row actions are initially visible and clickable
horae-browser-checks> CHECK: menu keyboard, disabled items, outside click and repeated opening
horae-browser-checks> PASS: menu keyboard, disabled items, outside click and repeated opening
horae-browser-checks> CHECK: scroll and resize dismiss without leaving stale menus or focus
horae-browser-checks> PASS: scroll and resize dismiss without leaving stale menus or focus
horae-browser-checks> CHECK: short-screen menu scrolls internally and supports reverse Tab and Space
horae-browser-checks> PASS: short-screen menu scrolls internally and supports reverse Tab and Space
horae-browser-checks> CHECK: filter and calendar menus keep their callbacks and survive route changes
horae-browser-checks> PASS: filter and calendar menus keep their callbacks and survive route changes
horae-browser-checks> CHECK: Timesheet keeps the shared calendar and week selection behavior
horae-browser-checks> PASS: Timesheet keeps the shared calendar and week selection behavior
horae-browser-checks> PASS: mobile content width, keyboard toggle, Escape/focus restoration and navigation close
horae-browser-checks> PASS: short mobile viewport reaches account navigation and preserves desktop collapse state
horae-browser-checks> PASS: desktop drag width survives mobile and breakpoint transitions
horae-browser-checks> PASS: keyboard selection, mixed/select-all states and Actions count
horae-browser-checks> PASS: search, status and client changes clear selection
horae-browser-checks> PASS: keyboard selection/confirmation and no document overflow in both themes at three widths
horae-browser-checks> PASS: cancel/focus, failure retry, pending guard and archive/reactivate confirmation
horae-browser-checks> PASS: oversized selection cannot submit and members have no bulk controls
horae-browser-checks> PASS: real atomic archive/reactivate, anonymous/member/inactive denial, manager/admin and safe retries
horae-browser-checks> PASS: bulk success restores focus and recovers the list without stale actions
horae-browser-checks> PASS: bulk pending restores focus and recovers the list without stale actions
horae-browser-checks> PASS: bulk failure restores focus and recovers the list without stale actions
horae-browser-checks> PASS: client activation failure remains visible with form closed, open and cancelled
horae-browser-checks> PASS: successful form save preserves action error; real status retry clears it and refreshes the row
horae-browser-checks> PASS: archive failure survives menu dismissal; shared edit failure preserves input and safely retries
horae-browser-checks> PASS: assignment removal failure remains visible with form closed, open and cancelled
horae-browser-checks> PASS: initial catalog/draft failures show loading and recovery without rendering or saving an empty replacement
horae-browser-checks> PASS: design typography, section spacing, quiet dividers and theme-aware task information
horae-browser-checks> PASS: project types retain native keyboard selection and the designed icon/option-card states
horae-browser-checks> PASS: searchable client selection, pending/error recovery and explicit/inherited currency
horae-browser-checks> PASS: client default-rate currency rejects relabelling, preserves failed input and permits compatible edits
horae-browser-checks> PASS: task and teammate searches handle pending results, keyboard selection, duplicates and retry
horae-browser-checks> PASS: real client, team, task restrictions, billing and invoice defaults survive reload
horae-browser-checks> PASS: all invoice-default errors identify and focus their field, preserve input and recover at three widths
horae-browser-checks> PASS: basic-field rejections retain values, link visible errors and restore focus at three widths
horae-browser-checks> PASS: failed resume and retry preserve the existing draft without a replacement write
horae-browser-checks> PASS: invalid hourly rates and hour/fee budgets retain input and focus their error at three widths
horae-browser-checks> PASS: a stale email configuration rejection can be corrected without sending mail
horae-browser-checks> PASS: task/person rate, cost and budget errors identify their row and preserve other inputs at three widths
horae-browser-checks> PASS: single/monthly fees, empty schedules and individual milestone errors recover without changing another row
horae-browser-checks> PASS: responsive form, conditional panels and Cancel preserve the draft
horae-browser-checks> PASS: edits made while a save is pending are saved after its acknowledgement
horae-browser-checks> PASS: lost acknowledgement retries the same request and preserves newer edits
horae-browser-checks> PASS: real finalized budget reaches the authorized Projects endpoint and display
horae-browser-checks> PASS: a new tagged project reaches reports and both exports; pending/error filters never show stale rows
horae-browser-checks> PASS: project invoice defaults prefill real estimates, stale reviews cannot generate, and editable draft values persist independently
horae-browser-checks> PASS: lost creation response is idempotent and clears the completed draft
horae-browser-checks> PASS: stale tabs cannot overwrite saved input; explicit discard removes only the draft
horae-browser-checks> PASS: invoice selection, partial balances, exact excess, dirty/pending navigation, reload recovery, account-scoped replay, storage failure safety and authorized per-occurrence project balances
horae-browser-checks> PASS: New Project task rates and editor capabilities preserve configured/legacy modes, source currencies, unknown/zero recovery, scoped budgets and keyboard recovery across three widths
horae-browser-checks> PASS: organization admin real-session UI, draft ownership, progress/export scope and financial redaction
horae-browser-checks> PASS: organization manager real-session UI, draft ownership, progress/export scope and financial redaction
horae-browser-checks> PASS: project lead real-session UI, draft ownership, progress/export scope and financial redaction
horae-browser-checks> PASS: project admin real-session UI, draft ownership, progress/export scope and financial redaction
horae-browser-checks> PASS: visible assigned member real-session UI, draft ownership, progress/export scope and financial redaction
horae-browser-checks> PASS: private assigned member real-session UI, draft ownership, progress/export scope and financial redaction
horae-browser-checks> PASS: revoked assignment and outsider identities cannot grant project-wide access
horae-browser-checks> PASS: inactive/anonymous requests are denied and a manager creates a real project without private overrides
horae-browser-checks> ExpectError: expect(locator).toContainText(expected) failed
horae-browser-checks>
horae-browser-checks> Locator: locator('.np-page').locator('header').getByRole('status')
horae-browser-checks> Expected substring: "Draft saved at"
horae-browser-checks> Received string: "Saving draft…"
horae-browser-checks> Timeout: 5000ms
horae-browser-checks>
horae-browser-checks> Call log:
horae-browser-checks> - Expect "to.have.text" with timeout 5000ms
horae-browser-checks> - waiting for locator('.np-page').locator('header').getByRole('status')
horae-browser-checks> 6 × locator resolved to Saving draft…
horae-browser-checks> - unexpected value "Saving draft…"
horae-browser-checks>
horae-browser-checks> at captureRawStack (/nix/store/9l763mr9j5rhgkjcwkgsig6vspvz3hc3-playwright-test-1.60.0/lib/node_modules/playwright-core/lib/coreBundle.js:3130:17)
horae-browser-checks> at callMatcherAsStep (/nix/store/9l763mr9j5rhgkjcwkgsig6vspvz3hc3-playwright-test-1.60.0/lib/node_modules/playwright/lib/matchers/expect.js:12873:57)
horae-browser-checks> at Object.toContainText (/nix/store/9l763mr9j5rhgkjcwkgsig6vspvz3hc3-playwright-test-1.60.0/lib/node_modules/playwright/lib/matchers/expect.js:12863:23)
horae-browser-checks> at saved (/nix/store/zn2f4srw60ixzmxqx0dgar80gvbmzry4-browser/new-project-task-errors.cjs:30:76)
horae-browser-checks> at /nix/store/zn2f4srw60ixzmxqx0dgar80gvbmzry4-browser/new-project-task-errors.cjs:89:15 {
horae-browser-checks> matcherResult: {
horae-browser-checks> name: 'toContainText',
horae-browser-checks> expected: 'Draft saved at',
horae-browser-checks> message: '\x1B[2mexpect(\x1B[22m\x1B[31mlocator\x1B[39m\x1B[2m).\x1B[22mtoContainText\x1B[2m(\x1B[22m\x1B[32mexpected\x1B[39m\x1B[2m)\x1B[22m failed\n' +
horae-browser-checks> '\n' +
horae-browser-checks> "Locator: locator('.np-page').locator('header').getByRole('status')\n" +
horae-browser-checks> 'Expected substring: \x1B[32m"Draft saved at"\x1B[39m\n' +
horae-browser-checks> 'Received string: \x1B[31m"Saving draft…"\x1B[39m\n' +
horae-browser-checks> 'Timeout: 5000ms\n' +
horae-browser-checks> '\n' +
horae-browser-checks> 'Call log:\n' +
horae-browser-checks> '\x1B[2m - Expect "to.have.text" with timeout 5000ms\x1B[22m\n' +
horae-browser-checks> "\x1B[2m - waiting for locator('.np-page').locator('header').getByRole('status')\x1B[22m\n" +
horae-browser-checks> '\x1B[2m 6 × locator resolved to Saving draft…
\x1B[22m\n' +
horae-browser-checks> '\x1B[2m - unexpected value "Saving draft…"\x1B[22m\n',
horae-browser-checks> pass: false,
horae-browser-checks> actual: 'Saving draft…',
horae-browser-checks> log: [
horae-browser-checks> ' - Expect "to.have.text" with timeout 5000ms',
horae-browser-checks> " - waiting for locator('.np-page').locator('header').getByRole('status')",
horae-browser-checks> ' 6 × locator resolved to Saving draft…
',
horae-browser-checks> ' - unexpected value "Saving draft…"'
horae-browser-checks> ],
horae-browser-checks> timeout: 5000,
horae-browser-checks> ariaSnapshot: '- status: Saving draft…'
horae-browser-checks> }
horae-browser-checks> }
horae-browser-checks> 2026-10-08T03:02:36.433031Z WARN horae: DEV_LOGIN=1: /auth/dev-login signs anyone in as an admin without a password. Use it for local development only.
horae-browser-checks> 2026-10-08T03:11:08.049389Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=3.148661811s elapsed_secs=3.148661811 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:08.427172Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.490407983s elapsed_secs=2.490407983 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:14.551181Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=3.526939136s elapsed_secs=3.526939136 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:19.569578Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=5.787000216s elapsed_secs=5.787000216 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:21.512098Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.765694457s elapsed_secs=2.765694457 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:25.526625Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.055278358s elapsed_secs=1.055278358 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:36.554556Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=9.30914075s elapsed_secs=9.30914075 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:50.967390Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=4.443481758s elapsed_secs=4.443481758 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:56.721278Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=32.333772236s elapsed_secs=32.333772236 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:11:56.762695Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=17.044340147s elapsed_secs=17.044340147 slow_threshold=1s
horae-browser-checks> 2026-10-08 03:02:35.608 UTC [18] LOG: starting PostgreSQL 17.10 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit
horae-browser-checks> 2026-10-08 03:02:35.608 UTC [18] LOG: listening on Unix socket "/tmp/horae-browser.qD0yvt/.s.PGSQL.5432"
horae-browser-checks> 2026-10-08 03:02:35.612 UTC [21] LOG: database system was shut down at 2026-10-08 03:02:35 UTC
horae-browser-checks> 2026-10-08 03:02:35.621 UTC [18] LOG: database system is ready to accept connections
horae-browser-checks> 2026-10-08 03:02:36.159 UTC [36] LOG: could not receive data from client: Connection reset by peer
horae-browser-checks> 2026-10-08 03:02:36.164 UTC [37] LOG: could not send data to client: Broken pipe
horae-browser-checks> 2026-10-08 03:02:36.164 UTC [37] FATAL: connection to client lost
horae-browser-checks> 2026-10-08 03:07:35.712 UTC [19] LOG: checkpoint starting: time
horae-browser-checks> 2026-10-08 03:09:35.707 UTC [19] LOG: checkpoint complete: wrote 1183 buffers (7.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=119.907 s, sync=0.001 s, total=119.994 s; sync files=0, longest=0.000 s, average=0.000 s; distance=6458 kB, estimate=6458 kB; lsn=0/1B78C20, redo lsn=0/1B3B2E0
horae-browser-checks> 2026-10-08 03:11:58.801 UTC [18] LOG: received fast shutdown request
horae-browser-checks> 2026-10-08 03:11:58.808 UTC [18] LOG: aborting any active transactions
horae-browser-checks> 2026-10-08 03:11:58.821 UTC [18] LOG: background worker "logical replication launcher" (PID 24) exited with exit code 1
horae-browser-checks> 2026-10-08 03:11:58.822 UTC [19] LOG: shutting down
horae-browser-checks> 2026-10-08 03:11:58.823 UTC [19] LOG: checkpoint starting: shutdown immediate
horae-browser-checks> 2026-10-08 03:11:58.905 UTC [19] LOG: checkpoint complete: wrote 156 buffers (1.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.070 s, sync=0.001 s, total=0.084 s; sync files=0, longest=0.000 s, average=0.000 s; distance=388 kB, estimate=5851 kB; lsn=0/1B9C508, redo lsn=0/1B9C508
horae-browser-checks> 2026-10-08 03:11:58.930 UTC [18] LOG: database system is shut down
error: build of '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv' on 'grpc://farm.numtide.com:50051?client-cert=/run/secrets/vars/per-machine/bld1/nix-farm-ci/crt&client-key=/run/secrets/vars/per-machine/bld1/nix-farm-ci/key&connect-timeout=20&system=aarch64-linux' failed: Cannot build '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/5jk1h8dvi5dvs5kjhs0hwn28abq7iwa2-horae-browser-checks
Last 25 log lines:
> 2026-10-08T03:11:08.427172Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.490407983s elapsed_secs=2.490407983 slow_threshold=1s
> 2026-10-08T03:11:14.551181Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=3.526939136s elapsed_secs=3.526939136 slow_threshold=1s
> 2026-10-08T03:11:19.569578Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=5.787000216s elapsed_secs=5.787000216 slow_threshold=1s
> 2026-10-08T03:11:21.512098Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.765694457s elapsed_secs=2.765694457 slow_threshold=1s
> 2026-10-08T03:11:25.526625Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.055278358s elapsed_secs=1.055278358 slow_threshold=1s
> 2026-10-08T03:11:36.554556Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=9.30914075s elapsed_secs=9.30914075 slow_threshold=1s
> 2026-10-08T03:11:50.967390Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=4.443481758s elapsed_secs=4.443481758 slow_threshold=1s
> 2026-10-08T03:11:56.721278Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=32.333772236s elapsed_secs=32.333772236 slow_threshold=1s
> 2026-10-08T03:11:56.762695Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=17.044340147s elapsed_secs=17.044340147 slow_threshold=1s
> 2026-10-08 03:02:35.608 UTC [18] LOG: starting PostgreSQL 17.10 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit
> 2026-10-08 03:02:35.608 UTC [18] LOG: listening on Unix socket "/tmp/horae-browser.qD0yvt/.s.PGSQL.5432"
> 2026-10-08 03:02:35.612 UTC [21] LOG: database system was shut down at 2026-10-08 03:02:35 UTC
> 2026-10-08 03:02:35.621 UTC [18] LOG: database system is ready to accept connections
> 2026-10-08 03:02:36.159 UTC [36] LOG: could not receive data from client: Connection reset by peer
> 2026-10-08 03:02:36.164 UTC [37] LOG: could not send data to client: Broken pipe
> 2026-10-08 03:02:36.164 UTC [37] FATAL: connection to client lost
> 2026-10-08 03:07:35.712 UTC [19] LOG: checkpoint starting: time
> 2026-10-08 03:09:35.707 UTC [19] LOG: checkpoint complete: wrote 1183 buffers (7.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=119.907 s, sync=0.001 s, total=119.994 s; sync files=0, longest=0.000 s, average=0.000 s; distance=6458 kB, estimate=6458 kB; lsn=0/1B78C20, redo lsn=0/1B3B2E0
> 2026-10-08 03:11:58.801 UTC [18] LOG: received fast shutdown request
> 2026-10-08 03:11:58.808 UTC [18] LOG: aborting any active transactions
> 2026-10-08 03:11:58.821 UTC [18] LOG: background worker "logical replication launcher" (PID 24) exited with exit code 1
> 2026-10-08 03:11:58.822 UTC [19] LOG: shutting down
> 2026-10-08 03:11:58.823 UTC [19] LOG: checkpoint starting: shutdown immediate
> 2026-10-08 03:11:58.905 UTC [19] LOG: checkpoint complete: wrote 156 buffers (1.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.070 s, sync=0.001 s, total=0.084 s; sync files=0, longest=0.000 s, average=0.000 s; distance=388 kB, estimate=5851 kB; lsn=0/1B9C508, redo lsn=0/1B9C508
> 2026-10-08 03:11:58.930 UTC [18] LOG: database system is shut down
For full logs, run:
nix log /nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv
error: Cannot build '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/5jk1h8dvi5dvs5kjhs0hwn28abq7iwa2-horae-browser-checks
nixbot: transient error detected, retrying once
this derivation will be built:
/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv
building '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv' on 'grpc://farm.numtide.com:50051?client-cert=/run/secrets/vars/per-machine/bld1/nix-farm-ci/crt&client-key=/run/secrets/vars/per-machine/bld1/nix-farm-ci/key&connect-timeout=21&system=aarch64-linux'
building '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv' on farm.numtide.com:50051
horae-browser-checks> elastic-arm-0b1d6a93: building pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv
horae-browser-checks> initdb: warning: enabling "trust" authentication for local connections
horae-browser-checks> initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
horae-browser-checks> waiting for server to start.... done
horae-browser-checks> server started
horae-browser-checks> 2026-10-08T03:13:34.349665Z INFO horae: Running migrations...
horae-browser-checks> 2026-10-08T03:13:34.709226Z INFO horae: Migrations complete.
horae-browser-checks> 2026-10-08T03:13:34.741640Z INFO horae::seed: Seeding demo data…
horae-browser-checks> 2026-10-08T03:13:34.830269Z INFO horae::seed: Seed complete.
horae-browser-checks> 2026-10-08T03:13:34.842200Z INFO horae::seed: time_entries: 10 rows
horae-browser-checks> 2026-10-08T03:13:34.842834Z INFO horae::seed: clients: 2 rows
horae-browser-checks> PASS: header actions, supported metrics and labelled native budget progress
horae-browser-checks> PASS: design heading, compact row typography and inline project type
horae-browser-checks> PASS: filtered empty state resets search, scope and client
horae-browser-checks> PASS: narrow/short layouts, keyboard row actions and export formats
horae-browser-checks> PASS: loading spend is unavailable until the read completes
horae-browser-checks> PASS: failed spend is unavailable; retry restores real amounts
horae-browser-checks> PASS: configured monthly consumption replaces lifetime spend only after loading
horae-browser-checks> PASS: keyboard-accessible scope breakdown exposes overruns without layout overflow
horae-browser-checks> PASS: failed configured budget never falls back to lifetime figures
horae-browser-checks> locator.ariaSnapshot: Timeout 30000ms exceeded.
horae-browser-checks> Call log:
horae-browser-checks> - waiting for locator('body')
horae-browser-checks>
horae-browser-checks> at /nix/store/zn2f4srw60ixzmxqx0dgar80gvbmzry4-browser/projects-design.cjs:321:90 {
horae-browser-checks> name: 'TimeoutError'
horae-browser-checks> }
horae-browser-checks> 2026-10-08T03:13:34.953174Z WARN horae: DEV_LOGIN=1: /auth/dev-login signs anyone in as an admin without a password. Use it for local development only.
horae-browser-checks> 2026-10-08T03:14:11.698437Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.03421695s elapsed_secs=1.03421695 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:21.175626Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.095349979s elapsed_secs=1.095349979 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:21.641646Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=3.106311946s elapsed_secs=3.106311946 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:24.182821Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=2.004892815s elapsed_secs=2.004892815 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:26.507257Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.507582934s elapsed_secs=1.507582934 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:27.851323Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.560586096s elapsed_secs=1.560586096 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:31.562088Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.320009263s elapsed_secs=1.320009263 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:32.185474Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=2.668447913s elapsed_secs=2.668447913 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:35.076457Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.67342364s elapsed_secs=1.67342364 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:35.754819Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.639186822s elapsed_secs=1.639186822 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:39.543512Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=1.12329726s elapsed_secs=1.12329726 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:40.723281Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.787434467s elapsed_secs=1.787434467 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:43.282986Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.427356845s elapsed_secs=1.427356845 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:44.809936Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.550733606s elapsed_secs=1.550733606 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:45.747916Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=10.028326278s elapsed_secs=10.028326278 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:49.596880Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.156545854s elapsed_secs=2.156545854 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:50.575286Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.663449826s elapsed_secs=1.663449826 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:50.819215Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT ps.project_id, ps.org_id FROM …" db.statement="\n\nSELECT ps.project_id, ps.org_id FROM project_settings ps\n JOIN projects p ON p.id = ps.project_id AND p.org_id = ps.org_id\n WHERE ps.alert_enabled AND p.active AND ps.project_id > $1\n ORDER BY ps.project_id LIMIT 100\n" rows_affected=0 rows_returned=0 elapsed=4.559051529s elapsed_secs=4.5590515289999995 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:53.477254Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.411615434s elapsed_secs=1.411615434 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:14:56.854646Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.00133484s elapsed_secs=1.00133484 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:15:02.968388Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=3.50546767s elapsed_secs=3.50546767 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:15:09.757751Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=14.30244938s elapsed_secs=14.30244938 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:15:20.284501Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=3.050003098s elapsed_secs=3.050003098 slow_threshold=1s
horae-browser-checks> 2026-10-08T03:15:20.346264Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=14.058526646s elapsed_secs=14.058526646 slow_threshold=1s
horae-browser-checks> 2026-10-08 03:13:34.021 UTC [18] LOG: starting PostgreSQL 17.10 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit
horae-browser-checks> 2026-10-08 03:13:34.024 UTC [18] LOG: listening on Unix socket "/tmp/horae-browser.2IqbMz/.s.PGSQL.5432"
horae-browser-checks> 2026-10-08 03:13:34.032 UTC [21] LOG: database system was shut down at 2026-10-08 03:13:33 UTC
horae-browser-checks> 2026-10-08 03:13:34.041 UTC [18] LOG: database system is ready to accept connections
horae-browser-checks> 2026-10-08 03:15:28.491 UTC [18] LOG: received fast shutdown request
horae-browser-checks> 2026-10-08 03:15:28.499 UTC [18] LOG: aborting any active transactions
horae-browser-checks> 2026-10-08 03:15:28.508 UTC [18] LOG: background worker "logical replication launcher" (PID 24) exited with exit code 1
horae-browser-checks> 2026-10-08 03:15:28.516 UTC [19] LOG: shutting down
horae-browser-checks> 2026-10-08 03:15:28.517 UTC [19] LOG: checkpoint starting: shutdown immediate
horae-browser-checks> 2026-10-08 03:15:28.625 UTC [19] LOG: checkpoint complete: wrote 1150 buffers (7.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.098 s, sync=0.001 s, total=0.110 s; sync files=0, longest=0.000 s, average=0.000 s; distance=5994 kB, estimate=5994 kB; lsn=0/1AC71C0, redo lsn=0/1AC71C0
horae-browser-checks> 2026-10-08 03:15:28.636 UTC [18] LOG: database system is shut down
error: build of '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv' on 'grpc://farm.numtide.com:50051?client-cert=/run/secrets/vars/per-machine/bld1/nix-farm-ci/crt&client-key=/run/secrets/vars/per-machine/bld1/nix-farm-ci/key&connect-timeout=21&system=aarch64-linux' failed: Cannot build '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/5jk1h8dvi5dvs5kjhs0hwn28abq7iwa2-horae-browser-checks
Last 25 log lines:
> 2026-10-08T03:14:39.543512Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=1.12329726s elapsed_secs=1.12329726 slow_threshold=1s
> 2026-10-08T03:14:40.723281Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.787434467s elapsed_secs=1.787434467 slow_threshold=1s
> 2026-10-08T03:14:43.282986Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.427356845s elapsed_secs=1.427356845 slow_threshold=1s
> 2026-10-08T03:14:44.809936Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.550733606s elapsed_secs=1.550733606 slow_threshold=1s
> 2026-10-08T03:14:45.747916Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=10.028326278s elapsed_secs=10.028326278 slow_threshold=1s
> 2026-10-08T03:14:49.596880Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.156545854s elapsed_secs=2.156545854 slow_threshold=1s
> 2026-10-08T03:14:50.575286Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.663449826s elapsed_secs=1.663449826 slow_threshold=1s
> 2026-10-08T03:14:50.819215Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT ps.project_id, ps.org_id FROM …" db.statement="\n\nSELECT ps.project_id, ps.org_id FROM project_settings ps\n JOIN projects p ON p.id = ps.project_id AND p.org_id = ps.org_id\n WHERE ps.alert_enabled AND p.active AND ps.project_id > $1\n ORDER BY ps.project_id LIMIT 100\n" rows_affected=0 rows_returned=0 elapsed=4.559051529s elapsed_secs=4.5590515289999995 slow_threshold=1s
> 2026-10-08T03:14:53.477254Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.411615434s elapsed_secs=1.411615434 slow_threshold=1s
> 2026-10-08T03:14:56.854646Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.00133484s elapsed_secs=1.00133484 slow_threshold=1s
> 2026-10-08T03:15:02.968388Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=3.50546767s elapsed_secs=3.50546767 slow_threshold=1s
> 2026-10-08T03:15:09.757751Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=14.30244938s elapsed_secs=14.30244938 slow_threshold=1s
> 2026-10-08T03:15:20.284501Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=3.050003098s elapsed_secs=3.050003098 slow_threshold=1s
> 2026-10-08T03:15:20.346264Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=14.058526646s elapsed_secs=14.058526646 slow_threshold=1s
> 2026-10-08 03:13:34.021 UTC [18] LOG: starting PostgreSQL 17.10 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit
> 2026-10-08 03:13:34.024 UTC [18] LOG: listening on Unix socket "/tmp/horae-browser.2IqbMz/.s.PGSQL.5432"
> 2026-10-08 03:13:34.032 UTC [21] LOG: database system was shut down at 2026-10-08 03:13:33 UTC
> 2026-10-08 03:13:34.041 UTC [18] LOG: database system is ready to accept connections
> 2026-10-08 03:15:28.491 UTC [18] LOG: received fast shutdown request
> 2026-10-08 03:15:28.499 UTC [18] LOG: aborting any active transactions
> 2026-10-08 03:15:28.508 UTC [18] LOG: background worker "logical replication launcher" (PID 24) exited with exit code 1
> 2026-10-08 03:15:28.516 UTC [19] LOG: shutting down
> 2026-10-08 03:15:28.517 UTC [19] LOG: checkpoint starting: shutdown immediate
> 2026-10-08 03:15:28.625 UTC [19] LOG: checkpoint complete: wrote 1150 buffers (7.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.098 s, sync=0.001 s, total=0.110 s; sync files=0, longest=0.000 s, average=0.000 s; distance=5994 kB, estimate=5994 kB; lsn=0/1AC71C0, redo lsn=0/1AC71C0
> 2026-10-08 03:15:28.636 UTC [18] LOG: database system is shut down
For full logs, run:
nix log /nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv
error: Cannot build '/nix/store/pf1g8vdr6pl2n9z6hapvznp2grixsxks-horae-browser-checks.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/5jk1h8dvi5dvs5kjhs0hwn28abq7iwa2-horae-browser-checks