this derivation will be built:
/nix/store/91gjbjrsmlfcmng50kr3ji53bpjbzlfc-horae-browser-checks.drv
building '/nix/store/91gjbjrsmlfcmng50kr3ji53bpjbzlfc-horae-browser-checks.drv' on 'grpc://farm.numtide.com:50051?client-cert=/run/secrets/vars/per-machine/bld1/nix-farm-ci/crt&client-key=/run/secrets/vars/per-machine/bld1/nix-farm-ci/key&connect-timeout=21&system=aarch64-linux'
building '/nix/store/91gjbjrsmlfcmng50kr3ji53bpjbzlfc-horae-browser-checks.drv' on farm.numtide.com:50051
horae-browser-checks> elastic-arm-976bb8e2: building 91gjbjrsmlfcmng50kr3ji53bpjbzlfc-horae-browser-checks.drv
horae-browser-checks> initdb: warning: enabling "trust" authentication for local connections
horae-browser-checks> initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
horae-browser-checks> waiting for server to start.... done
horae-browser-checks> server started
horae-browser-checks> 2026-10-08T02:06:44.340263Z INFO horae: Running migrations...
horae-browser-checks> 2026-10-08T02:06:44.934191Z INFO horae: Migrations complete.
horae-browser-checks> 2026-10-08T02:06:45.006821Z INFO horae::seed: Seeding demo data…
horae-browser-checks> 2026-10-08T02:06:45.068033Z INFO horae::seed: Seed complete.
horae-browser-checks> 2026-10-08T02:06:45.096436Z INFO horae::seed: time_entries: 10 rows
horae-browser-checks> 2026-10-08T02:06:45.101681Z INFO horae::seed: clients: 2 rows
horae-browser-checks> PASS: header actions, supported metrics and labelled native budget progress
horae-browser-checks> PASS: design heading, compact row typography and inline project type
horae-browser-checks> PASS: filtered empty state resets search, scope and client
horae-browser-checks> PASS: narrow/short layouts, keyboard row actions and export formats
horae-browser-checks> PASS: loading spend is unavailable until the read completes
horae-browser-checks> PASS: failed spend is unavailable; retry restores real amounts
horae-browser-checks> PASS: configured monthly consumption replaces lifetime spend only after loading
horae-browser-checks> PASS: keyboard-accessible scope breakdown exposes overruns without layout overflow
horae-browser-checks> PASS: failed configured budget never falls back to lifetime figures
horae-browser-checks> PASS: empty-state guidance and action visibility for admin
horae-browser-checks> PASS: empty-state guidance and action visibility for manager
horae-browser-checks> PASS: empty-state guidance and action visibility for member
horae-browser-checks> PASS: shared Menu, Combobox, empty-state and navigation-icon defaults are unchanged
horae-browser-checks> PASS: /clients header at 320px
horae-browser-checks> PASS: /clients header at 390px
horae-browser-checks> PASS: /clients header at 768px
horae-browser-checks> PASS: /clients header at 1280px
horae-browser-checks> PASS: /clients header at 1440px
horae-browser-checks> PASS: /projects header at 320px
horae-browser-checks> PASS: /projects header at 390px
horae-browser-checks> PASS: /projects header at 768px
horae-browser-checks> PASS: /projects header at 1280px
horae-browser-checks> PASS: /projects header at 1440px
horae-browser-checks> PASS: /invoices header at 320px
horae-browser-checks> PASS: /invoices header at 390px
horae-browser-checks> PASS: /invoices header at 768px
horae-browser-checks> PASS: /invoices header at 1280px
horae-browser-checks> PASS: /invoices header at 1440px
horae-browser-checks> PASS: /reports header at 320px
horae-browser-checks> PASS: /reports header at 390px
horae-browser-checks> PASS: /reports header at 768px
horae-browser-checks> PASS: /reports header at 1280px
horae-browser-checks> PASS: /reports header at 1440px
horae-browser-checks> PASS: /admin/users header at 320px
horae-browser-checks> PASS: /admin/users header at 390px
horae-browser-checks> PASS: /admin/users header at 768px
horae-browser-checks> PASS: /admin/users header at 1280px
horae-browser-checks> PASS: /admin/users header at 1440px
horae-browser-checks> PASS: /approvals header at 320px
horae-browser-checks> PASS: /approvals header at 390px
horae-browser-checks> PASS: /approvals header at 768px
horae-browser-checks> PASS: /approvals header at 1280px
horae-browser-checks> PASS: /approvals header at 1440px
horae-browser-checks> PASS: /settings header at 320px
horae-browser-checks> PASS: /settings header at 390px
horae-browser-checks> PASS: /settings header at 768px
horae-browser-checks> PASS: /settings header at 1280px
horae-browser-checks> PASS: /settings header at 1440px
horae-browser-checks> PASS: /admin/importers header at 320px
horae-browser-checks> PASS: /admin/importers header at 390px
horae-browser-checks> PASS: /admin/importers header at 768px
horae-browser-checks> PASS: /admin/importers header at 1280px
horae-browser-checks> PASS: /admin/importers header at 1440px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 320px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 390px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 768px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 1280px
horae-browser-checks> PASS: /timesheet/week/2026-10-05 header at 1440px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 320px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 390px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 768px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 1280px
horae-browser-checks> PASS: /timesheet/day/2026-10-05 header at 1440px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 320px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 390px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 768px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 1280px
horae-browser-checks> PASS: /timesheet/calendar/2026-10-05 header at 1440px
horae-browser-checks> PASS: project names and badges stay within aligned columns; scrolling to last-row actions opens editing
horae-browser-checks> PASS: small-screen timesheet pager, date picker and view controls remain usable
horae-browser-checks> PASS: admin subnavigation and its content remain reachable on mobile
horae-browser-checks> PASS: day strip scrolls to Sunday and long entry labels do not cover actions
horae-browser-checks> CHECK: late popover initialization restores missing focus without overriding deliberate focus
horae-browser-checks> PASS: late popover initialization restores missing focus without overriding deliberate focus
horae-browser-checks> CHECK: content reflow preserves an open menu while deliberate scrolling still dismisses it
horae-browser-checks> PASS: content reflow preserves an open menu while deliberate scrolling still dismisses it
horae-browser-checks> CHECK: first and last project-row actions are initially visible and clickable
horae-browser-checks> PASS: first and last project-row actions are initially visible and clickable
horae-browser-checks> CHECK: menu keyboard, disabled items, outside click and repeated opening
horae-browser-checks> PASS: menu keyboard, disabled items, outside click and repeated opening
horae-browser-checks> CHECK: scroll and resize dismiss without leaving stale menus or focus
horae-browser-checks> PASS: scroll and resize dismiss without leaving stale menus or focus
horae-browser-checks> CHECK: short-screen menu scrolls internally and supports reverse Tab and Space
horae-browser-checks> PASS: short-screen menu scrolls internally and supports reverse Tab and Space
horae-browser-checks> CHECK: filter and calendar menus keep their callbacks and survive route changes
horae-browser-checks> PASS: filter and calendar menus keep their callbacks and survive route changes
horae-browser-checks> CHECK: Timesheet keeps the shared calendar and week selection behavior
horae-browser-checks> PASS: Timesheet keeps the shared calendar and week selection behavior
horae-browser-checks> PASS: mobile content width, keyboard toggle, Escape/focus restoration and navigation close
horae-browser-checks> PASS: short mobile viewport reaches account navigation and preserves desktop collapse state
horae-browser-checks> PASS: desktop drag width survives mobile and breakpoint transitions
horae-browser-checks> PASS: keyboard selection, mixed/select-all states and Actions count
horae-browser-checks> PASS: search, status and client changes clear selection
horae-browser-checks> PASS: keyboard selection/confirmation and no document overflow in both themes at three widths
horae-browser-checks> PASS: cancel/focus, failure retry, pending guard and archive/reactivate confirmation
horae-browser-checks> PASS: oversized selection cannot submit and members have no bulk controls
horae-browser-checks> PASS: real atomic archive/reactivate, anonymous/member/inactive denial, manager/admin and safe retries
horae-browser-checks> PASS: bulk success restores focus and recovers the list without stale actions
horae-browser-checks> PASS: bulk pending restores focus and recovers the list without stale actions
horae-browser-checks> PASS: bulk failure restores focus and recovers the list without stale actions
horae-browser-checks> PASS: client activation failure remains visible with form closed, open and cancelled
horae-browser-checks> PASS: successful form save preserves action error; real status retry clears it and refreshes the row
horae-browser-checks> PASS: archive failure survives menu dismissal; shared edit failure preserves input and safely retries
horae-browser-checks> PASS: assignment removal failure remains visible with form closed, open and cancelled
horae-browser-checks> PASS: initial catalog/draft failures show loading and recovery without rendering or saving an empty replacement
horae-browser-checks> PASS: design typography, section spacing, quiet dividers and theme-aware task information
horae-browser-checks> PASS: project types retain native keyboard selection and the designed icon/option-card states
horae-browser-checks> PASS: searchable client selection, pending/error recovery and explicit/inherited currency
horae-browser-checks> PASS: client default-rate currency rejects relabelling, preserves failed input and permits compatible edits
horae-browser-checks> PASS: task and teammate searches handle pending results, keyboard selection, duplicates and retry
horae-browser-checks> PASS: real client, team, task restrictions, billing and invoice defaults survive reload
horae-browser-checks> PASS: all invoice-default errors identify and focus their field, preserve input and recover at three widths
horae-browser-checks> PASS: basic-field rejections retain values, link visible errors and restore focus at three widths
horae-browser-checks> PASS: failed resume and retry preserve the existing draft without a replacement write
horae-browser-checks> PASS: invalid hourly rates and hour/fee budgets retain input and focus their error at three widths
horae-browser-checks> PASS: a stale email configuration rejection can be corrected without sending mail
horae-browser-checks> PASS: task/person rate, cost and budget errors identify their row and preserve other inputs at three widths
horae-browser-checks> PASS: single/monthly fees, empty schedules and individual milestone errors recover without changing another row
horae-browser-checks> PASS: responsive form, conditional panels and Cancel preserve the draft
horae-browser-checks> PASS: edits made while a save is pending are saved after its acknowledgement
horae-browser-checks> PASS: lost acknowledgement retries the same request and preserves newer edits
horae-browser-checks> PASS: real finalized budget reaches the authorized Projects endpoint and display
horae-browser-checks> PASS: a new tagged project reaches reports and both exports; pending/error filters never show stale rows
horae-browser-checks> PASS: project invoice defaults prefill real estimates, stale reviews cannot generate, and editable draft values persist independently
horae-browser-checks> PASS: lost creation response is idempotent and clears the completed draft
horae-browser-checks> PASS: stale tabs cannot overwrite saved input; explicit discard removes only the draft
horae-browser-checks> locator.fill: Timeout 30000ms exceeded.
horae-browser-checks> Call log:
horae-browser-checks> - waiting for getByLabel('Period from', { exact: true })
horae-browser-checks> - locator resolved to
horae-browser-checks> - fill("2026-09-01")
horae-browser-checks> - attempting fill action
horae-browser-checks> - waiting for element to be visible, enabled and editable
horae-browser-checks>
horae-browser-checks> at newFeeInvoice (/nix/store/zn2f4srw60ixzmxqx0dgar80gvbmzry4-browser/invoice-preparation.cjs:382:61)
horae-browser-checks> at async /nix/store/zn2f4srw60ixzmxqx0dgar80gvbmzry4-browser/invoice-preparation.cjs:394:5 {
horae-browser-checks> name: 'TimeoutError'
horae-browser-checks> }
horae-browser-checks> {
horae-browser-checks> url: 'http://127.0.0.1:8093/invoices',
horae-browser-checks> errors: [],
horae-browser-checks> page: 'locator.ariaSnapshot: Timeout 3000ms exceeded.\n' +
horae-browser-checks> 'Call log:\n' +
horae-browser-checks> "\x1B[2m - waiting for locator('body')\x1B[22m\n"
horae-browser-checks> }
horae-browser-checks> locator.fill: Timeout 30000ms exceeded.
horae-browser-checks> Call log:
horae-browser-checks> - waiting for getByLabel('Period from', { exact: true })
horae-browser-checks> - locator resolved to
horae-browser-checks> - fill("2026-09-01")
horae-browser-checks> - attempting fill action
horae-browser-checks> - waiting for element to be visible, enabled and editable
horae-browser-checks>
horae-browser-checks> at newFeeInvoice (/nix/store/zn2f4srw60ixzmxqx0dgar80gvbmzry4-browser/invoice-preparation.cjs:382:61)
horae-browser-checks> at async /nix/store/zn2f4srw60ixzmxqx0dgar80gvbmzry4-browser/invoice-preparation.cjs:394:5 {
horae-browser-checks> name: 'TimeoutError'
horae-browser-checks> }
horae-browser-checks> 2026-10-08T02:06:45.269632Z WARN horae: DEV_LOGIN=1: /auth/dev-login signs anyone in as an admin without a password. Use it for local development only.
horae-browser-checks> 2026-10-08T02:15:12.725537Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.482293674s elapsed_secs=1.4822936740000001 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:17.019803Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=2.91848476s elapsed_secs=2.91848476 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:18.418983Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=3.058116801s elapsed_secs=3.058116801 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:21.711505Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=1.665913306s elapsed_secs=1.665913306 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:23.706861Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.535439749s elapsed_secs=2.535439749 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:26.944320Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=2.30882417s elapsed_secs=2.30882417 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:28.131817Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.642449193s elapsed_secs=1.642449193 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:30.869378Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=1.10031702s elapsed_secs=1.10031702 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:33.525880Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=1.104311499s elapsed_secs=1.104311499 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:35.735983Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=4.417892636s elapsed_secs=4.417892636 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:35.380825Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.242801501s elapsed_secs=1.242801501 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:39.666761Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=3.422733467s elapsed_secs=3.422733467 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:40.419356Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.026787768s elapsed_secs=2.026787768 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:45.619923Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.328468365s elapsed_secs=2.328468365 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:49.702375Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=3.608141976s elapsed_secs=3.6081419759999998 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:50.100757Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.701662856s elapsed_secs=1.701662856 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:54.455871Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=1.651911309s elapsed_secs=1.651911309 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:58.553351Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.0236872s elapsed_secs=1.0236872 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:15:59.291302Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=5.532521251s elapsed_secs=5.532521251 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:02.630624Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=16.72551392s elapsed_secs=16.72551392 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:05.001405Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.364774235s elapsed_secs=2.364774235 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:05.001362Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT ps.project_id, ps.org_id FROM …" db.statement="\n\nSELECT ps.project_id, ps.org_id FROM project_settings ps\n JOIN projects p ON p.id = ps.project_id AND p.org_id = ps.org_id\n WHERE ps.alert_enabled AND p.active AND ps.project_id > $1\n ORDER BY ps.project_id LIMIT 100\n" rows_affected=0 rows_returned=0 elapsed=1.401527778s elapsed_secs=1.401527778 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:04.819080Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=5.516441856s elapsed_secs=5.516441856 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:09.793076Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=1.400579778s elapsed_secs=1.400579778 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:13.179731Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=5.187092985s elapsed_secs=5.187092985 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:14.455122Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.39059102s elapsed_secs=1.39059102 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:20.576337Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=4.019030304s elapsed_secs=4.019030304 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:22.021011Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=6.264136212s elapsed_secs=6.264136212 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:27.152086Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=1.422454172s elapsed_secs=1.4224541720000001 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:29.658883Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=1.107304298s elapsed_secs=1.107304298 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:30.612961Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=6.566085609s elapsed_secs=6.566085609 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:33.439821Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.802005948s elapsed_secs=1.8020059480000001 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:43.740434Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=9.360566888s elapsed_secs=9.360566888 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:49.658331Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=13.594550453s elapsed_secs=13.594550453 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:56.772113Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=9.1663031s elapsed_secs=9.1663031 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:16:59.690147Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=4.743391306s elapsed_secs=4.743391306 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:04.212322Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=1.882476167s elapsed_secs=1.882476167 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:07.943264Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.939104632s elapsed_secs=1.939104632 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:12.677957Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=10.531491728s elapsed_secs=10.531491728 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:22.277673Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=12.413766345s elapsed_secs=12.413766344999999 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:24.380313Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=37.520474462s elapsed_secs=37.520474462 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:29.449586Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_job_uploads u …" db.statement="\n\nDELETE FROM horae_job_uploads u\n USING horae_jobs j\n WHERE u.job_id = j.id\n AND j.status = 'succeeded'\n AND j.finished_at < now() - interval '1 day'\n" rows_affected=0 rows_returned=0 elapsed=2.133469342s elapsed_secs=2.133469342 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:29.485354Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=10.519745267s elapsed_secs=10.519745267 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:34.772367Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=2.687217631s elapsed_secs=2.687217631 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:39.446650Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT ps.project_id, ps.org_id FROM …" db.statement="\n\nSELECT ps.project_id, ps.org_id FROM project_settings ps\n JOIN projects p ON p.id = ps.project_id AND p.org_id = ps.org_id\n WHERE ps.alert_enabled AND p.active AND ps.project_id > $1\n ORDER BY ps.project_id LIMIT 100\n" rows_affected=0 rows_returned=0 elapsed=11.960335397s elapsed_secs=11.960335397 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:39.446608Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.811196149s elapsed_secs=1.8111961490000001 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:39.601670Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=5.591269683s elapsed_secs=5.591269683 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:44.274530Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.326089329s elapsed_secs=2.326089329 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:46.670645Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=1.153678247s elapsed_secs=1.153678247 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:53.480125Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.065374911s elapsed_secs=1.065374911 slow_threshold=1s
horae-browser-checks> 2026-10-08T02:17:54.774283Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.894242606s elapsed_secs=1.8942426060000002 slow_threshold=1s
horae-browser-checks> 2026-10-08 02:06:43.932 UTC [18] LOG: starting PostgreSQL 17.10 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit
horae-browser-checks> 2026-10-08 02:06:43.932 UTC [18] LOG: listening on Unix socket "/tmp/horae-browser.aKxtZZ/.s.PGSQL.5432"
horae-browser-checks> 2026-10-08 02:06:43.934 UTC [21] LOG: database system was shut down at 2026-10-08 02:06:43 UTC
horae-browser-checks> 2026-10-08 02:06:43.937 UTC [18] LOG: database system is ready to accept connections
horae-browser-checks> 2026-10-08 02:06:45.103 UTC [47] LOG: could not send data to client: Broken pipe
horae-browser-checks> 2026-10-08 02:06:45.103 UTC [47] FATAL: connection to client lost
horae-browser-checks> 2026-10-08 02:11:44.049 UTC [19] LOG: checkpoint starting: time
horae-browser-checks> 2026-10-08 02:13:42.447 UTC [19] LOG: checkpoint complete: wrote 1159 buffers (7.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=118.375 s, sync=0.001 s, total=118.397 s; sync files=0, longest=0.000 s, average=0.000 s; distance=6147 kB, estimate=6147 kB; lsn=0/1B10720, redo lsn=0/1AED6D0
horae-browser-checks> 2026-10-08 02:16:44.731 UTC [19] LOG: checkpoint starting: time
horae-browser-checks> 2026-10-08 02:17:30.427 UTC [19] LOG: checkpoint complete: wrote 139 buffers (0.8%); 0 WAL file(s) added, 0 removed, 0 recycled; write=43.853 s, sync=0.035 s, total=45.704 s; sync files=0, longest=0.000 s, average=0.000 s; distance=289 kB, estimate=5561 kB; lsn=0/1B35EF0, redo lsn=0/1B35E60
horae-browser-checks> 2026-10-08 02:17:57.979 UTC [18] LOG: received fast shutdown request
horae-browser-checks> 2026-10-08 02:17:57.981 UTC [18] LOG: aborting any active transactions
horae-browser-checks> 2026-10-08 02:17:57.989 UTC [18] LOG: background worker "logical replication launcher" (PID 24) exited with exit code 1
horae-browser-checks> 2026-10-08 02:17:57.997 UTC [19] LOG: shutting down
horae-browser-checks> 2026-10-08 02:17:58.021 UTC [19] LOG: checkpoint starting: shutdown immediate
horae-browser-checks> 2026-10-08 02:17:58.029 UTC [19] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.032 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=5005 kB; lsn=0/1B35FA0, redo lsn=0/1B35FA0
horae-browser-checks> 2026-10-08 02:17:58.069 UTC [18] LOG: database system is shut down
error: build of '/nix/store/91gjbjrsmlfcmng50kr3ji53bpjbzlfc-horae-browser-checks.drv' on 'grpc://farm.numtide.com:50051?client-cert=/run/secrets/vars/per-machine/bld1/nix-farm-ci/crt&client-key=/run/secrets/vars/per-machine/bld1/nix-farm-ci/key&connect-timeout=21&system=aarch64-linux' failed: Cannot build '/nix/store/91gjbjrsmlfcmng50kr3ji53bpjbzlfc-horae-browser-checks.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/nqa0cd063pwny9d15pq9sc5h2d25c4k3-horae-browser-checks
Last 25 log lines:
> 2026-10-08T02:17:34.772367Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="DELETE FROM horae_jobs WHERE …" db.statement="\n\nDELETE FROM horae_jobs\n WHERE status IN ('succeeded', 'failed', 'cancelled')\n AND finished_at < now() - interval '30 days'\n" rows_affected=0 rows_returned=0 elapsed=2.687217631s elapsed_secs=2.687217631 slow_threshold=1s
> 2026-10-08T02:17:39.446650Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT ps.project_id, ps.org_id FROM …" db.statement="\n\nSELECT ps.project_id, ps.org_id FROM project_settings ps\n JOIN projects p ON p.id = ps.project_id AND p.org_id = ps.org_id\n WHERE ps.alert_enabled AND p.active AND ps.project_id > $1\n ORDER BY ps.project_id LIMIT 100\n" rows_affected=0 rows_returned=0 elapsed=11.960335397s elapsed_secs=11.960335397 slow_threshold=1s
> 2026-10-08T02:17:39.446608Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="UPDATE horae_jobs SET status …" db.statement="\n\nUPDATE horae_jobs\n SET status = CASE WHEN cancellation_requested THEN 'cancelled' ELSE 'failed' END,\n lease_until = NULL, worker_id = NULL, claim_token = NULL,\n finished_at = now(), updated_at = now(),\n last_error = CASE WHEN cancellation_requested THEN NULL\n ELSE COALESCE(last_error, 'Job attempt limit reached after interruption') END\n WHERE (attempts >= max_attempts OR cancellation_requested)\n AND (status = 'queued' OR (status = 'running' AND lease_until < now()))\n" rows_affected=0 rows_returned=0 elapsed=1.811196149s elapsed_secs=1.8111961490000001 slow_threshold=1s
> 2026-10-08T02:17:39.601670Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=5.591269683s elapsed_secs=5.591269683 slow_threshold=1s
> 2026-10-08T02:17:44.274530Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=2.326089329s elapsed_secs=2.326089329 slow_threshold=1s
> 2026-10-08T02:17:46.670645Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH due AS ( …" db.statement="\n\nWITH due AS (\n UPDATE time_entries te\n SET notified_long_running_at = now()\n FROM organizations o\n WHERE o.id = te.org_id\n AND te.is_running = true\n AND te.notified_long_running_at IS NULL\n AND te.started_at < now() - make_interval(mins => o.long_timer_minutes)\n RETURNING te.id, te.org_id, te.user_id, te.project_id, te.task_id,\n te.spent_date, te.minutes, te.billable, te.notes, te.started_at\n )\n SELECT id, org_id, user_id, project_id, task_id,\n spent_date as \"spent_date: chrono::NaiveDate\",\n minutes, billable, notes,\n started_at as \"started_at!: chrono::DateTime\",\n (EXTRACT(EPOCH FROM (now() - started_at)) / 60)::int as \"running_minutes!\"\n FROM due\n" rows_affected=0 rows_returned=0 elapsed=1.153678247s elapsed_secs=1.153678247 slow_threshold=1s
> 2026-10-08T02:17:53.480125Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id FROM horae_outbox\n WHERE delivered_at IS NULL AND failed_at IS NULL AND available_at <= now() AND event_kind = $2\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED LIMIT 1\n )\n UPDATE horae_outbox o\n SET attempts = o.attempts + 1,\n claim_token = $1,\n available_at = now() + interval '5 minutes'\n FROM candidate\n WHERE o.id = candidate.id\n RETURNING o.id, o.org_id, o.event_kind, o.payload, o.attempts,\n o.claim_token as \"claim_token!\"\n" rows_affected=0 rows_returned=0 elapsed=1.065374911s elapsed_secs=1.065374911 slow_threshold=1s
> 2026-10-08T02:17:54.774283Z WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="WITH candidate AS ( …" db.statement="\n\nWITH candidate AS (\n SELECT id\n FROM horae_jobs\n WHERE attempts < max_attempts AND NOT cancellation_requested\n AND ((status = 'queued' AND available_at <= now())\n OR (status = 'running' AND lease_until < now()))\n ORDER BY available_at, created_at\n FOR UPDATE SKIP LOCKED\n LIMIT 1\n )\n UPDATE horae_jobs j\n SET status = 'running', attempts = j.attempts + 1,\n lease_until = now() + $1::int * interval '1 second', worker_id = $2,\n claim_token = $3,\n started_at = COALESCE(j.started_at, now()), updated_at = now()\n FROM candidate\n WHERE j.id = candidate.id\n RETURNING j.id, j.org_id, j.payload\n" rows_affected=0 rows_returned=0 elapsed=1.894242606s elapsed_secs=1.8942426060000002 slow_threshold=1s
> 2026-10-08 02:06:43.932 UTC [18] LOG: starting PostgreSQL 17.10 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit
> 2026-10-08 02:06:43.932 UTC [18] LOG: listening on Unix socket "/tmp/horae-browser.aKxtZZ/.s.PGSQL.5432"
> 2026-10-08 02:06:43.934 UTC [21] LOG: database system was shut down at 2026-10-08 02:06:43 UTC
> 2026-10-08 02:06:43.937 UTC [18] LOG: database system is ready to accept connections
> 2026-10-08 02:06:45.103 UTC [47] LOG: could not send data to client: Broken pipe
> 2026-10-08 02:06:45.103 UTC [47] FATAL: connection to client lost
> 2026-10-08 02:11:44.049 UTC [19] LOG: checkpoint starting: time
> 2026-10-08 02:13:42.447 UTC [19] LOG: checkpoint complete: wrote 1159 buffers (7.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=118.375 s, sync=0.001 s, total=118.397 s; sync files=0, longest=0.000 s, average=0.000 s; distance=6147 kB, estimate=6147 kB; lsn=0/1B10720, redo lsn=0/1AED6D0
> 2026-10-08 02:16:44.731 UTC [19] LOG: checkpoint starting: time
> 2026-10-08 02:17:30.427 UTC [19] LOG: checkpoint complete: wrote 139 buffers (0.8%); 0 WAL file(s) added, 0 removed, 0 recycled; write=43.853 s, sync=0.035 s, total=45.704 s; sync files=0, longest=0.000 s, average=0.000 s; distance=289 kB, estimate=5561 kB; lsn=0/1B35EF0, redo lsn=0/1B35E60
> 2026-10-08 02:17:57.979 UTC [18] LOG: received fast shutdown request
> 2026-10-08 02:17:57.981 UTC [18] LOG: aborting any active transactions
> 2026-10-08 02:17:57.989 UTC [18] LOG: background worker "logical replication launcher" (PID 24) exited with exit code 1
> 2026-10-08 02:17:57.997 UTC [19] LOG: shutting down
> 2026-10-08 02:17:58.021 UTC [19] LOG: checkpoint starting: shutdown immediate
> 2026-10-08 02:17:58.029 UTC [19] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.032 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=5005 kB; lsn=0/1B35FA0, redo lsn=0/1B35FA0
> 2026-10-08 02:17:58.069 UTC [18] LOG: database system is shut down
For full logs, run:
nix log /nix/store/91gjbjrsmlfcmng50kr3ji53bpjbzlfc-horae-browser-checks.drv
error: Cannot build '/nix/store/91gjbjrsmlfcmng50kr3ji53bpjbzlfc-horae-browser-checks.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/nqa0cd063pwny9d15pq9sc5h2d25c4k3-horae-browser-checks