Operations

Keep an OrcAI instance running and find out why something stopped.

Creating accountsCopy Anchor Link

Creating an account needs an invitation. The endpoint that creates accounts requires the ID of a pending, unexpired invitation for the same address, so there is no open account creation to turn off, and no setting turns one on.

The first account of an instance comes from the init flow at /init, which writes the user directly, so the rule does not apply to it. Every later account is created from an invitation link an organisation admin sends. Creating the account accepts that invitation and writes the membership, while other invitations stay pending until the user accepts them. Membership, invitation status and permission intent commit atomically, and permission delivery retries from the outbox after the commit.

The rule does not affect existing accounts, so sign-in, password reset and invitations to further organisations keep working.

Permission deliveryCopy Anchor Link

Every change to memberships, groups, grants, bots, blocks, and assets commits its SpiceDB relationship changes to the authz_outbox table in the same transaction. The app delivers them to SpiceDB in commit order, one deliverer at a time, right after the request and every 10 seconds in the background.

A failed delivery retries with backoff up to 8 minutes apart until it succeeds, whatever the cause: SpiceDB unreachable, timing out, refusing the app's credentials, or rejecting the change, for example because a deploy runs the new code before the new SpiceDB schema is applied. Meanwhile the failing event holds back every newer one, so permissions stay consistent with the database but stop updating until the cause is fixed.

Each failed attempt logs an authz.delivery_blocked line with the event's eventId, seq, attempts, ageSeconds, the gRPC status as code, and the cause. It is a warning at first and an error once the event is older than 5 minutes. A blocked queue therefore shows as a series of these lines for the same eventId. To see it in the database:

SELECT id, seq, attempts, created_at, next_attempt_at
FROM authz_outbox
WHERE status IN ('pending', 'failed')
ORDER BY seq
LIMIT 10;

Fix the cause the log line names, for example by applying the SpiceDB schema. The next attempt then delivers the event and every one behind it.

An event can also be skipped by hand when it can never succeed:

UPDATE authz_outbox SET status = 'dead_letter' WHERE id = '<eventId>';

Delivery then moves on to the next event. SpiceDB is missing the skipped change, so correct the affected relationships, for example by repeating the change in the app. An event whose payload the app cannot read is moved to dead_letter automatically and logged as authz.dead_letter at error level.

MigrationsCopy Anchor Link

Docker Compose applies the database migrations and the SpiceDB schema with one-shot services each time the stack starts. The SpiceDB schema is written only after the database migrations succeed, so a failed migration leaves both unchanged. The one-shot services do not restart, so a failed migration stops the start at once. Coolify ignores them in the application status once they have exited. On a manual setup, apply both with:

bun run stack migrate

Inspect SpiceDB schema migration status with:

bun run stack exec -- bun run --filter @orcai/spice-db status

Package scripts do not read the root .env, so both commands go through bun run stack, which loads it.

HealthCopy Anchor Link

GET /api/health answers 200 when PostgreSQL and SpiceDB answer within three seconds, and 503 otherwise. It needs no session and returns no body, so it is safe to expose to a load balancer or orchestrator. The Compose app healthcheck uses it.

Instance admins see the full picture on the Health page (/instance/health): every dependency, the background processes, queue counts, email delivery, versions and the app's configuration with secrets hidden. See Instance management.

Email deliveryCopy Anchor Link

The workers send email from the notification_outbox table. Each sent row records its delivery_channel: smtp, or log when the workers have no SMTP configuration and only wrote the message to their log. A log row is still marked sent, so check the channel, or the Health page, which warns when the latest email only reached the log. The app process has no SMTP settings of its own.

Retry delays start at 30 seconds and double after each failed attempt, with up to 10 seconds of random jitter. After the tenth failed attempt the row is marked dead, and its last_error keeps the reason. The notification-outbox-cleanup-job runs once a day and deletes sent rows 30 days after they were sent and dead rows 30 days after they were given up.

WorkersCopy Anchor Link

Keep workers running alongside the app. If workers are stopped, uploads and scheduled maintenance jobs may remain queued, and no email goes out, including invitations and password resets.

Every process that runs pg-boss registers in its instance registry under a name: the app as app, the workers as workers, each with its host and process id. The Health page lists them and fails its workers check when no workers process is live.

Processing (process-asset-job) and indexing (vectorize-asset-job) jobs run up to four times, a minute apart, and an attempt times out after an hour. A job whose attempts are used up moves to its dead letter queue (process-asset-dead-letter-job, vectorize-asset-dead-letter-job), whose worker marks the asset, or the asset in that repository, as failed. Metadata suggestions (generate-asset-metadata-job) use the same settings, and their dead letter queue (generate-asset-metadata-dead-letter-job) marks the suggestion as failed. A dead letter job that fails is tried again up to six times, with pauses that double from about 5 seconds, so about 5 to 10 minutes in all. Every five minutes, preparation-sweep-job marks as failed any asset, repository entry or suggestion that still waits although no job is left to finish it, so a person can prepare it again. Each process creates the queues it works, with their retry and dead letter settings, at startup. The failed job, with its last error, stays in the pg-boss tables until retention removes it, and the worker logs carry the error once for every failed attempt.

The app server works data exports (data-export-job) instead of the workers, one at a time per app process, because their chat transcripts need the app's code. The app creates the queue and its dead letter queue (data-export-dead-letter-job) itself, with the same retry and timeout settings. The workers delete expired archives, so keep them running. An export whose attempts are used up is marked failed, and the user can start a new one. Stopping the app interrupts a running export, and pg-boss retries it.

The activity-cleanup-job runs every hour and keeps each user's activity bounded. It deletes an entry seven days after its work finished, or once the user dismissed it, and never deletes an entry whose work is still running. Failures and suggestions that need a user's attention are read from the assets and exports themselves, so the cleanup never removes one.

Upload storage lifecycleCopy Anchor Link

Uploads land under staging/ in the uploads bucket. Creating the asset moves the file to its final key, so staging/ only holds files no asset was created from, such as an upload abandoned mid-flight.

Each time the workers start, they put the lifecycle rule expire-staged-uploads on the uploads bucket: objects under staging/ expire one day after upload, and incomplete multipart uploads are aborted one day after they started. Other lifecycle rules on the bucket stay as they are. The rule never matches an asset's file. If the storage provider rejects it, the workers log the error and keep running.

The storage provider enforces the rule, not OrcAI. SeaweedFS runs lifecycle in its plugin worker, once a day, so a staged file can stay up to about two days. A storage provider without bucket lifecycle support keeps staged leftovers until you remove them.

Data export archivesCopy Anchor Link

Data export archives land under exports/ in the same bucket and can be downloaded for 24 hours. The workers delete them, so removal does not depend on the storage provider. The data-export-expire-job runs every ten minutes and deletes every object under exports/ older than 25 hours (the 24 hour download window plus the hour an export may take to build), including archives whose export or user no longer exists. The app refuses a download after 24 hours even before the job has run.

Asset preparationCopy Anchor Link

If OCR or extraction fails, check that:

• The worker process is running.
• Tesseract is installed when workers run on the host.
• KREUZBERG_OCR_LANGUAGE only names installed language packs.
• The organisation's utility model has the image capability. It describes images, so without it image files fail and figures inside files are skipped.
• The object storage credentials and bucket setup are valid.

RetrievalCopy Anchor Link

If indexing fails or retrieval quality is poor, check:

• The Embeddings check on the Health page. It passes once embeddings are set up and the active Qdrant collection exists with the model's dimensions.
• The stored endpoint, model, dimensions and collection in the Health page's configuration table, which also shows whether a key is set.
• Qdrant availability and API key.

The embedding setting lives in the PostgreSQL database, not in the environment, so a PostgreSQL backup includes it. Its API key is encrypted with ENCRYPTION_KEY, so a restore needs the same key. Embeddings describes the setting.

SpiceDB connectivityCopy Anchor Link

If SpiceDB connections fail, verify:

• SPICEDB_ENDPOINT.
• SPICEDB_TOKEN.
• SPICEDB_SECURITY matches your deployment mode.

While SpiceDB is unreachable, users can still list and switch their organisations, but every page and action that checks a permission shows "Permissions could not be checked right now." and the health check answers 503. Requests wait at most SPICEDB_READ_TIMEOUT (5 seconds by default) for a permission check before they fail.