122 lines
6.4 KiB
Markdown
122 lines
6.4 KiB
Markdown
# Sync Feature — Implementation Notes
|
||
|
||
## DB: better-sqlite3 (migrated from @libsql/client)
|
||
|
||
**Why migrated:** `@libsql/client` caused SIGSEGV (exit code 139) on Azure App Service due to Rust native module incompatibility with seccomp profile.
|
||
|
||
**Key constraints of better-sqlite3:**
|
||
- All operations are **synchronous** — no `await` on db calls
|
||
- `db.transaction()` callback **must be synchronous** — `async (tx) => {}` throws `TypeError: Transaction function cannot return a promise`
|
||
- Drizzle ORM for better-sqlite3: `drizzle-orm/better-sqlite3` + `drizzle-orm/better-sqlite3/migrator`
|
||
- Result object uses `.changes` (not `.rowsAffected`) → use `(result as any).changes ?? (result as any).rowsAffected ?? 0`
|
||
|
||
**Config:**
|
||
- `drizzle.config.ts`: `url` without `file:` prefix
|
||
- Docker base: `node:22-slim` (prebuilt binaries available)
|
||
- No build tools needed in Dockerfile (prebuilts used on Linux)
|
||
|
||
**Migration:** `migrate(db, { migrationsFolder })` called synchronously at startup in `connection.ts`.
|
||
|
||
## sync-service.ts
|
||
|
||
`pushChanges` processes entities in FK order: contexts → topics → historyEntries → ratings → imageBlobs → pages → notebooks → pageNotebooks.
|
||
|
||
`upsertEntity` logic: insert if not exists, update if `clientVersion > serverVersion`, else record conflict.
|
||
|
||
**No transaction wrapper** — the old `db.transaction(async ...)` was dead code and was removed. Each entity is upserted individually.
|
||
|
||
## API Keys (M2M Auth)
|
||
|
||
Added `api_keys` table in schema.ts. Format: `ka_<base64url-32bytes>`. Only SHA-256 hash stored in DB.
|
||
|
||
Auth middleware (`auth.ts`) detects `ka_...` Bearer tokens **before** JWT path, looks up by hash, fires `lastUsedAt` update async.
|
||
|
||
Routes: `GET /api/api-keys`, `POST /api/api-keys`, `DELETE /api/api-keys/:id` (soft-delete).
|
||
|
||
Raw key returned **once** on creation — not recoverable after that.
|
||
|
||
## Browser API Key (client-side)
|
||
|
||
`browserApiKey` store in `authStore.ts` backed by `localStorage['ka-note-browser-key']`.
|
||
|
||
`getAccessToken()` returns browser key first, then falls back to MSAL.
|
||
|
||
`isAuthenticated` derived from `account OR browserApiKey`.
|
||
|
||
UI in Settings → "Browser-Authentifizierung" section.
|
||
|
||
Use case: avoids MSAL silent refresh failures (Azure AD returning HTTP 400 on refresh token).
|
||
|
||
**401 auto-invalidation:** `authFetch` in `apiClient.ts` detects 401 responses when a browser key was used and calls `setBrowserApiKey(null)` — clears `localStorage` and the store. `isAuthenticated` becomes `false`, restoring the MSAL login UI. Prevents the stuck state where a revoked/expired key blocks both sync and manual login.
|
||
|
||
## Deploy
|
||
|
||
`deploy.ps1` uses normal `docker build` with layer caching. Only changed layers are pushed to ACR.
|
||
|
||
`COPY server/ server/` triggers a cache miss whenever server source changes → tsc reruns automatically. `--no-cache` is not needed and was removed (it caused all layers to be rebuilt and pushed on every deploy, >40 MB).
|
||
|
||
## Delta Push (client-side filtering)
|
||
|
||
`pushAll(since)` only sends entities with `updatedAt > since.toISOString()`. A read-only client sends 0 entities and skips the HTTP request entirely.
|
||
|
||
`fullSync()` passes `since=null` → pushes everything (used after DB reset or first sync).
|
||
|
||
**Root cause for all-conflicts:** `upsertEntity` accepts only `clientVersion > serverVersion`. After a pull, client has `version == serverVersion` → equal-version entities are never pushed (filtered out by delta push). Without delta push, a read-only client would produce 700+ conflicts on every sync cycle.
|
||
|
||
## Client identification (`X-Client-Id` header)
|
||
|
||
Each browser instance generates a stable random ID in `localStorage['ka-clientId']`. The header value is `Browser·OS·id` (e.g. `Edge·Win·a3f2`, `Safari·iOS·x9k2`).
|
||
|
||
Server logs `client:` field in `[sync/push]` and `[sync/pull]` entries.
|
||
|
||
## ECONNRESET on POST /api/sync/push
|
||
|
||
`Error: aborted / ECONNRESET` logged as 500 is a **client-side disconnect**, not a server error. Hono logs 500 because response couldn't be sent. Causes: tab backgrounded on iOS, Azure idle timeout, browser request timeout.
|
||
|
||
Not actionable on server side unless response time is consistently >10s (indicates oversized payload, e.g. first full sync after DB reset).
|
||
|
||
## Paginated Pull & Push (since v1.2.6)
|
||
|
||
**Root cause:** Single pull response was 6.25 MB (all tables at once). iOS Safari aborted both push and pull on full sync (e.g. after local DB reset).
|
||
|
||
### Pull — table-by-table pagination
|
||
|
||
`GET /api/sync/pull?since=&table=historyEntries&limit=500&offset=0`
|
||
|
||
- `table` param routes to `pullTable()` in `sync-service.ts`
|
||
- `limit+1` trick detects `hasMore` without extra COUNT query
|
||
- **Stable sort**: `ORDER BY updatedAt ASC, id ASC` — mandatory for correct offset pagination (ties on `updatedAt` alone cause skipped/duplicate rows)
|
||
- Response: same `SyncPullResponse` shape + optional `hasMore` / `nextOffset`
|
||
- Legacy path (no `?table=`): `pullChanges()` unchanged, returns all at once
|
||
|
||
**Pull order** (FK-safe, imageBlobs excluded — lazy loaded):
|
||
```
|
||
contexts → notebooks → topics → pageNotebooks → pages → ratings → historyEntries → tasks
|
||
```
|
||
|
||
Client loops per table until `hasMore === false`, merges each page into Dexie in a **small per-page transaction** (previously one big transaction across all 9 tables).
|
||
|
||
`lastSyncAt` is set only after all tables complete (last page of last table).
|
||
|
||
### Push — table-by-table batches
|
||
|
||
Client sends one table at a time in FK order, 500 entities per batch (10 for imageBlobs). Server `pushChanges()` already handles sparse changes (empty arrays skipped) — no server-side changes needed.
|
||
|
||
**Push order** (FK-safe, imageBlobs last):
|
||
```
|
||
contexts → topics → historyEntries → ratings → pages → notebooks → pageNotebooks → tasks → imageBlobs
|
||
```
|
||
|
||
### imageBlobs — lazy on-demand
|
||
|
||
imageBlobs are **excluded from the pull loop** (1 blob = 1–3 MB base64, pagination alone doesn't scale).
|
||
|
||
- `GET /api/sync/blob/:id` — fetches single blob by ID, returns same `ImageBlob` format
|
||
- `getImageUrl(id)` in `imageStore.ts`: checks Dexie → if miss, fetches from server → stores in Dexie → returns objectURL
|
||
- Push remains eager (blobs must reach server so other devices can lazy-fetch)
|
||
- Offline: missing blob → `getImageUrl()` returns `null` → `ka-img:ID` stays unreplaced in markdown → no crash
|
||
|
||
### Concurrency guard
|
||
|
||
`_syncInProgress` flag in `syncService.ts` prevents overlapping sync runs (30s auto-interval + visibilitychange + manual can fire simultaneously; with pagination a sync takes much longer).
|