added search
This commit is contained in:
parent
cb0f38b7f8
commit
684519384b
|
|
@ -0,0 +1,136 @@
|
||||||
|
# Full-Text Search
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Ka-Note implements a hybrid full-text search strategy: small in-memory corpora (contexts, page titles) are filtered client-side; the large corpus (history entry text, page body) is indexed server-side using SQLite FTS5 and queried via HTTP.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Search tiers
|
||||||
|
|
||||||
|
| Entity | Where | Method |
|
||||||
|
|---|---|---|
|
||||||
|
| Contexts (name) | Client only | Substring on in-memory Svelte store |
|
||||||
|
| Pages (title) | Client only | Substring on in-memory Svelte store |
|
||||||
|
| HistoryEntries (text) | Server FTS5 | Debounced HTTP GET /api/search |
|
||||||
|
| Pages (body) | Server FTS5 | Debounced HTTP GET /api/search |
|
||||||
|
|
||||||
|
History entries are the primary scaling concern (years of daily journals → tens of thousands of rows). SQLite FTS5 with BM25 ranking handles this efficiently without additional infrastructure.
|
||||||
|
|
||||||
|
### Offline fallback
|
||||||
|
|
||||||
|
When the server is unreachable, CommandBar falls back to local results (contexts, page titles) only and shows a notice: "Server nicht erreichbar — nur lokale Ergebnisse".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Server
|
||||||
|
|
||||||
|
### FTS5 tables
|
||||||
|
|
||||||
|
Migration: `server/drizzle/0013_fts_search.sql`
|
||||||
|
|
||||||
|
Two virtual tables using the `unicode61` tokenizer (handles German umlauts correctly, no stemming):
|
||||||
|
|
||||||
|
- `fts_history` — content table backed by `history_entries` (columns: `text`)
|
||||||
|
- `fts_pages` — content table backed by `pages` (columns: `title`, `body`)
|
||||||
|
|
||||||
|
Both tables are populated via `INSERT INTO fts_*(...) VALUES('rebuild')` on first migration run.
|
||||||
|
|
||||||
|
### Index maintenance
|
||||||
|
|
||||||
|
FTS index is updated synchronously after every write, covering all server-side write paths:
|
||||||
|
|
||||||
|
| Write path | File | FTS update |
|
||||||
|
|---|---|---|
|
||||||
|
| Sync push (primary client sync) | `sync-service.ts` → `pushChanges()` | after each upsert |
|
||||||
|
| Trash / soft-delete | `routes/trash.ts` | after batch update |
|
||||||
|
| AI bundle upload (ZIP) | `ai-export-service.ts` → `applyOps()` | after each op |
|
||||||
|
| AI legacy JSON upload | `ai-export-service.ts` → `applyOps()` | after each op |
|
||||||
|
| Startup drift recovery | `index.ts` `setImmediate` | full rebuild if mismatch > 10 |
|
||||||
|
|
||||||
|
All paths use `better-sqlite3` prepared statements. Shared helper `applyOps()` in `ai-export-service.ts` handles both upload variants. Soft-deleted rows are removed from FTS; active rows are re-indexed via `INSERT OR REPLACE … SELECT`.
|
||||||
|
|
||||||
|
**Startup consistency check:** On each server start, row counts of `history_entries` (non-deleted) and `fts_history` are compared. If the difference exceeds 10, both FTS tables are rebuilt via `INSERT INTO fts_*(fts_*) VALUES('rebuild')`. This guards against index drift after DB restores or backup imports.
|
||||||
|
|
||||||
|
### Raw SQLite access
|
||||||
|
|
||||||
|
File: `server/src/db/connection.ts`
|
||||||
|
|
||||||
|
The `better-sqlite3` instance is exported as `sqlite` alongside the Drizzle `db`. This is needed for FTS prepared statements (Drizzle has no FTS5 DSL).
|
||||||
|
|
||||||
|
### Search endpoint
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/search?q=<query>&limit=<n>
|
||||||
|
Authorization: Bearer <token>
|
||||||
|
```
|
||||||
|
|
||||||
|
Response:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"history": [
|
||||||
|
{ "id": "...", "topicId": "...", "date": "2025-01-15", "snippet": "...text..." }
|
||||||
|
],
|
||||||
|
"pages": [
|
||||||
|
{ "id": "...", "title": "Page Title", "snippet": "...body text..." }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `q` must be ≥ 2 characters; shorter queries return empty results.
|
||||||
|
- `limit` is capped at 20 server-side.
|
||||||
|
- Each word in `q` is automatically appended with `*` for prefix matching (`"term"*`).
|
||||||
|
- Results are ranked by BM25 (`ORDER BY rank`).
|
||||||
|
- FTS5 query errors (invalid syntax from special characters) return empty results instead of HTTP 500.
|
||||||
|
- Soft-deleted entries are excluded via the FTS delete-on-soft-delete strategy.
|
||||||
|
|
||||||
|
File: `server/src/routes/search.ts`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Client
|
||||||
|
|
||||||
|
### Settings store
|
||||||
|
|
||||||
|
File: `client/src/lib/stores/settings.ts`
|
||||||
|
|
||||||
|
Generic key-value settings backed by a Dexie `settings` table (version 13). Provides:
|
||||||
|
|
||||||
|
- `getSetting<T>(key, default)` — async one-time read
|
||||||
|
- `setSetting<T>(key, value)` — async write
|
||||||
|
- `settingStore<T>(key, default)` — reactive Svelte store backed by `liveQuery`
|
||||||
|
|
||||||
|
The `searchResultsLimit` store (default: 3) controls how many server results are requested.
|
||||||
|
|
||||||
|
### CommandBar integration
|
||||||
|
|
||||||
|
File: `client/src/lib/components/CommandBar.svelte`
|
||||||
|
|
||||||
|
In navigate mode (query ≥ 2 chars, not starting with `/`):
|
||||||
|
|
||||||
|
1. **Immediately (sync):** Filters `$contextsQuery` and `$pagesQuery` by substring match on name/title.
|
||||||
|
2. **After 250ms debounce:** Calls `authFetch('/api/search?q=...&limit=...')` using the existing `apiClient` helper.
|
||||||
|
3. **On success:** Server results are appended after local results. Pages already found by title match are deduplicated.
|
||||||
|
4. **On error:** `isOffline = true`, a footer notice is shown, local results remain visible.
|
||||||
|
5. **Total results** are capped at 10.
|
||||||
|
|
||||||
|
History results deep-link to `/context/daily-log?date=YYYY-MM-DD`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Settings
|
||||||
|
|
||||||
|
| Key | Type | Default | Description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `searchResultsLimit` | number | 3 | Max server search results per entity type |
|
||||||
|
|
||||||
|
To change: write to Dexie via `setSetting('searchResultsLimit', 5)` or add a Settings UI field.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scaling notes
|
||||||
|
|
||||||
|
- FTS5 + BM25 scales to millions of rows. No action needed as data grows.
|
||||||
|
- The `unicode61` tokenizer handles Unicode correctly. Stemming can be added later by changing `tokenize='unicode61'` to `tokenize='porter unicode61'` in the migration.
|
||||||
|
- If topic title search needs FTS in future, add `fts_topics` following the same pattern.
|
||||||
|
- Offline full-text search for history (e.g. via MiniSearch in a Web Worker) is a possible v2 enhancement.
|
||||||
|
|
@ -1 +1 @@
|
||||||
1.1.75
|
1.1.77
|
||||||
|
|
@ -15,6 +15,8 @@
|
||||||
pageNameExists,
|
pageNameExists,
|
||||||
} from "$lib/db/repositories";
|
} from "$lib/db/repositories";
|
||||||
import { newId, today } from "$lib/db/helpers";
|
import { newId, today } from "$lib/db/helpers";
|
||||||
|
import { authFetch } from "$lib/auth/apiClient";
|
||||||
|
import { searchResultsLimit } from "$lib/stores/settings";
|
||||||
|
|
||||||
const contextsQuery = allActiveContexts();
|
const contextsQuery = allActiveContexts();
|
||||||
const pagesQuery = allPages();
|
const pagesQuery = allPages();
|
||||||
|
|
@ -25,6 +27,77 @@
|
||||||
let recentContextIds = $state<string[]>([]);
|
let recentContextIds = $state<string[]>([]);
|
||||||
let isMac = $state(false);
|
let isMac = $state(false);
|
||||||
|
|
||||||
|
// Server FTS results
|
||||||
|
interface ServerResult {
|
||||||
|
id: string;
|
||||||
|
type: "nav-history" | "nav-wiki";
|
||||||
|
icon: string;
|
||||||
|
label: string;
|
||||||
|
badge: string;
|
||||||
|
action: () => void;
|
||||||
|
}
|
||||||
|
let serverResults = $state<ServerResult[]>([]);
|
||||||
|
let isOffline = $state(false);
|
||||||
|
let searchTimer: ReturnType<typeof setTimeout> | null = null;
|
||||||
|
|
||||||
|
$effect(() => {
|
||||||
|
const q = query.trim();
|
||||||
|
if (q.length < 2 || q.startsWith("/")) {
|
||||||
|
serverResults = [];
|
||||||
|
isOffline = false;
|
||||||
|
if (searchTimer) clearTimeout(searchTimer);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (searchTimer) clearTimeout(searchTimer);
|
||||||
|
searchTimer = setTimeout(async () => {
|
||||||
|
try {
|
||||||
|
const limit = $searchResultsLimit;
|
||||||
|
const res = await authFetch(`/api/search?q=${encodeURIComponent(q)}&limit=${limit}`);
|
||||||
|
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||||
|
const data = await res.json() as {
|
||||||
|
history: { id: string; topicId: string; date: string; snippet: string }[];
|
||||||
|
pages: { id: string; title: string; snippet: string }[];
|
||||||
|
};
|
||||||
|
const localPageIds = new Set(
|
||||||
|
($pagesQuery ?? [])
|
||||||
|
.filter((p) => p.title.toLowerCase().includes(q.toLowerCase()))
|
||||||
|
.map((p) => p.id),
|
||||||
|
);
|
||||||
|
const combined: ServerResult[] = [
|
||||||
|
...data.history.map((h) => ({
|
||||||
|
id: `hist-${h.id}`,
|
||||||
|
type: "nav-history" as const,
|
||||||
|
icon: "📓",
|
||||||
|
label: h.snippet.replace(/<[^>]+>/g, ""),
|
||||||
|
badge: `JOURNAL ${h.date}`,
|
||||||
|
action: () => {
|
||||||
|
closeBar();
|
||||||
|
goto(`/context/daily-log?date=${h.date}`);
|
||||||
|
},
|
||||||
|
})),
|
||||||
|
...data.pages
|
||||||
|
.filter((p) => !localPageIds.has(p.id))
|
||||||
|
.map((p) => ({
|
||||||
|
id: `page-fts-${p.id}`,
|
||||||
|
type: "nav-wiki" as const,
|
||||||
|
icon: "📄",
|
||||||
|
label: p.title,
|
||||||
|
badge: "WIKI",
|
||||||
|
action: () => {
|
||||||
|
closeBar();
|
||||||
|
goto(`/wiki/${p.id}`);
|
||||||
|
},
|
||||||
|
})),
|
||||||
|
];
|
||||||
|
serverResults = combined;
|
||||||
|
isOffline = false;
|
||||||
|
} catch {
|
||||||
|
isOffline = true;
|
||||||
|
serverResults = [];
|
||||||
|
}
|
||||||
|
}, 250);
|
||||||
|
});
|
||||||
|
|
||||||
onMount(() => {
|
onMount(() => {
|
||||||
isMac = navigator.platform.toUpperCase().indexOf("MAC") >= 0;
|
isMac = navigator.platform.toUpperCase().indexOf("MAC") >= 0;
|
||||||
});
|
});
|
||||||
|
|
@ -474,7 +547,7 @@
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return searchResults;
|
return [...searchResults, ...serverResults].slice(0, 10);
|
||||||
});
|
});
|
||||||
|
|
||||||
// Reset selection when query changes
|
// Reset selection when query changes
|
||||||
|
|
@ -627,6 +700,11 @@
|
||||||
Keine Ergebnisse für "{query}"
|
Keine Ergebnisse für "{query}"
|
||||||
</div>
|
</div>
|
||||||
{/if}
|
{/if}
|
||||||
|
{#if isOffline}
|
||||||
|
<div class="px-4 py-1.5 text-[11px] text-zinc-500 border-t border-zinc-700">
|
||||||
|
Server nicht erreichbar — nur lokale Ergebnisse
|
||||||
|
</div>
|
||||||
|
{/if}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
{/if}
|
{/if}
|
||||||
|
|
|
||||||
|
|
@ -12,6 +12,12 @@ export interface ImageBlob {
|
||||||
version: number;
|
version: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface AppSetting {
|
||||||
|
key: string;
|
||||||
|
value: unknown;
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
export class KaNoteDB extends Dexie {
|
export class KaNoteDB extends Dexie {
|
||||||
contexts!: EntityTable<AgendaContext, 'id'>;
|
contexts!: EntityTable<AgendaContext, 'id'>;
|
||||||
topics!: EntityTable<Topic, 'id'>;
|
topics!: EntityTable<Topic, 'id'>;
|
||||||
|
|
@ -22,6 +28,7 @@ export class KaNoteDB extends Dexie {
|
||||||
pages!: EntityTable<Page, 'id'>;
|
pages!: EntityTable<Page, 'id'>;
|
||||||
notebooks!: EntityTable<Notebook, 'id'>;
|
notebooks!: EntityTable<Notebook, 'id'>;
|
||||||
pageNotebooks!: EntityTable<PageNotebook, 'id'>;
|
pageNotebooks!: EntityTable<PageNotebook, 'id'>;
|
||||||
|
settings!: EntityTable<AppSetting, 'key'>;
|
||||||
|
|
||||||
constructor() {
|
constructor() {
|
||||||
super('ka-note');
|
super('ka-note');
|
||||||
|
|
@ -126,6 +133,10 @@ export class KaNoteDB extends Dexie {
|
||||||
if (p.isFavorite === undefined) p.isFavorite = false;
|
if (p.isFavorite === undefined) p.isFavorite = false;
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
this.version(13).stores({
|
||||||
|
settings: '&key',
|
||||||
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,32 @@
|
||||||
|
import { liveQuery } from 'dexie';
|
||||||
|
import { readable } from 'svelte/store';
|
||||||
|
import { db } from '$lib/db/schema.js';
|
||||||
|
|
||||||
|
function now(): string {
|
||||||
|
return new Date().toISOString();
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function getSetting<T>(key: string, defaultValue: T): Promise<T> {
|
||||||
|
const row = await db.settings.get(key);
|
||||||
|
return row !== undefined ? (row.value as T) : defaultValue;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function setSetting<T>(key: string, value: T): Promise<void> {
|
||||||
|
await db.settings.put({ key, value, updatedAt: now() });
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Reactive store for a single setting. Updates when the DB changes. */
|
||||||
|
export function settingStore<T>(key: string, defaultValue: T) {
|
||||||
|
return readable<T>(defaultValue, (set) => {
|
||||||
|
const subscription = liveQuery(async () => {
|
||||||
|
const row = await db.settings.get(key);
|
||||||
|
return row !== undefined ? (row.value as T) : defaultValue;
|
||||||
|
}).subscribe({
|
||||||
|
next: (v) => set(v),
|
||||||
|
error: (e) => console.error('[settings] liveQuery error', e),
|
||||||
|
});
|
||||||
|
return () => subscription.unsubscribe();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export const searchResultsLimit = settingStore<number>('searchResultsLimit', 3);
|
||||||
|
|
@ -0,0 +1,7 @@
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS fts_history USING fts5(id UNINDEXED, user_id UNINDEXED, text, date UNINDEXED, topic_id UNINDEXED, content='history_entries', content_rowid='rowid', tokenize='unicode61');
|
||||||
|
--> statement-breakpoint
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS fts_pages USING fts5(id UNINDEXED, user_id UNINDEXED, title, body, content='pages', content_rowid='rowid', tokenize='unicode61');
|
||||||
|
--> statement-breakpoint
|
||||||
|
INSERT INTO fts_history(fts_history) VALUES('rebuild');
|
||||||
|
--> statement-breakpoint
|
||||||
|
INSERT INTO fts_pages(fts_pages) VALUES('rebuild');
|
||||||
|
|
@ -92,6 +92,13 @@
|
||||||
"when": 1772004047537,
|
"when": 1772004047537,
|
||||||
"tag": "0012_chunky_stature",
|
"tag": "0012_chunky_stature",
|
||||||
"breakpoints": true
|
"breakpoints": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idx": 13,
|
||||||
|
"version": "6",
|
||||||
|
"when": 1772100000000,
|
||||||
|
"tag": "0013_fts_search",
|
||||||
|
"breakpoints": true
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
Binary file not shown.
Binary file not shown.
|
|
@ -21,6 +21,7 @@ sqlite.pragma('journal_mode = WAL');
|
||||||
sqlite.pragma('foreign_keys = ON');
|
sqlite.pragma('foreign_keys = ON');
|
||||||
|
|
||||||
export const db = drizzle(sqlite, { schema });
|
export const db = drizzle(sqlite, { schema });
|
||||||
|
export { sqlite };
|
||||||
|
|
||||||
// Run migrations on startup
|
// Run migrations on startup
|
||||||
const migrationsFolder = path.resolve(__dirname, '../../drizzle');
|
const migrationsFolder = path.resolve(__dirname, '../../drizzle');
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,9 @@ import adminRoutes from './routes/admin.js';
|
||||||
import pushRoutes from './routes/push.js';
|
import pushRoutes from './routes/push.js';
|
||||||
import backupRoutes from './routes/backup.js';
|
import backupRoutes from './routes/backup.js';
|
||||||
import apiKeyRoutes from './routes/api-keys.js';
|
import apiKeyRoutes from './routes/api-keys.js';
|
||||||
|
import searchRoutes from './routes/search.js';
|
||||||
import { runScheduledBackup, runIfMissed, checkIntegrity } from './lib/backup-service.js';
|
import { runScheduledBackup, runIfMissed, checkIntegrity } from './lib/backup-service.js';
|
||||||
|
import { sqlite } from './db/connection.js';
|
||||||
|
|
||||||
const app = new OpenAPIHono();
|
const app = new OpenAPIHono();
|
||||||
|
|
||||||
|
|
@ -27,13 +29,26 @@ app.onError((err, c) => {
|
||||||
return c.json({ error: 'internal server error', detail: err.message }, 500);
|
return c.json({ error: 'internal server error', detail: err.message }, 500);
|
||||||
});
|
});
|
||||||
|
|
||||||
// Integrity check at startup
|
// Integrity check + FTS consistency check at startup
|
||||||
setImmediate(() => {
|
setImmediate(() => {
|
||||||
try {
|
try {
|
||||||
checkIntegrity();
|
checkIntegrity();
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
console.error('[db] startup integrity_check threw:', e);
|
console.error('[db] startup integrity_check threw:', e);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const heCount = (sqlite.prepare('SELECT COUNT(*) AS n FROM history_entries WHERE deleted_at IS NULL').get() as { n: number }).n;
|
||||||
|
const ftsCount = (sqlite.prepare('SELECT COUNT(*) AS n FROM fts_history').get() as { n: number }).n;
|
||||||
|
if (Math.abs(heCount - ftsCount) > 10) {
|
||||||
|
console.warn(`[fts] Index mismatch (history_entries=${heCount}, fts_history=${ftsCount}), rebuilding...`);
|
||||||
|
sqlite.prepare("INSERT INTO fts_history(fts_history) VALUES('rebuild')").run();
|
||||||
|
sqlite.prepare("INSERT INTO fts_pages(fts_pages) VALUES('rebuild')").run();
|
||||||
|
console.log('[fts] Rebuild complete');
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
console.error('[fts] startup check threw:', e);
|
||||||
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
// Public routes
|
// Public routes
|
||||||
|
|
@ -81,6 +96,10 @@ app.route('/api/backup', backupRoutes);
|
||||||
app.use('/api/api-keys/*', authMiddleware);
|
app.use('/api/api-keys/*', authMiddleware);
|
||||||
app.route('/api/api-keys', apiKeyRoutes);
|
app.route('/api/api-keys', apiKeyRoutes);
|
||||||
|
|
||||||
|
app.use('/api/search/*', authMiddleware);
|
||||||
|
app.use('/api/search', authMiddleware);
|
||||||
|
app.route('/api/search', searchRoutes);
|
||||||
|
|
||||||
// OpenAPI spec + Scalar UI
|
// OpenAPI spec + Scalar UI
|
||||||
app.openAPIRegistry.registerComponent('securitySchemes', 'BearerAuth', {
|
app.openAPIRegistry.registerComponent('securitySchemes', 'BearerAuth', {
|
||||||
type: 'http',
|
type: 'http',
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import { randomUUID } from 'crypto';
|
import { randomUUID } from 'crypto';
|
||||||
import { db } from '../db/connection.js';
|
import { db, sqlite } from '../db/connection.js';
|
||||||
import { aiLocks, contexts, topics, historyEntries, ratings, imageBlobs, pages, notebooks, pageNotebooks } from '../db/schema.js';
|
import { aiLocks, contexts, topics, historyEntries, ratings, imageBlobs, pages, notebooks, pageNotebooks } from '../db/schema.js';
|
||||||
import { eq, and, sql } from 'drizzle-orm';
|
import { eq, and, sql } from 'drizzle-orm';
|
||||||
import { zipSync, unzipSync, strToU8, strFromU8 } from 'fflate';
|
import { zipSync, unzipSync, strToU8, strFromU8 } from 'fflate';
|
||||||
|
|
@ -8,6 +8,34 @@ import { generateAiReadme } from './ai-agent-readme.js';
|
||||||
|
|
||||||
const LOCK_EXPIRY_HOURS = Number(process.env.AI_LOCK_EXPIRY_HOURS ?? 24);
|
const LOCK_EXPIRY_HOURS = Number(process.env.AI_LOCK_EXPIRY_HOURS ?? 24);
|
||||||
|
|
||||||
|
const stmtFtsHistoryUpsert = sqlite.prepare(`INSERT OR REPLACE INTO fts_history(rowid, id, user_id, text, date, topic_id) SELECT rowid, id, user_id, text, date, topic_id FROM history_entries WHERE id = ? AND user_id = ?`);
|
||||||
|
const stmtFtsHistoryDelete = sqlite.prepare(`DELETE FROM fts_history WHERE id = ? AND user_id = ?`);
|
||||||
|
const stmtFtsPagesUpsert = sqlite.prepare(`INSERT OR REPLACE INTO fts_pages(rowid, id, user_id, title, body) SELECT rowid, id, user_id, title, body FROM pages WHERE id = ? AND user_id = ?`);
|
||||||
|
const stmtFtsPagesDelete = sqlite.prepare(`DELETE FROM fts_pages WHERE id = ? AND user_id = ?`);
|
||||||
|
|
||||||
|
type AiTableDef = typeof contexts | typeof topics | typeof historyEntries | typeof ratings | typeof imageBlobs | typeof pages | typeof notebooks | typeof pageNotebooks;
|
||||||
|
|
||||||
|
async function applyOps(ops: Array<{ action: 'insert' | 'update'; table: AiTableDef; row: Record<string, unknown> }>, userId: string): Promise<number> {
|
||||||
|
let accepted = 0;
|
||||||
|
for (const op of ops) {
|
||||||
|
if (op.action === 'insert') {
|
||||||
|
await db.insert(op.table).values(op.row as never);
|
||||||
|
} else {
|
||||||
|
await db.update(op.table).set(op.row as never)
|
||||||
|
.where(and(sql`${op.table.id} = ${op.row.id}`, sql`${op.table.userId} = ${userId}`));
|
||||||
|
}
|
||||||
|
accepted++;
|
||||||
|
if (op.table === historyEntries) {
|
||||||
|
if (op.row.deletedAt) stmtFtsHistoryDelete.run(op.row.id, userId);
|
||||||
|
else stmtFtsHistoryUpsert.run(op.row.id, userId);
|
||||||
|
} else if (op.table === pages) {
|
||||||
|
if (op.row.deletedAt) stmtFtsPagesDelete.run(op.row.id, userId);
|
||||||
|
else stmtFtsPagesUpsert.run(op.row.id, userId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return accepted;
|
||||||
|
}
|
||||||
|
|
||||||
function now(): string {
|
function now(): string {
|
||||||
return new Date().toISOString();
|
return new Date().toISOString();
|
||||||
}
|
}
|
||||||
|
|
@ -195,7 +223,7 @@ export interface AiUploadResult {
|
||||||
conflicts: Array<{ entityType: string; entityId: string; clientVersion: number; serverVersion: number }>;
|
conflicts: Array<{ entityType: string; entityId: string; clientVersion: number; serverVersion: number }>;
|
||||||
}
|
}
|
||||||
|
|
||||||
type TableDef = typeof contexts | typeof topics | typeof historyEntries | typeof ratings | typeof imageBlobs | typeof pages | typeof notebooks | typeof pageNotebooks;
|
type TableDef = AiTableDef;
|
||||||
|
|
||||||
async function checkConflict(
|
async function checkConflict(
|
||||||
table: TableDef,
|
table: TableDef,
|
||||||
|
|
@ -389,17 +417,7 @@ export async function applyUploadFromZip(
|
||||||
return { result: { accepted: 0, skipped, conflicts }, conflict: true };
|
return { result: { accepted: 0, skipped, conflicts }, conflict: true };
|
||||||
}
|
}
|
||||||
|
|
||||||
let accepted = 0;
|
const accepted = await applyOps(ops, userId);
|
||||||
for (const op of ops) {
|
|
||||||
if (op.action === 'insert') {
|
|
||||||
await db.insert(op.table).values(op.row as never);
|
|
||||||
} else {
|
|
||||||
await db.update(op.table).set(op.row as never)
|
|
||||||
.where(and(sql`${op.table.id} = ${op.row.id}`, sql`${op.table.userId} = ${userId}`));
|
|
||||||
}
|
|
||||||
accepted++;
|
|
||||||
}
|
|
||||||
|
|
||||||
return { result: { accepted, skipped, conflicts }, conflict: false };
|
return { result: { accepted, skipped, conflicts }, conflict: false };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -489,16 +507,6 @@ export async function applyUpload(
|
||||||
return { result: { accepted: 0, skipped, conflicts }, conflict: true };
|
return { result: { accepted: 0, skipped, conflicts }, conflict: true };
|
||||||
}
|
}
|
||||||
|
|
||||||
let accepted = 0;
|
const accepted = await applyOps(ops, userId);
|
||||||
for (const op of ops) {
|
|
||||||
if (op.action === 'insert') {
|
|
||||||
await db.insert(op.table).values(op.row as never);
|
|
||||||
} else {
|
|
||||||
await db.update(op.table).set(op.row as never)
|
|
||||||
.where(and(sql`${op.table.id} = ${op.row.id}`, sql`${op.table.userId} = ${userId}`));
|
|
||||||
}
|
|
||||||
accepted++;
|
|
||||||
}
|
|
||||||
|
|
||||||
return { result: { accepted, skipped, conflicts }, conflict: false };
|
return { result: { accepted, skipped, conflicts }, conflict: false };
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
import { db } from '../db/connection.js';
|
import { db, sqlite } from '../db/connection.js';
|
||||||
import { contexts, topics, historyEntries, ratings, imageBlobs, pages, notebooks, pageNotebooks } from '../db/schema.js';
|
import { contexts, topics, historyEntries, ratings, imageBlobs, pages, notebooks, pageNotebooks } from '../db/schema.js';
|
||||||
import { and, gt, eq, sql, isNotNull, lt } from 'drizzle-orm';
|
import { and, gt, eq, sql, isNotNull, lt } from 'drizzle-orm';
|
||||||
import type {
|
import type {
|
||||||
|
|
@ -8,6 +8,24 @@ import type {
|
||||||
|
|
||||||
type TableDef = typeof contexts | typeof topics | typeof historyEntries | typeof ratings | typeof imageBlobs | typeof pages | typeof notebooks | typeof pageNotebooks;
|
type TableDef = typeof contexts | typeof topics | typeof historyEntries | typeof ratings | typeof imageBlobs | typeof pages | typeof notebooks | typeof pageNotebooks;
|
||||||
|
|
||||||
|
// FTS index maintenance (prepared statements for performance)
|
||||||
|
const stmtFtsHistoryUpsert = sqlite.prepare(`
|
||||||
|
INSERT OR REPLACE INTO fts_history(rowid, id, user_id, text, date, topic_id)
|
||||||
|
SELECT rowid, id, user_id, text, date, topic_id
|
||||||
|
FROM history_entries WHERE id = ? AND user_id = ?
|
||||||
|
`);
|
||||||
|
const stmtFtsHistoryDelete = sqlite.prepare(`
|
||||||
|
DELETE FROM fts_history WHERE id = ? AND user_id = ?
|
||||||
|
`);
|
||||||
|
const stmtFtsPagesUpsert = sqlite.prepare(`
|
||||||
|
INSERT OR REPLACE INTO fts_pages(rowid, id, user_id, title, body)
|
||||||
|
SELECT rowid, id, user_id, title, body
|
||||||
|
FROM pages WHERE id = ? AND user_id = ?
|
||||||
|
`);
|
||||||
|
const stmtFtsPagesDelete = sqlite.prepare(`
|
||||||
|
DELETE FROM fts_pages WHERE id = ? AND user_id = ?
|
||||||
|
`);
|
||||||
|
|
||||||
function now(): string {
|
function now(): string {
|
||||||
return new Date().toISOString();
|
return new Date().toISOString();
|
||||||
}
|
}
|
||||||
|
|
@ -242,7 +260,14 @@ export async function pushChanges(request: SyncPushRequest, userId: string): Pro
|
||||||
purgedAt: he.purgedAt ?? null,
|
purgedAt: he.purgedAt ?? null,
|
||||||
version: he.version,
|
version: he.version,
|
||||||
};
|
};
|
||||||
if (await upsertEntity(historyEntries, row, conflicts, 'historyEntry', userId)) accepted++;
|
if (await upsertEntity(historyEntries, row, conflicts, 'historyEntry', userId)) {
|
||||||
|
accepted++;
|
||||||
|
if (he.deletedAt) {
|
||||||
|
stmtFtsHistoryDelete.run(he.id, userId);
|
||||||
|
} else {
|
||||||
|
stmtFtsHistoryUpsert.run(he.id, userId);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for (const rat of rats) {
|
for (const rat of rats) {
|
||||||
|
|
@ -278,7 +303,14 @@ export async function pushChanges(request: SyncPushRequest, userId: string): Pro
|
||||||
|
|
||||||
for (const pg of pgs) {
|
for (const pg of pgs) {
|
||||||
const row = { id: pg.id, userId, title: pg.title, body: pg.body, isPrivate: pg.isPrivate, isFavorite: pg.isFavorite ?? false, sortOrder: pg.sortOrder, updatedAt: pg.updatedAt, deletedAt: pg.deletedAt, purgedAt: pg.purgedAt ?? null, version: pg.version };
|
const row = { id: pg.id, userId, title: pg.title, body: pg.body, isPrivate: pg.isPrivate, isFavorite: pg.isFavorite ?? false, sortOrder: pg.sortOrder, updatedAt: pg.updatedAt, deletedAt: pg.deletedAt, purgedAt: pg.purgedAt ?? null, version: pg.version };
|
||||||
if (await upsertEntity(pages, row, conflicts, 'page', userId)) accepted++;
|
if (await upsertEntity(pages, row, conflicts, 'page', userId)) {
|
||||||
|
accepted++;
|
||||||
|
if (pg.deletedAt) {
|
||||||
|
stmtFtsPagesDelete.run(pg.id, userId);
|
||||||
|
} else {
|
||||||
|
stmtFtsPagesUpsert.run(pg.id, userId);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for (const nb of nbs) {
|
for (const nb of nbs) {
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,80 @@
|
||||||
|
import { Hono } from 'hono';
|
||||||
|
import { sqlite } from '../db/connection.js';
|
||||||
|
import type { AuthEnv } from '../middleware/auth.js';
|
||||||
|
|
||||||
|
const search = new Hono<AuthEnv>();
|
||||||
|
|
||||||
|
interface HistoryResult {
|
||||||
|
id: string;
|
||||||
|
topicId: string;
|
||||||
|
date: string;
|
||||||
|
snippet: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PageResult {
|
||||||
|
id: string;
|
||||||
|
title: string;
|
||||||
|
snippet: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const stmtSearchHistory = sqlite.prepare(`
|
||||||
|
SELECT
|
||||||
|
id,
|
||||||
|
topic_id AS topicId,
|
||||||
|
date,
|
||||||
|
snippet(fts_history, 2, '<mark>', '</mark>', '...', 12) AS snippet
|
||||||
|
FROM fts_history
|
||||||
|
WHERE fts_history MATCH ?
|
||||||
|
AND user_id = ?
|
||||||
|
AND id NOT IN (
|
||||||
|
SELECT id FROM history_entries WHERE deleted_at IS NOT NULL AND user_id = ?
|
||||||
|
)
|
||||||
|
ORDER BY rank
|
||||||
|
LIMIT ?
|
||||||
|
`);
|
||||||
|
|
||||||
|
const stmtSearchPages = sqlite.prepare(`
|
||||||
|
SELECT
|
||||||
|
id,
|
||||||
|
title,
|
||||||
|
snippet(fts_pages, 2, '<mark>', '</mark>', '...', 12) AS snippet
|
||||||
|
FROM fts_pages
|
||||||
|
WHERE fts_pages MATCH ?
|
||||||
|
AND user_id = ?
|
||||||
|
AND id NOT IN (
|
||||||
|
SELECT id FROM pages WHERE deleted_at IS NOT NULL AND user_id = ?
|
||||||
|
)
|
||||||
|
ORDER BY rank
|
||||||
|
LIMIT ?
|
||||||
|
`);
|
||||||
|
|
||||||
|
search.get('/', (c) => {
|
||||||
|
const auth = c.get('auth');
|
||||||
|
const userId = auth.userId;
|
||||||
|
|
||||||
|
const q = c.req.query('q')?.trim() ?? '';
|
||||||
|
const limit = Math.min(Number(c.req.query('limit') ?? 5), 20);
|
||||||
|
|
||||||
|
if (q.length < 2) {
|
||||||
|
return c.json({ history: [], pages: [] });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build FTS5 prefix query: each word gets a trailing * for prefix matching
|
||||||
|
const ftsQuery = q
|
||||||
|
.split(/\s+/)
|
||||||
|
.filter(Boolean)
|
||||||
|
.map((t) => `"${t.replace(/"/g, '')}"*`)
|
||||||
|
.join(' ');
|
||||||
|
|
||||||
|
try {
|
||||||
|
const history = stmtSearchHistory.all(ftsQuery, userId, userId, limit) as HistoryResult[];
|
||||||
|
const pages = stmtSearchPages.all(ftsQuery, userId, userId, limit) as PageResult[];
|
||||||
|
return c.json({ history, pages });
|
||||||
|
} catch (err) {
|
||||||
|
// FTS5 query syntax errors (e.g. special chars) → return empty rather than 500
|
||||||
|
console.warn('[search] FTS query error:', err instanceof Error ? err.message : err);
|
||||||
|
return c.json({ history: [], pages: [] });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
export default search;
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import { Hono } from 'hono';
|
import { Hono } from 'hono';
|
||||||
import { db } from '../db/connection.js';
|
import { db, sqlite } from '../db/connection.js';
|
||||||
import { contexts, topics, historyEntries, ratings } from '../db/schema.js';
|
import { contexts, topics, historyEntries, ratings } from '../db/schema.js';
|
||||||
import { and, eq, inArray, sql } from 'drizzle-orm';
|
import { and, eq, inArray, sql } from 'drizzle-orm';
|
||||||
import { handle } from '../lib/route-utils.js';
|
import { handle } from '../lib/route-utils.js';
|
||||||
|
|
@ -74,6 +74,8 @@ trash.delete('/', handle('trash/delete', async (c) => {
|
||||||
if (historyIds.size > 0) {
|
if (historyIds.size > 0) {
|
||||||
await db.update(historyEntries).set({ deletedAt: ts, purgedAt: ts, updatedAt: ts, version: sql`${historyEntries.version} + 1` })
|
await db.update(historyEntries).set({ deletedAt: ts, purgedAt: ts, updatedAt: ts, version: sql`${historyEntries.version} + 1` })
|
||||||
.where(and(eq(historyEntries.userId, userId), inArray(historyEntries.id, [...historyIds])));
|
.where(and(eq(historyEntries.userId, userId), inArray(historyEntries.id, [...historyIds])));
|
||||||
|
const stmtDel = sqlite.prepare('DELETE FROM fts_history WHERE id = ? AND user_id = ?');
|
||||||
|
for (const id of historyIds) stmtDel.run(id, userId);
|
||||||
}
|
}
|
||||||
if (topicIds.size > 0) {
|
if (topicIds.size > 0) {
|
||||||
await db.update(topics).set({ deletedAt: ts, purgedAt: ts, updatedAt: ts, version: sql`${topics.version} + 1` })
|
await db.update(topics).set({ deletedAt: ts, purgedAt: ts, updatedAt: ts, version: sql`${topics.version} + 1` })
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue