Commit Graph

50 Commits

Author SHA1 Message Date
Nathan.fooo 3211a93caa
chore: update expire time for database row (#1158) 2025-01-14 10:15:34 +08:00
nathan 1a5c1714db Merge branch 'main' into stateless 2024-12-24 14:59:04 +08:00
Nathan.fooo 1131818eb7
fix: support index document in the background, fix stack overflow when calling rayon::spawn (#1099)
* chore: batch index

* chore: format log

* chore: index workspace

* chore: fix stack overflow

* chore: background index

* chore: clippy

* chore: filter tasks

* chore: clippy

* chore: add metrics

* chore: fix test
2024-12-24 14:30:17 +08:00
Bartosz Sypytkowski e85e60c0f2 chore: revert from editing collab 2024-12-17 05:43:04 +01:00
Bartosz Sypytkowski d120860312 chore: remove state vector from redis udpates 2024-12-17 04:29:41 +01:00
Bartosz Sypytkowski 1b35326f0a Merge branch 'main' into stateless 2024-12-16 04:47:26 +01:00
Nathan.fooo 2ab1e94b99
chore: create embed in background (#1071)
* refactor: create indexer scheduler

* refactor: replace current calls

* chore: use rayon thread pool

* chore: index unindexed collab using thread pool

* chore: index collab when write to disk

* chore: add test

* chore: add test

* chore: add test

* chore: fix client api deps

* chore: chat with document test

* chore: fix test

* chore: clippy

* chore: update test
2024-12-15 12:29:51 +08:00
Bartosz Sypytkowski ed07104aeb chore: remove from editing collab param in get encode collab method 2024-12-14 09:42:15 +01:00
Bartosz Sypytkowski 41e9302ff9 Merge branch 'main' into stateless 2024-12-12 16:00:45 +01:00
Nathan.fooo af38efe6d9
feat: sync document through http request (#1064)
* chore: query embedding

* chore: create embeddings

* chore: apply update to editing collab

* refactor: web-update

* chore: calculate missing update the sv is not none

* chore: add test

* chore: fix audit

* chore: commit sqlx

* chore: fix client api

* test: add

* chore: clippy

* chore: fix collab drop when save
2024-12-12 14:53:07 +08:00
Zack 254bc23a6e
feat: insert database row into selected database (#1029)
* feat: insert database row into selected database

* feat: inserting database row

* feat: add impl for cell

* feat: insert row selection and mulitselection

* feat: support datetime field type

* fix: other fields like rich text

* feat: add database field server impl

* feat: add client api and tests

* feat: use to json value impl from collab

* feat: use add database cell impl from collab

* feat: update to latest collab

* chore: upgrade collab and fix tests

* chore: review issues

* fix: code review feedback

* feat: filter only allow supported field types

* feat: support more field types

* feat: support created at and last modified

* feat: add timestamp cell for created at and modified at fields

* chore: linting and formatting

* fix: add created at and last modified

* fix: ci add service dependency of appflowy cloud on admin frontend

* chore: trigger ci

* feat: add logging for admin frontend client signin

* fix: server logs if error

* fix: create admin confirmation without email

* fix: ci add service dependency of appflowy cloud on admin frontend

* fix: server logs if error

* chore: update collab
2024-12-11 15:47:29 +08:00
Bartosz Sypytkowski c4514ca5bf chore: add explanation 2024-12-10 10:12:33 +01:00
Bartosz Sypytkowski 9971c23749 fix: update redis cache when inserting collabs in batches 2024-12-10 10:07:54 +01:00
Nathan.fooo a598aed5ec
chore: remove empty map when query is empty (#1050) 2024-12-07 10:20:53 +08:00
Bartosz Sypytkowski 66deaf3fa3
chore: move collab cache for appflowy collaborate service and add metrics (#1045)
* chore: move collab cache for appflowy collaborate service and add metrics

* chore: add histogram for measuring time spent in postgres transactions

* chore: delete blobs in chunks

* chore: bring back redis cache filling by import worker

* chore: cargo fmt

* chore: make import workspace update existing collabs

* chore: bulk insert collab deduplicate records
2024-12-06 19:55:51 +08:00
khorshuheng aa64b7cd6b fix: web update should update both disk and mem cache 2024-12-05 16:51:56 +08:00
Bartosz Sypytkowski 5e7b5f87fa chore: merge branch main into stateless 2024-12-03 09:10:01 +01:00
Bartosz Sypytkowski 9ff6f1c744
S3 collab cache (#1028)
* chore: collab cache for S3

* chore: adjust disk cache api to accomodate s3

* chore: move postgres dependent ops to disk cache

* chore: replace blob inserts from pg to s3

* chore: delete blob and collab exist now use s3

* chore: fix clippy erorrs

* chore: post rebase fixes

* chore: fix clippy warnings

* chore: fix imports

* chore: make snapshots work over S3

* chore: remove dead code

* chore: use compressed snapshots

* chore: add zstd compression

* chore: introduce collab size threshold to keep smaller collabs in postgres

* chore: remove collabs from S3 if they were put to postgres

* chore: update tests
2024-12-03 13:08:55 +08:00
Bartosz Sypytkowski 1932607ca9 Merge branch 'main' into stateless 2024-11-18 05:43:16 +01:00
Bartosz Sypytkowski 3799966f12
chore: store pending collab writes in memory (#1000)
* chore: write immediatelly actually writes immediatelly

* chore: fix clippy errors

* chore: add metrics to new storage queue impl

* chore: set collab batch write capacity to the same as on main branch
2024-11-17 13:25:42 +08:00
Bartosz Sypytkowski 401cc3ad25 Merge branch 'main' into stateless 2024-11-16 07:22:34 +01:00
Khor Shu Heng d0c212ff10
refactor: remove enforcer group (#948) 2024-11-15 14:08:12 +08:00
Bartosz Sypytkowski 3e612e2db6 chore: create method for receiving the most recent snapshot 2024-11-05 07:47:17 +01:00
Bartosz Sypytkowski f9397eaaf2 chore: remove realtime shared state 2024-11-04 07:58:16 +01:00
Khor Shu Heng 9eb1c36c92
chore: access control interface to return error instead of boolean (#914) 2024-10-21 20:20:14 +08:00
khorshuheng 383629ab1c chore: simplify not enough permission error 2024-10-16 15:10:31 +08:00
Khor Shu Heng 8c14612f98
feat: refactor access control to enable swapping implementation (#859) 2024-10-10 13:04:47 +08:00
Nathan.fooo 33e63fae5c
chore: try to fix pg lock timeout (#864)
* chore: try to fix pg lock timeout

* chore: add logs for insert collab

* chore: add timeout for duplicate

* chore: timeout for pending write
2024-10-07 16:49:50 +08:00
Nathan.fooo 38ff1874e4
chore: using copy to improve write speed of batch collab api endpoint (#862)
* chore: using copy to improve write speed of batch collab api endpoint

* chore: insert to redis

* chore: fix compile
2024-10-07 08:58:19 +08:00
Nathan.fooo 9c0dffacef
feat: implement import notion zip file api endpoint (#840)
* chore: implement api endpoint

* chore: worker

* chore: async zip

* chore: test bulk insert

* chore: insert collab

* chore: custom task

* chore: consume un acked message

* chore: fix compile

* chore: add test

* test: update

* chore: save import record

* chore: save import record

* chore: fix ci

* chore: remove unused deps

* chore: update docker file

* chore: build worker images

* chore: use small int

* chore: use small int

* chore: rm protoc deps in runtime

* chore: move collab cache to database crate

* chore: update test

* chore: rm health api endpoint

* chore: clippy

* chore: update ci

* chore: add test

* chore: upgrade collab

* chore: clippy

* chore: update test

* chore: use custom host

* chore: config nginx

* chore: install cert
2024-10-03 22:22:29 +08:00
Khor Shu Heng 11732324b5
feat: support fetching encoded collab in batch from memory (#837) 2024-10-02 10:22:40 +08:00
Nathan.fooo a4b885e13b
chore: set exp time for collab base on collab type (#820)
* chore: set exp time for collab base on collab type

* chore: fix test
2024-09-13 19:04:56 +08:00
Nathan.fooo 5d356718fa
fix: using spawn_blocking in order to not block the runtime (#793)
* fix: using blocking in order to not block the runtime

* chore: increase deps count
2024-09-05 09:19:34 +08:00
Zack 826546c5cb
feat: duplicate published doc (#700)
* chore: test collab sync

* feat: folder view for user workspace

* feat: add private indicator

* chore: use collab cache instead

* chore: initial impl

* chore: cargo sqlx

* fix: write back to collab cache

* fix: assign page id

* fix: text map

* chore: connect api and refactor

* chore: switch to using mem storage

* fix: collab type

* feat: use group manager to manage sync

* feat: try add send command instead

* chore: add client api code

* feat: try use sink and stream from group collab

* chore: disable sync first

* fix: insert page before stopping group

* feat: add extra for view

* feat: add metadata to doc

* fix: icon

* fix: page circular dep

* fix: page circular dep

* fix: live update

* feat: database initial impl

* feat: workspace database

* chore: link database

* fix: database validation

* fix: workspace database oid

* fix: workspace database oid

* fix: specify view layout

* feat: add txn when inserting collab and rework database views

* fix: parent view id for new database

* fix: database view for folder

* fix: database ids

* fix: database row id

* fix: main database layout and write database before folder

* fix: metadata for non database main view

* chore: database in doc wip

* chore: filter visible views

* chore: use structure value instead of json

* chore: database in doc wip

* fix: remove child item push

* fix: main database view meta

* chore: rework inline database in doc

* fix: make duplicate more error forgiving

* chore: use get instead of get key value

* chore: merge with main

* feat: set max depth for folder view

* chore: update dependency counter

* chore: remove unneeded arg

* chore: simplify collab new from source

* chore: remove unneeded param

* chore: rename and doc

* feat: rework inline database and use duplicated ref

* chore: simplify params

* chore: fmt

* feat: deduplicate database wip

* fix: compatible with newer appflowy version

* fix: database views

* feat: database in doc

* chore: cargo clippy

* chore: update dockerfile rust toolchain

* fix: doc inline db parent id

* fix: database id

* fix: document data modification

* chore: minor refactor and todo

* chore: remove unneeded nesting

* fix: enforce ordering for views to add

* chore: add doc and test case for folder view

* fix: remove the need for cloning published collab

* fix: some review

* chore: move folder conversion logic

* fix: remove collab folder deps on shared entity

* chore: cargo fmt

* fix: compile

* chore: remove group manager dependency

* fix: dep count

* fix: add messages sent by server

* fix: dep count

* chore: add test for doc ref and inline doc database

* chore: cargo clippy

* chore: add more test scenarios

* chore: fix tests

* chore: get database id

* chore: update collab

* chore: add more assert and database row id checks

* fix: suggestions from review

* chore: sqlx

* fix: accumulate collab before insert

* chore: add tokio spawn blocking for encoding

* fix: reduce limit for publish collabs

---------

Co-authored-by: Bartosz Sypytkowski <b.sypytkowski@gmail.com>
Co-authored-by: nathan <nathan@appflowy.io>
2024-09-03 09:12:29 +08:00
Nathan.fooo 365c64c5c2
chore: use colla storage instead of select from db directly (#782) 2024-09-02 21:25:21 +08:00
Nathan.fooo 299680c14a
chore: decode encode collab using spawn blocking (#781) 2024-09-02 20:27:40 +08:00
Bartosz Sypytkowski 2af1999375
chore: added locks with timeouts (#765)
* chore: added locks with timeouts

* chore: roll back collab locks in client api
2024-08-29 16:13:27 +08:00
Nathan.fooo a21c55cbcc
chore: modify log level for some cases (#754)
* chore: modify log level for some cases

* chore: update logs
2024-08-27 14:57:00 +08:00
Nathan.fooo 8cc410ce65
chore: revamp get collab api request (#752)
* chore: stop retry if record is not found
2024-08-26 23:41:06 +08:00
nathan b4d79b954b chore: update logs 2024-08-23 10:30:52 +08:00
Nathan.fooo 1a053172c2
chore: write collab to disk after first sync step2 (#739)
* chore: write collab to disk after first sync step2

* chore: clippy

* chore: fix test & disable history plugin

* chore: upgrade appflowy collab that disbale document undo redo
2024-08-22 23:11:20 +08:00
Bartosz Sypytkowski f2fa4d7c22 chore: post rebase fixes 2024-08-14 05:26:38 +02:00
Bartosz Sypytkowski a706c312b3 chore: making embeddings code compile 2024-06-28 05:40:41 +02:00
Bartosz Sypytkowski b79fe41af3 chore: moving embeddings into collabparams 2024-06-28 05:39:56 +02:00
Bartosz Sypytkowski b1ebbecc5f chore: moving appflowy indexer into appflowy cloud 2024-06-28 05:39:53 +02:00
Nathan.fooo a6539dd963
chore: shorten default write interval (#661) 2024-06-28 10:06:09 +08:00
Bartosz Sypytkowski 7ad105ee34 fix: index all of the document blocks 2024-06-14 11:23:19 +02:00
Khor Shu Heng 67d9fad7d7
chore: migrate collab storage to appflowy-collaborate (#566)
* chore: migrate collab storage to appflowy-collaborate

* fix: clippy error

* chore: remove handler

---------

Co-authored-by: nathan <nathan@appflowy.io>
2024-05-21 10:06:26 +08:00
Khor Shu Heng c79267d42a
feat: migrate collab access control to collaboration service (#548)
* feat: migrate collab access control to collaborate service

* feat: migrate workspace access control to separate crate
2024-05-15 15:24:07 +08:00
Khor Shu Heng d9d874ab8a
feat: migrate collab cache to collaborate service (#524)
Co-authored-by: nathan <nathan@appflowy.io>
2024-05-09 23:12:13 +08:00