Commit Graph

55 Commits

Author SHA1 Message Date
khorshuheng aa64b7cd6b fix: web update should update both disk and mem cache 2024-12-05 16:51:56 +08:00
Bartosz Sypytkowski 9ff6f1c744
S3 collab cache (#1028)
* chore: collab cache for S3

* chore: adjust disk cache api to accomodate s3

* chore: move postgres dependent ops to disk cache

* chore: replace blob inserts from pg to s3

* chore: delete blob and collab exist now use s3

* chore: fix clippy erorrs

* chore: post rebase fixes

* chore: fix clippy warnings

* chore: fix imports

* chore: make snapshots work over S3

* chore: remove dead code

* chore: use compressed snapshots

* chore: add zstd compression

* chore: introduce collab size threshold to keep smaller collabs in postgres

* chore: remove collabs from S3 if they were put to postgres

* chore: update tests
2024-12-03 13:08:55 +08:00
khorshuheng 484cd595e0 feat: add support for create database pages via API 2024-11-26 15:58:16 +08:00
Bartosz Sypytkowski f9397eaaf2 chore: remove realtime shared state 2024-11-04 07:58:16 +01:00
Khor Shu Heng 9eb1c36c92
chore: access control interface to return error instead of boolean (#914) 2024-10-21 20:20:14 +08:00
Nathan.fooo 33e63fae5c
chore: try to fix pg lock timeout (#864)
* chore: try to fix pg lock timeout

* chore: add logs for insert collab

* chore: add timeout for duplicate

* chore: timeout for pending write
2024-10-07 16:49:50 +08:00
Nathan.fooo 38ff1874e4
chore: using copy to improve write speed of batch collab api endpoint (#862)
* chore: using copy to improve write speed of batch collab api endpoint

* chore: insert to redis

* chore: fix compile
2024-10-07 08:58:19 +08:00
Khor Shu Heng 11732324b5
feat: support fetching encoded collab in batch from memory (#837) 2024-10-02 10:22:40 +08:00
Zack 826546c5cb
feat: duplicate published doc (#700)
* chore: test collab sync

* feat: folder view for user workspace

* feat: add private indicator

* chore: use collab cache instead

* chore: initial impl

* chore: cargo sqlx

* fix: write back to collab cache

* fix: assign page id

* fix: text map

* chore: connect api and refactor

* chore: switch to using mem storage

* fix: collab type

* feat: use group manager to manage sync

* feat: try add send command instead

* chore: add client api code

* feat: try use sink and stream from group collab

* chore: disable sync first

* fix: insert page before stopping group

* feat: add extra for view

* feat: add metadata to doc

* fix: icon

* fix: page circular dep

* fix: page circular dep

* fix: live update

* feat: database initial impl

* feat: workspace database

* chore: link database

* fix: database validation

* fix: workspace database oid

* fix: workspace database oid

* fix: specify view layout

* feat: add txn when inserting collab and rework database views

* fix: parent view id for new database

* fix: database view for folder

* fix: database ids

* fix: database row id

* fix: main database layout and write database before folder

* fix: metadata for non database main view

* chore: database in doc wip

* chore: filter visible views

* chore: use structure value instead of json

* chore: database in doc wip

* fix: remove child item push

* fix: main database view meta

* chore: rework inline database in doc

* fix: make duplicate more error forgiving

* chore: use get instead of get key value

* chore: merge with main

* feat: set max depth for folder view

* chore: update dependency counter

* chore: remove unneeded arg

* chore: simplify collab new from source

* chore: remove unneeded param

* chore: rename and doc

* feat: rework inline database and use duplicated ref

* chore: simplify params

* chore: fmt

* feat: deduplicate database wip

* fix: compatible with newer appflowy version

* fix: database views

* feat: database in doc

* chore: cargo clippy

* chore: update dockerfile rust toolchain

* fix: doc inline db parent id

* fix: database id

* fix: document data modification

* chore: minor refactor and todo

* chore: remove unneeded nesting

* fix: enforce ordering for views to add

* chore: add doc and test case for folder view

* fix: remove the need for cloning published collab

* fix: some review

* chore: move folder conversion logic

* fix: remove collab folder deps on shared entity

* chore: cargo fmt

* fix: compile

* chore: remove group manager dependency

* fix: dep count

* fix: add messages sent by server

* fix: dep count

* chore: add test for doc ref and inline doc database

* chore: cargo clippy

* chore: add more test scenarios

* chore: fix tests

* chore: get database id

* chore: update collab

* chore: add more assert and database row id checks

* fix: suggestions from review

* chore: sqlx

* fix: accumulate collab before insert

* chore: add tokio spawn blocking for encoding

* fix: reduce limit for publish collabs

---------

Co-authored-by: Bartosz Sypytkowski <b.sypytkowski@gmail.com>
Co-authored-by: nathan <nathan@appflowy.io>
2024-09-03 09:12:29 +08:00
Nathan.fooo 365c64c5c2
chore: use colla storage instead of select from db directly (#782) 2024-09-02 21:25:21 +08:00
Nathan.fooo 388d38428b
chore: revamp post realtime endpoint (#536)
* chore: revamp  post realtime endpoint

* chore: fix clippy

* chore: fix fmt
2024-05-08 22:06:37 +08:00
Nathan.fooo 7b4b9d0c10
chore: folder data check (#496)
* chore: folder data check

* chore: stop observe remote change when specific type of collaboration object override with incorrect data

* chore: commit sqlx file
2024-04-26 13:20:06 +08:00
Nathan.fooo a0593b5502
chore: refactor return response with object id (#494)
* chore: refactor return response with object id

* chore: fix ci

* chore: fix ci
2024-04-24 12:07:54 +08:00
Nathan.fooo 970fbaed57
feat: using transaction to write multiple records (#477)
* chore: impl batch write

* chore: update test

* chore: using save point to optimize write

* chore: update docs

* chore: fix test

* chore: increase write speed

* chore: use redis sorted set

* chore: fallback to write to disk directly when fail to queue

* chore: fix test

* chore: fix test

* chore: update log

* chore: add metrics
2024-04-19 15:22:57 +08:00
Nathan.fooo 4168d72390
chore: using redis conn for each action (#476)
* chore: batch insert

* chore: opti redis conn

* chore: fix test
2024-04-17 11:41:50 +08:00
Nathan.fooo 8ab7815740
chore: reduce pool lock (#475)
* chore: reduce pool lock

* chore: clippy
2024-04-16 20:26:26 +08:00
Nathan.fooo baeeb2c5ce
chore: do not wait insert to redis finish when calling get collab (#468) 2024-04-13 10:26:05 +08:00
Nathan.fooo b1c3d779ba
chore: reduce save collab db operations (#455)
* chore: reduce save collab db operations

* chore: clippy

* chore: add test

* chore: add test

* chore: add test
2024-04-08 14:50:51 +08:00
Nathan.fooo d77fb77a3a
feat: validate collab data before writing to disk (#425)
* chore: validate collab

* chore: update test

* chore: partition key

* chore: save collab

* chore: save collab
2024-03-29 14:55:51 +08:00
Nathan.fooo d30cf93a1f
feat: sync protocol (#421)
* chore: send full update when it's first time to sync

* chore: update last sync at
2024-03-27 21:39:47 +08:00
Nathan.fooo 3e36dae50d
chore: remove collab ac cache (#420)
* chore: remove collab ac cache

* chore: update member write workspace role

* chore: fix test

* chore: clippy
2024-03-26 21:36:24 +08:00
Nathan.fooo c015ee7c7e
fix: peer sync (#408) 2024-03-22 21:32:41 +08:00
Nathan.fooo a31c246b67
feat: Access control group (#395)
* chore: access control for group

* chore: access control for workspace

* chore: add test

* chore: enforce group

* chore: update docs
2024-03-19 12:57:08 +08:00
Nathan.fooo 40fc18a499
feat: only enforce when permission change (#382)
* feat: only enforce when permission change

* chore: fix client api ci

* chore: fix test

* chore: fix test
2024-03-13 17:14:43 +08:00
Nathan.fooo 0e57de98d8
refactor: Access control (#361)
* refactor: access level

* refactor: access control

* refactor: enforce action

* refactor: collab cache

* chore: fix test

* chore: fix test

* chore: fix test

* chore: fix test

* chore: commit migration file

* chore: commit migration file
2024-03-03 12:55:12 +08:00
Nathan.fooo 5ed6a14943
chore: reduce lock required (#353)
* chore: spawn local task

* chore: spawn local task

* chore: spawn local task

* chore: spawn local task

* chore: fix test

* chore: fix test

* chore: clippy

* chore: clippy

* chore: clippy

* chore: clippy
2024-02-27 18:50:44 +08:00
Nathan.fooo deffd074b9
chore: check collab before insert (#349)
* chore: check collab before insert

* chore: fix audit

* chore: speed test

* chore: add expirate duration for cache encode collab

* chore: fix test
2024-02-25 12:38:07 +08:00
Nathan.fooo 10fc8dd2fe
chore: use Redis cache for enforcer (#348)
* chore: use redis cache

* chore: use redis to cache enforce result

* fix: test

* fix: test

* fix: test

* fix: test
2024-02-24 15:45:53 +08:00
Nathan.fooo b71e443d3f
refactor: write snapshot (#345)
* chore: increase max connection size of pg (#344)

* chore: refactor snapshot

* chore: fix test

* chore: fix test

* chore: fix test

* chore: metric

* chore: clippy

* chore: fix test

* chore: fix test
2024-02-23 14:36:06 +08:00
Nathan.fooo b73e7045dc
chore: use redis as encoded collab cache (#342) 2024-02-23 00:36:37 +08:00
Nathan.fooo 5cd16d7544
chore: load user id and uuid when start (#329)
* chore: load user id and uuid when start

* chore: fix test

* chore: fix test
2024-02-20 05:43:33 +08:00
Nathan.fooo a1494f0d09
feat: Realtime metrics (#309)
* chore: add metric

* chore: add metric

* chore: clippy

* chore: clippy

* ci: set rust version

* ci: set rust version

* ci: set rust version
2024-02-09 12:02:54 +08:00
Nathan.fooo e1307f4f5d
chore: cache within memory size (#305)
* chore: cache within memory size

* chore: update
2024-02-08 08:06:41 +08:00
Nathan.fooo 03471f3af7
chore: update feature of actix (#304)
* chore: update feature of actix

* chore: add logs
2024-02-08 02:32:25 +08:00
Nathan.fooo 6f2faff21f
chore: fix log (#301) 2024-02-07 08:15:57 +08:00
Nathan.fooo d497f2ab62
fix: pg lock (#299)
* chore: fix pg lock

* chore: fix pg lock
2024-02-06 13:44:39 +08:00
Nathan.fooo 1cfb38d7a2
chore: fix pg lock (#298) 2024-02-06 08:57:10 +08:00
Nathan.fooo 1a7866e651
chore: use redis to cache collab data (#294)
* chore: use redis to cache collab data

* chore: create table

* chore: use cache

* chore: period check group status

* chore: remove trait

* chore: update logs

* chore: update logs

* chore: update logs

* chore: update logs
2024-02-05 09:54:16 +08:00
Nathan.fooo 2d8579caab
Logs for apply update (#293)
* chore: add logs

* chore: add logs

* chore: clippy
2024-02-05 01:37:28 +08:00
Nathan.fooo 4886d3d474
chore: rename error message (#247) 2024-01-05 07:29:06 +08:00
Nathan.fooo c5fe447250
feat: bump collab (#243)
* chore: bump collab

* chore: bump collab

* chore: bump collab
2024-01-03 08:40:26 +08:00
Nathan.fooo a455c9de8e
chore: refactor logs (#236)
* chore: add logs

* chore: return ws state

* fix: init the collab from disk
2023-12-27 05:20:40 +08:00
Nathan.fooo a07cc0b903
feat: Insert multiple collab (#233)
* chore: remove panic hook

* chore: support insert multiple collabs
2023-12-25 02:03:41 +08:00
Nathan.fooo 453329dc0c
feat: try to resotre from snapshot when fail to initialize the collab in CollabStoragePlugin (#222) 2023-12-18 02:27:49 +08:00
Nathan.fooo 69ed6ff4e8
feat: create snapshot for collab (#221) 2023-12-17 13:38:55 +08:00
Nathan.fooo b39621e389
refactor: reduce select all workspace memebers call (#216)
* refactor: reduce select all workspace memebers call

* chore: docker build

* refactor: move tests

* chore: expose enforcer on tests

* refactor: update workspace member when recv pg notification

* chore: update test

* chore: commit sqlx file
2023-12-17 02:46:05 +08:00
Nathan.fooo 29b39f9ba3
feat: Client api url (#162)
* chore: expose client base url
2023-11-13 20:43:04 +08:00
Nathan.fooo 41e01b91fc
feat: update collab rev and fix some bugs (#148)
* chore: remove todo

* chore: update collab rev

* chore: fmt

* chore: fmt

* chore: cache collab user

* fix: internal error

* ci: fix test

* chore: update collab rev

* ci: fix
2023-11-05 22:43:20 +08:00
Nathan.fooo 9c911a3a94
refactor: Error (#143)
* chore: remove api

* chore: update log

* refactor: error

* chore: fmt

* ci: fix test

* ci: fix

* chore: remove unstable
2023-11-02 17:17:34 +08:00
Nathan.fooo d0d2e916a7
refactor: hiding sqlx row entitis from client api (#132) 2023-10-22 22:10:03 +08:00