29cb70a362
fix: synchronization issue due to incorrect collab origin ( #1054 )
2024-12-09 14:19:00 +08:00
a598aed5ec
chore: remove empty map when query is empty ( #1050 )
2024-12-07 10:20:53 +08:00
66deaf3fa3
chore: move collab cache for appflowy collaborate service and add metrics ( #1045 )
...
* chore: move collab cache for appflowy collaborate service and add metrics
* chore: add histogram for measuring time spent in postgres transactions
* chore: delete blobs in chunks
* chore: bring back redis cache filling by import worker
* chore: cargo fmt
* chore: make import workspace update existing collabs
* chore: bulk insert collab deduplicate records
2024-12-06 19:55:51 +08:00
aa64b7cd6b
fix: web update should update both disk and mem cache
2024-12-05 16:51:56 +08:00
b2f8a95ec4
chore: setup new collab in the db
2024-12-05 06:29:19 +01:00
5e7b5f87fa
chore: merge branch main into stateless
2024-12-03 09:10:01 +01:00
51bf9a85c7
chore: make collab storage switch between s3 and postgres configurable ( #1035 )
2024-12-03 14:09:09 +08:00
9ff6f1c744
S3 collab cache ( #1028 )
...
* chore: collab cache for S3
* chore: adjust disk cache api to accomodate s3
* chore: move postgres dependent ops to disk cache
* chore: replace blob inserts from pg to s3
* chore: delete blob and collab exist now use s3
* chore: fix clippy erorrs
* chore: post rebase fixes
* chore: fix clippy warnings
* chore: fix imports
* chore: make snapshots work over S3
* chore: remove dead code
* chore: use compressed snapshots
* chore: add zstd compression
* chore: introduce collab size threshold to keep smaller collabs in postgres
* chore: remove collabs from S3 if they were put to postgres
* chore: update tests
2024-12-03 13:08:55 +08:00
3e3bf693fd
chore: upload api endpoint for web upload ( #1024 )
...
* chore: upload api endpoint for web upload
* chore: use rust 1.78
* fix: test
2024-11-28 09:09:09 +08:00
4fb6df5c14
chore: reduce number of SQL statements executed when inserting/updating collab ( #1022 )
...
* chore: reduce number of SQL statements executed when inserting new collab
* chore: add workspace id check
* chore: fix previous query
* chore: upsert collabs and embddings as stored procedures
* chore: fix clippy errors
* chore: fix stored procedure to compare workspaces correctly
2024-11-27 13:00:00 +08:00
d6317813d7
chore: allow dead code
2024-11-20 09:06:00 +01:00
b91a5fac8e
chore: post merge fixes
2024-11-20 08:56:16 +01:00
d0f9059f3f
Merge branch 'main' into stateless
2024-11-20 08:36:28 +01:00
fd98351473
chore: adjust redis env ( #1003 )
2024-11-17 14:26:59 +08:00
3799966f12
chore: store pending collab writes in memory ( #1000 )
...
* chore: write immediatelly actually writes immediatelly
* chore: fix clippy errors
* chore: add metrics to new storage queue impl
* chore: set collab batch write capacity to the same as on main branch
2024-11-17 13:25:42 +08:00
d798c81ba4
chore: support split by text len ( #1002 )
...
* chore: support split by text len
* chore: update docs
* chore: update tests
2024-11-17 12:45:20 +08:00
655f13bc27
chore: search and chat ( #999 )
...
* chore: add test for search and chat
* chore: update test
* chore: update test
* chore: update ci
* chore: fix security audio
* chore: multiple core docker build
* chore: multiple core docker build
* chore: update ci
* chore: update model setting
* chore: test ci
* chore: use tiktoken to calcualte token length
* chore: remove env
* chore: use spawn_blocking with condition
* chore: docs
* chore: clippy
* chore: clippy
* chore: docker logs
* chore: pass message id
* chore: clippy
2024-11-16 14:52:12 +08:00
06b7ce0a29
chore: post merge fixes
2024-11-16 07:33:18 +01:00
401cc3ad25
Merge branch 'main' into stateless
2024-11-16 07:22:34 +01:00
d0c212ff10
refactor: remove enforcer group ( #948 )
2024-11-15 14:08:12 +08:00
b10a0e7829
chore: fix typo
2024-11-15 13:45:25 +08:00
97f9ff3dd8
fix: byte index 8000 is not a char boundary ( #995 )
...
* chore: fix split text boundary error and add related tests
* chore: reduce clone
2024-11-15 13:38:08 +08:00
f6fef9918b
chore: split document content into multiple embeddings ( #990 )
...
* chore: split document content into multiple embeddings
* chore: fallback to uuid v4 generation due to unmaintained transient dependency
* chore: fix lint errors
2024-11-14 14:15:52 +08:00
91b0c50722
chore: remove logs ( #986 )
2024-11-13 13:59:23 +08:00
9778843746
chore: disable history service ( #985 )
...
* chore: disable history service
* chore: fix build
2024-11-13 10:46:15 +08:00
eb54947ab7
chore: add config option for redis stream group workers count
2024-11-08 04:39:44 +01:00
b9ca480143
chore: add retry limit to which collab snapshot can be made
2024-11-07 13:12:17 +01:00
bc49d73b40
chore: split redis stream readers to active and one shot
2024-11-07 11:43:19 +01:00
2d9a024554
chore: use redis stream router
2024-11-07 07:18:20 +01:00
f3cd597832
chore: fix clippy errors
2024-11-06 10:57:32 +01:00
3fc49a8b21
chore: turn off collab preindexing
2024-11-06 10:10:55 +01:00
863e405263
chore: add more precise logging on get unindexed collabs
2024-11-06 08:04:07 +01:00
99203cc7f3
chore: fix flaky tests
2024-11-05 07:48:14 +01:00
25f0461389
chore: added metrics for new stateless operations
2024-11-05 07:48:14 +01:00
e86c9de316
chore: when saving collab snapshot, do not load it unless there are pending updates in redis
2024-11-05 07:48:14 +01:00
961c482995
chore: prevent hot loops in collab updates stream
2024-11-05 07:48:14 +01:00
2511bd720d
chore: fix clippy errors
2024-11-05 07:48:14 +01:00
ce793fa03e
chore: make collab group prune grace period configurable
2024-11-05 07:48:14 +01:00
395424bfd2
chore: detect and re-request missing updates on the server side
2024-11-05 07:48:14 +01:00
a99f986270
chore: removed appflowy history
2024-11-05 07:48:12 +01:00
81dc66fa7b
chore: fix clippy warnings
2024-11-05 07:47:17 +01:00
f684d3623d
chore: fixed collab ack
2024-11-05 07:47:17 +01:00
fc1a7d28e2
chore: better stress test scenario
2024-11-05 07:47:17 +01:00
de4016659e
chore: add break in cancellation of snapshot task
2024-11-05 07:47:17 +01:00
d624cdf49b
chore: turn off separate appflowy history process
2024-11-05 07:47:17 +01:00
a2d3d7feb5
chore: add perf tests
2024-11-05 07:47:17 +01:00
279a599b66
chore: fix early return from redis collab update stream reader
2024-11-05 07:47:17 +01:00
85e25f887c
chore: don't try to prune empty redis stream
2024-11-05 07:47:17 +01:00
d0b7c7d8e3
chore: update stream - live and current queries
2024-11-05 07:47:17 +01:00
61820d7226
chore: more traces in collab group
2024-11-05 07:47:17 +01:00
a1546909c3
chore: remove server sync protocol (not used anymore)
2024-11-05 07:47:17 +01:00
00f5a5bb7e
chore: code cleanup
2024-11-05 07:47:17 +01:00
3e612e2db6
chore: create method for receiving the most recent snapshot
2024-11-05 07:47:17 +01:00
9c674d5777
chore: changes to make appflowy cloud compile under the latest appflowy collab
2024-11-05 07:47:17 +01:00
bee7caedac
chore: attach indexer to collab persister
2024-11-05 07:47:17 +01:00
97335a94ec
chore: basics of snapshots (unoptimized)
2024-11-05 07:47:17 +01:00
253bb97825
chore: a dirty draft of snapshot saving
2024-11-05 07:47:17 +01:00
9164c494b1
chore: redis stream for awareness
2024-11-05 07:47:17 +01:00
d26d0ff1a7
chore: collab streams
2024-11-05 07:47:17 +01:00
0d6b595ee8
chore: create collab update sink and stream
2024-11-05 07:47:17 +01:00
fe3611cc88
chore: remove broadcast and introduce lease
2024-11-05 07:47:17 +01:00
d3d3378b86
chore: draft of stateless collab group
2024-11-05 07:47:17 +01:00
6aed9609b9
chore: removed realtime shared state tests
2024-11-04 09:21:10 +01:00
356cf5d132
chore: fix clippy errors
2024-11-04 08:29:32 +01:00
f105c2dbc2
chore: fix clippy errors
2024-11-04 08:20:56 +01:00
f9397eaaf2
chore: remove realtime shared state
2024-11-04 07:58:16 +01:00
9eb1c36c92
chore: access control interface to return error instead of boolean ( #914 )
2024-10-21 20:20:14 +08:00
ffee47d0f7
chore: remove workspace and collab listener ( #893 )
2024-10-17 17:43:00 +08:00
8ab5da4aef
fix: limit concurrency level of batch get encode collab ( #892 )
2024-10-17 13:29:27 +08:00
383629ab1c
chore: simplify not enough permission error
2024-10-16 15:10:31 +08:00
1616ceaf4f
chore: use workspace deps ( #880 )
2024-10-14 16:02:22 +08:00
8c14612f98
feat: refactor access control to enable swapping implementation ( #859 )
2024-10-10 13:04:47 +08:00
33e63fae5c
chore: try to fix pg lock timeout ( #864 )
...
* chore: try to fix pg lock timeout
* chore: add logs for insert collab
* chore: add timeout for duplicate
* chore: timeout for pending write
2024-10-07 16:49:50 +08:00
38ff1874e4
chore: using copy to improve write speed of batch collab api endpoint ( #862 )
...
* chore: using copy to improve write speed of batch collab api endpoint
* chore: insert to redis
* chore: fix compile
2024-10-07 08:58:19 +08:00
9c0dffacef
feat: implement import notion zip file api endpoint ( #840 )
...
* chore: implement api endpoint
* chore: worker
* chore: async zip
* chore: test bulk insert
* chore: insert collab
* chore: custom task
* chore: consume un acked message
* chore: fix compile
* chore: add test
* test: update
* chore: save import record
* chore: save import record
* chore: fix ci
* chore: remove unused deps
* chore: update docker file
* chore: build worker images
* chore: use small int
* chore: use small int
* chore: rm protoc deps in runtime
* chore: move collab cache to database crate
* chore: update test
* chore: rm health api endpoint
* chore: clippy
* chore: update ci
* chore: add test
* chore: upgrade collab
* chore: clippy
* chore: update test
* chore: use custom host
* chore: config nginx
* chore: install cert
2024-10-03 22:22:29 +08:00
11732324b5
feat: support fetching encoded collab in batch from memory ( #837 )
2024-10-02 10:22:40 +08:00
4c54481d70
chore: bump collab version ( #822 )
...
* chore: bump collab version
* chore: update deps count
2024-09-14 23:59:50 +08:00
a4b885e13b
chore: set exp time for collab base on collab type ( #820 )
...
* chore: set exp time for collab base on collab type
* chore: fix test
2024-09-13 19:04:56 +08:00
1ccff73e49
chore: bump appflowy collab ( #814 )
...
* chore: bump appflowy collab
* chore: fix build
2024-09-12 00:28:26 +08:00
fff93e6083
chore: Adjust gen history interval ( #812 )
...
* chore: adjust generate history interval
* chore: disable verbose log
2024-09-11 13:45:53 +08:00
07d6001cf8
chore: check before write to hisotry ( #809 )
...
* chore: check before write to hisotry
* chore: clippy
* chore: clippy
* chore: calculate edit count
* chore: clippy
* chore: exit loop
* chore: exit loop
* chore: update log
2024-09-10 22:10:59 +08:00
5d356718fa
fix: using spawn_blocking in order to not block the runtime ( #793 )
...
* fix: using blocking in order to not block the runtime
* chore: increase deps count
2024-09-05 09:19:34 +08:00
826546c5cb
feat: duplicate published doc ( #700 )
...
* chore: test collab sync
* feat: folder view for user workspace
* feat: add private indicator
* chore: use collab cache instead
* chore: initial impl
* chore: cargo sqlx
* fix: write back to collab cache
* fix: assign page id
* fix: text map
* chore: connect api and refactor
* chore: switch to using mem storage
* fix: collab type
* feat: use group manager to manage sync
* feat: try add send command instead
* chore: add client api code
* feat: try use sink and stream from group collab
* chore: disable sync first
* fix: insert page before stopping group
* feat: add extra for view
* feat: add metadata to doc
* fix: icon
* fix: page circular dep
* fix: page circular dep
* fix: live update
* feat: database initial impl
* feat: workspace database
* chore: link database
* fix: database validation
* fix: workspace database oid
* fix: workspace database oid
* fix: specify view layout
* feat: add txn when inserting collab and rework database views
* fix: parent view id for new database
* fix: database view for folder
* fix: database ids
* fix: database row id
* fix: main database layout and write database before folder
* fix: metadata for non database main view
* chore: database in doc wip
* chore: filter visible views
* chore: use structure value instead of json
* chore: database in doc wip
* fix: remove child item push
* fix: main database view meta
* chore: rework inline database in doc
* fix: make duplicate more error forgiving
* chore: use get instead of get key value
* chore: merge with main
* feat: set max depth for folder view
* chore: update dependency counter
* chore: remove unneeded arg
* chore: simplify collab new from source
* chore: remove unneeded param
* chore: rename and doc
* feat: rework inline database and use duplicated ref
* chore: simplify params
* chore: fmt
* feat: deduplicate database wip
* fix: compatible with newer appflowy version
* fix: database views
* feat: database in doc
* chore: cargo clippy
* chore: update dockerfile rust toolchain
* fix: doc inline db parent id
* fix: database id
* fix: document data modification
* chore: minor refactor and todo
* chore: remove unneeded nesting
* fix: enforce ordering for views to add
* chore: add doc and test case for folder view
* fix: remove the need for cloning published collab
* fix: some review
* chore: move folder conversion logic
* fix: remove collab folder deps on shared entity
* chore: cargo fmt
* fix: compile
* chore: remove group manager dependency
* fix: dep count
* fix: add messages sent by server
* fix: dep count
* chore: add test for doc ref and inline doc database
* chore: cargo clippy
* chore: add more test scenarios
* chore: fix tests
* chore: get database id
* chore: update collab
* chore: add more assert and database row id checks
* fix: suggestions from review
* chore: sqlx
* fix: accumulate collab before insert
* chore: add tokio spawn blocking for encoding
* fix: reduce limit for publish collabs
---------
Co-authored-by: Bartosz Sypytkowski <b.sypytkowski@gmail.com>
Co-authored-by: nathan <nathan@appflowy.io>
2024-09-03 09:12:29 +08:00
365c64c5c2
chore: use colla storage instead of select from db directly ( #782 )
2024-09-02 21:25:21 +08:00
299680c14a
chore: decode encode collab using spawn blocking ( #781 )
2024-09-02 20:27:40 +08:00
abd96d8b56
chore: add APPFLOWY_COLLAB_REMOVE_BATCH_SIZE env to control num of co… ( #779 )
...
* chore: add APPFLOWY_COLLAB_REMOVE_BATCH_SIZE env to control num of collabs to be removed in one tick
* chore: clippy
2024-09-02 11:24:40 +08:00
c2a5369313
chore: set chat request timeout ( #777 )
...
* chore: set chat request timeout
* chore: clippy
2024-09-01 15:16:11 +08:00
6972f9c4ab
feat: Measure sync ( #772 )
...
* chore: redesign collab sync protocol to enable injection of metric tracking
* chore: track metrics for apply update on the server side
* chore: close locks
* chore: post rebase fixes
2024-08-30 21:48:19 +08:00
f75413566e
chore: reconnect realtime metrics
2024-08-30 09:56:51 +02:00
1f9a03a83d
chore: enable lock timeout feature ( #771 )
2024-08-30 14:24:34 +08:00
3e701a1f8b
chore: remove no require data error log ( #766 )
...
* chore: remove no require data error log
* chore: clippy
2024-08-30 13:55:53 +08:00
5b2b717fe7
chore: remove redundant metrics collector
2024-08-30 05:14:44 +02:00
54b811125e
chore: wrap spawn blocking when decoding collab ( #767 )
2024-08-29 22:53:33 +08:00
2af1999375
chore: added locks with timeouts ( #765 )
...
* chore: added locks with timeouts
* chore: roll back collab locks in client api
2024-08-29 16:13:27 +08:00
e7bf221ff0
feat: update built-in README for new users ( #757 )
...
* feat: add getting_started, desktop_guide and mobile_guide json
* feat: update built-in readme data
* feat: support adding database as template
* feat: support creating nested documents
* chore: refactor code
* chore: refactor code
* chore: remove println
* chore: update collab version
* feat: update built-in readme and image ref
* chore: remove template icon
* feat: use database::create_with_view to create collab
* chore: add template tests
* chore: update collab version
* chore: remove debug code
* chore: add comment
* chore: fix lint & test
* fix: code review
* fix: rust lint
* chore: update comment
* test: add default workspace test
* test: add default user workspace test
* chore: use workspace database body
* chore: fix typo and clippy
* fix: insert_folder_data_success test
* fix: insert_folder_data_success test
* feat: update content in gettting_started
* chore: update to-dos json
---------
Co-authored-by: nathan <nathan@appflowy.io>
2024-08-28 19:40:23 +08:00
8beac5c85f
chore: reuse group collab for indexing ( #737 )
...
* chore: reuse group collab for indexing
* chore: fix linting errors
* chore: post rebase fixes
2024-08-28 00:13:41 +08:00
ab14568bbf
chore: use env to use control multiple thread runtime or not ( #760 )
2024-08-28 00:13:24 +08:00
fee2c37aa4
chore: disable multiple thread in group
2024-08-27 18:12:11 +08:00
c8ee6b43ee
chore: try to fix inifinite sync ( #756 )
2024-08-27 16:41:19 +08:00
a21c55cbcc
chore: modify log level for some cases ( #754 )
...
* chore: modify log level for some cases
* chore: update logs
2024-08-27 14:57:00 +08:00
Khor Shu Heng
Nathan.fooo
Bartosz Sypytkowski
khorshuheng
nathan
Zack
Lucas.Xu