Commit Graph

20 Commits

Author SHA1 Message Date
Bartosz Sypytkowski 66deaf3fa3
chore: move collab cache for appflowy collaborate service and add metrics (#1045)
* chore: move collab cache for appflowy collaborate service and add metrics

* chore: add histogram for measuring time spent in postgres transactions

* chore: delete blobs in chunks

* chore: bring back redis cache filling by import worker

* chore: cargo fmt

* chore: make import workspace update existing collabs

* chore: bulk insert collab deduplicate records
2024-12-06 19:55:51 +08:00
Bartosz Sypytkowski 9ff6f1c744
S3 collab cache (#1028)
* chore: collab cache for S3

* chore: adjust disk cache api to accomodate s3

* chore: move postgres dependent ops to disk cache

* chore: replace blob inserts from pg to s3

* chore: delete blob and collab exist now use s3

* chore: fix clippy erorrs

* chore: post rebase fixes

* chore: fix clippy warnings

* chore: fix imports

* chore: make snapshots work over S3

* chore: remove dead code

* chore: use compressed snapshots

* chore: add zstd compression

* chore: introduce collab size threshold to keep smaller collabs in postgres

* chore: remove collabs from S3 if they were put to postgres

* chore: update tests
2024-12-03 13:08:55 +08:00
Nathan.fooo 3e3bf693fd
chore: upload api endpoint for web upload (#1024)
* chore: upload api endpoint for web upload

* chore: use rust 1.78

* fix: test
2024-11-28 09:09:09 +08:00
Nathan.fooo 06a6ea2985
chore: retry upload file when get s3 service error (#993)
* chore: retry upload file when get s3 service error

* chore: ingore upload payload error
2024-11-14 14:39:36 +08:00
Nathan.fooo 9629d4cefa
chore: add limit for importing zip file (#938)
* chore: add limitation for import zip file

* chore: support upload big file

* chore: implement client api

* chore: implement client api

* chore: implement client api

* chore: update logs

* chore: check file size

* chore: last process at

* chore: set content type

* chore: fix test

* chore: try test

* chore: temporary disable test
2024-10-28 08:51:34 +08:00
Nathan.fooo 9c0dffacef
feat: implement import notion zip file api endpoint (#840)
* chore: implement api endpoint

* chore: worker

* chore: async zip

* chore: test bulk insert

* chore: insert collab

* chore: custom task

* chore: consume un acked message

* chore: fix compile

* chore: add test

* test: update

* chore: save import record

* chore: save import record

* chore: fix ci

* chore: remove unused deps

* chore: update docker file

* chore: build worker images

* chore: use small int

* chore: use small int

* chore: rm protoc deps in runtime

* chore: move collab cache to database crate

* chore: update test

* chore: rm health api endpoint

* chore: clippy

* chore: update ci

* chore: add test

* chore: upgrade collab

* chore: clippy

* chore: update test

* chore: use custom host

* chore: config nginx

* chore: install cert
2024-10-03 22:22:29 +08:00
Khor Shu Heng abae8d2d1b
feat: Support s3 as published collab storage (#798)
* chore: remove unused published collab methods

* feat: support s3 as published collab storage
2024-09-11 20:52:33 +08:00
Khor Shu Heng 52f33e4d85
feat: endpoint for avatar upload (#736) 2024-08-23 16:36:09 +08:00
Nathan.fooo a3af38162e
chore: fix upload filem meta key by adding parent dir as prefix (#635)
* chore: fix upload filem meta key by adding parent dir as prefix

* chore: fix test
2024-06-20 21:12:28 +08:00
Nathan.fooo 30f87457f3
feat: support uploading file with multiple part (#625)
* refactor: replace crate

* chore: remove s3-rust crate

* chore: test multiple upload

* chore: client api

* chore: client api

* chore: v1 version upload,download

* chore: fix test

* chore: fix test

* chore: remove workspace test

* chore: fix wasm build

* chore: update path

* chore: update path

* chore: rename

* chore: expose file entity

* chore: introduce client-api-entity

* chore: fix build

* chore: new url

* chore: fix test

* chore: merge main

* chore: fix clippy
2024-06-19 13:06:58 +08:00
Nathan.fooo 359433f14c
fix: pool timeout when there are too many put blob requests (#533)
* fix: pool timeout when there are too many put blob requests

* chore: fix clippy

* fix: dont use transaction while putting data

---------

Co-authored-by: Zack Fu Zi Xiang <speed2exe@live.com.sg>
2024-05-08 12:20:06 +08:00
Zack Fu Zi Xiang c8133302c1
feat: remove storage limit 2024-03-19 22:13:59 +08:00
Zack 97d1bb532c
feat: revamp storage api (#251)
* feat: revamp storage api

* feat: client api add method to generate url

* feat: remove some unused deps

* feat: upgrade deps

* feat: add mime to resp
2024-01-11 00:32:11 +08:00
Nathan.fooo 8e38ae7b69
refactor: remove rust-s3, sqlx from client-api (#245)
* refactor: remove rust-s3, sqlx from client-api

* chore: fmt

* chore: add ci check

* chore: add ci check

* chore: add ci check
2024-01-05 06:12:01 +08:00
Nathan.fooo 9c911a3a94
refactor: Error (#143)
* chore: remove api

* chore: update log

* refactor: error

* chore: fmt

* ci: fix test

* ci: fix

* chore: remove unstable
2023-11-02 17:17:34 +08:00
Nathan.fooo 8c1c5d0cd9
feat: insert uid instead of gen it from db (#139) 2023-10-25 12:18:33 +08:00
Nathan.fooo d0d2e916a7
refactor: hiding sqlx row entitis from client api (#132) 2023-10-22 22:10:03 +08:00
Nathan.fooo 1aba1f0cf4
feat: workspace usage test (#129)
* feat: workspace usage test

* test: add tests
2023-10-22 17:52:10 +08:00
Fu Zi Xiang 574d2661a1
chore: fix linting 2023-10-13 12:12:12 +08:00
Nathan.fooo ef726092be
refactor: file storage (#110)
* refactor: return file id after save to disk

* test: add tests

* chore: mine type

* test: disbale two client sync test

* chore: get file with url

* refactor: client api
2023-10-11 21:36:57 +08:00