diff options
| author | Dhravya Shah <[email protected]> | 2024-06-18 17:58:46 -0500 |
|---|---|---|
| committer | Dhravya Shah <[email protected]> | 2024-06-18 17:58:46 -0500 |
| commit | f4bb71e8f7e07bb2e919b7f222d5acb2905eb8f2 (patch) | |
| tree | 7310dc521ef3559055bbe71f50c3861be2fa0503 /apps/cf-ai-backend/src/utils/chonker.ts | |
| parent | darkmode by default - so that the colors don't f up on lightmode devices (diff) | |
| parent | Create Embeddings for Canvas (diff) | |
| download | supermemory-default-darkmode.tar.xz supermemory-default-darkmode.zip | |
Diffstat (limited to 'apps/cf-ai-backend/src/utils/chonker.ts')
| -rw-r--r-- | apps/cf-ai-backend/src/utils/chonker.ts | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/apps/cf-ai-backend/src/utils/chonker.ts b/apps/cf-ai-backend/src/utils/chonker.ts index 39d4b458..c63020be 100644 --- a/apps/cf-ai-backend/src/utils/chonker.ts +++ b/apps/cf-ai-backend/src/utils/chonker.ts @@ -1,5 +1,8 @@ import nlp from "compromise"; +/** + * Split text into chunks of specified max size with some overlap for continuity. + */ export default function chunkText( text: string, maxChunkSize: number, |