feat (docs): web crawler connector (#593)

2 files changed, 27 insertions, 7 deletions

diff --git a/apps/docs/memory-api/connectors/creating-connection.mdx b/apps/docs/memory-api/connectors/creating-connection.mdx
index 39abc47a..a3d1e257 100644
--- a/apps/docs/memory-api/connectors/creating-connection.mdx
+++ b/apps/docs/memory-api/connectors/creating-connection.mdx
@@ -13,9 +13,15 @@ const client = new Supermemory({
   apiKey: process.env['SUPERMEMORY_API_KEY'], // This is the default and can be omitted
 });
 
+// For OAuth providers (notion, google-drive, onedrive)
 const connection = await client.connections.create('notion');
-
 console.debug(connection.authLink);
+
+// For web-crawler (no OAuth required)
+const webCrawlerConnection = await client.connections.create('web-crawler', {
+  metadata: { startUrl: 'https://docs.example.com' }
+});
+console.debug(webCrawlerConnection.id); // authLink will be null
 ```
 
 ```python Python
@@ -57,12 +63,14 @@ curl --request POST \
 
 ### Parameters
 
-- `provider`: The provider to connect to. Currently supported providers are `notion`, `google-drive`, `one-drive`
+- `provider`: The provider to connect to. Currently supported providers are `notion`, `google-drive`, `onedrive`, `web-crawler`
 - `redirectUrl`: The URL to redirect to after the connection is created (your app URL)
+    - Note: For `web-crawler`, this is optional as no OAuth flow is required
 - `containerTags`: Optional. For partitioning users, organizations, etc. in your app.
     - Example: `["user_123", "project_alpha"]`
 - `metadata`: Optional. Any metadata you want to associate with the connection.
     - This metadata is added to every document synced from this connection.
+    - For `web-crawler`, must include `startUrl` in metadata: `{"startUrl": "https://example.com"}`
 - `documentLimit`: Optional. The maximum number of documents to sync from this connection.
     - Default: 10,000
     - This can be used to limit costs and sync a set number of documents for a specific user.
@@ -80,6 +88,10 @@ supermemory sends a response with the following schema:
 }
 ```
 
-You can use the `authLink` to redirect the user to the provider's login page.
+For most providers (notion, google-drive, onedrive), you can use the `authLink` to redirect the user to the provider's login page.
+
+<Note>
+**Web Crawler Exception:** For `web-crawler` provider, `authLink` and `expiresIn` will be `null` since no OAuth flow is required. The connection is established immediately upon creation.
+</Note>
 
 Next up, managing connections. 
diff --git a/apps/docs/memory-api/connectors/overview.mdx b/apps/docs/memory-api/connectors/overview.mdx
index 8727b68c..e7f9d479 100644
--- a/apps/docs/memory-api/connectors/overview.mdx
+++ b/apps/docs/memory-api/connectors/overview.mdx
@@ -1,26 +1,34 @@
 ---
 title: 'Connectors Overview'
 sidebarTitle: 'Overview'
-description: 'Sync external connections like Google Drive, Notion, OneDrive with supermemory'
+description: 'Sync external connections like Google Drive, Notion, OneDrive, Web Crawler with supermemory'
 ---
 
-supermemory can sync external connections like Google Drive, Notion, OneDrive with more coming soon.
+supermemory can sync external connections like Google Drive, Notion, OneDrive, and Web Crawler.
 
 ### The Flow
 
+For OAuth-based connectors (Notion, Google Drive, OneDrive):
 1. Make a `POST` request to `/v3/connections/{provider}`
 2. supermemory will return an `authLink` which you can redirect the user to
 3. The user will be redirected to the provider's login page
 4. User is redirected back to your app's `redirectUrl`
 
+For Web Crawler:
+1. Make a `POST` request to `/v3/connections/web-crawler` with `startUrl` in metadata
+2. Connection is established immediately (no OAuth required)
+3. Crawling begins automatically
+
 ![Connectors Flow](/images/connectors-flow.png)
 
 ## Sync frequency
 
 supermemory syncs documents:
-- **A document is modified or created (Webhook recieved)**
+- **A document is modified or created (Webhook received)**
     - Note that not all providers are synced via webhook (Instant sync right now)
     - `Google-Drive` and `Notion` documents are synced instantaneously
-- Every **four hours**
+    - `Web-Crawler` uses scheduled recrawling instead of webhooks
+- Every **four hours** (for OAuth-based connectors)
+- **Scheduled recrawling** (for Web Crawler - sites recrawled if not synced in 7+ days)
 - On **Manual Sync** (API call)
     - You can call `/v3/connections/{provider}/sync` to sync documents manually