LLM Ready Data Connectors

This repository contains a collection of data connectors built by Mendable AI. These connectors are designed to output data in a specific format suitable for LLMs vectorization.

Key Features

• 🛠️ Easy Integration: Quick setup for immediate use
• 🎯 LLM Specific: Unified formats for LLM compatibility
• 🔒 Nango Authorization: You can use your Nango account to authorize the connectors
• 🔗 Diverse Sources: Unified access to various data sources
• 🏷️ Strong Typing: Improves developer experience
• 🔄 Continuous Updates: Regularly updated with new connectors
• 🤝 Community Support: Active community for troubleshooting and support
• 🚀 High Performance: Optimized for speed and efficiency
• 🛡️ Secure: Authentication with OAuth2.0 for most data providers
• 💯 Open Source: Community-driven development

Available Connectors

The following connectors are currently available:

• ✅ Files (.md, .txt, .csv, and .pdf powered by LlamaParse)
• ✅ GitHub (Private and Public repos)
• ✅ Google Drive
• ✅ Notion (pages, need to grant access)
• ✅ Text
• ✅ Web Scraper (Crawler, URLs, Sitemap)
• ✅ Zendesk
• ✅ YouTube (Whole Channel and Video)
• ✅ Jira
• ✅ Confluence (Wiki Pages)
• ✅ Salesforce (accounts, articles, contacts, deals, tickets)

We are working hard on transitioning all of our connectors to this repository. If you need a connector that is not available here, please open an issue or submit a PR.

Installation

To install the connectors, run the following command:

npm install @mendable/data-connectors

Usage

To use these connectors, you need to create a data connector with the provider of your choice. Here is an example:

import { createDataConnector } from "@mendable/data-connectors";

const webDataConnector = createDataConnector({
  provider: "web-scraper",
});

webDataConnector.setOptions({
  urls: ["https://docs.mendable.ai"],
  mode:"single_urls",
})

const documents = await webDataConnector.getDocuments();

Authorization

For data connectors that require some sort of authorization such as Google Drive one of the following methods can be used:

import { createDataConnector } from "@mendable/data-connectors";

const googleDriveDataConnector = createDataConnector({
  provider: "google-drive",
});

// You can use normal google authorization, with OAuth access token or...
await googleDriveDataConnector.authorize({
  access_token: "<>",
})

// You can use Nango authorization, which is a lot easier and will handle all the Auth part for you
await googleDriveDataConnector.authorizeNango({
  nango_connection_id: "YOUR NANGO CONNECTION ID"
})

const documents = await googleDriveDataConnector.getDocuments();

Here is the .env.example file for the connectors. You can copy this file and rename it to .env and fill in the values. You only need to fill these values for the ones you plan on using.

NANGO_SECRET_KEY=<> // This is the secret key for your Nango account


GOOGLE_DRIVE_CLIENT_ID=<>
GOOGLE_DRIVE_CLIENT_SECRET=<>
GOOGLE_DRIVE_REDIRECT_URI=<>

SCRAPING_BEE_API_KEY=<>
NANGO_CONNECTION_ID_TEST=<>

Output Format

The output of the data connectors is a Document object. The structure of the Document object is as follows:

export class Document {
    content: string; // The content of the document
    provider: string; // The provider of the document
    id?: string; // The unique identifier of the document
    createdAt?: Date; // The date when the document was created
    updatedAt?: Date; // The date when the document was last updated
    type?: string; // The type of the document
    metadata: {
        sourceURL?: string, // The source URL of the document, optional but should almost always contain.
        [key: string]: any; // Any additional metadata associated with the document
    }
}

Contributors

Big thanks to all our contributors: @nickscamara, @rafasideguide, @mogery, @eciarla