> ## Documentation Index
> Fetch the complete documentation index at: https://docs.edux.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Connector

> Access knowledge from Web Pages

## How it works

The Web Connector scrapes sites based on a base URL.

* It only indexes files from the same domain and containing the same base path.
* It will index pages reachable via hyperlinks from the base URL.
* The text contents are cleaned up via some heuristics and some metadata such as the page Title is extracted.

## Setting up

### Authorization

* As long as the page is reachable, no additional authorization is necessary.

### Indexing

1. Navigate to the Admin Dashboard and select the **Web** Connector.
2. Input the base URL to index and click on Index.

![WebConnector](https://mintlify.s3-us-west-1.amazonaws.com/edux/images/connectors/web/WebConnector.png)

To see the status of the indexing, visit the Connectors Status page (top left).

![WebConnectorStatus](https://mintlify.s3-us-west-1.amazonaws.com/edux/images/connectors/web/WebConnectorStatus.png)