Connect Web as a Source so Dosu can consult public websites and documentation when it answers a question. This page explains how the Web Source works, what it does and does not do with website content, and how to add it to a Library.
What connecting Web gives you#
Web lets Dosu pull in up-to-date information from public pages when it needs it. When someone asks a question and the answer depends on an external reference (a public docs site, an upstream project page, a standards document), Dosu can search the web in real time, read the relevant pages, and fold what it finds into the answer alongside your other Sources. Responses include citations back to the pages Dosu used, so your team can check the original.
The Web is the only Source that contains no content of its own. It is a live lookup, not a stored copy.
How Web Search is different#
This is the important distinction, so it is worth stating plainly. With Web, Dosu does not crawl, index, or store website content. There is no copy of any page kept in your knowledge base.
- Web information is accessed only at query time, through real-time search, when Dosu is answering a question.
- Nothing from the web is added to your Documents or your indexed knowledge.
- A Web Source shows as synced in the app right after you add it. That status is only for consistency with other Sources in the interface. No syncing actually happens, because there is nothing to sync.
By comparison, a GitHub or Slack Source is indexed and kept up to date over time. Web is fetched per request and then gone.
Availability and prerequisites#
- Web is generally available and selectable today.
- Web is a Source, so it is added inside a Library. Open or create one at Libraries.
- Dosu can only read public pages. Web does not support sites that require a login, and there is no way to access an internal or password-protected knowledge base.
How to connect Web Search#
- Go to Libraries and open the Library you want Dosu to use the web for, or create one at create a Library.
- Add a Source and choose Web.
- Enter the website you want Dosu to consult.
- Optionally narrow the scope with allowed URL patterns. Specify one relative pattern per line, for example
/docs/*, to keep Dosu focused on the parts of a site that matter. Leave it empty to allow the whole site. - Save. The Web Source becomes available to Dosu right away.
You can add more than one Web Source to a Library, and you can manage or remove them later from the same place.
What Dosu does with it#
When a question comes in, Dosu searches your connected Sources for relevant context and writes an answer that synthesizes what it found, with citations linking back to the source material. If a Web Source is connected, public pages are included in that search and retrieved live for that request. Web results carry their own citation so readers can tell which part of an answer came from the open web. This works wherever Dosu answers, including the Chat experience and the GitHub and Slack Agents.
Notes and limits#
- Because Web is fetched per request, the quality and freshness of an answer depend on what is publicly reachable at that moment.
- Web does not produce Documents on its own. To turn an answer that drew on the web into a saved artifact, use Save This Answer or Generate Docs (see Related).
- Allowed URL patterns scope which pages Dosu may consult. They do not turn Web into an indexed Source. The behavior is still real-time only.