Category / Section
How to Add Web Pages as a Data Source for AI Knowledge Base
Published:
To enrich the knowledge base of the BoldDesk AI Copilot/AI Agent with web pages, you can configure the URLs of these pages. This feature will then crawl and process the contents of the specified web pages.
Steps to Add a New Web Page
- Navigate to
KB -> Artificial Intelligence -> Source -> Choose Web Pagestab. - Click on
Add Web Page. - Fill in the brand information and submit the URL you want to process.
Important Notes:
- Only URLs that are publicly accessible will be processed. Private or protected URLs will be excluded.
- Providing a root domain, such as https://www.bolddesk.com/, will result in the processing of all pages within that domain. If a specific path is provided, such as https://www.bolddesk.com/pricing, only the nested pages within that path (e.g., https://www.bolddesk.com/pricing/team-based) will be processed. Other URLs will not be crawled.
- Additionally, during crawling, only links to pages within the same domain will be considered. Links to external domains will be disregarded.
- Sitemap URLs can also be utilized, and they will be processed accordingly.
- To simplify whitelisting and approval through the robots.txt file, we provide a dedicated user agent, BoldDesk-Bot. If you have restrictions in place for external crawlers, you can grant access to BoldDesk by whitelisting this specific user agent.
