mirror of
https://github.com/lobehub/lobehub.git
synced 2026-09-20 04:56:13 +08:00
Arvin Xu
be3d4c53d5
Search1API's service-quality audit flagged four patterns of wasted calls from LobeHub: URLs that fail schema validation (422), identical retries of empty search results, crawls of icons/fonts/media, and retries of authoritative dead links (404/410, now billed as a completed answer). - web-crawler: add a URL preflight (`normalizeCrawlUrl` repairs bare domains, markdown links and stray punctuation; `getNonDocumentExtension` rejects image/font/media/archive files) that runs before any provider is contacted. Trimming is done with linear-time character loops rather than anchored `[…]+$` regexes, which backtrack quadratically on model-controlled input. - web-crawler: typed `InvalidUrlError` / `UnsupportedContentError` / `HTTPStatusError(status)`, 410 treated like 404, `naive` rejects non-document Content-Types, and a single `isRetryableCrawlError` rule. A confirmed 404/410 short-circuits the impl fallback chain and the error result carries `retryable: false` plus model-facing guidance. - server SearchService: `crawlWithRetry` no longer retries non-retryable results. - prompts: empty search results and `<error>` crawl items now tell the model the answer is final and how to rewrite instead of resending; error items keep their url. Web-browsing manifest and system role updated accordingly. - builtin-tool-web-browsing had no vitest config so its test file never ran; add one and fix the stale assertion. Co-authored-by: Arvin Xu <arvin.x@lobehub.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@lobechat/web-crawler
LobeHub's built-in web crawling module for intelligent extraction of web content and conversion to Markdown format.
📝 Introduction
@lobechat/web-crawler is a core component of LobeHub responsible for intelligent web content crawling and processing. It extracts valuable content from various webpages, filters out distracting elements, and generates structured Markdown text.
🛠️ Core Features
- Intelligent Content Extraction: Identifies main content based on Mozilla Readability algorithm
- Multi-level Crawling Strategy: Supports multiple crawling implementations including basic crawling, Jina, Search1API, and Browserless rendering
- Custom URL Rules: Handles specific website crawling logic through a flexible rule system
🤝 Contribution
Web structures are diverse and complex. We welcome community contributions for specific website crawling rules. You can participate in improvements through:
How to Contribute URL Rules
- Add new rules to the urlRules.ts file
- Rule example:
// Example: handling specific websites
const url = [
// ... other URL matching rules
{
// URL matching pattern, supports regex
urlPattern: 'https://example.com/articles/(.*)',
// Optional: URL transformation, redirects to an easier-to-crawl version
urlTransform: 'https://example.com/print/$1',
// Optional: specify crawling implementation, supports 'naive', 'jina', 'search1api', and 'browserless'
impls: ['naive', 'jina', 'search1api', 'browserless'],
// Optional: content filtering configuration
filterOptions: {
// Whether to enable Readability algorithm for filtering distracting elements
enableReadability: true,
// Whether to convert to plain text
pureText: false,
},
},
];
Rule Submission Process
- Fork the LobeHub repository
- Add or modify URL rules
- Submit a Pull Request describing:
- Target website characteristics
- Problems solved by the rule
- Test cases (example URLs)
📌 Note
This is an internal module of LobeHub ("private": true), designed specifically for LobeHub and not published as a standalone package.