Web Scraping Route Management#

RSSHub's 1,600+ routes scrape upstream websites and APIs to generate RSS feeds. Because scrapers are tightly coupled to the upstream site's HTML structure, URL layout, and API contracts, upstream changes are the primary cause of route breakage. This article covers how scraping routes are structured, the categories of upstream changes that break them, and the fix patterns used to restore them.

Route Structure#

Every scraping route is a TypeScript module under lib/routes/<namespace>/ that exports a Route object and an async handler function. The core scraping pipeline is:

1. Fetch the upstream page or API with ofetch (auto-retry on 4xx/5xx, proxy fallback) or the got wrapper 1.
2. Parse HTML with cheerio's load(), then query via CSS selectors — e.g., $('table.list-table tbody tr') 2.
3. Map DOM elements to DataItem objects, extracting title, link, date (parseDate utility, 3), category, image, and enclosure fields.
4. Return a Data object with feed metadata and items 4.

The Route object also declares a radar field — an array of URL-pattern rules used by the RSSHub Radar browser extension to auto-detect the right RSSHub route from the user's current webpage 5. Radar rules must be updated alongside handler URLs since they encode the same upstream URL assumptions.

Key files for a typical route:

Common Upstream Changes That Break Routes#

1. URL Structure Changes#

Upstream sites reorganize URL paths, switch from query parameters to path segments, or migrate domains. The route's hardcoded base URL or URL-construction logic becomes stale.

Example — LHRatings redesign (Issue #22832, open as of 2026-07-26):
The route constructed URLs as research.html?type={type} 7, but the site redesigned to use path-based URLs (lists/92.html, lists/93.html, etc.), returning 404 on the old pattern 8.

Required fix: Update the URL-building logic to map the type parameter to the new path (e.g., type=1 → lists/92.html). The CSS selector (table.list-table tbody tr, 2) also needs re-verification since a redesign typically changes the HTML structure 9.

Example — Zaobao finance route (Issue #22707):
The zfinance section hardcoded /finance/realtime but the upstream page moved to /realtime/finance, causing 404/503 errors. Fix: update the URL path in the route and submit via PR #22705 10.

Example — JIHS IDWR route (Issue #22684):
The 2026 IDWR pages dropped the /jp/ URL segment (old: .../idwr/jp/idwr/{year}/, new: .../idwr/idwr/{year}/index.html), causing 404s for year=2026 11. Both the handler URL and the Radar source path needed updating.

2. CSS Selector / HTML Structure Changes#

When an upstream site redesigns its frontend, existing CSS selectors target elements that no longer exist or have new class names.

Example — CHNMuseum redesign (PR #22383):
Three routes were broken by a CHNMuseum frontend overhaul 12:

• News list selector: ul.cj_xushuliebao_list li → ul.xly_list_ts li
• Link selector: a → a.titles
• Date selector: span.date → .times span.sp
• Exhibition selector: ul[id="div"] a.recurl → ul[id="div"] > li > a

The fix also introduced dayjs for more robust date formatting and a seenLinks Set to deduplicate items, since the new HTML structure could surface duplicates 12.

3. API Response Schema Changes#

For routes that call JSON APIs, the upstream may restructure the response payload, changing field names or nesting.

Example — Fanbox API change (Issue #22817):
On 2026-07-22, Fanbox's post.listCreator endpoint moved the post array from body (array) to body.posts (nested object), causing TypeError: postListResponse.body.map is not a function 13. Fix: update the accessor (body.map() → body.posts.map()) and the TypeScript type in types.ts.

4. Route Parameter Remapping#

When upstream URL semantics change (e.g., numeric IDs replace keyword slugs), route parameters and Radar targets must be remapped. The radar[].target function in the Route object extracts URL parameters and maps them to RSSHub route parameters 14; this function must also be updated when source URL structure changes.

Fix Checklist#

When an upstream change is reported:

1. Identify the failure mode — 404/503 (URL change), empty feed or selector mismatch (HTML change), or TypeError/undefined (API schema change).
2. Verify the new upstream URL by visiting the site and checking the current path/query structure.
3. Update the URL-building logic in the handler — base URL, path construction, or query-parameter-to-path mappings.
4. Re-inspect CSS selectors with the new HTML — open DevTools on the live page, find the new selector for the list container, item link, date, and image.
5. Update Radar rules (radar[].source and radar[].target) to match the new URL pattern 5.
6. Update TypeScript types if the API response schema changed.
7. Test locally with pnpm dev and confirm the feed returns items.

Maintainer Triage Workflow#

RSSHub uses a GitHub Actions bot to automatically parse issue reports, identify the broken route path, and tag the route's listed maintainer(s) 15 16. Maintainers are listed in the maintainers field of the Route object 17. If a maintainer does not respond, any contributor can submit a fix PR.

Issues recur for the same routes when upstream sites undergo periodic redesigns — JIHS broke twice across different domain migrations 11, and LHRatings is experiencing its first known breakage from a full site overhaul 18.

References#

• LHRatings research route implementation — canonical example of URL construction, CSS selector use, and Radar configuration
• CHNMuseum redesign fix (PR #22383) — complete before/after example of selector and parameter remapping
• LHRatings breakage issue #22832 18 — active URL restructuring case
• Zaobao URL change issue #22707 10 — path segment reorder case
• Fanbox API schema change issue #22817 13 — JSON API response breakage
• JIHS IDWR route — known URL breakage — recurring domain/path migration case
• lib/utils/ofetch.ts · lib/utils/parse-date.ts — shared scraping utilities