XHS API Response Parsing#

Overview#

XHS-Downloader processes XiaoHongShu posts by extracting structured data from server-rendered HTML responses. The parsing pipeline has three sequential stages: HTML → JSON extraction, JSON → Namespace wrapping, and Namespace → typed field extraction for metadata and download links.

Pipeline Architecture#

Stage 1: HTML → JSON (Converter)#

Source: source/expansion/converter.py

Converter.run() chains three steps:

1. _extract_object(html) 1 — parses the HTML via lxml XPath (//script/text()), then finds the script starting with window.__INITIAL_STATE__.
2. _convert_object(text) 2 — strips the window.__INITIAL_STATE__= prefix, removes illegal YAML control characters, then calls yaml.safe_load() to produce a Python dict.
3. _filter_object(data) 3 — navigates the dict with two fallback key paths:
   • Phone: noteData → data → noteData
   • PC: note → noteDetailMap → [-1] → note

Returns the note-level dict, or {} on failure.

Stage 2: Dict → Namespace Object#

Source: source/expansion/namespace.py

Namespace.__init__ calls generate_data_object(), which recursively converts every nested dict into a SimpleNamespace and every list into a list of SimpleNamespaces (or scalars).

The primary accessor is safe_extract(attribute_chain, default):

• Splits attribute_chain on . and traverses attributes via getattr.
• Supports bracket-index notation: "imageList[0]" indexes into a list element.
• Returns default (empty string by default) at any missing step — never raises.

The class method Namespace.object_extract(data_object, attribute_chain) applies the same logic to an already-unwrapped SimpleNamespace (used when iterating list items like individual images).

In app.py, the Namespace is created at __generate_data_object() or via json_to_namespace() for script-server mode.

Stage 3: Field Extraction#

Metadata — Explore#

Source: source/application/explore.py

Explore.run(data) returns a flat dict with all post metadata:

4

Content type is classified by __classify_works(): "video" type with a single imageList entry → 视频；"video" with multiple → 图集；"normal" → 图文.

Image Download Links — Image#

Source: source/application/image.py

Image.get_image_link(data, format_):

1. Extracts imageList via data.safe_extract("imageList", []).
2. Tries urlDefault per item first, falls back to url 5.
3. __extract_image_token(url) strips the CDN prefix and variant suffix: "/".join(url.split("/")[5:]).split("!")[0].
4. Builds final URLs:
   • auto format: https://sns-img-bd.xhscdn.com/{token} 6
   • Fixed format (png/webp/jpeg/heic/avif): https://ci.xiaohongshu.com/{token}?imageView2/format/{format_} 7
5. Also returns live-image links per item from stream.h264[0].masterUrl 8.

Returns (download_urls: list, live_urls: list) → stored as container["下载地址"] and container["动图地址"].

Video Download Links — Video#

Source: source/application/video.py

Video.deal_video_link(data, preference) first checks video.consumer.originVideoKey for a direct CDN key; if absent, collects H.264/H.265 stream objects from video.media.stream.h264 / video.media.stream.h265, sorts by preference (resolution / bitrate / size), and returns the best backupUrls[0] or masterUrl. (See Video Quality Selection)

Container Assembly (app.py)#

Source: source/application/app.py

__deal_extract() orchestrates the full pipeline:

1. Fetches HTML → wraps in Namespace 9.
2. Calls _extract_data() → Explore.run() → metadata dict 10.
3. Merges 作品链接 into the dict and calls _deal_download_tasks() 11.
4. _deal_download_tasks() routes by 作品类型: __extract_video() or __extract_image() populate 下载地址 and 动图地址.
5. save_data() serializes lists to space-separated strings before writing to the data recorder.

The script-server path (deal_script_tasks) accepts pre-parsed JSON directly and wraps it via json_to_namespace() 12, skipping the HTML fetching stage.

Key Source Files#