Table Column Field Normalization#

When RAGFlow's table parser (rag/app/table.py) ingests an Excel, CSV, or TXT file, it must map human-readable column headers — including Chinese-language headers — into storage field names that conform to the Elasticsearch dynamic template convention. It does this through two sequential transforms: pinyin normalization and type-suffix appending.

How It Works#

1. Header Deduplication#

Before normalization, duplicate column names are made unique by _deduplicate_column_names(), which appends an incrementing numeric suffix to any repeated header. Multi-level (merged-cell) headers are joined with a hyphen 1.

2. Type Inference#

column_data_type() inspects each column's values and assigns one of five types: text, int, float, datetime, or bool. The winning type is the plurality type across all non-null values. Integers exceeding 2⁶³−1 are promoted to float.

3. Pinyin Conversion#

Inside chunk(), a Pinyin instance converts each column header to ASCII pinyin 2:

py_clmns = [PY.get_pinyins(re.sub(r"(/.*|（…）|\(…\))", "", str(n)), "_")[0] for n in clmns]

Before conversion, synonym/enum annotations are stripped — e.g., gender/sex(male, female) → gender 3. For purely ASCII headers, pinyin conversion is a no-op. For Chinese headers like 姓名, the result is xing_ming.

4. Type-Suffix Mapping#

The fields_map dict 4 defines the suffix for each inferred type:

clmns_map is then built by concatenating the lowercased pinyin name with the suffix 5:

clmns_map = [(py_clmns[i].lower() + fields_map[clmn_tys[i]], display_name) for i in range(len(clmns))]

Example: A Chinese column 姓名 inferred as text → storage key xing_ming_tks.

These suffixes are matched by dynamic field templates in conf/mapping.json, which automatically assign the correct Elasticsearch field type without any explicit schema definition 6.

5. Storage Engine Branching#

The resulting field name depends on the active storage engine 7:

• Elasticsearch: uses the full <pinyin>_<suffix> key (e.g., xing_ming_tks)
• Infinity / OceanBase: uses the plain pinyin key without suffix (e.g., xing_ming), and stores typed data in a chunk_data dict

For ES text columns, a <pinyin>_raw companion field is also stored with the unprocessed string value alongside the tokenized _tks field 8.

6. Column Roles#

If the knowledge base is configured with table_column_mode = "manual", each column's role can be indexing, vectorize, metadata, or both 9. Columns with metadata or both are stored as typed fields in the chunk; columns with indexing/vectorize roles contribute to content_with_weight for full-text/vector search only 10.

7. field_map Persistence#

After processing all sheets, the merged field_map (pinyin+suffix → display name) is persisted to the knowledge base record via KnowledgebaseService.update_parser_config() 11. This map drives the column-role UI selector so users see original column names rather than storage keys.

Key Entry Points#