Writing Extracted CAD Metadata to Parquet

Extracted CAD and BIM attributes are heterogeneous by nature — a block attribute, an XDATA value and an IFC property set have nothing structurally in common — so the table that holds them needs a small stable core of typed columns plus a map column for everything else. Partition by source so a re-extraction replaces one partition, and write provenance alongside so a query result can be traced to a delivery. This page is part of Metadata Extraction Strategies.

Why the Schema Is Mostly a Map

The three extraction mechanisms produce different shapes. A block attribute is a tag and a string. XDATA is a nested tree under an application identifier. An IFC property set is a named bag of typed values. Any schema wide enough to hold all of them as columns is mostly nulls, and any schema narrow enough to be useful excludes most of what was extracted.

The design that survives is the same one that works for property sets in a database: a handful of columns every source can supply, and a map for the rest.

The core columns are the ones a query filters or joins on — a stable identifier, the source file, the entity type, the layer or class, and a reference to where the geometry lives. Everything else goes in a map<string, string>, which Parquet stores efficiently and which needs no migration when a new delivery brings a new attribute.

Types in the map are a deliberate loss. A value that matters enough to be filtered numerically should be promoted to a typed column; a value that is read and displayed does not need to be. Deciding that per attribute, rather than trying to infer types across sources, is what keeps the table stable.

Production-Ready Script

# pyarrow>=14.0, Python 3.9+
from __future__ import annotations

from dataclasses import dataclass, asdict
from datetime import datetime, timezone
from pathlib import Path
import json

import pyarrow as pa
import pyarrow.parquet as pq

# The core is explicit and small. Everything else lives in the attributes map.
SCHEMA = pa.schema([
    pa.field("entity_id", pa.string(), nullable=False),      # stable, per source
    pa.field("source_file", pa.string(), nullable=False),
    pa.field("source_format", pa.string(), nullable=False),  # dxf | ifc | gml
    pa.field("entity_type", pa.string(), nullable=False),
    pa.field("layer_or_class", pa.string()),
    pa.field("geometry_ref", pa.string()),                   # key into the geometry store
    pa.field("attributes", pa.map_(pa.string(), pa.string())),
])


@dataclass(frozen=True)
class WriteProvenance:
    extractor_version: str
    source_file: str
    source_mtime: float
    rows: int
    written_at: str


def _as_map(d: dict) -> list[tuple[str, str]]:
    """Coerce deliberately: None becomes an absent key, everything else a string."""
    return [(str(k), str(v)) for k, v in d.items() if v is not None]


def write_partition(
    records: list[dict],
    root: Path,
    *,
    source_file: str,
    source_format: str,
    extractor_version: str,
) -> WriteProvenance:
    """One partition per source file — a re-extraction replaces it, not the dataset."""
    if not records:
        raise ValueError(f"{source_file}: nothing to write")

    table = pa.Table.from_pydict({
        "entity_id": [r["entity_id"] for r in records],
        "source_file": [source_file] * len(records),
        "source_format": [source_format] * len(records),
        "entity_type": [r["entity_type"] for r in records],
        "layer_or_class": [r.get("layer_or_class") for r in records],
        "geometry_ref": [r.get("geometry_ref") for r in records],
        "attributes": [_as_map(r.get("attributes") or {}) for r in records],
    }, schema=SCHEMA)

    partition = root / f"source_format={source_format}" / f"source={Path(source_file).stem}"
    partition.mkdir(parents=True, exist_ok=True)
    pq.write_table(table, partition / "part-0.parquet", compression="zstd")

    prov = WriteProvenance(
        extractor_version=extractor_version,
        source_file=source_file,
        source_mtime=Path(source_file).stat().st_mtime,
        rows=len(records),
        written_at=datetime.now(timezone.utc).isoformat(),
    )
    (partition / "_provenance.json").write_text(json.dumps(asdict(prov), indent=2))
    return prov


def read_dataset(root: Path, *, columns: list[str] | None = None) -> pa.Table:
    """Column projection is the reason this is Parquet and not CSV."""
    return pq.read_table(root, columns=columns)


if __name__ == "__main__":
    prov = write_partition(
        [{"entity_id": "3vB2YO", "entity_type": "IfcWall",
          "layer_or_class": "IfcWall", "geometry_ref": "geom/3vB2YO",
          "attributes": {"FireRating": "EI60", "LoadBearing": "True"}}],
        Path("./metadata"), source_file="model.ifc", source_format="ifc",
        extractor_version="1.4.0",
    )
    print(prov)

Key implementation notes:

• The schema is declared, not inferred. Inference from the first chunk is how a column becomes a string because one early row had a stray value.
• _as_map drops None rather than storing an empty string, so absence and emptiness stay distinguishable — a distinction that matters in this domain more than most.
• Partitioning on source format and source file means a re-extraction of one drawing rewrites one directory. Rewriting the whole dataset for one changed input is the failure mode this avoids.
• Provenance is written per partition, so a query result can be traced to the extractor version and the source file’s state at extraction time.
• zstd compression is a good default for this data: string-heavy, highly repetitive, and read far more often than written.

Compatibility Matrix

Fallback Strategies

1. A reader cannot handle the map column. Some older engines do not. Explode the map into key-value rows in a companion table, or promote the keys that matter into typed columns.

2. Attribute names collide across sources. Prefix by source format or by property set, as with the IFC property set flattening. Collisions in a map are silent overwrites.

3. Partitions proliferate. One partition per drawing on a delivery of thousands produces many small files. Group by delivery rather than by file where individual re-extraction is not needed.

4. A value should have been numeric. Promote it to a typed column in the core schema and backfill. This is a migration and should be a deliberate one; the map exists so it can be deferred until the value is worth it.

5. Provenance drifts from the data. Write it in the same operation as the table, as above, so a partition without provenance means an interrupted write and can be re-run.

FAQ

Related Pages

• Metadata Extraction Strategies — parent reference on the three metadata mechanisms this table normalises
• Extracting Block Attributes from CAD Files with ezdxf — one of the extractors that feeds this table
• Mapping IFC Property Sets to PostGIS Columns — the same typed-core-plus-tail design in a database