Skip to content

The web engine does not flatten nested JSON objects, so a nested column is dropped there but profiled in Python #992

Closed

Description

@yakew7

Where: assets/profiler-engine.js recordsToTable() / parseJSON(), faircode/loaders_extra.py read_table() (the json_normalize call, #844), tests/test_json_edge_cases.py.

Current gap: the Python loader flattens a records-style JSON whose values are objects into dotted columns (loc.state), so {"loc": {"state": "TX"}} becomes a profiled column. The JS parser keeps the object as one cell, and since #973 such a column is skipped, so the same file gives different dimensions on the CLI and on the web.

Repro:

$ cat nested.json     # 30 records: {"sex": "m"|"f", "loc": {"state": "TX"|"CA"}}
$ python3 -c "from faircode.loaders_extra import read_table; print(list(read_table('nested.json').columns))"
['sex', 'loc.state']
$ node scripts/engine-js.js profile-json nested.json | python3 -c "import json,sys; print([d['name'] for d in json.load(sys.stdin)['dimensions']])"
['sex']

Suggested approach: flatten nested plain objects in recordsToTable() with the same dotted-name rule as pandas.json_normalize (lists stay as cells), and add the file above to the JSON parity tests.

Activity

  1. added
    bugSomething isn't working
    help wantedExtra attention is needed
    on Oct 15, 2026
  2. pinned this issue on Oct 15, 2026
  3. yakew7 commented on Oct 16, 2026

    @yakew7
    OwnerAuthor

    Fixed in 368201f (on main). Full suite green (941 passed).

  4. unpinned this issue on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions