flatterer integration

[jpmckinney]

I have a local repository (spoonbill-ng) that I last touched in February 2022, to try to use flatterer purely for flattening, while keeping the other high-level features of Spoonbill. It has a NOTES.txt file with the remaining code to read/review for integration.

Another option is to store some of the metadata files created by Flatterer, when the Data Registry converts the JSON to Excel and CSV. Those metadata files can then be used to do basic table customizations in spoonbill-web like: decide which tables and columns to include, and how to rename the headings. With this approach, the "shape" of the tables is unchanged.

If we want to offer other shapes (like combining documents into one sheet) we can do that by editing (pre-processing) the JSON (would need to add our own _link columns?). In terms of pre-processing, I think we can’t implement "roll-up multiple" easily (unless we name JSON fields like "fieldname/0"?). I tagged some issues that involve pre-processing the data before it’s flattened.

See also this Slack thread. Some notes:

• For translated field titles as column headings, we just need to provide an extended and translated schema to Flatterer.

[jpmckinney]

Here's the NOTES.txt:

Details

---- flatterer

---- spoonbill references from spoonbill-web

FileAnalyzer: analyzes files
- .__init__(workdir, root_tables=ROOT_TABLES, combined_tables=COMBINED_TABLES)
  TODO: change initialization? copy/paste
- .analyze_file(filepaths, with_preview=True)
  finds data array, calls spec.process_items on each array entry (SEE BELOW), sets table order
  TODO: determine use case for multiple file paths, then edit as needed
- .pkg_type ("releases", "records")
  easy to determine with ijson, using similar code as in Kingfisher Process
  TODO: eliminate the need for this
- .order (returns the order of the top-level properties in the schema, renaming "tender" to "tenders")
  TODO: copy/paste
- .spec (DataPreprocessor instance)
  SEE BELOW

DataPreprocessor: 
- .tables
  dict of table name to Table instance (SEE BELOW)
  TODO: incorporate logic from main() into FileAnalyzer
- [.process_items]
  counts rows per table, builds table previews (maybe more?)[TODO]
  TODO: replace with analyze
- DataPreprocessor.restore()
  restores state from dump file
  TODO: copy/paste?
- .dump()
  writes state to dump file
  TODO: copy/paste?
- .multiple_values
  whether JSON is concatenated
  TODO: eliminate the need for this
- .pkg_type
  "releases" or "records"
  TODO: eliminate the need for this
- .schema
  the release or record schema
  TODO: copy/paste?
- .total_items
  the number of releases or records
  DONE

Table: (relates to frontend logic) [TODO]
unclear
- .data
- .headers
both
- .name
- .parent

    :param name: Table name
           str
    :param parent: Parent table, None if this table is root table
           object = field(default_factory=dict)

spec dataclass
- .additional_columns
- .arrays
- .child_tables
- .columns
- .combined_columns
- .preview_rows
- .preview_rows_combined
- .splitted
- .titles
- .total_rows

    :param additional_columns: Columns identified in dataset but not in schema
           Mapping[str, Column] = field(default_factory=OrderedDict)
    :param columns: Columns extracted from schema for split version of this table
           Mapping[str, Column] = field(default_factory=OrderedDict)
    :param combined_columns: Columns extracted from schema for unsplit version of this table
           Mapping[str, Column] = field(default_factory=OrderedDict)

    :param arrays: Table array columns and maximum items (not the total count) in each array
           Mapping[str, int] = field(default_factory=dict)
    :param child_tables: List of possible child tables
           List[str] = field(default_factory=list)
    :param titles: All human-friendly column titles, extracted from the schema
           Mapping[str, str] = field(default_factory=dict)

    :param splitted: This table should be splitted
           bool = False
    :param total_rows: Total available rows in this table
           int = 0

    :param preview_rows: Generated preview for split version of this table
           Sequence[dict] = field(default_factory=list)
    :param preview_rows_combined: Generated preview for unsplit version of this table
           Sequence[dict] = field(default_factory=list)

- unused: is_root, is_combined, path, rolled_up, types

    :param is_combined: This table contains data collected from different paths
           bool = False
    :param is_root: This table is root table
           bool = False
    :param path: List of paths to gather data to this table
           List[str]
    :param rolled_up: This table should be ated from its parent
           bool = False
    :param types: All paths matched to this table with corresponding object type on each path
           Mapping[str, List[str]] = field(default_factory=dict)

    array_columns: Mapping[str, Column] = field(default_factory=OrderedDict)
    array_positions: Mapping[str, str] = field(default_factory=dict)

[Django model
- .array_tables
- .column_headings
- .heading
- .id
- .include
- .mergeable
- .should_split
- .split]

Column: [TODO]
- .id str The JSON path without indexes
- .hits int The number of times the column contains data during analysis

Functions that take DataPreprocessor as input:
- retrieve_tables() [spec.tables]
- get_column_headings()
- get_options_for_table()
- _split_table()
- store_preview_csv()

FileFlattener: flattens files
- .__init__
- .flatten_file

-- copied to ng

FlattenOptions: simple dataclass to describe the tables' design

get_reader: opens the file with gzip or not

schema/heading-related
- SchemaHeaderExtractor
- nonschema_title_formatter

constants
- CURRENT_SCHEMA_TAG
- DEFAULT_SCHEMA_URL
- COMBINED_TABLES
- ROOT_TABLES
- TABLE_THRESHOLD

[jpmckinney]

I've pushed the code as a branch (no shared history): https://github.com/open-contracting/spoonbill/tree/flatterer