iterate_columnar_dicts

iterate_columnar_dicts(inp)

Iterates columnar dict data as rows.

Useful for constructing rows/objects out from pyarrow.Table or pyarrow.RecordBatch.

Example:

@classmethod
def create_from_pyarrow_table(cls, table: pa.Table) -> "PairUniverse":
    pairs = {}
    for batch in table.to_batches(max_chunksize=5000):
        d = batch.to_pydict()
        for row in iterate_columnar_dicts(d):
            pairs[row["pair_id"]] = DEXPair.from_dict(row)

    return PairUniverse(pairs=pairs)

Parameters:

inp (Dict[str, list]) – Input dictionary of lists e.g. one from :py:method:`pyarrow.RecordBatch.to_pydict`. All lists in the input must be equal length.

Returns:

Iterable that gives one dictionary per row after transpose

Return type:

Iterable[Dict[str, object]]