Skip to content

Large Datasets: LazyData

LazyData holds a loader per field instead of the field itself, and loads a field the first time an expression reads it. An expression that uses only close never loads volume or earnings.

from qex import qex, LazyData

data = LazyData({
    'close':    lambda: pd.read_parquet('data/close.parquet'),
    'volume':   lambda: pd.read_parquet('data/volume.parquet'),
    'earnings': lambda: pd.read_parquet('data/earnings.parquet'),
})

result = qex("rank(returns(20))").evaluate(data)    # loads close only

A loaded field is kept, so it is read from disk once. Values can be loaders or frames already in memory, mixed.

Describing the fields

descriptions documents each field where the data is defined:

data = LazyData(
    data={
        'close':    lambda: pd.read_parquet('close.parquet'),
        'earnings': lambda: pd.read_parquet('earnings.parquet'),
    },
    descriptions={
        'close':    'Daily close, adjusted for splits and dividends',
        'earnings': 'Quarterly EPS surprise, (actual - estimate) / |estimate|, on the announcement date',
    },
)

print(data.describe())            # every field and its description
print(data.describe('earnings'))  # one field
data.descriptions                 # the dict

LazyData

Wrapper for data dict that supports lazy loading via callables.

Values can be DataFrames or callables that return DataFrames. Callables are invoked on first access and cached.

Supports optional field descriptions for self-documenting data, enabling AI agents to understand available fields without external docs.

Example

data = LazyData({ 'close': lambda: pd.read_parquet('close.parquet'), 'volume': volume_df, # Already loaded })

'close' is loaded only when first accessed

close_data = data['close']

Example with descriptions

data = LazyData( data={ 'close': close_df, 'volume': volume_df, 'sector': sector_df, }, descriptions={ 'close': 'Daily closing price, adjusted for splits and dividends', 'volume': 'Daily trading volume in shares', 'sector': 'GICS sector classification string', } )

data.describe() # Pretty-print available fields

descriptions property

descriptions: Dict[str, str]

Return the field descriptions dictionary.

describe

describe(field: Optional[str] = None) -> str

Pretty-print available fields with descriptions.

Parameters:

Name Type Description Default
field Optional[str]

If provided, describe only this field. Otherwise describe all.

None

Returns:

Type Description
str

Formatted string describing the field(s).

Example output

Available fields: close: Daily closing price, adjusted for splits and dividends volume: Daily trading volume in shares sector: GICS sector classification string

__repr__

__repr__() -> str

Return a concise representation.