Large Datasets: LazyData¶
LazyData holds a loader per field instead of the field itself, and loads a field the first time an expression reads
it. An expression that uses only close never loads volume or earnings.
from qex import qex, LazyData
data = LazyData({
'close': lambda: pd.read_parquet('data/close.parquet'),
'volume': lambda: pd.read_parquet('data/volume.parquet'),
'earnings': lambda: pd.read_parquet('data/earnings.parquet'),
})
result = qex("rank(returns(20))").evaluate(data) # loads close only
A loaded field is kept, so it is read from disk once. Values can be loaders or frames already in memory, mixed.
Describing the fields¶
descriptions documents each field where the data is defined:
data = LazyData(
data={
'close': lambda: pd.read_parquet('close.parquet'),
'earnings': lambda: pd.read_parquet('earnings.parquet'),
},
descriptions={
'close': 'Daily close, adjusted for splits and dividends',
'earnings': 'Quarterly EPS surprise, (actual - estimate) / |estimate|, on the announcement date',
},
)
print(data.describe()) # every field and its description
print(data.describe('earnings')) # one field
data.descriptions # the dict
LazyData ¶
Wrapper for data dict that supports lazy loading via callables.
Values can be DataFrames or callables that return DataFrames. Callables are invoked on first access and cached.
Supports optional field descriptions for self-documenting data, enabling AI agents to understand available fields without external docs.
Example
data = LazyData({ 'close': lambda: pd.read_parquet('close.parquet'), 'volume': volume_df, # Already loaded })
'close' is loaded only when first accessed¶
close_data = data['close']
Example with descriptions
data = LazyData( data={ 'close': close_df, 'volume': volume_df, 'sector': sector_df, }, descriptions={ 'close': 'Daily closing price, adjusted for splits and dividends', 'volume': 'Daily trading volume in shares', 'sector': 'GICS sector classification string', } )
data.describe() # Pretty-print available fields
describe ¶
Pretty-print available fields with descriptions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
field
|
Optional[str]
|
If provided, describe only this field. Otherwise describe all. |
None
|
Returns:
| Type | Description |
|---|---|
str
|
Formatted string describing the field(s). |
Example output
Available fields: close: Daily closing price, adjusted for splits and dividends volume: Daily trading volume in shares sector: GICS sector classification string