Group Operations¶
A cross-sectional operator (rank, demean, zscore, quantile) compares each stock with every other stock that
day. A group operator does the same thing inside each group: a stock is compared only with the stocks that share
its label. That is how a signal is made sector-neutral, size-neutral, or neutral to anything else you can label.
Every example on this page uses the same nine stocks on one day:
| A | B | C | D | E | F | G | H | I | |
|---|---|---|---|---|---|---|---|---|---|
field('value') |
9 | 1 | 5 | 2 | 8 | 3 | 7 | 4 | 6 |
field('size') |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
group('sector') |
Tech | Tech | Fin | Fin | Tech | Fin | Tech | — | Tech |
H has no sector label (— is missing).
The operators¶
| Operation | Each stock gets |
|---|---|
group_rank(x, groups) |
its percentile within its group, in (0, 1] |
group_demean(x, groups) |
x minus its group's mean |
group_quantile(x, groups, n) |
its bucket within its group, 1 (lowest) to n |
group_sum(x, groups) |
its group's total (every member gets the same number; missing if the group has no values) |
group_std(x, groups) |
the standard deviation of x across its group's members that day |
group_count_valid(x, groups, period) |
how many of its group's values are present, over a rolling window |
groups is either a group's name ('sector', looked up in the data), or any expression that gives each stock
a label (below).
Every group operator works on one day's cross-section, except group_count_valid, which counts over a window. Each
member gets its group's number from that day, even a member whose own value is missing. A stock with no label, or
outside the universe, gets no value.
Labels given on fewer dates than the data (monthly sector labels on daily prices) hold from their date until the next row: each day takes the latest label row on or before it, never a later one, and a day before the first row has no label. (Before qex 0.12.0 a missing date took the last row, a look-ahead.)
Compare each operator with its whole-market version:
| A | B | C | D | E | F | G | H | I | |
|---|---|---|---|---|---|---|---|---|---|
demean(field('value')) |
4 | −4 | 0 | −3 | 3 | −2 | 2 | −1 | 1 |
group_demean(field('value'), 'sector') |
2.8 | −5.2 | 1.67 | −1.33 | 1.8 | −0.33 | 0.8 | — | −0.2 |
rank(field('value')) |
1 | 0.11 | 0.56 | 0.22 | 0.89 | 0.33 | 0.78 | 0.44 | 0.67 |
group_rank(field('value'), 'sector') |
1 | 0.2 | 1 | 0.33 | 0.8 | 0.67 | 0.6 | — | 0.4 |
group_sum(field('value'), 'sector') |
31 | 31 | 10 | 10 | 31 | 10 | 31 | — | 31 |
C has the highest value among the financials, so its group_rank is 1, though it is only in the middle of the market.
The Tech mean is 6.2 and the Fin mean is 3.33, so group_demean takes out the difference between the two sectors:
a book built from it is long and short within each sector, with no net bet on Tech against Fin.
group_demean(returns(20), 'sector') # sector-neutral momentum
group_rank(-returns(5), 'sector') # reversal, ranked within each sector
adv(20) / group_sum(adv(20), 'sector') # each stock's share of its sector's trading
A stock with no label¶
A stock with no label is in no group, so a group operator gives it no value, even when every other field is present. H above has none, and the reason recorded for its missing value is no group.
The other stocks are not affected: H is in no group, so it moves no group's mean or ranks. A book takes no position in H. Two ways to keep it, when you want to:
| H | the others | |
|---|---|---|
coalesce(group_demean(field('value'), 'sector'), 0) |
0 | unchanged |
group_demean(field('value'), coalesce(group('sector'), 'Other')) |
0 | unchanged |
The first says "a stock with no sector sits at its sector's average". The second puts every unlabelled stock in a
group of its own, named 'Other', and here gives the same answer, because H is alone in it and a group of one
demeans to 0. With two unlabelled stocks the second would demean them against each other: a pair trade between two
unrelated names, chosen by a gap in the data. Prefer the first.
Labels from any expression¶
Anything that gives each stock a label on each day can be a grouping: a number, a string, the result of a comparison. The common case is a bucket:
| A | B | C | D | E | F | G | H | I | |
|---|---|---|---|---|---|---|---|---|---|
quantile(field('size'), 3) |
1 | 1 | 1 | 2 | 2 | 2 | 3 | 3 | 3 |
group_demean(field('value'), quantile(field('size'), 3)) |
4 | −4 | 0 | −2.33 | 3.67 | −1.33 | 1.33 | −1.67 | 0.33 |
group_demean(returns(20), quantile(field('mcap'), 5)) # size-neutral momentum
group_rank(returns(20), quantile(volatility(60), 3)) # rank within volatility terciles
let size = quantile(field('mcap'), 3) in group_demean(returns(20), size) # a grouping can be named
A name and the expression that gives the same labels are the same grouping:
group_demean(x, 'sector') and group_demean(x, group('sector')) agree everywhere.
Things to know:
- Buckets are recomputed every day. A stock near a boundary switches groups from one day to the next, and each
switch moves its position: a source of turnover, like a universe that flickers. Bucket a slow quantity
(
quantile(ts_mean(field('mcap'), 21), 5)rather than today's market cap) to damp it. - Buckets come from the universe.
quantileis computed over the universe's members, so each bucket holds its share of the members, and a stock outside the universe has no bucket. - A missing input means no label. A stock with no market cap has no size bucket, so a size-neutral signal has
no value for it;
explain_missingsaysno data, at field('mcap'). - Labels compare by value.
1and1.0are one group;0is a label like any other. - A grouping must give a label per stock per day:
group_demean(x, 3)is refused, with a message saying whatgroupscan be.
Double sorts¶
A double sort buckets stocks on two characteristics. There are two kinds, and they differ.
Independent: bucket each characteristic over the whole market. Each stock's cell is the pair.
quantile(field('size'), 3) # size tercile
quantile(field('value'), 3) # value tercile, among all stocks
Dependent (conditional): bucket on the first, then on the second within each bucket of the first. Value is then measured against stocks of similar size:
| A | B | C | D | E | F | G | H | I | |
|---|---|---|---|---|---|---|---|---|---|
| size tercile | 1 | 1 | 1 | 2 | 2 | 2 | 3 | 3 | 3 |
| value tercile, independent | 3 | 1 | 2 | 1 | 3 | 1 | 3 | 2 | 2 |
| value tercile, dependent | 3 | 1 | 2 | 1 | 3 | 2 | 3 | 1 | 2 |
F's value of 3 is in the bottom third of the market but in the middle of its size bucket (2, 3, 8); H's 4 is the reverse. In a dependent sort every size bucket splits into value buckets of the same size (as nearly as the counts allow), so a long-short on value is not secretly a bet on size.
A long-short book from the dependent sort, long the top value tercile and short the bottom within each size bucket:
let v = group_quantile(field('value'), quantile(field('size'), 3), 3)
in where(v == 3, 1, where(v == 1, -1, None))
The middle tercile gets None, so it takes no position.
group_quantile(x, groups, n) is quantile applied inside each group: the same as writing out, for each bucket,
quantile(where(b == k, x, None), n), which is how a dependent sort had to be written before qex 0.8.0. Like
quantile, it buckets by percentile, so a group with fewer stocks than buckets skips some buckets, and a group of one
is in the top bucket.
Crossing groupings¶
group_cross(a, b, ...) combines two or more groupings into one: each stock's label is its labels joined with |.
Its arguments are names or label expressions.
| A | B | C | D | E | F | G | H | I | |
|---|---|---|---|---|---|---|---|---|---|
group_cross('sector', quantile(field('size'), 3)) |
Tech|1 | Tech|1 | Fin|1 | Fin|2 | Tech|2 | Fin|2 | Tech|3 | — | Tech|3 |
group_demean(field('value'), group_cross(...)) |
4 | −4 | 0 | −0.5 | 0 | 0.5 | 0.5 | — | −0.5 |
group_demean(returns(20), group_cross('sector', quantile(field('mcap'), 3))) # sector- and size-neutral
where(group_cross('sector', quantile(field('mcap'), 3)) == 'Technology|3', 1, None)
A stock missing any part has no label (H has no sector). Crossing multiplies the number of groups and shrinks each
one: here C and E are alone in their cells, so they demean to 0 and take no position. Count the stocks per cell
before trusting a cross on a real universe (group_count_valid(x, groups, 1) gives each stock its cell's count).
A label containing | is refused, since 'a|b' with 'c' and 'a' with 'b|c' would join to the same label.
Numbers and labels¶
Every qex expression gives a value per date and stock. Most give numbers. A few give labels:
group('sector'), group_cross(...), a string such as 'Other', and coalesce or where whose branches are labels.
Labels can be:
- compared, with
==and!=:group('sector') == 'Technology'is 1, 0, or missing; - used to group: the
groupsof any group operator, and the parts ofgroup_cross; - passed through
coalesceandwhere, as long as both sides are labels (or one isNone).
Anywhere else needs numbers, and qex refuses the expression when it is read, before anything runs:
| Written | Refused with |
|---|---|
rank(group('sector')) |
group('sector') gives labels, not numbers, and rank() needs numbers. Compare labels (group('sector') == 'Technology') or group by them (group_demean(x, group('sector'))). |
coalesce(group('sector'), 0) |
coalesce(group('sector'), 0) mixes labels and numbers: both sides must give the same kind |
Numbers can serve as labels (a quantile bucket is a number), so a group operator accepts either. A field read with
field('name') is taken to be numbers; read a field of labels with group('name'). An alpha must give numbers.
Inside the cross-sectional universe¶
Like every cross-sectional operator, a group operator compares only the stocks in the universe that day
(Universes); a stock outside it gets no value. Labels from an expression are computed the same way, so
quantile(field('mcap'), 5) buckets the members only.
See also¶
- Groups in the operator reference, with a worked example of each.
- Missing values:
None, why a value is missing, three-valued logic andcoalesce.