Toolkit

This chapter provides formal definitions and properties of each toolkit function.

Synopsis

One-Sample Estimators

  • Center(x)\operatorname{Center}(\mathbf{x}) — robust location; median of pairwise averages.
    Like the mean but stable with outliers; tolerates up to 29% corrupted data.
  • CenterBounds(x,misrate)\operatorname{CenterBounds}(\mathbf{x}, \mathrm{misrate}) — bounds on center with error rate =misrate= \mathrm{misrate}.
    Exact under weak symmetry.
  • Spread(x)\operatorname{Spread}(\mathbf{x}) — robust dispersion; median of pairwise absolute differences.
    Same units as data; tolerates up to 29% corrupted data.
  • RelSpread(x)\operatorname{RelSpread}(\mathbf{x}) — relative dispersion =Spread/Center= \operatorname{Spread} / \lvert \operatorname{Center} \rvert.
    Dimensionless; compares variability across scales.

Two-Sample Estimators

  • Shift(x,y)\operatorname{Shift}(\mathbf{x}, \mathbf{y}) — robust location difference; median of pairwise differences.
    Negative means first sample tends to be lower.
  • ShiftBounds(x,y,misrate)\operatorname{ShiftBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate}) — bounds on shift with error rate =misrate= \mathrm{misrate}.
    If bounds exclude zero, the difference is reliable.
  • Ratio(x,y)\operatorname{Ratio}(\mathbf{x}, \mathbf{y}) — robust multiplicative ratio via log-space shift.
    For positive-valued quantities (latency, price, concentration).
  • RatioBounds(x,y,misrate)\operatorname{RatioBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate}) — bounds on ratio with error rate =misrate= \mathrm{misrate}.
    If bounds exclude 1, the multiplicative difference is reliable.
  • AvgSpread(x,y)\operatorname{AvgSpread}(\mathbf{x}, \mathbf{y}) — pooled robust dispersion, weighted by sample sizes.
  • Disparity(x,y)\operatorname{Disparity}(\mathbf{x}, \mathbf{y}) — robust effect size =Shift/AvgSpread= \operatorname{Shift} / \operatorname{AvgSpread} (robust Cohens d).

Randomization

  • Rng(s)\operatorname{Rng}(s) — deterministic pseudorandom generator from seed ss.
    Identical sequences across all supported languages.
  • r.Sample(x,k)r.\operatorname{Sample}(\mathbf{x}, k) — select kk elements without replacement.
  • r.Shuffle(x)r.\operatorname{Shuffle}(\mathbf{x}) — uniformly random permutation.
  • r.Resample(x,k)r.\operatorname{Resample}(\mathbf{x}, k) — select kk elements with replacement.

One-Sample Estimators

Center

Center(x)=Median1ijnxi+xj2\operatorname{Center}(\mathbf{x}) = \underset{1 \leq i \leq j \leq n}{\operatorname{Median}} \frac{x_i + x_j}{2}

Robust measure of location (central tendency).

  • Also known as — Hodges-Lehmann estimator, pseudomedian
  • Asymptotic — median of the average of two random measurements from XX
  • ComplexityO(n2logn)O(n^2 \log n) naive, O(nlogn)O(n \log n) fast (see Fast Center)
  • Domain — any real numbers
  • Unit — same as measurements

Properties

  • Shift equivariance Center(x+k)=Center(x)+k\operatorname{Center}(\mathbf{x} + k) = \operatorname{Center}(\mathbf{x}) + k
  • Scale equivariance Center(kx)=kCenter(x)\operatorname{Center}(k \cdot \mathbf{x}) = k \cdot \operatorname{Center}(\mathbf{x})

Example

  • Center([0, 2, 4, 6, 8]) = 4
  • Center(x + 10) = 14 Center(3x) = 12

References

  • Hodges & Lehmann 1963
  • Sen 1963

Center\operatorname{Center} is the recommended default for representing where the data is. It works like the familiar mean but does not break when the data contains a few bad measurements or outliers. Up to 29% of data can be corrupted before Center\operatorname{Center} becomes unreliable. When data is clean, Center\operatorname{Center} is nearly as precise as the mean (95% efficiency), so the added protection comes at almost no cost. When uncertain whether to use mean, median, or something else, start with Center\operatorname{Center}.

CenterBounds

CenterBounds(x,misrate)=[w(kleft),w(kright)]\operatorname{CenterBounds}(\mathbf{x}, \mathrm{misrate}) = [w_{(k_{\text{left}})}, w_{(k_{\text{right}})}]

where w=xi+xj2\mathbf{w} = { \frac{x_i + x_j}{2} } (pairwise averages, sorted) for iji \leq j, kleft=SignedRankMargin/2)+1k_{\text{left}} = \lfloor \operatorname{SignedRankMargin} / \rfloor2) + 1, kright=NSignedRankMargin/2)k_{\text{right}} = N - \lfloor \operatorname{SignedRankMargin} / \rfloor2), and N=n(n+1)2N = \frac{n(n+1)}{2}

Robust bounds on Center(x)\operatorname{Center}(\mathbf{x}) with specified coverage.

  • Also known as — Wilcoxon signed-rank confidence interval for Hodges-Lehmann pseudomedian
  • Interpretationmisrate\mathrm{misrate} is probability that true center falls outside bounds
  • Domain — any real numbers, n2n \geq 2, misrate21n\mathrm{misrate} \geq 2^{1-n}
  • Unit — same as measurements
  • Note — assumes weak symmetry and weak continuity; exact for n63n \leq 63, Edgeworth approximation for n>63n > 63

Properties

  • Shift equivariance CenterBounds(x+k,misrate)=CenterBounds(x,misrate)+k\operatorname{CenterBounds}(\mathbf{x} + k, \mathrm{misrate}) = \operatorname{CenterBounds}(\mathbf{x}, \mathrm{misrate}) + k
  • Scale equivariance CenterBounds(kx,misrate)=kCenterBounds(x,misrate)\operatorname{CenterBounds}(k \cdot \mathbf{x}, \mathrm{misrate}) = k \cdot \operatorname{CenterBounds}(\mathbf{x}, \mathrm{misrate})

Example

  • CenterBounds([1..10], 0.01) = [2.5, 8.5] where Center = 5.5
  • Bounds fail to cover true center with probability misrate\approx \mathrm{misrate}

CenterBounds\operatorname{CenterBounds} provides not just the estimated center but also the uncertainty of that estimate. The function returns an interval of plausible center values given the data. Set misrate\mathrm{misrate} to control how often the bounds might fail to contain the true center: use 10310^{-3} for everyday analysis or 10610^{-6} for critical decisions where errors are costly. These bounds require weak symmetry but no specific distributional form. If the bounds exclude some reference value, that suggests the true center differs reliably from that value.

Spread

Spread(x)=Median1i<jnxixj\operatorname{Spread}(\mathbf{x}) = \underset{1 \leq i < j \leq n}{\operatorname{Median}} \lvert x_i - x_j \rvert

Robust measure of dispersion (variability, scatter).

  • Also known as — Shamos scale estimator
  • Asymptotic — median of the absolute difference between two random measurements from XX
  • ComplexityO(n2logn)O(n^2 \log n) naive, O(nlogn)O(n \log n) fast (see Fast Spread)
  • Domain — any real numbers
  • Assumptionssparity(x)
  • Unit — same as measurements

Properties

  • Shift invariance Spread(x+k)=Spread(x)\operatorname{Spread}(\mathbf{x} + k) = \operatorname{Spread}(\mathbf{x})
  • Scale equivariance Spread(kx)=kSpread(x)\operatorname{Spread}(k \cdot \mathbf{x}) = \lvert k \rvert \cdot \operatorname{Spread}(\mathbf{x})
  • Non-negativity Spread(x)0\operatorname{Spread}(\mathbf{x}) \geq 0

Example

  • Spread([0, 2, 4, 6, 8]) = 4
  • Spread(x + 10) = 4 Spread(2x) = 8

References

  • Shamos 1976

Spread\operatorname{Spread} measures how much measurements vary from each other. It serves the same purpose as standard deviation but does not explode with outliers or heavy-tailed data. The result comes in the same units as the measurements, so if Spread\operatorname{Spread} is 5 milliseconds, that indicates how much values typically differ. Like Center\operatorname{Center}, it tolerates up to 29% corrupted data. When comparing variability across datasets, Spread\operatorname{Spread} gives a reliable answer even when standard deviation would be misleading or infinite.

RelSpread

RelSpread(x)=Spread(x)Center(x)\operatorname{RelSpread}(\mathbf{x}) = \frac{\operatorname{Spread}(\mathbf{x})}{\lvert \operatorname{Center}(\mathbf{x}) \rvert}

Relative dispersion normalized by location.

  • Also known as — robust coefficient of variation
  • DomainCenter(x)0\operatorname{Center}(\mathbf{x}) \neq 0
  • Assumptionspositivity(x)
  • Unit — dimensionless

Properties

  • Scale invariance RelSpread(kx)=RelSpread(x)\operatorname{RelSpread}(k \cdot \mathbf{x}) = \operatorname{RelSpread}(\mathbf{x})
  • Non-negativity RelSpread(x)0\operatorname{RelSpread}(\mathbf{x}) \geq 0

Example

  • RelSpread([1, 3, 5, 7, 9]) = 0.8
  • RelSpread(5x) = 0.8

RelSpread\operatorname{RelSpread} compares how noisy different datasets are, even if they have completely different scales or units. A dataset centered around 100 with spread of 10 has the same relative variability as one centered around 1000 with spread of 100. Both show 10% relative variation, and RelSpread\operatorname{RelSpread} captures exactly this. This makes it useful for comparing measurement quality across different experiments, instruments, or physical quantities where absolute numbers are not directly comparable.

Two-Sample Estimators

Shift

Shift(x,y)=Median1in  1jm(xiyj)\operatorname{Shift}(\mathbf{x}, \mathbf{y}) = \underset{1 \leq i \leq n\; 1 \leq j \leq m}{\operatorname{Median}} (x_i - y_j)

Robust measure of location difference between two samples.

  • Also known as — Hodges-Lehmann estimator for two samples
  • Asymptotic — median of the difference between random measurements from XX and YY
  • ComplexityO(mnlog(mn))O(m n \log(m n)) naive, O((m+n)logL)O((m+n) \log L) fast (see Fast Shift)
  • Domain — any real numbers
  • Unit — same as measurements

Properties

  • Self-difference Shift(x,x)=0\operatorname{Shift}(\mathbf{x}, \mathbf{x}) = 0
  • Shift equivariance Shift(x+kx,y+ky)=Shift(x,y)+kxky\operatorname{Shift}(\mathbf{x} + k_x, \mathbf{y} + k_y) = \operatorname{Shift}(\mathbf{x}, \mathbf{y}) + k_x - k_y
  • Scale equivariance Shift(kx,ky)=kShift(x,y)\operatorname{Shift}(k \cdot \mathbf{x}, k \cdot \mathbf{y}) = k \cdot \operatorname{Shift}(\mathbf{x}, \mathbf{y})
  • Antisymmetry Shift(x,y)=Shift(y,x)\operatorname{Shift}(\mathbf{x}, \mathbf{y}) = -\operatorname{Shift}(\mathbf{y}, \mathbf{x})

Example

  • Shift([0, 2, 4, 6, 8], [10, 12, 14, 16, 18]) = -10
  • Shift(y, x) = -Shift(x, y)

References

  • Hodges & Lehmann 1963
  • Sidak et al. 1999

Shift\operatorname{Shift} measures how much one group differs from another. When comparing response times between version A and version B, Shift\operatorname{Shift} tells by how many milliseconds A is faster or slower than B. A negative result means the first group tends to be lower; positive means it tends to be higher. Unlike comparing means, Shift\operatorname{Shift} handles outliers gracefully and works well with skewed data. The result comes in the same units as your measurements, making it easy to interpret.

ShiftBounds

ShiftBounds(x,y,misrate)=[z(kleft),z(kright)]\operatorname{ShiftBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate}) = [z_{(k_{\text{left}})}, z_{(k_{\text{right}})}]

where z=xiyj\mathbf{z} = { x_i - y_j } (sorted), kleft=PairwiseMargin/2+1k_{\text{left}} = \lfloor \operatorname{PairwiseMargin} / 2 \rfloor + 1, kright=nmPairwiseMargin/2k_{\text{right}} = n m - \lfloor \operatorname{PairwiseMargin} / 2 \rfloor

Robust bounds on Shift(x,y)\operatorname{Shift}(\mathbf{x}, \mathbf{y}) with specified coverage.

  • Also known as — distribution-free confidence interval for Hodges-Lehmann
  • Interpretationmisrate\mathrm{misrate} is probability that true shift falls outside bounds
  • Domain — any real numbers, misrate2(n+mn)\mathrm{misrate} \geq \frac{2}{\binom{n+m}{n}}
  • Unit — same as measurements
  • Note — assumes weak continuity (ties from measurement resolution are tolerated but may yield conservative bounds)

Properties

  • Shift invariance ShiftBounds(x+k,y+k,misrate)=ShiftBounds(x,y,misrate)\operatorname{ShiftBounds}(\mathbf{x} + k, \mathbf{y} + k, \mathrm{misrate}) = \operatorname{ShiftBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate})
  • Scale equivariance ShiftBounds(kx,ky,misrate)=kShiftBounds(x,y,misrate)\operatorname{ShiftBounds}(k \cdot \mathbf{x}, k \cdot \mathbf{y}, \mathrm{misrate}) = k \cdot \operatorname{ShiftBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate})

Example

  • ShiftBounds([1..30], [21..50], 1e-4) = [-30, -10] where Shift = -20
  • Bounds fail to cover true shift with probability misrate\approx \mathrm{misrate}

ShiftBounds\operatorname{ShiftBounds} provides not just the estimated shift but also the uncertainty of that estimate. The function returns an interval of plausible shift values given the data. Set misrate\mathrm{misrate} to control how often the bounds might fail to contain the true shift: use 10310^{-3} for everyday analysis or 10610^{-6} for critical decisions where errors are costly. These bounds require no assumptions about your data distribution, so they remain valid for any continuous measurements. If the bounds exclude zero, that suggests a reliable difference between the two groups.

Ratio

Ratio(x,y)=exp(Shift(logx,logy))\operatorname{Ratio}(\mathbf{x}, \mathbf{y}) = \exp(\operatorname{Shift}(\log \mathbf{x}, \log \mathbf{y}))

Robust measure of scale ratio between two samples — the multiplicative dual of Shift\operatorname{Shift}.

  • Asymptotic — geometric median of pairwise ratios xiyj\frac{x_i}{y_j} (via log-space aggregation)
  • Domainxi>0x_i > 0, yj>0y_j > 0
  • Assumptionspositivity(x), positivity(y)
  • Unit — dimensionless
  • Complexity — naive O(mnlog(mn))O(m n \log(m n)), fast O((m+n)logL)O((m + n) \log L) via FastRatio

Properties

  • Self-ratio Ratio(x,x)=1\operatorname{Ratio}(\mathbf{x}, \mathbf{x}) = 1
  • Scale equivariance Ratio(kxx,kyy)=(kxky)Ratio(x,y)\operatorname{Ratio}(k_x \cdot \mathbf{x}, k_y \cdot \mathbf{y}) = (\frac{k_x}{k_y}) \cdot \operatorname{Ratio}(\mathbf{x}, \mathbf{y})
  • Multiplicative antisymmetry Ratio(x,y)=1/Ratio(y,x)\operatorname{Ratio}(\mathbf{x}, \mathbf{y}) = 1 / \operatorname{Ratio}(\mathbf{y}, \mathbf{x})

Example

  • Ratio([1, 2, 4, 8, 16], [2, 4, 8, 16, 32]) = 0.5
  • Ratio(x, x) = 1 Ratio(2x, 5y) = 0.4 · Ratio(x, y)

Relationship to Shift

Ratio\operatorname{Ratio} is the multiplicative analog of Shift\operatorname{Shift}. While Shift\operatorname{Shift} computes the median of pairwise differences xiyjx_i - y_j, Ratio\operatorname{Ratio} computes the median of pairwise ratios xiyj\frac{x_i}{y_j} via log-transformation. This relationship is expressed formally as:

Ratio(x,y)=exp(Shift(logx,logy))\operatorname{Ratio}(\mathbf{x}, \mathbf{y}) = \exp(\operatorname{Shift}(\log \mathbf{x}, \log \mathbf{y}))

The log-transformation converts multiplicative relationships to additive ones, allowing the fast Shift\operatorname{Shift} algorithm to compute the result efficiently. The exp-transformation converts back to the ratio scale.

Ratio\operatorname{Ratio} is appropriate for multiplicative relationships rather than additive differences. If one system is twice as fast or prices are 30% lower, the underlying thinking is in ratios. A result of 0.5 means the first group is typically half the size of the second; 2.0 means twice as large. This estimator is appropriate for quantities like prices, response times, and concentrations where relative comparisons make more sense than absolute ones. Both samples must contain strictly positive values.

RatioBounds

RatioBounds(x,y,misrate)=exp(ShiftBounds(logx,logy,misrate))\operatorname{RatioBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate}) = \exp(\operatorname{ShiftBounds}(\log \mathbf{x}, \log \mathbf{y}, \mathrm{misrate}))

Robust bounds on Ratio(x,y)\operatorname{Ratio}(\mathbf{x}, \mathbf{y}) with specified coverage — the multiplicative dual of ShiftBounds\operatorname{ShiftBounds}.

  • Also known as — distribution-free confidence interval for Hodges-Lehmann ratio
  • Interpretationmisrate\mathrm{misrate} is probability that true ratio falls outside bounds
  • Domainxi>0x_i > 0, yj>0y_j > 0, misrate2(n+mn)\mathrm{misrate} \geq \frac{2}{\binom{n+m}{n}}
  • Assumptionspositivity(x), positivity(y)
  • Unit — dimensionless
  • Note — assumes weak continuity (ties from measurement resolution are tolerated but may yield conservative bounds)

Properties

  • Scale invariance RatioBounds(kx,ky,misrate)=RatioBounds(x,y,misrate)\operatorname{RatioBounds}(k \cdot \mathbf{x}, k \cdot \mathbf{y}, \mathrm{misrate}) = \operatorname{RatioBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate})
  • Scale equivariance RatioBounds(kxx,kyy,misrate)=(kxky)RatioBounds(x,y,misrate)\operatorname{RatioBounds}(k_x \cdot \mathbf{x}, k_y \cdot \mathbf{y}, \mathrm{misrate}) = (\frac{k_x}{k_y}) \cdot \operatorname{RatioBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate})
  • Multiplicative antisymmetry RatioBounds(x,y,misrate)=1/RatioBounds(y,x,misrate)\operatorname{RatioBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate}) = 1 / \operatorname{RatioBounds}(\mathbf{y}, \mathbf{x}, \mathrm{misrate}) (bounds reversed)

Example

  • RatioBounds([1..30], [10..40], 1e-4) where Ratio ≈ 0.5 yields bounds containing 0.50.5
  • Bounds fail to cover true ratio with probability misrate\approx \mathrm{misrate}

Relationship to ShiftBounds

RatioBounds\operatorname{RatioBounds} is computed via log-transformation:

RatioBounds(x,y,misrate)=exp(ShiftBounds(logx,logy,misrate))\operatorname{RatioBounds}(\mathbf{x}, \mathbf{y}, \mathrm{misrate}) = \exp(\operatorname{ShiftBounds}(\log \mathbf{x}, \log \mathbf{y}, \mathrm{misrate}))

This means if ShiftBounds\operatorname{ShiftBounds} returns [a,b][a, b] for the log-transformed samples, RatioBounds\operatorname{RatioBounds} returns [ea,eb][e^a, e^b].

RatioBounds\operatorname{RatioBounds} provides not just the estimated ratio but also the uncertainty of that estimate. The function returns an interval of plausible ratio values given the data. Set misrate\mathrm{misrate} to control how often the bounds might fail to contain the true ratio: use 10310^{-3} for everyday analysis or 10610^{-6} for critical decisions where errors are costly. These bounds require no assumptions about your data distribution, so they remain valid for any continuous positive measurements. If the bounds exclude 11, that suggests a reliable multiplicative difference between the two groups.

AvgSpread

AvgSpread(x,y)=nSpread(x)+mSpread(y)n+m\operatorname{AvgSpread}(\mathbf{x}, \mathbf{y}) = \frac{n \cdot \operatorname{Spread}(\mathbf{x}) + m \cdot \operatorname{Spread}(\mathbf{y})}{n + m}

Weighted average of dispersions (pooled scale).

  • Also known as — robust pooled standard deviation
  • Domain — any real numbers
  • Assumptionssparity(x), sparity(y)
  • Unit — same as measurements
  • CaveatAvgSpread(x,y)Spread(xuniony)\operatorname{AvgSpread}(\mathbf{x}, \mathbf{y}) \neq \operatorname{Spread}(\mathbf{x} union \mathbf{y}) (pooled scale, not concatenated spread)

Properties

  • Self-average AvgSpread(x,x)=Spread(x)\operatorname{AvgSpread}(\mathbf{x}, \mathbf{x}) = \operatorname{Spread}(\mathbf{x})
  • Symmetry AvgSpread(x,y)=AvgSpread(y,x)\operatorname{AvgSpread}(\mathbf{x}, \mathbf{y}) = \operatorname{AvgSpread}(\mathbf{y}, \mathbf{x})
  • Scale equivariance AvgSpread(kx,ky)=kAvgSpread(x,y)\operatorname{AvgSpread}(k \cdot \mathbf{x}, k \cdot \mathbf{y}) = \lvert k \rvert \cdot \operatorname{AvgSpread}(\mathbf{x}, \mathbf{y})
  • Mixed scaling AvgSpread(k1x,k2x)=k1+k22Spread(x)\operatorname{AvgSpread}(k_1 \cdot \mathbf{x}, k_2 \cdot \mathbf{x}) = \frac{\lvert k_1 \rvert + \lvert k_2 \rvert}{2} \cdot \operatorname{Spread}(\mathbf{x})

Example

  • AvgSpread(x, y) = 5 where Spread(x) = 6, Spread(y) = 4, n = m
  • AvgSpread(x, y) = AvgSpread(y, x)

AvgSpread\operatorname{AvgSpread} provides a single number representing the typical variability across two groups. It combines the spread of both samples, giving more weight to larger samples since they provide more reliable estimates. This pooled spread serves as a common reference scale, essential for expressing a difference in relative terms. Disparity\operatorname{Disparity} uses AvgSpread\operatorname{AvgSpread} internally to normalize the shift into a scale-free effect size.

Disparity

Disparity(x,y)=Shift(x,y)/AvgSpread(x,y)\operatorname{Disparity}(\mathbf{x}, \mathbf{y}) = \operatorname{Shift}(\mathbf{x}, \mathbf{y}) / \operatorname{AvgSpread}(\mathbf{x}, \mathbf{y})

Robust effect size (shift normalized by pooled dispersion).

  • Also known as — robust Cohens d (Cohen 1988; estimates differ due to robust construction)
  • DomainAvgSpread(x,y)>0\operatorname{AvgSpread}(\mathbf{x}, \mathbf{y}) > 0
  • Assumptionssparity(x), sparity(y)
  • Unit — spread units

Properties

  • Location invariance Disparity(x+k,y+k)=Disparity(x,y)\operatorname{Disparity}(\mathbf{x} + k, \mathbf{y} + k) = \operatorname{Disparity}(\mathbf{x}, \mathbf{y})
  • Scale invariance Disparity(kx,ky)=sign(k)Disparity(x,y)\operatorname{Disparity}(k \cdot \mathbf{x}, k \cdot \mathbf{y}) = \operatorname{sign}(k) \cdot \operatorname{Disparity}(\mathbf{x}, \mathbf{y})
  • Antisymmetry Disparity(x,y)=Disparity(y,x)\operatorname{Disparity}(\mathbf{x}, \mathbf{y}) = -\operatorname{Disparity}(\mathbf{y}, \mathbf{x})

Example

  • Disparity(x, y) = 0.4 where Shift = 2, AvgSpread = 5
  • Disparity(x + c, y + c) = Disparity(x, y) Disparity(kx, ky) = Disparity(x, y)

Disparity\operatorname{Disparity} expresses a difference between groups in a way that does not depend on the original measurement units. A disparity of 0.5 means the groups differ by half a spread unit; 1.0 means one full spread unit. Being dimensionless allows comparison of effect sizes across different studies, metrics, or measurement scales. What counts as a large or small disparity depends entirely on the domain and what matters practically in a given application. Do not rely on universal thresholds; interpret the number in context.

Randomization

Rng

Rng(s)\operatorname{Rng}(s)

Deterministic pseudorandom number generator from seed ss.

  • Algorithm — xoshiro256++ with SplitMix64 seeding (see Pseudorandom Number Generation)
  • Seed types — integer seed or string seed (hashed via FNV-1a)
  • Determinism — identical sequences across all supported languages
  • Period225612^{256} - 1

Notation

  • XX, YY random variables (generators of real measurements)

Properties

  • Reproducibility Rng(s)\operatorname{Rng}(s) produces identical sequence for same ss
  • Independence different seeds produce uncorrelated sequences

Example

  • Rng("demo-uniform") — string seed for reproducible demos
  • Rng("experiment-1") — string seed for named experiments

Rng\operatorname{Rng} provides reproducible random numbers. The same seed produces exactly the same sequence of values, identical across Python, TypeScript, R, C#, Kotlin, Rust, and Go. Passing a descriptive string like "experiment-1" makes code self-documenting. Each draw advances the generators internal state, so independent random streams require separate generators with different seeds.

Sample

r.Sample(x,k)r.\operatorname{Sample}(\mathbf{x}, k)

Select kk elements from sample x\mathbf{x} without replacement using generator rr.

  • Algorithm — selection sampling (Fan, Muller, Rezucha 1962), see Pseudorandom Number Generation
  • ComplexityO(n)O(n) time, single pass
  • Output — preserves original order of selected elements
  • Domain1kn1 \leq k \leq n

Notation

  • x=(x1,,xn)\mathbf{x} = (x_1, \ldots, x_n), y=(y1,,ym)\mathbf{y} = (y_1, \ldots, y_m) samples (n,m1n, m \geq 1)
  • xix_i, yjy_j individual measurements

Properties

  • Simple random sample each kk-subset has equal probability
  • Order preservation selected elements appear in order of first occurrence
  • Determinism same generator state produces same selection

Example

  • Sample([1, 2, 3, 4, 5], 3, Rng("demo-sample")) — select 3 elements
  • Sample(x, n, r) = x — selecting all elements returns original order

Sample\operatorname{Sample} picks a random subset of data without replacement. Common uses include bootstrap resampling, creating cross-validation splits, or reducing a large dataset to a manageable size. Every possible subset of size kk has equal probability of being selected, and the selected elements keep their original order. To make your subsampling reproducible, combine it with a seeded generator: Sample(data, 100, Rng("training-set")) will always select the same 100 elements.

Shuffle

r.Shuffle(x)r.\operatorname{Shuffle}(\mathbf{x})

Uniformly random permutation of sample x\mathbf{x} using generator rr.

  • Algorithm — Fisher-Yates (Knuth shuffle), see Pseudorandom Number Generation
  • ComplexityO(n)O(n) time, O(1)O(1) additional space
  • Output — new array (does not modify input)

Properties

  • Uniformity each of n!n! permutations has equal probability
  • Determinism same generator state produces same permutation

Example

  • Shuffle([1, 2, 3, 4, 5], Rng("demo-shuffle")) — shuffled copy
  • Shuffle(x, r) preserves multiset (same elements, different order)

Shuffle\operatorname{Shuffle} produces a random reordering of data. This is essential for permutation tests and useful for eliminating any bias from the original ordering. Every possible arrangement has exactly equal probability, which is required for valid statistical inference. The function returns a new shuffled array and leaves the original data unchanged. For reproducible results, pass a seeded generator: Shuffle(data, Rng("experiment-1")) will always produce the same permutation.

Resample

r.Resample(x,k)r.\operatorname{Resample}(\mathbf{x}, k)

Select kk elements from sample x\mathbf{x} with replacement using generator rr.

  • Algorithm — uniform sampling with replacement
  • ComplexityO(k)O(k) time
  • Output — new array with kk elements (may contain duplicates)
  • Domaink0k \geq 0, sample size n1n \geq 1

Notation

  • x=(x1,,xn)\mathbf{x} = (x_1, \ldots, x_n) sample (n1n \geq 1)
  • xix_i individual measurements

Properties

  • Independence each selection is independent with equal probability 1n\frac{1}{n}
  • Duplicates same element may appear multiple times in output
  • Determinism same generator state produces same selection

Example

  • Resample([1, 2, 3, 4, 5], 3, Rng("demo-resample")) — select 3 with replacement
  • Resample(x, n, r) — bootstrap sample of same size as original

Resample\operatorname{Resample} picks elements with replacement, allowing the same element to be selected multiple times. This is essential for bootstrap methods where we simulate new samples from the observed data. Unlike Sample\operatorname{Sample} (without replacement), Resample\operatorname{Resample} can produce outputs larger than the input and will typically contain duplicate values. For reproducible bootstrap, combine with a seeded generator: Resample(data, n, Rng("bootstrap-1")).

Auxiliary

Median

Median(x)={x(n+12)ifnis odd(x(n2)+x(n2+1))/2ifnis even\begin{aligned} \operatorname{Median}(\mathbf{x}) = \begin{cases} x_{(\frac{n+1}{2})} & \text{if} n \text{is odd} \\ (x_{(\frac{n}{2})} + x_{(\frac{n}{2}+1)}) / 2 & \text{if} n \text{is even} \end{cases} \end{aligned}

The value splitting a sorted sample into two equal parts.

  • Also known as — 50th percentile, second quartile (Q2)
  • Asymptotic — value where P(XMedian)=0.5P(X \leq \operatorname{Median}) = 0.5
  • ComplexityO(n)O(n) with selection, O(nlogn)O(n \log n) with sorting
  • Domain — any real numbers
  • Unit — same as measurements

Notation

  • x(1),,x(n)x_{(1)}, \ldots, x_{(n)} order statistics (sorted sample)

Properties

  • Shift equivariance Median(x+k)=Median(x)+k\operatorname{Median}(\mathbf{x} + k) = \operatorname{Median}(\mathbf{x}) + k
  • Scale equivariance Median(kx)=kMedian(x)\operatorname{Median}(k \cdot \mathbf{x}) = k \cdot \operatorname{Median}(\mathbf{x})

Example

  • Median([1, 2, 3, 4, 5]) = 3
  • Median([1, 2, 3, 4]) = 2.5

Median\operatorname{Median} provides maximum protection against outliers and corrupted data. It achieves a 50% breakdown point, meaning that up to half of the data can be arbitrarily bad before the estimate becomes meaningless. However, this extreme robustness comes at a cost: the median is less precise than Center\operatorname{Center} when data is clean. For most practical applications, Center\operatorname{Center} offers a better tradeoff (29% breakdown with 95% efficiency). Reserve Median\operatorname{Median} for situations with suspected contamination levels above 29% or when the strongest possible robustness guarantee is needed.

PairwiseMargin

PairwiseMargin(n,m,misrate)\operatorname{PairwiseMargin}(n, m, \mathrm{misrate})

Exclusion count for dominance-based bounds.

  • Purpose — determines extreme pairwise differences to exclude when constructing bounds
  • Based on — distribution of Dominance(x,y)=i=1nj=1m1(xi>yj)\operatorname{Dominance}(\mathbf{x}, \mathbf{y}) = \sum_{i=1}^n \sum_{j=1}^m \mathbb{1}(x_i > y_j) under random sampling
  • Returns — total margin split evenly between lower and upper tails
  • Used byShiftBounds\operatorname{ShiftBounds} to select appropriate order statistics
  • Complexity — exact for small samples, approximated for large (see Fast PairwiseMargin)
  • Domainn,m1n, m \geq 1, misrate>2(n+mn)\mathrm{misrate} > \frac{2}{\binom{n+m}{n}} (minimum achievable)
  • Unit — count
  • Note — assumes weak continuity (ties from measurement resolution are tolerated)

Properties

  • Symmetry PairwiseMargin(n,m,misrate)=PairwiseMargin(m,n,misrate)\operatorname{PairwiseMargin}(n, m, \mathrm{misrate}) = \operatorname{PairwiseMargin}(m, n, \mathrm{misrate})
  • Bounds 0PairwiseMargin(n,m,misrate)nm0 \leq \operatorname{PairwiseMargin}(n, m, \mathrm{misrate}) \leq n m

Example

  • PairwiseMargin(30, 30, 1e-6) = 276
  • PairwiseMargin(30, 30, 1e-4) = 390
  • PairwiseMargin(30, 30, 1e-3) = 464

This is a supporting function that ShiftBounds\operatorname{ShiftBounds} uses internally, so most users do not need to call it directly. It calculates how many extreme pairwise differences should be excluded when constructing bounds, based on sample sizes and the desired error rate. A lower misrate (higher confidence) results in a smaller margin, which produces wider bounds. The function automatically chooses between exact computation for small samples and a fast approximation for large samples.

SignedRankMargin

SignedRankMargin(n,misrate)\operatorname{SignedRankMargin}(n, \mathrm{misrate})

Exclusion count for one-sample signed-rank based bounds.

  • Purpose — determines extreme pairwise averages to exclude when constructing bounds
  • Based on — Wilcoxon signed-rank distribution under weak symmetry
  • Returns — total margin split evenly between lower and upper tails
  • Used byCenterBounds\operatorname{CenterBounds} to select appropriate order statistics
  • Complexity — exact for n63n \leq 63, approximated for larger (see Fast SignedRankMargin)
  • Domainn2n \geq 2, misrate21n\mathrm{misrate} \geq 2^{1-n}
  • Unit — count
  • Note — assumes weak symmetry and weak continuity

Properties

  • Bounds 0SignedRankMargin(n,misrate)n(n+1)20 \leq \operatorname{SignedRankMargin}(n, \mathrm{misrate}) \leq \frac{n(n+1)}{2}
  • Monotonicity lower misrate \rightarrow smaller margin \rightarrow wider bounds

Example

  • SignedRankMargin(10, 0.05) = 18
  • SignedRankMargin(30, 1e-4) = 112
  • SignedRankMargin(100, 1e-6) = 706

This is a supporting function that CenterBounds\operatorname{CenterBounds} uses internally, so most users do not need to call it directly. It calculates how many extreme pairwise averages should be excluded when constructing bounds, based on sample size and the desired error rate. A lower misrate (higher confidence) results in a smaller margin, which produces wider bounds. The function automatically chooses between exact computation for small samples and a fast approximation for large samples.