stats
Descriptive statistics over a list of numbers: mean, median, variance, standard deviation and percentiles.
import stats
The stats package answers the first questions you ask of a list of numbers: how many, the total, the typical value, the spread and the extremes. Use it for reports, monitoring (latency percentiles), grading and checking a simulation. It covers descriptive statistics only; there is no mode, correlation, regression or histogram. For scalar maths use math, for fitting models ml, and to generate the numbers in the first place rand.
Every function takes a Vector<f64> and never changes it. Whole numbers have to be converted with as f64 on the way in. Median and Percentile sort a private copy on each call, so asking for p50, p90 and p99 sorts three times; that is fine for thousands of values and worth knowing for millions.
Variance and standard deviation come in two forms. Variance and Stdev treat the data as a sample of a larger group and divide by n−1; PopVariance and PopStdev treat it as the whole group and divide by n. The sample form is the usual choice.
What people get wrong:
- Trusting the result of an empty list. Nothing fails:
Mean,Median,Min,MaxandPercentilereturn 0, which looks like real data. Checklen()first. - Reporting the mean of skewed data. One slow request or one very rich customer moves the mean a long way; the median and percentiles do not move.
- Averaging growth rates with
Mean. Factors that multiply needGeomean.
A summary of a class
The usual job is a short report on a list of numbers: how many, the middle, the spread and the extremes. Put the values in a Vector<f64> once and call the functions you need; none of them changes the vector. The standard deviation also lets you flag values that are far from the rest: a value more than one standard deviation from the mean is unusual in a small group.
import stats
import conv
func main:
names := ["Sara", "Reza", "Mina", "Ali", "Neda", "Omid", "Leila"]
mut scores := Vector {} as Vector<f64>
defer scores.Free()
each s in [14.5, 17.0, 19.5, 11.0, 16.0, 8.5, 18.0]:
scores.Push(s)
end
mean := stats.Mean(scores)
sd := stats.Stdev(scores)
println "students:", scores.Len()
println "mean: ", conv.FormatFloatPrec(mean, 2)
println "median: ", stats.Median(scores)
println "stdev: ", conv.FormatFloatPrec(sd, 2)
println "range: ", stats.Min(scores), "to", stats.Max(scores)
println "top 25%: ", stats.Percentile(scores, 75.0), "and above"
each i, name in names:
z := (scores.Get(i) - mean) / sd
if z > 1.0:
println name, "is well above the class:", conv.FormatFloatPrec(z, 2)
else z < -1.0:
println name, "needs help:", conv.FormatFloatPrec(z, 2)
end
end
endA class or a survey is usually a sample, so Stdev is the right choice. Use PopStdev only when the list holds every member of the group you are describing, for example all the machines in a cluster.
Response times: median and percentiles
One slow request can pull the mean far away from what most users see. In this example nine requests take about 40 ms and one takes 1200 ms; the mean says 157 ms, while the median (the 50th percentile) says 41.5 ms. Monitoring therefore uses percentiles: p90 is the time 90% of requests stay under, and p99 shows the worst cases.
import stats
import conv
func show(label: str, x: f64):
println label, conv.FormatFloatPrec(x, 1), "ms"
end
func main:
mut ms := Vector {} as Vector<f64>
defer ms.Free()
each t in [42.0, 38.0, 45.0, 40.0, 39.0, 41.0, 44.0, 37.0, 43.0, 1200.0]:
ms.Push(t)
end
show("mean:", stats.Mean(ms))
show("p50: ", stats.Percentile(ms, 50.0))
show("p90: ", stats.Percentile(ms, 90.0))
show("p99: ", stats.Percentile(ms, 99.0))
show("max: ", stats.Max(ms))
budget := 100.0
if stats.Percentile(ms, 90.0) > budget:
println "alert: p90 is over the budget of", budget, "ms"
else:
println "p90 is within the budget"
end
endPercentile interpolates between the two nearest values, so with few samples the high percentiles land between a normal value and the outlier. Collect a few hundred values before you trust p99.
Growth rates and the geometric mean
When values are multiplied, as with yearly returns or growth factors, the arithmetic mean overstates the result. A gain of 50% followed by a loss of 40% is not an average gain. Geomean gives the single factor that, repeated, reaches the same end value.
import stats
import conv
func main:
mut factors := Vector {} as Vector<f64>
defer factors.Free()
each f in [1.50, 0.60, 1.20, 1.10]:
factors.Push(f)
end
mut value := 1000.0
each f in factors:
value *= f
end
println "after four years:", conv.FormatFloatPrec(value, 2)
arith := stats.Mean(factors)
geo := stats.Geomean(factors)
println "arithmetic mean factor:", conv.FormatFloatPrec(arith, 4)
println "geometric mean factor: ", conv.FormatFloatPrec(geo, 4)
mut a := 1000.0
mut g := 1000.0
repeat 4:
a *= arith
g *= geo
end
println "replayed with the arithmetic mean:", conv.FormatFloatPrec(a, 2)
println "replayed with the geometric mean: ", conv.FormatFloatPrec(g, 2)
endReading numbers from text
Data usually arrives as text, with empty cells, labels such as n/a and sometimes Persian digits. Check each cell with types.IsNumeric from types, turn Persian digits into English ones with str.ToEnglishDigits and parse with conv.ParseFloat. Count what you skipped, so that a report on bad data does not look like a report on good data.
import stats
import str
import conv
import types
func main throws:
line := "12.5, 15, ۱۸, n/a, 20, , 9.75"
mut values := Vector {} as Vector<f64>
defer values.Free()
mut skipped := 0
each cell in str.Split(line, ","):
t := str.Trim(cell)
if types.IsNumeric(t):
values.Push(try conv.ParseFloat(str.ToEnglishDigits(t)))
else:
skipped++
end
end
println "read", values.Len(), "numbers, skipped", skipped
println "sum: ", stats.Sum(values)
println "mean: ", stats.Mean(values)
println "median:", stats.Median(values)
endEmpty and tiny lists
The functions do not fail on awkward input; they return a number, and that number can mislead. An empty vector gives 0 for the mean, the median and the maximum. One value has a variance of 0. A zero or a negative value makes the geometric mean 0, and a percentile outside 0 to 100 is moved to the nearest end. Check len() before you report anything.
import stats
func main:
empty := Vector {} as Vector<f64>
println "empty:", stats.Mean(empty), stats.Median(empty), stats.Max(empty)
mut one := Vector {} as Vector<f64>
defer one.Free()
one.Push(7.0)
println "one value:", stats.Variance(one), stats.PopVariance(one)
mut rates := Vector {} as Vector<f64>
defer rates.Free()
each r in [1.2, 0.0, 1.5]:
rates.Push(r)
end
println "geomean with a zero:", stats.Geomean(rates)
println "percentile 150 is clamped:", stats.Percentile(rates, 150.0)
if empty.Len() == 0:
println "no data, nothing to report"
end
end| Question | Use |
|---|---|
| A typical value | Median, or Mean when there are no outliers |
| How spread out the values are | Stdev (sample) or PopStdev (whole population) |
| What most requests or users see | Percentile with 50, 90 or 99 |
| Average growth per period | Geomean |
| Totals and extremes | Sum, Min, Max |
Functions
Sum
func Sum(v: Vector<f64>): f64import stats
func main:
mut scores := Vector {} as Vector<f64>
defer scores.Free()
each s in [12.0, 18.0, 15.0, 19.0, 11.0]:
scores.Push(s)
end
println stats.Sum(scores), stats.Mean(scores), stats.Median(scores)
println stats.Min(scores), stats.Max(scores)
println stats.Variance(scores), stats.PopVariance(scores)
println stats.Stdev(scores), stats.PopStdev(scores)
println stats.Geomean(scores), stats.Percentile(scores, 90.0)
endMean
func Mean(v: Vector<f64>): f64The example for this item is shown with the ones above: stats-Mean.salam
Min
func Min(v: Vector<f64>): f64The example for this item is shown with the ones above: stats-Mean.salam
Max
func Max(v: Vector<f64>): f64The example for this item is shown with the ones above: stats-Mean.salam
Median
func Median(v: Vector<f64>): f64The example for this item is shown with the ones above: stats-Mean.salam
Variance
func Variance(v: Vector<f64>): f64The example for this item is shown with the ones above: stats-Mean.salam
PopVariance
func PopVariance(v: Vector<f64>): f64The example for this item is shown with the ones above: stats-Mean.salam
Stdev
func Stdev(v: Vector<f64>): f64Variance. Fewer than two values give 0.The example for this item is shown with the ones above: stats-Mean.salam
PopStdev
func PopStdev(v: Vector<f64>): f64PopVariance.The example for this item is shown with the ones above: stats-Mean.salam
Geomean
func Geomean(v: Vector<f64>): f64The example for this item is shown with the ones above: stats-Mean.salam
Percentile
func Percentile(v: Vector<f64>, p: f64): f64pth percentile, with p from 0 to 100, interpolating linearly between the two nearest sorted values (the method of Excel PERCENTILE.INC and the NumPy default). p outside that range is moved to the nearest end; 0 gives the minimum and 100 the maximum. Sorts a copy on each call.The example for this item is shown with the ones above: stats-Mean.salam