Sketching: Hash function tricks used to estimate useful statistical properties.

Flajolet-Martin Sketches (HyperLogLog)

Estimating Count-Distinct

Count Sketches

Estimating Count-GroupBy

Count-Min Sketches

Estimating Count-GroupBy-TopK

A brief digression

The Coin Flip Game

Tails (🐕)

Get a point and flip again.

Heads (👽)

Game over.

If I told you that in a series of games, my best score was $N$, you might expect that I played $2^{N+1}$ games.

To do that, I only need to track my top score!

Idea: Simulate coin flips with a hash function

... take the index of the lowest-order nonzero bit

Estimate: $2^{3+1} = 16$

Duplicates can't raise the top score!

Problem: Noisy estimate!

Idea 1: Instead of your top score, track the lowest score you have not gotten yet ($R$).

Estimate: $\frac{2^R}{\phi} = \frac{2^{2}}{0.77351} \approx 5.2$

Idea 2: Compute several estimates in parallel and average estimates.

Flajolet-Martin Sketches

($\approx$ HyperLogLog)

1. For each record...
   1. Hash each record
   2. Find the index of the lowest-order non-zero bit
   3. Add the index of the bit to a set
2. Find $R$, the lowest index not in the set
3. Estimate Count-Distinct as $\frac{2^R}{\phi}$ ($\phi \approx 0.77351$)
4. Repeat (in parallel) as needed

Idea: Keep only one counter!

No... seriously

$$Total \approx \texttt{COUNT_OF}(O_i) \cdot \delta(O_i) + 0$$

Running total was $-1$

Not... so... great

Problem 1: All of the objects use the same counter (no way to differentiate an estimate for $O_1$ from $O_2$).

Problem 2: The estimate is really noisy

Idea 1: Multiple Buckets ($h(x)$ picks a bucket)

Idea 2: Multiple Trials ($h \rightarrow h_1, h_2, \ldots$; $\delta \rightarrow \delta_1, \delta_2, \ldots$)

Objects Seen: $$

Objects Seen: $O_2$

Objects Seen: $O_2,O_1$

Objects Seen: $O_2,O_1,O_4$

Objects Seen: $O_2,O_1,O_4,O_1$

Objects Seen: $O_2,O_1,O_4,O_1,O_2$

In practice, use Median and not Mode to combine trials

Idea: Give up. Drop $\delta$.

Count-Min Sketch