$Q(1) = ae + af + bg$

Set Semantics

$Q(1) = ae + af + bg$

$a = b = e = f = g = T \rightarrow$ $T$

Bag Semantics

$Q(1) = ae + af + bg$

$\{a = 3; b = 1; e = 1; f = 1; g = 3 \} \rightarrow$ $12$

How Provenance

$Q(1) = ae + af + bg$

$\{a = t_1; b = t_2; e = t_5; f = t_6; g = t_7 \}\rightarrow$
$\{\{t_1, t_5\}, \{t_1, t_6\}, \{t_2, t_7\}\}$

Access Control

With $\left< [0, 5], \min, \max, 0, 5 \right>$
(0 = minimum security, 5 = top secret)

$Q(1) = ae + af + bg$

$\{a = 5; b = 1; e = 2; f = 3; g = 2 \}\rightarrow$$2$

Representing Provenance

Relational/Annotation

Extra columns in each table store provenance information.

External

Provenance stored in external data-structures or tables.

Generating Provenance

Query Rewriting

$Q \rightarrow Q'$, where $Q'$ computes the provenance.

Operator Instrumentation

Specialized operators construct provenance.

Generating Provenance

Eager

Compute provenance as part of query results.

Lazy

Store query: Compute provenance as-needed.

Extra columns store annotations

      PROVENANCE OF SELECT A FROM R NATURAL JOIN S
    

We'll be using bag-relational algebra.

$R \rightarrow \pi_{R.*, \phi \leftarrow \texttt{ROWID}}(R)$

$\pi_A(R) \rightarrow ?$

$\pi_{A, \phi}(R)$

$\sigma_c(R) \rightarrow ?$

$\sigma_c(R)$

$R \times S \rightarrow ?$

$\rho_{\phi \rightarrow \phi_1} (R) \times \rho_{\phi \rightarrow \phi_2} (S)$

$\pi_A(R) \rightarrow \pi_{A, \phi_1, \ldots, \phi_N}(R)$

$R \times S \rightarrow ?$

$R$ (resp., $S$) has $M$ (resp., $N$) annotation columns.

$R \times \rho_{\phi_{1} \rightarrow \phi_{M+1}, \ldots, \phi_{N} \rightarrow \phi_{M+N}} (S)$

(Rename $S$'s columns, appending to $R$'s)

$R \uplus S \rightarrow ?$

If $R$ has $M$ annotation columns
and $S$ has $N > M$ columns:

$\pi_{R.*, \phi_{M+1} \leftarrow \texttt{NULL}, \ldots, \phi_{N} \leftarrow \texttt{NULL}} (R) \uplus S$

(Pad the narrower relation with $\texttt{NULL}$s)

Idea: Construct provenance by combining multiple rows.

$\delta\big(\pi_{A}(Q_1)\big)$?

$\delta R \rightarrow ?$

$R$

$_A\gamma_{SUM(B)}(R) \rightarrow ?$

$\big(_A\gamma_{SUM(B)}(R)\big) \bowtie_{A} \big( \pi_{A, \phi_1, \ldots, \phi_{M}} R\big)$

Plug in "F" for $R_1$ (i.e., simulate removing $R_1$ from the input)

Plug in "F" for $R_1$ (i.e., simulate removing $R_1$ from the input)

Plug in "F" for $R_1$ (i.e., simulate removing $R_1$ from the input)

Not quite a provenance polynomial...

...but still correct for lineage queries

(and there are other solutions)

... but it works better for lazy evaluation

                  SELECT DISTINCT A, COUNT 
                  FROM (PROVENANCE OF Q_1)
                  WHERE phi_1 = 'R_1'
    

Observation 1: predicate on $\phi$ can be pushed down.

Observation 2: DISTINCT would be a no-op (group-by attributes are always a key) if not for provenance additions.

This query can be made to run very fast!

The power of the relational representation is that it can be queried and optimized using an existing database. Everything is just a query.

Goal 1: Find the input tuples that produced a given output tuple (backward query).

Goal 2: Find the output tuples that result from a given input tuple (forward query).

Combine for interactive visualizations

Basic Idea: Each operator emits two structures:
$\textbf{id}_{in} \rightarrow \{ \textbf{id}_{out} \}$ and $\textbf{id}_{out} \rightarrow \{ \textbf{id}_{in} \}$

A "Forward index" and a "Backward index"

Again, we'll be using bag-relational algebra.

Optimizations

Multimap

Avoid DB overheads by storing forward/backward indexes in special in-mem datastructure

Simple Array where possible

Further specialize layout for 1-1 provenance operators.

Size selection

Avoid (expensive) structure reallocation costs by using DB statistics for join selectivity, number of rows

Deferred Provenance

2-stage execution: First results (and fixed-size metadata), then provenance

Recovering Provenance

$_A\gamma_{COUNT}\big((R \bowtie S) \bowtie T\big) \rightarrow \\ \textbf{fwd}_{\bowtie_1}, \textbf{fwd}_{\bowtie_2}, \textbf{fwd}_{\gamma}, \textbf{back}_{\bowtie_1}, \textbf{back}_{\bowtie_2}, \textbf{back}_{\gamma}$

What tuples went in to $\left< A: 1, COUNT: 3 \right>$
(output tuple 1)?

$$\pi_{\textbf{back}_{\bowtie_2}.in}\big(\big( (\textbf{back}_{\gamma} \bowtie_{\textbf{back}_{\gamma}.in = \textbf{back}_{\bowtie_2}.out} \textbf{back}_{\bowtie_2})$$ $$\bowtie_{\textbf{back}_{\bowtie_2}.in = \textbf{back}_{\bowtie_1}.out} \textbf{back}_{\bowtie_1}\big)\big)$$