Catalyzer provides...

• Fold operators for all standard aggregation functions.
• Aggregate logical plan operator

So where's the challenge?

Test queries will be posted tonight...

Try them!

Sparkisms

• New Placeholders
• Interpreting the Aggregate operator
• Interfacing with Spark's fold operators

UnresolvedFunction

(e.g., REGEXP_EXTRACT(target, "1(3{2})7", 1))

      UnresolvedFunction(
        name = FunctionIdentifier("REGEXP_EXTRACT"), 
        arguments = Seq(
          UnresolvedAttribute(Seq("target")),
          Literal("1(3{2})7", StringType),
          Literal(1, IntegerType)
        ),
        distinct = false,
        filter = None,
        ignoreNulls = false
      )
    

Replace like UnresolvedAlias, UnresolvedAttribute

... but with what?

FunctionRegistry

      case UnresolvedFunction(name, arguments, isDistinct, filter, ignoreNulls) =>
      {
        val builder = 
          FunctionRegistry.builtin
            .lookupFunctionBuilder(name)
            .getOrElse {
              throw new RuntimeException(
                s"Unable to resolve function `${name}`"
              )
            }
        builder(arguments) // returns the replacement expression node.
      }
    

Functions

      val builder = FunctionRegistry.builtin
                      .lookupFunctionBuilder("REGEXP_EXTRACT").get
      builder(
        Attribute("target"),
        Literal("1(3{2})7", StringType),
        Literal(1, IntegerType)
      )
    

↓

      RegExpExtract(
        Attribute("target"),
        Literal("1(3{2})7", StringType),
        1
      )
    

Functions

      val builder = FunctionRegistry.builtin
                      .lookupFunctionBuilder("SUM").get
      builder(
        Attribute("target")
      )
    

↓

      Sum(
        Attribute("target")
      )
    

Aggregates

An expressions subclassing:

• AggregateFunction
  • DeclarativeAggregate
  • ImperativeAggregate

Parsing SQL

  SELECT ... FROM R
  WHERE ...
        

  SELECT ... FROM R
  GROUP BY ...
        

      SELECT REGEXP_EXTRACT(...) FROM R
    

vs

      SELECT SUM(...) FROM R
    

How does the parser distinguish these cases?

It doesn't

      SELECT SUM(A) FROM R
    

↓

      Project(Seq(
        UnresolvedFunction("SUM", Seq(
          UnresolvedAttribute("A")
        ))
      ), ...)
    

      Project(Seq(
        UnresolvedFunction("SUM", Seq(
          UnresolvedAttribute("A")
        ))
      ), ...)
    

↓

      Project(Seq(
        Sum(Attribute("A"))
      ), ...)
    

Now you can tell it's an aggregate.

Basic Guideline: If any expression is an AggregateFunction, the entire Project node should be an Aggregate instead.

      Project(targets, child) => 
        Aggregate(Seq(), targets, child)
    

Supporting everything Spark supports
will be a lot more work.

1. Assign input tuple to a group based on groupingAttributes
2. Accumulate for each aggregate in aggregateExpressions
3. Repeat for all input tuples
4. "Render" result based on aggregateExpressions

COUNT(*)

Init

$0$

Fold(Accum, New)

$Accum + 1$

SUM(A)

Init

$0$

Fold(Accum, New)

$Accum + New$

AVG(A)

Init

$\{ sum = 0, count = 0 \}$

Fold(Accum, New)

$\{ sum = Accum.sum + New, \\\;count = Accum.count + 1\}$

Finalize(Accum)

$\frac{Accum.sum}{Accum.count}$

Basic Aggregate Pattern

Init

Define a starting value for the accumulator

Fold(Accum, New)

Merge a new value into the accumulator

Finalize(Accum)

Extract the aggregate from the accumulator.

What does the accumulator look like for each aggregate?

Aggregation Buffers

AggregationFunction.aggBufferAttributes

The attributes that the aggregation function is requesting.

Allocate an InternalRow
with this schema for each function.

Aggregation Buffers

• One Buffer Per Aggregate, Per Group
• One Buffer Per Group (Aggregates Share)

↙ To be Discussed ↘

↖ What Spark Does ↗

DeclarativeAggregates

updateExpressions

How to manage Unresolved Attributes?

Input

[Buffer, InputRow]

Schema (for resolution)

agg.aggBufferAttributes
   ++ child.output

Adjust based on how you implemented your aggregation buffer.

DeclarativeAggregates

evaluateExpression

Input

Buffer

Schema (for resolution)

agg.aggBufferAttributes