[DMDB] Sorting, selection, projection, aggregation

semester4/dmdb/parts/04_query-processing/02_projection/00_intro.tex

@@ -0,0 +1,11 @@
+\subsubsection{Duplicate Elimination}
+This is done using either hashing or sorting.
+The hashing approach hashes an element and checks in the existing map if this element exists already. If not, it is inserted, else it is skipped.
+Finally the map is returned, or transformed and returned.
+
+For large datasets, we may want to create a hash table of size $B$, then run multiple dedupe passes, on each subset where duplicates could be,
+then return the union'd map.
+
+A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
+However, this tends to be more expensive than the hash-based approach for low amounts of data, but for larger amounts, very similar.
+The sort-based approach also has the benefit of producing sorted data, which may be useful for future operators.

semester4/dmdb/parts/04_query-processing/02_projection/01_set-operations.tex

@@ -0,0 +1,21 @@
+\subsubsection{Set Operations}
+\paragraph{Union / Union All}
+$R \cup S$ is the operation to perform.
+We can again use a sort-based or hash-based approach:
+\begin{itemize}
+    \item \bi{Sort-based}: We first sort both relations on all attributes and merge the sorted relations, while eliminating duplicates (except if \texttt{UNION ALL})
+    \item \bi{Hash-based}: We first hash-partition both $R$ and $S$.
+          We then build an in-memory hash table for each partition $R_p$ of $R$ and probe with tuples in corresponding $S_p$ of $S$ and add to the output, if not duplicate.
+          (Again only if not \texttt{UNION ALL})
+\end{itemize}
+
+
+\paragraph{Difference}
+$R - S$ is the operation to perform.
+We can again use a sort-based or hash-based approach:
+\begin{itemize}
+    \item \bi{Sort-based}: We first sort both relations on all attributes and merge the sorted relations, while eliminating tuples from $R$ that exist in $S$
+    \item \bi{Hash-based}: We first hash-partition both $R$ and $S$.
+          We then build an in-memory hash table for each partition $R_p$ of $R$ and probe with tuples in corresponding $S_p$ of $S$ and add to the output,
+          if it does not exist in the hash table.
+\end{itemize}