[DMDB] Fix issues, improve query processing with notes on time complexities

semester4/dmdb/parts/03_systems/02_indexing/06_art-trees.tex

@@ -0,0 +1,16 @@
+Adaptive Radix Trees (ARTs) are mainly used to ensure primary key constraints and to speed up point and highly selective queries (selectivity <0.1\%).
+It can also be manually created using \texttt{CREATE INDEX} and are automatically created for columns with a \texttt{UNIQUE} or \texttt{PRIMARY KEY} constraint.
+
+The above applies to DuckDB, which is often similar to Postgres.
+ARTs are not balanced and the internal node size is modified to accommodate the data distribution, thus not requiring nodes to have fixed sizes, which would contain empty pointers.
+
+The internal node size however typically isn't \textit{entirely} variable, but often comes in four different sizes, facilitating traversal.
+Each node stores the keys and pointers to the next node to follow for a given key.
+
+Common optimizations includes path compression, where particularly with long keys, some nodes could potentially only contain a single pointer.
+These nodes can be collapsed and removed to make the tree smaller (i.e. the pointer in the parent is updated to directly point to the data instead).
+If required by insertions, nodes can be added as needed (akin to B+ trees).
+
+For speed, ART require indexed values to be binary-comparable.
+This can become an issue with data types such as signed numbers (extra bit for sign), strings (termination character) and nulls (must be mapped to a non-value sequence).
+Thus, the data needs to be transformed to be able to index it and search the tree.

semester4/dmdb/parts/03_systems/02_indexing/07_r-trees.tex

@@ -0,0 +1,8 @@
+R-Trees are typically used for geospatial datasets, where searching for spatial relationship is expensive (requiring a full table scan).
+
+R-Trees create indexes on bounding boxes as opposed to points in 2- or 3D space.
+We can locate the smallest bounding box where a tuple may lay by traversing the index and looking up the leaf to see if the point is there (or to look for a near neighbour).
+
+For more information, see \hlhref{https://en.wikipedia.org/wiki/R-tree}{Wikipedia}
+
+{\scriptsize (yes, they actually just linked to Wikipedia, so it probably isn't very exam-relevant)}

semester4/dmdb/parts/03_systems/02_indexing/08_other-optimizations.tex

@@ -0,0 +1,7 @@
+For groups of data that don't change often, we can compute statistics on insert, such as \texttt{sum}, \texttt{avg}, \texttt{min}, \texttt{max}, etc.
+These aggregates can be used to check whether data needed is within that group.
+This also enable computing (some) aggregates without reading the entire table and is commonly used in data mining and data cubes.
+
+A concept that is not used in DB engines, is count min sketch. It can be very useful in data streaming engines and large scale data processing.
+What this does is it computes statistics on how many times a given value was seen in a list or stream of data and is a variation of the bloom filter,
+but with several arrays and counting. The overestimation in count is bounded by the size in the matrix.

semester4/dmdb/parts/04_query-processing/00_sorting/01_merge-sort.tex

@@ -12,10 +12,19 @@ The cost for this is for the 1-page sorted runs (i.e. sorting the original pages
 The total cost is given by $2 \cdot M \cdot N$ I/O operations, with $M$ the number of passes / phases (i.e. how many merge steps there are) and $N$ being the number of frames.
 
 For Two-Way Merge Sort, the number of passes, $M$ is given by $M = \lceil \log_2(N) \rceil + 1$,
-so the total cost to sort is $2N(\lceil \log_2 N \rceil + 1)$
+so the total cost to sort is \cost{$2N(\lceil \log_2 N \rceil + 1)$}
 
 The primary goal for sorting is of course efficiency. Primarily we want to reduce the amount of extra RAM used, the disk I/O (as that is slow) and trying to reduce random accesses
 in favour of sequential accesses. And of course, reducing CPU cost is still a concern.
 
 We can increase the number of frames to $B$ for the sorted runs, and then perform a $B - 1$-way merge.
-In that case, the number of operations are given by $2N (1 + \lceil \log_{B - 1} \lceil N / B \rceil \rceil)$
+\coloredbox{orange}{I/O cost for merge sort}{
+    (the $2$ comes from the fact that each pass reads and writes $N$ pages)
+    \[
+        2N \cdot P \text{ with } P = (1 + \lceil \log_{B - 1} \lceil N / B \rceil \rceil) \text{ the number of passes}
+    \]
+}
+
+If for instance, we want to sort $N = 600$ pages with $B = 50$ buffers, pass $0$ creates $\ceil{600 / 50} = 12$ runs,
+which fits in a single merge pass because $12 \leq B - 1$, which is derived from the logarithm.
+Thus we have $2$ passes and the cost in I/Os is $2 \cdot 2 \text{ passes } \cdot 600$.

semester4/dmdb/parts/04_query-processing/00_sorting/02_replacement-sort.tex

@@ -24,7 +24,7 @@ It works as follows:
 Informally, it is searching for an always larger value once it has committed to a certain value.
 It picks the first value to be the lowest value in the current frames.
 
-A new run is started if all values in the loaded frames are defered or used already.
+A new run is started if all values in the loaded frames are deferred or used already.
 
 This means that after the sort run is complete, a merge is still needed.
 
@@ -33,6 +33,8 @@ This means that after the sort run is complete, a merge is still needed.
 
 \bi{Worst case}: $\tco{B - 1}$ elements in a run (reversed data)
 
-\bi{Average case}: $\tct{2B}$ elements in a run. In that case, the cost of sorting is $2N(1 + \lceil \log_{B - 1} \lceil N / 2B \rceil \rceil)$
+\bi{Average case}: $\tct{2B}$ elements in a run.
 
-Thus, Quicksort is fatster, but longer runs often mean fewer passes, saving time.
+In any case, the cost of sorting is \cost{$2N(1 + \lceil \log_{B - 1} \lceil N / E \rceil \rceil)$}, with $E$ the number of elements ($2B$ in the average case)
+
+Thus, Quicksort is faster, but longer runs often mean fewer passes, saving time.

semester4/dmdb/parts/04_query-processing/00_sorting/03_sorting-with-b-trees.tex

@@ -2,5 +2,5 @@
 If a table has a B+ Index on the sorted column, we could retrieve the records in order by traversing the leaf pages.
 In case the B+ tree is clustered, this is a good idea, if it is not, then it could be very bad (due to random accesses).
 
-The operating principle is very simple, abusing the linked lists in the leafs of the B+ tree.
+The operating principle is very simple, abusing the linked lists in the leafs of the B+ tree, leading to each page being fetched just once,
-If the data is not clustered, we have one I/O per record, which is bad.
+instead of one page fetch (= I/O) per record, if the data is not clustered, we have one I/O per record, which is bad.

semester4/dmdb/parts/04_query-processing/01_selection/00_intro.tex

@@ -1,6 +1,6 @@
 Two terms important here are \textit{logical selection}, which describes \bi{what} we want to select and \textit{physical selection},
 which describes \bi{how} the algorithm or procedure works that actually retrieves, or filters, the data.
 
-The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is the number of pages in each relation.
+The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is \cost{$N$}, where $N$ is the number of pages in each relation.
 Alternatively, we can use \textit{index scan}, where we use an index to retrieve the matching rows.
 The cost then of course depends on the index used and if said index can even be used to generate the resulsts needed. We will cover that in more detail now.

semester4/dmdb/parts/04_query-processing/01_selection/01_index-scan.tex

@@ -6,7 +6,7 @@
     \item \bi{Bitmap Index}: $\tco{\text{size of bitmap index}} + X$, {\color{red} $X$ depends on clustering again}
 \end{itemize}
 
-The I/O cost for B+ Tree Index Scan is $\texttt{tree height} + \texttt{\# leaf pages} + \texttt{\# file pages}$
+The I/O cost for B+ Tree Index Scan is \cost{$\texttt{tree height} + \texttt{\#leaf pages} + \texttt{\#file pages}$}
 
 \inlineexample{Computation example}
 Given a relation $R$ with $N =$ one million records. There are 100 records on a page and we have a B+ Tree with the data entries $<k, rid>$

semester4/dmdb/parts/04_query-processing/02_projection/00_intro.tex

@@ -1,11 +1,6 @@
-\subsubsection{Duplicate Elimination}
+For a basic \texttt{SELECT A, B FROM R} query, we scan the file and for each tuple output $A, B$.
-This is done using either hashing or sorting.
+Obviously, that means that for $N$ tuples and ratio of included tuples of $Q$, we have \cost{$N + N \cdot Q$} I/O operations.
-The hashing approach hashes an element and checks in the existing map if this element exists already. If not, it is inserted, else it is skipped.
-Finally the map is returned, or transformed and returned.
 
-For large datasets, we may want to create a hash table of size $B$, then run multiple dedupe passes, on each subset where duplicates could be,
+For a \texttt{SELECT DISTINCT A, B FROM R} query, we scan the file and eliminate duplicates before outputting.
-then return the union'd map.
+If this is also combined with a sort, we can prefer the sort-based approach and deduplicate after search,
-
+so we are projecting, then sorting and finally deduplicating using the sort method.
-A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
-However, this tends to be more expensive than the hash-based approach for low amounts of data, but for larger amounts, very similar.
-The sort-based approach also has the benefit of producing sorted data, which may be useful for future operators.

semester4/dmdb/parts/04_query-processing/02_projection/01_duplicate-elimination.tex

@@ -0,0 +1,63 @@
+\subsubsection{Duplicate Elimination}
+This is done using either hashing or sorting.
+
+\paragraph{Hash-based approach}
+Assume we have $B$ memory buffers.
+Then the hashing approach works as follows:
+\begin{enumerate}
+    \item Project out attributes and splitting the input into $B - 1$ partitions using a hash function $h1$.
+    \item After that, all duplicates will be in the same partition. Since there most likely are further collisions, for each partition from (1), we do the following:
+          \begin{enumerate}
+              \item If the partition fits into memory, then we are done with this step
+              \item If not, we run step recursively run step 1 again on this partition
+          \end{enumerate}
+    \item We return all keys from the hash table
+\end{enumerate}
+A partition from step 2 fits into memory if $\frac{f \cdot T}{B - 1} < B$ (or approximately $B > \sqrt{f \cdot T}$),
+where $T$ is the number of pages after the projection and $f$ is a \textit{fudge factor}, typically $f \approx 1.2$.
+
+\coloredbox{orange}{Cost of hash-based duplicate elimination}{
+    $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{elim}(R')$, with
+    \[
+        \texttt{elim}(R') = \sum_{T \in R'} \begin{cases}
+            \texttt{elim}(T)   & \text{if } \frac{f \cdot L(T)}{B - 1} \geq B \\
+            \texttt{length}(T) & \text{else}
+        \end{cases}
+    \]
+    where $R'$ is the relation $R$ after the projection. $\texttt{cnt}(R)$ is the number of pages and $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot Q$,
+    with $Q$ the ratio of kept and total attributes in the relation.
+}
+
+\paragraph{Sort-based approach}
+A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
+
+The cost here is $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cost}_\texttt{sort}(R') + \texttt{cnt}(R')$.
+Broken down:
+\begin{itemize}
+    \item $\texttt{cnt}(R)$ I/Os for the initial scan
+    \item first $\texttt{cnt}(R')$ I/Os for storing the projected $R$ (denoted $R'$) on disk
+    \item $\texttt{cost}_\texttt{sort}(R')$ I/Os to sort the remaining attributes (see \ref{sec:external-sort})
+    \item second $\texttt{cnt}(R')$ I/Os to scan the sorted result and eliminate duplicates
+\end{itemize}
+
+
+There is also an optimized version that uses a modified sort algorithm that during sorting projects out the attributes, with cost
+\cost{$\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cnt}(R')$}. The first count is to read, sort and project; second count is to store the result to disk;
+third count is to eliminate duplicates.
+
+
+\paragraph{Comparison}
+The sorting approach has the benefit of better handling data skew and that the result is sorted.
+The I/O costs are comparable if $B$ is large, both algorithms need 2 passes over the data.
+For low numbers of records, the hash-based approach is likely cheaper
+
+
+\paragraph{Index-based approach}
+Here, we only scan the index, and we don't visit the relation. The projection attributes need to be a subset of the index attributes.
+We can project distinct on $A$ using a B+ tree on $A, B$.
+We then only apply the projection algorithm only to the data entries.
+
+Then, if an ordered index contains all projection attributes as prefix of the search key,
+we first retrieve index data entries in order, discard unwanted fields and compare adjacent entries to eliminate duplicates (if needed).
+
+If an index is smaller than a relation, then it is faster to scan the index.

semester4/dmdb/parts/04_query-processing/03_aggregation.tex

@@ -1,10 +1,10 @@
 And again, we have the option to do this via sorting or hashing:
 \begin{itemize}
     \item \bi{Sorting}: Sort on the attribute. Then scan the sorted tuples, computing running aggregate (such as max/min, average, etc).
-          When a new group is encountered, then we output the aggregate.
+          When a new group is encountered, then we output the aggregate. The cost is \cost{$\texttt{cost}_\texttt{sort}(R) + \texttt{cnt}(R)$}.
           Alternatively, we can already compute the aggregates on the last step of sorting (the merge) to save some time.
-          The limiting factor then becomes the sorting.
+          The limiting factor then becomes the sorting and the cost is \cost{$\texttt{cost}_\texttt{sort}(R)$}
     \item \bi{Hashing}: We hash the attribute and now each hash table entry is a group of all records with this value of the attribute.
-          For each one of these groups, we compute the aggregate.
+          For each one of these groups, we compute the aggregate. The cost is \cost{$\texttt{cnt}(R)$}
-          If the table is too large for memory, we use a two-step approach, as before
+          If the table is too large for memory, we use a two-step approach, as before. The cost is then very similar \cost{$\texttt{cnt}(R) + B$} for $B$ tables.
 \end{itemize}

semester4/dmdb/parts/04_query-processing/04_join/00_simple-join.tex

@@ -13,4 +13,6 @@
         \EndProcedure
     \end{algorithmic}
 \end{algorithm}
-Here, the max output size is $T_R \cdot T_S$, where $T_X$ is the number of tuples for realtion $X$.
+Here, the max output size is $T_R \cdot T_S$, where $T_X$ is the number of tuples for relation $X$.
+
+The cost is obviously \cost{$\texttt{cnt}(R) \cdot \texttt{cnt}(S)$}, with $\texttt{cnt}(X)$ the number of tuples in $X$.

semester4/dmdb/parts/04_query-processing/04_join/01_nested-loops.tex

@@ -13,5 +13,5 @@
     \end{algorithmic}
 \end{algorithm}
 
-The cost here is $P_R + P_R \cdot P_S$ I/Os, where $P_X$ is the number of pages in the relation $X$.
+The cost here is \cost{$P_R + P_R \cdot P_S$ I/Os}, where $P_X$ is the number of pages in the relation $X$.
 Furthermore, the DBMS should put the smaller relation on the outer loop to improve performance.

semester4/dmdb/parts/04_query-processing/04_join/02_index-nested-loops.tex

@@ -13,7 +13,7 @@
     \end{algorithmic}
 \end{algorithm}
 
-The cost here is $P_R + T_R \cdot C(I_S)$ I/Os, with $P_X$ the number of pages in the relation $X$, $T_X$ the number of tuples in said relation
+The cost here is \cost{$P_R + T_R \cdot C(I_S)$ I/Os}, with $P_X$ the number of pages in the relation $X$, $T_X$ the number of tuples in said relation
 and $C(I_X)$ is the cost to probe the index $I_X$ of relation $X$ and depends on the clustering of data and type of index.
 
 This operation needs three buffer frames. $1$ for $R$, $1$ for output and $1$ to traverse $I_S$

semester4/dmdb/parts/04_query-processing/04_join/03_block-nested-loops.tex

@@ -16,7 +16,7 @@
 We have $B$ buffer frames, of which one is reserved for the pages of $S$ and one is reserved for the output, thus the $B - 2$ blocks.
 Each of these blocks houses $P_R / (B - 2))$ pages of $R$, reducing search costs by only scanning $S$ once for every block.
 
-The cost here is $P_R + (P_R / (B - 2) \cdot P_S$ I/Os.
+The cost here is \cost{$P_R + (P_R / (B - 2) \cdot P_S$ I/Os}.
 If however, $R$ fits into memory, we have cost $P_R + P_S$ I/Os.
 We can reduce that further by building as second step (right after the first for loop) a in-memory hash table for the tuples in the $R$ pages block.
 This reduces the cost of comparisons, however, we need $(B - 2) \cdot f$ memory now.

semester4/dmdb/parts/04_query-processing/04_join/04_block-index-nested-loops.tex

@@ -14,7 +14,7 @@
     \end{algorithmic}
 \end{algorithm}
 
-The cost here is $P_R + T_R \cdot C(I_S)$ I/Os.
+The cost here is \cost{$P_R + T_R \cdot C(I_S)$ I/Os}.
 We sort the tuples in each buffer frame on the predicate key, which leads to tuples in $R$ with the same key only requiring a single index search.
 In addition, tuples with ``similar'' values in $R$ will likely match with keys on the same pages of $I_S$, which will already be in the buffer pool.
 

semester4/dmdb/parts/04_query-processing/04_join/05_sort-merge-join.tex

@@ -2,11 +2,11 @@
 We first sort $R$ and $S$ on the join attributes with external merge sort (see Section~\ref{sec:external-sort}). We then read the sorted files and merge.
 This is of course not great for performance, but if the relations are already sorted, we can skip the first step.
 
-The cost (if there are \bi{no duplicates}) is $\texttt{Sort}(S) + \texttt{Sort}(R) + P_R + P_S$ I/Os.
+The cost (if there are \bi{no duplicates}) is \cost{$\texttt{Sort}(S) + \texttt{Sort}(R) + P_R + P_S$ I/Os}.
 
 If there are duplicates (i.e. the keys are not unique, say we are joining on a column that doesn't have the UNIQUE constraint)
 then we need to move the pointer for the join back on the second relation to make sure we don't miss any tuples.
 
-The cost now of course isn't linear anymore and worst case quadratic, $\tco{\texttt{Sort}(S) + \texttt{Sort}(R) + P_R \cdot P_S}$.
+The cost now of course isn't linear anymore and worst case quadratic, \cost{$\tco{\texttt{Sort}(S) + \texttt{Sort}(R) + P_R \cdot P_S}$}.
 
 Compared to BNLJ, its performance doesn't depend on the amount of memory available, but tends to be a tad slower than BNLJ, if BNLJ is provided with ample memory.

semester4/dmdb/parts/04_query-processing/04_join/06_optimized-smj.tex

@@ -1,10 +1,10 @@
 \subsubsection{Optimized Sort Merge Join ((O)SMJ)}
 In this section, we are exploring how we can optimize sort and merge with extra memory.
 
-Sorting can be optimized using replacement sort, the output is $\frac{P_R + P_S}{2B}$ sorted runs of length $2B$.
+Sorting can be optimized using replacement sort, the output is \cost{$\frac{P_R + P_S}{2B}$} sorted runs of length $2B$.
 
 That of course means that the logical optimization is to do a $K$-way merge to merge the sorted runs and join the relations at the same time.
 If we have $B$ frames, so we can do a $B-1$ way merge, otherwise, we merge runs from each relation separately to reduce the number of runs
 until we have less than $B$ sorted runs, then we perform the join.
 
-The cost drops to $3 \cdot (P_R + P_S)$ I/Os, but we need memory such that $B^2 \geq \max\{ P_R, P_S \}$
+The cost drops to \cost{$3 \cdot (P_R + P_S)$ I/Os}, but we need memory such that $B^2 \geq \max\{ P_R, P_S \}$

semester4/dmdb/parts/04_query-processing/04_join/08_partitioned-hash-join.tex

@@ -20,4 +20,4 @@ The benefit of this approach is that in the third step, we are only operating on
 What we ideally do in the above algorithm is to do \textit{recursive partitioning}, i.e. as already \textit{somewhat} described, we partition each partition again,
 if it has more than one page. This further reduces access times.
 
-The cost for this is $3 \cdot (P_S + P_R)$, thus both OSMJ and PHJ are very similar in performance.
+The cost for this is \cost{$3 \cdot (P_S + P_R)$}, thus both OSMJ and PHJ are very similar in performance.

semester4/dmdb/parts/04_query-processing/main.tex

@@ -1,4 +1,7 @@
+\newcommand\cost[1]{\shade{orange}{#1}}
 \section{Query Processing}
+In this section, knowing the time complexities by heart is very important for the exam.
+They are highlighted \cost{like this}.
 
 \subsection{Sorting}
 \input{parts/04_query-processing/00_sorting/00_intro.tex}
@@ -7,15 +10,18 @@
 \input{parts/04_query-processing/00_sorting/03_sorting-with-b-trees.tex}
 % \input{parts/04_query-processing/00_sorting/}
 
+\newpage
 \subsection{Selection}
 \input{parts/04_query-processing/01_selection/00_intro.tex}
 \input{parts/04_query-processing/01_selection/01_index-scan.tex}
 \input{parts/04_query-processing/01_selection/02_index-matching.tex}
 % \input{parts/04_query-processing/01_selection/}
 
+\newpage
 \subsection{Projection}
 \input{parts/04_query-processing/02_projection/00_intro.tex}
-\input{parts/04_query-processing/02_projection/01_set-operations.tex}
+\input{parts/04_query-processing/02_projection/01_duplicate-elimination.tex}
+\input{parts/04_query-processing/02_projection/02_set-operations.tex}
 % \input{parts/04_query-processing/02_projection/}
 
 \subsection{Aggregation}