[DMDB] Notes on query processing, better summary for those, some exam notes

semester4/dmdb/parts/04_query-processing/00_sorting/01_merge-sort.tex

@@ -9,9 +9,9 @@ If there is ever just one frame (group) left, we add an empty one implicitly and
 
 The cost for this is for the 1-page sorted runs (i.e. sorting the original pages) is a single pass (read + write pass).
 
-The total cost is given by $2 \cdot M \cdot N$ I/O operations, with $M$ the number of passes / phases (i.e. how many merge steps there are) and $N$ being the number of frames.
+The total cost is given by $2 \cdot P \cdot N$ I/O operations, with $P$ the number of passes / phases (i.e. how many merge steps there are) and $N$ being the number of frames.
 
-For Two-Way Merge Sort, the number of passes, $M$ is given by $M = \lceil \log_2(N) \rceil + 1$,
+For Two-Way Merge Sort, the number of passes, $P$ is given by $P = \lceil \log_2(N) \rceil + 1$,
 so the total cost to sort is \cost{$2N(\lceil \log_2 N \rceil + 1)$}
 
 The primary goal for sorting is of course efficiency. Primarily we want to reduce the amount of extra RAM used, the disk I/O (as that is slow) and trying to reduce random accesses

semester4/dmdb/parts/04_query-processing/00_sorting/02_replacement-sort.tex

@@ -29,11 +29,11 @@ A new run is started if all values in the loaded frames are deferred or used alr
 This means that after the sort run is complete, a merge is still needed.
 
 \paragraph{Performance}
-\bi{Best case}: $\tcl{N}$ elements in a run (thus a single run, data is already sorted)
+\bi{Best case}: $N$ elements in a run (thus a single run, data is already sorted)
 
-\bi{Worst case}: $\tco{B - 1}$ elements in a run (reversed data)
+\bi{Worst case}: $B - 1$ elements in a run (reversed data)
 
-\bi{Average case}: $\tct{2B}$ elements in a run.
+\bi{Average case}: $2B$ elements in a run.
 
 In any case, the cost of sorting is \cost{$2N(1 + \lceil \log_{B - 1} \lceil N / E \rceil \rceil)$}, with $E$ the number of elements ($2B$ in the average case)
 

semester4/dmdb/parts/04_query-processing/01_selection/00_intro.tex

@@ -1,6 +1,7 @@
 Two terms important here are \textit{logical selection}, which describes \bi{what} we want to select and \textit{physical selection},
 which describes \bi{how} the algorithm or procedure works that actually retrieves, or filters, the data.
 
-The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is \cost{$N$}, where $N$ is the number of pages in each relation.
+The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is \cost{$N \div P_F$},
+where $N$ is the number of records in the relation and $P_F$ the number of records per page.
 Alternatively, we can use \textit{index scan}, where we use an index to retrieve the matching rows.
 The cost then of course depends on the index used and if said index can even be used to generate the resulsts needed. We will cover that in more detail now.

semester4/dmdb/parts/04_query-processing/01_selection/01_index-scan.tex

@@ -1,12 +1,19 @@
 \subsubsection{Index Scan}
 \begin{itemize}
     \item \bi{Hash Index}: {\color{ForestGreen} $\tco{1}$, we read the bucket and possibly the overflow buckets.} {\color{red} Can only be used for equality predicates}
-    \item \bi{B+ Tree Index}: $\tco{\log_F(N) + X}$, with $F$ fanout, $N$ the number of leaf nodes and $X$ the ratio of number of selected tuples and tuples per page.
-              {\color{red} $X$ can be up to 1 per selected tuple with an unclustered index}. Optimization: we could sort the RIDs.
+    \item \bi{B+ Tree Index}: $\tco{\log_F(N) + X}$ (or simply $\tco{h + X}$, with $h$ the height of the tree),
+          with $F$ fanout, $N$ the number of leaf nodes and $X$ the ratio of number of selected tuples and tuples per page.
+              {\color{red} $X$ can be up to 1 per selected tuple with an unclustered index, thus $N$ for all}. Optimization: we could sort the RIDs.
     \item \bi{Bitmap Index}: $\tco{\text{size of bitmap index}} + X$, {\color{red} $X$ depends on clustering again}
 \end{itemize}
 
-The I/O cost for B+ Tree Index Scan is \cost{$\texttt{tree height} + \texttt{\#leaf pages} + \texttt{\#file pages}$}
+The I/O cost for B+ Tree Index Scan is \cost{$\texttt{tree height} + \texttt{\#leaf pages} + \texttt{\#file pages}$}.
+Note that the number of leaf pages and file pages are multiplied with the selectivity of the predicate and are given by
+(for $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot S$, with $S$ the selectivity of the predicate):
+\begin{itemize}
+    \item \texttt{\#leaf pages}: $\texttt{cnt}(R') \div (P_L \cdot F_L)$, with $P_L$ the \#records per leaf page and $F_L$ the fill factor for the leaf.
+    \item \texttt{\#file pages}: $\texttt{cnt}(R') \div P_F$, with $P_F$ the number of records per file page.
+\end{itemize}
 
 \inlineexample{Computation example}
 Given a relation $R$ with $N =$ one million records. There are 100 records on a page and we have a B+ Tree with the data entries $<k, rid>$
@@ -17,6 +24,6 @@ Then the cost is computed as follows:
     \item 3 internal nodes to parse, $\ceil{\log_F(N)}$
     \item Number of result records = $1,000,000 * 1\% = 10,000$ (this is the selectivity)
     \item Number of leaf pages pointing to the results records = $10,000 / (500 \cdot 0.67) = 30$
-    \item Number of pages in the heap file that hold the result records $= 10,000 / 100 = 30$
+    \item Number of pages in the heap file that hold the result records $= 10,000 / 100 = 100$
 \end{itemize}
 Then, the total cost is $3 + 30 + 100 = 133$

semester4/dmdb/parts/04_query-processing/02_projection/00_intro.tex

@@ -1,5 +1,6 @@
 For a basic \texttt{SELECT A, B FROM R} query, we scan the file and for each tuple output $A, B$.
 Obviously, that means that for $N$ tuples and ratio of included tuples of $Q$, we have \cost{$N + N \cdot Q$} I/O operations.
+($N$ to read, $N \cdot Q$ to write the filtered tuples)
 
 For a \texttt{SELECT DISTINCT A, B FROM R} query, we scan the file and eliminate duplicates before outputting.
 If this is also combined with a sort, we can prefer the sort-based approach and deduplicate after search,

semester4/dmdb/parts/04_query-processing/02_projection/01_duplicate-elimination.tex

@@ -13,25 +13,21 @@ Then the hashing approach works as follows:
           \end{enumerate}
     \item We return all keys from the hash table
 \end{enumerate}
-A partition from step 2 fits into memory if $\frac{f \cdot T}{B - 1} < B$ (or approximately $B > \sqrt{f \cdot T}$),
-where $T$ is the number of pages after the projection and $f$ is a \textit{fudge factor}, typically $f \approx 1.2$.
+A partition from step 2 fits into memory if $\frac{f \cdot R'}{B - 1} < B$ (or approximately $B > \sqrt{f \cdot R'}$),
+where $R'$ is the number of pages after the projection and $f$ is a \textit{fudge factor}, typically $f \approx 1.2$.
 
 \coloredbox{orange}{Cost of hash-based duplicate elimination}{
-    $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{elim}(R')$, with
-    \[
-        \texttt{elim}(R') = \sum_{T \in R'} \begin{cases}
-            \texttt{elim}(T)   & \text{if } \frac{f \cdot L(T)}{B - 1} \geq B \\
-            \texttt{length}(T) & \text{else}
-        \end{cases}
-    \]
-    where $R'$ is the relation $R$ after the projection. $\texttt{cnt}(R)$ is the number of pages and $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot Q$,
+    $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{repart}(R') + \texttt{cnt}(R')$, with $\texttt{repart}(R') = \sum_{i = 1}^{N} 2 \cdot \texttt{cnt}(R')$,
+    with $N$ the number of times we had to re-partitioning the data to (typically) $B = (B - 1)^2$ partitions. The $\texttt{cnt}(R')$ is for dedupe.
+
+    $R'$ is the relation $R$ after the projection. $\texttt{cnt}(R)$ is the number of pages and $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot Q$,
     with $Q$ the ratio of kept and total attributes in the relation.
 }
 
 \paragraph{Sort-based approach}
 A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
 
-The cost here is $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cost}_\texttt{sort}(R') + \texttt{cnt}(R')$.
+The cost here is \cost{$\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cost}_\texttt{sort}(R') + \texttt{cnt}(R')$}.
 Broken down:
 \begin{itemize}
     \item $\texttt{cnt}(R)$ I/Os for the initial scan