[DMDB] Notes on query processing, better summary for those, some exam notes

semester4/dmdb/code/sql/ddl/create.sql

@@ -1,6 +1,7 @@
 CREATE TABLE TableName (
-    Attribute integer,
+    Attribute integer UNIQUE,
-    OtherAttribute varchar (30),
+    OtherAttribute varchar(30) NOT NULL, -- Not null
-    NextAttribute character (2) default "AP", -- default value, if unset on insert
+    NextAttribute character(2) default "AP", -- default value, if unset on insert
-    PRIMARY KEY (Attribute) -- primary key, as in RA
+    PRIMARY KEY (Attribute), -- primary key, as in RA
+    CHECK (Attribute > 1)
 );

semester4/dmdb/data-modelling-databases-summary.tex

@@ -90,6 +90,7 @@ In addition, the end user does not have to worry about optimization too much. Th
 \include{parts/05_concurrency-control-recovery/main.tex}
 \include{parts/06_vector-search/main.tex}
 \include{parts/07_data/main.tex}
+\include{parts/08_quick-overview/main.tex}
 
 
 \printGlossary

semester4/dmdb/parts/01_sql/01_ddl.tex

@@ -27,6 +27,8 @@ Since the primary key must be unique for each entry, it may be useful to configu
 
 \inputcodewithfilename{sql}{}{code/sql/ddl/create.sql}
 
+Note that PostgreSQL doesn't support \texttt{AUTO\_INCREMENT} constraints, instead use the \texttt{SERIAL} (or \texttt{BIGSERIAL} type)
+
 
 % TODO: Make sure all the sql statements actually execute in pgsql
 \subsubsection{Updating / Altering Tables}

semester4/dmdb/parts/01_sql/03_query-language/00_intro.tex

@@ -13,7 +13,7 @@ To rename, we can set \texttt{I = Column as Name, Column2 as Name2}, etc
               Rename            & $\rho_{a, b, c} R$         & \texttt{SELECT A as a, \ldots, B as c FROM R;} \\
               Union             & $R_1 \cup R_2$             & \texttt{R1 UNION R2;}                          \\
               Difference        & $R_1 - R_2$                & \texttt{R1 EXCEPT R2;}                         \\
-              Intersection      & $R_1 - R_2$                & \texttt{R1 INTERSECT R2;}                      \\
+              Intersection      & $R_1 \cap R_2$             & \texttt{R1 INTERSECT R2;}                      \\
 \end{tables}
 
 Since \sql\ implements \gls{bag} and not set semantics, there is a \texttt{DISTINCT} keyword, which is used to remove duplicates.
@@ -26,3 +26,12 @@ In addition, to apply \textit{bag semantics}, as opposed to \textit{set semantic
 
 We can also name the tables, e.g. $\texttt{T} = \texttt{One o, Two t}$, and then access columns from a specific table using
 $\texttt{I} = \texttt{o.Col, b.Col}$, or the like.
+
+String concatenation works using \texttt{CONCAT('string', 'string', 'string', ...)}, or using \texttt{'string' || 'string' || \dots}. Note that SQL uses single quote for Strings,
+and double quotes for renaming columns (using \texttt{AS} statements, or in \texttt{SELECT})
+
+Dates work as you'd expect. We can create a new date using \texttt{Date('ISO-date-string')}. \texttt{DATETIME} combines date and time and \texttt{TIME} is just the time.
+To compute time delta, we can use \texttt{DATEDIFF('interval', DateOne, DateTwo)}, where the interval can be:
+\texttt{year, quarter, month, dayofyear, day, week, weekday, hour, minute, second, millisecond}.
+
+The current date, time, year, etc is provided using \texttt{current\_date} (etc).

semester4/dmdb/parts/01_sql/03_query-language/01_basic-operators.tex

@@ -31,3 +31,5 @@ or in \texttt{WHERE} clauses like this:
 
 \inlineremark You may have noticed that in the above query, the same table was used twice.
 This is referred to as a \bi{Self-Join} and can come in handy when comparing values in a single table.
+
+We can use comparison operators also with subqueries containing aggregations, if they return a single result. \TODO Check that this doesn't work with more than a single result

semester4/dmdb/parts/04_query-processing/00_sorting/01_merge-sort.tex

@@ -9,9 +9,9 @@ If there is ever just one frame (group) left, we add an empty one implicitly and
 
 The cost for this is for the 1-page sorted runs (i.e. sorting the original pages) is a single pass (read + write pass).
 
-The total cost is given by $2 \cdot M \cdot N$ I/O operations, with $M$ the number of passes / phases (i.e. how many merge steps there are) and $N$ being the number of frames.
+The total cost is given by $2 \cdot P \cdot N$ I/O operations, with $P$ the number of passes / phases (i.e. how many merge steps there are) and $N$ being the number of frames.
 
-For Two-Way Merge Sort, the number of passes, $M$ is given by $M = \lceil \log_2(N) \rceil + 1$,
+For Two-Way Merge Sort, the number of passes, $P$ is given by $P = \lceil \log_2(N) \rceil + 1$,
 so the total cost to sort is \cost{$2N(\lceil \log_2 N \rceil + 1)$}
 
 The primary goal for sorting is of course efficiency. Primarily we want to reduce the amount of extra RAM used, the disk I/O (as that is slow) and trying to reduce random accesses

semester4/dmdb/parts/04_query-processing/00_sorting/02_replacement-sort.tex

@@ -29,11 +29,11 @@ A new run is started if all values in the loaded frames are deferred or used alr
 This means that after the sort run is complete, a merge is still needed.
 
 \paragraph{Performance}
-\bi{Best case}: $\tcl{N}$ elements in a run (thus a single run, data is already sorted)
+\bi{Best case}: $N$ elements in a run (thus a single run, data is already sorted)
 
-\bi{Worst case}: $\tco{B - 1}$ elements in a run (reversed data)
+\bi{Worst case}: $B - 1$ elements in a run (reversed data)
 
-\bi{Average case}: $\tct{2B}$ elements in a run.
+\bi{Average case}: $2B$ elements in a run.
 
 In any case, the cost of sorting is \cost{$2N(1 + \lceil \log_{B - 1} \lceil N / E \rceil \rceil)$}, with $E$ the number of elements ($2B$ in the average case)
 

semester4/dmdb/parts/04_query-processing/01_selection/00_intro.tex

@@ -1,6 +1,7 @@
 Two terms important here are \textit{logical selection}, which describes \bi{what} we want to select and \textit{physical selection},
 which describes \bi{how} the algorithm or procedure works that actually retrieves, or filters, the data.
 
-The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is \cost{$N$}, where $N$ is the number of pages in each relation.
+The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is \cost{$N \div P_F$},
+where $N$ is the number of records in the relation and $P_F$ the number of records per page.
 Alternatively, we can use \textit{index scan}, where we use an index to retrieve the matching rows.
 The cost then of course depends on the index used and if said index can even be used to generate the resulsts needed. We will cover that in more detail now.

semester4/dmdb/parts/04_query-processing/01_selection/01_index-scan.tex

@@ -1,12 +1,19 @@
 \subsubsection{Index Scan}
 \begin{itemize}
     \item \bi{Hash Index}: {\color{ForestGreen} $\tco{1}$, we read the bucket and possibly the overflow buckets.} {\color{red} Can only be used for equality predicates}
-    \item \bi{B+ Tree Index}: $\tco{\log_F(N) + X}$, with $F$ fanout, $N$ the number of leaf nodes and $X$ the ratio of number of selected tuples and tuples per page.
+    \item \bi{B+ Tree Index}: $\tco{\log_F(N) + X}$ (or simply $\tco{h + X}$, with $h$ the height of the tree),
-              {\color{red} $X$ can be up to 1 per selected tuple with an unclustered index}. Optimization: we could sort the RIDs.
+          with $F$ fanout, $N$ the number of leaf nodes and $X$ the ratio of number of selected tuples and tuples per page.
+              {\color{red} $X$ can be up to 1 per selected tuple with an unclustered index, thus $N$ for all}. Optimization: we could sort the RIDs.
     \item \bi{Bitmap Index}: $\tco{\text{size of bitmap index}} + X$, {\color{red} $X$ depends on clustering again}
 \end{itemize}
 
-The I/O cost for B+ Tree Index Scan is \cost{$\texttt{tree height} + \texttt{\#leaf pages} + \texttt{\#file pages}$}
+The I/O cost for B+ Tree Index Scan is \cost{$\texttt{tree height} + \texttt{\#leaf pages} + \texttt{\#file pages}$}.
+Note that the number of leaf pages and file pages are multiplied with the selectivity of the predicate and are given by
+(for $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot S$, with $S$ the selectivity of the predicate):
+\begin{itemize}
+    \item \texttt{\#leaf pages}: $\texttt{cnt}(R') \div (P_L \cdot F_L)$, with $P_L$ the \#records per leaf page and $F_L$ the fill factor for the leaf.
+    \item \texttt{\#file pages}: $\texttt{cnt}(R') \div P_F$, with $P_F$ the number of records per file page.
+\end{itemize}
 
 \inlineexample{Computation example}
 Given a relation $R$ with $N =$ one million records. There are 100 records on a page and we have a B+ Tree with the data entries $<k, rid>$
@@ -17,6 +24,6 @@ Then the cost is computed as follows:
     \item 3 internal nodes to parse, $\ceil{\log_F(N)}$
     \item Number of result records = $1,000,000 * 1\% = 10,000$ (this is the selectivity)
     \item Number of leaf pages pointing to the results records = $10,000 / (500 \cdot 0.67) = 30$
-    \item Number of pages in the heap file that hold the result records $= 10,000 / 100 = 30$
+    \item Number of pages in the heap file that hold the result records $= 10,000 / 100 = 100$
 \end{itemize}
 Then, the total cost is $3 + 30 + 100 = 133$

semester4/dmdb/parts/04_query-processing/02_projection/00_intro.tex

@@ -1,5 +1,6 @@
 For a basic \texttt{SELECT A, B FROM R} query, we scan the file and for each tuple output $A, B$.
 Obviously, that means that for $N$ tuples and ratio of included tuples of $Q$, we have \cost{$N + N \cdot Q$} I/O operations.
+($N$ to read, $N \cdot Q$ to write the filtered tuples)
 
 For a \texttt{SELECT DISTINCT A, B FROM R} query, we scan the file and eliminate duplicates before outputting.
 If this is also combined with a sort, we can prefer the sort-based approach and deduplicate after search,

semester4/dmdb/parts/04_query-processing/02_projection/01_duplicate-elimination.tex

@@ -13,25 +13,21 @@ Then the hashing approach works as follows:
           \end{enumerate}
     \item We return all keys from the hash table
 \end{enumerate}
-A partition from step 2 fits into memory if $\frac{f \cdot T}{B - 1} < B$ (or approximately $B > \sqrt{f \cdot T}$),
+A partition from step 2 fits into memory if $\frac{f \cdot R'}{B - 1} < B$ (or approximately $B > \sqrt{f \cdot R'}$),
-where $T$ is the number of pages after the projection and $f$ is a \textit{fudge factor}, typically $f \approx 1.2$.
+where $R'$ is the number of pages after the projection and $f$ is a \textit{fudge factor}, typically $f \approx 1.2$.
 
 \coloredbox{orange}{Cost of hash-based duplicate elimination}{
-    $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{elim}(R')$, with
+    $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{repart}(R') + \texttt{cnt}(R')$, with $\texttt{repart}(R') = \sum_{i = 1}^{N} 2 \cdot \texttt{cnt}(R')$,
-    \[
+    with $N$ the number of times we had to re-partitioning the data to (typically) $B = (B - 1)^2$ partitions. The $\texttt{cnt}(R')$ is for dedupe.
-        \texttt{elim}(R') = \sum_{T \in R'} \begin{cases}
+
-            \texttt{elim}(T)   & \text{if } \frac{f \cdot L(T)}{B - 1} \geq B \\
+    $R'$ is the relation $R$ after the projection. $\texttt{cnt}(R)$ is the number of pages and $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot Q$,
-            \texttt{length}(T) & \text{else}
-        \end{cases}
-    \]
-    where $R'$ is the relation $R$ after the projection. $\texttt{cnt}(R)$ is the number of pages and $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot Q$,
     with $Q$ the ratio of kept and total attributes in the relation.
 }
 
 \paragraph{Sort-based approach}
 A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
 
-The cost here is $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cost}_\texttt{sort}(R') + \texttt{cnt}(R')$.
+The cost here is \cost{$\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cost}_\texttt{sort}(R') + \texttt{cnt}(R')$}.
 Broken down:
 \begin{itemize}
     \item $\texttt{cnt}(R)$ I/Os for the initial scan

semester4/dmdb/parts/08_quick-overview/00_sql.tex

@@ -0,0 +1,12 @@
+\subsection{SQL}
+DuckDB (and PostgreSQL) features an \texttt{EXPLAIN} and \texttt{EXPLAIN ANALYZE} query,
+pretty-printing the query plan and pretty-printing the query plan and profiling the query execution, respectively
+
+A few useful commands for debugging and writing queries (for DuckDB and PostgreSQL via \texttt{psql} CLI):
+\begin{tables}{p{4cm}p{4cm}p{8cm}}{DuckDB & PostgreSQL          & Description}
+              \texttt{SHOW TABLES;}       & \verb|\dt|          & Show all tables                         \\
+              \texttt{SHOW tablename;}    & \verb|\d tablename| & Show schema of table \texttt{tablename} \\
+\end{tables}
+Note that \texttt{SHOW} and \texttt{DESCRIBE} are aliases in DuckDB. They can also be used on queries!.
+
+To see all \texttt{psql} commands, run \verb|\?|.

semester4/dmdb/parts/08_quick-overview/02_system.tex

@@ -0,0 +1,21 @@
+\subsection{System}
+For computing the usable space, deduct the overhead \textit{after} multiplying with the free fraction,
+e.g. for a block size of 4kB (with kB actually being KiB), header and directories having a combined size of 100 bytes and \texttt{PCTFREE} set to 20\%,
+the available space is $\ceil{4096 \cdot 0.8 - 100} = 3177$.
+
+To compute the maximum number of tuples in a slotted page, remember that it also has a header with a certain number of bytes for the slotting.
+Add that to the number of bytes per tuple when computing, even though it doesn't count to the tuple size!
+
+Row stores are better if we need to access many columns, column stores otherwise. This is (primarily) due to indexing.
+
+
+\subsubsection{Indexing}
+The global depth is the maximum Local Depth of all buckets.
+
+The hash key is the actual value of the column to be hashed typically.
+Thus, an assignment may be very misleading in that the block ID and hash key are asked.
+The hash key is the value before hashing, the block ID afterwards is the block the hashing algorithm assigned it to.
+
+When splitting a block, its $N$ entries must be repartitioned. There are $2^N$ possible ways in which to do this.
+So, there is a $\frac{2}{2^N}$ probability for there to be a \bi{recursive overflow}, an overflow in which all keys end up in the same bucket again
+and need further splitting.

semester4/dmdb/parts/08_quick-overview/03_qp.tex

@@ -0,0 +1,43 @@
+\subsection{Query Processing}
+\subsubsection{Sorting}
+Given $B$ frames of memory and $N$ records, we ahve
+\begin{itemize}
+    \item \bi{Merge Sort}: $2N \cdot P$, with $P = (1 + \ceil{\log_{B - 1}\ceil{N \div B}})$ the number of passes.
+          After the first pass, $\ceil{N \div B}$ number of sorted runs were created (typically)
+    \item \bi{Replacement Sort / Heap Sort}: $2N \cdot P$, with $P = (1 + \ceil{\log_{B - 1} \ceil{N \div E}})$ the number of passes, with $E$ the number of records per run,
+          given by $E = 2B$ in the average case, $E = N$ in the best case (thus single run, data already sorted), $B - 1$ (worst case, reversed data)
+\end{itemize}
+
+
+\subsubsection{Selection}
+\begin{itemize}
+    \item \bi{File Scan}: $N \div P_F$, with $P_F$ the number of pages, because we need to run through all pages
+    \item \bi{B+ Tree}: $\texttt{height}(T) + \texttt{cnt}(P_L) + \texttt{cnt}(P_F)$, $P_L$ the leaf pages and $P_F$ the file pages.
+          \begin{itemize}
+              \item For a clustered index, $\texttt{cnt}(P_F) = \texttt{cnt}(R') \div (F_L \cdot R_L)$, with $R_L$ the number of records per leaf page and $F_L$ the fill factor.
+              \item For an unclustered index, $\texttt{cnt}(P_F) = \texttt{cnt}(R')$, with $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot S$, with $S$ the selectivity of the predicate.
+          \end{itemize}
+    \item \bi{Hash Index}: Cost is $1 + \texttt{cnt}(R') \div \texttt{cnt}(P_L) + \texttt{cnt}(R')$, where the $1$ is to do the lookup and
+          $\texttt{cnt}(R') \div P_L$, with $P_L$ the number of records per Leaf.
+\end{itemize}
+
+
+\subsubsection{Projection}
+\begin{itemize}
+    \item \bi{Partitioning}: $\texttt{Cost}_\texttt{part}(R) = \texttt{cnt}(R) + \texttt{cnt}(R')$ with
+          $\texttt{cnt}(R') = \texttt{cnt}(R) Q$, with $Q$ the fraction of selected attributes divided by total attributes
+    \item \bi{Sort-Based}: The number of sorted runs are $M = \ceil{\texttt{cnt}(R') \div B}$, with the merge passes $P = \ceil{\log_{B - 1}(M)}$, total cost:
+          $2 \cdot N \cdot P + \texttt{Cost}_\texttt{part}(R)$
+    \item \bi{Hash-Based}:
+          \begin{itemize}
+              \item \bi{Minimum $B$ for $M$-pass}: $B > \sqrt{\texttt{cnt}(R') \cdot f}$, with $f$ the Minister of Magic Factor (for Books $\leq$ 5),
+                    aka. Fudge Factor, typically $f \approx 1.2$.
+              \item \bi{Below threshold}: To check if we need recursive partitioning, we use the un-approximated version of the above: $\frac{f \cdot \texttt{cnt}(R')}{B - 1} < B$
+              \item \bi{Cost}: The base cost is $\texttt{Cost}_\texttt{part}(R) + \texttt{cnt}(R')$ for partitioning and duplicate elimination later on
+                    \begin{itemize}
+                        \item If below threshold, that's our cost
+                        \item If above threshold, then we need to repartition, which costs $2 \cdot \texttt{cnt}(R')$ (once each for write and read) for each time we do this action.
+                    \end{itemize}
+          \end{itemize}
+\end{itemize}
+Note that both the sort- and hash-based approaches perform approximately equally well at larger $B$, since both algorithms need two passes over the data.

semester4/dmdb/parts/08_quick-overview/main.tex

@@ -0,0 +1,7 @@
+\section{Quick Overview}
+This section aims to give you the most important things to remember in a very concise manner.
+
+\input{parts/08_quick-overview/00_sql.tex}
+\input{parts/08_quick-overview/01_theory.tex}
+\input{parts/08_quick-overview/02_system.tex}
+\input{parts/08_quick-overview/03_qp.tex}