[DMDB] Sorting, selection, projection, aggregation

2026-07-27 15:24:37 +02:00 · 2026-07-17 16:08:58 +02:00
parent 653a807848
commit 89009ef7d3
13 changed files with 161 additions and 6 deletions
@@ -5,8 +5,7 @@ which corresponds to $\Pi_{\texttt{I}}(\sigma_{\texttt{C}} \texttt{T})$.
 We can set \texttt{I = *} if we want to return all columns for a table.
 To rename, we can set \texttt{I = Column as Name, Column2 as Name2}, etc

-% TODO: Make RA an acronym, too
-\inlinetheorem Every SPJR \acrshort{RA} expression can be written in \texttt{SELECT ... FROM ... WHERE ...} form:
+\inlinetheorem Every SPJR \acrshort{ra} expression can be written in \texttt{SELECT ... FROM ... WHERE ...} form:
 \begin{tables}{lll}{Operation   & Notation                   & SQL}
              Selection         & $\sigma_c(R)$              & \texttt{SELECT * FROM R WHERE c;}              \\
              Projection        & $\Pi_{A_1, \ldots, A_n} R$ & \texttt{SELECT A1, \ldots, An FROM R;}         \\
@@ -1,4 +1,3 @@
-\section{Sorting}
 Sorting is crucial in DBMS due to the possibility that the user requests sorted data using an \texttt{ORDER BY} clause.
 In addtion, it can help speed up some operators.

@@ -1,5 +1,4 @@
-\subsection{Merge Sort}
-\subsubsection{Merging Sorted Runs}
+\subsubsection{Merge Sort}
 When we sort each page on load into memory, we then have to merge all pages together, or more precisely, combine elements into pages in correct order.
 For that, we keep pointers to each pair of frames, then we first take first element of either the first or second frame, depending on which one comes first in the order,
 and copy it into the empty frame. We apply the same again to fill up the empty frame (to a threshold or fully). When the frame is full, we create a second one and link it.
@@ -13,4 +12,10 @@ The cost for this is for the 1-page sorted runs (i.e. sorting the original pages
 The total cost is given by $2 \cdot M \cdot N$ I/O operations, with $M$ the number of passes / phases (i.e. how many merge steps there are) and $N$ being the number of frames.

 For Two-Way Merge Sort, the number of passes, $M$ is given by $M = \lceil \log_2(N) \rceil + 1$,
-so the total cost to sort is $2N(\lceil \log_2 N \rceil + 1)
+so the total cost to sort is $2N(\lceil \log_2 N \rceil + 1)$
+
+The primary goal for sorting is of course efficiency. Primarily we want to reduce the amount of extra RAM used, the disk I/O (as that is slow) and trying to reduce random accesses
+in favour of sequential accesses. And of course, reducing CPU cost is still a concern.
+
+We can increase the number of frames to $B$ for the sorted runs, and then perform a $B - 1$-way merge.
+In that case, the number of operations are given by $2N (1 + \lceil \log_{B - 1} \lceil N / B \rceil \rceil)$
@@ -0,0 +1,38 @@
+\subsubsection{Replacement Sort / Heap Sort}
+It has the advantage that it can produce longer runs with the same amount of memory, on average $2B$ (instead of $B$) with $B$ frames of memory.
+It is used as the sorting algorithm in pass $0$ above.
+
+It works as follows:
+\begin{algorithm}
+    \caption{Replacement Sort}
+    \begin{algorithmic}[1]
+        \Procedure{ReplacementSort}{$R$}
+            \State Read $B - 1$ pages of relation $R$, store in a heap (priority queue)
+            \For{every page}
+                \If{Top of heap is smaller than the end of the sorted run}
+                    \State Continue with next iteration
+                \ElsIf{No element is left in memory that is larger than last element of current sorted run}
+                    \State create new run
+                \Else
+                    \State load page, remove smaller records and place them on the current sorted run, until a frame is freed.
+                \EndIf
+            \EndFor
+        \EndProcedure
+    \end{algorithmic}
+\end{algorithm}
+
+Informally, it is searching for an always larger value once it has committed to a certain value.
+It picks the first value to be the lowest value in the current frames.
+
+A new run is started if all values in the loaded frames are defered or used already.
+
+This means that after the sort run is complete, a merge is still needed.
+
+\paragraph{Performance}
+\bi{Best case}: $\tcl{N}$ elements in a run (thus a single run, data is already sorted)
+
+\bi{Worst case}: $\tco{B - 1}$ elements in a run (reversed data)
+
+\bi{Average case}: $\tct{2B}$ elements in a run. In that case, the cost of sorting is $2N(1 + \lceil \log_{B - 1} \lceil N / 2B \rceil \rceil)$
+
+Thus, Quicksort is fatster, but longer runs often mean fewer passes, saving time.
@@ -0,0 +1,6 @@
+\subsubsection{Sorting with B+ Trees}
+If a table has a B+ Index on the sorted column, we could retrieve the records in order by traversing the leaf pages.
+In case the B+ tree is clustered, this is a good idea, if it is not, then it could be very bad (due to random accesses).
+
+The operating principle is very simple, abusing the linked lists in the leafs of the B+ tree.
+If the data is not clustered, we have one I/O per record, which is bad.
@@ -0,0 +1,6 @@
+Two terms important here are \textit{logical selection}, which describes \bi{what} we want to select and \textit{physical selection},
+which describes \bi{how} the algorithm or procedure works that actually retrieves, or filters, the data.
+
+The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is the number of pages in each relation.
+Alternatively, we can use \textit{index scan}, where we use an index to retrieve the matching rows.
+The cost then of course depends on the index used and if said index can even be used to generate the resulsts needed. We will cover that in more detail now.
@@ -0,0 +1,22 @@
+\subsubsection{Index Scan}
+\begin{itemize}
+    \item \bi{Hash Index}: {\color{ForestGreen} $\tco{1}$, we read the bucket and possibly the overflow buckets.} {\color{red} Can only be used for equality predicates}
+    \item \bi{B+ Tree Index}: $\tco{\log_F(N) + X}$, with $F$ fanout, $N$ the number of leaf nodes and $X$ the ratio of number of selected tuples and tuples per page.
+              {\color{red} $X$ can be up to 1 per selected tuple with an unclustered index}. Optimization: we could sort the RIDs.
+    \item \bi{Bitmap Index}: $\tco{\text{size of bitmap index}} + X$, {\color{red} $X$ depends on clustering again}
+\end{itemize}
+
+The I/O cost for B+ Tree Index Scan is $\texttt{tree height} + \texttt{\# leaf pages} + \texttt{\# file pages}$
+
+\inlineexample{Computation example}
+Given a relation $R$ with $N =$ one million records. There are 100 records on a page and we have a B+ Tree with the data entries $<k, rid>$
+and it has a capacity of 500 data entries on each leaf. It also has 3 internal node levels and a fill factor of $0.67$.
+
+Then the cost is computed as follows:
+\begin{itemize}
+    \item 3 internal nodes to parse, $\ceil{\log_F(N)}$
+    \item Number of result records = $1,000,000 * 1\% = 10,000$ (this is the selectivity)
+    \item Number of leaf pages pointing to the results records = $10,000 / (500 \cdot 0.67) = 30$
+    \item Number of pages in the heap file that hold the result records $= 10,000 / 100 = 30$
+\end{itemize}
+Then, the total cost is $3 + 30 + 100 = 133$
@@ -0,0 +1,15 @@
+\subsubsection{Index Matching}
+The predicates in the queries can contain comparison operators, as well as \texttt{AND} and \texttt{OR}, allowing conjuncts and disjuncts.
+
+Index Matching is the process of choosing the proper index for the task.
+For conjuncts only, we can use:
+\begin{itemize}
+    \item \bi{Hash Indexes}. Constraints: Only equality operations and predicate has to contain all index attributes
+    \item \bi{B+-Tree}. Constraints: The attributes in the predicates need to be prefixes of the search key
+\end{itemize}
+
+Here, the indexes need to match \bi{every} predicate in the disjunction.
+For instance, if we have a hash index on $A$ and a B+ tree on $B$, and the query is \verb|A = 7 OR B > 5|.
+In that case, we can either file scan or use both indexes (we fetch the RIDs and take the union)
+
+However, index matching is a hard problem.
@@ -0,0 +1,11 @@
+\subsubsection{Duplicate Elimination}
+This is done using either hashing or sorting.
+The hashing approach hashes an element and checks in the existing map if this element exists already. If not, it is inserted, else it is skipped.
+Finally the map is returned, or transformed and returned.
+
+For large datasets, we may want to create a hash table of size $B$, then run multiple dedupe passes, on each subset where duplicates could be,
+then return the union'd map.
+
+A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
+However, this tends to be more expensive than the hash-based approach for low amounts of data, but for larger amounts, very similar.
+The sort-based approach also has the benefit of producing sorted data, which may be useful for future operators.
@@ -0,0 +1,21 @@
+\subsubsection{Set Operations}
+\paragraph{Union / Union All}
+$R \cup S$ is the operation to perform.
+We can again use a sort-based or hash-based approach:
+\begin{itemize}
+    \item \bi{Sort-based}: We first sort both relations on all attributes and merge the sorted relations, while eliminating duplicates (except if \texttt{UNION ALL})
+    \item \bi{Hash-based}: We first hash-partition both $R$ and $S$.
+          We then build an in-memory hash table for each partition $R_p$ of $R$ and probe with tuples in corresponding $S_p$ of $S$ and add to the output, if not duplicate.
+          (Again only if not \texttt{UNION ALL})
+\end{itemize}
+
+
+\paragraph{Difference}
+$R - S$ is the operation to perform.
+We can again use a sort-based or hash-based approach:
+\begin{itemize}
+    \item \bi{Sort-based}: We first sort both relations on all attributes and merge the sorted relations, while eliminating tuples from $R$ that exist in $S$
+    \item \bi{Hash-based}: We first hash-partition both $R$ and $S$.
+          We then build an in-memory hash table for each partition $R_p$ of $R$ and probe with tuples in corresponding $S_p$ of $S$ and add to the output,
+          if it does not exist in the hash table.
+\end{itemize}
@@ -0,0 +1,10 @@
+And again, we have the option to do this via sorting or hashing:
+\begin{itemize}
+    \item \bi{Sorting}: Sort on the attribute. Then scan the sorted tuples, computing running aggregate (such as max/min, average, etc).
+          When a new group is encountered, then we output the aggregate.
+          Alternatively, we can already compute the aggregates on the last step of sorting (the merge) to save some time.
+          The limiting factor then becomes the sorting.
+    \item \bi{Hashing}: We hash the attribute and now each hash table entry is a group of all records with this value of the attribute.
+          For each one of these groups, we compute the aggregate.
+          If the table is too large for memory, we use a two-step approach, as before
+\end{itemize}
@@ -0,0 +1,23 @@
+\section{Query Processing}
+
+\subsection{Sorting}
+\input{parts/04_query-processing/00_sorting/00_intro.tex}
+\input{parts/04_query-processing/00_sorting/01_merge-sort.tex}
+\input{parts/04_query-processing/00_sorting/02_replacement-sort.tex}
+\input{parts/04_query-processing/00_sorting/03_sorting-with-b-trees.tex}
+% \input{parts/04_query-processing/00_sorting/}
+
+\subsection{Selection}
+\input{parts/04_query-processing/01_selection/00_intro.tex}
+\input{parts/04_query-processing/01_selection/01_index-scan.tex}
+\input{parts/04_query-processing/01_selection/02_index-matching.tex}
+% \input{parts/04_query-processing/01_selection/}
+
+\subsection{Projection}
+\input{parts/04_query-processing/02_projection/00_intro.tex}
+\input{parts/04_query-processing/02_projection/01_set-operations.tex}
+% \input{parts/04_query-processing/02_projection/}
+
+\subsection{Aggregation}
+\input{parts/04_query-processing/03_aggregation.tex}
+% \input{parts/04_query-processing/}