diff --git a/semester4/dmdb/data-modelling-databases-summary.pdf b/semester4/dmdb/data-modelling-databases-summary.pdf index 7eed211..0074504 100644 Binary files a/semester4/dmdb/data-modelling-databases-summary.pdf and b/semester4/dmdb/data-modelling-databases-summary.pdf differ diff --git a/semester4/dmdb/parts/03_systems/02_indexing/06_art-trees.tex b/semester4/dmdb/parts/03_systems/02_indexing/06_art-trees.tex index e69de29..924a6f3 100644 --- a/semester4/dmdb/parts/03_systems/02_indexing/06_art-trees.tex +++ b/semester4/dmdb/parts/03_systems/02_indexing/06_art-trees.tex @@ -0,0 +1,16 @@ +Adaptive Radix Trees (ARTs) are mainly used to ensure primary key constraints and to speed up point and highly selective queries (selectivity <0.1\%). +It can also be manually created using \texttt{CREATE INDEX} and are automatically created for columns with a \texttt{UNIQUE} or \texttt{PRIMARY KEY} constraint. + +The above applies to DuckDB, which is often similar to Postgres. +ARTs are not balanced and the internal node size is modified to accommodate the data distribution, thus not requiring nodes to have fixed sizes, which would contain empty pointers. + +The internal node size however typically isn't \textit{entirely} variable, but often comes in four different sizes, facilitating traversal. +Each node stores the keys and pointers to the next node to follow for a given key. + +Common optimizations includes path compression, where particularly with long keys, some nodes could potentially only contain a single pointer. +These nodes can be collapsed and removed to make the tree smaller (i.e. the pointer in the parent is updated to directly point to the data instead). +If required by insertions, nodes can be added as needed (akin to B+ trees). + +For speed, ART require indexed values to be binary-comparable. +This can become an issue with data types such as signed numbers (extra bit for sign), strings (termination character) and nulls (must be mapped to a non-value sequence). +Thus, the data needs to be transformed to be able to index it and search the tree. diff --git a/semester4/dmdb/parts/03_systems/02_indexing/07_r-trees.tex b/semester4/dmdb/parts/03_systems/02_indexing/07_r-trees.tex index e69de29..57a5c25 100644 --- a/semester4/dmdb/parts/03_systems/02_indexing/07_r-trees.tex +++ b/semester4/dmdb/parts/03_systems/02_indexing/07_r-trees.tex @@ -0,0 +1,8 @@ +R-Trees are typically used for geospatial datasets, where searching for spatial relationship is expensive (requiring a full table scan). + +R-Trees create indexes on bounding boxes as opposed to points in 2- or 3D space. +We can locate the smallest bounding box where a tuple may lay by traversing the index and looking up the leaf to see if the point is there (or to look for a near neighbour). + +For more information, see \hlhref{https://en.wikipedia.org/wiki/R-tree}{Wikipedia} + +{\scriptsize (yes, they actually just linked to Wikipedia, so it probably isn't very exam-relevant)} diff --git a/semester4/dmdb/parts/03_systems/02_indexing/08_other-optimizations.tex b/semester4/dmdb/parts/03_systems/02_indexing/08_other-optimizations.tex index e69de29..0585bf4 100644 --- a/semester4/dmdb/parts/03_systems/02_indexing/08_other-optimizations.tex +++ b/semester4/dmdb/parts/03_systems/02_indexing/08_other-optimizations.tex @@ -0,0 +1,7 @@ +For groups of data that don't change often, we can compute statistics on insert, such as \texttt{sum}, \texttt{avg}, \texttt{min}, \texttt{max}, etc. +These aggregates can be used to check whether data needed is within that group. +This also enable computing (some) aggregates without reading the entire table and is commonly used in data mining and data cubes. + +A concept that is not used in DB engines, is count min sketch. It can be very useful in data streaming engines and large scale data processing. +What this does is it computes statistics on how many times a given value was seen in a list or stream of data and is a variation of the bloom filter, +but with several arrays and counting. The overestimation in count is bounded by the size in the matrix. diff --git a/semester4/dmdb/parts/04_query-processing/00_sorting/01_merge-sort.tex b/semester4/dmdb/parts/04_query-processing/00_sorting/01_merge-sort.tex index 8cb3a5e..ebb0b18 100644 --- a/semester4/dmdb/parts/04_query-processing/00_sorting/01_merge-sort.tex +++ b/semester4/dmdb/parts/04_query-processing/00_sorting/01_merge-sort.tex @@ -12,10 +12,19 @@ The cost for this is for the 1-page sorted runs (i.e. sorting the original pages The total cost is given by $2 \cdot M \cdot N$ I/O operations, with $M$ the number of passes / phases (i.e. how many merge steps there are) and $N$ being the number of frames. For Two-Way Merge Sort, the number of passes, $M$ is given by $M = \lceil \log_2(N) \rceil + 1$, -so the total cost to sort is $2N(\lceil \log_2 N \rceil + 1)$ +so the total cost to sort is \cost{$2N(\lceil \log_2 N \rceil + 1)$} The primary goal for sorting is of course efficiency. Primarily we want to reduce the amount of extra RAM used, the disk I/O (as that is slow) and trying to reduce random accesses in favour of sequential accesses. And of course, reducing CPU cost is still a concern. We can increase the number of frames to $B$ for the sorted runs, and then perform a $B - 1$-way merge. -In that case, the number of operations are given by $2N (1 + \lceil \log_{B - 1} \lceil N / B \rceil \rceil)$ +\coloredbox{orange}{I/O cost for merge sort}{ + (the $2$ comes from the fact that each pass reads and writes $N$ pages) + \[ + 2N \cdot P \text{ with } P = (1 + \lceil \log_{B - 1} \lceil N / B \rceil \rceil) \text{ the number of passes} + \] +} + +If for instance, we want to sort $N = 600$ pages with $B = 50$ buffers, pass $0$ creates $\ceil{600 / 50} = 12$ runs, +which fits in a single merge pass because $12 \leq B - 1$, which is derived from the logarithm. +Thus we have $2$ passes and the cost in I/Os is $2 \cdot 2 \text{ passes } \cdot 600$. diff --git a/semester4/dmdb/parts/04_query-processing/00_sorting/02_replacement-sort.tex b/semester4/dmdb/parts/04_query-processing/00_sorting/02_replacement-sort.tex index cee2032..6eabe24 100644 --- a/semester4/dmdb/parts/04_query-processing/00_sorting/02_replacement-sort.tex +++ b/semester4/dmdb/parts/04_query-processing/00_sorting/02_replacement-sort.tex @@ -24,7 +24,7 @@ It works as follows: Informally, it is searching for an always larger value once it has committed to a certain value. It picks the first value to be the lowest value in the current frames. -A new run is started if all values in the loaded frames are defered or used already. +A new run is started if all values in the loaded frames are deferred or used already. This means that after the sort run is complete, a merge is still needed. @@ -33,6 +33,8 @@ This means that after the sort run is complete, a merge is still needed. \bi{Worst case}: $\tco{B - 1}$ elements in a run (reversed data) -\bi{Average case}: $\tct{2B}$ elements in a run. In that case, the cost of sorting is $2N(1 + \lceil \log_{B - 1} \lceil N / 2B \rceil \rceil)$ +\bi{Average case}: $\tct{2B}$ elements in a run. -Thus, Quicksort is fatster, but longer runs often mean fewer passes, saving time. +In any case, the cost of sorting is \cost{$2N(1 + \lceil \log_{B - 1} \lceil N / E \rceil \rceil)$}, with $E$ the number of elements ($2B$ in the average case) + +Thus, Quicksort is faster, but longer runs often mean fewer passes, saving time. diff --git a/semester4/dmdb/parts/04_query-processing/00_sorting/03_sorting-with-b-trees.tex b/semester4/dmdb/parts/04_query-processing/00_sorting/03_sorting-with-b-trees.tex index 1ce6391..8074241 100644 --- a/semester4/dmdb/parts/04_query-processing/00_sorting/03_sorting-with-b-trees.tex +++ b/semester4/dmdb/parts/04_query-processing/00_sorting/03_sorting-with-b-trees.tex @@ -2,5 +2,5 @@ If a table has a B+ Index on the sorted column, we could retrieve the records in order by traversing the leaf pages. In case the B+ tree is clustered, this is a good idea, if it is not, then it could be very bad (due to random accesses). -The operating principle is very simple, abusing the linked lists in the leafs of the B+ tree. -If the data is not clustered, we have one I/O per record, which is bad. +The operating principle is very simple, abusing the linked lists in the leafs of the B+ tree, leading to each page being fetched just once, +instead of one page fetch (= I/O) per record, if the data is not clustered, we have one I/O per record, which is bad. diff --git a/semester4/dmdb/parts/04_query-processing/01_selection/00_intro.tex b/semester4/dmdb/parts/04_query-processing/01_selection/00_intro.tex index b3d10ca..1576b49 100644 --- a/semester4/dmdb/parts/04_query-processing/01_selection/00_intro.tex +++ b/semester4/dmdb/parts/04_query-processing/01_selection/00_intro.tex @@ -1,6 +1,6 @@ Two terms important here are \textit{logical selection}, which describes \bi{what} we want to select and \textit{physical selection}, which describes \bi{how} the algorithm or procedure works that actually retrieves, or filters, the data. -The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is the number of pages in each relation. +The options include an \textit{file scan}, where we scan the entire file and thus the I/O cost is \cost{$N$}, where $N$ is the number of pages in each relation. Alternatively, we can use \textit{index scan}, where we use an index to retrieve the matching rows. The cost then of course depends on the index used and if said index can even be used to generate the resulsts needed. We will cover that in more detail now. diff --git a/semester4/dmdb/parts/04_query-processing/01_selection/01_index-scan.tex b/semester4/dmdb/parts/04_query-processing/01_selection/01_index-scan.tex index f0bfb5d..71c6d57 100644 --- a/semester4/dmdb/parts/04_query-processing/01_selection/01_index-scan.tex +++ b/semester4/dmdb/parts/04_query-processing/01_selection/01_index-scan.tex @@ -6,7 +6,7 @@ \item \bi{Bitmap Index}: $\tco{\text{size of bitmap index}} + X$, {\color{red} $X$ depends on clustering again} \end{itemize} -The I/O cost for B+ Tree Index Scan is $\texttt{tree height} + \texttt{\# leaf pages} + \texttt{\# file pages}$ +The I/O cost for B+ Tree Index Scan is \cost{$\texttt{tree height} + \texttt{\#leaf pages} + \texttt{\#file pages}$} \inlineexample{Computation example} Given a relation $R$ with $N =$ one million records. There are 100 records on a page and we have a B+ Tree with the data entries $$ diff --git a/semester4/dmdb/parts/04_query-processing/02_projection/00_intro.tex b/semester4/dmdb/parts/04_query-processing/02_projection/00_intro.tex index 587b5a9..a400405 100644 --- a/semester4/dmdb/parts/04_query-processing/02_projection/00_intro.tex +++ b/semester4/dmdb/parts/04_query-processing/02_projection/00_intro.tex @@ -1,11 +1,6 @@ -\subsubsection{Duplicate Elimination} -This is done using either hashing or sorting. -The hashing approach hashes an element and checks in the existing map if this element exists already. If not, it is inserted, else it is skipped. -Finally the map is returned, or transformed and returned. +For a basic \texttt{SELECT A, B FROM R} query, we scan the file and for each tuple output $A, B$. +Obviously, that means that for $N$ tuples and ratio of included tuples of $Q$, we have \cost{$N + N \cdot Q$} I/O operations. -For large datasets, we may want to create a hash table of size $B$, then run multiple dedupe passes, on each subset where duplicates could be, -then return the union'd map. - -A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates. -However, this tends to be more expensive than the hash-based approach for low amounts of data, but for larger amounts, very similar. -The sort-based approach also has the benefit of producing sorted data, which may be useful for future operators. +For a \texttt{SELECT DISTINCT A, B FROM R} query, we scan the file and eliminate duplicates before outputting. +If this is also combined with a sort, we can prefer the sort-based approach and deduplicate after search, +so we are projecting, then sorting and finally deduplicating using the sort method. diff --git a/semester4/dmdb/parts/04_query-processing/02_projection/01_duplicate-elimination.tex b/semester4/dmdb/parts/04_query-processing/02_projection/01_duplicate-elimination.tex new file mode 100644 index 0000000..82cfe64 --- /dev/null +++ b/semester4/dmdb/parts/04_query-processing/02_projection/01_duplicate-elimination.tex @@ -0,0 +1,63 @@ +\subsubsection{Duplicate Elimination} +This is done using either hashing or sorting. + +\paragraph{Hash-based approach} +Assume we have $B$ memory buffers. +Then the hashing approach works as follows: +\begin{enumerate} + \item Project out attributes and splitting the input into $B - 1$ partitions using a hash function $h1$. + \item After that, all duplicates will be in the same partition. Since there most likely are further collisions, for each partition from (1), we do the following: + \begin{enumerate} + \item If the partition fits into memory, then we are done with this step + \item If not, we run step recursively run step 1 again on this partition + \end{enumerate} + \item We return all keys from the hash table +\end{enumerate} +A partition from step 2 fits into memory if $\frac{f \cdot T}{B - 1} < B$ (or approximately $B > \sqrt{f \cdot T}$), +where $T$ is the number of pages after the projection and $f$ is a \textit{fudge factor}, typically $f \approx 1.2$. + +\coloredbox{orange}{Cost of hash-based duplicate elimination}{ + $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{elim}(R')$, with + \[ + \texttt{elim}(R') = \sum_{T \in R'} \begin{cases} + \texttt{elim}(T) & \text{if } \frac{f \cdot L(T)}{B - 1} \geq B \\ + \texttt{length}(T) & \text{else} + \end{cases} + \] + where $R'$ is the relation $R$ after the projection. $\texttt{cnt}(R)$ is the number of pages and $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot Q$, + with $Q$ the ratio of kept and total attributes in the relation. +} + +\paragraph{Sort-based approach} +A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates. + +The cost here is $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cost}_\texttt{sort}(R') + \texttt{cnt}(R')$. +Broken down: +\begin{itemize} + \item $\texttt{cnt}(R)$ I/Os for the initial scan + \item first $\texttt{cnt}(R')$ I/Os for storing the projected $R$ (denoted $R'$) on disk + \item $\texttt{cost}_\texttt{sort}(R')$ I/Os to sort the remaining attributes (see \ref{sec:external-sort}) + \item second $\texttt{cnt}(R')$ I/Os to scan the sorted result and eliminate duplicates +\end{itemize} + + +There is also an optimized version that uses a modified sort algorithm that during sorting projects out the attributes, with cost +\cost{$\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cnt}(R')$}. The first count is to read, sort and project; second count is to store the result to disk; +third count is to eliminate duplicates. + + +\paragraph{Comparison} +The sorting approach has the benefit of better handling data skew and that the result is sorted. +The I/O costs are comparable if $B$ is large, both algorithms need 2 passes over the data. +For low numbers of records, the hash-based approach is likely cheaper + + +\paragraph{Index-based approach} +Here, we only scan the index, and we don't visit the relation. The projection attributes need to be a subset of the index attributes. +We can project distinct on $A$ using a B+ tree on $A, B$. +We then only apply the projection algorithm only to the data entries. + +Then, if an ordered index contains all projection attributes as prefix of the search key, +we first retrieve index data entries in order, discard unwanted fields and compare adjacent entries to eliminate duplicates (if needed). + +If an index is smaller than a relation, then it is faster to scan the index. diff --git a/semester4/dmdb/parts/04_query-processing/02_projection/01_set-operations.tex b/semester4/dmdb/parts/04_query-processing/02_projection/02_set-operations.tex similarity index 100% rename from semester4/dmdb/parts/04_query-processing/02_projection/01_set-operations.tex rename to semester4/dmdb/parts/04_query-processing/02_projection/02_set-operations.tex diff --git a/semester4/dmdb/parts/04_query-processing/03_aggregation.tex b/semester4/dmdb/parts/04_query-processing/03_aggregation.tex index 1c03d86..d290a86 100644 --- a/semester4/dmdb/parts/04_query-processing/03_aggregation.tex +++ b/semester4/dmdb/parts/04_query-processing/03_aggregation.tex @@ -1,10 +1,10 @@ And again, we have the option to do this via sorting or hashing: \begin{itemize} \item \bi{Sorting}: Sort on the attribute. Then scan the sorted tuples, computing running aggregate (such as max/min, average, etc). - When a new group is encountered, then we output the aggregate. + When a new group is encountered, then we output the aggregate. The cost is \cost{$\texttt{cost}_\texttt{sort}(R) + \texttt{cnt}(R)$}. Alternatively, we can already compute the aggregates on the last step of sorting (the merge) to save some time. - The limiting factor then becomes the sorting. + The limiting factor then becomes the sorting and the cost is \cost{$\texttt{cost}_\texttt{sort}(R)$} \item \bi{Hashing}: We hash the attribute and now each hash table entry is a group of all records with this value of the attribute. - For each one of these groups, we compute the aggregate. - If the table is too large for memory, we use a two-step approach, as before + For each one of these groups, we compute the aggregate. The cost is \cost{$\texttt{cnt}(R)$} + If the table is too large for memory, we use a two-step approach, as before. The cost is then very similar \cost{$\texttt{cnt}(R) + B$} for $B$ tables. \end{itemize} diff --git a/semester4/dmdb/parts/04_query-processing/04_join/00_simple-join.tex b/semester4/dmdb/parts/04_query-processing/04_join/00_simple-join.tex index e0e49ba..718e49c 100644 --- a/semester4/dmdb/parts/04_query-processing/04_join/00_simple-join.tex +++ b/semester4/dmdb/parts/04_query-processing/04_join/00_simple-join.tex @@ -13,4 +13,6 @@ \EndProcedure \end{algorithmic} \end{algorithm} -Here, the max output size is $T_R \cdot T_S$, where $T_X$ is the number of tuples for realtion $X$. +Here, the max output size is $T_R \cdot T_S$, where $T_X$ is the number of tuples for relation $X$. + +The cost is obviously \cost{$\texttt{cnt}(R) \cdot \texttt{cnt}(S)$}, with $\texttt{cnt}(X)$ the number of tuples in $X$. diff --git a/semester4/dmdb/parts/04_query-processing/04_join/01_nested-loops.tex b/semester4/dmdb/parts/04_query-processing/04_join/01_nested-loops.tex index c018ea1..e4ed545 100644 --- a/semester4/dmdb/parts/04_query-processing/04_join/01_nested-loops.tex +++ b/semester4/dmdb/parts/04_query-processing/04_join/01_nested-loops.tex @@ -13,5 +13,5 @@ \end{algorithmic} \end{algorithm} -The cost here is $P_R + P_R \cdot P_S$ I/Os, where $P_X$ is the number of pages in the relation $X$. +The cost here is \cost{$P_R + P_R \cdot P_S$ I/Os}, where $P_X$ is the number of pages in the relation $X$. Furthermore, the DBMS should put the smaller relation on the outer loop to improve performance. diff --git a/semester4/dmdb/parts/04_query-processing/04_join/02_index-nested-loops.tex b/semester4/dmdb/parts/04_query-processing/04_join/02_index-nested-loops.tex index a85c83c..60a990f 100644 --- a/semester4/dmdb/parts/04_query-processing/04_join/02_index-nested-loops.tex +++ b/semester4/dmdb/parts/04_query-processing/04_join/02_index-nested-loops.tex @@ -13,7 +13,7 @@ \end{algorithmic} \end{algorithm} -The cost here is $P_R + T_R \cdot C(I_S)$ I/Os, with $P_X$ the number of pages in the relation $X$, $T_X$ the number of tuples in said relation +The cost here is \cost{$P_R + T_R \cdot C(I_S)$ I/Os}, with $P_X$ the number of pages in the relation $X$, $T_X$ the number of tuples in said relation and $C(I_X)$ is the cost to probe the index $I_X$ of relation $X$ and depends on the clustering of data and type of index. This operation needs three buffer frames. $1$ for $R$, $1$ for output and $1$ to traverse $I_S$ diff --git a/semester4/dmdb/parts/04_query-processing/04_join/03_block-nested-loops.tex b/semester4/dmdb/parts/04_query-processing/04_join/03_block-nested-loops.tex index 1170aaf..a8cefea 100644 --- a/semester4/dmdb/parts/04_query-processing/04_join/03_block-nested-loops.tex +++ b/semester4/dmdb/parts/04_query-processing/04_join/03_block-nested-loops.tex @@ -16,7 +16,7 @@ We have $B$ buffer frames, of which one is reserved for the pages of $S$ and one is reserved for the output, thus the $B - 2$ blocks. Each of these blocks houses $P_R / (B - 2))$ pages of $R$, reducing search costs by only scanning $S$ once for every block. -The cost here is $P_R + (P_R / (B - 2) \cdot P_S$ I/Os. +The cost here is \cost{$P_R + (P_R / (B - 2) \cdot P_S$ I/Os}. If however, $R$ fits into memory, we have cost $P_R + P_S$ I/Os. We can reduce that further by building as second step (right after the first for loop) a in-memory hash table for the tuples in the $R$ pages block. This reduces the cost of comparisons, however, we need $(B - 2) \cdot f$ memory now. diff --git a/semester4/dmdb/parts/04_query-processing/04_join/04_block-index-nested-loops.tex b/semester4/dmdb/parts/04_query-processing/04_join/04_block-index-nested-loops.tex index cc2d981..364dfc3 100644 --- a/semester4/dmdb/parts/04_query-processing/04_join/04_block-index-nested-loops.tex +++ b/semester4/dmdb/parts/04_query-processing/04_join/04_block-index-nested-loops.tex @@ -14,7 +14,7 @@ \end{algorithmic} \end{algorithm} -The cost here is $P_R + T_R \cdot C(I_S)$ I/Os. +The cost here is \cost{$P_R + T_R \cdot C(I_S)$ I/Os}. We sort the tuples in each buffer frame on the predicate key, which leads to tuples in $R$ with the same key only requiring a single index search. In addition, tuples with ``similar'' values in $R$ will likely match with keys on the same pages of $I_S$, which will already be in the buffer pool. diff --git a/semester4/dmdb/parts/04_query-processing/04_join/05_sort-merge-join.tex b/semester4/dmdb/parts/04_query-processing/04_join/05_sort-merge-join.tex index 854c223..359696b 100644 --- a/semester4/dmdb/parts/04_query-processing/04_join/05_sort-merge-join.tex +++ b/semester4/dmdb/parts/04_query-processing/04_join/05_sort-merge-join.tex @@ -2,11 +2,11 @@ We first sort $R$ and $S$ on the join attributes with external merge sort (see Section~\ref{sec:external-sort}). We then read the sorted files and merge. This is of course not great for performance, but if the relations are already sorted, we can skip the first step. -The cost (if there are \bi{no duplicates}) is $\texttt{Sort}(S) + \texttt{Sort}(R) + P_R + P_S$ I/Os. +The cost (if there are \bi{no duplicates}) is \cost{$\texttt{Sort}(S) + \texttt{Sort}(R) + P_R + P_S$ I/Os}. If there are duplicates (i.e. the keys are not unique, say we are joining on a column that doesn't have the UNIQUE constraint) then we need to move the pointer for the join back on the second relation to make sure we don't miss any tuples. -The cost now of course isn't linear anymore and worst case quadratic, $\tco{\texttt{Sort}(S) + \texttt{Sort}(R) + P_R \cdot P_S}$. +The cost now of course isn't linear anymore and worst case quadratic, \cost{$\tco{\texttt{Sort}(S) + \texttt{Sort}(R) + P_R \cdot P_S}$}. Compared to BNLJ, its performance doesn't depend on the amount of memory available, but tends to be a tad slower than BNLJ, if BNLJ is provided with ample memory. diff --git a/semester4/dmdb/parts/04_query-processing/04_join/06_optimized-smj.tex b/semester4/dmdb/parts/04_query-processing/04_join/06_optimized-smj.tex index 76bcb36..e44dc68 100644 --- a/semester4/dmdb/parts/04_query-processing/04_join/06_optimized-smj.tex +++ b/semester4/dmdb/parts/04_query-processing/04_join/06_optimized-smj.tex @@ -1,10 +1,10 @@ \subsubsection{Optimized Sort Merge Join ((O)SMJ)} In this section, we are exploring how we can optimize sort and merge with extra memory. -Sorting can be optimized using replacement sort, the output is $\frac{P_R + P_S}{2B}$ sorted runs of length $2B$. +Sorting can be optimized using replacement sort, the output is \cost{$\frac{P_R + P_S}{2B}$} sorted runs of length $2B$. That of course means that the logical optimization is to do a $K$-way merge to merge the sorted runs and join the relations at the same time. If we have $B$ frames, so we can do a $B-1$ way merge, otherwise, we merge runs from each relation separately to reduce the number of runs until we have less than $B$ sorted runs, then we perform the join. -The cost drops to $3 \cdot (P_R + P_S)$ I/Os, but we need memory such that $B^2 \geq \max\{ P_R, P_S \}$ +The cost drops to \cost{$3 \cdot (P_R + P_S)$ I/Os}, but we need memory such that $B^2 \geq \max\{ P_R, P_S \}$ diff --git a/semester4/dmdb/parts/04_query-processing/04_join/08_partitioned-hash-join.tex b/semester4/dmdb/parts/04_query-processing/04_join/08_partitioned-hash-join.tex index a7776f9..5e849e4 100644 --- a/semester4/dmdb/parts/04_query-processing/04_join/08_partitioned-hash-join.tex +++ b/semester4/dmdb/parts/04_query-processing/04_join/08_partitioned-hash-join.tex @@ -20,4 +20,4 @@ The benefit of this approach is that in the third step, we are only operating on What we ideally do in the above algorithm is to do \textit{recursive partitioning}, i.e. as already \textit{somewhat} described, we partition each partition again, if it has more than one page. This further reduces access times. -The cost for this is $3 \cdot (P_S + P_R)$, thus both OSMJ and PHJ are very similar in performance. +The cost for this is \cost{$3 \cdot (P_S + P_R)$}, thus both OSMJ and PHJ are very similar in performance. diff --git a/semester4/dmdb/parts/04_query-processing/main.tex b/semester4/dmdb/parts/04_query-processing/main.tex index 63b54f2..6a00f85 100644 --- a/semester4/dmdb/parts/04_query-processing/main.tex +++ b/semester4/dmdb/parts/04_query-processing/main.tex @@ -1,4 +1,7 @@ +\newcommand\cost[1]{\shade{orange}{#1}} \section{Query Processing} +In this section, knowing the time complexities by heart is very important for the exam. +They are highlighted \cost{like this}. \subsection{Sorting} \input{parts/04_query-processing/00_sorting/00_intro.tex} @@ -7,15 +10,18 @@ \input{parts/04_query-processing/00_sorting/03_sorting-with-b-trees.tex} % \input{parts/04_query-processing/00_sorting/} +\newpage \subsection{Selection} \input{parts/04_query-processing/01_selection/00_intro.tex} \input{parts/04_query-processing/01_selection/01_index-scan.tex} \input{parts/04_query-processing/01_selection/02_index-matching.tex} % \input{parts/04_query-processing/01_selection/} +\newpage \subsection{Projection} \input{parts/04_query-processing/02_projection/00_intro.tex} -\input{parts/04_query-processing/02_projection/01_set-operations.tex} +\input{parts/04_query-processing/02_projection/01_duplicate-elimination.tex} +\input{parts/04_query-processing/02_projection/02_set-operations.tex} % \input{parts/04_query-processing/02_projection/} \subsection{Aggregation}