diff --git a/semester4/dmdb/assets/db-system-memory.png b/semester4/dmdb/assets/db-system-memory.png new file mode 100644 index 0000000..b32eb8f Binary files /dev/null and b/semester4/dmdb/assets/db-system-memory.png differ diff --git a/semester4/dmdb/assets/join-ordering-example.png b/semester4/dmdb/assets/join-ordering-example.png new file mode 100644 index 0000000..cff79ae Binary files /dev/null and b/semester4/dmdb/assets/join-ordering-example.png differ diff --git a/semester4/dmdb/data-modelling-databases-summary.pdf b/semester4/dmdb/data-modelling-databases-summary.pdf index 73c7ed3..50e2226 100644 Binary files a/semester4/dmdb/data-modelling-databases-summary.pdf and b/semester4/dmdb/data-modelling-databases-summary.pdf differ diff --git a/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/01_rewriting-rules.tex b/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/01_rewriting-rules.tex index 847b1d1..e7aed86 100644 --- a/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/01_rewriting-rules.tex +++ b/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/01_rewriting-rules.tex @@ -18,4 +18,70 @@ The following rules were discussed in the lectures: \[ \Pi_{t_1}(\Pi_{t_2}(E)) = \Pi_{t_1}(E) \] + This avoids going over the data twice + \item Selections can be combined with cartesian products and theta joins + \[ + \sigma_\theta(E_1 \times E_2) = E_1 \bowtie_\theta E_2 + \qquad \sigma_{\theta_1}(E_1 \bowtie_{\theta_2} E_2) = E_1 \bowtie_{\theta_1 \land \theta_2} E_2 + \] + This combines the join and selection (thus avoiding having to go over the data twice) + \item Theta-join operations (and natural joins) are commutative. + \[ + E_1 \bowtie_\theta E_2 = E_2 \bowtie_\theta E_1 + \] + This is useful so we ccan choose as the smaller table as the outer table or as the inner one the one with an index. + \item Natural join operations are associative, so are theta joins (with restrictions) + \[ + (E_1 \bowtie E_2) \bowtie E_3 = E_1 \bowtie (E_2 \bowtie E_3) + \qquad (E_1 \bowtie_{\theta_1} E_2) \bowtie_{\theta_2 \land \theta_3} = E_1 \bowtie_{\theta_1 \land \theta_3} (E_2 \bowtie_{\theta_2} E_3) + \] + This allows for joins to be performed in different orders, allowing us to do the most selective first (fewer rows) + \item Pushdown selection: + \[ + \sigma_\theta (E_1 \bowtie E_2) = \sigma_\theta(E_1) \bowtie E_2 + \] + This allows us to first remove the data that is not needed before doing the more expensive operations + \item Projections distribute over theta joins (applies if $\theta$ involves only attributes from $L_1 \cup L_2$ of course) + \[ + \Pi_{L_1 \cup L_2}(E_1 \bowtie_\theta E_2) = \Pi_{L_1}(E_1) \bowtie_\theta \Pi_{L_2}(E_2) + \] + This allows the table width to be reduced before the join to reduce \bi{data movement} + \item Union and Intersection are commutative + \[ + E_1 \cup E_2 = E_2 \cup E_1 \qquad E_1 \cap E_2 = E_2 \cap E_1 + \] + This allows us to perform one side before the other, depending on resource constraints + \item Set union and intersection are associative + \[ + (E_1 \cup E_2) \cup E_3 = E_1 \cup (E_2 \cup E_3) + \qquad (E_1 \cap E_2) \cap E_3 = E_1 \cap (E_2 \cap E_3) + \] + This allows us to change the order in which operations are done and how the operator tree is executed. + \item The selection operation distributes over $\cup, \cap$ and $-$ + \[ + \sigma_\theta(E_1 - E_2) = \sigma_\theta(E_1) - \sigma_\theta(E_2) + \qquad \sigma_\theta(E_1 \cup E_2) = \sigma_\theta(E_1) \cup \sigma_\theta(E_2) + \qquad \sigma_\theta(E_1 \cap E_2) = \sigma_\theta(E_1) \cap \sigma_\theta(E_2) + \] + In addition, we have + \[ + \sigma_\theta(E_1 - E_2) = \sigma_\theta(E_1) - E_2 + \qquad \sigma_\theta(E_1 \cap E_2) = \sigma_\theta(E_1) \cap E_2 + \qquad \text{not for } \cup + \] + This allows us to remove unneeded tuples before performing the next operation, + which may be involving comparisons + \item The projection operation distributes over union + \[ + \Pi_L(E_1 \cup E_2) = \Pi_L(E_1) \cup \Pi_L(E_2) + \] + This allows us to remove columns before the union, requiring less \bi{data movement} \end{enumerate} + +\begin{figure}[h!] + \begin{center} + \includegraphics[width=1\textwidth]{assets/join-ordering-example.png} + \end{center} + \caption{Example application of the rewriting rules. + (Figure from lecture slides 08, slide 21)} +\end{figure} diff --git a/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/02_execution-model.tex b/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/02_execution-model.tex new file mode 100644 index 0000000..78da200 --- /dev/null +++ b/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/02_execution-model.tex @@ -0,0 +1,31 @@ +\subsubsection{Execution Model} +There are a number of ways in which the execution can happen + + +\paragraph{Materialization Model} +Here, each operator processes its entire input all at once and then emits its output all at once. + +This is ideal, if the intermediate results are not much larger than the final results, as very little overhead is created. +The efficiency of this approach however is more limited the larger the intermediate results are. + + +\paragraph{Iterator Model} +Here, each operator is an iterator, it takes as input a set of streams of tuples (each of which provides a \texttt{next()} function) and outputs a stream of tuples, +that can in turn again be accessed using a \texttt{next()} function. +To get a result, we run \texttt{root.next()} again and again, until no more data is available to be pulled in. + +This is a \bi{Volcano Mode}, data flows from bottom to top. + +This method has the benefits of there being a single interface for all operators, thus we don't (need to) know what the previous operator did. +Furthermore, iterators are easy to implement and there are little to no memory overheads and the whole thing can be pipelined, parallelized and distributed. + +On the other hand, method calls introduce a large overhead and suffer from poor instruction cache locality. + + +\paragraph{Vectorization Model} +This method is similar to the iterator model, but instead of a single tuple, the \texttt{next()} invocation returns a batch of tuples. + +We can also use vector instructions on the batch of tuple, which reduces overhead by requiring fewer function and instruction calls. + +Of course, the limitation here is that the hardware needs to support vector instructions, though nowadays that is hardly an issue, +since AVX instructions are supported on CPUs as old as the Sandy Bridge Architecture (e.g. Intel Core i7-2600K) for Intel (2011) and Bulldozer (e.g. AMD FX-8100) for AMD (2011). diff --git a/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/03_cost-model.tex b/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/03_cost-model.tex new file mode 100644 index 0000000..1fe195d --- /dev/null +++ b/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/03_cost-model.tex @@ -0,0 +1,66 @@ +\subsubsection{Cost Model} +A cost model is used to \bi{estimate} the computation cost of a given physical plan, without actually running it. + +Since the ccost of running each operator depends on the input and output size of it, we can easily convert this into cost. + +Thus, to estimate the cost, we need to know the runtime of an operator and the number of inputs and outputs. +The first of these two aspects is of course very easily attainable, whereas the \bi{cardinality estimation}, +i.e. the estimation of the number of tuples an operator returns, is very hard. + +To make this a bit easier, commonly a \bi{selectivity} or \bi{reduction factor} is used, which tells us how much of the input an operator returns. + +Below some reduction factor estimators (we assume that the values are uniformly distributed) + +\begin{tables}{lllll}{Predicate & Reduction Factor & Predicate Example} + \texttt{col = val} & \texttt{1 / \#UniqueValues(col)} & \texttt{col = 2} \\ + \texttt{col > val} & $|\max(\texttt{col}) - \texttt{val}| \div (\max(\texttt{col}) - \min(\texttt{col}))$ & \texttt{col > 5} \\ + \texttt{col < val} & $|\texttt{val} - \min(\texttt{col})| \div (\max(\texttt{col}) - \min(\texttt{col}))$ & \texttt{col < 5} \\ + \texttt{col1 = col2} & $1 \div \max(\#\texttt{UniqueValues(col1)}, \#\texttt{UniqueValues(col2)})$ & \texttt{t.name = s.name} \\ +\end{tables} + +This table shows the overall concept for the cardinality estimation. +For each predication, a reduction factor is determined, as each predicate ``filters'' out some tuples. + +Thus, the resulting cardinality is $\max(\#\text{tuples}) \cdot \prod_{r \in RF} r$, where $RF$ is the \gls{bag} of all reduction factors. + +In all of the above estimators, we assumed that the values are uniformly distributed. +This of course is not always true --- there are many cases where data is heavily skewed --- in which cases, histograms are used. + +Two types of histograms are commonly used: +\begin{itemize} + \item \bi{Equi-Width}: Here, the heights (tuple counts) of the buckets differ, as we fix the values for each bucket. + This is useful for identifying hot-spots. + \item \bi{Equi-Depth} (or Equi-Height): Here, the width differs, because we want all buckets to contain the same number of tuples. + The size of a range helps with cardinality estimates + \item \bi{Singleton} (or Frequency): Plots how often a single item appears in a table. + This is very useful to compute the selectivity of queries +\end{itemize} + +\newpage +To enable quick processing of the cost function, the following metadata, or statistics, are typically stored: +\begin{multicols}{2} + \bi{Table statistics} + \begin{itemize} + \item Number of rows + \item Number of blocks + \item Average row length + \end{itemize} + \bi{Column statistics} + \begin{itemize} + \item Number of distinct values (NDV) in columns + \item Number of nulls in column + \item Data distribution (histogram) + \end{itemize} + \bi{Extended statistics} + \begin{itemize} + \item Index statistics + \item Number of leaf blocks + \item Levels + \item Clustering factor + \end{itemize} + \bi{System statistics} + \begin{itemize} + \item I/O performance and utilization + \item CPU performance and utilization + \end{itemize} +\end{multicols} diff --git a/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/04_search.tex b/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/04_search.tex new file mode 100644 index 0000000..cc17e8c --- /dev/null +++ b/semester4/dmdb/parts/03_systems/00_query-optimization/01_search-space/04_search.tex @@ -0,0 +1,36 @@ +\subsubsection{Search} +We now know how to generate a set of semantically equivalent physical plans and how to estimate the cost for each of them. +The question that remains is how to efficiently find the best plan. + +Since real-world queries can be very complex, searching for the best physical plan can be hard. +There are a few options, but we only covered a simple approach. + +We can constrain the search space by only considering \bi{left-deep join trees}, i.e. trees that have only leafs on the right hand side of each join statement. +The rationale behind this is that many of these are fully pipelined plans and no intermediate results have to be written to temporary files. +This is the concept \texttt{System R}, the first relational database used. + +An example for a non-pipelined left-deep join tree is one in volving sort operations. +This brings the complexity down to \tco{n!}, where $n$ is the number of realations in the join. + +\begin{itemize} + \item Enumerate join orders (different left deep trees) + \item Enumerate plans for each operator (hash join, sort merge join, nested loop join, \dots) + \item Enumerate access methods for each operator (B+-Tree, hash table, sequential scan, \dots) +\end{itemize} + +This can be achieved using dynamic programming. + +We then find the best 1-relation plan, then the best 2-relation plan by finding ways to join a relation with the best 1-relation plan, etc. + +While this is still slow, it's faster than enumerating all possible join trees. + +Later, systems started introducing heuristics to reduce the number of choices that must be made in a cost-based fashion. + +Heuristic optimization transforms the query-tree using a set of rules typically improve execution performance: +\begin{itemize} + \item Perform selection early (reduces the number of tuples to process) + \item Perform projection early (reduces the number of attributes to process) + \item Perform most restrictive selection and join operations before other similar operations +\end{itemize} + +Some systems only use heuristics, while others combine them with partial cost-based optimization. diff --git a/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/00_intro.tex b/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/00_intro.tex new file mode 100644 index 0000000..82b041f --- /dev/null +++ b/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/00_intro.tex @@ -0,0 +1,18 @@ +\subsubsection{Storage Management} +To make database engines easy to use, some abstractions have to happen. +\begin{figure}[h!] + \begin{center} + \includegraphics[width=0.7\textwidth]{assets/db-system-memory.png} + \end{center} + \caption{Overview of a typical abstraction for a database system + (Figure from lecture slides 09, slide 3)} +\end{figure} + +In the design of database engines it is commonly assumed that the data is stored on spinning disks, i.e. HDDs, with a high latency for seeks (random accesses) +and that most of the DB is not in memory. + +Very modern database systems may assume that a higher percentage of the DB is stored in memory due to the higher quantity of memory available, +as well as lower access latencies due to the more readily available SSDs and high-speed NASes. + +Some database systems operate similarly to operating systems in certain aspects, in that they manage their own processes and all of the memory +to get the absolute maximum performance out of the systems. diff --git a/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/01_blocks-pages.tex b/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/01_blocks-pages.tex new file mode 100644 index 0000000..e69de29 diff --git a/semester4/dmdb/parts/03_systems/main.tex b/semester4/dmdb/parts/03_systems/main.tex index 276a34d..562bd33 100644 --- a/semester4/dmdb/parts/03_systems/main.tex +++ b/semester4/dmdb/parts/03_systems/main.tex @@ -4,5 +4,14 @@ \input{parts/03_systems/00_query-optimization/00_intro.tex} \input{parts/03_systems/00_query-optimization/01_search-space/00_basics.tex} \input{parts/03_systems/00_query-optimization/01_search-space/01_rewriting-rules.tex} -% \input{parts/03_systems/00_query-optimization/01_search-space/} -% \input{parts/03_systems/00_query-optimization/} +\input{parts/03_systems/00_query-optimization/01_search-space/02_execution-model.tex} +\input{parts/03_systems/00_query-optimization/01_search-space/03_cost-model.tex} +\input{parts/03_systems/00_query-optimization/01_search-space/04_search.tex} + +\newpage +\subsection{Database Engines} +\input{parts/03_systems/01_database-engines/00_memory/00_intro.tex} +% \input{parts/03_systems/01_database-engines/00_memory/} +% \input{parts/03_systems/01_database-engines/} + +% \input{parts/03_systems/}