mirror of
https://github.com/janishutz/eth-summaries.git
synced 2026-07-27 21:29:09 +02:00
[DMDB] Query optimization, basics of memory management
This commit is contained in:
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 52 KiB |
Binary file not shown.
+66
@@ -18,4 +18,70 @@ The following rules were discussed in the lectures:
|
||||
\[
|
||||
\Pi_{t_1}(\Pi_{t_2}(E)) = \Pi_{t_1}(E)
|
||||
\]
|
||||
This avoids going over the data twice
|
||||
\item Selections can be combined with cartesian products and theta joins
|
||||
\[
|
||||
\sigma_\theta(E_1 \times E_2) = E_1 \bowtie_\theta E_2
|
||||
\qquad \sigma_{\theta_1}(E_1 \bowtie_{\theta_2} E_2) = E_1 \bowtie_{\theta_1 \land \theta_2} E_2
|
||||
\]
|
||||
This combines the join and selection (thus avoiding having to go over the data twice)
|
||||
\item Theta-join operations (and natural joins) are commutative.
|
||||
\[
|
||||
E_1 \bowtie_\theta E_2 = E_2 \bowtie_\theta E_1
|
||||
\]
|
||||
This is useful so we ccan choose as the smaller table as the outer table or as the inner one the one with an index.
|
||||
\item Natural join operations are associative, so are theta joins (with restrictions)
|
||||
\[
|
||||
(E_1 \bowtie E_2) \bowtie E_3 = E_1 \bowtie (E_2 \bowtie E_3)
|
||||
\qquad (E_1 \bowtie_{\theta_1} E_2) \bowtie_{\theta_2 \land \theta_3} = E_1 \bowtie_{\theta_1 \land \theta_3} (E_2 \bowtie_{\theta_2} E_3)
|
||||
\]
|
||||
This allows for joins to be performed in different orders, allowing us to do the most selective first (fewer rows)
|
||||
\item Pushdown selection:
|
||||
\[
|
||||
\sigma_\theta (E_1 \bowtie E_2) = \sigma_\theta(E_1) \bowtie E_2
|
||||
\]
|
||||
This allows us to first remove the data that is not needed before doing the more expensive operations
|
||||
\item Projections distribute over theta joins (applies if $\theta$ involves only attributes from $L_1 \cup L_2$ of course)
|
||||
\[
|
||||
\Pi_{L_1 \cup L_2}(E_1 \bowtie_\theta E_2) = \Pi_{L_1}(E_1) \bowtie_\theta \Pi_{L_2}(E_2)
|
||||
\]
|
||||
This allows the table width to be reduced before the join to reduce \bi{data movement}
|
||||
\item Union and Intersection are commutative
|
||||
\[
|
||||
E_1 \cup E_2 = E_2 \cup E_1 \qquad E_1 \cap E_2 = E_2 \cap E_1
|
||||
\]
|
||||
This allows us to perform one side before the other, depending on resource constraints
|
||||
\item Set union and intersection are associative
|
||||
\[
|
||||
(E_1 \cup E_2) \cup E_3 = E_1 \cup (E_2 \cup E_3)
|
||||
\qquad (E_1 \cap E_2) \cap E_3 = E_1 \cap (E_2 \cap E_3)
|
||||
\]
|
||||
This allows us to change the order in which operations are done and how the operator tree is executed.
|
||||
\item The selection operation distributes over $\cup, \cap$ and $-$
|
||||
\[
|
||||
\sigma_\theta(E_1 - E_2) = \sigma_\theta(E_1) - \sigma_\theta(E_2)
|
||||
\qquad \sigma_\theta(E_1 \cup E_2) = \sigma_\theta(E_1) \cup \sigma_\theta(E_2)
|
||||
\qquad \sigma_\theta(E_1 \cap E_2) = \sigma_\theta(E_1) \cap \sigma_\theta(E_2)
|
||||
\]
|
||||
In addition, we have
|
||||
\[
|
||||
\sigma_\theta(E_1 - E_2) = \sigma_\theta(E_1) - E_2
|
||||
\qquad \sigma_\theta(E_1 \cap E_2) = \sigma_\theta(E_1) \cap E_2
|
||||
\qquad \text{not for } \cup
|
||||
\]
|
||||
This allows us to remove unneeded tuples before performing the next operation,
|
||||
which may be involving comparisons
|
||||
\item The projection operation distributes over union
|
||||
\[
|
||||
\Pi_L(E_1 \cup E_2) = \Pi_L(E_1) \cup \Pi_L(E_2)
|
||||
\]
|
||||
This allows us to remove columns before the union, requiring less \bi{data movement}
|
||||
\end{enumerate}
|
||||
|
||||
\begin{figure}[h!]
|
||||
\begin{center}
|
||||
\includegraphics[width=1\textwidth]{assets/join-ordering-example.png}
|
||||
\end{center}
|
||||
\caption{Example application of the rewriting rules.
|
||||
(Figure from lecture slides 08, slide 21)}
|
||||
\end{figure}
|
||||
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
\subsubsection{Execution Model}
|
||||
There are a number of ways in which the execution can happen
|
||||
|
||||
|
||||
\paragraph{Materialization Model}
|
||||
Here, each operator processes its entire input all at once and then emits its output all at once.
|
||||
|
||||
This is ideal, if the intermediate results are not much larger than the final results, as very little overhead is created.
|
||||
The efficiency of this approach however is more limited the larger the intermediate results are.
|
||||
|
||||
|
||||
\paragraph{Iterator Model}
|
||||
Here, each operator is an iterator, it takes as input a set of streams of tuples (each of which provides a \texttt{next()} function) and outputs a stream of tuples,
|
||||
that can in turn again be accessed using a \texttt{next()} function.
|
||||
To get a result, we run \texttt{root.next()} again and again, until no more data is available to be pulled in.
|
||||
|
||||
This is a \bi{Volcano Mode}, data flows from bottom to top.
|
||||
|
||||
This method has the benefits of there being a single interface for all operators, thus we don't (need to) know what the previous operator did.
|
||||
Furthermore, iterators are easy to implement and there are little to no memory overheads and the whole thing can be pipelined, parallelized and distributed.
|
||||
|
||||
On the other hand, method calls introduce a large overhead and suffer from poor instruction cache locality.
|
||||
|
||||
|
||||
\paragraph{Vectorization Model}
|
||||
This method is similar to the iterator model, but instead of a single tuple, the \texttt{next()} invocation returns a batch of tuples.
|
||||
|
||||
We can also use vector instructions on the batch of tuple, which reduces overhead by requiring fewer function and instruction calls.
|
||||
|
||||
Of course, the limitation here is that the hardware needs to support vector instructions, though nowadays that is hardly an issue,
|
||||
since AVX instructions are supported on CPUs as old as the Sandy Bridge Architecture (e.g. Intel Core i7-2600K) for Intel (2011) and Bulldozer (e.g. AMD FX-8100) for AMD (2011).
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
\subsubsection{Cost Model}
|
||||
A cost model is used to \bi{estimate} the computation cost of a given physical plan, without actually running it.
|
||||
|
||||
Since the ccost of running each operator depends on the input and output size of it, we can easily convert this into cost.
|
||||
|
||||
Thus, to estimate the cost, we need to know the runtime of an operator and the number of inputs and outputs.
|
||||
The first of these two aspects is of course very easily attainable, whereas the \bi{cardinality estimation},
|
||||
i.e. the estimation of the number of tuples an operator returns, is very hard.
|
||||
|
||||
To make this a bit easier, commonly a \bi{selectivity} or \bi{reduction factor} is used, which tells us how much of the input an operator returns.
|
||||
|
||||
Below some reduction factor estimators (we assume that the values are uniformly distributed)
|
||||
|
||||
\begin{tables}{lllll}{Predicate & Reduction Factor & Predicate Example}
|
||||
\texttt{col = val} & \texttt{1 / \#UniqueValues(col)} & \texttt{col = 2} \\
|
||||
\texttt{col > val} & $|\max(\texttt{col}) - \texttt{val}| \div (\max(\texttt{col}) - \min(\texttt{col}))$ & \texttt{col > 5} \\
|
||||
\texttt{col < val} & $|\texttt{val} - \min(\texttt{col})| \div (\max(\texttt{col}) - \min(\texttt{col}))$ & \texttt{col < 5} \\
|
||||
\texttt{col1 = col2} & $1 \div \max(\#\texttt{UniqueValues(col1)}, \#\texttt{UniqueValues(col2)})$ & \texttt{t.name = s.name} \\
|
||||
\end{tables}
|
||||
|
||||
This table shows the overall concept for the cardinality estimation.
|
||||
For each predication, a reduction factor is determined, as each predicate ``filters'' out some tuples.
|
||||
|
||||
Thus, the resulting cardinality is $\max(\#\text{tuples}) \cdot \prod_{r \in RF} r$, where $RF$ is the \gls{bag} of all reduction factors.
|
||||
|
||||
In all of the above estimators, we assumed that the values are uniformly distributed.
|
||||
This of course is not always true --- there are many cases where data is heavily skewed --- in which cases, histograms are used.
|
||||
|
||||
Two types of histograms are commonly used:
|
||||
\begin{itemize}
|
||||
\item \bi{Equi-Width}: Here, the heights (tuple counts) of the buckets differ, as we fix the values for each bucket.
|
||||
This is useful for identifying hot-spots.
|
||||
\item \bi{Equi-Depth} (or Equi-Height): Here, the width differs, because we want all buckets to contain the same number of tuples.
|
||||
The size of a range helps with cardinality estimates
|
||||
\item \bi{Singleton} (or Frequency): Plots how often a single item appears in a table.
|
||||
This is very useful to compute the selectivity of queries
|
||||
\end{itemize}
|
||||
|
||||
\newpage
|
||||
To enable quick processing of the cost function, the following metadata, or statistics, are typically stored:
|
||||
\begin{multicols}{2}
|
||||
\bi{Table statistics}
|
||||
\begin{itemize}
|
||||
\item Number of rows
|
||||
\item Number of blocks
|
||||
\item Average row length
|
||||
\end{itemize}
|
||||
\bi{Column statistics}
|
||||
\begin{itemize}
|
||||
\item Number of distinct values (NDV) in columns
|
||||
\item Number of nulls in column
|
||||
\item Data distribution (histogram)
|
||||
\end{itemize}
|
||||
\bi{Extended statistics}
|
||||
\begin{itemize}
|
||||
\item Index statistics
|
||||
\item Number of leaf blocks
|
||||
\item Levels
|
||||
\item Clustering factor
|
||||
\end{itemize}
|
||||
\bi{System statistics}
|
||||
\begin{itemize}
|
||||
\item I/O performance and utilization
|
||||
\item CPU performance and utilization
|
||||
\end{itemize}
|
||||
\end{multicols}
|
||||
@@ -0,0 +1,36 @@
|
||||
\subsubsection{Search}
|
||||
We now know how to generate a set of semantically equivalent physical plans and how to estimate the cost for each of them.
|
||||
The question that remains is how to efficiently find the best plan.
|
||||
|
||||
Since real-world queries can be very complex, searching for the best physical plan can be hard.
|
||||
There are a few options, but we only covered a simple approach.
|
||||
|
||||
We can constrain the search space by only considering \bi{left-deep join trees}, i.e. trees that have only leafs on the right hand side of each join statement.
|
||||
The rationale behind this is that many of these are fully pipelined plans and no intermediate results have to be written to temporary files.
|
||||
This is the concept \texttt{System R}, the first relational database used.
|
||||
|
||||
An example for a non-pipelined left-deep join tree is one in volving sort operations.
|
||||
This brings the complexity down to \tco{n!}, where $n$ is the number of realations in the join.
|
||||
|
||||
\begin{itemize}
|
||||
\item Enumerate join orders (different left deep trees)
|
||||
\item Enumerate plans for each operator (hash join, sort merge join, nested loop join, \dots)
|
||||
\item Enumerate access methods for each operator (B+-Tree, hash table, sequential scan, \dots)
|
||||
\end{itemize}
|
||||
|
||||
This can be achieved using dynamic programming.
|
||||
|
||||
We then find the best 1-relation plan, then the best 2-relation plan by finding ways to join a relation with the best 1-relation plan, etc.
|
||||
|
||||
While this is still slow, it's faster than enumerating all possible join trees.
|
||||
|
||||
Later, systems started introducing heuristics to reduce the number of choices that must be made in a cost-based fashion.
|
||||
|
||||
Heuristic optimization transforms the query-tree using a set of rules typically improve execution performance:
|
||||
\begin{itemize}
|
||||
\item Perform selection early (reduces the number of tuples to process)
|
||||
\item Perform projection early (reduces the number of attributes to process)
|
||||
\item Perform most restrictive selection and join operations before other similar operations
|
||||
\end{itemize}
|
||||
|
||||
Some systems only use heuristics, while others combine them with partial cost-based optimization.
|
||||
@@ -0,0 +1,18 @@
|
||||
\subsubsection{Storage Management}
|
||||
To make database engines easy to use, some abstractions have to happen.
|
||||
\begin{figure}[h!]
|
||||
\begin{center}
|
||||
\includegraphics[width=0.7\textwidth]{assets/db-system-memory.png}
|
||||
\end{center}
|
||||
\caption{Overview of a typical abstraction for a database system
|
||||
(Figure from lecture slides 09, slide 3)}
|
||||
\end{figure}
|
||||
|
||||
In the design of database engines it is commonly assumed that the data is stored on spinning disks, i.e. HDDs, with a high latency for seeks (random accesses)
|
||||
and that most of the DB is not in memory.
|
||||
|
||||
Very modern database systems may assume that a higher percentage of the DB is stored in memory due to the higher quantity of memory available,
|
||||
as well as lower access latencies due to the more readily available SSDs and high-speed NASes.
|
||||
|
||||
Some database systems operate similarly to operating systems in certain aspects, in that they manage their own processes and all of the memory
|
||||
to get the absolute maximum performance out of the systems.
|
||||
@@ -4,5 +4,14 @@
|
||||
\input{parts/03_systems/00_query-optimization/00_intro.tex}
|
||||
\input{parts/03_systems/00_query-optimization/01_search-space/00_basics.tex}
|
||||
\input{parts/03_systems/00_query-optimization/01_search-space/01_rewriting-rules.tex}
|
||||
% \input{parts/03_systems/00_query-optimization/01_search-space/}
|
||||
% \input{parts/03_systems/00_query-optimization/}
|
||||
\input{parts/03_systems/00_query-optimization/01_search-space/02_execution-model.tex}
|
||||
\input{parts/03_systems/00_query-optimization/01_search-space/03_cost-model.tex}
|
||||
\input{parts/03_systems/00_query-optimization/01_search-space/04_search.tex}
|
||||
|
||||
\newpage
|
||||
\subsection{Database Engines}
|
||||
\input{parts/03_systems/01_database-engines/00_memory/00_intro.tex}
|
||||
% \input{parts/03_systems/01_database-engines/00_memory/}
|
||||
% \input{parts/03_systems/01_database-engines/}
|
||||
|
||||
% \input{parts/03_systems/}
|
||||
|
||||
Reference in New Issue
Block a user