diff --git a/semester4/dmdb/data-modelling-databases-summary.pdf b/semester4/dmdb/data-modelling-databases-summary.pdf index 3eabfa0..ac2a302 100644 Binary files a/semester4/dmdb/data-modelling-databases-summary.pdf and b/semester4/dmdb/data-modelling-databases-summary.pdf differ diff --git a/semester4/dmdb/data-modelling-databases-summary.tex b/semester4/dmdb/data-modelling-databases-summary.tex index 0caf89b..aedb534 100644 --- a/semester4/dmdb/data-modelling-databases-summary.tex +++ b/semester4/dmdb/data-modelling-databases-summary.tex @@ -7,11 +7,11 @@ \input{glossary/main.tex} \setGlossaryMarkup - \usepackage{lmodern} \setFontType{sans} \newcommand{\sql}{\acrshort{sql}} +\renewcommand{\descriptorNameDisplay}[1]{\textbf{#1}} \setup{Data Modelling \& Databases} diff --git a/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/03_data-in-blocks-and-pages.tex b/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/03_data-in-blocks-and-pages.tex index 757b047..5a1f186 100644 --- a/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/03_data-in-blocks-and-pages.tex +++ b/semester4/dmdb/parts/03_systems/01_database-engines/00_memory/03_data-in-blocks-and-pages.tex @@ -18,6 +18,7 @@ There are some ways to improve the performance: \paragraph{Slotted Pages} +\label{sec:slotted-pages} To find tuples within pages, slotted pages are used often. In slotted pages, each page has a header containing checksum, version, transaction visibility, compression information, utilization, etc. diff --git a/semester4/dmdb/parts/03_systems/02_indexing/00_access-methods.tex b/semester4/dmdb/parts/03_systems/02_indexing/00_access-methods.tex new file mode 100644 index 0000000..f24cd9a --- /dev/null +++ b/semester4/dmdb/parts/03_systems/02_indexing/00_access-methods.tex @@ -0,0 +1,13 @@ +\subsubsection{Access methods} +\inlinedefinition[Access Methods] typically refers to the operations and data structures used for reading data from the buffer cache into the private space of the worker thread + +The basic access method is \bi{sequential scan}, where all pages of the table are read from beginning to end. +Indexing is used to avoid having to scan the whole table. + +There are many indexing techniques, but we will focus on three indexing techniques in this section, hashing, B+ trees and bitmaps. + +Hashing is primarily used to randomize access to the data, finding data based on a key (hash and access concept) +and both as an index for tables and used to implement operators. + +B+ trees on the other hand are used to order data along the values of one attribute, but require traversal, unless we are looking for consecutive values. +They are thus only used as an indexing mechanism. diff --git a/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/00_b-trees.tex b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/00_b-trees.tex new file mode 100644 index 0000000..2e6bf68 --- /dev/null +++ b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/00_b-trees.tex @@ -0,0 +1,6 @@ +\subsubsection{B+ Trees} +\paragraph{B-Tree} +In B-Trees of order $k$, each node (except the root) has between $k \div 2$ and $k$ child nodes. +The root node has at least two children, unless it is also a leaf and any non-leaf node contains at exactly $k - 1$ keys. + +Even if database systems say they use B-Trees, they do in fact use B+ trees. diff --git a/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/01_basics.tex b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/01_basics.tex new file mode 100644 index 0000000..0d5e47d --- /dev/null +++ b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/01_basics.tex @@ -0,0 +1,59 @@ +\paragraph{Basics} +B+ Trees are B-Trees, but the data (or the pointers to it) is at the leaves only, which are organized as a linked list. +They are balanced trees, their inner nodes correspond to blocks and the leaf nodes correspond to blocks. +The blocks are organized like slotted pages for variable length data (see Section \ref{sec:slotted-pages}) + +The keys in the inner nodes may not correspond to the actual data and are instead used as separators. +Typically, leaf nodes contain pointers to the tuples \texttt{(value, key)}, where \texttt{value} is the actually indexed value +and \texttt{key} is the pointer to the full data tuple, typically as a row id or tuple id. +There are some systems that allow storing the data directly on the leaves. + +Some systems create a B+ tree index by default for all tables and index the key. If there is no key, the engine assigns random keys and indexes them. + + +\paragraph{Clustered Indexes} +An index orders the table by the attribute it is indexing, but the tuples in the table might not be ordered. +This is where \bi{clustered indexes} come into play: They force the tuples to be stored in the same order as the index indicates, +thus, the table is physically stored in a sorted manner. This is typically done only for the primary key, +and automatically done in systems that store data in the leaf nodes directly. + +This eliminates random memory accesses caused by looking up ranges, since the tuples are stored in consecutive order, according to the index. + +This is most useful for tables that are infrequently updated, since they require maintenance. + + +\paragraph{What to index?} +This is a crucial question, as it can massively speed things up, but equally significantly slow things down. + +B+ trees can use one or \textit{more} attributes as the key to the index. +If we have just one attribute, those are the values. +If there are several attributes, it builds a composite key, which is useful when we are looking for combinations of values on certain attributes, +or a table is searched by several attributes. + +We then compare the keys lexicographically: +\[ + (a_1, a_2) < (b_1, b_2) \Leftrightarrow (a_1 < b_1) \lor (a_1 = b_1 \land a_2 < b_2) +\] +Some values can also be left unspecified, but this is typically only done to the ones at the beginning of the key. + +\newpage +\paragraph{Composite Index} +For a \acrshort{sql} statement such as +\begin{code}{sql} + SELECT department_id, last_name, salary + FROM employees WHERE salary > 5000 + ORDER BY department_id, last_name; +\end{code} +if we assume a composite index on \texttt{department\_id, last\_name, salary}, then the data is sorted by the three attributes in that order. + +We can then perform a \bi{full index scan}, where read the data from the leaves in sorted order and filter on salary. +This avoids having to scan the entire table and the results is also already sorted. + + +\paragraph{Non-unique values} +A B+ tree index can also be built on attributes containing also non-unique values. + +This is a problem, because we cannot find all the duplicates with the basic design. To resolve this, we can repeat the key at the leaf nodes for every duplicated entry, +or we can store the key once, but point to a linked list of all matching entries. + +If the data is stored in the leaves, we append the tuple ID to know what tuple the entry refers to, because otherwise, they are all the same. diff --git a/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/02_operations.tex b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/02_operations.tex new file mode 100644 index 0000000..a2441e4 --- /dev/null +++ b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/02_operations.tex @@ -0,0 +1,55 @@ +\paragraph{Operations} +\subparagraph{Direct Lookup} +We traverse the tree from the root, then within each node, we use binary search to look for the correct entry. +When we reach a leaf node, we return the corresponding pointer and the position. + + +\subparagraph{Range Lookup} +We return all tuples that match, by first finding the first tuple matching, the continue traversal until the last match is found. +We can determine the last match easily because the tree's leafs are lexicographically sorted. +Since the leafs are linked as linked lists, we can simply move along the linked lists, cutting down on processing time. + + +\subparagraph{Inserting into the index} +We first look up the corresponding leaf. Then, if there is space, we simply insert the value. +If there is no space in the leaf, we split the leaf into two new leafs and insert the new item on the corresponding leaf. + +All that remains to do now is to insert the separator on the parent node. +If that node is full, we split the node in two and insert the separator into the parent node and propagate up to the root, if necessary. + + +\subparagraph{Deleting from the index} +We proceed by looking up the corresponding leaf, then we remove the entry from the leaf. +If the leaf now is less than half full, we check the neighboring leaf +\begin{itemize} + \item If it is more than half full, we balance the two leafs + \item If it is half full, we merge both leaves +\end{itemize} +For both operations, we update the separator in the parent node. + + +\paragraph{Concurrent Access} +Indexes are heavily used while they are being maintained, thus creating conflicting, concurrent accesses. + +A way to prevent this is using \bi{lock coupling}, where we lock a page and its parent and move down the tree. + +This prevents problems when the page must be split, however, this is not enough, if we have to go further up to apply the changes. + +To prevent this, on every node we check if there is space for one more entry. If not, we split the node. +This way, the split of a page never leads to propagating changes. + +Alternatively, we can try lock coupling, and if we have to do a change that propagates up, we abort, release everything and start again from the top, +but now lock the entire path. + + +\paragraph{Bulk Inserts} +To create a B+ Tree index, we proceed as follows (the tree is created bottom up (from the leaves up)) +\begin{enumerate} + \item Sort the data + \item Use the sorted data to fill one block after the other + \item Remember the largest value in each block + \item create the inner nodes by using the largest value in each block as separator + \item iterate upwards to the next level, until there is only one block left (the root) +\end{enumerate} +If the blocks are filled completely, we get a clustered and compact tree. +However, since we usually expect that data has to be updated, it is advised to leave some space in the each block to accommodate that without expensive propagating changes. diff --git a/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/03_optimizations.tex b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/03_optimizations.tex new file mode 100644 index 0000000..86d055f --- /dev/null +++ b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/03_optimizations.tex @@ -0,0 +1,27 @@ +\paragraph{Optimizations} +\shade{ForestGreen}{Reverse Index} +If the indexed attribute is a sequence and new items are constantly produced, we may encounter an issue where values next to each other go into the same block, +leading to concurrent updates fighting to insert. Depending on how updates are handled, many copies could be produced. + +Reverse indexing solves that issue by reversing the indices, meaning that consecutive indices go to different pages. + + +\shade{ForestGreen}{Many Keys} Depending on attributes, many keys can become expensive to process. +There are several possible optimizations: +\begin{itemize} + \item Replace separators that correspond to actual keys with shorter separators with the same effect (can be hard) + \item Factor common prefix (this makes a lot of sense and is usually easy, too) +\end{itemize} + + +\shade{ForestGreen}{Ignoring rules} We can also ignore the rules of the B+ trees and e.g. not merge nodes when they do not have enough data, +delaying a merge (and instead rebuilding the tree periodically) + + +\shade{ForestGreen}{Variable Length Keys} This is a bit of a problem. We can mitigate it by no longer forcing the number of entries to lay between $k \div 2$ and $k$, +and using smaller values as separators, placing long values in leafs. + +\shade{ForestGreen}{Making sure there is space} We don't ever fill blocks to the max and make sure there is space on creation. + + +\shade{ForestGreen}{Depth vs Breadth} For slow devices, prefer larger node sizes, thus shallower trees; For fast devices, prefer deeper trees. diff --git a/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/04_uses.tex b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/04_uses.tex new file mode 100644 index 0000000..07f179b --- /dev/null +++ b/semester4/dmdb/parts/03_systems/02_indexing/00_b-trees/04_uses.tex @@ -0,0 +1,37 @@ +\paragraph{When to use B+ Trees} +\begin{itemize} + \item For exact match queries (using \texttt{=} or \texttt{in}). This avoids having to compare against the entire table + \item Range queries (using \texttt{<} and/or \texttt{<}). + \item Full scans for sorted output (as the tuples are inherently sorted in the tree) +\end{itemize} + + +\paragraph{When not to use B+ Trees} +\begin{itemize} + \item The index stores pointers to tuples + \item For larger number of accesses, more random memory accesses happen, which is slower. +\end{itemize} + + +\paragraph{Columns vs Rows} +B+ Trees can be used to index columns, but are primarily used in row store systems, +because Column Stores optimize sequential reads and can use vector operations, as well as compressions. + +A B+ Tree would point to an entry, but only after a lot of pointer chasing. + +And then there is also the point that they require extra storage, which for column stores just doesn't make sense. + + + +\paragraph{Filtered Indexes} +As the name implies, this is a B+ Tree that indexes only parts of a table, which commonly are the most used ranges. +This of course means that the index is smaller and thus faster to traverse. +It defines a range predicate over the indexed key. + + +\paragraph{How are they used?} +\begin{itemize} + \item JOIN: if the inner table is indexed, a nested loop join will look up the tuple on the index, reducing comparisons + \item ORDER BY: If the sorting is on the indexed key, we can easily retrieve tuples in order + \item IN or NOT IN: In nested queries, the index can be used to check if a tuple in the outer query is in the result of the inner query. +\end{itemize} diff --git a/semester4/dmdb/parts/03_systems/main.tex b/semester4/dmdb/parts/03_systems/main.tex index 48a2e90..47d2556 100644 --- a/semester4/dmdb/parts/03_systems/main.tex +++ b/semester4/dmdb/parts/03_systems/main.tex @@ -19,4 +19,13 @@ % \input{parts/03_systems/01_database-engines/00_memory/} % \input{parts/03_systems/01_database-engines/} +\newpage +\subsection{Indexing} +\input{parts/03_systems/02_indexing/00_access-methods.tex} +\input{parts/03_systems/02_indexing/00_b-trees/00_b-trees.tex} +\input{parts/03_systems/02_indexing/00_b-trees/01_basics.tex} +\input{parts/03_systems/02_indexing/00_b-trees/02_operations.tex} +\input{parts/03_systems/02_indexing/00_b-trees/03_optimizations.tex} +% \input{parts/03_systems/02_indexing/00_b-trees/} +% \input{parts/03_systems/02_indexing/} % \input{parts/03_systems/}