[DMDB] Add concurrency control and recovery

Oops, luckily still figured out that that was missing...
2026-07-27 21:29:09 +02:00 · 2026-07-25 18:58:39 +02:00
parent 45b3f178a7
commit c82ceca4e0
29 changed files with 475 additions and 22 deletions
@@ -0,0 +1,6 @@
+Data formats heavily influence how efficient a database system is.
+For a 1GB data, JSON is about three times larger than necessary due to field names and text encoding.
+For the TPC-H lineitem table, the CSV file is 15.95GB, whereas Parquet Snappy and ZSTD are 3.78 and 3.22 Gigabytes respectively,
+so factor 5 smaller. Furthermore, a raw CSV scanner is about three times slower than a Parquet Scanner.
+
+If we scale that up to Terabytes or more, compute already is dominated by I/O 80-90\% of the time, so it really is crucial that we have an efficient format.
@@ -0,0 +1,46 @@
+\subsubsection{Encodings}
+Typical text encodings are \texttt{CSV}, \texttt{XML}\footnote{you have to be quite a bit of a moron to think that this is the best of the text encodings, but all right}
+and \texttt{JSON}. Of course, there are also the (in my objective opinion) best formats\footnote{best for configuration files, not (necessarily) in general}
+like \texttt{conf} and \texttt{yaml}, as well as \texttt{TOML}, \texttt{INI} and more.
+
+These are good for being human readable, but not if you have terabytes of data.
+From a practical example, I know that a 1GB JSON file in Python's built in parser takes about 1s to load.
+
+This is why we need (direct) binary encodings for the data, such as the Apache formats.
+
+Examples include Avro (row-oriented format, including a schema describing the fields and types, this is the header of the file), Parquet (Compressed column store),
+CarbonData (Compressed column store with indexes) and more.
+
+When designing formats, decisions need to be made on the file metadata, format layout, type system, encoding schemes, compression, filters and how to nest the data.
+
+\bi{Metadata} Ideally the files are self-contained to allow for portability, and for space efficiency reasons (even though it may be beneficial for other reasons),
+schemas are only stored once per file, in either the header (e.g. for Avro) or in the footer (e.g. Parquet). Other metadata however is more oftne stored once per frame.
+
+The \bi{format layout} is the physical organization of the bytes on the disk. We can have row stores, column stores or hybrids:
+\begin{itemize}
+    \item N-ary Storage Model (NSM): All attributes stored contiguously in a single page (Row store)
+    \item Decomposition Storage Model (DSM): Store single attributes for all tuples contiguously in a block of data (Column store)
+    \item PAX: Horizontally partition data into row groups, then vertically partition the attributes into column chunks.
+\end{itemize}
+
+For the \bi{type system}, we choose what kind of types we want to support, be it primitives such as numbers and strings, or more complex types such as arrays.
+Furthermore, we choose how they are physically represented, such as IEEE-754 encoding for floating point numbers, and logically, if they are complex types based on other types,
+i.e. how they abstract a more primitive type.
+
+For \bi{encoding schemes} include:
+\begin{itemize}
+    \item \textit{Dictionary encoding}: We find all unique items of a column and build a dictionary with integer keys and then only store the integers.
+          This is ideal for columns with few distinct values and worse than not doing anything for columns with the UNIQUE constraint set.
+    \item \textit{Run-Length encoding}: If we have many repeating values, called runs, we replace each run with a pair \texttt{(count, value)}.
+          This can be done on the data itself, or on the binary representation of said data.
+    \item \textit{Delta Encoding}: Store the first value (as-is), then store the difference between this value and the next.
+          Since these deltas are often small, we can fit them into fewer bits.
+\end{itemize}
+
+For \bi{Block compression}, we can use typical compression algorithms, such as \texttt{LZ4} (which by the way is VERY impressive for compression ratio),
+\texttt{Snappy}, \texttt{Zstd} (which \texttt{pacman} uses for the packages (\texttt{.pkg.tar.zst} files)), or older algorithms such as \texttt{gzip} or \texttt{xz}
+(if you remember the whole malware debacle they had, you may not want to use that anymore).
+
+
+\bi{Filters} in the files are primarily used to improve traverse performance, by filtering out pages that simply don't need to be searched, using some metadata.
+For that, statistics are stored in the metadata, such as the min/max, null count, cardinality and more.
@@ -0,0 +1,15 @@
+Cloud systems need to be very reliable, no data is allowed to be lost.
+In basic versions of so-called \bi{shared nothing instances}, this is not as much guaranteed as it should be,
+because all DBs systems have their own disks and they don't replicate or exchange their data.
+
+To improve fault tolerance, make blocks small, like 1MB and replicate each block on a different compute node and ideally on a separate S3 node,
+which serves as a backup. That's the 3-2 taken care of of the 3-2-1 backup strategy. This should then also be replicated off-site to mitigate the risk of destruction of all nodes
+from a natural disaster, building collapse, fire or similar.
+
+Modern clouds separate compute and storage entirely. Typically, compute servers have no disk and are booted using iSCSI,
+or they have a small disk for the base operating system in them.
+
+In systems with disaggregated Compute, Memory and Storage, each layer can scale independently, allowing large memory shuffles to take place and allows
+for larger numbers of blocks for sorting and joins, which, as we have seen, can massively boost performance.
+
+The Big Query system is one such example.
@@ -0,0 +1,77 @@
+\subsection{Key Value Store}
+\label{sec:key-value-store}
+The rise of the internet has shown that there is the need for a system that can more quickly retrieve data than the older implementations using blocks, files and objects.
+The goal is to be able to access a chunk of data that does not need to be parsed (unlike objects), but can be of various sizes (unlike blocks)
+and needs no system-supported names (like file systems).
+
+The solution to this problem is called a \textit{Key Value Store} (KVS).
+
+Most \acrshort{sql} databases follow the \acrshort{acid} concept to maintain consitency, etc.
+The \acrshort{acid} concept gives very strong guarantees, thus the work is done by the database engine as opposed to the application.
+However, it is expensive, both in throughput and latency and does not scale well.
+
+Key-Value stores get rid of EVERYTHING.
+They are schema-less, \acrshort{nosql}, no \acrshort{acid}, (typically) \acrshort{base} based stores.
+The most well known examples of \acrshort{nosql} DBs are \texttt{Redis} and \texttt{MongoDB},
+with \texttt{Redis} being a true Key-Value Store (however usually referred to as a Key-Value Cache).
+
+\subsubsection{Architecture}
+\begin{itemize}
+    \item Data is stored as a key-value pair. The key is used to locate and retrieve the data. The value is a binary object of arbitrary size
+    \item The data is organized as a distributed hash table. We hash the key to get the location for the value
+    \item KV pairs stored in several places for availability
+    \item Access is \gls{quorum}-based for consistency
+    \item Typically eventual consistency
+    \item The physical storage uses already reserved buckets of different sizes and objects are located in the corresponding buckets.
+    \item This can be run in main memory for very fast access, with larger blobs stored in storage.
+\end{itemize}
+
+
+\subsubsection{Advantages and Disadvantages}
+\begin{multicols}{2}
+    \bi{Advantages}
+    \begin{itemize}
+        \item Very fast lookups (due to hashing)
+        \item Easy to scale to many machines (simply add more copies / machines). This is great for the cloud's elasticity
+        \item Can be implemented in main memory of machines (giving even faster lookups)
+        \item Useful in many applications (looking up user profiles, retrieving user's web pages and info)
+        \item Scalable and easy to get going (no schema needed)
+    \end{itemize}
+
+    \bi{Disadvantages}
+    \begin{itemize}
+        \item No easy support for queries beyond simple key requests
+        \item this means that there are no joins, no ranges, no queries on attributes, etc (because there are none)
+        \item Depending on implementation, data can be inconsistent
+        \item Maintaining consistency is up to the application
+        \item Used in large scale deployments (worldwide), so keeping data in sync can be challenging
+    \end{itemize}
+\end{multicols}
+As mentioned above, one major drawback of \acrshort{nosql} DBs is that they move the complexity to the application.
+This means that some operations are very costly to implement (range queries, joins, etc), as they require reading all data from the DB.
+
+
+\subsubsection{Uses of KVS}
+As a result of the above disadvantages, KVS are typically used as a cache or specialized system and not as a replacement for \acrshort{sql} databases.
+
+They are often used as a main memory cache to avoid having to access the cloud storage, resulting in higher throughput and lower latency.
+
+This is especially useful for commonly run queries, such as loading all videos of a commonly accessed YouTube Channel.
+In that case especially it is easy to invalidate the cache line and updating it, as updates are very rare and they can trigger a cache invalidation for that channel.
+
+
+\subsubsection{Distributed KVS}
+How do we distribute a KVS across multiple systems?
+A simple strategy would be to assign data to machines using modulo $n$ for $n$ machines on the hash.
+In theory that is not a bad idea, but what if we remove one machine? Or if we add one? Then we have to move ALL of the data.
+If one machine fails, in the worst case, all data could possibly need to be re-hashed, possibly moving ALL of the data to different machines.
+
+A better solution is to use horizontal partitioning of the data using hashing.
+Each machine deals with the data points in the clockwise direction before the next machine.
+In other words, each machine deals with the data in a certain range of hashes, the next machine deals with the subsequent ones, etc.
+
+Thus, on fail, or exit in any other way, only the machine after the failed / exited machine needs to modify the data.
+When adding nodes, the next machine hands some of the responsibilities over to the newly inserted machine.
+
+To account for possible failure of machines, we need to keep multiple copies over multiple machines (ahead in counter-clockwise orientation)
+to prevent data loss.
@@ -0,0 +1,10 @@
+\subsection{Cloud-Native Databases}
+Many of the traditional database engines may run into overhead due to heavy I/O traffic, especially when mirroring.
+
+As a result, many companies have improved implementations of SQL databases to make them suitable to cloud infrastructure.
+
+Some of the improvements include caching. Micro-partitions in the Snowflake system is a system to facilitate query processing.
+Each one of these partitions has sizes between 50 and 500 megabytes, and has metadata describing what is inside, which can also be read without reading the entire partition.
+
+Furthermore, Amazon S3, the storage platform, does not support in-place updates, files are replaced in their entirety.
+This means that old data can easily be recovered.
@@ -0,0 +1,10 @@
+\subsection{The Future}
+``The IT industry is, above all, an industry''. This is very much true and trends and developments are driven by demand and market.
+For hardware, the investment is very high for new developments and only pay off at large sale volumes.
+Hardware moves slowly, software even slower.
+
+Furthermore, the number of foundries has gone down significantly since 2002 (130nm architecture),
+where today, there are only three bleeding edge foundries left, of which one has over 50\% market share in bleeding edge semiconductor manufacturing, TSMC.
+Samsung and Intel both have lost out on major deals due to a number of issues (Nvidia Ampere was the last major GPU series manufactured by Samsung).
+
+CPUs are low latency (very fast), but low throughput in terms of parallelism, whereas GPUs are the exact opposite.
@@ -0,0 +1,13 @@
+\section{Data}
+\subsection{Formats}
+\input{parts/07_data/00_formats/00_intro.tex}
+\input{parts/07_data/00_formats/01_encodings.tex}
+% \input{parts/07_data/00_formats/}
+
+\subsection{Cloud systems}
+\input{parts/07_data/01_cloud-systems/00_intro.tex}
+\input{parts/07_data/01_cloud-systems/01_key-value-store.tex}
+\input{parts/07_data/01_cloud-systems/02_cloud-native-databases.tex}
+\input{parts/07_data/01_cloud-systems/03_future.tex}
+% \input{parts/07_data/01_cloud-systems/}
+% \input{parts/07_data/}