[DMDB] Data formats and cloud basics, start vector search

semester4/dmdb/data-modelling-databases-summary.tex

@@ -84,6 +84,8 @@ In addition, the end user does not have to worry about optimization too much. Th
 \include{parts/02_theory-background/main.tex}
 \include{parts/03_systems/main.tex}
 \include{parts/04_query-processing/main.tex}
+\include{parts/05_vector-search/main.tex}
+\include{parts/06_data/main.tex}
 
 
 \printGlossary

semester4/dmdb/parts/05_vector-search/00_intro-trees-clustering.tex

@@ -0,0 +1,15 @@
+Vector search is for example used in Semantic Search and of course search engines.
+It is also used to find relevant data to augment an LLM prompt (Deep Research or whatever that type of feature is called).
+
+\subsection{Trees/Clustering}
+Since nearest neighbour search (NNS) is not viable for large scale datasets (which they typically are),
+we use clustering algorithms such as K-Means to find $K$ cluster centroids.
+We then assign to each of the centroids the vectors that belong to this cluster
+
+Then, for search, we can check which cluster(s) the query vector is closest to.
+Due to the low number of centroids, this is comparatively cheap to do.
+We can then search all the vectors in the corresponding clusters, which is much cheaper due to the much lower number of vectors to search.
+Still, Vector Search is a very expensive operation due to the large number of vectors to be searched, even if the search itself is $\tco{n}$,
+with $n$ being the number of vectors to search through.
+
+On insert, we search the the closest centroid and simply assign the vector to that cluster.

semester4/dmdb/parts/05_vector-search/00_rag.tex

@@ -1 +0,0 @@
-\subsection{Retrieval Augmented Generation}

semester4/dmdb/parts/05_vector-search/main.tex

@@ -0,0 +1,3 @@
+\section{Vector Search}
+\input{parts/05_vector-search/00_intro-trees-clustering.tex}
+% \input{parts/05_vector-search/}

semester4/dmdb/parts/06_data/00_formats/00_intro.tex

@@ -0,0 +1,6 @@
+Data formats heavily influence how efficient a database system is.
+For a 1GB data, JSON is about three times larger than necessary due to field names and text encoding.
+For the TPC-H lineitem table, the CSV file is 15.95GB, whereas Parquet Snappy and ZSTD are 3.78 and 3.22 Gigabytes respectively,
+so factor 5 smaller. Furthermore, a raw CSV scanner is about three times slower than a Parquet Scanner.
+
+If we scale that up to Terabytes or more, compute already is dominated by I/O 80-90\% of the time, so it really is crucial that we have an efficient format.

semester4/dmdb/parts/06_data/00_formats/01_encodings.tex

@@ -0,0 +1,46 @@
+\subsubsection{Encodings}
+Typical text encodings are \texttt{CSV}, \texttt{XML}\footnote{you have to be quite a bit of a moron to think that this is the best of the text encodings, but all right}
+and \texttt{JSON}. Of course, there are also the (in my objective opinion) best formats\footnote{best for configuration files, not (necessarily) in general}
+like \texttt{conf} and \texttt{yaml}, as well as \texttt{TOML}, \texttt{INI} and more.
+
+These are good for being human readable, but not if you have terabytes of data.
+From a practical example, I know that a 1GB JSON file in Python's built in parser takes about 1s to load.
+
+This is why we need (direct) binary encodings for the data, such as the Apache formats.
+
+Examples include Avro (row-oriented format, including a schema describing the fields and types, this is the header of the file), Parquet (Compressed column store),
+CarbonData (Compressed column store with indexes) and more.
+
+When designing formats, decisions need to be made on the file metadata, format layout, type system, encoding schemes, compression, filters and how to nest the data.
+
+\bi{Metadata} Ideally the files are self-contained to allow for portability, and for space efficiency reasons (even though it may be beneficial for other reasons),
+schemas are only stored once per file, in either the header (e.g. for Avro) or in the footer (e.g. Parquet). Other metadata however is more oftne stored once per frame.
+
+The \bi{format layout} is the physical organization of the bytes on the disk. We can have row stores, column stores or hybrids:
+\begin{itemize}
+    \item N-ary Storage Model (NSM): All attributes stored contiguously in a single page (Row store)
+    \item Decomposition Storage Model (DSM): Store single attributes for all tuples contiguously in a block of data (Column store)
+    \item PAX: Horizontally partition data into row groups, then vertically partition the attributes into column chunks.
+\end{itemize}
+
+For the \bi{type system}, we choose what kind of types we want to support, be it primitives such as numbers and strings, or more complex types such as arrays.
+Furthermore, we choose how they are physically represented, such as IEEE-754 encoding for floating point numbers, and logically, if they are complex types based on other types,
+i.e. how they abstract a more primitive type.
+
+For \bi{encoding schemes} include:
+\begin{itemize}
+    \item \textit{Dictionary encoding}: We find all unique items of a column and build a dictionary with integer keys and then only store the integers.
+          This is ideal for columns with few distinct values and worse than not doing anything for columns with the UNIQUE constraint set.
+    \item \textit{Run-Length encoding}: If we have many repeating values, called runs, we replace each run with a pair \texttt{(count, value)}.
+          This can be done on the data itself, or on the binary representation of said data.
+    \item \textit{Delta Encoding}: Store the first value (as-is), then store the difference between this value and the next.
+          Since these deltas are often small, we can fit them into fewer bits.
+\end{itemize}
+
+For \bi{Block compression}, we can use typical compression algorithms, such as \texttt{LZ4} (which by the way is VERY impressive for compression ratio),
+\texttt{Snappy}, \texttt{Zstd} (which \texttt{pacman} uses for the packages (\texttt{.pkg.tar.zst} files)), or older algorithms such as \texttt{gzip} or \texttt{xz}
+(if you remember the whole malware debacle they had, you may not want to use that anymore).
+
+
+\bi{Filters} in the files are primarily used to improve traverse performance, by filtering out pages that simply don't need to be searched, using some metadata.
+For that, statistics are stored in the metadata, such as the min/max, null count, cardinality and more.

semester4/dmdb/parts/06_data/01_cloud-systems/00_intro.tex

@@ -0,0 +1,15 @@
+Cloud systems need to be very reliable, no data is allowed to be lost.
+In basic versions of so-called \bi{shared nothing instances}, this is not as much guaranteed as it should be,
+because all DBs systems have their own disks and they don't replicate or exchange their data.
+
+To improve fault tolerance, make blocks small, like 1MB and replicate each block on a different compute node and ideally on a separate S3 node,
+which serves as a backup. That's the 3-2 taken care of of the 3-2-1 backup strategy. This should then also be replicated off-site to mitigate the risk of destruction of all nodes
+from a natural disaster, building collapse, fire or similar.
+
+Modern clouds separate compute and storage entirely. Typically, compute servers have no disk and are booted using iSCSI,
+or they have a small disk for the base operating system in them.
+
+In systems with disaggregated Compute, Memory and Storage, each layer can scale independently, allowing large memory shuffles to take place and allows
+for larger numbers of blocks for sorting and joins, which, as we have seen, can massively boost performance.
+
+The Big Query system is one such example.

semester4/dmdb/parts/06_data/main.tex

@@ -0,0 +1,10 @@
+\section{Data}
+\subsection{Formats}
+\input{parts/06_data/00_formats/00_intro.tex}
+\input{parts/06_data/00_formats/01_encodings.tex}
+% \input{parts/06_data/00_formats/}
+
+\subsection{Cloud systems}
+\input{parts/06_data/01_cloud-systems/00_intro.tex}
+% \input{parts/06_data/01_cloud-systems/}
+% \input{parts/06_data/}