[DMDB] Finish first version of summary

semester4/dmdb/glossary/acronyms.tex

@@ -1,7 +1,10 @@
 \newacronym{sql}{SQL}{Structured Query Language}
+\newacronym{nosql}{NoSQL}{SQL-less database (commonly a Key-Value Store, see Section~\ref{sec:key-value-store})}
 \newacronym{dbms}{DBMS}{Database Management System}
 \newacronym{simd}{SIMD}{Single Instruction Multiple Data}
 \newacronym{avxa}{AVX}{Advanced Vector Extensions, also see \gls{avx}}
 \newacronym{ra}{RA}{Relational Algebra, see Section~\ref{sec:relational-algebra}}
 \newacronym{ann}{ANN}{Approximate Nearest Neighbour Search}
 \newacronym{knn}{KNN}{$K$-Nearest Neighbour Search}
+\newacronym{acid}{ACID}{Atomicitty, Consistency, Isolation, Durability}
+\newacronym{base}{BASE}{Basically Available, Soft State, Eventually Consistent}

semester4/dmdb/glossary/defs.tex

@@ -2,3 +2,10 @@
     name={superkey},
     description={See Section \ref{sec:functional-dependency}}
 }
+\newglossaryentry{quorum}{
+    name={quorum},
+    description={
+            See \href{https://en.wikipedia.org/wiki/Quorum_(distributed_computing)}{this Wikipedia article}.
+            TL;DR: It is a synchronization system for distributed computing.
+        }
+}

semester4/dmdb/parts/05_vector-search/03_filtered-vector-search.tex

@@ -2,7 +2,7 @@
 The benefit of supporting vector search in DBMS is that this makes it easy to use for the end user and can be optimized for large datasets.
 
 \subsection{Filtered Vector Search (FVS)}
-This allows us to query both structured and unstructured data. In concept, this is SQL + Vectors, sometimes plus joins, subquery expressions,
+This allows us to query both structured and unstructured data. In concept, this is \acrshort{sql} + Vectors, sometimes plus joins, subquery expressions,
 multiple vector searches (on both sides of the join). Sometimes, the dataset isn't even a vector + tag stored in main memory.
 
 Indices are typically built on unfiltered data. Thus, we need to think about the proper execution method. Suppose we are to retrieve $K$ rows with filtering:

semester4/dmdb/parts/06_data/01_cloud-systems/01_key-value-store.tex

@@ -0,0 +1,77 @@
+\subsection{Key Value Store}
+\label{sec:key-value-store}
+The rise of the internet has shown that there is the need for a system that can more quickly retrieve data than the older implementations using blocks, files and objects.
+The goal is to be able to access a chunk of data that does not need to be parsed (unlike objects), but can be of various sizes (unlike blocks)
+and needs no system-supported names (like file systems).
+
+The solution to this problem is called a \textit{Key Value Store} (KVS).
+
+Most \acrshort{sql} databases follow the \acrshort{acid} concept to maintain consitency, etc.
+The \acrshort{acid} concept gives very strong guarantees, thus the work is done by the database engine as opposed to the application.
+However, it is expensive, both in throughput and latency and does not scale well.
+
+Key-Value stores get rid of EVERYTHING.
+They are schema-less, \acrshort{nosql}, no \acrshort{acid}, (typically) \acrshort{base} based stores.
+The most well known examples of \acrshort{nosql} DBs are \texttt{Redis} and \texttt{MongoDB},
+with \texttt{Redis} being a true Key-Value Store (however usually referred to as a Key-Value Cache).
+
+\subsubsection{Architecture}
+\begin{itemize}
+    \item Data is stored as a key-value pair. The key is used to locate and retrieve the data. The value is a binary object of arbitrary size
+    \item The data is organized as a distributed hash table. We hash the key to get the location for the value
+    \item KV pairs stored in several places for availability
+    \item Access is \gls{quorum}-based for consistency
+    \item Typically eventual consistency
+    \item The physical storage uses already reserved buckets of different sizes and objects are located in the corresponding buckets.
+    \item This can be run in main memory for very fast access, with larger blobs stored in storage.
+\end{itemize}
+
+
+\subsubsection{Advantages and Disadvantages}
+\begin{multicols}{2}
+    \bi{Advantages}
+    \begin{itemize}
+        \item Very fast lookups (due to hashing)
+        \item Easy to scale to many machines (simply add more copies / machines). This is great for the cloud's elasticity
+        \item Can be implemented in main memory of machines (giving even faster lookups)
+        \item Useful in many applications (looking up user profiles, retrieving user's web pages and info)
+        \item Scalable and easy to get going (no schema needed)
+    \end{itemize}
+
+    \bi{Disadvantages}
+    \begin{itemize}
+        \item No easy support for queries beyond simple key requests
+        \item this means that there are no joins, no ranges, no queries on attributes, etc (because there are none)
+        \item Depending on implementation, data can be inconsistent
+        \item Maintaining consistency is up to the application
+        \item Used in large scale deployments (worldwide), so keeping data in sync can be challenging
+    \end{itemize}
+\end{multicols}
+As mentioned above, one major drawback of \acrshort{nosql} DBs is that they move the complexity to the application.
+This means that some operations are very costly to implement (range queries, joins, etc), as they require reading all data from the DB.
+
+
+\subsubsection{Uses of KVS}
+As a result of the above disadvantages, KVS are typically used as a cache or specialized system and not as a replacement for \acrshort{sql} databases.
+
+They are often used as a main memory cache to avoid having to access the cloud storage, resulting in higher throughput and lower latency.
+
+This is especially useful for commonly run queries, such as loading all videos of a commonly accessed YouTube Channel.
+In that case especially it is easy to invalidate the cache line and updating it, as updates are very rare and they can trigger a cache invalidation for that channel.
+
+
+\subsubsection{Distributed KVS}
+How do we distribute a KVS across multiple systems?
+A simple strategy would be to assign data to machines using modulo $n$ for $n$ machines on the hash.
+In theory that is not a bad idea, but what if we remove one machine? Or if we add one? Then we have to move ALL of the data.
+If one machine fails, in the worst case, all data could possibly need to be re-hashed, possibly moving ALL of the data to different machines.
+
+A better solution is to use horizontal partitioning of the data using hashing.
+Each machine deals with the data points in the clockwise direction before the next machine.
+In other words, each machine deals with the data in a certain range of hashes, the next machine deals with the subsequent ones, etc.
+
+Thus, on fail, or exit in any other way, only the machine after the failed / exited machine needs to modify the data.
+When adding nodes, the next machine hands some of the responsibilities over to the newly inserted machine.
+
+To account for possible failure of machines, we need to keep multiple copies over multiple machines (ahead in counter-clockwise orientation)
+to prevent data loss.

semester4/dmdb/parts/06_data/01_cloud-systems/02_cloud-native-databases.tex

@@ -0,0 +1,10 @@
+\subsection{Cloud-Native Databases}
+Many of the traditional database engines may run into overhead due to heavy I/O traffic, especially when mirroring.
+
+As a result, many companies have improved implementations of SQL databases to make them suitable to cloud infrastructure.
+
+Some of the improvements include caching. Micro-partitions in the Snowflake system is a system to facilitate query processing.
+Each one of these partitions has sizes between 50 and 500 megabytes, and has metadata describing what is inside, which can also be read without reading the entire partition.
+
+Furthermore, Amazon S3, the storage platform, does not support in-place updates, files are replaced in their entirety.
+This means that old data can easily be recovered.

semester4/dmdb/parts/06_data/01_cloud-systems/03_future.tex

@@ -0,0 +1,10 @@
+\subsection{The Future}
+``The IT industry is, above all, an industry''. This is very much true and trends and developments are driven by demand and market.
+For hardware, the investment is very high for new developments and only pay off at large sale volumes.
+Hardware moves slowly, software even slower.
+
+Furthermore, the number of foundries has gone down significantly since 2002 (130nm architecture),
+where today, there are only three bleeding edge foundries left, of which one has over 50\% market share in bleeding edge semiconductor manufacturing, TSMC.
+Samsung and Intel both have lost out on major deals due to a number of issues (Nvidia Ampere was the last major GPU series manufactured by Samsung).
+
+CPUs are low latency (very fast), but low throughput in terms of parallelism, whereas GPUs are the exact opposite.

semester4/dmdb/parts/06_data/main.tex

@@ -6,5 +6,8 @@
 
 \subsection{Cloud systems}
 \input{parts/06_data/01_cloud-systems/00_intro.tex}
+\input{parts/06_data/01_cloud-systems/01_key-value-store.tex}
+\input{parts/06_data/01_cloud-systems/02_cloud-native-databases.tex}
+\input{parts/06_data/01_cloud-systems/03_future.tex}
 % \input{parts/06_data/01_cloud-systems/}
 % \input{parts/06_data/}