diff --git a/semester4/dmdb/data-modelling-databases-summary.pdf b/semester4/dmdb/data-modelling-databases-summary.pdf index 8aeec5a..6c805d0 100644 Binary files a/semester4/dmdb/data-modelling-databases-summary.pdf and b/semester4/dmdb/data-modelling-databases-summary.pdf differ diff --git a/semester4/dmdb/glossary/acronyms.tex b/semester4/dmdb/glossary/acronyms.tex index 721c7b7..15d7c6f 100644 --- a/semester4/dmdb/glossary/acronyms.tex +++ b/semester4/dmdb/glossary/acronyms.tex @@ -1,7 +1,10 @@ \newacronym{sql}{SQL}{Structured Query Language} +\newacronym{nosql}{NoSQL}{SQL-less database (commonly a Key-Value Store, see Section~\ref{sec:key-value-store})} \newacronym{dbms}{DBMS}{Database Management System} \newacronym{simd}{SIMD}{Single Instruction Multiple Data} \newacronym{avxa}{AVX}{Advanced Vector Extensions, also see \gls{avx}} \newacronym{ra}{RA}{Relational Algebra, see Section~\ref{sec:relational-algebra}} \newacronym{ann}{ANN}{Approximate Nearest Neighbour Search} \newacronym{knn}{KNN}{$K$-Nearest Neighbour Search} +\newacronym{acid}{ACID}{Atomicitty, Consistency, Isolation, Durability} +\newacronym{base}{BASE}{Basically Available, Soft State, Eventually Consistent} diff --git a/semester4/dmdb/glossary/defs.tex b/semester4/dmdb/glossary/defs.tex index 241e0c7..3ae0adb 100644 --- a/semester4/dmdb/glossary/defs.tex +++ b/semester4/dmdb/glossary/defs.tex @@ -2,3 +2,10 @@ name={superkey}, description={See Section \ref{sec:functional-dependency}} } +\newglossaryentry{quorum}{ + name={quorum}, + description={ + See \href{https://en.wikipedia.org/wiki/Quorum_(distributed_computing)}{this Wikipedia article}. + TL;DR: It is a synchronization system for distributed computing. + } +} diff --git a/semester4/dmdb/parts/05_vector-search/03_filtered-vector-search.tex b/semester4/dmdb/parts/05_vector-search/03_filtered-vector-search.tex index 7c04d26..38ebd66 100644 --- a/semester4/dmdb/parts/05_vector-search/03_filtered-vector-search.tex +++ b/semester4/dmdb/parts/05_vector-search/03_filtered-vector-search.tex @@ -2,7 +2,7 @@ The benefit of supporting vector search in DBMS is that this makes it easy to use for the end user and can be optimized for large datasets. \subsection{Filtered Vector Search (FVS)} -This allows us to query both structured and unstructured data. In concept, this is SQL + Vectors, sometimes plus joins, subquery expressions, +This allows us to query both structured and unstructured data. In concept, this is \acrshort{sql} + Vectors, sometimes plus joins, subquery expressions, multiple vector searches (on both sides of the join). Sometimes, the dataset isn't even a vector + tag stored in main memory. Indices are typically built on unfiltered data. Thus, we need to think about the proper execution method. Suppose we are to retrieve $K$ rows with filtering: diff --git a/semester4/dmdb/parts/06_data/01_cloud-systems/01_key-value-store.tex b/semester4/dmdb/parts/06_data/01_cloud-systems/01_key-value-store.tex new file mode 100644 index 0000000..10b641a --- /dev/null +++ b/semester4/dmdb/parts/06_data/01_cloud-systems/01_key-value-store.tex @@ -0,0 +1,77 @@ +\subsection{Key Value Store} +\label{sec:key-value-store} +The rise of the internet has shown that there is the need for a system that can more quickly retrieve data than the older implementations using blocks, files and objects. +The goal is to be able to access a chunk of data that does not need to be parsed (unlike objects), but can be of various sizes (unlike blocks) +and needs no system-supported names (like file systems). + +The solution to this problem is called a \textit{Key Value Store} (KVS). + +Most \acrshort{sql} databases follow the \acrshort{acid} concept to maintain consitency, etc. +The \acrshort{acid} concept gives very strong guarantees, thus the work is done by the database engine as opposed to the application. +However, it is expensive, both in throughput and latency and does not scale well. + +Key-Value stores get rid of EVERYTHING. +They are schema-less, \acrshort{nosql}, no \acrshort{acid}, (typically) \acrshort{base} based stores. +The most well known examples of \acrshort{nosql} DBs are \texttt{Redis} and \texttt{MongoDB}, +with \texttt{Redis} being a true Key-Value Store (however usually referred to as a Key-Value Cache). + +\subsubsection{Architecture} +\begin{itemize} + \item Data is stored as a key-value pair. The key is used to locate and retrieve the data. The value is a binary object of arbitrary size + \item The data is organized as a distributed hash table. We hash the key to get the location for the value + \item KV pairs stored in several places for availability + \item Access is \gls{quorum}-based for consistency + \item Typically eventual consistency + \item The physical storage uses already reserved buckets of different sizes and objects are located in the corresponding buckets. + \item This can be run in main memory for very fast access, with larger blobs stored in storage. +\end{itemize} + + +\subsubsection{Advantages and Disadvantages} +\begin{multicols}{2} + \bi{Advantages} + \begin{itemize} + \item Very fast lookups (due to hashing) + \item Easy to scale to many machines (simply add more copies / machines). This is great for the cloud's elasticity + \item Can be implemented in main memory of machines (giving even faster lookups) + \item Useful in many applications (looking up user profiles, retrieving user's web pages and info) + \item Scalable and easy to get going (no schema needed) + \end{itemize} + + \bi{Disadvantages} + \begin{itemize} + \item No easy support for queries beyond simple key requests + \item this means that there are no joins, no ranges, no queries on attributes, etc (because there are none) + \item Depending on implementation, data can be inconsistent + \item Maintaining consistency is up to the application + \item Used in large scale deployments (worldwide), so keeping data in sync can be challenging + \end{itemize} +\end{multicols} +As mentioned above, one major drawback of \acrshort{nosql} DBs is that they move the complexity to the application. +This means that some operations are very costly to implement (range queries, joins, etc), as they require reading all data from the DB. + + +\subsubsection{Uses of KVS} +As a result of the above disadvantages, KVS are typically used as a cache or specialized system and not as a replacement for \acrshort{sql} databases. + +They are often used as a main memory cache to avoid having to access the cloud storage, resulting in higher throughput and lower latency. + +This is especially useful for commonly run queries, such as loading all videos of a commonly accessed YouTube Channel. +In that case especially it is easy to invalidate the cache line and updating it, as updates are very rare and they can trigger a cache invalidation for that channel. + + +\subsubsection{Distributed KVS} +How do we distribute a KVS across multiple systems? +A simple strategy would be to assign data to machines using modulo $n$ for $n$ machines on the hash. +In theory that is not a bad idea, but what if we remove one machine? Or if we add one? Then we have to move ALL of the data. +If one machine fails, in the worst case, all data could possibly need to be re-hashed, possibly moving ALL of the data to different machines. + +A better solution is to use horizontal partitioning of the data using hashing. +Each machine deals with the data points in the clockwise direction before the next machine. +In other words, each machine deals with the data in a certain range of hashes, the next machine deals with the subsequent ones, etc. + +Thus, on fail, or exit in any other way, only the machine after the failed / exited machine needs to modify the data. +When adding nodes, the next machine hands some of the responsibilities over to the newly inserted machine. + +To account for possible failure of machines, we need to keep multiple copies over multiple machines (ahead in counter-clockwise orientation) +to prevent data loss. diff --git a/semester4/dmdb/parts/06_data/01_cloud-systems/02_cloud-native-databases.tex b/semester4/dmdb/parts/06_data/01_cloud-systems/02_cloud-native-databases.tex new file mode 100644 index 0000000..06bf193 --- /dev/null +++ b/semester4/dmdb/parts/06_data/01_cloud-systems/02_cloud-native-databases.tex @@ -0,0 +1,10 @@ +\subsection{Cloud-Native Databases} +Many of the traditional database engines may run into overhead due to heavy I/O traffic, especially when mirroring. + +As a result, many companies have improved implementations of SQL databases to make them suitable to cloud infrastructure. + +Some of the improvements include caching. Micro-partitions in the Snowflake system is a system to facilitate query processing. +Each one of these partitions has sizes between 50 and 500 megabytes, and has metadata describing what is inside, which can also be read without reading the entire partition. + +Furthermore, Amazon S3, the storage platform, does not support in-place updates, files are replaced in their entirety. +This means that old data can easily be recovered. diff --git a/semester4/dmdb/parts/06_data/01_cloud-systems/03_future.tex b/semester4/dmdb/parts/06_data/01_cloud-systems/03_future.tex new file mode 100644 index 0000000..3bcad3a --- /dev/null +++ b/semester4/dmdb/parts/06_data/01_cloud-systems/03_future.tex @@ -0,0 +1,10 @@ +\subsection{The Future} +``The IT industry is, above all, an industry''. This is very much true and trends and developments are driven by demand and market. +For hardware, the investment is very high for new developments and only pay off at large sale volumes. +Hardware moves slowly, software even slower. + +Furthermore, the number of foundries has gone down significantly since 2002 (130nm architecture), +where today, there are only three bleeding edge foundries left, of which one has over 50\% market share in bleeding edge semiconductor manufacturing, TSMC. +Samsung and Intel both have lost out on major deals due to a number of issues (Nvidia Ampere was the last major GPU series manufactured by Samsung). + +CPUs are low latency (very fast), but low throughput in terms of parallelism, whereas GPUs are the exact opposite. diff --git a/semester4/dmdb/parts/06_data/main.tex b/semester4/dmdb/parts/06_data/main.tex index 044a612..6bec7d9 100644 --- a/semester4/dmdb/parts/06_data/main.tex +++ b/semester4/dmdb/parts/06_data/main.tex @@ -6,5 +6,8 @@ \subsection{Cloud systems} \input{parts/06_data/01_cloud-systems/00_intro.tex} +\input{parts/06_data/01_cloud-systems/01_key-value-store.tex} +\input{parts/06_data/01_cloud-systems/02_cloud-native-databases.tex} +\input{parts/06_data/01_cloud-systems/03_future.tex} % \input{parts/06_data/01_cloud-systems/} % \input{parts/06_data/}