[DMDB] Add concurrency control and recovery

Oops, luckily still figured out that that was missing...

semester4/dmdb/data-modelling-databases-summary.tex

@@ -84,8 +84,9 @@ In addition, the end user does not have to worry about optimization too much. Th
 \include{parts/02_theory-background/main.tex}
 \include{parts/03_systems/main.tex}
 \include{parts/04_query-processing/main.tex}
-\include{parts/05_vector-search/main.tex}
+\include{parts/05_concurrency-control-recovery/main.tex}
-\include{parts/06_data/main.tex}
+\include{parts/06_vector-search/main.tex}
+\include{parts/07_data/main.tex}
 
 
 \printGlossary

semester4/dmdb/glossary/acronyms.tex

@@ -6,5 +6,5 @@
 \newacronym{ra}{RA}{Relational Algebra, see Section~\ref{sec:relational-algebra}}
 \newacronym{ann}{ANN}{Approximate Nearest Neighbour Search}
 \newacronym{knn}{KNN}{$K$-Nearest Neighbour Search}
-\newacronym{acid}{ACID}{Atomicitty, Consistency, Isolation, Durability}
+\newacronym{acid}{ACID}{Atomicitty, Consistency, Isolation, Durability, also see Section~\ref{sec:acid}}
 \newacronym{base}{BASE}{Basically Available, Soft State, Eventually Consistent}

semester4/dmdb/parts/05_concurrency-control-recovery/00_transactions.tex

@@ -0,0 +1,9 @@
+\subsection{Transactions}
+Transactions are accesses to the database with the purpose of \textit{modifying the data}.
+The focus is of course on correctness, recovery and state management.
+
+Transactions typically are expensive compared to queries due to the focus on correctness.
+
+This means we need to employ concurrency control (ensuring data remains consistent with transactions and queries running at the same time,
+definition of consistency, correctness, conflicts and problematic situations, as well as locking).
+We furthermore want to implement recovery, to ensure that data remains consistent even when unexpected failures occur.

semester4/dmdb/parts/05_concurrency-control-recovery/01_acid.tex

@@ -0,0 +1,44 @@
+\subsection{ACID}
+\label{sec:acid}
+\acrshort{acid} is the conventional notion of correctness (informal because it's an acronym):
+\begin{itemize}
+    \item \bi{Atomicity}: A group of operations must take place in their entirety or not at all
+    \item \bi{Consistency}: Operations should take the database from a correct state to another correct state
+    \item \bi{Isolation}: Concurrent execution of operations should yield predictable and correct results
+    \item \bi{Durability}: The DB needs to remember the state it is in at all moments, even when failures occur
+\end{itemize}
+
+\inlinedefinition[Database state] is the actual values stored in a database at a given point in time (in memory and storage).
+
+\inlinedefinition[Consistent state] is a database state that is the result of \textit{correctly} applying \textit{operations} to the DB.
+
+\inlinedefinition[Correct application] The model of correct executions of operations is sequential executions.
+
+\inlinedefinition[Operations] The operations for transactions are inserts, updates and deletion of tuples.
+
+\inlinedefinition[Consistency] The assumption of correctness of transactions is based on two things:
+\bi{(1)} the user doesn't apply wrong transactions and \bi{(2)} the DBMS has mechanisms to prevent incorrect modifications, such as violations of constraints, etc.
+
+\inlinedefinition[Durability] ensures that the changes of a completed transaction are remembered by the system and can be recovered.
+This is often more strict and DBMS will keep the DB st ate and a history of all transactions.
+
+The transaction history is captured in the DB log. The state at a given moment is captured in snapshots.
+The log and snapshots allow to go back and forward in the history of the database by undoing or redoing transactions from a given snapshot.
+Thus, very similar in concept to what \texttt{git} uses.
+
+
+\subsubsection{Atomicity}
+\inlinedefinition[Atomicity] dictates that a transaction only takes effect if it can be executed in its entirety, or in other words,
+a transaction has to be executed in its entirety or not at all.
+The issue with this is that violations are costly to correct.
+
+Thus, atomicity requires isolation. If a transaction can see the intermediate state created by another transaction, then its input is not guaranteed to be consistent,
+implying that its output is not guaranteed to be consistent, either.
+
+The DBMS get around this issue by isolating the transactions, such that they behave as if they were alone in the database.
+This of course means that we need to know when a transaction is completed so we can \textit{commit} it, if there are no conflicts.
+If there are conflicts, we have to resolve them, rerun the transaction (note that this can be very expensive) or to prevent them altogether.
+
+The isolation is enforced through concurrency control mechanisms to prevent conflicts.
+The canonical concurrency control mechanism in DBs is locking.
+We use locking policies to ensure that the transactions are isolated.

semester4/dmdb/parts/05_concurrency-control-recovery/02_transaction-model.tex

@@ -0,0 +1,35 @@
+\subsection{Transaction Model}
+\inlinedefinition Some terms for Transaction model:
+\begin{itemize}
+    \item \bi{Begin of transaction (BOT)}: Often implicit
+    \item \bi{Commit}: Transaction has finished, database confirms to client when all changes of the transaction have been made persistent
+    \item \bi{Abort/rollback}: transaction is cancelled, all changes made already are rolled back
+    \item $a <_T b$: $a$ happens before $b$, is a \textit{partial order}, which implies that $a$ will be done before $b$.
+\end{itemize}
+
+In real systems, transactions are often associated with sessions and all SQL statements after an update statement are considered to be part of the transaction.
+This means they are explicitly managed, using a \texttt{BEGIN TRANSACTION} or \texttt{BEGIN} statement and ended with either a \texttt{COMMIT} or \texttt{ROLLBACK} statement.
+
+So, we have the following statements:
+\begin{itemize}
+    \item \texttt{COMMIT}: Transaction will not make any order
+    \item \texttt{ROLLBACK}: Roll back all changes
+    \item \texttt{SAVEPOINT $<$name$>$}: Create a save point / history point, used to roll back some changes and try some others in case of failure.
+    \item \texttt{ROLLBACK TO SAVEPOINT $<$name$>$}: Roll back to a specific save point.
+    \item \texttt{RELEASE SAVEPOINT $<$name$>$}: Delete a save point (doesn't delete the changes made before, just the log entry)
+\end{itemize}
+
+\subsubsection{Operations}
+Below a list of very important syntax for the exam (where the subscript $i$ denotes that this operation is part of transaction $T_i$):
+\begin{itemize}
+    \item Read operations: $r_i[x]$ or $r_i(x)$ is an access to tuple $x$ without modifying it.
+    \item Write operations: $w_i[x]$ or $r_i(x)$ represents a write operation on the tuple $x$.
+    \item Abort: $a_j$ is an abort of transaction $T_j$
+    \item Commit $c_j$ is the commit of transaction $T_j$
+\end{itemize}
+Conflicting operations are two operations on the same item with at least one of them being a write operation, such as $r_1[x] w_2[x]$ or $w_1[y] w_2[y]$.
+
+\subsubsection{History}
+A history $H$ is a partially ordered $<_H$ sequence of operations from a set of transactions.
+If two operations are ordered within a transaction, they are equally ordered in the history.
+If two operations $p$ and $q$ conflict, then they are ordered with respect to each other ($p <_H q$ or $q <_H p$)

semester4/dmdb/parts/05_concurrency-control-recovery/03_concurrency-control.tex

@@ -0,0 +1,39 @@
+\newpage
+\subsection{Concurrency Control}
+The goal of concurrency control is to ensure the correct executions of concurrent transactions.
+The baseline for correctness is, as previously mentioned, serial execution.
+
+\inlinedefinition[Serial History] A history $H$ is serial if, for every pair of two transactions $T_i$ and $T_j$ that appear in $H$,
+either all operations from $T_i$ appear before all all operations of $T_j$, or vice versa.
+A serial history with only committed transactions is correct by definition, because transactions are isolated and each transaction starts in a consistent state
+and leaves the DB in a consistent state.
+
+\inlinedefinition[Equivalent History] Two histories are equivalent if and only if
+\begin{enumerate}
+    \item They are over the same transactions and contain the same operations
+    \item Conflicting operations of non-aborted transactions are ordered in the same way in both histories
+\end{enumerate}
+
+\inlinedefinition[Serializable History] A history is serializable if and only it is equivalent to a serial history.
+
+\inlinetheorem[Serializability] A history is serializable if and only if its serializability graph is acyclic.
+The seerializabililty graph is constructed from the history graph as follows:
+\begin{enumerate}
+    \item The start node is the transaction the first operation.
+    \item Then, follow the operations, adding any new transaction to the tree and connect any further dependencies.
+\end{enumerate}
+More visually, see the below graphics
+
+\begin{figure}[h!]
+    \begin{center}
+        \begin{subfigure}{0.49\textwidth}
+            \includegraphics[width=0.9\textwidth]{assets/serializability-graph-history.png}
+            \caption{The history}
+        \end{subfigure}
+        \begin{subfigure}{0.49\textwidth}
+            \includegraphics[width=0.7\textwidth]{assets/serializability-graph-built.png}
+            \caption{The resulting serializability graph}
+        \end{subfigure}
+    \end{center}
+    \caption{Building a serializability graph (Figure from lecture slides 19, slide 37 and 38)}
+\end{figure}

semester4/dmdb/parts/05_concurrency-control-recovery/04_two-phase-locking.tex

@@ -0,0 +1,54 @@
+\subsection{2 Phase Locking (2PL)}
+This type of locking is widely used since many decades and is the canonical implementation of concurrency control.
+Of course, alternatives exist and the locking implementation has a huge impact on performance and scalability.
+
+Concurrency control is implemented using locking, typically using a locking table.
+The locks cover tuples, tables, indexes, blocks and more.
+
+Locks have different semantics that indicate whether they can be shared or are exclusive.
+
+Compatibility matrix:
+\begin{tables}{cccc}{ & NL         & S          & X}
+              S       & \checkmark & \checkmark & -  \\
+              X       & \checkmark & -          & -  \\
+\end{tables}
+
+\subsubsection{The protocol}
+\begin{enumerate}
+    \item Before accessing an object, a transaction must acquire lock
+    \item A transaction acquires a lock only once. Lock upgrades are possible
+    \item A transaction is blocked if the lock request cannot be granted according to the compatibility matrix
+    \item A transaction goes through two phases:
+          \begin{enumerate}
+              \item Growth: Acquire locks, but never release locks
+              \item Shrink: Release locks, but never acquire locks
+          \end{enumerate}
+    \item At EOT (commit or abort), all locks must be released.
+\end{enumerate}
+
+The reason for the two phases is that 2PL prevents histories where a transaction must be aborted due to concurrency control issues.
+An operation is thus not allowed unless we are sure it is safe.
+
+Without the two phases, the following can happen:
+$L_1[x] w_1[x] U_1[x] L_2[x] w_2[x] U_2[y] L_2[y] w_2[y]$, etc, thus meaning that if $T_1$ now access $y$ or $x$ for read or write, the history becomes non-serializable.
+
+The basic 2PL doesn't say anything about aborting or committing transactions.
+The issue is that if we are to undo the changes of $T_1$, we would also undo some of the changes from $T_2$, which would then in turn not be restorable anymore.
+
+\subsubsection{Strict 2PL}
+Strict 2PL combines 2PL with strict histories. This means that we enforce 2PL, but we also keep the read and write locks until the transaction commits or aborts.
+This results in serializable and strict histories.
+All DB engines implement strict 2PL.
+
+
+\subsubsection{Deadlocks Detection}
+In theory, we could use a wait-for graph, which is however expensive.
+For example, for $T_1 \rightarrow T_2 \rightarrow T_3 \rightarrow T_4 \rightarrow T_1$, we could abort $T_3$ to resolve the cycle.
+
+In practice, no wait-for graphs are built. Transactions and queries have timers. If the timer expires, the transaction or query is aborted.
+Deadlocks are considered rare enough to prefer making mistakes every now and then (aborting a transaction that was fine),
+rather than implementing something as expensive as a wait-for graph.
+
+Implications from this are for example that long transactions may block queries and that those queries may abort because the timer expired.
+
+However, this is only common in poorly written applications.

semester4/dmdb/parts/05_concurrency-control-recovery/05_snapshot-isolation.tex

@@ -0,0 +1,17 @@
+\subsection{Snapshot isolation}
+This was initially used by Oracle and is a form of Multi-Version Concurrency Control (MVCC).
+
+MVCC applies serializability, but writes generate a new version and reads read from a concrete version of the DB.
+This means that the whole DB can be implemented (almost) entirely without locks.
+
+When a transaction starts, it receives a timestamp $T$.
+Read operations only have access to items committed before $T$.
+All write operations are carried out in a separate buffer (shadow paging) and only become visible after a commit.
+When a transaction commits, it checks for conflicts. If there are, the first-committer-wins rule is applied, i.e. if we try to commit and there are conflicts,
+this transaction has to rerun (or be aborted).
+
+However, there are scenarios, such as $r_1[x] r_1[y] r_2[x] r_2[y] w_1[x] w_2[x] c_1 c_2$, where the history is not serializable:
+$T_2 \rightarrow T_1 \rightarrow T_2$.
+Depending on the application, this can still be correct (if this is for a ticketing system), but can also be disastrous (for e.g. a bank).
+
+If done properly however, it has significant performance advantages, as well as in distributed settings with replication.

semester4/dmdb/parts/05_concurrency-control-recovery/06_recovery.tex

@@ -0,0 +1,77 @@
+\subsection{Recovery}
+Recovery has to make sure that, even in the case of failures, the changes from transactions that have not completed are not visible
+and those of committed transactions are recorded and visible.
+We define undesirable situations and classify histories according to these situations.
+Furthermore, we want to define how histories can avoid such situations.
+
+
+\subsubsection{Recovery procedures}
+\begin{enumerate}[label=R\arabic*]
+    \item Undo all changes from transactions
+    \item Redo the changes from committed transactions
+    \item Undo the changes that remain in the system from active transactions
+    \item Read consistent snapshot from backup, and, if available, apply log
+\end{enumerate}
+R1 is used for abort/rollback of single transactions, R2 and R3 are used on system crash (where we lose the memory, but disks survive)
+and R4 is used for recovery from a system crash with disk loss.
+
+This means that we can undo the changes of an active transaction (we keep a copy of the value before it was modified; Shadow pages is a mechanism that would support this),
+we can redo changes made by committed transactions (we keep a persistent record of all the changes made by committed transactions, this why there is a redo log)
+and we have a consistent snapshot to start with (either move forward by taking a snapshot and apply committed transactions,
+or move backward by taking the current state and go back to a consistent state by undoing changes from transactions that should be there).
+
+
+\subsubsection{Undo - Redo}
+The changes of an aborted transaction are undone typically by restoring the \bi{before image} of the modified value.
+
+Changes are redone by restoring the \bi{after image} of the modified value.
+
+Databases typically log transactions by keeping before and after images of their changes.
+
+
+\subsubsection{Recovery on histories}
+A transaction $T_1$ reads from another transaction $T_2$ if $T_1$ reads a value written by $T_2$ at a time when $T_2$ was not aborted.
+
+We have:
+\begin{itemize}
+    \item \bi{Recoverable} (RC) history: If $T_i$ reads from $T_j$ and commits, then $c_j < c_i$
+    \item \bi{Avoids Cascading Aborts} (ACA) history: If $T_i$ reads $x$ from $T_j$, then $c_j < r_i[x]$
+    \item \bi{Strict} (ST) history: If $T_i$ reads from, or overwrites, a value written by $T_j$ then $c_j < r_i[x] / w_i[x]$ or $a_j < r_i[x] / w_i[x]$
+\end{itemize}
+
+
+\paragraph{Recoverable}
+There is no need to undo a committed transaction because it read the wrong data and they commit in their serialization order.
+
+Suppose we have the following sequence: $\ldots w_2[x] r_1[x] c_1 \ldots$.
+If $T_2$ aborts, then $T_1$ has read invalid data and since $T_1$ has committed, the data has returned to the user.
+Recovery now involves correcting the application that has processed the data and might have acted upon it, which typically is a very expensive procedure.
+This is avoided by making sure that $T_1$ does not commit until $T_2$ commits and if $T_2$ aborts, then we can safely abort $T_1$,
+since the results have not been returned to the user yet.
+
+
+\paragraph{ACA}
+Aborting a transaction does not cause aborting others and transactions only read from committed transactions
+
+Suppose we have the following sequence: $\ldots w_2[x] r_1[x] a_2 \ldots$.
+When $T_2$ aborts, we have to abort $T_1$ because it has read invalid data (the change made by $T_2$, which will be undone as part of rollback of $T_2$).
+If those cascading aborts happen regularly, the performance will degrade and possibly further transactions will be aborted.
+This is avoided by making sure that uncommitted data is never read (\textit{read committed}).
+
+
+\paragraph{Strict}
+Undoing a transaction does not undo the changes of other transactions. Transactions further don't read or overwrite updates of uncommitted transactions.
+
+Suppose we have the following sequence: $\ldots w_2[x] w_1[x] a_2 \ldots$.
+If we undo $w_2[x]$, we would also remove the changes made by $T_1$. Thus, undoing $T_2$ implies aborting $T_1$.
+Like in ACA, non-strict execution leads to cascading aborts of transactions.
+This is avoided by not letting a value be read or updated unless it is committed.
+
+
+\paragraph{Why recoverability matters}
+Fixing errors with recovery typically are very expensive:
+\begin{itemize}
+    \item Not RC: running my program issues a report, but the data that I got was later removed because a different program aborted $\rightarrow$ report is invalid
+    \item Not ACA: Thrashing behaviour when transactions keep aborting each other (my program slows down, or makes no progress at all because of other programs)
+    \item Not Strict: recovery after a failure becomes very complex (or impossible)
+\end{itemize}

semester4/dmdb/parts/05_concurrency-control-recovery/07_recovery-procedures.tex

@@ -0,0 +1,127 @@
+\subsection{Recovery Procedures}
+\inlinedefinition[Recovery Manager] A recovery manager implements the recovery procedure, which depends on how the system functions.
+It may or may not require both, one or none of UNDO and REDO.
+
+\subsubsection{From the buffer cache's perspective}
+\begin{itemize}
+    \item \bi{Steal Policy}: An uncommitted transaction is allowed to overwrite the changes of a committed transaction in persistent storage.
+          This logically means that we need to be able to undo transactions.
+    \item \bi{Force Policy}: All changes made by a transaction must be in persistent storage before the transaction commits.
+          This requires flushing all blocks with updates from the transaction and if not in-place, it will require being able to redo the transaction.
+    \item \bi{Steal/No-Force}: UNDO/REDO, is the most common approach
+\end{itemize}
+If for updates tuples are locked, blocks may still be updatable.
+Thus if the block is copied to storage, it is possible that some changes are committed while others are not.
+If failures occur, there is no guarantee that the data in storage is 100\% consistent, thus requiring undo and redo.
+
+\subsubsection{Recovery Procedure for UNDO/REDO}
+\begin{itemize}
+    \item READ: simply read the value from the block in the buffer cache
+    \item WRITE: Create log entry (before and after image), append it to persistent log and write the after image to the block on the buffer cache.
+    \item COMMIT: We write a persistent log entry indicating that the transaction has committed
+    \item ABORT: For all updates, we restore the before image using the log entry
+\end{itemize}
+
+The Procedure:
+\begin{enumerate}
+    \item Start from the end of the log and work backwards
+    \item Keep a list of UNDONE items and another for REDONE items
+    \item Procedure terminates when all items are in either the UNDONE or REDONE list or we reach the beginning of the log.
+    \item For each log entry, we look at the data item $x$ being accessed and if $x$ is in none of the two lists:
+          \begin{itemize}
+              \item If the log entry is of a committed transaction, apply the after image, add $x$ to the REDONE list
+              \item If the log entry is of an aborted transaction, apply the before image, add $x$ to the UNDONE list
+          \end{itemize}
+\end{enumerate}
+The procedure ignores the data stored on disk as it could correspond to uncommitted transactions, it only takes it as starting point for the recovery.
+An alternative way of thinking about the procedure is as follows:
+\begin{algorithm}
+    \caption{Undo-Redo Recovery}
+    \begin{algorithmic}[1]
+        \Procedure{UndoRedoRecovery}{}
+            \For{each item in the DB}
+                \State Find last committed transaction modifying the item and REDO it
+                \If{no committed transaction modified item}
+                    \State Find the first aborted transaction that modified the item and \texttt{UNDO} the modification
+                \EndIf
+                \If{If no transaction has touched the item}
+                    \State Value is correct (we assume that we start from a consistent state)
+                \EndIf
+            \EndFor
+        \EndProcedure
+    \end{algorithmic}
+\end{algorithm}
+This is not the way it's done in practice, because there typically are many more items than log entries.
+
+\shade{ForestGreen}{Advantages} Only forced I/O are log records, buffer cache manager has a lot of freedom
+(no flushing dirty pages, I/O on data is minimized and only triggered for replacement policies) and we allow writing dirty data to disk, simplifying buffer management.
+
+However, recovery is more complicated and takes more time, but normal operations are only minimally affected.
+Furthermore, queries are not affected since there is no forced I/O of data.
+Transactions are affected, because we need to write every operation to the log.
+
+
+\subsubsection{UNDO, no REDO}
+\begin{itemize}
+    \item READ: simply read the value from the block in the buffer cache
+    \item WRITE: Create log entry (before image), append it to persistent log and write the after image to the block on the buffer cache.
+    \item COMMIT: Flush all dirty values modified by transaction if still in cache. We write a persistent log entry indicating that the transaction has committed
+    \item ABORT: For all updates, we restore the before image using the log entry
+\end{itemize}
+The Procedure:
+\begin{enumerate}
+    \item Start from the end of the log and work backwards
+    \item Keep a list of UNDONE items
+    \item Procedure terminates when all items are in either the UNDONE list or we reach the beginning of the log.
+    \item For each log entry, we look at the data item $x$ being accessed and if $x$ is not in the UNDONE list and the transaction is aborted,
+          we UNDO the changes by using the before image, adding $x$ to the UNDONE list
+\end{enumerate}
+It relies on the fact that all committed values are in persistent storage and therefore, they have not been lost.
+
+However, it has forced I/O on all dirty blocks touched by the transaction when it commits,
+log records no longer need to include after images, reducing log record size and the recovery procedure is shorter because it only involves undoing aborted transactions.
+
+However, this trade-off does not pay off in practice, as we still need to write the log with every update and while it requires smaller log records,
+it forces I/O on data blocks, which are often much larger.
+And finally, flushing the buffer cache interferes with its operations.
+
+
+\subsubsection{no UNDO, REDO}
+\begin{itemize}
+    \item READ: If the transaction did not write the item before, read the value from the block in the buffer cache. Otherwise, read the value from the temporary buffer
+    \item WRITE: Create log entry (after image), append it to persistent log and write the after image to the block on the buffer cache.
+    \item COMMIT: Apply all updates in the temporary buffer to the actual blocks. We write a persistent log entry indicating that the transaction has committed
+    \item ABORT: Discard the temporary buffer
+\end{itemize}
+The Procedure:
+\begin{enumerate}
+    \item Start from the end of the log and work backwards
+    \item Keep a list of REDONE items
+    \item Procedure terminates when all items are in either the REDONE list or we reach the beginning of the log.
+    \item For each log entry, we look at the data item $x$ being accessed and if $x$ is not in the REDONE list and the transaction is committed,
+          we REDO the changes by using the after image, adding $x$ to the REDONE list
+\end{enumerate}
+It relies on the fact that there are never dirty blocks in the buffer cache. All data there is committed and is the last committed version.
+
+It has forced I/O only on the log records and they are also smaller.
+Furthermore, the recovery procedure is shorter, only involving redoing the last committed transaction that touched the item.
+It is similar to snapshot isolation.
+
+
+\subsubsection{No UNDO, no REDO}
+This procedure is rarely used in conventional DBs. It does not require a log.
+However, it requires to be able to write all changes made by a transaction to persistent storage in \bi{a single atomic action}.
+It relies on directories to keep track of the changes.
+\begin{itemize}
+    \item READ: If the transaction did not write the item before, use current directory to find the latest committed copy.
+          Otherwise, read the value from the shadow directory of that transaction to find the updated copy
+    \item WRITE: Write to a buffer and add a pointer in the shadow directory for the transactions
+    \item COMMIT: Create a full directory by merging the current one and the shadow directory of the transaction.
+          Then, we swap the pointer indicating the latest committed directory.
+    \item ABORT: Discard the buffer(s) and the shadow directory
+\end{itemize}
+There is no recovery procedure here, which is the exact point.
+
+It is not used in practice, although some ideas can be applied partially.
+One of the reasons for that is expensive pointer indirections for accesses and requires garbage collection and moves data all the time,
+which poses a challenge for clustered indexes or hash clustered tables.

semester4/dmdb/parts/05_concurrency-control-recovery/08_implementation-of-recovery.tex

@@ -0,0 +1,38 @@
+\subsection{Implementation of Recovery}
+The most common implementation of these ideas is Write Ahead Logging (WAL).
+There, we separate the storage used for the log and data. The log contains enough information for whatever policy is chosen and log records corresponding to a change in the DB
+must be written to the log before changes to the data in the buffer cache are flushed to permanent storage.
+The COMMIT record in the log is us ed to mark the end of a transaction.
+
+This is typically used to implement UNDO/REDO (on 2PL-based systems) or no-UNDO/REDO (on Snapshot Isolation (SI)-based systems)
+
+
+\subsubsection{Checkpoints}
+A log would grow forever if we don't do anything. Storage would then become an issue and recovery time would increase.
+Instead, Checkpoints are used. At a checkpoint, we do the following:
+\begin{itemize}
+    \item Push all dirty blocks to the disk
+    \item Push all the logs in the log buffer to disk
+    \item We now have an active translation table and dirty page table
+    \item We mark the log with a checkpoint label and flush it to the log
+\end{itemize}
+Then, recovery happens from a checkpoint instead from the beginning. However note that a checkpoint is not necessarily a consistent copy of the DB.
+
+
+\subsubsection{Recovery with WAL and checkpoints}
+Also called ARIES style recovery, if we use WAL, the recovery with a checkpoint works as follows:
+
+\begin{algorithm}
+    \caption{WAL checkpoint recovery}
+    \begin{algorithmic}[1]
+        \Procedure{WALCheckpointRecovery}{}
+            \State Find latest completed checkpoint in the log
+            \While{Traversing the log to the end}
+                \State identify active transactions at time of crash and dirty pages that may not have made it to disk at time of crash for each element
+            \EndWhile
+            \State Apply all updates (REDO) starting from the log entry matching the lowest SCN in the dirty pages
+            \State UNDO all transactions that were active at the time of the crash
+        \EndProcedure
+    \end{algorithmic}
+\end{algorithm}
+

semester4/dmdb/parts/05_concurrency-control-recovery/main.tex

@@ -0,0 +1,12 @@
+\newpage
+\section{Concurrency Control and Recovery}
+\input{parts/05_concurrency-control-recovery/00_transactions.tex}
+\input{parts/05_concurrency-control-recovery/01_acid.tex}
+\input{parts/05_concurrency-control-recovery/02_transaction-model.tex}
+\input{parts/05_concurrency-control-recovery/03_concurrency-control.tex}
+\input{parts/05_concurrency-control-recovery/04_two-phase-locking.tex}
+\input{parts/05_concurrency-control-recovery/05_snapshot-isolation.tex}
+\input{parts/05_concurrency-control-recovery/06_recovery.tex}
+\input{parts/05_concurrency-control-recovery/07_recovery-procedures.tex}
+\input{parts/05_concurrency-control-recovery/08_implementation-of-recovery.tex}
+% \input{parts/05_concurrency-control-recovery/}

semester4/dmdb/parts/05_vector-search/main.tex

@@ -1,6 +0,0 @@
-\section{Vector Search}
-\input{parts/05_vector-search/00_intro-trees-clustering.tex}
-\input{parts/05_vector-search/01_hsnw-index.tex}
-\input{parts/05_vector-search/02_hash-index.tex}
-\input{parts/05_vector-search/03_filtered-vector-search.tex}
-% \input{parts/05_vector-search/}

semester4/dmdb/parts/06_data/main.tex

@@ -1,13 +0,0 @@
-\section{Data}
-\subsection{Formats}
-\input{parts/06_data/00_formats/00_intro.tex}
-\input{parts/06_data/00_formats/01_encodings.tex}
-% \input{parts/06_data/00_formats/}
-
-\subsection{Cloud systems}
-\input{parts/06_data/01_cloud-systems/00_intro.tex}
-\input{parts/06_data/01_cloud-systems/01_key-value-store.tex}
-\input{parts/06_data/01_cloud-systems/02_cloud-native-databases.tex}
-\input{parts/06_data/01_cloud-systems/03_future.tex}
-% \input{parts/06_data/01_cloud-systems/}
-% \input{parts/06_data/}

semester4/dmdb/parts/06_vector-search/main.tex

@@ -0,0 +1,6 @@
+\section{Vector Search}
+\input{parts/06_vector-search/00_intro-trees-clustering.tex}
+\input{parts/06_vector-search/01_hsnw-index.tex}
+\input{parts/06_vector-search/02_hash-index.tex}
+\input{parts/06_vector-search/03_filtered-vector-search.tex}
+% \input{parts/06_vector-search/}

semester4/dmdb/parts/07_data/main.tex

@@ -0,0 +1,13 @@
+\section{Data}
+\subsection{Formats}
+\input{parts/07_data/00_formats/00_intro.tex}
+\input{parts/07_data/00_formats/01_encodings.tex}
+% \input{parts/07_data/00_formats/}
+
+\subsection{Cloud systems}
+\input{parts/07_data/01_cloud-systems/00_intro.tex}
+\input{parts/07_data/01_cloud-systems/01_key-value-store.tex}
+\input{parts/07_data/01_cloud-systems/02_cloud-native-databases.tex}
+\input{parts/07_data/01_cloud-systems/03_future.tex}
+% \input{parts/07_data/01_cloud-systems/}
+% \input{parts/07_data/}