diff --git a/semester4/dmdb/assets/serializability-graph-built.png b/semester4/dmdb/assets/serializability-graph-built.png new file mode 100644 index 0000000..7add31e Binary files /dev/null and b/semester4/dmdb/assets/serializability-graph-built.png differ diff --git a/semester4/dmdb/assets/serializability-graph-history.png b/semester4/dmdb/assets/serializability-graph-history.png new file mode 100644 index 0000000..379d4af Binary files /dev/null and b/semester4/dmdb/assets/serializability-graph-history.png differ diff --git a/semester4/dmdb/data-modelling-databases-summary.pdf b/semester4/dmdb/data-modelling-databases-summary.pdf index 6c805d0..4b59d8b 100644 Binary files a/semester4/dmdb/data-modelling-databases-summary.pdf and b/semester4/dmdb/data-modelling-databases-summary.pdf differ diff --git a/semester4/dmdb/data-modelling-databases-summary.tex b/semester4/dmdb/data-modelling-databases-summary.tex index 4bce89c..bdd4b49 100644 --- a/semester4/dmdb/data-modelling-databases-summary.tex +++ b/semester4/dmdb/data-modelling-databases-summary.tex @@ -84,8 +84,9 @@ In addition, the end user does not have to worry about optimization too much. Th \include{parts/02_theory-background/main.tex} \include{parts/03_systems/main.tex} \include{parts/04_query-processing/main.tex} -\include{parts/05_vector-search/main.tex} -\include{parts/06_data/main.tex} +\include{parts/05_concurrency-control-recovery/main.tex} +\include{parts/06_vector-search/main.tex} +\include{parts/07_data/main.tex} \printGlossary diff --git a/semester4/dmdb/glossary/acronyms.tex b/semester4/dmdb/glossary/acronyms.tex index 15d7c6f..152bbd9 100644 --- a/semester4/dmdb/glossary/acronyms.tex +++ b/semester4/dmdb/glossary/acronyms.tex @@ -6,5 +6,5 @@ \newacronym{ra}{RA}{Relational Algebra, see Section~\ref{sec:relational-algebra}} \newacronym{ann}{ANN}{Approximate Nearest Neighbour Search} \newacronym{knn}{KNN}{$K$-Nearest Neighbour Search} -\newacronym{acid}{ACID}{Atomicitty, Consistency, Isolation, Durability} +\newacronym{acid}{ACID}{Atomicitty, Consistency, Isolation, Durability, also see Section~\ref{sec:acid}} \newacronym{base}{BASE}{Basically Available, Soft State, Eventually Consistent} diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/00_transactions.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/00_transactions.tex new file mode 100644 index 0000000..01b49ab --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/00_transactions.tex @@ -0,0 +1,9 @@ +\subsection{Transactions} +Transactions are accesses to the database with the purpose of \textit{modifying the data}. +The focus is of course on correctness, recovery and state management. + +Transactions typically are expensive compared to queries due to the focus on correctness. + +This means we need to employ concurrency control (ensuring data remains consistent with transactions and queries running at the same time, +definition of consistency, correctness, conflicts and problematic situations, as well as locking). +We furthermore want to implement recovery, to ensure that data remains consistent even when unexpected failures occur. diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/01_acid.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/01_acid.tex new file mode 100644 index 0000000..2cc7135 --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/01_acid.tex @@ -0,0 +1,44 @@ +\subsection{ACID} +\label{sec:acid} +\acrshort{acid} is the conventional notion of correctness (informal because it's an acronym): +\begin{itemize} + \item \bi{Atomicity}: A group of operations must take place in their entirety or not at all + \item \bi{Consistency}: Operations should take the database from a correct state to another correct state + \item \bi{Isolation}: Concurrent execution of operations should yield predictable and correct results + \item \bi{Durability}: The DB needs to remember the state it is in at all moments, even when failures occur +\end{itemize} + +\inlinedefinition[Database state] is the actual values stored in a database at a given point in time (in memory and storage). + +\inlinedefinition[Consistent state] is a database state that is the result of \textit{correctly} applying \textit{operations} to the DB. + +\inlinedefinition[Correct application] The model of correct executions of operations is sequential executions. + +\inlinedefinition[Operations] The operations for transactions are inserts, updates and deletion of tuples. + +\inlinedefinition[Consistency] The assumption of correctness of transactions is based on two things: +\bi{(1)} the user doesn't apply wrong transactions and \bi{(2)} the DBMS has mechanisms to prevent incorrect modifications, such as violations of constraints, etc. + +\inlinedefinition[Durability] ensures that the changes of a completed transaction are remembered by the system and can be recovered. +This is often more strict and DBMS will keep the DB st ate and a history of all transactions. + +The transaction history is captured in the DB log. The state at a given moment is captured in snapshots. +The log and snapshots allow to go back and forward in the history of the database by undoing or redoing transactions from a given snapshot. +Thus, very similar in concept to what \texttt{git} uses. + + +\subsubsection{Atomicity} +\inlinedefinition[Atomicity] dictates that a transaction only takes effect if it can be executed in its entirety, or in other words, +a transaction has to be executed in its entirety or not at all. +The issue with this is that violations are costly to correct. + +Thus, atomicity requires isolation. If a transaction can see the intermediate state created by another transaction, then its input is not guaranteed to be consistent, +implying that its output is not guaranteed to be consistent, either. + +The DBMS get around this issue by isolating the transactions, such that they behave as if they were alone in the database. +This of course means that we need to know when a transaction is completed so we can \textit{commit} it, if there are no conflicts. +If there are conflicts, we have to resolve them, rerun the transaction (note that this can be very expensive) or to prevent them altogether. + +The isolation is enforced through concurrency control mechanisms to prevent conflicts. +The canonical concurrency control mechanism in DBs is locking. +We use locking policies to ensure that the transactions are isolated. diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/02_transaction-model.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/02_transaction-model.tex new file mode 100644 index 0000000..e2cd6e2 --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/02_transaction-model.tex @@ -0,0 +1,35 @@ +\subsection{Transaction Model} +\inlinedefinition Some terms for Transaction model: +\begin{itemize} + \item \bi{Begin of transaction (BOT)}: Often implicit + \item \bi{Commit}: Transaction has finished, database confirms to client when all changes of the transaction have been made persistent + \item \bi{Abort/rollback}: transaction is cancelled, all changes made already are rolled back + \item $a <_T b$: $a$ happens before $b$, is a \textit{partial order}, which implies that $a$ will be done before $b$. +\end{itemize} + +In real systems, transactions are often associated with sessions and all SQL statements after an update statement are considered to be part of the transaction. +This means they are explicitly managed, using a \texttt{BEGIN TRANSACTION} or \texttt{BEGIN} statement and ended with either a \texttt{COMMIT} or \texttt{ROLLBACK} statement. + +So, we have the following statements: +\begin{itemize} + \item \texttt{COMMIT}: Transaction will not make any order + \item \texttt{ROLLBACK}: Roll back all changes + \item \texttt{SAVEPOINT $<$name$>$}: Create a save point / history point, used to roll back some changes and try some others in case of failure. + \item \texttt{ROLLBACK TO SAVEPOINT $<$name$>$}: Roll back to a specific save point. + \item \texttt{RELEASE SAVEPOINT $<$name$>$}: Delete a save point (doesn't delete the changes made before, just the log entry) +\end{itemize} + +\subsubsection{Operations} +Below a list of very important syntax for the exam (where the subscript $i$ denotes that this operation is part of transaction $T_i$): +\begin{itemize} + \item Read operations: $r_i[x]$ or $r_i(x)$ is an access to tuple $x$ without modifying it. + \item Write operations: $w_i[x]$ or $r_i(x)$ represents a write operation on the tuple $x$. + \item Abort: $a_j$ is an abort of transaction $T_j$ + \item Commit $c_j$ is the commit of transaction $T_j$ +\end{itemize} +Conflicting operations are two operations on the same item with at least one of them being a write operation, such as $r_1[x] w_2[x]$ or $w_1[y] w_2[y]$. + +\subsubsection{History} +A history $H$ is a partially ordered $<_H$ sequence of operations from a set of transactions. +If two operations are ordered within a transaction, they are equally ordered in the history. +If two operations $p$ and $q$ conflict, then they are ordered with respect to each other ($p <_H q$ or $q <_H p$) diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/03_concurrency-control.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/03_concurrency-control.tex new file mode 100644 index 0000000..433dfc0 --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/03_concurrency-control.tex @@ -0,0 +1,39 @@ +\newpage +\subsection{Concurrency Control} +The goal of concurrency control is to ensure the correct executions of concurrent transactions. +The baseline for correctness is, as previously mentioned, serial execution. + +\inlinedefinition[Serial History] A history $H$ is serial if, for every pair of two transactions $T_i$ and $T_j$ that appear in $H$, +either all operations from $T_i$ appear before all all operations of $T_j$, or vice versa. +A serial history with only committed transactions is correct by definition, because transactions are isolated and each transaction starts in a consistent state +and leaves the DB in a consistent state. + +\inlinedefinition[Equivalent History] Two histories are equivalent if and only if +\begin{enumerate} + \item They are over the same transactions and contain the same operations + \item Conflicting operations of non-aborted transactions are ordered in the same way in both histories +\end{enumerate} + +\inlinedefinition[Serializable History] A history is serializable if and only it is equivalent to a serial history. + +\inlinetheorem[Serializability] A history is serializable if and only if its serializability graph is acyclic. +The seerializabililty graph is constructed from the history graph as follows: +\begin{enumerate} + \item The start node is the transaction the first operation. + \item Then, follow the operations, adding any new transaction to the tree and connect any further dependencies. +\end{enumerate} +More visually, see the below graphics + +\begin{figure}[h!] + \begin{center} + \begin{subfigure}{0.49\textwidth} + \includegraphics[width=0.9\textwidth]{assets/serializability-graph-history.png} + \caption{The history} + \end{subfigure} + \begin{subfigure}{0.49\textwidth} + \includegraphics[width=0.7\textwidth]{assets/serializability-graph-built.png} + \caption{The resulting serializability graph} + \end{subfigure} + \end{center} + \caption{Building a serializability graph (Figure from lecture slides 19, slide 37 and 38)} +\end{figure} diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/04_two-phase-locking.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/04_two-phase-locking.tex new file mode 100644 index 0000000..fb340ae --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/04_two-phase-locking.tex @@ -0,0 +1,54 @@ +\subsection{2 Phase Locking (2PL)} +This type of locking is widely used since many decades and is the canonical implementation of concurrency control. +Of course, alternatives exist and the locking implementation has a huge impact on performance and scalability. + +Concurrency control is implemented using locking, typically using a locking table. +The locks cover tuples, tables, indexes, blocks and more. + +Locks have different semantics that indicate whether they can be shared or are exclusive. + +Compatibility matrix: +\begin{tables}{cccc}{ & NL & S & X} + S & \checkmark & \checkmark & - \\ + X & \checkmark & - & - \\ +\end{tables} + +\subsubsection{The protocol} +\begin{enumerate} + \item Before accessing an object, a transaction must acquire lock + \item A transaction acquires a lock only once. Lock upgrades are possible + \item A transaction is blocked if the lock request cannot be granted according to the compatibility matrix + \item A transaction goes through two phases: + \begin{enumerate} + \item Growth: Acquire locks, but never release locks + \item Shrink: Release locks, but never acquire locks + \end{enumerate} + \item At EOT (commit or abort), all locks must be released. +\end{enumerate} + +The reason for the two phases is that 2PL prevents histories where a transaction must be aborted due to concurrency control issues. +An operation is thus not allowed unless we are sure it is safe. + +Without the two phases, the following can happen: +$L_1[x] w_1[x] U_1[x] L_2[x] w_2[x] U_2[y] L_2[y] w_2[y]$, etc, thus meaning that if $T_1$ now access $y$ or $x$ for read or write, the history becomes non-serializable. + +The basic 2PL doesn't say anything about aborting or committing transactions. +The issue is that if we are to undo the changes of $T_1$, we would also undo some of the changes from $T_2$, which would then in turn not be restorable anymore. + +\subsubsection{Strict 2PL} +Strict 2PL combines 2PL with strict histories. This means that we enforce 2PL, but we also keep the read and write locks until the transaction commits or aborts. +This results in serializable and strict histories. +All DB engines implement strict 2PL. + + +\subsubsection{Deadlocks Detection} +In theory, we could use a wait-for graph, which is however expensive. +For example, for $T_1 \rightarrow T_2 \rightarrow T_3 \rightarrow T_4 \rightarrow T_1$, we could abort $T_3$ to resolve the cycle. + +In practice, no wait-for graphs are built. Transactions and queries have timers. If the timer expires, the transaction or query is aborted. +Deadlocks are considered rare enough to prefer making mistakes every now and then (aborting a transaction that was fine), +rather than implementing something as expensive as a wait-for graph. + +Implications from this are for example that long transactions may block queries and that those queries may abort because the timer expired. + +However, this is only common in poorly written applications. diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/05_snapshot-isolation.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/05_snapshot-isolation.tex new file mode 100644 index 0000000..5a9f84f --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/05_snapshot-isolation.tex @@ -0,0 +1,17 @@ +\subsection{Snapshot isolation} +This was initially used by Oracle and is a form of Multi-Version Concurrency Control (MVCC). + +MVCC applies serializability, but writes generate a new version and reads read from a concrete version of the DB. +This means that the whole DB can be implemented (almost) entirely without locks. + +When a transaction starts, it receives a timestamp $T$. +Read operations only have access to items committed before $T$. +All write operations are carried out in a separate buffer (shadow paging) and only become visible after a commit. +When a transaction commits, it checks for conflicts. If there are, the first-committer-wins rule is applied, i.e. if we try to commit and there are conflicts, +this transaction has to rerun (or be aborted). + +However, there are scenarios, such as $r_1[x] r_1[y] r_2[x] r_2[y] w_1[x] w_2[x] c_1 c_2$, where the history is not serializable: +$T_2 \rightarrow T_1 \rightarrow T_2$. +Depending on the application, this can still be correct (if this is for a ticketing system), but can also be disastrous (for e.g. a bank). + +If done properly however, it has significant performance advantages, as well as in distributed settings with replication. diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/06_recovery.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/06_recovery.tex new file mode 100644 index 0000000..339a3f6 --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/06_recovery.tex @@ -0,0 +1,77 @@ +\subsection{Recovery} +Recovery has to make sure that, even in the case of failures, the changes from transactions that have not completed are not visible +and those of committed transactions are recorded and visible. +We define undesirable situations and classify histories according to these situations. +Furthermore, we want to define how histories can avoid such situations. + + +\subsubsection{Recovery procedures} +\begin{enumerate}[label=R\arabic*] + \item Undo all changes from transactions + \item Redo the changes from committed transactions + \item Undo the changes that remain in the system from active transactions + \item Read consistent snapshot from backup, and, if available, apply log +\end{enumerate} +R1 is used for abort/rollback of single transactions, R2 and R3 are used on system crash (where we lose the memory, but disks survive) +and R4 is used for recovery from a system crash with disk loss. + +This means that we can undo the changes of an active transaction (we keep a copy of the value before it was modified; Shadow pages is a mechanism that would support this), +we can redo changes made by committed transactions (we keep a persistent record of all the changes made by committed transactions, this why there is a redo log) +and we have a consistent snapshot to start with (either move forward by taking a snapshot and apply committed transactions, +or move backward by taking the current state and go back to a consistent state by undoing changes from transactions that should be there). + + +\subsubsection{Undo - Redo} +The changes of an aborted transaction are undone typically by restoring the \bi{before image} of the modified value. + +Changes are redone by restoring the \bi{after image} of the modified value. + +Databases typically log transactions by keeping before and after images of their changes. + + +\subsubsection{Recovery on histories} +A transaction $T_1$ reads from another transaction $T_2$ if $T_1$ reads a value written by $T_2$ at a time when $T_2$ was not aborted. + +We have: +\begin{itemize} + \item \bi{Recoverable} (RC) history: If $T_i$ reads from $T_j$ and commits, then $c_j < c_i$ + \item \bi{Avoids Cascading Aborts} (ACA) history: If $T_i$ reads $x$ from $T_j$, then $c_j < r_i[x]$ + \item \bi{Strict} (ST) history: If $T_i$ reads from, or overwrites, a value written by $T_j$ then $c_j < r_i[x] / w_i[x]$ or $a_j < r_i[x] / w_i[x]$ +\end{itemize} + + +\paragraph{Recoverable} +There is no need to undo a committed transaction because it read the wrong data and they commit in their serialization order. + +Suppose we have the following sequence: $\ldots w_2[x] r_1[x] c_1 \ldots$. +If $T_2$ aborts, then $T_1$ has read invalid data and since $T_1$ has committed, the data has returned to the user. +Recovery now involves correcting the application that has processed the data and might have acted upon it, which typically is a very expensive procedure. +This is avoided by making sure that $T_1$ does not commit until $T_2$ commits and if $T_2$ aborts, then we can safely abort $T_1$, +since the results have not been returned to the user yet. + + +\paragraph{ACA} +Aborting a transaction does not cause aborting others and transactions only read from committed transactions + +Suppose we have the following sequence: $\ldots w_2[x] r_1[x] a_2 \ldots$. +When $T_2$ aborts, we have to abort $T_1$ because it has read invalid data (the change made by $T_2$, which will be undone as part of rollback of $T_2$). +If those cascading aborts happen regularly, the performance will degrade and possibly further transactions will be aborted. +This is avoided by making sure that uncommitted data is never read (\textit{read committed}). + + +\paragraph{Strict} +Undoing a transaction does not undo the changes of other transactions. Transactions further don't read or overwrite updates of uncommitted transactions. + +Suppose we have the following sequence: $\ldots w_2[x] w_1[x] a_2 \ldots$. +If we undo $w_2[x]$, we would also remove the changes made by $T_1$. Thus, undoing $T_2$ implies aborting $T_1$. +Like in ACA, non-strict execution leads to cascading aborts of transactions. +This is avoided by not letting a value be read or updated unless it is committed. + + +\paragraph{Why recoverability matters} +Fixing errors with recovery typically are very expensive: +\begin{itemize} + \item Not RC: running my program issues a report, but the data that I got was later removed because a different program aborted $\rightarrow$ report is invalid + \item Not ACA: Thrashing behaviour when transactions keep aborting each other (my program slows down, or makes no progress at all because of other programs) + \item Not Strict: recovery after a failure becomes very complex (or impossible) +\end{itemize} diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/07_recovery-procedures.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/07_recovery-procedures.tex new file mode 100644 index 0000000..3d8325e --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/07_recovery-procedures.tex @@ -0,0 +1,127 @@ +\subsection{Recovery Procedures} +\inlinedefinition[Recovery Manager] A recovery manager implements the recovery procedure, which depends on how the system functions. +It may or may not require both, one or none of UNDO and REDO. + +\subsubsection{From the buffer cache's perspective} +\begin{itemize} + \item \bi{Steal Policy}: An uncommitted transaction is allowed to overwrite the changes of a committed transaction in persistent storage. + This logically means that we need to be able to undo transactions. + \item \bi{Force Policy}: All changes made by a transaction must be in persistent storage before the transaction commits. + This requires flushing all blocks with updates from the transaction and if not in-place, it will require being able to redo the transaction. + \item \bi{Steal/No-Force}: UNDO/REDO, is the most common approach +\end{itemize} +If for updates tuples are locked, blocks may still be updatable. +Thus if the block is copied to storage, it is possible that some changes are committed while others are not. +If failures occur, there is no guarantee that the data in storage is 100\% consistent, thus requiring undo and redo. + +\subsubsection{Recovery Procedure for UNDO/REDO} +\begin{itemize} + \item READ: simply read the value from the block in the buffer cache + \item WRITE: Create log entry (before and after image), append it to persistent log and write the after image to the block on the buffer cache. + \item COMMIT: We write a persistent log entry indicating that the transaction has committed + \item ABORT: For all updates, we restore the before image using the log entry +\end{itemize} + +The Procedure: +\begin{enumerate} + \item Start from the end of the log and work backwards + \item Keep a list of UNDONE items and another for REDONE items + \item Procedure terminates when all items are in either the UNDONE or REDONE list or we reach the beginning of the log. + \item For each log entry, we look at the data item $x$ being accessed and if $x$ is in none of the two lists: + \begin{itemize} + \item If the log entry is of a committed transaction, apply the after image, add $x$ to the REDONE list + \item If the log entry is of an aborted transaction, apply the before image, add $x$ to the UNDONE list + \end{itemize} +\end{enumerate} +The procedure ignores the data stored on disk as it could correspond to uncommitted transactions, it only takes it as starting point for the recovery. +An alternative way of thinking about the procedure is as follows: +\begin{algorithm} + \caption{Undo-Redo Recovery} + \begin{algorithmic}[1] + \Procedure{UndoRedoRecovery}{} + \For{each item in the DB} + \State Find last committed transaction modifying the item and REDO it + \If{no committed transaction modified item} + \State Find the first aborted transaction that modified the item and \texttt{UNDO} the modification + \EndIf + \If{If no transaction has touched the item} + \State Value is correct (we assume that we start from a consistent state) + \EndIf + \EndFor + \EndProcedure + \end{algorithmic} +\end{algorithm} +This is not the way it's done in practice, because there typically are many more items than log entries. + +\shade{ForestGreen}{Advantages} Only forced I/O are log records, buffer cache manager has a lot of freedom +(no flushing dirty pages, I/O on data is minimized and only triggered for replacement policies) and we allow writing dirty data to disk, simplifying buffer management. + +However, recovery is more complicated and takes more time, but normal operations are only minimally affected. +Furthermore, queries are not affected since there is no forced I/O of data. +Transactions are affected, because we need to write every operation to the log. + + +\subsubsection{UNDO, no REDO} +\begin{itemize} + \item READ: simply read the value from the block in the buffer cache + \item WRITE: Create log entry (before image), append it to persistent log and write the after image to the block on the buffer cache. + \item COMMIT: Flush all dirty values modified by transaction if still in cache. We write a persistent log entry indicating that the transaction has committed + \item ABORT: For all updates, we restore the before image using the log entry +\end{itemize} +The Procedure: +\begin{enumerate} + \item Start from the end of the log and work backwards + \item Keep a list of UNDONE items + \item Procedure terminates when all items are in either the UNDONE list or we reach the beginning of the log. + \item For each log entry, we look at the data item $x$ being accessed and if $x$ is not in the UNDONE list and the transaction is aborted, + we UNDO the changes by using the before image, adding $x$ to the UNDONE list +\end{enumerate} +It relies on the fact that all committed values are in persistent storage and therefore, they have not been lost. + +However, it has forced I/O on all dirty blocks touched by the transaction when it commits, +log records no longer need to include after images, reducing log record size and the recovery procedure is shorter because it only involves undoing aborted transactions. + +However, this trade-off does not pay off in practice, as we still need to write the log with every update and while it requires smaller log records, +it forces I/O on data blocks, which are often much larger. +And finally, flushing the buffer cache interferes with its operations. + + +\subsubsection{no UNDO, REDO} +\begin{itemize} + \item READ: If the transaction did not write the item before, read the value from the block in the buffer cache. Otherwise, read the value from the temporary buffer + \item WRITE: Create log entry (after image), append it to persistent log and write the after image to the block on the buffer cache. + \item COMMIT: Apply all updates in the temporary buffer to the actual blocks. We write a persistent log entry indicating that the transaction has committed + \item ABORT: Discard the temporary buffer +\end{itemize} +The Procedure: +\begin{enumerate} + \item Start from the end of the log and work backwards + \item Keep a list of REDONE items + \item Procedure terminates when all items are in either the REDONE list or we reach the beginning of the log. + \item For each log entry, we look at the data item $x$ being accessed and if $x$ is not in the REDONE list and the transaction is committed, + we REDO the changes by using the after image, adding $x$ to the REDONE list +\end{enumerate} +It relies on the fact that there are never dirty blocks in the buffer cache. All data there is committed and is the last committed version. + +It has forced I/O only on the log records and they are also smaller. +Furthermore, the recovery procedure is shorter, only involving redoing the last committed transaction that touched the item. +It is similar to snapshot isolation. + + +\subsubsection{No UNDO, no REDO} +This procedure is rarely used in conventional DBs. It does not require a log. +However, it requires to be able to write all changes made by a transaction to persistent storage in \bi{a single atomic action}. +It relies on directories to keep track of the changes. +\begin{itemize} + \item READ: If the transaction did not write the item before, use current directory to find the latest committed copy. + Otherwise, read the value from the shadow directory of that transaction to find the updated copy + \item WRITE: Write to a buffer and add a pointer in the shadow directory for the transactions + \item COMMIT: Create a full directory by merging the current one and the shadow directory of the transaction. + Then, we swap the pointer indicating the latest committed directory. + \item ABORT: Discard the buffer(s) and the shadow directory +\end{itemize} +There is no recovery procedure here, which is the exact point. + +It is not used in practice, although some ideas can be applied partially. +One of the reasons for that is expensive pointer indirections for accesses and requires garbage collection and moves data all the time, +which poses a challenge for clustered indexes or hash clustered tables. diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/08_implementation-of-recovery.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/08_implementation-of-recovery.tex new file mode 100644 index 0000000..597d964 --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/08_implementation-of-recovery.tex @@ -0,0 +1,38 @@ +\subsection{Implementation of Recovery} +The most common implementation of these ideas is Write Ahead Logging (WAL). +There, we separate the storage used for the log and data. The log contains enough information for whatever policy is chosen and log records corresponding to a change in the DB +must be written to the log before changes to the data in the buffer cache are flushed to permanent storage. +The COMMIT record in the log is us ed to mark the end of a transaction. + +This is typically used to implement UNDO/REDO (on 2PL-based systems) or no-UNDO/REDO (on Snapshot Isolation (SI)-based systems) + + +\subsubsection{Checkpoints} +A log would grow forever if we don't do anything. Storage would then become an issue and recovery time would increase. +Instead, Checkpoints are used. At a checkpoint, we do the following: +\begin{itemize} + \item Push all dirty blocks to the disk + \item Push all the logs in the log buffer to disk + \item We now have an active translation table and dirty page table + \item We mark the log with a checkpoint label and flush it to the log +\end{itemize} +Then, recovery happens from a checkpoint instead from the beginning. However note that a checkpoint is not necessarily a consistent copy of the DB. + + +\subsubsection{Recovery with WAL and checkpoints} +Also called ARIES style recovery, if we use WAL, the recovery with a checkpoint works as follows: + +\begin{algorithm} + \caption{WAL checkpoint recovery} + \begin{algorithmic}[1] + \Procedure{WALCheckpointRecovery}{} + \State Find latest completed checkpoint in the log + \While{Traversing the log to the end} + \State identify active transactions at time of crash and dirty pages that may not have made it to disk at time of crash for each element + \EndWhile + \State Apply all updates (REDO) starting from the log entry matching the lowest SCN in the dirty pages + \State UNDO all transactions that were active at the time of the crash + \EndProcedure + \end{algorithmic} +\end{algorithm} + diff --git a/semester4/dmdb/parts/05_concurrency-control-recovery/main.tex b/semester4/dmdb/parts/05_concurrency-control-recovery/main.tex new file mode 100644 index 0000000..0b87a7e --- /dev/null +++ b/semester4/dmdb/parts/05_concurrency-control-recovery/main.tex @@ -0,0 +1,12 @@ +\newpage +\section{Concurrency Control and Recovery} +\input{parts/05_concurrency-control-recovery/00_transactions.tex} +\input{parts/05_concurrency-control-recovery/01_acid.tex} +\input{parts/05_concurrency-control-recovery/02_transaction-model.tex} +\input{parts/05_concurrency-control-recovery/03_concurrency-control.tex} +\input{parts/05_concurrency-control-recovery/04_two-phase-locking.tex} +\input{parts/05_concurrency-control-recovery/05_snapshot-isolation.tex} +\input{parts/05_concurrency-control-recovery/06_recovery.tex} +\input{parts/05_concurrency-control-recovery/07_recovery-procedures.tex} +\input{parts/05_concurrency-control-recovery/08_implementation-of-recovery.tex} +% \input{parts/05_concurrency-control-recovery/} diff --git a/semester4/dmdb/parts/05_vector-search/main.tex b/semester4/dmdb/parts/05_vector-search/main.tex deleted file mode 100644 index 8774e64..0000000 --- a/semester4/dmdb/parts/05_vector-search/main.tex +++ /dev/null @@ -1,6 +0,0 @@ -\section{Vector Search} -\input{parts/05_vector-search/00_intro-trees-clustering.tex} -\input{parts/05_vector-search/01_hsnw-index.tex} -\input{parts/05_vector-search/02_hash-index.tex} -\input{parts/05_vector-search/03_filtered-vector-search.tex} -% \input{parts/05_vector-search/} diff --git a/semester4/dmdb/parts/06_data/main.tex b/semester4/dmdb/parts/06_data/main.tex deleted file mode 100644 index 6bec7d9..0000000 --- a/semester4/dmdb/parts/06_data/main.tex +++ /dev/null @@ -1,13 +0,0 @@ -\section{Data} -\subsection{Formats} -\input{parts/06_data/00_formats/00_intro.tex} -\input{parts/06_data/00_formats/01_encodings.tex} -% \input{parts/06_data/00_formats/} - -\subsection{Cloud systems} -\input{parts/06_data/01_cloud-systems/00_intro.tex} -\input{parts/06_data/01_cloud-systems/01_key-value-store.tex} -\input{parts/06_data/01_cloud-systems/02_cloud-native-databases.tex} -\input{parts/06_data/01_cloud-systems/03_future.tex} -% \input{parts/06_data/01_cloud-systems/} -% \input{parts/06_data/} diff --git a/semester4/dmdb/parts/05_vector-search/00_intro-trees-clustering.tex b/semester4/dmdb/parts/06_vector-search/00_intro-trees-clustering.tex similarity index 100% rename from semester4/dmdb/parts/05_vector-search/00_intro-trees-clustering.tex rename to semester4/dmdb/parts/06_vector-search/00_intro-trees-clustering.tex diff --git a/semester4/dmdb/parts/05_vector-search/01_hsnw-index.tex b/semester4/dmdb/parts/06_vector-search/01_hsnw-index.tex similarity index 100% rename from semester4/dmdb/parts/05_vector-search/01_hsnw-index.tex rename to semester4/dmdb/parts/06_vector-search/01_hsnw-index.tex diff --git a/semester4/dmdb/parts/05_vector-search/02_hash-index.tex b/semester4/dmdb/parts/06_vector-search/02_hash-index.tex similarity index 100% rename from semester4/dmdb/parts/05_vector-search/02_hash-index.tex rename to semester4/dmdb/parts/06_vector-search/02_hash-index.tex diff --git a/semester4/dmdb/parts/05_vector-search/03_filtered-vector-search.tex b/semester4/dmdb/parts/06_vector-search/03_filtered-vector-search.tex similarity index 100% rename from semester4/dmdb/parts/05_vector-search/03_filtered-vector-search.tex rename to semester4/dmdb/parts/06_vector-search/03_filtered-vector-search.tex diff --git a/semester4/dmdb/parts/06_vector-search/main.tex b/semester4/dmdb/parts/06_vector-search/main.tex new file mode 100644 index 0000000..e83a4a7 --- /dev/null +++ b/semester4/dmdb/parts/06_vector-search/main.tex @@ -0,0 +1,6 @@ +\section{Vector Search} +\input{parts/06_vector-search/00_intro-trees-clustering.tex} +\input{parts/06_vector-search/01_hsnw-index.tex} +\input{parts/06_vector-search/02_hash-index.tex} +\input{parts/06_vector-search/03_filtered-vector-search.tex} +% \input{parts/06_vector-search/} diff --git a/semester4/dmdb/parts/06_data/00_formats/00_intro.tex b/semester4/dmdb/parts/07_data/00_formats/00_intro.tex similarity index 100% rename from semester4/dmdb/parts/06_data/00_formats/00_intro.tex rename to semester4/dmdb/parts/07_data/00_formats/00_intro.tex diff --git a/semester4/dmdb/parts/06_data/00_formats/01_encodings.tex b/semester4/dmdb/parts/07_data/00_formats/01_encodings.tex similarity index 100% rename from semester4/dmdb/parts/06_data/00_formats/01_encodings.tex rename to semester4/dmdb/parts/07_data/00_formats/01_encodings.tex diff --git a/semester4/dmdb/parts/06_data/01_cloud-systems/00_intro.tex b/semester4/dmdb/parts/07_data/01_cloud-systems/00_intro.tex similarity index 100% rename from semester4/dmdb/parts/06_data/01_cloud-systems/00_intro.tex rename to semester4/dmdb/parts/07_data/01_cloud-systems/00_intro.tex diff --git a/semester4/dmdb/parts/06_data/01_cloud-systems/01_key-value-store.tex b/semester4/dmdb/parts/07_data/01_cloud-systems/01_key-value-store.tex similarity index 100% rename from semester4/dmdb/parts/06_data/01_cloud-systems/01_key-value-store.tex rename to semester4/dmdb/parts/07_data/01_cloud-systems/01_key-value-store.tex diff --git a/semester4/dmdb/parts/06_data/01_cloud-systems/02_cloud-native-databases.tex b/semester4/dmdb/parts/07_data/01_cloud-systems/02_cloud-native-databases.tex similarity index 100% rename from semester4/dmdb/parts/06_data/01_cloud-systems/02_cloud-native-databases.tex rename to semester4/dmdb/parts/07_data/01_cloud-systems/02_cloud-native-databases.tex diff --git a/semester4/dmdb/parts/06_data/01_cloud-systems/03_future.tex b/semester4/dmdb/parts/07_data/01_cloud-systems/03_future.tex similarity index 100% rename from semester4/dmdb/parts/06_data/01_cloud-systems/03_future.tex rename to semester4/dmdb/parts/07_data/01_cloud-systems/03_future.tex diff --git a/semester4/dmdb/parts/07_data/main.tex b/semester4/dmdb/parts/07_data/main.tex new file mode 100644 index 0000000..3846e69 --- /dev/null +++ b/semester4/dmdb/parts/07_data/main.tex @@ -0,0 +1,13 @@ +\section{Data} +\subsection{Formats} +\input{parts/07_data/00_formats/00_intro.tex} +\input{parts/07_data/00_formats/01_encodings.tex} +% \input{parts/07_data/00_formats/} + +\subsection{Cloud systems} +\input{parts/07_data/01_cloud-systems/00_intro.tex} +\input{parts/07_data/01_cloud-systems/01_key-value-store.tex} +\input{parts/07_data/01_cloud-systems/02_cloud-native-databases.tex} +\input{parts/07_data/01_cloud-systems/03_future.tex} +% \input{parts/07_data/01_cloud-systems/} +% \input{parts/07_data/}