[DMDB] Add concurrency control and recovery

Oops, luckily still figured out that that was missing...
2026-07-27 21:29:09 +02:00 · 2026-07-25 18:58:39 +02:00
parent 45b3f178a7
commit c82ceca4e0
29 changed files with 475 additions and 22 deletions
@@ -0,0 +1,49 @@
+\subsection{Hierarchical Navigable Small World Index (HNSW Index)}
+The name used in the lecture was HSNW (Hierarchical Small Navigable World) Index.
+
+This is a layered graph concept, at insert, first all nodes are inserted into layer 0 and each is connected to $M \approx 3 - 4$ neighbours.
+Then, the edge count is $\tco{N \cdot M}$. With $N = 100$ million and $M = 16$, we have $1.6B$ edges, with a typical memory footprint of $4-8$GB for $100M$ 32-bit Edge IDs.
+Then, some nodes (typically about $N_{L_k} = N / M^k$ to layer $k$) are promoted to layer 1, connected to $M$ nearest neighbours in $L1$ only.
+This process is repeated a few times until the number of nodes in the layer is low enough.
+This number depends on a number of factors, primarily how coarse grained we want the initial search to be.
+
+Then, after building, we prune edges, in a process called RNG pruning.
+Here we prune an edge $(u, v)$ if there exist edges $(u, w)$ and $(w, v)$, such that $d(u, w) < d(u, v)$ and $d(w, v) < d(u, v)$.
+Important is that both conditions have to be met, as the goal is to remove the longest redundant edge from the graph.
+This is done using a greedy algorithm.
+
+
+\subsubsection{Search}
+We note that greedy search always makes progress in this index because the graph stays navigable.
+
+HNSW index search works by starting at a fixed entry, then doing \textit{one} greedy hop only.
+The hop goes towards the neighbour on this layer that is closer to the query.
+This is often done using Beam Search (see below) to find the $K$ closest vectors.
+
+After some number of hops, there are no more neighbours in layer $LN$ that are closer to check, so descend one layer and repeat.
+Traversal of the layer costs $\tco{M_L}$, with $M_L$ the number of edges on layer $L$.
+Higher up layers have fewer edges, thus making traversal cheap.
+
+The parameter \texttt{ef\_search} controls the recall vs latency.
+Set to $1$ it is approximately greedy, for $\texttt{ef\_search} \gg 1$ means more effort, i.e. more vectors visited.
+
+\paragraph{Beam Search}
+This is another greedy search algorithm, which chooses the next node to visit from a beam of size $N$.
+The goal is to find the closest node of the query, with $d$ the distance to the query.
+
+The procedure is simple:
+\begin{algorithm}
+    \caption{Beam Search}
+    \begin{algorithmic}[1]
+        \Procedure{BeamSearch}{$G$, start\_node}
+            \State $\texttt{Beam} \gets$ new beam data structure
+            \State \texttt{Beam.add(start\_node)}
+            \While{\texttt{beam} in \texttt{Beam}}
+                \State Check distances of neighbours and add them to \texttt{Beam}
+                \State Truncate \texttt{Beam} to size \texttt{max\_beam} \Comment{This is a tunable parameter}
+            \EndWhile
+            \State \Return \texttt{closest}
+        \EndProcedure
+    \end{algorithmic}
+\end{algorithm}
+