mirror of
https://github.com/janishutz/eth-summaries.git
synced 2026-07-27 21:29:09 +02:00
64 lines
3.5 KiB
TeX
64 lines
3.5 KiB
TeX
\subsubsection{Duplicate Elimination}
|
|
This is done using either hashing or sorting.
|
|
|
|
\paragraph{Hash-based approach}
|
|
Assume we have $B$ memory buffers.
|
|
Then the hashing approach works as follows:
|
|
\begin{enumerate}
|
|
\item Project out attributes and splitting the input into $B - 1$ partitions using a hash function $h1$.
|
|
\item After that, all duplicates will be in the same partition. Since there most likely are further collisions, for each partition from (1), we do the following:
|
|
\begin{enumerate}
|
|
\item If the partition fits into memory, then we are done with this step
|
|
\item If not, we run step recursively run step 1 again on this partition
|
|
\end{enumerate}
|
|
\item We return all keys from the hash table
|
|
\end{enumerate}
|
|
A partition from step 2 fits into memory if $\frac{f \cdot T}{B - 1} < B$ (or approximately $B > \sqrt{f \cdot T}$),
|
|
where $T$ is the number of pages after the projection and $f$ is a \textit{fudge factor}, typically $f \approx 1.2$.
|
|
|
|
\coloredbox{orange}{Cost of hash-based duplicate elimination}{
|
|
$\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{elim}(R')$, with
|
|
\[
|
|
\texttt{elim}(R') = \sum_{T \in R'} \begin{cases}
|
|
\texttt{elim}(T) & \text{if } \frac{f \cdot L(T)}{B - 1} \geq B \\
|
|
\texttt{length}(T) & \text{else}
|
|
\end{cases}
|
|
\]
|
|
where $R'$ is the relation $R$ after the projection. $\texttt{cnt}(R)$ is the number of pages and $\texttt{cnt}(R') = \texttt{cnt}(R) \cdot Q$,
|
|
with $Q$ the ratio of kept and total attributes in the relation.
|
|
}
|
|
|
|
\paragraph{Sort-based approach}
|
|
A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
|
|
|
|
The cost here is $\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cost}_\texttt{sort}(R') + \texttt{cnt}(R')$.
|
|
Broken down:
|
|
\begin{itemize}
|
|
\item $\texttt{cnt}(R)$ I/Os for the initial scan
|
|
\item first $\texttt{cnt}(R')$ I/Os for storing the projected $R$ (denoted $R'$) on disk
|
|
\item $\texttt{cost}_\texttt{sort}(R')$ I/Os to sort the remaining attributes (see \ref{sec:external-sort})
|
|
\item second $\texttt{cnt}(R')$ I/Os to scan the sorted result and eliminate duplicates
|
|
\end{itemize}
|
|
|
|
|
|
There is also an optimized version that uses a modified sort algorithm that during sorting projects out the attributes, with cost
|
|
\cost{$\texttt{Cost}(R) = \texttt{cnt}(R) + \texttt{cnt}(R') + \texttt{cnt}(R')$}. The first count is to read, sort and project; second count is to store the result to disk;
|
|
third count is to eliminate duplicates.
|
|
|
|
|
|
\paragraph{Comparison}
|
|
The sorting approach has the benefit of better handling data skew and that the result is sorted.
|
|
The I/O costs are comparable if $B$ is large, both algorithms need 2 passes over the data.
|
|
For low numbers of records, the hash-based approach is likely cheaper
|
|
|
|
|
|
\paragraph{Index-based approach}
|
|
Here, we only scan the index, and we don't visit the relation. The projection attributes need to be a subset of the index attributes.
|
|
We can project distinct on $A$ using a B+ tree on $A, B$.
|
|
We then only apply the projection algorithm only to the data entries.
|
|
|
|
Then, if an ordered index contains all projection attributes as prefix of the search key,
|
|
we first retrieve index data entries in order, discard unwanted fields and compare adjacent entries to eliminate duplicates (if needed).
|
|
|
|
If an index is smaller than a relation, then it is faster to scan the index.
|