mirror of
https://github.com/janishutz/eth-summaries.git
synced 2026-07-27 21:29:09 +02:00
[DMDB] Fix issues, improve query processing with notes on time complexities
This commit is contained in:
@@ -1,11 +1,6 @@
|
||||
\subsubsection{Duplicate Elimination}
|
||||
This is done using either hashing or sorting.
|
||||
The hashing approach hashes an element and checks in the existing map if this element exists already. If not, it is inserted, else it is skipped.
|
||||
Finally the map is returned, or transformed and returned.
|
||||
For a basic \texttt{SELECT A, B FROM R} query, we scan the file and for each tuple output $A, B$.
|
||||
Obviously, that means that for $N$ tuples and ratio of included tuples of $Q$, we have \cost{$N + N \cdot Q$} I/O operations.
|
||||
|
||||
For large datasets, we may want to create a hash table of size $B$, then run multiple dedupe passes, on each subset where duplicates could be,
|
||||
then return the union'd map.
|
||||
|
||||
A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
|
||||
However, this tends to be more expensive than the hash-based approach for low amounts of data, but for larger amounts, very similar.
|
||||
The sort-based approach also has the benefit of producing sorted data, which may be useful for future operators.
|
||||
For a \texttt{SELECT DISTINCT A, B FROM R} query, we scan the file and eliminate duplicates before outputting.
|
||||
If this is also combined with a sort, we can prefer the sort-based approach and deduplicate after search,
|
||||
so we are projecting, then sorting and finally deduplicating using the sort method.
|
||||
|
||||
Reference in New Issue
Block a user