[DMDB] Fix issues, improve query processing with notes on time complexities

This commit is contained in:
2026-07-27 14:41:27 +02:00
parent d197d01f24
commit be2e079959
22 changed files with 142 additions and 34 deletions
@@ -1,11 +1,6 @@
\subsubsection{Duplicate Elimination}
This is done using either hashing or sorting.
The hashing approach hashes an element and checks in the existing map if this element exists already. If not, it is inserted, else it is skipped.
Finally the map is returned, or transformed and returned.
For a basic \texttt{SELECT A, B FROM R} query, we scan the file and for each tuple output $A, B$.
Obviously, that means that for $N$ tuples and ratio of included tuples of $Q$, we have \cost{$N + N \cdot Q$} I/O operations.
For large datasets, we may want to create a hash table of size $B$, then run multiple dedupe passes, on each subset where duplicates could be,
then return the union'd map.
A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
However, this tends to be more expensive than the hash-based approach for low amounts of data, but for larger amounts, very similar.
The sort-based approach also has the benefit of producing sorted data, which may be useful for future operators.
For a \texttt{SELECT DISTINCT A, B FROM R} query, we scan the file and eliminate duplicates before outputting.
If this is also combined with a sort, we can prefer the sort-based approach and deduplicate after search,
so we are projecting, then sorting and finally deduplicating using the sort method.