mirror of
https://github.com/janishutz/eth-summaries.git
synced 2026-07-28 09:49:11 +02:00
12 lines
868 B
TeX
12 lines
868 B
TeX
\subsubsection{Duplicate Elimination}
|
|
This is done using either hashing or sorting.
|
|
The hashing approach hashes an element and checks in the existing map if this element exists already. If not, it is inserted, else it is skipped.
|
|
Finally the map is returned, or transformed and returned.
|
|
|
|
For large datasets, we may want to create a hash table of size $B$, then run multiple dedupe passes, on each subset where duplicates could be,
|
|
then return the union'd map.
|
|
|
|
A sort-based approach is also very easy to understand: We sort the records and then discard, on a run through the sorted records, the ones that are duplicates.
|
|
However, this tends to be more expensive than the hash-based approach for low amounts of data, but for larger amounts, very similar.
|
|
The sort-based approach also has the benefit of producing sorted data, which may be useful for future operators.
|