Summary of issue: The former question showed some TeX code that used a list of strings which were labels derived from references in the text. However, that list could contain duplicates, and that code did not deduplicate that list. The latter question was focused on deduplicating that list after collecting it from the text.
However, David Carlisle indicated in chat that it might be possible to create a list without duplicates in the first instance. I had assumed it would be faster and easier to create a list with duplicates, and then remove the duplicates, but perhaps not. In any case, I'd be interested in a solution that uses this approach.
The relevant exchange started at
where David wrote
I can't see why you'd need \clist_remove_duplicates:N if duplicates are a problem then just don't put them in the list.
I responded with
@DavidCarlisle well, you'd have to check the list at every addition. One could do that, certainly, but it's probably more efficient to remove dupes at the end.
To which David responded:
@FaheemMitha I doubt it
and followed up with
@FaheemMitha You are adding ref so if \csname r@#1\endcsname is already defined you have seen this ref before and don't need to add it again. There is need to build a list with duplicates
You are presumably saying that code in the original question can be modified so dupes are not added.
to which he responded
@FaheemMitha yes although not testing r@#1 (as that just tells you that there is a label) but you could make each ref leave a csname so you know you have seen it (this trades space for time)