Deleting duplicates from a comma-separated list of strings

Question

This is a followup to Difference in compilation time between two slightly different versions.

As became obvious when asking that question, the speed of that compilation is seriously affected by the length of the comma separated list \ReferencedID. This is the list of the labels collected from the text. But this list may contain duplicates, and in fact in the example in my question, it does. So, how can one delete duplicates from a comma-separated list?

For concreteness, the list is my example is:

\gdef \ReferencedIDs{,2014.11.14,2014.11.14,2014.11.17,2014.11.14,2014.07.18.tm,2014.11.12.tm,2013.10.05.kanjur,2013.10.05.powai,2013.10.06,2013.10.28.zhm,2013.10.17,2013.10.22,2013.10.14.ks,2013.10.06,2013.10.17,2013.10.22,2013.10.24,2013.10.28,2013.10.30,2013.11.11,2014.09.05.tm,2014.07.18.tm,2014.11.12.tm,2014.11.17,2015.01.06,2015.01.06}

Answer 1

Assuming \ReferencedIDs has already been formed as a comma separated list, you can combine the macros I have already shown in seqsplit does not work unless the string is given as an argument directly

\documentclass[12pt]{article}
\usepackage[T1]{fontenc}

\usepackage{seqsplit}
\usepackage{xparse}

\ExplSyntaxOn
\cs_set_eq:NN \faheem_seqsplit:n \seqsplit
\cs_generate_variant:Nn \faheem_seqsplit:n { V }
\clist_new:N \l_faheem_referenced_ids_clist

\NewDocumentCommand{\printIDs}{s}
 {
  \clist_set:NV \l_faheem_referenced_ids_clist \ReferencedIDs
  \IfBooleanF{#1}
  { \clist_remove_duplicates:N \l_faheem_referenced_ids_clist }
  \faheem_seqsplit:V \l_faheem_referenced_ids_clist
 }
\ExplSyntaxOff

\def\ReferencedIDs{,2014.11.14,2014.11.14,2014.11.17,2014.11.14,2014.07.18.tm,%
  2014.11.12.tm,2013.10.05.kanjur,2013.10.05.powai,2013.10.06,2013.10.28.zhm,%
  2013.10.17,2013.10.22,2013.10.14.ks,2013.10.06,2013.10.17,2013.10.22,2013.10.24,%
  2013.10.28,2013.10.30,2013.11.11,2014.09.05.tm,2014.07.18.tm,2014.11.12.tm,%
  2014.11.17,2015.01.06,2015.01.06}


\begin{document}

This is the original list with duplicates, \printIDs*

Here is the list with duplicates removed, \printIDs

\end{document}

Answer 2

Thanks to Joseph Wright for showing me how to do this using expl3. The following code is largely self-explanatory. The line that removes the duplicates is

\clist_remove_duplicates:N \l_ReferencedIDs

For documentation, see the interface3.pdf file, found on TeX Live, at least, with texdoc interface3.

\documentclass[12pt]{article}
\usepackage{seqsplit}
\usepackage{expl3}
\usepackage[T1]{fontenc}
\let\oldseqsplit\seqsplit% Copy \seqsplit
\renewcommand{\seqsplit}{% Redefine \seqsplit to...
\expandafter\oldseqsplit\expandafter}% ...expand its argument before processing it

\gdef \ReferencedIDs{,2014.11.14,2014.11.14,2014.11.17,2014.11.14,2014.07.18.tm,2014.11.12.tm,2013.10.05.kanjur,2013.10.05.powai,2013.10.06,2013.10.28.zhm,2013.10.17,2013.10.22,2013.10.14.ks,2013.10.06,2013.10.17,2013.10.22,2013.10.24,2013.10.28,2013.10.30,2013.11.11,2014.09.05.tm,2014.07.18.tm,2014.11.12.tm,2014.11.17,2015.01.06,2015.01.06}
\ExplSyntaxOn
\clist_set:NV \l_ReferencedIDs \ReferencedIDs % initialise list
\clist_remove_duplicates:N \l_ReferencedIDs % remove duplicates
\gdef \DeduplicatedReferencedIDs{\l_ReferencedIDs}
\ExplSyntaxOff

\begin{document}

This is the original list with duplicates, \seqsplit{\ReferencedIDs}.

Here is the list with duplicates removed, \seqsplit{\DeduplicatedReferencedIDs}.

\end{document}

NOTE: The \seqsplit{\DeduplicatedReferencedIDs} does not wrap correctly; I don't know why.