0

I just want to filter a given database for duplicates and remove them. In my specific case the database is loaded from an external .csv data and has only one single column. But to keep it simple in this MWE, I just created the DB floorRawData and filled it up with some data.

Because I did not find any out of the box function to remove duplicates, I tried it on my own like this:

\documentclass{article}
\usepackage{datatool}

\begin{document}
\DTLnewdb{floorFilteredData}
\DTLnewdb{floorRawData}

\DTLnewrow{floorRawData}
\DTLnewdbentry{floorRawData}{FloorRaw}{groundFloor}
\DTLnewrow{floorRawData}
\DTLnewdbentry{floorRawData}{FloorRaw}{upperFloor}
\DTLnewrow{floorRawData}
\DTLnewdbentry{floorRawData}{FloorRaw}{groundFloor}
\DTLnewrow{floorRawData}
\DTLnewdbentry{floorRawData}{FloorRaw}{basementFloor}
\DTLdisplaydb{floorRawData}\\ % Result as expected!

\newcommand{\noDBentry}{FALSE}

\DTLforeach*{floorRawData}{\floorRaw=FloorRaw}
{
    \noindent FloorRaw Element: \floorRaw\\
    \DTLifdbempty{floorFilteredData}
    {
        \DTLnewrow{floorFilteredData}
        \DTLnewdbentry{floorFilteredData}{FloorFiltered}{\floorRaw}
        \\\DTLdisplaydb{floorFilteredData}\\[5mm]
    }
    {
        \DTLforeach{floorFilteredData}{\floorFiltered=FloorFiltered}
        {
            FloorFiltered Element: \floorFiltered\\ % Why is this element "upperFloor" and not "groundFloor"?
            \ifthenelse{\equal{\floorRaw}{\floorFiltered}}
            {
                \renewcommand{\noDBentry}{TRUE}
            }{}
        }
        \ifthenelse{\equal{\noDBentry}{FALSE}}
        {
            Add new data: \floorFiltered % Never called!
            \DTLnewrow{floorFilteredData}
            \DTLnewdbentry{floorFilteredData}{FloorFiltered}{\floorRaw}
        }
        {
            \renewcommand{\noDBentry}{FALSE}
        }  
    }
}

\DTLdisplaydb{floorFilteredData}\\

\end{document}

In this MWE I just parse through the floorRawData DB and check if the current element is already part of my floorFilteredData DB. When it is part, then I will not add a new element in floorFilteredData DB otherwise I want to add this element.

So at the end I expect a floorFilteredData DB where in every line is a different \floorFiltered element.

But as you can see, the \floorFiltered element always seems to be the same element like \floorRaw.

Therefore my expectation is, that \DTLforeach{floorFilteredData}{\floorFiltered=FloorFiltered} will not start from the beginning.

I had a look at the datatool package manual p.58 https://texdoc.org/serve/datatool/0 But I didn't get the correct clue out of this...

Nested \foreach

OUTPUT enter image description here

1 Answer 1

0

After hours of searching, I found this great FAQ from the datatool package maintainer Nicola L. C. Talbot :) I would like to highlight this topic even more in the package manual on p.43.

The solution is to add \dtlexpandnewvalue before you are adding DB elements programmatically. And that's it!

\documentclass{article}
\usepackage{datatool}

\begin{document}
\DTLnewdb{floorFilteredData}
\DTLnewdb{floorRawData}

\DTLnewrow{floorRawData}
\DTLnewdbentry{floorRawData}{FloorRaw}{groundFloor}
\DTLnewrow{floorRawData}
\DTLnewdbentry{floorRawData}{FloorRaw}{upperFloor}
\DTLnewrow{floorRawData}
\DTLnewdbentry{floorRawData}{FloorRaw}{groundFloor}
\DTLnewrow{floorRawData}
\DTLnewdbentry{floorRawData}{FloorRaw}{basementFloor}
\DTLdisplaydb{floorRawData}\\ % Result as expected!

\newcommand{\noDBentry}{FALSE}
\dtlexpandnewvalue % Just add this line

\DTLforeach*{floorRawData}{\floorRaw=FloorRaw}
{
    \DTLifdbempty{floorFilteredData}
    {
        \DTLnewrow{floorFilteredData}
        \DTLnewdbentry{floorFilteredData}{FloorFiltered}{\floorRaw}
    }
    {
        \DTLforeach{floorFilteredData}{\floorFiltered=FloorFiltered}
        {
            \ifthenelse{\equal{\floorRaw}{\floorFiltered}}
            {
                \renewcommand{\noDBentry}{TRUE}
            }{}
        }
        \ifthenelse{\equal{\noDBentry}{FALSE}}
        {
            \DTLnewrow{floorFilteredData}
            \DTLnewdbentry{floorFilteredData}{FloorFiltered}{\floorRaw}
        }
        {
            \renewcommand{\noDBentry}{FALSE}
        }  
    }
}

\DTLdisplaydb{floorFilteredData}\\

\end{document}

OUTPUT Output

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .