# Datatool gobbling spaces

I've come across a strange problem with datatool. If the first character of the first column of a row is immediately followed by a space, that space gets gobbled somehow. Am I doing something wrong or is this a bug?

\begin{filecontents}{\jobname.csv}
Sentence1,Sentence2,Sentence3
The child played a new game.,He played it three times with his friends.,He loved the game.
I visited my childhood home last week.,It's just as I remembered it.,My house has always been white.
It's just as I remembered it.,I visited my childhood home last week.,My house has always been white.
\end{filecontents}
\documentclass{article}
\usepackage{datatool}
\begin{document}
\DTLforeach{sentences}{%
\sOne=Sentence1,\sTwo=Sentence2,\sThree=Sentence3}
{%
\sOne\par
\sTwo\par
\sThree\par
}
\end{document}


Here's the output. Notice that the sentence in column 1 of the CSV file (marked with the red arrow) has the gobbled space, but the exact same sentence in column 2 (marked with the blue arrow) doesn't have the space gobbled.

• Sure seems like a bug. Strange that it has not been noticed before. TeXLive 2015 also reproduces this problem. – Peter Grill Apr 4 '17 at 0:21
• Any one letter word (or a single braced group) will show the same behavior. Apparently the macro for testing the beginning of a row eats two arguments, so a space in between them is lost. – egreg Apr 4 '17 at 7:27

This bug has now been fixed. If you encounter it, make sure you have the latest version of datatool.

The problem is with \dtl@trim which is supposed to trim the trailing space at the end of each line (caused by the end of line character). It's also supposed to discard the terminating \par that occurs on the final \read before the end of file is reached.

I think the following patch should fix the problem (but I need to test further to make sure it doesn't cause any unwanted side-effects). The argument of \dtl@trim is always a control sequence (set by \read for \DTLloaddb).

\makeatletter
\renewcommand{\dtl@trim}[1]{%
\if#1\par
\def\@dtl@trmstr{}%
\else
\expandafter\@dtl@start@trim#1\@dtl@end@trim
\fi
\let#1=\@dtl@trmstr
}

\def\@dtl@start@trim#1 \@dtl@end@trim{%
\def\@dtl@trmstr{#1}%
}

\makeatother


Complete MWE:

\begin{filecontents}{\jobname.csv}
Sentence1,Sentence2,Sentence3
The child played a new game.,He played it three times with his friends.,He loved the game.
I visited my childhood home last week.,It's just as I remembered it.,My house has always been white.
It's just as I remembered it.,I visited my childhood home last week.,My house has always been white.
\end{filecontents}
\documentclass{article}
\usepackage{datatool}

\makeatletter
\renewcommand{\dtl@trim}[1]{%
\if#1\par
\def\@dtl@trmstr{}%
\else
\expandafter\@dtl@start@trim#1\@dtl@end@trim
\fi
\let#1=\@dtl@trmstr
}

\def\@dtl@start@trim#1 \@dtl@end@trim{%
\def\@dtl@trmstr{#1}%
}

\makeatother

\begin{document}
\DTLforeach{sentences}{%
\sOne=Sentence1,\sTwo=Sentence2,\sThree=Sentence3}
{%
\sOne\par
\sTwo\par
\sThree\par
}
\end{document}


• Probably \ifx\par#1 is better; it will fail if the line starts with {\par} or \par, however. – egreg Apr 4 '17 at 20:12
• @egreg #1 is a control sequence containing the line, so #1 is never \par, but may be a control sequence whose replacement text is \par. Perhaps better would be something like \def\@dtl@par{\par} and then \ifx#1\@dtl@par. – Nicola Talbot Apr 4 '17 at 20:28

This does not fix the datatool bug, but rather shows an alternate approach using readarray. It creates an array \Sentence[<i>,<j>], where i and j are the row and column of the \Sentence array.

\documentclass{article}
\begin{filecontents*}{\jobname.csv}
The child played a new game.,He played it three times with his friends.,He loved the game.
I visited my childhood home last week.,It's just as I remembered it.,My house has always been white.
It's just as I remembered it.,I visited my childhood home last week.,My house has always been white.
\end{filecontents*}
\begin{document}
\foreach\x in {1,2,...,\SentenceROWS}
{\foreach\y in {1,2,...,\SentenceCOLS}{\Sentence[\x,\y]\par}}
\end{document}


• Thanks Steven. There are plenty of workarounds I can use (and the real example is more complex than this) but this might be useful to others. It's certainly lower overhead than datatool. – Alan Munn Apr 4 '17 at 2:39
• @AlanMunn Understood. I just find the listofitems package, which is incorporated into readarray, to be very versatile in these matters. But I understand the real desire is to debug datatool. – Steven B. Segletes Apr 4 '17 at 2:41

I propose a different patch than Nicola’s.

\RequirePackage{filecontents}
\begin{filecontents}{\jobname.csv}
Sentence1,Sentence2,Sentence3
The child played a new game.,He played it three times with his friends.,He loved the game.
I visited my childhood home last week.,It's just as I remembered it.,My house has always been white.
It's just as I remembered it.,I visited my childhood home last week.,My house has always been white.
{}A visited my childhood home last week.,It's just as I remembered it.,My house has always been white.
{Bbb} visited my childhood home last week.,It's just as I remembered it.,My house has always been white.
X,X,X
X
\end{filecontents}
\documentclass{article}
\usepackage{datatool}

\makeatletter
\let\saved@dtl@starttrim\@dtl@starttrim
\long\def\@dtl@starttrim#1{%
\def\fix@dtl@starttrim@first{#1}%
\futurelet\next\fix@dtl@starttrim@second
}
\def\fix@dtl@starttrim@second{%
\if\noexpand\next\@sptoken
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
{\expandafter\saved@dtl@starttrim\expandafter{\fix@dtl@starttrim@first}{}}%
{\expandafter\saved@dtl@starttrim\expandafter{\fix@dtl@starttrim@first}}%
}
\makeatother

\begin{document}
\DTLforeach{sentences}{%
\sOne=Sentence1,\sTwo=Sentence2,\sThree=Sentence3}
{%
\sOne\par
\sTwo\par
\sThree\par\kern1pt\hrule\kern1pt
}
\end{document}


The trimming macro checks whether the first token is followed by a space, but absorbing just one token (or braced group). It takes different actions according to whether the token is a space or not.

• What are the relative merits/issues of each? – Alan Munn Apr 4 '17 at 17:44
• @AlanMunn Only Nicola can judge. However, \if#1\par looks very suspicious. – egreg Apr 4 '17 at 17:52
• Now you're just being coy. ;-) – Alan Munn Apr 4 '17 at 17:57
• Your patch loses the grouping. Add \show\sOne at the start of the loop: {} is missing from {}A visited... and the group is stripped from {Bbb} visited... – Nicola Talbot Nov 12 '17 at 13:07
• @NicolaTalbot Then a previous test for \bgroup is necessary. – egreg Nov 12 '17 at 13:28