# Find the locations of blank lines in .csv file

This is related to a previous question I asked - Plotting data blocks of varying length using PGFPlots and gnuplot.

Here is a minimum working example:

% !TeX TXS-program:compile = txs:///pdflatex/[--shell-escape]
\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots,filecontents}

\begin{filecontents*}{data.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents*}

\begin{document}
\begin{tikzpicture}
\begin{axis} [width=0.5\textwidth,height=7cm,
]
\addplot gnuplot [raw gnuplot, mark=none, black]{
set datafile separator comma;
plot "<(sed -n '3,6p' data.csv)" using 1:2 with lines;
};
\addplot gnuplot [raw gnuplot, mark=none, red]{
set datafile separator comma;
plot "<(sed -n '10,16p' data.csv)" using 1:2 with lines;
};
\end{axis}
\end{tikzpicture}
\end{document}


My question is, is there a way to get TeX to locate the blank lines in the .csv file? Using datatool or something like that? If I could get locations of the blank lines, it could tell me that (a) there are two data sets, and (b) I could work out the length of each data set.

In this example, the blank lines are at line 7 and line 17. So data set 1 goes from line 3 to 6 (7-1) and data set 2 goes from 10 (7+3) to 16 (17-1). This would enable me to automatically generate the gnuplot commands and enable me to answer Plotting data blocks of varying length using PGFPlots and gnuplot.

• Sad to say, the \readdef from my other answer appears to skip over blank lines in the input, and so is unable to detect them explicitly. – Steven B. Segletes May 1 at 3:03

The use of xstring package may help:

\documentclass[a4paper,12pt]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{filecontents,xstring}
\begin{filecontents}{data.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"Amplitude","notes: data set 3",
X,Y,
1,2
3,4
5,6,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents}
\newcount\cntblanklines
\newcount\currentdataline
\begingroup
\catcode0 12
\begingroup\everyeof{\noexpand}\endlinechar0\xdef\datas{\csname @@input\endcsname data.csv }\endgroup
\StrCount\datas{^^00^^00}[\nbblanklines]\global\let\nbblanklines\nbblanklines
\loop
\ifnum\cntblanklines<\nbblanklines\relax
\StrCut\datas{^^00^^00}\currentdatas\datas
\StrCount\currentdatas{^^00}[\currentblocklength]%
\expandafter\xdef\csname blockbegin\romannumeral\cntblanklines\endcsname{\number\numexpr\currentdataline+3}%
\expandafter\xdef\csname   blockend\romannumeral\cntblanklines\endcsname{\number\numexpr\currentblocklength+1+\currentdataline}%
\repeat
\endgroup%
\begin{document}
Number of blank lines : \nbblanklines

Block 1 : \blockbegini--\blockendi

Block 2 : \blockbeginii--\blockendii

Block 3 : \blockbeginiii--\blockendiii
\end{document}


I originally modified the definition of \readdef (readarray package) to not ignore blank lines in the input. However, here I EDITED to define \simplereaddef that eliminates the overhead of the readarray package altogether.

\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots,filecontents,listofitems}
\makeatletter
\catcode\endlinechar=9 %
\def#3{}%
\repeat%
\catcode\endlinechar=5 %
}
\makeatother
\begin{filecontents*}{mydata.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents*}
\begin{document}
\setsepchar{\\}
\noindent\foreachitem\x\in\myarray[]{%
\ifnum\xcnt<\listlen\myarray[]\relax%
\expandafter\ifx\expandafter\relax\x\relax Line \xcnt{} blank\\\fi%
\fi}
\end{document}


SUPPLEMENT

Since the overall desire is not just to find the blank records of the file, but to use them to obtain sublists of the original file that are delimited by blank records. Here is one way, EDITED to use listofitems nested lists.

In the \sublist output, I also add a leading #) to each sub-record to show that the sublist is not merely a block of text, but individually accessible records within the sub-block of text.

EDITED to use a \simplereaddef macro, instead of modifying that from \readarray package.

\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots,filecontents,pgffor,listofitems}
\makeatletter
\catcode\endlinechar=9 %
\def#3{}%
\repeat%
\catcode\endlinechar=5 %
}
\makeatother
\begin{filecontents*}{mydata.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents*}
\newcommand\sublist[1]{SUBLIST #1:\\\foreachitem\x\in\myarray[#1]{\xcnt) \x\\}\par}
\begin{document}
\simplereaddef[\\]{mydata.csv}\mydata% OPTIONAL ARG IS record-delim (DEFAULT ,)
\setsepchar{\\\\/\\}% OF FORM {2X record-delim / record-delim}
\ignoreemptyitems
Number of non-empty sublists: \listlen\myarray[]

\sublist{1}

\sublist{3}

\sublist{2}
\end{document}


• The final value \xcnt{} = 22 which is unexpected as is the blank 20 line I would have expected \xcnt{} to be 20 at end of pass. The aim of the question (one of a series of many) is how to capture the feedback eg given that first response is 7 and second is 17 the difference is 10 and subtracting 3 we get the desired value of 7 entries starting at 10th position running to 16th (inclusive) but datatool will subsequently need range as lines 9-15 – user170109 May 1 at 4:08
• @KJO The two "phantom" records arise for known reasons. Record 20 arises because filecontents adds a linefeed following the data on line 19. Most editors and the TeX \read routine digest this ending blank line as a record. The\readdef macro, reading this record, attaches a \read@array@sepchar (\) following it Then, when listofitems parses the record looking for instances of \\, it too will interpret the emptyness beyond the trailing \\ as a blank record (unless I set the ignoreblankitems declaration which defeats the purpose of the algorithm) . – Steven B. Segletes May 1 at 9:47
• @KJO Therefore, a record length of \listlen\myarray[] of 21 is totally expected. One can design the algorithm accordingly. – Steven B. Segletes May 1 at 9:50
• Thanks for the explanation and real world someone may have truly blank lines at the end I see you are commenting that 21 may be acceptable as I query so will test that aspect out, the main question is how to use the returned values easily in secondary actions – user170109 May 1 at 9:53
• @KJO Knowing these things, perhaps I should have set the final test loop to \noindent\foreachitem\x\in\myarray[]{% \ifnum\xcnt<\numexpr\listlen\myarray[]-1\relax% \expandafter\ifx\expandafter\relax\x\relax Line \xcnt{} blank\\\fi% \fi} – Steven B. Segletes May 1 at 9:57

This is an incomplete collection of thoughts. It is only to report that pgfplots does come with means to treat empty lines. And one can inject something in the scanline option (empty line=scanline, see pp. 45 of the manual). The main thing I am proposing here is to use this information by adding

  \xdef\BlockLength{\pgfplots@scanlinelength}%


to \pgfplotsscanlinelength@scanline@complete in order to keep track of the block lengths. What the following does is to go over the file, find out the length of the blocks (including the header) and to record them. This is neither elegant nor fully tested, let a lone a complete answer, but seems to survive some very basic checks.

\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots,filecontents}

\begin{filecontents*}{data.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"Amplitude","notes: data set 3",
X,Y,
3,5,
4,4,
5,3,
6,2,
7,1,

"Amplitude","notes: data set 4",
X,Y,
3,5,
4,4,
5,3,
6,2,
6,2,
7,1,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents*}
\makeatletter
\def\pgfplotsscanlinelength@scanline@complete{%
\ifnum\pgfplots@scanlinelength>0
\ifnum\c@pgfplots@scanlineindex=0
%
% \pgfplotsscanlinecomplete
% \pgfplotsscanlinecomplete
% \pgfplotsscanlinecomplete
% should have the same effect as a single statement. Do
% nothing here.
\else
\ifnum\pgfplots@scanlinelength=\c@pgfplots@scanlineindex\relax
\else
%\message{Found inconsistent scan line length: \pgfplots@scanlinelength\space vs. \the\c@pgfplots@scanlineindex\space near line \pgfplotstablelineno.}%
% special marker which means 'inconsistent scan line length found'
\def\pgfplots@scanlinelength{-2}%
\fi
\pgfplotsplothandlernotifyscanlinecomplete
\fi
\else
\ifnum\pgfplots@scanlinelength=-2
\else
\edef\pgfplots@scanlinelength{\the\c@pgfplots@scanlineindex}%
\xdef\BlockLength{\pgfplots@scanlinelength}%
\fi
%
\ifnum\c@pgfplots@scanlineindex>0
\pgfplotsplothandlernotifyscanlinecomplete
\fi
\fi
\c@pgfplots@scanlineindex=0
}
\makeatother
\newsavebox{\NonSense}
\begin{document}
\begin{lrbox}{\NonSense}
\begin{tikzpicture}
\pgfmathtruncatemacro{\DataLines}{\pgfplotsretval+1}
\typeout{\DataLines}
\def\SkipLength{0}
\foreach \X in {0,...,12}
{\begin{axis}
\addplot[empty line=scanline] table[x expr=0,y expr=0,skip first n=\SkipLength] {data.csv};
\end{axis}
\ifnum\X=0
\xdef\LstBlocks{\BlockLength}
\pgfmathtruncatemacro{\SkipLength}{\BlockLength+2}
\xdef\SkipLength{\SkipLength}
\else
\xdef\LstBlocks{\LstBlocks,\BlockLength}
\pgfmathtruncatemacro{\SkipLength}{\SkipLength+\BlockLength+2}
\xdef\SkipLength{\SkipLength}
\fi
\ifnum\SkipLength>\DataLines
\breakforeach
\fi
}
\end{tikzpicture}
\end{lrbox}
blocks:\LstBlocks
\end{document}


So the main point of this post is to report on this empty line thingy, hoping some expert may find it useful for a true and complete solution.

• Thanks marmot at first glance it seems a lot of inline coding (can some be \included as an external subroutine ?) but if it is much closer to user requirement for tweaking alongside plots then it may not help if its out of view – user170109 May 1 at 23:44
• @KJO Most of the inline coding is in fact just copied from the pgfplots code. I guess some expert will be able to patch the command in such a way that copying is no longer needed. Yet the point of this post is to say that pgplots has already some built-in means to detect the empty lines, it just does not broadcast them to us. Even more, it just gives up if the blocks are of different lengths, and the only thing I do is to hook in at the point it realizes this and make it broadcast the length of the block (meaning also that it must be possible to do that with LaTeX "only".) – user121799 May 2 at 2:25