Find the locations of blank lines in .csv file

Question

This is related to a previous question I asked - Plotting data blocks of varying length using PGFPlots and gnuplot.

Here is a minimum working example:

% !TeX TXS-program:compile = txs:///pdflatex/[--shell-escape]
\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots,filecontents}

\begin{filecontents*}{data.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents*}

\begin{document}
    \begin{tikzpicture}
        \begin{axis} [width=0.5\textwidth,height=7cm,
        ]
        \addplot gnuplot [raw gnuplot, mark=none, black]{
            set datafile separator comma;
            plot "<(sed -n '3,6p' data.csv)" using 1:2 with lines;
        };
            \addplot gnuplot [raw gnuplot, mark=none, red]{
        set datafile separator comma;
        plot "<(sed -n '10,16p' data.csv)" using 1:2 with lines;
    };
        \end{axis}
    \end{tikzpicture}
\end{document}

My question is, is there a way to get TeX to locate the blank lines in the .csv file? Using datatool or something like that? If I could get locations of the blank lines, it could tell me that (a) there are two data sets, and (b) I could work out the length of each data set.

In this example, the blank lines are at line 7 and line 17. So data set 1 goes from line 3 to 6 (7-1) and data set 2 goes from 10 (7+3) to 16 (17-1). This would enable me to automatically generate the gnuplot commands and enable me to answer Plotting data blocks of varying length using PGFPlots and gnuplot.

Answer 1

The use of xstring package may help:

\documentclass[a4paper,12pt]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{filecontents,xstring}
\begin{filecontents}{data.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"Amplitude","notes: data set 3",
X,Y,
1,2
3,4
5,6,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents}
\newcount\cntblanklines
\newcount\currentdataline
\begingroup
    \catcode0 12
    \begingroup\everyeof{\noexpand}\endlinechar0\xdef\datas{\csname @@input\endcsname data.csv }\endgroup
    \StrCount\datas{^^00^^00}[\nbblanklines]\global\let\nbblanklines\nbblanklines
    \loop
        \ifnum\cntblanklines<\nbblanklines\relax
        \advance\cntblanklines1
        \StrCut\datas{^^00^^00}\currentdatas\datas
        \StrCount\currentdatas{^^00}[\currentblocklength]%
        \expandafter\xdef\csname blockbegin\romannumeral\cntblanklines\endcsname{\number\numexpr\currentdataline+3}%
        \expandafter\xdef\csname   blockend\romannumeral\cntblanklines\endcsname{\number\numexpr\currentblocklength+1+\currentdataline}%
        \advance\currentdataline\numexpr\currentblocklength+2\relax
    \repeat
\endgroup%
\begin{document}
Number of blank lines : \nbblanklines

Block 1 : \blockbegini--\blockendi

Block 2 : \blockbeginii--\blockendii

Block 3 : \blockbeginiii--\blockendiii
\end{document}

Answer 2

I originally modified the definition of \readdef (readarray package) to not ignore blank lines in the input. However, here I EDITED to define \simplereaddef that eliminates the overhead of the readarray package altogether.

\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots,filecontents,listofitems}
\makeatletter
\newread\readfile
\newcommand\simplereaddef[3][,]{%
  \catcode\endlinechar=9 %
  \def#3{}%
  \openin\readfile=#2%
  \loop\unless\ifeof\readfile%
    \read\readfile to\readfileline % Reads a line of the file into \readfileline%
    \expandafter\g@addto@macro\expandafter#3\expandafter{\readfileline}%
    \g@addto@macro#3{#1}% ADD record-delim TO END OF EACH RECORD
  \repeat%
  \closein\readfile%
  \catcode\endlinechar=5 %
}
\makeatother
\begin{filecontents*}{mydata.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents*}
\begin{document}
\simplereaddef[\\]{mydata.csv}\mydata
\setsepchar{\\}
\readlist\myarray{\mydata}
\noindent\foreachitem\x\in\myarray[]{%
  \ifnum\xcnt<\listlen\myarray[]\relax%
    \expandafter\ifx\expandafter\relax\x\relax Line \xcnt{} blank\\\fi%
  \fi}
\end{document}

SUPPLEMENT

Since the overall desire is not just to find the blank records of the file, but to use them to obtain sublists of the original file that are delimited by blank records. Here is one way, EDITED to use listofitems nested lists.

In the \sublist output, I also add a leading #) to each sub-record to show that the sublist is not merely a block of text, but individually accessible records within the sub-block of text.

EDITED to use a \simplereaddef macro, instead of modifying that from \readarray package.

\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots,filecontents,pgffor,listofitems}
\makeatletter
\newread\readfile
\newcommand\simplereaddef[3][,]{%
  \catcode\endlinechar=9 %
  \def#3{}%
  \openin\readfile=#2%
  \loop\unless\ifeof\readfile%
    \read\readfile to\readfileline % Reads a line of the file into \readfileline%
    \expandafter\g@addto@macro\expandafter#3\expandafter{\readfileline}%
    \g@addto@macro#3{#1}% ADD record-delim TO END OF EACH RECORD
  \repeat%
  \closein\readfile%
  \catcode\endlinechar=5 %
}
\makeatother
\begin{filecontents*}{mydata.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents*}
\newcommand\sublist[1]{SUBLIST #1:\\\foreachitem\x\in\myarray[#1]{\xcnt) \x\\}\par}
\begin{document}
\simplereaddef[\\]{mydata.csv}\mydata% OPTIONAL ARG IS record-delim (DEFAULT ,)
\setsepchar{\\\\/\\}% OF FORM {2X record-delim / record-delim}
\ignoreemptyitems
\readlist\myarray{\mydata}
Number of non-empty sublists: \listlen\myarray[]

\sublist{1}

\sublist{3}

\sublist{2}
\end{document}

Answer 3

This is an incomplete collection of thoughts. It is only to report that pgfplots does come with means to treat empty lines. And one can inject something in the scanline option (empty line=scanline, see pp. 45 of the manual). The main thing I am proposing here is to use this information by adding

  \xdef\BlockLength{\pgfplots@scanlinelength}% 

to \pgfplotsscanlinelength@scanline@complete in order to keep track of the block lengths. What the following does is to go over the file, find out the length of the blocks (including the header) and to record them. This is neither elegant nor fully tested, let a lone a complete answer, but seems to survive some very basic checks.

\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots,filecontents}

\begin{filecontents*}{data.csv}
"Amplitude","notes: data set 1",
X,Y,
1,1,
2,2,
3,3,
4,4,

"Amplitude","notes: data set 2",
X,Y,
1,7,
2,6,
3,5,
4,4,
5,3,
6,2,
7,1,

"Amplitude","notes: data set 3",
X,Y,
3,5,
4,4,
5,3,
6,2,
7,1,

"Amplitude","notes: data set 4",
X,Y,
3,5,
4,4,
5,3,
6,2,
6,2,
7,1,

"CH1","notes: data set 1",
"CH1","notes: data set 2",
\end{filecontents*}
\makeatletter
\def\pgfplotsscanlinelength@scanline@complete{%
        \ifnum\pgfplots@scanlinelength>0
                \ifnum\c@pgfplots@scanlineindex=0
                        %
                        % \pgfplotsscanlinecomplete
                        % \pgfplotsscanlinecomplete
                        % \pgfplotsscanlinecomplete
                        % should have the same effect as a single statement. Do
                        % nothing here.
                \else
                        \ifnum\pgfplots@scanlinelength=\c@pgfplots@scanlineindex\relax
                        \else
%\message{Found inconsistent scan line length: \pgfplots@scanlinelength\space vs. \the\c@pgfplots@scanlineindex\space near line \pgfplotstablelineno.}%
                                % special marker which means 'inconsistent scan line length found'
                                \def\pgfplots@scanlinelength{-2}%
                        \fi
                        \pgfplotsplothandlernotifyscanlinecomplete
                \fi
        \else
                \ifnum\pgfplots@scanlinelength=-2
                \else
                        \edef\pgfplots@scanlinelength{\the\c@pgfplots@scanlineindex}%
                        \xdef\BlockLength{\pgfplots@scanlinelength}%
                \fi
                %
                \ifnum\c@pgfplots@scanlineindex>0
                        \pgfplotsplothandlernotifyscanlinecomplete
                \fi
        \fi
        \c@pgfplots@scanlineindex=0
        \pgfplotsutil@advancestringcounter\pgfplotsscanlineindex%
}
\makeatother
\newsavebox{\NonSense}
\begin{document}
\begin{lrbox}{\NonSense}
\begin{tikzpicture}
\pgfplotstableread[comment chars={"}]{data.csv}\loadedtable
\pgfplotstablegetrowsof{\loadedtable}
\pgfmathtruncatemacro{\DataLines}{\pgfplotsretval+1}
\typeout{\DataLines}
\def\SkipLength{0}
\foreach \X in {0,...,12}
{\begin{axis} 
    \addplot[empty line=scanline] table[x expr=0,y expr=0,skip first n=\SkipLength] {data.csv};
\end{axis}
\ifnum\X=0
\xdef\LstBlocks{\BlockLength}
\pgfmathtruncatemacro{\SkipLength}{\BlockLength+2}
\xdef\SkipLength{\SkipLength}
\else
\xdef\LstBlocks{\LstBlocks,\BlockLength}
\pgfmathtruncatemacro{\SkipLength}{\SkipLength+\BlockLength+2}
\xdef\SkipLength{\SkipLength}
\fi
\ifnum\SkipLength>\DataLines
\breakforeach
\fi
}
\end{tikzpicture}
\end{lrbox}
blocks:\LstBlocks
\end{document}

So the main point of this post is to report on this empty line thingy, hoping some expert may find it useful for a true and complete solution.