How to use underscores with pgfplotstable?

Question

I have an input file like this:

some_entry 123
another_entry 456
yet_another_entry 789

and I want to typeset it as a table with:

\pgfplotstabletypeset[col sep=space]{FILENAME}

Problem is that LaTeX seems to interpret the underscores as a command to typeset subscript.

Escaping the underscores in the input file does not work.

Answer 1

Here is a solution for the problem which works without category codes:

\documentclass{standalone}

\usepackage{pgfplotstable}

\begin{document}
\pgfplotstabletypeset[
  header=false,
  columns/0/.style={
    string type,
    postproc cell content/.code={%
        \pgfplotsutilstrreplace{_}{\_}{##1}%
        \pgfkeyslet{/pgfplots/table/@cell content}\pgfplotsretval
    },
  },
  ]{
some_entry 123
another_entry 456
yet_another_entry 789
}
\end{document}

the solution is based on the following observations:

1. you do not really have column names, right? I added header=false which causes pgfplotstable to assign column indices as names (starting with 0).

2. Your first column is of string type and not a number, so I added that as style.

3. This is, of course, the key part: LaTeX (not pgfplotstable) interpretes _ as math mode subscript token. So: we have the choice to render the entire text in math mode (not what we want) or we replace the subscript token by a suitable text token. That's what I did: I added a postprocessor which applies string search-and-replace: it replaces _ by \_.

The approach works for both inline tables and input files.

---

Note that the story would be slightly different if you had column names containing underscores - in this case, you would need to add column name such that the column name is typeset correctly and our example would become:

\documentclass{standalone}

\usepackage{pgfplotstable}

\begin{document}
\pgfplotstabletypeset[
  columns/A_d/.style={
    string type,
    column name=$A_d$,
    postproc cell content/.code={%
        \pgfplotsutilstrreplace{_}{\_}{##1}%
        \pgfkeyslet{/pgfplots/table/@cell content}\pgfplotsretval
    },
  },
  ]{
  A_d B
some_entry 123
another_entry 456
yet_another_entry 789
}
\end{document}

This works because math mode _ are actually expandable, so they can be part of column names.

Answer 2

\pgfplotstabletypeset[col sep=space,columns/colname1/.style={string type}]{FILENAME}

And the file should look like that:

colname1 colname2
some\_entry 123
another\_entry 456
yet_another\_entry 789

Answer 3

This is not really an answer - I was motivated by @egreg's comment to try to hack further, but I failed, and these are my notes; maybe they'll help eventually finding a hack/workaround.

The thing is, I have a .csv file that would otherwise pass in other software - and it would have passed in pgfplotstable too, if it wasn't for the damn underscores; so now I'd have to escape that file, and have copies of the same data in two files, which I dislike extremely.

So basically I tried to write a small regex script in Perl that would automatically escape nasty characters; and then with -shell-escape, capture the \write18 output, to be used as input for \pgfplotstableread; unfortunately, it seems that as long as we use a token/macro in its argument, the \pgfplotstableread will decide it's a filename, not inline content - and it will of course, fail finding such a file:

! Package pgfplots Error: Could not read table file '" etmp "'. In case you intended to provide inline data: maybe TeX screwed up your end-of-lines? [...]

or

! Package pgfplots Error: Could not read table file 'var name,var length,value, [...]

The few times when I did manage to get stuff interpreted as data (cannot remember how, though), the process would fail with:

\pgfplotstable@result ...futilensuremath {4}\\GDM_
                                                  LANG&\pgfutilensuremath {8...

I cannot tell exactly, but it looks like in this case, pgfplotstable actually tries to make command tokens out of the data (probably via \csname .. \endcsname) - and because of this, I guess no changing of catcodes would ever help (as can be seen by the example below).

Btw, the only thing I did manage to do once, but I couldn't reproduce twice, was to get Latex to somehow enter mathmode while typesetting the table, in which case it didn't choke on the underscores anymore, and could compile - but, it interpreted them, and typeset them, as subscripts. Interestingly enough, that occured while I was trying to disable math codes for underscore (note that I don't really know if math codes can be disabled; never found an explicit reference about that - just tried to change them wildly, and see if I can get a similar effect; I cannot).

Here I'd just like to note How can LaTeX code in a data file be read by pgfplotstable? :

pgfplotstable assumes that column names do not contain expandable material.
Its way of dealing with "display names" is to provide the display names explicitly using the column name key which is used during \pgfplotstabletypeset.
Protecting expandable material in column names is unsupported. It is unlikely that pgfplotstable will support such protection automatically in the future (because protection cannot be done during \edef which is a common use case of access to columns).
So, the answer is: no, this is generally unsupported. Period.

So here is the, currently uncompilable, example code (with comments) that I used so far - maybe it helps eventually finding a hack for this problem:

\documentclass{article}

% \usepackage{tikz} %graphicx
% \usepackage{xcolor} % \pagecolor
% \pagecolor{yellow!15}

\usepackage{pgfplotstable}
\usepackage{tikz}
\usepackage{filecontents}
\usepackage{trace}

\def\csvfile{test.csv}
\begin{filecontents*}{\csvfile}
%# this is my .csv table example; data generated with:
%# $ env | grep 'G\(NO\|DM\)' | column -t -s'=' | awk '// {printf("%s,%s,%s,%s\n", $1,length($1),$2,length($2));}' | sed 's/_/-/g'
var name,var length,value,value length
GNOME-KEYRING-CONTROL,21,/tmp/keyring-JcALFh,19
GDM-KEYBOARD-LAYOUT,19,us,2
GNOME-KEYRING-PID,17,1989,4
GDM_LANG,8,en-DK,5
GDMSESSION,10,gnome-2d,8
GNOME-DESKTOP-SESSION-ID,24,this-is-deprecated,18
\end{filecontents*}

% don't include # here, we need it as comment char? it needs to be, for \x
% don't escape dollar, though? cannot, also it alone is mathmode..
% also escaped \\?
% actually, the easiest is to add noexpand everywhere, instead of just a backslash??
\begin{filecontents*}{escdat.sh}
perl -ne '$_ =~ s/([@#%{}])/\\noexpand$1/g;
$_ =~ s/(\n)/^^J$1/g;
$_ =~ s/(_)/\\noexpand\\$1/g;
print $_' "$1"
\end{filecontents*}

% \let\)\relax % in the end, \{ causes ! TeX capacity exceeded,
% the problem is all these escapes end in commands!
% escape all in a group @#_\$%{} :
% echo -e " # $ % { _ } \n # # % % $ $ # @ " | perl -ne '$_ =~ s/([@#_\$%{}])/\\$1/g; print $_'
% https://tex.stackexchange.com/a/16794/2595
\begingroup\makeatletter\endlinechar=\m@ne\everyeof{\noexpand}
\edef\x{\endgroup\def\noexpand\getEscaped{\@@input|"bash escdat.sh \csvfile" }}\x
% \global\let\textunderscore\relax
% \typeout{\getEscaped}

\gdef\smallcheck{ %
  \typeout{BSL '\BSL' \the\catcode\expandafter`\BSL, %
    DLR '\DLR' \the\catcode\expandafter`\DLR, %
    NHS '\NHS' \the\catcode\expandafter`\NHS, %
  }
}

% https://tex.stackexchange.com/a/185740/2595
% https://tex.stackexchange.com/a/69294/2595

\begingroup
  \catcode `\|=0
  |catcode `|\=11
  |catcode `|$=11
  |catcode `|#=11
  |catcode `|_=11
  |gdef|LF{\n} % \LF becomes (ASCII) "\n" (verbatim char!)
  |gdef|n{\n} % \n becomes (ASCII) "\n" (verbatim char!)
  |gdef|ELF{\\n} % \ELF becomes (ASCII) "\\n" - escaped backslash for shell!
  |gdef|BSL{\} %
  |gdef|DLR{$} %
  |gdef|NHS{#} %
  |gdef|USC{_} %
%   |xdef|etmp{|getEscaped}
  |smallcheck % new catcodes here: BSL '\' 11, DLR '$' 11, NHS '#' 11,
|endgroup

% back to old catcodes here: BSL '\' 0, DLR '$' 3, NHS '#' 6,
\smallcheck

\gdef\textunderscore{}
\edef\etmp{\getEscaped}
\typeout{\etmp}

\traceon
% \begingroup
\global\let\mytable\relax
\edef\tmpcmd{\noexpand\pgfplotstableread[%
  col sep=comma,
  % hash/number sign # should be comment char by default.
  % # is: ASCII 35=0x23 hex
  %comment chars={\^^23},
  %ignore chars={,\^^24}, % 0x5c=92='\', 0x24=36='$' % bad, ! Improper alphabetic constant. for \uC macro:->\uC ,\$ - backslash?
  %comment chars={\#}, % can do without this
  ignore chars={\noexpand\\,\noexpand\$,\noexpand\{}, % \BSL, \DLR are: ! Improper alphabetic ...
  string replace*={\noexpand\USC}{...}, %nowork % {\_}{+},% nowork
  header=true,
% ]{./test.csv}\mytable
]{ ^^J
\# trying a hack \noexpand\\^^J
\etmp
}\noexpand\mytable %
} \tmpcmd
% ! Package pgfplots Error: Could not read table file '" etmp "'. In case you intended to provide inline data: maybe TeX screwed up your end-of-lines? Try `row sep=crcr' and terminate your lines with `\\' (refer to the pgfplotstable manual for details).
% means it is read with \string - no chance to expand a macro inside there
% \endgroup
% with the \edef and traceon:
% \GenericError  ...
%                                                   \endgroup
% <to be read again>
%                    \#
% \pgfplotstableread@filename ->"
%  \#
%                                      trying a hack \\
%  var name,var length,v...
% \pgfplotstableread@openfile ...tableread@filename
%                                                   .tex \ifeof \r@pgfplots@re...
% \pgfplotstableread@impl@ ...otstableread@openfile
%                                                   \def \pgfplotstableread@im...
% l.95 } \tmpcmd


\begin{document}
\begin{tikzpicture}

\begingroup
  \catcode`_=12
  \gdef\myunds{_}
  \typeout{catcode 1: \the\catcode`_} %catcode 1: 11
\endgroup %

\typeout{catcode 2: \the\catcode`_} % catcode 2: 8
% \typeout{catcode 3: \the\catcode`\myunds} % ! Improper alphabetic constant.
\typeout{catcode 3: \the\catcode\expandafter`\myunds} % catcode 3: 8

\typeout{myunds \myunds}

% https://tex.stackexchange.com/questions/62705/underscore-in-textmode-vs-mathmode

\node[] at (current bounding box.south) {
\resizebox{\textwidth}{!}{
\begingroup%
% \traceon%
\makeatletter\@makeother\_%
\catcode`\_=11 %
\catcode`_=12
\begingroup\lccode`~=`_
\lowercase{\endgroup\let~}\sb
\mathcode`\_="202B %\mathcode`_="8000
\gdef\_{W}
\pgfplotstabletypeset[%
  every row/.style={before row={\catcode`_=11\mathcode`\_="202B\def_{+}\relax}},
  font=\tiny,
  %   every row/.style={before row={}},
  string replace*={_}{\myunds},%
  columns/value/.style={string type},% ,column type={l}
  columns/var name/.style={string type},% ,column type={l}
]\mytable
\endgroup
% \makeatother%
}%
};

% for \pgfplotstabletypeset[]\etmp:
% ! Package pgfplots Error: Could not read table file 'var name,var length,value,
% value length
% GNOME-KEYRING-CONTROL,21,/tmp/keyring-JcALFh,19
% so it thinks its a a filename, which it isn't

\end{tikzpicture}

% trace gives:
% \pgfplotstable@result ...futilensuremath {4}\\GDM_
%                                                   LANG&\pgfutilensuremath {8...

\end{document}