4

I have a pretty good understanding of LaTex, but I've spent a while now digging up information on catcodes, but because they're lesser known, I haven't seen any applicable examples that will help with my problem.

For those of you familiar with any experimental work with data tables, you'll probably be aware that the standard way to store data as numbers in is a table, either tab or space delimited, with enters for a new line. This is how many data processing programs read and write data.

As a result, if I want to enter data into LaTex, I have to reformat my data table to include ampersands between columns and a double-backslash to end lines. Is there a straight forward way to create a new environment, which redefines spaces and tabs and double-backslashes, inserts a tabular environment, and then undoes the redefinitions so that they do not have any effect on the rest of the code, something like

\newenvironment{datatable}[2]  
\catcode{10}=\catcode{4} 
% catcode 10 is spaces and tabs, catcode 4 is an ampersand 

\catcode{5}=\\ % catcode 5 is the return character
% I haven't a clue how to define it as a double-backslash   

\begin{tabular}{#1}  
   \#2  
\end{tabular}  
% I have no idea how I would undo the definitions from above, because if I said   
\catcode{10}=\catcode{10} 
% I'm not sure it would make any sense
% catcode 10 already points to catcode 4
% and I haven't seen anyone use an undo command for catcodes yet 

\catcode{10}=\catcode{4} 
% catcode 10 is spaces and tabs, catcode 4 is an ampersand 

\catcode{5}=\\ 
% catcode 5 is the return character, I haven't a clue how to define it
% as a double-backslash   

\begin{tabular}{#1}  
   #2  
\end{tabular}  
% I have no idea how I would undo the definitions from above
% because if I said   
\catcode{10}=\catcode{10} 
% I'm not sure it would make any sense
% catcode 10 already points to catcode 4
% and I haven't seen anyone use an undo command for catcodes yet 

Hopefully it's clear what I want to do, but effectively I just want to get a better understanding of redefining catcodes, and then undoing redefinitions. Hopefully this question also helps some other Tex users as well. Thanks for your help!

Edit:
The solution does not require the use of catcodes. In my case, I require the ability to implement the solution on overleaf.com (which is just an online latex editor and compiler), but other than that, I have no restrictions.

3
  • Welcome to TeX.SE. Does the solution have to involve catcodes, or might a LuaLaTeX-based solution be acceptable as well? Please advise.
    – Mico
    Commented Jan 20, 2019 at 4:11
  • I've edited my original question to answer your comment. That was a good point that I did not clarify, so thank you for your comment!
    – Kraig
    Commented Jan 20, 2019 at 4:17
  • 1
    I think you simply need csvsimple or pgfplotstable or datatool packages.
    – CarLaTeX
    Commented Jan 20, 2019 at 4:23

3 Answers 3

5

TeX maintains a catcode array; this array has length 256 (indexed from 0) in 8-bit engines and 0x110000 in Unicode engines (XeTeX and LuaTeX). Each entry in the array should be a 4-bit number.

You assign an entry by a declaration of the form

\catcode<number> = <4-bit number>

(the = and the spaces around it are optional); the first <number> must be in the allowed range, that is, 0–255 for 8-bit engines (Knuth TeX, pdftex) or 0–1114111 for Unicode engines.

If used in a non assignment context, \catcode<number> returns the corresponding entry in the array. For instance, \the\catcode`a returns 11 (under standard setup).

Numbers can be input in one of the standard TeX ways: an integer in its decimal representation, an octal number, a hexadecimal number, or by character code:

\catcode 97 = 11
\catcode '141 = 11
\catcode "61 = 11
\catcode `a = 11
\catcode `\a = 11

are all equivalent and assign character a the catcode 11; more technically, they assign the number 11 to the entry in the catcode array indexed by 97. The <number> can also be anything that returns a number in the context (a counter's value, a \chardef token, a \mathchardef token, \numexpr, an internal array entry). Also the right-hand side in the assignment can be expressed in different formats. For instance

\catcode`a=\catcode`b

will assign a the same category code as b (whatever is the current category code of b. A common idiom is \catcode<number>=\active, where \active is a \chardef token whose value is 13.

Thus your \catcode{10}=\catcode{4} is syntactically wrong. Note that there's no way of telling “change all characters of category code 10 into category code 4” without looping through the array; in an 8-bit engine

\count255=0
\loop\ifnum\count255<256
  \ifnum\catcode\count255=10 \catcode\count255=4 \fi
  \advance\count255 by 1
\repeat

would change the characters of category code 10 into characters of category code 4.

When TeX is in the tokenization phase, interpreting input and forming tokens from it, it attaches to character tokens the corresponding category code.

Let's attack your problem. You want to input a file, but assigning different category codes to some characters, namely tab and spaces, but also interpreting the end-of-line as \\. Let's use a syntax such as

 \maketabularfromfile{<table specs>}{<filename>}

Here's a possible code:

\begin{filecontents*}{\jobname.dat}
a   b c
1   2 3
4   5 6
\end{filecontents*}

\documentclass{article}

\makeatletter
\newcommand{\maketabularfromfile}[2]{%
  % confine the changes
  \begingroup
  \catcode`\ =4   % space is column separator
  \catcode`\^^I=4 % tab is column separator
  \catcode`\^^M=\active % end-of-line is active
  \begingroup\lccode`~=`\^^M \lowercase{\endgroup\def~}{\\}%
  \begin{tabular}{#1}
  \@@input #2 % use the primitive \input
  \end{tabular}
  \endgroup
}
\makeatother

\begin{document}

\maketabularfromfile{ccc}{\jobname.dat}

\end{document}

I used <TAB>s as separators for the second column, just by way of example (the site will most likely convert them to spaces).

enter image description here

This is very crude, of course. But since there are packages such as datatool or cvssimple that do a much better job, I don't think it's worthwhile to reinvent the wheel.

1
  • I really appreciate your explanation. I think I will look into those other packages, but based on my question your answer is the best and most applicable!
    – Kraig
    Commented Jan 20, 2019 at 13:50
5

I think you're complicating your life, what you actually need is "read a text file and produce a table".

There are many packages for doing that. In the following MWE, you can find some examples with csvsimple and pgfplotstable, but there are others: datatool is more powerful, even if perhaps not for beginners. I recommend to you to read their documentation and leave \catcodes to egreg & Co.

I've made my example on Overleaf, with the usual pdfLaTeX compiler.

\documentclass{article}

\usepackage{booktabs}
\usepackage{siunitx}
\usepackage{csvsimple}
\usepackage{pgfplotstable}
\pgfplotsset{compat=1.14}
% you can set some option for all your table if you write this 
\pgfplotstableset{string type, 
    every head row/.style={before row=\toprule,after row=\midrule},
    every last row/.style={after row=\bottomrule}}

\usepackage{filecontents}% only needed to create the csv in this example
% the following filecontents* environments are needed only to create the text files, you don't need the if you already have yourfile....dat
\begin{filecontents*}{yourfilecommasep.dat}
Name,Surname,Length,Gender
Paulinho,van Duck,.56,M
Paulette,de la Quack,.52,F
\end{filecontents*}
\begin{filecontents*}{yourfilespacesep.dat}
Name Surname Length Gender
Paulinho {van Duck} .56 M
Paulette {de la Quack} .52 F
\end{filecontents*}

\begin{document}
\verb|pgfplotstable| accepts space, tab, comma, semicolon, colon, braces, and ampersand as 
separators.

\begin{center}
\pgfplotstabletypeset[
    col sep=space, 
    ]
    {yourfilespacesep.dat}
\end{center}

With \verb|\pgfplotstabletypeset|, you don't even need a file, you can type your data directly.

\begin{center}
\pgfplotstabletypeset[
    col sep=space, 
    ]
    {
    Name Surname Length Gender
    Paulinho {van Duck} .56 M
    Paulette {de la Quack} .52 F
    }
\end{center}

\verb|csvsimple| accepts comma (the default), semicolon, pipe, and tab as separators.

Here I also merged the two colums "Name" and "Surname", and inverted the two colums "Gender" and "Length":
\begin{center}
\begin{tabular}{
    lcS[table-format=1.2,round-mode=places]}
    \toprule
    Ducks and drakes & Gender & {Length} \\
    & & {(\si{\metre})} \\
    \midrule
    \csvreader[head to column names,
        late after line=\\]{yourfilecommasep.dat}{}%
        {\Name\ \Surname & \Gender & \Length}
    \bottomrule
\end{tabular}
\end{center}

\end{document}

enter image description here

2
  • This is a great answer, and I've been putting it to use. HOWEVER, in your preamble declarations you write " \pgfplotstableset{string type, .... " which seems to ruin custom column definitions, like precision. Removing string type fixes it all. I'm not sure what purpose it serves.
    – Kraig
    Commented Jan 20, 2019 at 15:03
  • @Kraig string type is because I put some alphabetical columns, but you can define the type you like for any column, see the package documentation.
    – CarLaTeX
    Commented Jan 20, 2019 at 15:51
5

Changing catcodes is always a bit tricky especially with space, and especially if here input may contained braced items with spaces too.

So I am changing only catcode of end-of-line.

I am illustrating here inline input, if data is in external file, small variant is needed. But anyhow, there are packages for this, of course method here is only a few lines of macros.

\documentclass{article}

\usepackage{array}
\usepackage{booktabs}

\makeatletter
\def\mytabular@end{\end{mytabular}}
\def\mytabular@parse#1 %
{%
  \if\relax\detokenize{#1}\relax
    \g@addto@macro\mytabular@line{\\}%
    \xdef\mytabular@tabular{%
       \unexpanded\expandafter{\mytabular@tabular}%
       \unexpanded\expandafter\expandafter\expandafter{\mytabular@line}%
       }%
    \expandafter\mytabular@again
  \else
    \g@addto@macro\mytabular@line{&#1}%
    \expandafter\mytabular@parse
  \fi
}%
% some complications to insert toprule but then midrule only once
% after header line
\def\mytabular@newline{%
    \let\mytabular@newline\mytabular@@newline
    \gdef\mytabular@line{\mytabular@atstart\toprule}%
}%
\def\mytabular@@newline{%
    \let\mytabular@newline\mytabular@@@newline
    \gdef\mytabular@line{\mytabular@atstart\midrule}%
}%
\def\mytabular@@@newline{%
    \gdef\mytabular@line{\mytabular@atstart{}}%
}%
\def\mytabular@atstart#1&{#1}%
\def\mytabular@finish{%
    \g@addto@macro\mytabular@tabular{\bottomrule\end{tabular}}%
    \end{mytabular}\mytabular@tabular
}%
\begingroup\catcode`\^^M\active
  \gdef\mytabular@again{\expandafter^^M}%
\endgroup
\def\zzzzzzzzzzzzzzzzzzzzz#1{%
\newenvironment{mytabular}[1]%
  {%
   \gdef\mytabular@tabular{\begin{tabular}{##1}}%
   \obeylines
   \begingroup\lccode`~`\^^M 
   \lowercase{\endgroup\def~####1~}{%
     \def\mytabular@tmp{####1}%
     \ifx\mytabular@tmp\mytabular@end
       \expandafter\mytabular@finish
     \else
       \mytabular@newline%
       \mytabular@parse####1 #1%
     \fi}%
  }%
  {}%
}\zzzzzzzzzzzzzzzzzzzzz{ }%
\makeatother

\begin{document}

\begin{mytabular}{cccc}
    Name Surname Length Gender
    Paulinho {van Duck} .56 M
    Paulette {de la Quack} .52 F
\end{mytabular}

\end{document}

enter image description here

1
  • due to @DavidCarlisle I must take extra precautions to not overwrite crucial macros (hence the one with a long name)
    – user4686
    Commented Jan 20, 2019 at 12:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .