5

I'm trying to understand how the TAB character of the keyboard behaves within verbatim. I have defined a new command using the v argument of the xparse, but, I get errors and a rather strange output so to speak. The package documentation says nothing about this specific character (If...I know it shouldn't be used, but,...).

This is my example file, with real tab character (thanks to @Ulrich for the tip):

\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand{\myxverb}{v}{#1}
\pagestyle{empty}
\begin{document}
%This is \verb*|verb with tbars and two keyboard TABs \t \t in argument| text, OK.\par
This is \verb*|verb with tbars and two keyboard TABs		in argument| text, OK.\par

%This is \myxverb|myxverb with tbars and two keyboard TABs \t \t in argument| text, NOT OK.\par
This is \myxverb|myxverb with tbars and two keyboard TABs		in argument| text, NOT OK.\par

%This is \myxverb{myxverb with brace and two keyboard TABs \t \t in argument} text, NOT OK.
This is \myxverb{myxverb with brace and two keyboard TABs		in argument} text, NOT OK.

\end{document}

An image of the output: outout Part of the .log file generated using pdflatex in TeXLive 2019 (update):

! LaTeX3 Error: The verbatim command '\myxverb' cannot be used inside an
(LaTeX3)        argument.

For immediate help type H <return>.
 ...                                              

l.10 ...|myxverb with tbars and two keyboard TABs   
                                                    in argument| text, NOT OK...

? 

! LaTeX3 Error: The verbatim command '\myxverb' cannot be used inside an
(LaTeX3)        argument.

For immediate help type H <return>.
 ...                                              

l.13 ...{myxverb with brace and two keyboard TABs   
                                                    in argument} text, NOT OK.
? 
! Too many }'s.
l.13 ... brace and two keyboard TABs        in argument}
                                                   text, NOT OK.
?

Any way to skip this problem?

Greetings

5
  • I think tabs 'fall through the cracks' here: one might argue that \verb is not ideal either. One can 'fix' with \edef\dospecials{\unexpanded\expandafter{\dospecials}\noexpand\do\noexpand\^^I}, but I'm not sure that's what you want ... – Joseph Wright Sep 12 '19 at 13:04
  • 1
    What is the expected output for tabs here anyway? – siracusa Sep 12 '19 at 13:24
  • Okay, this may help me, this is just a small example. I'm adapting some old files to a new template and I didn't want to modify the source files, in my real archive the argument is +v, as they are very old files, in some the character \t appears and in others not... the character TAB is quite a nuisance. I would expect them to be ignored (like a \verb) or turned into a space... getting -NoValue- is very strange to me :( – Pablo González L Sep 12 '19 at 13:34
  • I tried to find the source of the problem, but I failed. As Joseph mentioned, if ^^I (tab character) has catcode 12 (other), then the argument is parsed correctly. In the standard setting it has catcode 10 (space). While normal space characters work fine, the tab seems to throw the argument parser off the track. Workarounds are to set \catcode`\^^I=9 (ignore tabs completely) or \catcode`\^^I=13 \def^^I{ } (print them as spaces). – siracusa Sep 12 '19 at 14:56
  • Finding the source of the problem is quite interesting, that would explain why -NoValue- (I'm quite intrigued). You can write something else in the line of expl3 instead of \catcode. – Pablo González L Sep 12 '19 at 15:32
6

How horizontal tab characters are treated in LaTeX:

The horizontal tab character has code point number 9 in ASCII encoding and in many other character encodings.

In (La)TeX you can, using ^^-notation, write the horizontal tab character also as ^^I. ("I" is the ninth letter in the Latin alphabet.) Of course this is not possible in situations where the category code régime is changed in a way where ^ is not of category code 7(superscript). This is the case when the arguments of commands like \verb are read from the .tex-input-file and tokenized.

Usually the horizontal tab character has category code 10(space) in (La)TeX.

In (La)TeX any input character of category code 10(space) will be tokenized as an explicit space character token, i.e., as an explicit character token of character code 32 and category code 10(space).

Thus, if (La)TeX under usual category code régime in the .tex-input file encounters a horizontal tab character, it will, in case the reading-apparatus is in state M (middle of line), insert into the token stream an explicit space character token, i.e., an explicit character token of character code 32 and category code 10(space).

When using the \verb-command, the category code régime gets changed before having LaTeX read from the .tex-input-file and tokenize the argument for that \verb-command. Internally this is done by the command \dospecials. That command expands to a list whose elements are of pattern

\do⟨control-symbol-token whose name is formed by the
    character whose category-code is to be changed⟩

The minimal example

\show\dospecials\stop

reveals:

> \dospecials=macro:
->\do \ \do \\\do \{\do \}\do \$\do \&\do \#\do \^\do \_\do \%\do \~.

With the LaTeX 2e kernel's \verb-command and verbatim-environment the command \do in turn is (via \let) made equal to \@makeother when \dospecials is carried out.

Thus you can see that \dospecials, e.g., changes the category code of the space (\do \) but does not change the category code of the horizontal tab character. (Should be something like \do \^^I).

This means:

When the argument of \verb/\verb* gets tokenized (i.e. read from the .tex-input-file for deciding what tokens to insert into the token stream), the category code of the horizontal tab character is not changed and thus still is 10(space). Thus even in verbatim category code régime horizontal tab characters yield explicit space tokens=explicit character tokens of category code 10(space) and character code 32. In horizontal mode space tokens yield horizontal glue.
(This is the greatly shortened story with \verb.)

But why the error-messages in connection with xparse's verbatim-argument-story?

I looked into xparse.dtx (2019-05-28) and searched for the phrase "verb".

I found two paragraphs that seemed interesting to me:

% \begin{macro}{\@@_grab_v_aux_test:N}
% \begin{macro}
%   {
%     \@@_grab_v_aux_loop:N,
%     \@@_grab_v_aux_loop:NN,
%     \@@_grab_v_aux_loop_end:
%   }
%   Check that the opening delimiter is a character, setup category codes,
%   then start reading tokens one by one, keeping the delimiter as an argument.
%   If the verbatim was not nested, we will be grabbing one character
%   at each step. Unfortunately, it can happen that what follows the
%   verbatim argument is already tokenized. Thus, we check at each step
%   that the next token is indeed a \enquote{nice}
%   character, \emph{i.e.}, is not a character with
%   category code $1$ (begin-group), $2$ (end-group)
%   or $6$ (macro parameter), nor the space character,
%   with category code~$10$ and character code~$32$,
%   nor a control sequence.

Here you learn that xparse's mechanism for reading verbatim-arguments examines these arguments and throws an error-message when the examination yields that the argument contains, e.g., an explicit space token.

% \begin{macro}{\@@_grab_v_aux_catcodes:}
% \begin{macro}{\@@_grab_v_aux_abort:n}
%   In a standalone format, the list of special characters is kept
%   as a sequence, \cs{c_@@_special_chars_seq}, and we use
%   \tn{dospecials} in package mode.

Here you learn that xparse's mechanism for reading verbatim-arguments also uses \dospecials for adjusting the category code régime.

You already know that \dospecials leaves the category code of the horizontal tab character untouched, which means that this character still has category code 10(space) which means that its tokenization will (in case the reading-apparatus is in state M (middle of line)) yield an explicit space token, i.e., an explicit character token of category code 10(space) and character code 32.

Summa summarum:

Horizontal tab characters get tokenized as space tokens. During examination by xparse's mechanism for reading arguments verbatim these space tokens yield error messages.

What can be done about this?

Both the mechanism of the LaTeX 2e-kernel for reading arguments verbatim and xparse's mechanism for reading arguments verbatim leave the category code of the horizontal tab character unchanged.

So with both mechanisms there should be no problems with having changed the category code of the horizontal tab character in a way where tokenizing that character does not yield a space token, but, e.g., an explicit active horizontal tab character token.

If you do this, then, at the time of handling that verbatim-argument, the active horizontal tab character token must be defined to deliver whatever tokens you wish, e.g., for visualizing horizontal tabs.


In the example below I tried my very best at making the horizontal tab character yielding the horizontal tab key symbol, i.e., the symbol ⭾ which is represented by the html-entity &#11134; and which is printed on the tab key of many keyboards.

The macro I created for delivering that symbol is \mytabkeysymbol. With this macro I use \scalebox and \resizebox of the graphicx-package in order to get a box whose measurements don't exceed the measurements of the letter W in the current font.

This approach at obtaining the symbol is far from perfect. But it is sufficient for having a macro \mytabkeysymbol and for exhibiting that one's own catcode trickery can be done.

\documentclass{article}

\oddsidemargin=1cm
\addtolength\oddsidemargin{-1in}%
\addtolength\oddsidemargin{-\hoffset}%
\evensidemargin=\oddsidemargin
\textwidth=\paperwidth
\addtolength\textwidth{-2cm}%
\parindent=0ex

\usepackage{xparse}

%..........................................................    
% This is just some code for producing a horizontal-tab-
% key-symbol:
\usepackage{graphicx}
\newbox\myscratchboxa
\newbox\myscratchboxb
\newlength\myscratchlength
\newcommand\mytabkeysymbol{%
  \begingroup\setbox\myscratchboxa=\hbox{W}%
  \hbox to\wd\myscratchboxa{\hfill%
  \resizebox{.8\wd\myscratchboxa}{\ht\myscratchboxa}{%
  \leavevmode
  \hbox{%
    \scalebox{1}[.5]{%
      \setbox\myscratchboxa=\hbox{XM}%
      \setbox\myscratchboxb=\hbox{$\leftarrow$}%
      \ht\myscratchboxb=1.5\ht\myscratchboxb
      \myscratchlength2\ht\myscratchboxa
      \advance\myscratchlength-2\ht\myscratchboxb
      \myscratchlength=.5\myscratchlength
      \advance\myscratchlength\ht\myscratchboxb
      %
      \lower-\myscratchlength
      \hbox{%
        \vrule\box\myscratchboxb
      }%
    }%
  }%
  \llap{%
  \hbox{%
    \scalebox{1}[.5]{%
      \setbox\myscratchboxa=\hbox{XM}%
      \setbox\myscratchboxb=\hbox{$\rightarrow$}%
      \ht\myscratchboxb=1.5\ht\myscratchboxb
      \myscratchlength2\ht\myscratchboxa
      \advance\myscratchlength-2\ht\myscratchboxb
      \myscratchlength=.5\myscratchlength
      %
      \lower-\myscratchlength
      \hbox{%
        \box\myscratchboxb\vrule
      }%
    }%
  }%
  }%
  }\hfill}\endgroup
}%
% End of code for producing a horizontal-tab-key-symbol.
%..........................................................

\makeatletter

\newcommand\MakeHorizontalTabActive{\catcode`\^^I=13\relax}%

\begingroup
\MakeHorizontalTabActive
\@firstofone{%
  \endgroup
  \newcommand\MakeActiveHorizontalTabDeliverTabKeySymbol{%
    \let^^I=\mytabkeysymbol
  }%
  \newcommand\MakeActiveHorizontalTabDeliverInvisiblespace{%
    \let^^I=\@xobeysp
  }%
}%

%..........................................................
% Don't indent this snippet of code!
\begingroup
\MakeHorizontalTabActive
\catcode`\ =12\relax%
\@firstofone{%
\endgroup%
\newcommand\MakeActiveHorizontalTabActLikeSpace{%
\def^^I{%
\ifnum\the\catcode`\ =13\space\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi%
{\@xobeysp}{ }%
}%
}%
\newcommand\MakeActiveHorizontalTabDeliverVisiblespace{\let^^I= }%
}%
%..........................................................

\NewDocumentCommand{\myxverb}{m}{%
  \begingroup
  \MakeHorizontalTabActive
  \myinnerxverb{#1}%
}%
\NewDocumentCommand{\myinnerxverb}{ m v }{%
  \endgroup
  {\verbatim@font\frenchspacing#1#2}%
}%

\makeatother

\pagestyle{empty}

\begin{document}

text \verb*|This is verb* with tbars and two keyboard TABs \t \t in argument.| text

text {%
  \MakeHorizontalTabActive
  \MakeActiveHorizontalTabActLikeSpace
  \verb*|This is verb* with tbars and two keyboard TABs		in argument after changing keyboard TAB's catcode.|%
} text

text {%
  \MakeHorizontalTabActive
  \MakeActiveHorizontalTabDeliverTabKeySymbol
  \verb*|This is verb* with tbars and two keyboard TABs		in argument after changing keyboard TAB's catcode.|%
} text

text {%
  \MakeHorizontalTabActive
  \MakeActiveHorizontalTabActLikeSpace
  \verb|This is verb with tbars and two keyboard TABs		in argument after changing keyboard TAB's catcode.|%
} text

text {%
  \MakeHorizontalTabActive
  \MakeActiveHorizontalTabDeliverTabKeySymbol
  \verb|This is verb with tbars and two keyboard TABs		in argument after changing keyboard TAB's catcode.|%
} text


text
\myxverb{\MakeActiveHorizontalTabDeliverInvisiblespace}|This is myxverb with tbars and two keyboard TABs		in argument.|
text

text
\myxverb{\MakeActiveHorizontalTabDeliverVisiblespace}|This is myxverb with tbars and two keyboard TABs		in argument.|
text

text
\myxverb{\MakeActiveHorizontalTabDeliverTabKeySymbol}|This is myxverb with tbars and two keyboard TABs		in argument.|
text

\end{document}

enter image description here


By the way:

Seems you can prevent "the forum" from converting horizontal tab characters into sequences of spaces as follows:

Don't use `inline code`-notation for inline code and don't indent blocks of code by four spaces but use the html-tags <code>inline code</code> respective

<pre><code>block of code
block of code
block of code</code></pre>

. If you do this, you can/must use html-entities and the like for many characters which have a special meaning when parsing html-code.

E.g., you have to write &gt; for obtaining > and &lt; for obtaining <.

One benefit is: You can write the html-entity &#9; for obtaining a horizontal tab character in your code which does not get converted.

Seems to work on my system at least: When I copy-paste from the following code-block to my favorite text-editor, then I get five consecutive horizontal tab characters between A and B:

A					B

I wrote this code-block as <pre><code>A&#9;&#9;&#9;&#9;&#9;B</code></pre>.

2
  • Dear Ulrich Diez, your answer is more than complete and illustrative, the TAB character is really a nuisance. I always read his answers and explanations, of course, I think it's the first time I've seen him use xparse :) . Perhaps there should be some note about this (fairly common) character in the xparse user documentation. Thank you for everything. – Pablo González L Sep 13 '19 at 1:14
  • 1
    @PabloGonzálezL Thank you. I rarely use the tab-character in verbatim-listings. Tab-characters without catcode-change causing trouble is -eh- rather unusual. ;-) I think it is nice that you can use its active variant as a free pass for doing your own things even in verbatim catcode-régime. To be honest: With xparse I sense a trend of unneeded hiding from the user essential concepts like token, macro and expansion in favor of terms like variable and function which misleads many beginners into believing that transferring knowledge about Pascal/C/C++/Java etc to (La)TeX is trouble-free. – Ulrich Diez Sep 13 '19 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.