Keyboard TAB character in argument v (xparse)

Question

I'm trying to understand how the TAB character of the keyboard behaves within verbatim. I have defined a new command using the v argument of the xparse, but, I get errors and a rather strange output so to speak. The package documentation says nothing about this specific character (If...I know it shouldn't be used, but,...).

This is my example file, with real tab character (thanks to @Ulrich for the tip):

\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand{\myxverb}{v}{#1}
\pagestyle{empty}
\begin{document}
%This is \verb*|verb with tbars and two keyboard TABs \t \t in argument| text, OK.\par
This is \verb*|verb with tbars and two keyboard TABs		in argument| text, OK.\par

%This is \myxverb|myxverb with tbars and two keyboard TABs \t \t in argument| text, NOT OK.\par
This is \myxverb|myxverb with tbars and two keyboard TABs		in argument| text, NOT OK.\par

%This is \myxverb{myxverb with brace and two keyboard TABs \t \t in argument} text, NOT OK.
This is \myxverb{myxverb with brace and two keyboard TABs		in argument} text, NOT OK.

\end{document}

An image of the output: Part of the .log file generated using pdflatex in TeXLive 2019 (update):

! LaTeX3 Error: The verbatim command '\myxverb' cannot be used inside an
(LaTeX3)        argument.

For immediate help type H <return>.
 ...                                              

l.10 ...|myxverb with tbars and two keyboard TABs   
                                                    in argument| text, NOT OK...

? 

! LaTeX3 Error: The verbatim command '\myxverb' cannot be used inside an
(LaTeX3)        argument.

For immediate help type H <return>.
 ...                                              

l.13 ...{myxverb with brace and two keyboard TABs   
                                                    in argument} text, NOT OK.
? 
! Too many }'s.
l.13 ... brace and two keyboard TABs        in argument}
                                                   text, NOT OK.
?

Any way to skip this problem?

Greetings

Answer 1

How horizontal tab characters are treated in TeX (and thus also in LaTeX):

The horizontal tab character has code point number 9 in ASCII encoding and in many other character encodings.

In TeX you can, using ^^-notation, write the horizontal tab character also as ^^I. ("I" is the ninth letter in the Latin alphabet.) Of course this is not possible in situations where the category code régime is changed in a way where ^ is not of category code 7(superscript). This is the case when the arguments of commands like \verb are read from the .tex-input-file and tokenized.

Usually the horizontal tab character has category code 10(space) in TeX.

In TeX any input character of category code 10(space) will be tokenized as an explicit space token, i.e., as an explicit character token of character code 32 and category 10(space).

Thus, if TeX under usual category code régime in the .tex-input file encounters a horizontal tab character, it will, in case the reading-apparatus is in state M (middle of line), insert into the token stream an explicit space token, i.e., an explicit character token of character code 32 and category 10(space).

When in LaTeX using the \verb-command, the category code régime gets changed before having LaTeX read from the .tex-input-file and tokenize the argument for that \verb-command. Internally this is done by the command \dospecials. That command expands to a list whose elements are of pattern

\do⟨control-symbol-token whose name is formed by the
    character whose category-code is to be changed⟩

The minimal example

\show\dospecials\stop

reveals:

> \dospecials=macro:
->\do \ \do \\\do \{\do \}\do \$\do \&\do \#\do \^\do \_\do \%\do \~.

With the LaTeX 2e kernel's \verb-command and verbatim-environment the command \do in turn is (via \let) made equal to \@makeother when \dospecials is carried out.

Thus you can see that \dospecials, e.g., changes the category code of the space (\do \ ) but does not change the category code of the horizontal tab character. (Should be something like \do \^^I).

This means:

When the argument of \verb/\verb* gets tokenized (i.e. read from the .tex-input-file for deciding what tokens to insert into the token stream), the category code of the horizontal tab character is not changed and thus still is 10(space). Thus even in verbatim category code régime horizontal tab characters yield explicit space tokens=explicit character tokens of category 10(space) and character code 32. In horizontal mode space tokens yield horizontal glue.
(This is the greatly shortened story with \verb.)

But why the error-messages in connection with xparse's verbatim-argument-story?

I looked into xparse.dtx (2019-05-28) and searched for the phrase "verb".

I found two paragraphs that seemed interesting to me:

% \begin{macro}{\@@_grab_v_aux_test:N}
% \begin{macro}
%   {
%     \@@_grab_v_aux_loop:N,
%     \@@_grab_v_aux_loop:NN,
%     \@@_grab_v_aux_loop_end:
%   }
%   Check that the opening delimiter is a character, setup category codes,
%   then start reading tokens one by one, keeping the delimiter as an argument.
%   If the verbatim was not nested, we will be grabbing one character
%   at each step. Unfortunately, it can happen that what follows the
%   verbatim argument is already tokenized. Thus, we check at each step
%   that the next token is indeed a \enquote{nice}
%   character, \emph{i.e.}, is not a character with
%   category code $1$ (begin-group), $2$ (end-group)
%   or $6$ (macro parameter), nor the space character,
%   with category code~$10$ and character code~$32$,
%   nor a control sequence.

Here you learn that xparse's mechanism for reading verbatim-arguments examines these arguments and throws an error-message when the examination yields that the argument contains, e.g., an explicit space token.

% \begin{macro}{\@@_grab_v_aux_catcodes:}
% \begin{macro}{\@@_grab_v_aux_abort:n}
%   In a standalone format, the list of special characters is kept
%   as a sequence, \cs{c_@@_special_chars_seq}, and we use
%   \tn{dospecials} in package mode.

Here you learn that xparse's mechanism for reading verbatim-arguments also uses \dospecials for adjusting the category code régime.

You already know that \dospecials leaves the category code of the horizontal tab character untouched, which means that this character still has category code 10(space) which means that its tokenization will (in case the reading-apparatus is in state M (middle of line)) yield an explicit space token, i.e., an explicit character token of category 10(space) and character code 32.

Summa summarum:

Horizontal tab characters get tokenized as space tokens. During examination by xparse's mechanism for reading arguments verbatim these space tokens yield error messages.

What can be done about this?

Both the mechanism of the LaTeX 2e-kernel for reading arguments verbatim and xparse's mechanism for reading arguments verbatim leave the category code of the horizontal tab character unchanged.

So with both mechanisms there should be no problems with having changed the category code of the horizontal tab character in a way where tokenizing that character does not yield a space token, but, e.g., an explicit active horizontal tab character token.

If you do this, then, at the time of handling that verbatim-argument, the active horizontal tab character token must be defined to deliver whatever tokens you wish, e.g., for visualizing horizontal tabs.

---

In the example below I tried my very best at making the horizontal tab character yielding the horizontal tab key symbol, i.e., the symbol ⭾ which is represented by the html-entity ⭾ and which is printed on the tab key of many keyboards.

The macro I created for delivering that symbol is \mytabkeysymbol. With this macro I use \scalebox and \resizebox of the graphicx-package in order to get a box whose measurements don't exceed the measurements of the letter W in the current font.

This approach at obtaining the symbol is far from perfect. But it is sufficient for having a macro \mytabkeysymbol and for exhibiting that one's own catcode trickery can be done.

\documentclass{article}

\oddsidemargin=1cm
\addtolength\oddsidemargin{-1in}%
\addtolength\oddsidemargin{-\hoffset}%
\evensidemargin=\oddsidemargin
\textwidth=\paperwidth
\addtolength\textwidth{-2cm}%
\parindent=0ex

\usepackage{xparse}

%..........................................................    
% This is just some code for producing a horizontal-tab-
% key-symbol:
\usepackage{graphicx}
\newbox\myscratchboxa
\newbox\myscratchboxb
\newlength\myscratchlength
\newcommand\mytabkeysymbol{%
  \begingroup\setbox\myscratchboxa=\hbox{W}%
  \hbox to\wd\myscratchboxa{\hfill%
  \resizebox{.8\wd\myscratchboxa}{\ht\myscratchboxa}{%
  \leavevmode
  \hbox{%
    \scalebox{1}[.5]{%
      \setbox\myscratchboxa=\hbox{XM}%
      \setbox\myscratchboxb=\hbox{$\leftarrow$}%
      \ht\myscratchboxb=1.5\ht\myscratchboxb
      \myscratchlength2\ht\myscratchboxa
      \advance\myscratchlength-2\ht\myscratchboxb
      \myscratchlength=.5\myscratchlength
      \advance\myscratchlength\ht\myscratchboxb
      %
      \lower-\myscratchlength
      \hbox{%
        \vrule\box\myscratchboxb
      }%
    }%
  }%
  \llap{%
  \hbox{%
    \scalebox{1}[.5]{%
      \setbox\myscratchboxa=\hbox{XM}%
      \setbox\myscratchboxb=\hbox{$\rightarrow$}%
      \ht\myscratchboxb=1.5\ht\myscratchboxb
      \myscratchlength2\ht\myscratchboxa
      \advance\myscratchlength-2\ht\myscratchboxb
      \myscratchlength=.5\myscratchlength
      %
      \lower-\myscratchlength
      \hbox{%
        \box\myscratchboxb\vrule
      }%
    }%
  }%
  }%
  }\hfill}\endgroup
}%
% End of code for producing a horizontal-tab-key-symbol.
%..........................................................

\makeatletter

\newcommand\MakeHorizontalTabActive{\catcode`\^^I=13\relax}%

\begingroup
\MakeHorizontalTabActive
\@firstofone{%
  \endgroup
  \newcommand\MakeActiveHorizontalTabDeliverTabKeySymbol{%
    \let^^I=\mytabkeysymbol
  }%
  \newcommand\MakeActiveHorizontalTabDeliverInvisiblespace{%
    \let^^I=\@xobeysp
  }%
}%

%..........................................................
% Don't indent this snippet of code!
\begingroup
\MakeHorizontalTabActive
\catcode`\ =12\relax%
\@firstofone{%
\endgroup%
\newcommand\MakeActiveHorizontalTabActLikeSpace{%
\def^^I{%
\ifnum\the\catcode`\ =13\space\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi%
{\@xobeysp}{ }%
}%
}%
\newcommand\MakeActiveHorizontalTabDeliverVisiblespace{\let^^I= }%
}%
%..........................................................

\NewDocumentCommand{\myxverb}{m}{%
  \begingroup
  \MakeHorizontalTabActive
  \myinnerxverb{#1}%
}%
\NewDocumentCommand{\myinnerxverb}{ m v }{%
  \endgroup
  {\verbatim@font\frenchspacing#1#2}%
}%

\makeatother

\pagestyle{empty}

\begin{document}

text \verb*|This is verb* with tbars and two keyboard TABs \t \t in argument.| text

text {%
  \MakeHorizontalTabActive
  \MakeActiveHorizontalTabActLikeSpace
  \verb*|This is verb* with tbars and two keyboard TABs		in argument after changing keyboard TAB's catcode.|%
} text

text {%
  \MakeHorizontalTabActive
  \MakeActiveHorizontalTabDeliverTabKeySymbol
  \verb*|This is verb* with tbars and two keyboard TABs		in argument after changing keyboard TAB's catcode.|%
} text

text {%
  \MakeHorizontalTabActive
  \MakeActiveHorizontalTabActLikeSpace
  \verb|This is verb with tbars and two keyboard TABs		in argument after changing keyboard TAB's catcode.|%
} text

text {%
  \MakeHorizontalTabActive
  \MakeActiveHorizontalTabDeliverTabKeySymbol
  \verb|This is verb with tbars and two keyboard TABs		in argument after changing keyboard TAB's catcode.|%
} text


text
\myxverb{\MakeActiveHorizontalTabDeliverInvisiblespace}|This is myxverb with tbars and two keyboard TABs		in argument.|
text

text
\myxverb{\MakeActiveHorizontalTabDeliverVisiblespace}|This is myxverb with tbars and two keyboard TABs		in argument.|
text

text
\myxverb{\MakeActiveHorizontalTabDeliverTabKeySymbol}|This is myxverb with tbars and two keyboard TABs		in argument.|
text

\end{document}

---

By the way:

Seems you can prevent "the forum"/the StackExchage-platform from converting horizontal tab characters into sequences of spaces as follows:

Don't use `inline code`-notation for inline code and don't indent blocks of code by four spaces but use the html-tags <code>inline code</code> respective

<pre><code>block of code
block of code
block of code</code></pre>

. If you do this, you can/must use html-entities and the like for many characters which have a special meaning when parsing html-code.

E.g., you have to write > for obtaining > and < for obtaining <.

One benefit is: You can write the html-entity 	 for obtaining a horizontal tab character in your code which does not get converted.

Seems to work on my system at least: When I copy-paste from the following code-block to my favorite text-editor, then I get five consecutive horizontal tab characters between A and B:

A					B

I wrote this code-block as <pre><code>A&#9;&#9;&#9;&#9;&#9;B</code></pre>.

Answer 2

Side note, if you want to patch the patch xparse to handle TAB character, you can patch \__xparse_grab_v_aux_catcodes: macro.

%! TEX program = lualatex
\documentclass{article}
\usepackage{etoolbox}  % ← need this package
\begin{document}

\NewDocumentCommand{\aaa}{+v}{#1}

\ExplSyntaxOn
% ======== begin main code ========
\patchcmd {\__xparse_grab_v_aux_catcodes:} {
    \cs_set_eq:NN \do \char_set_catcode_other:N
    \dospecials
} {
    \cs_set_eq:NN \do \char_set_catcode_other:N
    \do \^^I
    \dospecials
} {} {}
% ======== end main code ========

% example use
\aaa+123456x%^^ 
    1234    
123+
\ExplSyntaxOff

\end{document}

Obviously, this is not guaranteed to work in future expl3/xparse versions.