# Macro inserted via \everypar in \obeylines context doesn't see some commands

I have the following code which works as expected except, as noted in the code, when a macro begins a line of the xltabular being built:

\documentclass{article}

\usepackage[papersize={5.5in,8.5in},margin=0.6in,bottom=0.7in]{geometry}
\usepackage{array}
\usepackage{multicol}
\usepackage{xparse}
\usepackage{xltabular}
\usepackage{etoolbox}

%% https://tex.stackexchange.com/questions/487572/obeylines-and-gappto-from-etoolbox
%% Collect the body of the xltabular in \tabbody:
\begingroup
\lccode~=\^^M
\lowercase{%
\endgroup
\def\tabline#1~{%
\xappto{\tabbody}{\unexpanded{#1\\\hline}}~
}
}

\makeatletter
\NewDocumentEnvironment{listit}{}{%
\gdef\tabbody{}
\noindent
\begin{minipage}{\textwidth}
\raggedcolumns
\begin{multicols}{2}
\col@number\@ne
\mathchardef\LT@end@pen=0 %
\begingroup
\offinterlineskip
\everypar={\tabline}
\obeylines
}{%
\end{multicols}
\end{minipage}
}
\makeatother

\NewDocumentCommand{\explain}{}{%
\endgroup
\vspace{-\baselineskip}
\begin{xltabular}{\linewidth}{X}
\hline\hline
\tabbody
\hline
\end{xltabular}
}

\begin{document}

\begin{listit}
Lorem ipsum dolor
sit amet, consectetur
sed do eiusmod
tempor incididunt
\bfseries See Note
ut labore
et dolore
magna aliqua.
\explain

Note: \verb+\bfseries+ by itself is ignored. \verb+\textbf{See Note}+ produces \verb+Extra \fi+ error.
\end{listit}

\end{document}


Using \leavevmode\bfseries See Note produces the expected output. Why is this necessary?

What am I missing?

# Diagnosis

The problem is that the lines of your \tabbody are collected by the \tabline macro, which is inserted by \everypar. However, the \everypar tokens are only inserted when a paragraph starts, and \bfseries used in vertical mode doesn't cause TeX to start a paragraph (i.e., to switch to horizontal mode). So, when one of the special lines designed to be read by \tabline starts with \bfseries, TeX is still in vertical mode when it finds the \bfseries (this is right after an end-of-line has been turned into \par). It expands it, and only later when a 〈horizontal command〉1 causes a switch to horizontal mode, it inserts \tabline which will collect the rest of the line. But when this happens, it is too late, the \bfseries has already been fully expanded and digested, it is not grabbed as part of the #1 by the inserted \tabline.

Example: suppose that TeX is collecting these lines:

tempor incididunt
\bfseries See Note


tempor incididunt is the first argument of the \tabline macro that was inserted by means of \everypar when the paragraph was started with the t from tempor incididunt (t is a 〈letter〉 and therefore a 〈horizontal command〉; when TeX finds it in vertical mode, it switches to horizontal mode to start a new paragraph). So, the expansion of this \tabline consumed tempor incididunt• from the input stream, where • represents an active end-of-line character (it is active because \obeylines was used above). After these tokens have been removed from the input stream, the replacement text of \tabline is inserted, with tempor incididunt substituted for #1:

\xappto{\tabbody}{\unexpanded{tempor incididunt\\\hline}}•


Once \xappto has been fully processed, TeX finds the • that was just inserted, which has been made \let-equivalent to \par by \obeylines:

\obeylines:
macro:->\catcode \^^M\active \let ^^M\par


Therefore, when this • token is digested, TeX ends the paragraph and switches to vertical mode. The next token from the input stream is \bfseries. This is a macro, therefore it is expanded. I pass on the details of its expansion (\protect\bfseries where the second \bfseries has a space at the end of its name, etc.). What is important is that this \bfseries doesn't start a new paragraph (its expansion doesn't contain any 〈horizontal command〉). This only happens when TeX digests the S from See Note, i.e., after \bfseries has been fully processed. At this point, TeX switches to horizontal mode because the S is a 〈horizontal command〉; it inserts the indentation box (invisible here because \parindent is 0pt), then the tokens stored in \everypar, in your case a single \tabline, and resumes normal processing of the input stream. The just-inserted \tabline token is expanded, this grabs See Note as the first argument (you see, \bfseries isn't present in this argument, it is already behind us), consumes the active end-of-line character that follows (because it is part of the 〈parameter text〉 of the macro), then the replacement text is inserted with See Note substituted for #1:

\xappto{\tabbody}{\unexpanded{See Note\\\hline}}•


(note that this \xappto call doesn't append the \bfseries to \tabbody, it has definitely been lost for \tabbody) and the process goes on, as we just explained.

# Proposed solution

I propose not to rely on \everypar. Rather, we can make the end-of-line character active in the innermost group of your listit environment definition and redefine the active end-of-line character to be \let-equal to \tabline (see my \listit@obeylines macro). This way, every end-of-line character in the special portion becomes equivalent to a \tabline token and is expanded right before the next line. This way, its #1 grabs everything until the following end-of-line, and thus no command is lost.

This method even allows you to have \par tokens among the text that is collected in \tabbody (see Paragraph break here:\par But... in the example below). Of course, it requires a means to stop the special collection process. Given your example, I decided that a line starting with \explain marks the end of this process (cf. \listit@checknext). Of course, one could use a different end marker if you wish.

\documentclass{article}
\usepackage{multicol}
\usepackage{xparse}
\usepackage{xltabular}
\usepackage{etoolbox}

\makeatletter

%% https://tex.stackexchange.com/questions/487572/obeylines-and-gappto-from-etoolbox
%% Collect the body of the xltabular in \tabbody:
\begingroup
\lccode~=\^^M
\lowercase{%
\endgroup
\long\def\tabline#1~{%
\xappto{\tabbody}{\unexpanded{#1\\\hline}}%
\futurelet\next\listit@checknext
}
\newcommand*{\listit@obeylines}{\catcode~=\active \let~=\tabline}
}

\newcommand*{\listit@checknext}{%
\ifx\next\explain
\let\next=\relax
\else
\let\next=\tabline
\fi
\next
}

\NewDocumentEnvironment{listit}{}{%
\gdef\tabbody{}
\noindent
\begin{minipage}{\textwidth}
\raggedcolumns
\begin{multicols}{2}
\col@number\@ne
\mathchardef\LT@end@pen=0 %
\begingroup
\listit@obeylines
}{%
\end{multicols}
\end{minipage}
}
\makeatother

\NewDocumentCommand{\explain}{}{%
\endgroup
\vspace{-\baselineskip}
\begin{xltabular}{\linewidth}{X}
\hline\hline
\tabbody
\hline\noalign{\vskip 4pt}%
\end{xltabular}
}

\begin{document}

\begin{listit}
Lorem ipsum dolor
sit amet, consectetur
sed do eiusmod
tempor incididunt
\bfseries See Note
ut labore
\textbf{See Note}
magna aliqua.
Paragraph break here:\par But we remain in the same line.''
\explain

Note: \verb+\bfseries+ is not ignored anymore. You can easily see that
\verb+\textbf{See Note}+ works fine too.
\end{listit}

\end{document}


Footnote

1. Such as a 〈letter〉, 〈otherchar〉 or \unhbox coming from the expansion of \leavevmode, among other possibilities (cf. TeXbook p. 283).