The issue is "obviously" (*) that the optional argument grabber tries to tokenize the newline token, which normally has catcode 5 ("end of line") before \obeylines
is executed which changes its catcode to 13 (active).
Since it's required to tokenize the following token to determine whether or not there is an optional argument, it's unavoidable.
(in theory it's possible to use scantokens
to "untokenize" it. In practice, it has a performance hit and without setting \everyeof
user might hit even more perplexing bugs & maybe confusing \errorcontextlines
as well)
For a workaround, set the catcode of the \endlinechar
before parsing the optional argument. The cleanest way to do that at the moment (that I know of) is to define auxiliary macro/environment.
%! TEX program = lualatex
\documentclass{article}
\NewDocumentEnvironment{aaa}{}{\obeylines ENV:}{}
\NewDocumentEnvironment{innerbbb}{o}{ENV:}{}
\NewDocumentEnvironment{bbb}{}{\obeylines\innerbbb}{\endinnerbbb}
\NewDocumentEnvironment{ccc}{o}{ENV:}{}
\begin{document}
\begin{aaa}
x
y
\end{aaa}
\begin{bbb}
x
y
\end{bbb}
\begin{bbb}[arg]
x
y
\end{bbb}
\begin{ccc}\par
x\par
y\par
\end{ccc}
\begin{ccc}[arg]\par
x\par
y\par
\end{ccc}
\end{document}
Side note, use the plain form \innerbbb
because Defining environments based on other ones: What's the right way?
Side note 2, this will make the error messages a bit confusing for runaway optional argument (i.e. you forgot to put a ]
), reporting the environment name being innerbbb
instead of bbb
! File ended while scanning use of \environment innerbbb .
(*) if you're sufficiently experienced in TeX programming