Can I detect whether LaTeX is in vertical mode?

Question

I am writing a command that creates a margin paragraph. This requires that this command is only used in vertical mode, which is fine. If the command is used in a float, LaTeX gives a "not in outer par mode" error, which is not very friendly. Is there a way to detect whether LaTeX is currently in vertical mode, so that I can write something helpful for the user?

Answer 1

The \ifvmode switch indicates if you are in vertical mode including internal vertical mode. There is also \ifhmode for the horizontal mode and \ifinner for "inner" mode.

See The TeXBook, Chapter 20: Definitions (also called Macros), page 209 for more details.

Note that it is sometimes required to add a \relax before \ifvmode if it is at the beginning of a macro which can be used in a location where tokens are expanded. For example in a tabular/\halign cell tokens are expanded until something unexpandable is found in order to look for a \noalign. In such cases the wrong mode might be still active, e.g. TeX is then in vertical mode for the first cell of a row AFAIK because the cell is not entered yet so TeX is still between rows.

Answer 2

As an alternative to trying to duplicate the tests that LaTeX is doing, you could just locally redefine \@parmoderr and or \@ehb to something more specific to your environment.

\@parmoderr is defined initially as

    \gdef\@parmoderr{%
  \@latex@error{Not in outer par mode}\@ehb}

which makes the "Not in outer par mode" error and then \@ehb is the help text you get if you type h which is initially defined as

\gdef\@ehb{%
  You've lost some text. \space \@ehc}
\gdef\@ehc{%
  Try typing \space <return> %
  \space to proceed.\MessageBreak
  If that doesn't work, type \space X <return> \space to quit.}

It would be a bad idea to globally change these definitions but reasonable to change them within a specific environment.