Command to display free-text answers from a survey, eliminating ascii control chars

Question

I'd like to create a report from a survey with free-text answers that is collected in a database. For each question there are typically many answers which I could present in an itemize environment, one answer per item. The problem I have is that an individual answer (in Latin1 encoding) might contain double spaces, returns and other control characters that should not matter in the presentation. Is there a suitable environment or command (implemented probably using catcode trickery) that smoothes these out, resulting in a nice even presentation?

\answer{Code with  double spaces, returns, linefeeds and other control character that should not matter here}

An alternative would be to filter all these out using the software that fetches the data from a database and generates the LaTeX code. However, I'd prefer a TeX/LaTeX solution. I'd be also interested in other creative solutions.

Answer 1

I would suggest to simply set all ASCII control characters (0-31) to catcode 9 (ignored) etc.:

\def\mycatcodes{%
% Control characters ASCII #0-31:
% Catcode 9 = "ignore"
\catcode0=9
\catcode1=9
\catcode2=9
\catcode3=9
...
\catcode30=9
\catcode31=9
% and DEL:
\catcode127=9
%
% You might want the line-end character to be a space:
\catcode13=10
% Might be also required:
\endlinechar=-1
%
% Verbatimise all special TeX characters:
\let\do\@makeother
\dospecials
%
% Let Spaces be spaces again:
\catcode`\ =10
}

\newenvironment{answer}{%
    \begingroup
    \mycatcodes
    \@answer
}{}%
% A macro which reads everything until a verbatim "\end{answer}":
\begingroup
\catcode`\|=0
\catcode`\(=1
\catcode`\)=2
\@makeother\{
\@makeother\}
\@makeother\\
|gdef|@answer#1\end{answer}(%
    |endgroup % end catcode changes
    |@@answer(#1)%
    |end(answer)% real \end{answer}
)
|endgroup
% Now a third macro is used which doesn't has the hassle with catcode changes
% like \@answer has!:
\def\@@answer#1{%
    % Does the typesetting and formatting of the answer, e.g.
    \texttt{#1}%
    % Note, that not all verbatim character (like `_`) are correctly displayed
    % by all fonts
}

In the document:

\begin{answer}
     Test with possible control characters (ignored) and LaTeX special characters 
     incl. { and }.
\end{answer}

As you already said, a filter script (e.g. Perl) might be also a good possibility.

Edit: Turned it into a environment

Answer 2

Here is how this could be done in ConTeXt.

\def\startignore
  {\dostartbuffer[ignore][startignore][stopignore]}

\def\stopignore
  {\pushcatcodetable
   \setcatcodetable\nilcatcodes
   \getbuffer[ignore]
   \popcatcodetable}

\setupbodyfont[cambria] 
%Any unicode font that contains greek used in the example will do

\starttext

\startignore
  Line
  feeds
  and multiple   spaces are ignored. The text may contain \undefined \TeX
  commands or ` or ' or _ or ^ or ωθατενερ
\stopignore

\stoptext

In principle, the same idea will work in LaTeX: capture the content of the environment verbatim, set nil catcodes (basically just set the catcode of space, end of line, form feed, tab, and end of file to their usual values; everything else is set to other), and retrieve the contents of the environment.