# Non-ascii character as argument

I want to do something special with some characters in the text. In this example that is just boldfacing them:

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

\newcommand\<[1]{\textbf{#1}}

\begin{document}
f\<oo b\<ar b\<öll
\end{document}


This works fine for the two first cases, but not for the non-ascci character "ö". There is an error message

! Package inputenc Error: Unicode char \u8:\check@icr not set up for use with LaTeX.


which is reported between the two bytes making up "ö" in utf8.

1. I know this works with XeLaTeX or LuaLaTeX (just by removing the inputenc line). It does not work in pdfLaTeX and (DVI)LaTeX.

2. One workaround is to write \<{ö}.

3. But is there a way to get this to work with pdflatex without that workaround?

(In the actual application an active character is used, since the point is to have something that disturbs the view of the source text as little as possible.)

-
I think the short answer here is 'no': something like ö is two bytes and thus two args for pdfTeX, while a 'normal' letter such as o is only one byte. Thus you can't grab an undelimited argument 'safely' while allowing for UTF-8 with an 8-bit engine. –  Joseph Wright May 12 '14 at 9:18
I feared that, but hoped that it would be possible to let the command catch just the first byte and then do something more. –  pst May 12 '14 at 9:20
You can imagine partial solutions, e.g. grab one token, test catcode, if active grab second token and re-insert. The problem is that you leave open edge cases where it will fail: no truly general solution exists, I think. –  Joseph Wright May 12 '14 at 9:22
@pst obviously it is possible to do that but you would have to do it for every command, that is you could define \< such that if its argument was an active character with definition that used the utf8 two byte handler macro, that it called a helper macro that grabbed the utf8 sequence braced it and then called the original \< but how many commands would you need to redefine? –  David Carlisle May 12 '14 at 9:23

You can do this, but I'm not sure you should:-)

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

%\newcommand\<[1]{\textbf{#1}}
\makeatletter
\def\<{\expandafter\zz}
\def\zz#1{%
\ifx\UTFviii@two@octets#1% could be 3 or 4 octets, but not today
\expandafter\zztwo
\else
\expandafter\zzone{#1}%
\fi}
\def\zztwo#1#2{\zzone{\UTFviii@two@octets#1#2}}
\makeatother
\def\zzone#1{\textbf{#1}}

\begin{document}

f\<oo b\<ar b\<öll

f\<{o}o b\<ar b\<{ö}ll

\end{document}

-
IMHO, this and egreg's solutions are only patches at bad level of processing. One problem is solved but somebody would write \futurelet\next\macro č and \macro expands to \ifx\next ě... and the problem is here again. –  wipet Jul 31 '14 at 15:14

I am sorry about my non-LaTeX answer but I would signify that we can treat with UTF-8 codes as one token i normal 8bit pdftex and we can avoid the complications shown in David's and egreg's answers. You can try to create the UTF-8 encoded file:

\input lmfonts

\def\<#1{{\bf#1}}
f\<oo b\<öll €\<€ f\<{oö} f\< öo
\bye


and process it by pdftex -fmt csplain test.tex.

Explanation: The format csplain is generated with encTeX extension of pdfTeX, which is able to interpret UTF-8 codes at input prosessor level and returns them as single tokens (byte or control sequence) to the token processor. It is able to return back to the log and \write files the original UTF-8 codes.

-
I were informed by some LaTeX users that they are using encTeX in LaTeX too: they added -enc option to their fmtutil.cnf file for latex and pdflatex formats and they re-generated these formats. They wrote \input utf8-t1 before \documentclass and they used [T1]{fontenc} and they don't use {inputenc} (which is in conflict). –  wipet Jul 30 '14 at 18:47
This is essentially the same as just saving the file in latin1 before processing it, but getting tex to do it on the fly on input. It suffers the same drawback (which you do not mention in your answer) that it doesn't work for arbitrary utf8 input, just the range covered by T1 so more or less latin1. (obviously the enc file could cover a bigger range, mapping to control codes, but it doesn't) –  David Carlisle Jul 31 '14 at 15:33
@DavidCarlisle No, this isn't the same as using one-byte preprocessed encoding. The number of supported UTF-8 codes are not limited by 256. Most of them can be treated as control sequences which returns back to UTF-8 codes during \write primitive. For example after \input cyrchars there is support not only of east europe characters but cyrillic too. This is more than 256 characters. –  wipet Jul 31 '14 at 16:57
For example the Euro character in the example above is internally treated as \euro control sequence, no as the byte-code. This control sequence can be set as macro which finds the extra characters font (appropriate to the current one) selects it, prints the character and returns back to the base font. When we write \write\file{€} (i.e.internally \write\file{\euro} the normal UTF-8 code of the Euro character is printed. –  wipet Jul 31 '14 at 17:18

A general answer, covering also the cases of three and four byte UTF-8 characters; \<ö or \<{ö} is allowed. If a space creeps in, like in the last example, it is removed.

Maybe a test for a control sequence should be added, in order to catch wrong input; as long as you have just characters or { after \<, it should be safe.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{textcomp}
\usepackage{lmodern}

\usepackage{xparse,l3regex}

\ExplSyntaxOn
\NewDocumentCommand{\<}{}
{
\pst_boldify:
}

\tl_new:N \l__pst_first_byte_tl
\cs_new_protected:Npn \pst_boldify:
{
\peek_catcode_ignore_spaces:NTF \c_group_begin_token
{
\textbf
}
{
\pst_boldify_aux:N
}
}

\cs_new_protected:Npn \pst_boldify_aux:N #1
{
\tl_set:Nx \l__pst_first_byte_tl
{
\int_compare:nT { #1 < 128 } { 0 }
\int_to_bin:n { #1 }
}
\regex_replace_once:nnN { 0[01]*\Z } { } \l__pst_first_byte_tl
\str_case:on { \l__pst_first_byte_tl }
{
{ }      { \textbf { #1 } }
{ 11 }   { \pst_do_bold:nn { #1 } }
{ 111 }  { \pst_do_bold:nnn { #1 } }
{ 1111 } { \pst_do_bold:nnnn { #1 } }
}
}
\cs_new_protected:Npn \pst_do_bold:nn #1 #2
{
\textbf{#1#2}
}
\cs_new_protected:Npn \pst_do_bold:nnn #1 #2 #3
{
\textbf{#1#2#3}
}
\cs_new_protected:Npn \pst_do_bold:nnnn #1 #2 #3 #4
{
\textbf{#1#2#3#4}
}

\ExplSyntaxOff

\begin{document}
f\<oo b\<öll €\<€ f\<{oö} f\< öo
\end{document}


The idea is that if the next token (after removing spaces) is a brace, just \textbf is executed. Otherwise, the next token is examined and converted to a character code in binary form; everything is removed from the first zero included so as to determine if the UTF-8 character we have to manage is one, two, three or four bytes. Finally, the appropriate decision is made.

Without a regular expression substitution, an arithmetic test can be performed; the definition of \pst_boldify_aux:N should then be

\cs_new_protected:Npn \pst_boldify_aux:N #1
{
\int_compare:nTF { #1<128 }
{
\textbf
}
{
\int_compare:nTF { 192 <= #1 < 224 }
{
\pst_do_bold:nn { #1 }
}
{
\int_compare:nTF { 224 <= #1 < 240 }
{
\pst_do_bold:nnn { #1 }
}
{
\pst_do_bold:nnnn { #1 }
}
}
}
}


with the rest as is (except that loading l3regex is not needed any more).

-
I mean that LaTeX isn't so intelligent to automatically add the backslash in the last but one line (the last usage of the \<) in your first exmaple. But maybe I don't understand this... –  wipet Jul 30 '14 at 16:59
@wipet Just a typo, and you know it. –  egreg Jul 30 '14 at 17:00
I've mechanically copied your line to my plain TeX example and I've expected exactly the same result as in your picture. But I was surprised :) –  wipet Jul 30 '14 at 18:23
Thanks! I couldn't use it right away -- evidently \int_to_bin:n was added in February this year, and I'm still using Texlive 2013 (shame on me!). –  pst Jul 31 '14 at 10:53
@pst Yes, it's in one of the last releases, the older function was called \int_to_binary:n`, IIRC. –  egreg Jul 31 '14 at 19:48