Spanish hyphenation and UTF8 characters in plain TeX with pdfcsplain, xetex and luatex

Question

I am trying to figure how to use OPmac for Spanish documents because the easy setting of the structure document in plain TeX. However, using pdfcsplain or xetex/luatex engines, I have no luck with the hyphenation patterns and I not sure of the best way to manage some UTF8 symbols, as I have little experience in plain TeX.

According to their web site, CSplain is designed for Czech and Slovak languages but it "is ready to load hyphenation patterns of 54 languages". although is not clear for me what is next. I guess that should be something like \esUnicode and/or \eslang but all my attempts have failed.

So, the first question is how to add Spanish hyphenation for these both minimal working examples:

For xetex/luatex:

\input opmac
\input ucode
\input cs-schola 
\margins/1 a5 (4,5,1,1)cm  
\typosize[12/14] 
\parindent10pt\parskip1em 
Fantástico, seguro que mañana vendré a tiempo.
\bye

As showed the image, the problem here is the lack of a hyphen in "mañana" (it should be "ma-ñana" or "maña-na), causing a overfull \hbox. The same problem using pdfcsplain and the default font:

\input opmac
\input utf8lat1 
\margins/1 a5 (4,5,1,1)cm  
\typosize[12/14] 
\parindent10pt\parskip1em 
Fantástico, seguro que mañana vendré a tiempo.
\bye

On the other hand, as showed above, any approach support accents (á, é, í, ... ) but some commom UTF8 symbols in Spanish documents (as ñ,Ñ,€) and some others (as →) are not. As far I can see, with xetex or luatex this can be solved with \input ucode and a TeX Gyre font (e.g. \input cs-schola):

\input opmac
\input ucode 
\input cs-schola 
Uno → Dos → Tres.  Me darán 50 € y 24 ¢.
\bye

Whereas in pdfcsplain with \input utf8lat1 I can use ñ instead of \~n but not ohters symbols as €. Although these might be mapped as explained in the CSplain page, that seem a very hard task if a need map too many:

\input opmac
\input utf8lat1 

% To avoid 
%WARNING: unknown UTF-8 code: `→ = ^^e2^^86^^92' 
%WARNING: unknown UTF-8 code: `€ = ^^e2^^82^^ac' 
%WARNING: unknown UTF-8 code: `ñ = ^^c3^^b1' 
% WARNING: unknown UTF-8 code: `¢ = ^^c2^^a2'

 \mubyte\eurochar ^^e2^^82^^ac\endmubyte%
 \def\eurochar{{\eurofont e}}%
 \font\eurofont=feymr10%

\mubyte\flecha ^^e2^^86^^92\endmubyte%
\def\flecha{$\rightarrow$}%

\mubyte\Cent ^^c2^^a2\endmubyte%
\def\Cent{céntimos}%

 Uno → Dos → Tres.  Mañana me darán 50 € y 24 ¢.

\bye

Without mapping the result will be:

Then, I wonder if respect the encoding there are some solution less restrictive (i.e., not limited to TeX Gyre fonts) or less laborious (i.e,, without mapping) to have the widest UTF8 spectrum available.

Answer 1

I don't know how to integrate OPmac with this; but if you follow the instructions in the manual of hyplain, with just a few additions, you can hyphenate Spanish.

1. Make a working directory spanishtest and cd to it

2. In the working directory, copy the file hylang.tex with

   cp $(kpsewhich hylang.tex) .
   

3. Edit the hyrules.tex file to become

   %%% This is hylang.tex (version 1.0), where language definitions
   %%% actually occur. The first one should always be
   %%% American English, for compatibility with plain TeX.
   %%%
   %%% Users can modify this file in order to define the
   %%% languages they need.
   %%%
   %%% Every language definition should be followed by a
   %%% \refinelanguage command where conventions specific to 
   %%% the language are set; users should at least provide 
   %%% the left and right hyphenation minima using
   %%% \hyphenmins{<left>}{<right>}
   %%%
   %%% In the third argument one puts what has to be done
   %%% when activating the language; in the fourth argument
   %%% what needs to be undone.
   
   %%% US English must always come first
   \definebaselanguage{en}{US}{hyphen} %%% <--- don't modify
   \refinelanguage{en}{US}{\hyphenmins{2}{3}}{}
   
   %%% fix the lccode tables
   \input unicode-letters.def
   
   %%% Spanish
   \definelanguage{es}{ES}{loadhyph-es}
   \refinelanguage{es}{ES}{\hyphenmins{2}{2}}{}
   
   %%% Add other languages if needed
   %%%
   %%% The arguments to \definelanguage are:
   %%% #1: the language code; it is an arbitrary string, use the 
   %%%     ISO two-letter language code for uniformity, or `nde' for
   %%%     new orthography German
   %%% #2: the nation code; use the uppercase ISO two-letter code
   %%% #3: the file with hyphenation patterns
   %%%
   %%% The arguments to \refinelanguage and \refinedialect are:
   %%% #1 and #2: a pair defined through \definelanguage or \definedialect
   %%% #3: commands to be executed when entering the language
   %%% #4: commands to be undone when entering a new language
   %%%
   % \definelanguage{xx}{YY}{xxhyph}
   % \refinelanguage{xx}{YY}{<something>}{<something>}
   %
   % \definedialect{aa}{BB}{xx}{XX}
   % \refinedialect{aa}{BB}{<something>}{<something>}
   
   % At last the fallback, a language with no patterns
   \definelanguage{zz}{ZZ}{zerohyph}
   % \refinelanguage{zz}{ZZ}{}{} % no need to set conventions
   
   %%% Aliases
   \addalias\US{en}{US}
   \addalias\SP{es}{ES}
   \addalias\ZZ{zz}{ZZ}
   \addalias\nohyphens{zz}{ZZ}
   
   \endinput
   

4. Dump a format

   xetex -ini -etex hyplain
   

5. End, for the moment

Now, write the file spanishtest.tex

%% we need to use OpenType fonts
\font\tenrm="CMU Serif" at 10pt
\tenrm

\selectlanguage{es}{ES}

\vbox{\hsize=0pt \overfullrule=0pt
\hskip0pt Fantástico, seguro que mañana vendré a tiempo.
}
\bye

Compile with

xetex -fmt hyplain spanishtest

Here's the result:

When you're satisfied, you can put hyplain.fmt in, for instance,

$TEXMFHOME/web2c/xetex/

so it will be available when called from any directory (with xetex -ini -etex, which you can define an alias of in your shell).

---

On the other hand, the default xetex.fmt format already includes support for all the languages defined in language.def, so everything works also without hyplain: just change the test file into

\font\tenrm="CMU Serif" at 10pt
\tenrm

\uselanguage{espanol}

\vbox{\hsize=0pt \overfullrule=0pt
\hskip0pt Fantástico, seguro que mañana vendré a tiempo.
}
\bye

Of course, OpenType fonts are necessary for hyphenation and general typesetting to succeed. But hyplain is more customizable, albeit a bit more difficult. Note that also LuaTeX can be used, so long as you add

\input luaotfload.sty

before defining OpenType fonts.

In any case, OPmac doesn't define OpenType fonts and you have to supply their definitions.

Answer 2

It seems that there is a mix of many problems. For example the hyphenation was mentioned in the title and in the first paragraph of your post but no more. You are doing something with missing characters and you are varying the ways: tex engine, format, font. So I am not sure where is the core of your question.

First of all: OPmac has nothing to do with your problem. You need to have the texengine+format+hyphenation ready to use with Spanish letters in your "plain TeX" and then you can use OPmac. Moreover, the format CSplain isn't needed when using OPmac.

Encoding comment: You are lucky that the letters of your language is encoded in ISO-8859-1 which is subset of T1 TeX encodning and of Unicode too. This means that you can load any T1 encoded or Unicode font and your language will work if the utf8->Unicode conversion is activated. This conversion is activated automatically in xetex or luatex.

There is one funny exception: the Eur symbol isn't present in T1 encoding even though this encoding was designed for European languages. The reason: T1 encoding was designed in 1992, i.e the Eur symbol didn't exist at that time.

Your first attempt using csplain (more precisely with pdftex + activated encTeX). The \input utf8lat1 activates conversion from utf8 only for those characters which are defined in ISO-8859-1. And it isn't right arrow nor Eur nor stroked c.

There is big difference if you apply \input t1code in CSplain document or not. The ñ symbol (for example) is converted to accent plus n if \input t1code isn't used and to Unicode=ISO-8859-1=T1 code of ñ if \input t1code (or \input ucode) is used. The first way (accent plus n) has disadvantage: the hyphenation is impossible and one advantage: it works with classical Knuth's fonts preloaded in plain TeX.

CSplain can run with classical Knuth's TeX or with pdftex or with xetex. It is ready to use in English, Czech and Slovak languages. Other languages can be used too, but csplain must be re-generated with appropriate hyphenation table. This is described in CSplain documentation. I'll do a revision of this documentation and I'll add a example of using CSplain in another language.

Answer 3

You may activate the Spanish patterns for csplain, required by default by OPmac, by removing the comment in the line that activates it in the hyphen.lan file, and then regenerate the csplain and pdfcsplain formats.