1

Writing a converter from ASCII-TeX to Unicode accents is not an easy task. Most converters I have seen fail terribly in not following macros, nested macros, or bear on unmaintainability due to the large set of search/replace patterns one needs to cover infinitely many ways to write something as simple as \'\i in TeX.

Biber seems to do a terrific job at it. It converts accented strings in a Bibliography entry and in the examples I have seen, without a hitch.

Would it be possible to harness this Biber routine to make a full converter for generic TeX files?

6
  • the file tuenc.def in the base latex directory has the data you need but what exactly do you want to convert? converting something like \verb|\"{a}| makes \"{a} is tricky if you want to just recognise the second of those cases to change to ä Commented Apr 23, 2018 at 17:39
  • TeX is not a final output format, most of the time PDF is the target, so the idea is to convert and have the same output... so the \verb|\"{a}| should stay as-is in source so to produce \"{a} in the final output format. This case in particular is simple because the nesting has only one level, but with TeX you can make infinitely harder ones to parse.
    – Paulo Ney
    Commented Apr 23, 2018 at 17:49
  • 1
    yes that is what I mean: so given any package can define a verb-like command that requires that kind of special handling, so an external convertor is tricky the tuenc.sty has the definitions that you need if you want to convert strings within tex, but hard to output a complete document in that case. Commented Apr 23, 2018 at 17:52
  • note that if i put the above string in a title field biber does the naive translation and breaks it as expected, producing \field{title}{\verb|ä| makes ä} Commented Apr 23, 2018 at 18:03
  • That is correct, Biber does not work in these instances and it would be desirable to. It is not just TUGboat that has articles with titles like that.
    – Paulo Ney
    Commented Apr 23, 2018 at 18:30

1 Answer 1

3

That aspect of biber is just doing a simple string replacement, it is not expecting complicated local definitions, if you use \verb or tabbing (which has local definitions of commands such as \=) or local definitions then you see that it is not taking any account of the tex structure.

The following document has a strange citation but shows tex constructs being passed to biber

\begin{filecontents}{\jobname.bib}
@misc {zzz,
author="Zzzz",
title={\verb|\"{a}| makes \"{a}},
journal={{\renewcommand\"{boo} \" a}},
publisher={
\begin{tabbing}
  a \= b
\end{tabbing}
}
}
\end{filecontents}
\documentclass{article}
\usepackage{biblatex}
\addbibresource{\jobname.bib}
\begin{document}

\begin{tabbing}
  a \= b
\end{tabbing}

\cite{zzz}
\printbibliography

\end{document}

the relevant fields generated in the .bbl file are

  \list{publisher}{1}{%
    {\begin{tabbing} a b̄ \end{tabbing}}%
  \field{journaltitle}{{\renewcommand\"{boo} ä}}
  \field{title}{\verb|ä| makes ä}

all of which would fail to give the correct result in latex, the \= command to set the tab in tabbing has been lost. the \" command is locally redefined as intended but its use has been replaced by ä and the \verb will not show the use of \"

Because of local definitions it would be very hard for any system to completely reliably change all instances of traditional ascii markup that corresponds to a unicode character by that character without breaking any "unxepected" uses of that string. The definitions in tuenc.def give unicode definitions for the constructs but that is expanded while expanding and executing the tex document and so the context is known and you are not trying to preserve the original tex structure.

For a personal document if you wanted to do this, a simple string replace using sed or perl or python etc would convert the document and perhaps require some manual fixing of broken edge cases, but a tool to reliably transform an existing collection of documents without breaking any of them would be much harder.

2
  • Indeed Biber mostly uses RegEx and has no notion of more complicated macro definitions or grouping.
    – moewe
    Commented Apr 23, 2018 at 20:00
  • @moewe shocking that publisher names set with tabbing are not part of the biblatex test suite:-) Commented Apr 23, 2018 at 20:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .