Full Unicode support for a document with a large number of languages

Question

I use LaTeX (in detail I use pdflatex from MikTeX) for automatically generating documents based on data gathered from the Internet. Of course this means that those pages can contain small parts of other languages such as Chinese, Korean, Arabic, Turkish, ...

Therefore I don't know what languages are used and where they are used within my document.

Until now I have replace all characters from foreign languages by a dot to avoid problems, but in some situations e.g. company names then end up as ..... instead of some Chinese characters.

Is it possible to build one LaTeX document that allows all available languages or at least a large set of languages?

If it is possible how do I know what Unicode characters are supported in this document and which not so I can filter them before generating the .tex sources?

Answer 1

Here is an example of a document that automatically detects several different scripts.

It’s not always possible to detect which language is using a given script, for example, whether you’re processing Arabic versus Persian, or Spanish versus French, Unfortunately, there are a few languages that write the same Unicode codepoints differently, such as Japanese Kanji and traditional Chinese, and you could not display those correctly without tagging.

The simplest solution is to select a font that supports a large number of scripts, such as FreeSerif or DejaVu Sans. No OpenType font can support all of Unicode, but you probably only care about languages that are still spoken today.

Answer 2

With xelatex and polyglossia one can make multilanguage pdf's. For asia languages one need ucharclasses too.

% !TEX TS-program = xelatex
% !TEX encoding = UTF-8 Unicode
\documentclass[12pt, a4paper]{article}

\usepackage{polyglossia}

\setmainlanguage[variant=british]{english}
\setotherlanguages{hebrew, greek, japanese}
\newfontfamily\hebrewfont{SBL Hebrew}
\newfontfamily\greekfont{SBL Greek}
\newfontfamily{\cjkfont}{WenQuanYi Zen Hei}
\usepackage[CJK]{ucharclasses}
\setDefaultTransitions{\defaultfont}{}
\setTransitionsForCJK{\cjkfont}{}

\title{Title}

\begin{document}

\section{First}
\textgreek{αταραξία}. That was in Greek using SBL Greek.

\texthebrew{קֹהֶלֶת}. That was in Hebrew using SBL Hebrew.

\textjapanese{東南西北} That was in CJK using WenQuanYi.

\end{document}