1

I am trying to find out more about how XeTeX implements line breaking for different languages using \XeTeXlinebreaklocale. From the XeTeX reference guide:

\XeTeXlinebreaklocaleLocale ID⟩: Defines how to break lines for multilingual text.

That's all the information I could find about it, leaving me with several questions:

  1. What are acceptable values for ⟨Locale ID⟩? In this question the OP arrives at the conclusion that ISO 639-1 language codes are used, but where can I find out which one of these are actually implemented and lead to changes in XeTeX's behavior?
  2. Where in my TeX Live system would I find a file containing the line breaking information specific to each locale? It seems like it must be baked in into XeTeX the engine directly as it works even if I only load what xetex.ini loads, which as far as I can see does not include or \input any file containing such information.
  3. Is there any way to find documentation on how exactly the line breaking algorithm changes when a \XeTeXlinebreaklocale command is issued?

Now let me add that I do know that certain XeLaTeX packages handle multilingual line breaking for me, but this is not what I am looking for; I am interested in getting to know what XeTeX does exactly.

3

well in xetex.web you end up here

procedure do_locale_linebreaks(s: integer; len: integer);
var
  offs, prevOffs, i: integer;
  use_penalty, use_skip: boolean;
begin
  if (XeTeX_linebreak_locale = 0) or (len = 1) then begin
    link(tail):=new_native_word_node(main_f, len);
    tail:=link(tail);
    for i:=0 to len - 1 do
      set_native_char(tail, i, native_text[s + i]);
    set_native_metrics(tail, XeTeX_use_glyph_metrics);
  end else begin
    use_skip:=XeTeX_linebreak_skip <> zero_glue;
    use_penalty:=XeTeX_linebreak_penalty <> 0 or not use_skip;
    linebreak_start(main_f, XeTeX_linebreak_locale, native_text + s, len);
    offs:=0;
    repeat
      prevOffs:=offs;
      offs:=linebreak_next;
      if offs > 0 then begin
        if prevOffs <> 0 then begin
          if use_penalty then
            tail_append(new_penalty(XeTeX_linebreak_penalty));
          if use_skip then
            tail_append(new_param_glue(XeTeX_linebreak_skip_code));
        end;
        link(tail):=new_native_word_node(main_f, offs - prevOffs);
        tail:=link(tail);
        for i:=prevOffs to offs - 1 do
          set_native_char(tail, i - prevOffs, native_text[s + i]);
        set_native_metrics(tail, XeTeX_use_glyph_metrics);
      end;
    until offs < 0;
  end
end;

so the locale ends up being passed to

linebreak_start(main_f, XeTeX_linebreak_locale, native_text + s, len);

As Marcel points out in comments below this function is defined in xetex_ext.c where it is called linebreakstart (general C-web name mapping rules)

linebreakstart(int f, integer localeStrNum, uint16_t* text, integer textLength)
...

Which in turn basically calls the ICU library ubrk_open function which implements the locale-specific linebreaking from UAX #14 with the locale data available in the ICU tree. Actual changes occur for all locales for which the file there contains a line:process line like zh

see also

http://userguide.icu-project.org/boundaryanalysis

7
  • 1
    You are right, linebreak_start is defined (as linebreakstart) in XeTeX_ext.c and mostly just calls ICU's ubrk_open. It implements UAX #14 with the locale data available in the ICU tree Actual changes occur for all locales for which the file there contains a line:process line like zh – Marcel Krüger Mar 18 '20 at 15:04
  • ah thanks, I lost the _ name mangling in going between web and C:-) – David Carlisle Mar 18 '20 at 15:10
  • when is (XeTeX_linebreak_locale = 0) or (len = 1) actually true? – wave Mar 18 '20 at 16:39
  • @wave er I think the first just if you have not used \XeTeXlinebreaklocale and the second if it's a one letter word, but I'd have to try to read (and understand:-) more of the web sources than I can do at the present time. – David Carlisle Mar 18 '20 at 16:42
  • @DavidCarlisle interestingly, some Japanese text is not broken into lines at all, creating very overfull hboxes, when I don't specify a \XeTeXlinebreaklocale (only after a newline, probably breaking at the space token inserted), but it does look normal if I set \XeTeXlinebreaklocale 0 (or to any value really). – wave Mar 18 '20 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.