4

Is there a program that you can use to generate unicode hyphenation pattern? I already know about opatgen but I can not compile that program since it requires a very old version of gcc.

  • if the language you are using has less than 256 characters (or actually less than 256 character classes in hyphenation patters) you could translate the unicode slots into the 0-255 range, use patgen and then translate the generated patterns back. – David Carlisle Oct 15 '13 at 8:39
  • Thanks David. If you just make this as an answer with only one line of code and instructions on how to generate the unicode hyphenation patterns, then I will accept your answer. – user22486 Oct 17 '13 at 4:10
2

I was going to leave it as a comment as I've never tried this but in principle...

If the language you are using has less than 256 characters (or actually less than 256 character classes in hyphenation patters) you could translate the unicode slots into the 0-255 range, use patgen and then translate the generated patterns back

I can think of two methods, if ABCD are your arabic letters and you have a hyphenation dictionary

AB-C  BC-DA

then you can use any unicode-aware editing tool to change ABCD to the latin alphabet abcd (it doesn't matter that the meaning is messed up, or if two letters are mapped to the same latin letter so long as all the letters mapped the same way take part in hyphenation the same way. so suppose you use the perl tr command

tr/ABCD/abca/;

Then your hyphenation dictionary looks like

ab-c bc-aa

Let patgen do its thing and produce a hyphenation pattern

a1b4c
b1c4a1a
.1a1b

or whatever, now to get back you just need to convert abc back to [AD]BC so if a pattern has two a in it then you will end up with four patterns for all combinations of A and D

A1B4C
D1B4C

B1C4A1A
B1C4A1D
B1C4D1A
B1C4D1D

.1A1B
.1D1B

Which is a pattern using the original letters, and hopefully some connection with reality.

While writing that I thought of a second method you could try. Save the unicode patten dictionary as utf-8 and then just give it to patgen anyway. To patgen each multiple byte sequence will be a single letter but there will be no hyphens between an initial byte (leading bit 0) and a following byte (leading bit 1) so when it has generated the patterns you should get patterns of the form (if you look at the result as latin1, or equivalently as a byte stream) of the form

<leading bit 0> <digit> <leading bit 1> <digit> <leading bit 1>

Just delete any <digit> before a character with leading bit 1 and you should reconstitute valid utf-8 sequences (discard any that are not valid).

Or of course a third method would be to stare at the C source of opatgen and see how to remove the obsolete constructs:-)

[I have marked this answer "community wiki" If anyone understands languages enough to test a solution then feel free to edit (or add your own answer and I'll delete this)]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy