10

I'm trying to write a (JavaScript) RegEx to turn arbitrary sequences of characters into (hopefully) unique command names.

Part of the solution would be to replace everything that does not match /[a-zA-Z]/ by something else, e.g. the empty string '' or, for digits, '1' by 'one' and so forth.

Experimentally I have found that while strings like '辛亥革命' and even '一二三' (that is, CJK digits) do work as legal names in XeLaTeX, neither '¥ÄÖå' nor '、。' do, presumably because of the respective first characters of these strings not being categorized as letters (catcode 11) by XeLaTeX.

So the question is: where are XeTeX's catcodes defined or documented? If I replace all Unicode codepoints that are not classified as letters—i.e. those not matching /[\p{Letter}]/gu—with something else, will all strings be turned into legal XeTeX names?

Edit: I'm fully aware that the resulting names may not be unique and that distinct filenames like foo-bar.txt and foobar.txt will both turn into foobartxt with a simple-minded elision of offending characters; however, disambiguation is not part of this question. That said, a solution that translates arbitrary sequences into unique names (hence allowing for roundtrip conversions) would also be interesting.

Update: I've come up with the below code; by going through a number of Unicode codepoints that are classed as \p{Letter} or \p{Mark} I found that the IPA Length Marker, ː, has a pleasant appearance and is a legal name character in TeX; it also is not treated as a word separator in my editor. I'm now using the below code to turn filenames into valid TeX names:

@TEX_partitioner      = 'ː'
@TEX_latindigitnames  =
  0:    'zero'
  1:    'one'
  2:    'two'
  3:    'three'
  4:    'four'
  5:    'five'
  6:    'six'
  7:    'seven'
  8:    'eight'
  9:    'nine'

@TEX_text_as_legal_name = ( text ) ->
  ### See https://github.com/latex3/unicode-data; thx to https://tex.stackexchange.com/a/514731/128255 ###
  validate.nonempty_text text
  R = text
  ### replace Latin digits by partitioner plus name: ###
  R = R.replace ///[0-9]///gu,                  ( $0        ) => @TEX_partitioner + @TEX_latindigitnames[ $0 ]
  ### insert partitioner between sequences of lower and upper case (aB -> aːB) ###
  R = R.replace ///(\p{Ll})(\p{Lu})///gu,       ( _, $1, $2 ) => $1 + @TEX_partitioner + $2
  ### replace each code point that is not a Unicode 'Letter' or 'Mark' with partitioner: ###
  R = R.replace ///[^\p{Letter}\p{Mark}]///gu,  @TEX_partitioner
  ### turn the result into lower case: ###
  return R.toLowerCase()

Here are some examples; on the left, the original filenames; on the right, the symbolic names that are usable in TeX documents:

'Roboto-Regular.ttf'                  'robotoːregularːttf'
'Roboto-Thin.ttf'                     'robotoːthinːttf'
'sarasa-gothic-0.6.0'                 'sarasaːgothicːːzeroːːsixːːzero'
'ebas927.ttf'                         'ebasːnineːtwoːsevenːttf'
'Source_Sans_Pro'                     'sourceːsansːpro'
'OFL.txt'                             'oflːtxt'
'SourceHanSerifSC-Heavy.otf'          'sourceːhanːserifːscːheavyːotf'
'TH-Khaai-TP0.ttf'                    'thːkhaaiːtpːzeroːttf'
'UnifrakturMaguntia16.ttf'            'unifrakturːmaguntiaːoneːsixːttf'
'书法家超明体.ttf'                      '书法家超明体ːttf'

Given how difficult it is to come up with a naming convention in TeX that does not lead to hard-to-read spaghetti names with Roman numerals, I'm quite pleased with this.

  • Depending on what you want to do with the final macro names, note that you can create command names from arbitrary character sequences using \csname...\endcsname. The actual macro name can be obtained after exactly one expansion step then. – siracusa Nov 3 '19 at 14:16
  • My experience in TeX has taught me to eschew any and all tricks when it comes to naming things, such as manipulating catcodes (beyond the standard way of temporarily allowing @ in names). My objective is to find names to be used in, e.g., font declarations such as \newfontface{\mktsFontfileːsunːextbːttf}{Sun-ExtB.ttf}[Path=\mktsPathsFontsHome/]. – John Frazer Nov 3 '19 at 15:26
  • 1
    Is that for personal usage only? To me it seems easier to let a user write things like \usefont{sarasa-gothic-0.6.0} instead of a longish macro name which has a word separator character in it that is hard to type on many keyboards. – siracusa Nov 3 '19 at 19:10
10

Category codes in Xe(La)TeX ad Lua(La)TeX are both set by the script load-unicode-data.tex, which picks up the code points from the Unicode Consortium. The script itself sets

  • \catcode 11 for all letters (Unicode class "L")
  • \catcode 11 for all combining marks (Unicode class "M")

Thus 'read UnicodeData.txt' is the best one can say here.

  • Great, that was the kind of pointer I was hoping for and that duckduckgo refused to come up with so far. – John Frazer Nov 3 '19 at 11:32
  • 3
    @JohnFrazer it helps that I wrote it! – Joseph Wright Nov 3 '19 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.