How do I identify the encoding and family of a new math font?

Question

In this question, sfranky asked to put a double accent over \bm{\phi}, produced by the bm package. Double accents are implemented in both the amsmath and accents packages, but those packages don't work together well with bm. What I found out myself: One can use \use@mathgroup to make it work.

Now in my answer to sfranky's question I proposed to use

\use@mathgroup{\M@OMS}{5}{\bm{\phi}}

to get a bold \phi that the accents package can handle. However, half of that was just educated guesses. OMS is the encoding, I think (and maybe it should be OML in this case), and 5 is the font family of the \bm{phi}. Altough it seems to work, surely this is not the Right Way of doing it.

Thus my question: Is there a way to automatically identify the encoding and family of the symbol I'm going to use?

Answer 1

It is possible to extract the family of a symbol by looking either at its \mathcode (this is for explicit characters like 1, a, etc.; for example, \the\mathcode`1 gives 28721 which is "7031 in hex) or by looking at it's \meaning which begins with \mathchar (e.g. \meaning\phi gives \mathchar"11E). The second of the four hex digits is the math family (if there are less, the leading digits are zeros). For example, 1 has mathcode "7031 so it's from family 0 (operators), and \phi has mathcode "011E so it's from family 1 (letters).

To get the family from the symbol, you can thus proceed this way: you look at the \meaning of the symbol. The symbols which normally get accents (1, \phi, etc.) typically have a \meaning which is either \mathchar".... or the character 1 or the letter a. For the first type, you must remove the \mathchar from the \meaning and keep only the number. For the second type, you must check if the \meaning begins with the, look at the \mathcode and then convert it to hex (for example by using the binhex.tex package). In each case, you might need to pad the number with leading zeros to extract the second digit reliably.

To be able to say whether the meaning of the character begins with \mathchar or with the, there are various ways. Here, I used the same way as amsmath.sty does for its automatic \dots. As \meaning produces catcode 12 characters, you must produce a \, an m, a t, etc. with catcode 12 to test against. A way of doing this is to use an uppercase/lowercase trick (the principle is the same as in my answer to the question How to make a real backslash (escape) character?): you take catcode 12 characters (e.g. !, ? etc.) and make their uppercase versions be \, m, etc. and then uppercase your whole definition.

Here's a code showing how to do all this. I've commented it, so I hope it's understandable.

\documentclass{article}

% the code works with all font packages
%\usepackage{lmodern}
%\usepackage{fourier}
%\usepackage[utopia]{mathdesign}
%\usepackage[charter]{mathdesign}
%\usepackage[garamond]{mathdesign}
%\usepackage{txfonts}
%\usepackage{pxfonts}
%\usepackage{mathpazo}

\usepackage{amsmath}
\usepackage{bm,accents}
\input{binhex}% for hex conversion

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%% CODE TO DETECT MATH FAMILY %%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
\makeatletter
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Utility macros to pad zeros (e.g. transform 34 into 0034)
% The delimiter is \empty so that if it is expanded, it does nothing
\def\@pad@four@zeros#1{\@pad@four@zeros@aux#1\empty\empty\empty\empty}%
\def\@pad@four@zeros@aux#1#2#3#4\empty{%
  \ifx#1\empty 0\fi\ifx#2\empty 0\fi\ifx#3\empty 0\fi\ifx\\#4\\0\fi#1#2#3#4%
}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Macro extracting the second digit (e.g. 7163 -> 1)
\def\@extract@second@of@four#1#2#3#4\@nil{#2}%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% new boolean which will be used to test if a \meaning begins with \mathcha (e.g. \mahchar"11E)
\newif\if@begins@by@mathchar@
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% uppercase trick to obtain characters with catcode 12
\begingroup
\uccode`!=`\\ % ! is \ with catcode 12
\uccode`?=`m  % ? is m with catcode 12
\uccode`,=`a  % , is a with catcode 12
\uccode`.=`t  % . is t with catcode 12
\uccode`;=`h  % ; is h with catcode 12
\uccode`/=`c  % / is c with catcode 12
\uccode`|=`r  % | is r with catcode 12
\uppercase{\endgroup
\def\@if@begins@by@mathchar@#1#2#3#4#5#6#7#8#9\@nil{%
  \@begins@by@mathchar@false % initiate boolean to false
  % test if #1#2#3#4#5#6#7#8 is !?,.;/;, i.e. \mathcha
  \ifx !#1\ifx ?#2\ifx ,#3\ifx .#4\ifx ;#5\ifx /#6\ifx ;#7\ifx ,#8%
    \@begins@by@mathchar@true % switch boolean to true
  \fi\fi\fi\fi\fi\fi\fi\fi
  \if@begins@by@mathchar@ % if boolean true
    \expandafter\@firstoftwo
  \else % if boolean false
    \expandafter\@secondoftwo
  \fi
}
% macro which strips prefix \mathchar
\gdef\@strip@mathchar@prefix!?,.;/;,|#1\@nil{#1}%
% end of uppercase trick
}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% new boolean which will be used to test if a \meaning begins with the (e.g. the character 1)
\newif\if@begins@by@the@
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% uppercase trick to obtain characters with catcode 12
\begingroup
\uccode`!=`t % ! is t with catcode 12
\uccode`?=`h % ? is h with catcode 12
\uccode`,=`e % , is e with catcode 12
\uppercase{\endgroup
\def\@if@begins@by@the@#1#2#3#4\@nil{%
  \@begins@by@the@false % initiate boolean to false
  % test if #1#2#3 is !?, i.e. the
  \ifx !#1\ifx ?#2\ifx ,#3%
    \@begins@by@the@true % switch boolean to true
  \fi\fi\fi
  \if@begins@by@the@ % if boolean true
    \expandafter\@firstoftwo
  \else % if boolean false
    \expandafter\@secondoftwo
  \fi
}
}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% macro which finds the font family of a math glyph
% does not work for characters modified by amsmath (because of the 
% \DOTSB before them) but as they are not normally accented, it 
% shouldn't be a problem
\def\find@font@family#1{%
% #1 = glyph from which to extract font family
  \edef\@the@mathcode@{-1}% dummy definition (-1 will give a 0 family)
% test if #1 begins with \mathchar
  \expandafter\@if@begins@by@mathchar@\meaning#1\@nil
%   if so, store the value of \mathchar inside \@the@mathcode@
    {\edef\@the@mathcode@{%
         \number\expandafter\@strip@mathchar@prefix\meaning#1\@nil}}%
%   otherwise, do nothing
    {}%
% test if #1 begins with the
  \expandafter\@if@begins@by@the@\meaning#1\@nil
%   if so, store the mathcode inside \@the@mathcode@
    {\edef\@the@mathcode@{\the\mathcode`#1}}%
%   otherwise, do nothing
    {}%
% convert the mathcode into hex with package binhex.tex
  \edef\@the@mathcode@hex{\hex{\@the@mathcode@}}%
% convert the hex number to a 4-digit one
  \edef\@the@mathcode@hex{\expandafter\@pad@four@zeros
                          \expandafter{\@the@mathcode@hex}}%
% store the second digit (which is the math family) in \@the@math@family
  \edef\@the@math@family{%
     \expandafter\@extract@second@of@four\@the@mathcode@hex\@nil}%
}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%% NEW \accbm MACRO %%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
\newcommand{\accbm}[1]{%
% store the font family inside \@the@math@family
  \find@font@family{#1}%
% assign counter \@tempcnta to \@the@math@family so that \bm@boldtable
% gives the right corresponding bold family offset
  \@tempcnta=\@the@math@family
% select the right \fam and apply it to \bm
% \@the@math@family+\bm@boldtable is the number of the bold family
  \use@mathgroup{}% not currently used by LaTeX so can be empty
                {\number\numexpr\@the@math@family+\bm@boldtable}%
                {\bm{#1}}%
}
%
\makeatother
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%% END OF CODE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\newcommand{\test}[1]{$\hat{\dot{\bm{#1}}} \hat{\dot{\accbm{#1}}} \bm{{\hat{\dot{#1}}}}$\par}

\test{\phi}
\test{A}
\test{1}
\test{\Gamma}
\test{+}

% should give 3 but gives 0 if amsmath is loaded
\makeatletter$\find@font@family{\sum}\@the@math@family$\makeatother

\end{document}

In the tests, I compare \hat{\dot{\bm{...}}}, \hat{\dot{\accbm{...}}}, and \bm{{\hat{\dot{...}}}} (the last puts the accents in bold, but works out of the box). Here's the result, from left to right, when the font is mathdesign/utopia:

---

A few explanations of the code.

1. About the encoding. The encoding information (e.g. \M@OMS, \M@OML, etc.) is not really used by LaTeX. The \meaning of these macros is normally \default@M . or \default@M \noaccents@ . where \default@M is empty and \noaccents@ is used by the math accent mechanism of amsmath to change between families 0 and 7 to avoid strange results when the accents are used inside a math alphabet (try $\mathcal{\hat{A}}$ with and without amsmath to see the effect).

Once you have the \fam number, you can extract the encoding in by looking at \the\textfont2 (which is typically \OMS/cmsy/m/n/10) and extracting the tokens between \ and the first /, which can be done thanks to a delimited macro:

\makeatletter
\def\@extract@encoding@aux#1#2/#3\@nil{#2}
\newcommand{\extractencoding}[1]{%
  \expandafter\@extract@encoding@aux\detokenize\expandafter{\the\textfont#1}/\@nil
}
\makeatother
\extractencoding{0} \extractencoding{1} \extractencoding{2} \extractencoding{3} \extractencoding{4}

If the font family is not currently used, the encoding extracted in this way will be "nullfont".

2. About the boldtable macro. The \bm@boldtable macro holds the offsets between the families and their bold versions, stored in an \ifcase structure in the following way:

\ifcase \@tempcnta 9\relax \or 9\relax \or 9\relax \or 9\relax \or \z@ \or \z@ \or \z@ \or \z@ \or \z@ \else \z@ \fi .

This means that the offset is 9 for \fam0 to \fam3 and is zero for all the others. To access these numbers, you must have the current \fam number stored inside the count register \@tempcnta, which you do with

\@tempcnta=\@the@math@family

and then you get the corresponding bold \fam number with

\number\numexpr\@the@math@family+\bm@boldtable

Answer 2

I'm not aware of any pre-existing mechanism for identifying the encoding and family of a symbol.

The encguide package may offer some illumination. It uses a command such as the following to print the the set of encodings in each family:

\ftable{cmr10}{OT1}

In particular, it prints each character belonging to a family and encoding as follows:

\font\X=cmr10
\X{\char86} % "V"

A list of families and encodings are at the bottom of encguide.tex (starting around line 1298).

That being said, I'm not sure if you can extract the family and encoding from a given symbol name or character. I tried using the ifthen package to make a comparison (i.e. \X\ifthenelse{\equal{V}{\char86}}{Same}{Different}), but the comparison doesn't work as expected. If a comparison were possible, one could compare the symbol against every character of every family.

Rather than building this reverse map, it could be a sensible alternative for sfranky (and others with similar issues) to use unicode via X_ƎT_EX or LuaTeX.

While not a solution, I hope this is helpful food for thought.

Answer 3

This is not really an answer to my question, but an explanation why it's the fault of the bm package that it doesn't identify the encoding and family. To give an example, \mathcal is defined in fontmath.ltx via

\DeclareSymbolFontAlphabet{\mathcal}{symbols}

In effect, this makes \mathcal expand to

\use@mathgroup\M@OMS\symsymbols

when used in math mode, i.e., both encoding (\M@OMS) and family (\symsymbols) are provided. This enables the amsmath and accents packages to place multiple accents correctly. Thus, the correct fix to the problem in question should be that eventually, \bm expands to the correct \use@mathgroup<encoding><family>.