10

I need help writing a parser for decimal numbers using only TeX and plain TeX. The set of accepted inputs is given by the regular expression:

(+-)?(0-9)*(.,)?(0-9)+

The parser need not do anything other than typeset the sign, the integer part and the fractional part.

EDIT:

It looks like this question needs some clarification so here it is. I am trying to design and implement a macro (similar to \num) that will read a number passed as an argument and format it in its own way. So far I've broken down this task to the following steps:

  1. check if the argument is a valid number, i.e. if it agrees with the regular expression above (this is the difficult part for me to do in TeX). I would specifically like to know if we can detect spaces in a macro argument, e.g. in "12 345", "12 .3", "12. 34", "- 1", "+ 1", and how to detect other non-digits in a macro argument such as primitive control sequences and letters (\num does it);

  2. "extract" the sign, the integer part and the decimal part, each in a separate macro (I recon this is easy to do once I know how to implement step 1);

  3. format and typeset the number given the macros with the sign, the integer and the fractional parts (already implemented).

So, basically I need help with step 1.

EDIT 2:

In the regular expression above, ? means zero or one occurrence of a character from the preceding set, * means zero or more occurrences, + -- one or more occurrences.

  • 1
    what form do you want the parse output? You say just typeset but \relax+99.123 would typeset +99.123 but presumably that isn't an answer? – David Carlisle Nov 11 '14 at 20:23
  • @DavidCarlisle The output format does not matter as long as the parser can "extract" the sign, the integer part and the fractional part from accepted inputs and signal an error when passed an unaccepted input. – SJU Nov 11 '14 at 20:27
  • 6
    You might want to read this paper on TUGboat – egreg Nov 11 '14 at 21:00
  • 2
    Does the parser need to be expandable? Are we allowed to make assumptions about catcodes or does the scanner need to check or ignore these? – Joseph Wright Nov 11 '14 at 21:01
  • You can easily check if there's a certain token in a macro, so it's easy to check for +, -, ., ,, and switch the appropiate flags and then execute a generic macro. – Manuel Nov 11 '14 at 22:24
10

EDITED to seamlessly handle dots and/or commas as the decimal separator.

REEDITED to provide error checking. The error handler works by pre-processing the string and removing "numerically valid" tokens from it (any numeral, leading + or -, and a single . or ,). If the processed string is not a null string at the end of this error-checking process, then it implies the number is formatted incorrectly and sets an error flag.

If no error is found, then the normal parsing is invoked to extract, successively, a leading + or -, the pre-decimal portion, and the post-decimal portion from the original argument.

\def\parsenum#1{%
\leavevmode\llap{#1} %REMOVE THIS ECHO LINE FOR GENERAL USE
  \errcheck{#1}%
  \if T\badchars \def\errcode{1}\def\psign{}\def\pint{}\def\pdec{}\else%
    \def\errcode{0}%
    \expandafter\parsedots#1,\relax%
    \expandafter\parsesign\pdots\relax\relax\relax%
  \fi%
}
\def\parsedots#1,#2\relax{\ifx\relax#2\relax\gdef\pdots{#1}\else%
  \parsedotshelper#1.#2\fi}
\def\parsedotshelper#1,{\gdef\pdots{#1}}
\def\parsesign#1#2\relax{%
  \if+#1\gdef\psign{#1}\parseint#2.\relax\relax\else%
    \if-#1\gdef\psign{#1}\parseint#2.\relax\relax\else%
      \gdef\psign{+}\parseint#1#2.\relax\relax%
    \fi%
  \fi%
}
\def\parseint#1.#2\relax{\gdef\pint{#1}\parsedec#2.\relax}
\def\parsedec#1.#2\relax{\gdef\pdec{#1}}

\def\errcheck#1{%
  \def\dwindling{}%
  \rmplus#1\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmminus\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmone\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmtwo\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmthree\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmfour\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmfive\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmsix\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmseven\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmeight\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmnine\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmzero\tmp\relax\relax%
  \edef\tmp{\dwindling}\def\dwindling{}%
  \expandafter\rmdot\tmp\relax\relax%
  \if F\dotfind%
    \edef\tmp{\dwindling}\def\dwindling{}%
    \expandafter\rmcomma\tmp\relax\relax%
  \fi
  \if\relax\dwindling\relax\gdef\badchars{F}\else\gdef\badchars{T}\fi%
}
\def\rmplus#1#2\relax{\if+#1\edef\dwindling{\dwindling#2}\else%
  \edef\dwindling{\dwindling#1#2}\fi%
}
\def\rmminus#1#2\relax{\if-#1\edef\dwindling{\dwindling#2}\else%
  \edef\dwindling{\dwindling#1#2}\fi%
}
\def\rmone#1#2\relax{\if1#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmone#2\relax\fi
}
\def\rmtwo#1#2\relax{\if2#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmtwo#2\relax\fi
}
\def\rmthree#1#2\relax{\if3#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmthree#2\relax\fi
}
\def\rmfour#1#2\relax{\if4#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmfour#2\relax\fi
}
\def\rmfive#1#2\relax{\if5#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmfive#2\relax\fi
}
\def\rmsix#1#2\relax{\if6#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmsix#2\relax\fi
}
\def\rmseven#1#2\relax{\if7#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmseven#2\relax\fi
}
\def\rmeight#1#2\relax{\if8#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmeight#2\relax\fi
}
\def\rmnine#1#2\relax{\if9#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmnine#2\relax\fi
}
\def\rmzero#1#2\relax{\if0#1\else\edef\dwindling{\dwindling#1}\fi%
  \ifx\relax#2\relax\else\expandafter\rmzero#2\relax\fi
}
\def\rmdot#1#2\relax{\if.#1\edef\dwindling{\dwindling#2}\def\dotfind{T}\else%
  \edef\dwindling{\dwindling#1}%
  \ifx\relax#2\relax\def\dotfind{F}\else\expandafter\rmdot#2\relax\fi\fi
}
\def\rmcomma#1#2\relax{\if,#1\edef\dwindling{\dwindling#2}\else%
  \edef\dwindling{\dwindling#1}%
  \ifx\relax#2\relax\else\expandafter\rmcomma#2\relax\fi\fi
}

\parsenum{123} [\psign][\pint][\pdec]:\errcode\par
\parsenum{+123.34} [\psign][\pint][\pdec]:\errcode\par
\parsenum{-123.} [\psign][\pint][\pdec]:\errcode\par
\def\x{-123}\parsenum{\x} [\psign][\pint][\pdec]:\errcode\par
\parsenum{+123,34} [\psign][\pint][\pdec]:\errcode\par
\parsenum{-123,} [\psign][\pint][\pdec]:\errcode\par
\def\x{-123,45}\parsenum{\x} [\psign][\pint][\pdec]:\errcode\par
\parsenum{-123,A} [\psign][\pint][\pdec]:\errcode\par
\parsenum{X-123,} [\psign][\pint][\pdec]:\errcode\par
\parsenum{12+3,} [\psign][\pint][\pdec]:\errcode\par
\parsenum{-12-3,} [\psign][\pint][\pdec]:\errcode\par
\parsenum{-123,2,5} [\psign][\pint][\pdec]:\errcode\par
\parsenum{-123,2.5} [\psign][\pint][\pdec]:\errcode\par
\bye

enter image description here

  • Doesn't detect control sequences in the macro argument such as in \parsenum{-12\relax3,2}. (I've edited the question to make it clearer) – SJU Nov 17 '14 at 18:25
  • @AngelTsankov That is correct. A "number" being passed to \parsenum is assumed to have no control sequences. How would one attempt to parse them anyhow? Throw them away? Retain them? What if the control sequence fell between the - and the 123? Does it belong to the minus or the number? I think it is reasonable to require numbers to contain no macros. – Steven B. Segletes Nov 17 '14 at 18:28
  • @AngelTsankov Or do you mean for a number with macros inside of it to be flagged with an error? – Steven B. Segletes Nov 17 '14 at 18:30
  • I was just wondering how the \num macro detects control sequences (such as \relax) that should not be part of a number and issues an error. See the edits to my question for more info – SJU Nov 17 '14 at 18:33
9

this version is not expandable but it doesn't need any terminator, it just gobbles up as far as the end of the number. It is restricted to integers that fit in a tex count register.

enter image description here

\newcount\ca
\newcount\cb
\def\p#1{%
\def\digitone{}%
\ifx+#1\def\sign{#1}\else
\ifx-#1\def\sign{#1}\else
\def\sign{}\def\digitone{#1}\fi\fi
\afterassignment\n\ca=1\digitone}

\def\n#1{%
\def\digitone{}%
\ifx,#1\def\point{#1}\else
\ifx.#1\def\point{.}\else
\def\point{}\def\digitone{#1}\fi\fi
\afterassignment\nn\cb=1\digitone}

\def\gobble#1{}

\def\nn{\par
[sign=\sign]
[n1=\expandafter\gobble\the\ca]
[point=\point]
[n2=\expandafter\gobble\the\cb]
\par}


\p 123 hello \p456,99 hafa \p ggg \p-12,88

\bye
  • By "latex count register" you surely mean "tex count register"? – morbusg Nov 12 '14 at 19:34
  • @morbusg well oddly enough they have the same limits on integer range:-) (fixed thanks) – David Carlisle Nov 12 '14 at 20:49
  • This is clever but it doesn't detect the control word in \p 12\relax34, nor does it properly handle the space in \p 12 345. For details, see the edits to the question – SJU Nov 17 '14 at 18:20
4

The code inside siunitx (v2) uses a loop over each token in the input before comparing this to the various lists of possible 'number parts'. As that process cannot make assumptions about the nature of the tokens and as it covers a variety of input forms, it's rather complex. I'm currently re-writing it all to be more efficient and clear but the need to work with a range of inputs means its still non-trivial.

With the more restricted input form given here a number of concepts can be hard-coded to produce a much faster parser. For example, the following uses a simple token-by-token loop over the input with the knowledge that the first token can be a + or -, other tokens can be 01234567890 and that exactly one . or , is allowed to break between the integer and decimal parts.

\catcode`\@=11 %
\def\num#1{%
  \begingroup
    \edef\num@temp{#1}%
    \def\num@int@part{}%
    \def\num@dec@part{}%
    \ifx\num@temp\empty
    \else
      \expandafter\num@first\num@temp\num@stop 
    \fi
    \ifx\num@int@part\empty
      \ifx\num@dec@part\empty
        \ERROR
      \fi
    \fi
    $
      \num@sign@part
      \ifx\num@int@part\empty
        0%
      \else
        \num@int@part
      \fi
      \ifx\num@dec@part\empty
      \else
        .\num@dec@part
      \fi  
    $%
  \endgroup
}
\def\num@first#1#2\num@stop{%
  \ifnum
    \ifx#1+ 1\fi
    \ifx#1- 1\fi
    0>0 %
    \def\num@sign@part{#1}%
    \expandafter\num@first@auxi
  \else
    \def\num@sign@part{}%
    \expandafter\num@first@auxii
  \fi
    {#1}{#2}%
}
\def\num@first@auxi#1#2{\num@int#2\num@stop}
\def\num@first@auxii#1#2{\num@int#1#2\num@stop}
\def\num@int#1{%
  \ifx\num@stop#1%
  \else
    \ifnum
      \ifx#1. 1\fi
      \ifx#1, 1\fi
      0>0 %
      \expandafter\expandafter\expandafter\num@dec
    \else
      \num@digit@check#1\num@int@part
      \expandafter\expandafter\expandafter\num@int
    \fi
  \fi
}
\def\num@dec#1{%
  \ifx\num@stop#1%
  \else
    \num@digit@check#1\num@dec@part
    \expandafter\num@dec
  \fi
}

\def\num@digit@check#1#2{%
  \begingroup
    \def\num@digit@check@aux##1#1{}%
    \toks0\expandafter{\num@digit@check@aux0123456789{}{}#1}%
    \edef\num@digit@check@aux{\the\toks0}%
  \expandafter\endgroup
  \ifx\num@digit@check@aux\empty
    \def#2{}%
    \ERROR
  \else
    \edef#2{#2#1}%
  \fi
}
\def\num@stop{\num@stop}
\long\def\gobble#1{}
\num{1234.56789010}
\bye

I've used a simple error trap: an undefined control sequence. A real version would of course need to be a bit more verbose here.

Spaces in arguments are awkward. In siunitx I simply ignore them, which is easy as a normal token-based mapping will do it at the TeX level. If you really want to check for spaces in such a case it is doable but more tedious and I'm not sure worth the effort. (Usually the input should be something 'reasonable'.)

In my revisions for siunitx I'm going with a path a bit more like Steven B. Segletes's answer. For me, that makes sense as I can divide up parts with different restrictions and only need a subset of tests. With the simpler number here such division is probably not much of a saving so I've gone with a single mapping.


An alternative approach to checking the digits is to use TeX, which limits us to the range for count registers but does avoid needing a loop over everything. That might for example read

\catcode`\@=11 %
\def\num#1{%
  \begingroup
    \edef\num@temp{#1}%
    \def\num@int@part{}%
    \def\num@dec@part{}%
    \ifx\num@temp\empty
    \else
      \expandafter\num@first\num@temp\num@stop 
    \fi
    \ifx\num@int@part\empty
      \ifx\num@dec@part\empty
        \ERROR
      \fi
    \fi
    $
      \num@sign@part
      \ifx\num@int@part\empty
        0%
      \else
        \num@int@part
      \fi
      \ifx\num@dec@part\empty
      \else
        .\num@dec@part
      \fi  
    $%
  \endgroup
}
\def\num@first#1#2\num@stop{%
  \ifnum
    \ifx#1+ 1\fi
    \ifx#1- 1\fi
    0>0 %
    \def\num@sign@part{#1}%
    \expandafter\num@first@auxi
  \else
    \def\num@sign@part{}%
    \expandafter\num@first@auxii
  \fi
    {#1}{#2}%
}
\def\num@first@auxi#1#2{\num@int{#2}}
\def\num@first@auxii#1#2{\num@int{#1#2}}
\def\num@int#1{%
  \ifx\relax#1\relax
    \expandafter\gobble
  \else
    \expandafter\num@int@auxi
  \fi
    {#1}%
}
\def\num@int@auxi#1{\num@int@auxii#1\num@stop}
\def\num@int@auxii#1#2\num@stop{%
  \ifnum
    \ifx#1. 1\fi
    \ifx#1, 1\fi
    0>0 %
      \num@dec{#1#2}%
  \else
    \num@digit@check{#1#2}\num@int@part\num@dec
  \fi
}
\def\num@dec#1{%
  \ifx\relax#1\relax
  \else
    \num@dec@auxi#1\num@stop
  \fi
}
\def\num@dec@auxi#1#2\num@stop{%
   \num@dec@auxii{#2}%
}
\def\num@dec@auxii#1{%
  \ifx\relax#1\relax
  \else
    \num@dec@auxiii#1\num@stop
  \fi
}
\def\num@dec@auxiii#1#2\num@stop{%
  \ifnum
    \ifx#1+ 1\fi
    \ifx#1- 1\fi
    0>0 %
    \ERROR
  \else
    \num@digit@check{#1#2}\num@dec@part\num@end
  \fi
}
\def\num@end#1{%
  \ifx\relax#1\relax\else\ERROR\fi
}
\def\num@digit@check#1#2#3{%
  \begingroup
    \def\num@digit@check@aux##1\relax{%
      \edef\num@digit@check@aux{%
        \def\noexpand#2{\the\count0 }%
        \noexpand#3{##1}%
      }%
    }%
    \afterassignment\num@digit@check@aux\count0=#1\relax
  \expandafter\endgroup
  \num@digit@check@aux
}

\def\num@stop{\num@stop}
\long\def\gobble#1{}

\num{1234.56789010}
\bye

Again, I've not forbidden every possible space: doable but tedious and I'd rather not!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.