# Syntax of TeX/LaTeX

{} and [] do some kind of grouping, and comma seems to be special as well. Being a programmer I find it frustrating to use Latex on incantation-only basis. I would like to know how does it exactly work.

What is exactly the syntax of TeX/LaTeX?
How is it parsed?
Does Latex introduces new commands only or changes the syntax as well?
Or maybe changing syntax locally is a inherent feature of Tex?

-
furthermore pstricks, tikz and other similar packages introduce their own syntax which can be intermixed with TeX/LaTeX. –  Dima Jul 27 '10 at 13:05
You already got a great answer from Andrew, adding to that, really the only way to parse TeX is by using the tex program itself. Indeed the all three actions parse/evaluate/output are intermixed in TeX and the results of any of these stages can (and often does) influence the behavior of the others. –  Juan A. Navarro Jul 27 '10 at 13:20
spelling s/seams/seems/ –  Yossi Farjoun Jul 27 '10 at 17:32

For a programmer, I would recommend reading the book "TeX by Topic" (available for free at http://www.eijkhout.net/tbt/). I think that that will give you the best answer to your general questions (which really are, in my opinion, a bit too general for this site; I would advise you to read TbT and then ask more focussed questions on particular aspects).

However, you mention a couple of specifics so let me try to answer them. {} are lexical:

\def\hello{world}
{\def\hello{hello}
\hello}
\hello


produces: "hello world".

The grouping given by square brackets is an illusion. Some commands in LaTeX start with "If the next character is [, gobble the rest up to ] and use it as the first argument.". This can cause problems if, for example, what you want to pass has a square bracket in it:

\newcommand{\hello}[2][hello]{#1 #2}
\hello{world}
\hello[greetings]{world}
\hello[Greetings [1] to the]{world}


Produces

hello world
greetings world
Greetings [1 worldo the]world


because the first argument to "hello" in the last line is "Greetings [1".

(This can be remedied by using braces to do proper grouping, \hello[{Greetings [1] to the}]{world} would work fine).

I've never heard of commas being special. Can you give an example?

-
As I say in my answer, you can deal with the lack of grouping for optional arguments with a bit of effort. We've done that for LaTeX3 in xparse, and I'd hope people will make use of this ability! –  Joseph Wright Jul 27 '10 at 15:40
You can put several citations, separated by commas in a \cite{} command. –  Yossi Farjoun Jul 27 '10 at 17:23
I'd quibble that that makes the comma special. It's the separator chosen by the person who coded that command, but that doesn't make the comma special in the say that, say, ^ and _ are special. –  Loop Space Jul 27 '10 at 17:45
I guess the point is that commas (and the equals sign) are special in the sense they are used a lot in dividing up input. Comma-separated lists are used all over the place in LaTeX, and key-value input is also popular. However, this works only when they are not special (i.e. commas need to have catcode 'other'). –  Joseph Wright Jul 27 '10 at 20:30
So commas can be made special if someone declares them to be special by using them in a special way. Fair enough, but I feel that's a different quality of being special than of those that TeX declares to be special at the outset. And if we're going down the catcode route then technically no character is special, rather there are special catcodes and certain characters have these to begin with. –  Loop Space Jul 27 '10 at 21:10

I would recommend reading The TeXbook, by Donald Knuth, at least for two reasons:

1. It is a big pleasure to read. Knuth's writing style is very clear and explanatory. Not to mention he is the author of TeX.
2. It has a excellent chapter which is almost literally the answer to your question: Chapter 7. How TeX Reads What You Type.''

I consider this book to be the best writing about the TeX low-level features.

-
+1 for The TeXbook. It is truly fantastic. Knuth starts off telling little white lies (he's very up front about this) at the beginning of the book, enabling one to read and learn easily. By the end, he's covered basically everything there is to know about TeX. Any exercise not marked with a "dangerous bend" sign should really be attempted. And those with the dangerous bend should be read, thought about, and if necessary, looked up in Appendix A. –  TH. Sep 13 '10 at 8:40
I upvoted this answer, but Knuth's writing style is not for everybody. Personally I like it, but I have met people that are put off TeX by his writing style completely. Also, you cannot really use The TeXbook for a quick lookup until you have read the whole book at least a few times (TbT is much better for that). –  Taco Hoekwater Sep 13 '10 at 8:46

The excellent TeX by Topic has already been mentioned, and I'd start there if you are a programmer looking for enlightenment. What you need to bear in mind is that TeX works with tokens with category codes. In the case of grouping, usually { and } are tokens with category codes 1 (begin group) and 2 (end group), respectively. So TeX will start a group when it reads { and finish it when it reads }.

For a simple macro such as

\newcommand*\example[1]{Stuff with #1}


the argument will be either a single token or a single group. Thus

\example{stuff}


will absorb 'stuff' as #1, but

\example stuff


will only pick up the 's' of 'stuff', as 's' is a single token.

LaTeX introduces the idea of optional arguments inside [ ... ]. As pointed out by Andrew, this isn't done by grouping, and so can give odd results. You can get around this: see how xparse manages optional arguments for example. LaTeX also uses ( ... ) for co-ordinate arguments, but again with no grouping.

As I said, category codes are important. ConTeXt uses [ and ] for grouping rather than { and }. So it's not the character you need to worry about, but the token (including the category code). This is a rather TeX-specific concept, so looking at TeX by Topic and The TeXbook is a good way to get to grips with it.

-
Second mention of catcodes (as far as I can tell)! Yippee! –  Loop Space Jul 27 '10 at 17:47
If you program TeX, you have to understand catcodes: 'tis how it is. –  Joseph Wright Jul 27 '10 at 20:08

Since TeX by Topic and the TeXbook have already been mentioned as particularly relevant to programmers, I think it would be egregious not to mention TeX: The Program. This is a beautiful—probably the crowning—example of Knuth's philosophy of literate programming; I am at absolute best an amateur programmer, and I found it very easy to dive in and read the parts I wanted (I, too, am deeply concerned with the parsing, and not so much right now with the layout) and leave out the rest.

As a warning, as Juan Navarro mentioned, any visions of getting a clean view of parsing, separate from the actual ‘compilation’ per se, will soon be dashed. What one would hope to think of from a modern point of view as distinct parts of the typesetting process are hopelessly intermingled; this is at least part of why people say that TeX is a terrible programming language. Chris Rowley argues for a modern approach to separating what should be separate (and people like Taco Hoekwater have done some beautiful things that should count as steps in that direction, even though I'm not sure that was their intention).

-