The BibTeX author mentions:
Here's how BibTeX works. It takes as input
.aux file produced by LaTeX on an earlier run;
.bst file (the style file), which specifies the general reference-list style and specifies how to format individual entries, and which is written by a style designer (called a wizard throughout this program) in a special-purpose language described in the BibTeX documentation - see the file
.bib file(s) constituting a database of all reference-list entries the user might ever hope to use.
A "special-purpose language", as described in Designing BibTeX Styles, is a form of "postfix stack language":
Basically the style file is a program, written in an unnamed language, that
tells BibTeX how to format the entries that will go in the reference list [..].
This programming language has ten commands [..]. These commands manipulate the language's objects: constants, variables, functions, the stack, and the entry list.
(Warning: The terminology [..] chosen for ease of explanation, is slightly different from BibTeX's. For example [..] "variables" and "functions" are both "functions" to BibTeX. [..])
There are two types of functions: built-in ones that BibTeX provides [..] and ones you define using either the
Your most time-consuming task, as a style designer, will be creating or modifying functions using the
FUNCTION command [..].
Another interesting, related read would be Taming the BeaST, which attempts to provide the "B to X of BibTeX" - as complete a reference as possible.
bibtex itself is written in Web (similar to TeX). While we know it as an executable file, the original
bibtex.web source is tangled into PASCAL code, and then compiled into a binary executable.