There are two ways to proceed. Regardless of which way you choose, make sure you re-run BibTeX using the new BST so that the BBL is properly updated.
1.) The Easy Way (in general)
The simplest way to accomplish what you want is to re-generate the bst using makebst. That is, from the command line, execute:
latex makebst
The resulting script will custom-build a docstrip bst driver for you. You can select all of the formatting options you like, including the sort order (if any) of the entries. Once you are finished, you will get a "dbj" file that you can use to generate your bst. For example, if the result is "mycustom.dbj", then you execute:
latex mycustom.dbj
and you will generate a "mycustom.bst". Moreover, this BST file will be natbib compatible.
2.) The Hard Way (for a general BST)
Alternatively, you can edit the bst file yourself and hack in an explicit sort order based on appearance in the manuscript. This is not difficult to do so long as you are comfortable with a postfix (think "reverse Polish notation") syntax. For example, if I open up the "abbrvnat.bst" file, I see that there is a block...
FUNCTION {presort}
{ calc.label
label sortify
" "
*
type$ "book" =
type$ "inbook" =
or
'author.editor.sort
{ type$ "proceedings" =
'editor.organization.sort
{ type$ "manual" =
'author.organization.sort
'author.sort
if$
}
if$
}
if$
" "
*
year field.or.null sortify
*
" "
*
cite$
*
#1 entry.max$ substring$
'sort.label :=
sort.label *
#1 entry.max$ substring$
'sort.key$ :=
}
This is the function that sets up the order of the entries. Within the function, you have to read the code in reverse. The first important block to note starts with type$ "book" =
and ends in if$
. The line type$ "book" =
tests if the type$
is equal to "book"
. The following line similarly tests if the type is inbook. The following "or" then takes the disjunction of those two, and the result is the test for the if$
at the end. The line following the or
is the first branch of the if$
and the curly-braced block following that line is the else branch of the if$
. The pattern continues inside the block. Basically, this section of code decides which field to sort by initially. The lines following this large section of code concatenate (*
in this language) those primary sort keys with secondary sort keys, like the year. This concatenation continues until eventually the sort.key$
is assigned. So it is our job to re-write this code to set the sort.key$
based on the order of appearance in the manuscript. So, borrowing from unsrtnat.bst, we replace that block of code with the following:
INTEGERS { seq.num }
FUNCTION {init.seq}
{ #0 'seq.num :=}
EXECUTE {init.seq}
FUNCTION {int.to.fix}
{ "000000000" swap$ int.to.str$ *
#-1 #10 substring$
}
FUNCTION {presort}
{ calc.label
label sortify
" "
*
seq.num #1 + 'seq.num :=
seq.num int.to.fix
'sort.label :=
sort.label *
#1 entry.max$ substring$
'sort.key$ :=
}
The result is that the sort key is consistent with the order the entries show up in the manuscript.
But, again, the best way to craft your own custom BST's is almost always to use latex makebst
, which allows you to set custom sort order. Plus, the resulting BST file is natbib compatible.