# How to find characters that LaTeX doesn't like?

I use exported .bib entries with biber, biblatex and \usepackage[utf8]{inputenc}, in my literature reports for my group. This means most standard annoying characters (à, etc) are handled automatically. However, there are a lot that are not. Also, it prints gibberish on the command line.

\u8:�

Is the closest I can get (via pdflatex file.tex >demo.txt), thought it gives actual gibberish on the command line, depending on what the symbol is. It also doesn't say anything about where in the bib file that character is, so I have to try and use several runs of kill here and run fully to guess the entry...

This means that quite often I have to search through my document trying to find the one character that is screwing it up. Often it isn't even a letter, but someone is using a non-ASCII hypen or some such. Is there an easy way to check for non-LaTeX approved characters in a file?

The closest I've found is some mode in emacs that turned non-ASCII characters red, but I forget how I did that, and I still had problems noticing one slightly red hyphen in a 3000 line file. Are there better tools? Or even someone who knows who to turn that mode back on?

• Emacs-specific option: C-M-s to start a regular expression search, then search for [[:nonascii:]]. From emacswiki May 2, 2014 at 13:12
• In Windows, you must set your locale (or at least code page) to UTF-8 for console apps to be able to display UTF-8 correctly. Also be sure to select a console font with good support for Unicode, such as Lucida Console or DejaVu Sans Mono. May 15, 2021 at 9:06

I had the same problem in preparing bibliography and I managed to solve it with a text editor Sublime Text. Open the tex file and Ctrl+F, make sure the regular expression (first button) is on and type in [^\x00-\x7F] to find. Special characters are circled.

• This works perfectly in Notepad++ and I use it all the time now, thank you. Sep 15, 2017 at 17:53
• The "Gremlins" extension for VSCode works well also Feb 28, 2021 at 3:19
• @Canageek Which command do you use with Notepad++ ? "find ... what ???" Nov 7, 2021 at 7:41
• @DuchampGérardH.E. [^\x00-\x7F] Nov 14, 2021 at 20:48
• @DuchampGérardH.E. Use the standard search dialog (Control-F) then use the regex radio button at the bottom Nov 15, 2021 at 0:34

I often have to perform this task as a production editor where the supplied files have mixed encodings. I wrote a small bash script called findnonascii that just runs grep:

#!/bin/sh

grep -n -P "[^|a-zA-Z\{\}\s%\./\-:;,0-9@=\\\\\"'_~\$\!&\\?+#\^<>\*]"$@


Sample file:

\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

\begin{document}
Sample character: õ

Another one: â

And again: ê

\end{document}


Output of findnonascii test.tex:

7:    Sample character: õ
9:    Another one: â
11:    And again: ê


Which gives the line numbers, so it narrows the search down a bit.

Edit:

Here's a Perl script that provides a platform-independent alternative:

#!/usr/bin/perl -w

use strict;
use warnings;
use feature 'unicode_strings';

if ($#ARGV == -1) { die "Syntax:$0 <filename>+\n";
}

foreach my $filename (@ARGV) { open (my$FH, $filename) or die "Can't open '$filename' $!\n"; my$linenum = 0;

while (<$FH>) {$linenum++;

if (/[^|a-zA-Z\{\}\s%\.\/\-:;,0-9@=\\\\\"'_~\$\!&\\?+#\^<>\*]/) { print$#ARGV > 0 ? "$filename " : '', "l.$linenum: ", $_; } } close$FH;
}

1;


Edit 2:

The following is a slight modification that will highlight the characters so they're easier to see (I don't know if it will work on Windows):

#!/usr/bin/perl -w

use strict;
use warnings;
use feature 'unicode_strings';
use Term::ANSIColor;

if ($#ARGV == -1) { die "Syntax:$0 <filename>+\n";
}

foreach my $filename (@ARGV) { open (my$FH, $filename) or die "Can't open '$filename' $!\n"; my$linenum = 0;

while (<$FH>) {$linenum++;

if (s/([^|a-zA-Z\{\}\s%\.\/\-:;,0-9@=\\\\\"'_~\$\!&\\?+#\^<>\*]+)/&highlight($1)/eg)
{
print $#ARGV > 0 ? "$filename " : '', "l.$linenum: ",$_;
}

}

close $FH; } sub highlight{ my$text = $_[0]; colored($text, 'on_bright_red');
}

1;


The pattern used above is a subset of ASCII since TeX generally doesn't like control characters (although I rarely encounter a LaTeX file with control codes). A simpler pattern is [^ -~] which excludes ([^...]) the range (start-end) from space (, 0x20) to tilde (~, 0x7E). Note that this range doesn't cover the TAB character (0x09), which (La)TeX usually interprets as a space. If you also want to ignore TAB from the search then use [^ -~\t]. Sophisticated text editors often allow regular expression searches and should accept that pattern.

• Great, now I need to go an install cygwin again... Jul 2, 2014 at 17:43
• @Canageek perl would provide a platform-independent alternative. In fact a perl script could be more informative as it would be possible to find the column as well as the line number in the event of a long line. Jul 2, 2014 at 19:06

VIM approach

I frequently have this problem when copying and pasting text. I also quite often enter accidentally an (invisibly) nonbreaking space (ALT-SPACE on a Mac keyboard). To identify such characters, do the following:

Start with :set hls to let VIM highlight all search results. Then search with /[<RANGE>] for characters in the ASCII code range between <128> and <255>. You can enter a character by its ASCII code by pressing CTRL-V and then enter three digits for the decimal ASCII code:

/[ CTRL-V128 - CTRL-V255 ] ENTER

All non-ASCII characters are highlighted, you can navigate between them with n and N as usual. To stop the highlighting of search results, use :set nohls.

• Can a vim user verify if this works for me? I have no idea how to use vim, being an emacs person. Jul 2, 2014 at 17:43

i've used the log file to help in such cases. in emacs two-window mode, with the log file in one window and the tex file in the other, i can mouse-over the unidentified character in the log, then go to the tex window, ^s to search, click the middle button to enter the search argument, then return to launch the search.

this requires a 3-button mouse, and sometimes several tries, but is the best approach i've found so far, since it doesn't require knowing what the unidentified character is, and the ^s` search is repeatable.

• It seems the windows version of emacs doesn't like this; I can't middle click on the search line. Not sure why... Jun 26, 2014 at 5:27

Probably the easiest method is to paste your text in one of the following websites:

Unicode character detector --- will annotate "weird" characters, so it gives a graphical overview of the type of characters used and helps you find the problem. This sounds similar to what you mentioned about "characters turning red".

Non-ASCII character replacer works probably better, especially for a large amount of text as you don't have to manually find the characters that gives you problems. On the other hand, it's harder to see where the problem was, which might have been handy at times.

I use WinEdt 10. It is possible to find inadmissible unicode characters using search (with the regular expression flag checked!) with the search line "\u".

For more details: WinEdt help -> Regular expressions and scroll down to "Extended RegEx".