How to find characters that LaTeX doesn't like?

Question

I use exported .bib entries with biber, biblatex and \usepackage[utf8]{inputenc}, in my literature reports for my group. This means most standard annoying characters (à, etc) are handled automatically. However, there are a lot that are not. Also, it prints gibberish on the command line.

\u8:�

Is the closest I can get (via pdflatex file.tex >demo.txt), thought it gives actual gibberish on the command line, depending on what the symbol is. It also doesn't say anything about where in the bib file that character is, so I have to try and use several runs of kill here and run fully to guess the entry...

This means that quite often I have to search through my document trying to find the one character that is screwing it up. Often it isn't even a letter, but someone is using a non-ASCII hypen or some such. Is there an easy way to check for non-LaTeX approved characters in a file?

The closest I've found is some mode in emacs that turned non-ASCII characters red, but I forget how I did that, and I still had problems noticing one slightly red hyphen in a 3000 line file. Are there better tools? Or even someone who knows who to turn that mode back on?

Answer 1

I had the same problem in preparing bibliography and I managed to solve it with a text editor Sublime Text. Open the tex file and Ctrl+F, make sure the regular expression (first button) is on and type in [^\x00-\x7F] to find. Special characters are circled.

Answer 2

I often have to perform this task as a production editor where the supplied files have mixed encodings. I wrote a small bash script called findnonascii that just runs grep:

#!/bin/sh

grep -n -P "[^|a-zA-Z\{\}\s%\./\-:;,0-9@=\\\\\"'\(\)_~\$\!&\`\?+#\^<>\[\]\*]" $@

Sample file:

\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

\begin{document}
Sample character: õ

Another one: â

And again: ê

\end{document}

Output of findnonascii test.tex:

7:    Sample character: õ
9:    Another one: â
11:    And again: ê

Which gives the line numbers, so it narrows the search down a bit.

Edit:

Here's a Perl script that provides a platform-independent alternative:

#!/usr/bin/perl -w

use strict;
use warnings;
use feature 'unicode_strings';

if ($#ARGV == -1)
{
   die "Syntax: $0 <filename>+\n";
}

foreach my $filename (@ARGV)
{
   open (my $FH, $filename)
      or die "Can't open '$filename' $!\n";

   my $linenum = 0;

   while (<$FH>)
   {
      $linenum++;

      if (/[^|a-zA-Z\{\}\s%\.\/\-:;,0-9@=\\\\\"'\(\)_~\$\!&\`\?+#\^<>\[\]\*]/)
      {
         print $#ARGV > 0 ? "$filename " : '', "l.$linenum: ", $_; 
      }
   }

   close $FH;
}

1;

Edit 2:

The following is a slight modification that will highlight the characters so they're easier to see (I don't know if it will work on Windows):

#!/usr/bin/perl -w

use strict;
use warnings;
use feature 'unicode_strings';
use Term::ANSIColor;

if ($#ARGV == -1)
{
   die "Syntax: $0 <filename>+\n";
}

foreach my $filename (@ARGV)
{
   open (my $FH, $filename)
      or die "Can't open '$filename' $!\n";

   my $linenum = 0;

   while (<$FH>)
   {
      $linenum++;

      if (s/([^|a-zA-Z\{\}\s%\.\/\-:;,0-9@=\\\\\"'\(\)_~\$\!&\`\?+#\^<>\[\]\*]+)/&highlight($1)/eg)
      {
         print $#ARGV > 0 ? "$filename " : '', "l.$linenum: ", $_;
      }

   }

   close $FH;
}

sub highlight{
  my $text = $_[0];

  colored($text, 'on_bright_red');
}

1;

The pattern used above is a subset of ASCII since TeX generally doesn't like control characters (although I rarely encounter a LaTeX file with control codes). A simpler pattern is [^ -~] which excludes ([^...]) the range (start-end) from space (, 0x20) to tilde (~, 0x7E). Note that this range doesn't cover the TAB character (0x09), which (La)TeX usually interprets as a space. If you also want to ignore TAB from the search then use [^ -~\t]. Sophisticated text editors often allow regular expression searches and should accept that pattern.

Answer 3

VIM approach

I frequently have this problem when copying and pasting text. I also quite often enter accidentally an (invisibly) nonbreaking space (ALT-SPACE on a Mac keyboard). To identify such characters, do the following:

Start with :set hls to let VIM highlight all search results. Then search with /[<RANGE>] for characters in the ASCII code range between <128> and <255>. You can enter a character by its ASCII code by pressing CTRL-V and then enter three digits for the decimal ASCII code:

/[ CTRL-V128 - CTRL-V255 ] ENTER

All non-ASCII characters are highlighted, you can navigate between them with n and N as usual. To stop the highlighting of search results, use :set nohls.

Answer 4

i've used the log file to help in such cases. in emacs two-window mode, with the log file in one window and the tex file in the other, i can mouse-over the unidentified character in the log, then go to the tex window, ^s to search, click the middle button to enter the search argument, then return to launch the search.

this requires a 3-button mouse, and sometimes several tries, but is the best approach i've found so far, since it doesn't require knowing what the unidentified character is, and the ^s search is repeatable.

Answer 5

Probably the easiest method is to paste your text in one of the following websites:

Unicode character detector --- will annotate "weird" characters, so it gives a graphical overview of the type of characters used and helps you find the problem. This sounds similar to what you mentioned about "characters turning red".

Non-ASCII character replacer works probably better, especially for a large amount of text as you don't have to manually find the characters that gives you problems. On the other hand, it's harder to see where the problem was, which might have been handy at times.

Answer 6

I use WinEdt 10. It is possible to find inadmissible unicode characters using search (with the regular expression flag checked!) with the search line "\u".

For more details: WinEdt help -> Regular expressions and scroll down to "Extended RegEx".