Search for tab in external file with filename in another external file (\catcode is not the problem)

Question

This is a quite specific question. I have some code and I'm completely stuck because it won't work for an inexplicable reason. I need to find in which line of an external file there is a tab. I use the ifthen and the xstring package to test everything. I tried to make it as short as possible (which is not quite short...).
If I simply have to search for a tab from a given file with name \fname there is no problem at all (this is really a minimal working example):

\documentclass{article}

\usepackage{ifthen}
\usepackage{xstring}

\begin{document}

\newcounter{nolines}  % counter NumberOfLines

\def\fname{FileOne.txt}  % define filename

OPENING \fname
\immediate\newread\file
\immediate\openin\file=\fname

\setcounter{nolines}{0}

\catcode`\^^I=11  % set tab (   ) to normal character
\loop\unless\ifeof\file  % loop until end of \file
  \stepcounter{nolines}  % counter +1

  \immediate\read\file to\fline  % read a line of \file
  \ifthenelse{\equal{\fline}{\par}}{% if \fline empty, very important check
    Line \thenolines: emtpy
  }{% if \fline not empty
    \IfSubStr{\fline}{  }{% if \fline contains 'tab'
      Line \thenolines: tab
    }{% if \fline does not contain 'tab'
      Line \thenolines: no tab.
    }
  }
\repeat  % go back to \loop
\catcode`\^^I=10  % revert the code of tab (    )

CLOSING \fname
\immediate\closein\file



\end{document}

The second argument of the command \IfSubStr is a tab! Change it if you copy/paste before running. An example of the text file:

1.This is some text.

3.Here comes a tab: !
4.And some more text.

Again change the space to tab (after a tab: of course) if you copy/paste. This gives the output:

OPENING File.txt
Line 1: no tab.
Line 2: emtpy
Line 3: tab
Line 4: no tab.
Line 5: emtpy
CLOSING File.txt

Which is entirely correct. But if I have to search for the file-name in another file (this file contains a lot of file-names, and there are a lot of those 'parent-files'. Thus manually is out of the question. The example is kept simple) it suddenly detects in every scanned line a tab (not empty lines). Replace \def\fname{FileOne.txt} % define filename with

\def\parentfile{File.txt}  % define parentfile instead

\immediate\newread\parent
\immediate\openin\parent=\parentfile

\immediate\read\parent to\fname  % read a line of the parentfile

\ifthenelse{\equal{\fname}{\par}}{% if \fname is empty
}{% if \fname is not emtpy

and add at the bottom before \end{document}

  }
\immediate\closein\parent

File.txt simply contains

FileOne.txt

This gives the unexpected output of

OPENING FileOne.txt
Line 1: tab
Line 2: emtpy
Line 3: tab
Line 4: tab
Line 5: emtpy
CLOSING FileOne.txt

Which is not what I wanted. Without changing anything to the 'recognize tab code', still the output is different for a reason I really do not understand and cannot find out (for 2 days now...)

Sorry again for the long question, but I couldn't make it shorter.

Answer 1

This is to do with tokenization, nothing to do with the 'parent file' business. When you put your 'inner' code inside the argument of \ifthenelse, TeX will tokenize it with the currently applicable category codes. You have (effectively)

\ifthenelse{\equal{\fname}{\par}}{% if \fname is empty
}{% if \fname is not emtpy
  % Other stuff

  \catcode`\^^I=11  % set tab (   ) to normal character
  \loop\unless\ifeof\file  % loop until end of \file
    \stepcounter{nolines}  % counter +1

    \immediate\read\file to\fline  % read a line of \file
    \ifthenelse{\equal{\fline}{\par}}{% if \fline empty, very important check
      Line \thenolines: emtpy
    }{% if \fline not empty
      \IfSubStr{\fline}{  }{% if \fline contains 'tab'
        Line \thenolines: tab
      }{% if \fline does not contain 'tab'
        Line \thenolines: no tab.
      }
    }
  \repeat  % go back to \loop
  \catcode`\^^I=10  % revert the code of tab (    )
}

in your second case. This means that TeX absorbs the tab characters with category code 10 ('space'), and your \catcode`\^^I=11 does nothing at all. You need to change the category code before TeX absorbs anything. That would be easiest to do using an auxiliary

\begingroup
  \catcode`\^^I=11  % set tab (   ) to normal character
  \gdef\mytestsystem{%
    \loop\unless\ifeof\file  % loop until end of \file
      \stepcounter{nolines}  % counter +1
      \catcode`\^^I=11  % set tab (   ) to normal character
      \immediate\read\file to\fline  % read a line of \file
      \ifthenelse{\equal{\fline}{\par}}{% if \fline empty, very important check
        Line \thenolines: emtpy
      }{% if \fline not empty
        \IfSubStr{\fline}{  }{% if \fline contains 'tab'
          Line \thenolines: tab
        }{% if \fline does not contain 'tab'
          Line \thenolines: no tab.
        }
      }
    \repeat  % go back to \loop
    \catcode`\^^I=10  % revert the code of tab (    )
  }
\endgroup

which can then be used without worrying about the nature of tabs at point of use.

As egreg notes, you need to set the catcode of the tab inside the auxiliary as well: this applies when the file is read, and so sets the tokenization which applies to \fline.

Answer 2

The problem is that you're now inserting the previous code as the argument to \ifthenelse, so the change in the category code of ^^I is not effective, as category codes are frozen once tokens have been read in.

Move the \catcode`\^^I=11 before the newly added \ifthenelse, or avoid \ifthenelse altogether:

\documentclass{article}

\def\apar{\par}
\newcounter{nolines}  % counter NumberOfLines
\newread\parent
\newread\file

\usepackage{xstring}

\begin{document}


%\def\fname{\jobname.dat}  % define filename

\def\parentfile{\jobname.fil}  % define parentfile instead
\openin\parent=\parentfile\relax
\read\parent to\fname  % read a line of the parentfile
\ifx\fname\apar\else
  OPENING \fname
  \openin\file=\fname\relax

  \setcounter{nolines}{0}
  \catcode`\^^I=11  % set tab to normal character
  \loop\unless\ifeof\file  % loop until end of \file
    \stepcounter{nolines}  % counter +1
    \read\file to\fline  % read a line of \file
    \ifx\fline\apar % if \fline empty, very important check
      Line \thenolines: empty
    \else
      \IfSubStr{\fline}{^^I}{% if \fline contains 'tab'
        Line \thenolines: tab
      }{% if \fline does not contain 'tab'
        Line \thenolines: no tab.
      }
  \fi
  \par
  \repeat  % go back to \loop
  \catcode`\^^I=10  % revert the code of tab

  CLOSING \fname
  \closein\file
\fi
\closein\parent


\end{document}

Of course saving the inner loop in a macro will vastly improve your document's appearance.

(I have changed the filenames to \jobname.dat for the "contents file" and to \jobname.fil for the file name file, just not to pollute my space.)

---

Here's a version with the definition; I've chosen to set the category code to 12 instead of 11, which might give problems if the read in file contains control sequences.

\documentclass{article}
\usepackage{xstring}

\def\apar{\par}
\newcounter{nolines}  % counter NumberOfLines
\newread\parent
\newread\file

\begingroup
\catcode`\^^I=12
\gdef\testfortab{%
  \ifx\fname\apar\else
  OPENING \fname
  \openin\file=\fname
  \par
  \setcounter{nolines}{0}
  \catcode`\^^I=12  % set tab to normal character
  \loop\unless\ifeof\file  % loop until end of \file
    \stepcounter{nolines}  % counter +1
    \read\file to\fline  % read a line of \file
    \ifx\fline\apar % if \fline empty, very important check
      Line \thenolines: empty
    \else
      \IfSubStr{\fline}{^^I}{% if \fline contains 'tab'
        Line \thenolines: tab
      }{% if \fline does not contain 'tab'
        Line \thenolines: no tab.
      }
  \fi
  \par
  \repeat  % go back to \loop
  \fi
  \catcode`\^^I=10  % revert the code of tab

  CLOSING \fname
  \closein\file
}
\endgroup



\begin{document}

\def\parentfile{\jobname.fil}
\openin\parent=\parentfile
\ifeof\parent\else
  \read\parent to\fname  % read a line of the parentfile
  \testfortab
\fi
\closein\parent

\end{document}

The juggling between \catcode`\^^I=12 and \catcode`\^^I=10 in the body of the definition may be avoided by enclosing the inner loop in a group: just saying

\begingroup\catcode`\^^I=12

instead of \catcode`\^^I=12, and \endgroup instead of \catcode`\^^I=10 at the end. It mostly depends on what you really have to do with \fname. If its usage is local, as in the example, it would probably be better. If you need to use it (or a modified version thereof, perhaps removing tabs) to input a file, then enclosing the loop in a group is not a good thing to do, because the whole file would be input inside the group.

An alternative version without grouping might be

\chardef\savedtabcatcode=\catcode`\^^I
\catcode`\^^I=12
\newcommand\testfortab{%
  \ifx\fname\apar\else
  OPENING \fname
  \openin\file=\fname\relax
  \par
  \setcounter{nolines}{0}
  \chardef\savedtabcatcode=\catcode`\^^I
  \catcode`\^^I=12  % set tab to normal character
  \loop\unless\ifeof\file  % loop until end of \file
    \stepcounter{nolines}  % counter +1
    \read\file to\fline  % read a line of \file
    \ifx\fline\apar % if \fline empty, very important check
      Line \thenolines: empty
    \else
      \IfSubStr{\fline}{^^I}{% if \fline contains 'tab'
        Line \thenolines: tab
      }{% if \fline does not contain 'tab'
        Line \thenolines: no tab.
      }
  \fi
  \par
  \repeat  % go back to \loop
  \fi
  \catcode`\^^I=\savedtabcatcode  % revert the code of tab

  CLOSING \fname
  \closein\file
}
\catcode`\^^I=\savedtabcatcode  % revert the code of tab

---

By the way, \immediate is for writing, not for reading: in front of \openin, \read and \closein it does nothing.