How to letter space (German) abbreviations automatically

Question

In German abbreviations like "z. B." should be written with a space in between. In good old typewriter days the usual way to write them was without space. And even today I often type them without a space. Is there a way to automatically replace them with a correctly spaced version (e.g. with \, in between)?

I'm on LuaLaTeX, so a Lua solution would be okay.

Example document:

\documentclass{article}
\usepackage[ngerman]{babel}
\begin{document}
Dies ist z.B. ein Test.
\end{document}

This input should result in an output equivalent to z.\,B. in the document.

Answer 1

(I rewrote this answer rather significantly after discussions with OP and after receiving very significant coding help from @EgorSkriptunoff)

Here's a solution that doesn't pre-specify a list of all abbreviations for which thinspace should be inserted after interior periods (aka "full stops"). Instead, it sets up a pattern matching function to capture u.a., u.a.m., u.v.a.m., z.Zt., Bem.d.Red. and many more such cases. (See the function insert_thinspaces in the code below for the exact pattern matches that are performed.)

Observe also the use of unicode.utf8.gsub instead of string.gsub inside the Lua function insert_thinspaces. This lets the code deal correctly with non-ASCII-encoded letters, such as ä and Ä, which may occur in abbreviations.

On the downside (potentially), this solution method doesn't capture abbreviations if they occur at the start of a sentence, e.g., Z.T. or U.U.; for what it's worth, your parallel answer currently doesn't catch such cases either, right?

The Lua function is assigned to the process_input_buffer callback via a LaTeX macro called \ExpandAbbrOn. If, for any reason, you need to suspend operation of the Lua function, simply execute the instruction \ExpandAbbrOff.

The code checks if the string to be processed lies inside a verbatim-like environment such as verbatim, Verbatim, and lstlisting; if that's the case, no processing is performed. And, with the latest iteration, the code now also ignores material that's in the arguments of inline-verbatim-like macros, such as \verb, \Verb, \lstinline, and \url. For sure, the contents of URL strings should never be processed by the Lua function, right?

% !TeX program = lualatex
\documentclass{article}
\usepackage[ngerman]{babel}
\usepackage{fancyvrb}        % for "Verbatim" env.
\usepackage[obeyspaces]{url} % for "\url" macro
\usepackage{listings}        % for "\lstinline" macro

%% Lua-side code:
\usepackage{luacode} % for 'luacode*' environment
\begin{luacode*}

-- Names of verbatim-like environments
local verbatim_env = { "[vV]erbatim" , "lstlisting" }

-- By default, we're *not* in a verbatim-like env.:
local in_verbatim_env = false 

-- Specify number of parameters for every macro; use neg. numbers 
-- for macros that support matching pair of curly braces {} 
local all_macros = {
          verb = 1,
          Verb = 1,
          lstinline = -1,
          url = -1
}

-- List all poss. delimiters
local all_delimiters = [[!"#$%&'*+,-./:;<=>?^_`|~()[]{}0123456789]]

-- Quick check if "s" contains an inline-verbatim-like macro:
function quick_check ( s )
  if s:find("\\[vV]erb") or s:find("\\url") or s:find("\\lstinline") then
    return true
  else
    return false
  end
end

-- Function to process the part of string "s" that 
-- does *not* contain inline-verbatim-like macros
local function insert_thinspaces ( s ) 
  s = unicode.utf8.gsub ( s , 
      "(%l%.)(%a%l?%l?%.)(%a%l?%l?%.)(%a%l?%l?%.)", 
      "%1\\,%2\\,%3\\,%4" ) -- e.g. "u.v.a.m.", "w.z.b.w."
  s = unicode.utf8.gsub ( s , 
      "(%l%.)(%a%l?%l?%.)(%a%l?%l?%.)", 
      "%1\\,%2\\,%3" ) -- e.g., "a.d.Gr." 
  s = unicode.utf8.gsub ( s , 
      "(%u%l%l?%.)(%a%l?%l?%.)(%a%l?%l?%.)", 
      "%1\\,%2\\,%3" ) -- e.g., "Anm.d.Red."
  s = unicode.utf8.gsub ( s , 
      "(%l%.)(%a%l?%l?%.)", 
      "%1\\,%2" ) -- e.g., "z.T.", "z.Zt.", "v.Chr."
  return s
end

-- Finally, the main Lua function:
function expand_abbr ( s )
  -- Check if we're entering or exiting a verbatim-like env.;
  -- if so, reset the 'in_verbatim_env' "flag" and break.
  for i,p in ipairs ( verbatim_env ) do
    if s:find( "\\begin{" .. p .. "}" ) then
      in_verbatim_env = true
      break
    elseif s:find( "\\end{" .. p .. "}" ) then
      in_verbatim_env = false
      break
    end
  end
  -- Potentially modify "s" only if *not* in a verbatim-like env.:
  if not in_verbatim_env then
    -- Quick check if "s" contains one or more inlike-verbatim-like macros:
    if quick_check ( s ) then
      -- See https://stackoverflow.com/a/45688711/1014365 for the source
      -- of the following code. Many many thanks, @EgorSkriptunoff!!
      s = s:gsub("\\([%a@]+)",
        function(macro_name)
          if all_macros[macro_name] then
            return
              "\1\\"..macro_name
              ..(all_macros[macro_name] < 0 and "\2" or "\3")
              :rep(math.abs(all_macros[macro_name]) + 1)
          end
        end
        )
      repeat
        local old_length = #s
        repeat
          local old_length = #s
          s = s:gsub("\2(\2+)(%b{})", "%2%1")
        until old_length == #s
        s = s:gsub("[\2\3]([\2\3]+)((["..all_delimiters:gsub("%p", "%%%0").."])(.-)%3)", "%2%1")
      until old_length == #s
      s = ("\2"..s.."\1"):gsub("[\2\3]+([^\2\3]-)\1", insert_thinspaces):gsub("[\1\2\3]", "")
    else
      -- Since no inline-verbatim-like macro found in "s", invoke 
      -- the Lua function 'insert_thinspaces' directly.
      s = insert_thinspaces ( s )
    end
  end
  return(s)
end

\end{luacode*}

%% LaTeX-side code: Macros to assign 'expand_abbr' 
%% to LuaTeX's 'process_input_buffer' callback.
\newcommand\ExpandAbbrOn{\directlua{%
  luatexbase.add_to_callback("process_input_buffer", 
  expand_abbr, "expand_abbreviations")}}
\newcommand\ExpandAbbrOff{\directlua{%
  luatexbase.remove_from_callback("process_input_buffer", 
  "expand_abbreviations")}}
\AtBeginDocument{\ExpandAbbrOn} % enabled by default

%% Just for this example:
\setlength\parindent{0pt}
\obeylines

\begin{document}

Dies ist u.U. ein Test.
\begin{Verbatim}
Dies ist u.U. ein Test.
\end{Verbatim}

z.B. u.a. u.Ä. u.ä. u.U. a.a.O. d.h. i.e. v.a.
i.e.S. z.T. m.E. i.d.F. z.Z. u.v.m. z.Zt.
u.v.a.m. b.z.b.w. v.Chr. a.d.Gr. Anm.d.Red.

\begin{verbatim}
z.B. u.a. u.Ä. u.ä. u.U. a.a.O. d.h. i.e. v.a.
i.e.S. z.T. m.E. i.d.F. z.Z. u.v.m. z.Zt.
u.v.a.m. b.z.b.w. v.Chr. a.d.Gr. Anm.d.Red.
\end{verbatim}

U.S.A. U.K. % should *not* be processed

\lstinline|u.a. u.Ä. u.ä.|; \Verb$u.a. u.Ä. u.ä.$

% nested verbatim-like macros
\Verb+u.U.\lstinline|u.U.|u.U.+  \lstinline+u.U.\[email protected][email protected].+

% 2 URL strings
u.U. \url{u.U.aaaa.z.T.bbb_u.v.a.m.com} u.U.
u.U. \url?u.U.aaaa.z.T.bbb_u.v.a.m.com? u.U.
\end{document}

Answer 2

Here's a simple solution to the problem. It uses the Lua callback process_input_buffer to scan each input line for one of the given abbreviations and insert a small space in there.

For that action you only need to specify the abbreviation (table key) you want to replace with a spaced version (table value). That mechanism can, of course, be used to simply replace any content (table key) with some other (table value).

This solution also enables you to use verbatim input, but you have to make the verbatim environment known to the script. If the script wouldn't check for verbatim environments those changes would be visible to the reader.

You should note that the function works, but may not be the fastest as it always checks every line whether it is a verbatim start/end for every verbatim environment known to the script.

Update: I have simplified user input (dicitonary) by generating the required spaces automatically. The most problematic part mentioned in the discussions are abbreviations at sentence start and in inline verbatim. The first one may be handled with this version (check for a dot, a space and then the abbreviation with capitalized letter) but the second one may be very hard to detect and change.

\documentclass{article}
\usepackage[ngerman]{babel}
\usepackage{luacode}
\begin{luacode}

local tabbr = {"z.B.","u.a.","u.Ä.","u.ä.","u.U.","a.a.O.","d.h.","i.e.",
  "i.e.S.","v.a.","z.T.","m.E.","i.d.F.","z.Z.","u.v.m.","u.v.a.m.",
  "z.Hd."}
local verbenv = {"[vV]erbatim","lstlisting"}
local tsub = {}
local inverb = false

function createsubstitutes()
  for i,p in ipairs(tabbr) do
    tsub[p] = unicode.utf8.gsub(p:sub(1,p:len()-1), "%.", ".\\,") .. "."
  end
end

function expandabbr(s)
    for i,p in ipairs(verbenv) do
      if s:find("\\begin{" .. p .. "}") then
        inverb = true
        break
      end
      if s:find("\\end{" .. p .. "}") then
        inverb = false
        break
      end
    end
    if not inverb then
      for k,v in pairs(tsub) do
        s = unicode.utf8.gsub(s, k:gsub("(%.)","%%%1"), v)
      end
    end
  return(s)
end

\end{luacode}
\AtBeginDocument{%
  \luaexec{createsubstitutes()}
  \luaexec{luatexbase.add_to_callback("process_input_buffer", expandabbr, "expand_abbreviations")}%
}
\begin{document}
Dies ist z.B. ein Test.\\
In dieser Zeile gibt es z.B. \verb+z.B.+
\begin{verbatim}
Test z.B.
\end{verbatim}
\end{document}

Answer 3

Just a minor tweak to @Mico's answer. I put the replacement in a loop to match abbreviations of unknown length. The drawback is, that each chunk of a abbreviation must meet one of the defined patterns.

For example o.B.d.A. would get evaluated by the second gsub pattern "(%l.)(%a)" and replaced by o.\,B.d.\,A.. In the next loop the pattern wouldn't match, because the chunk B.d. doesn't. I guess the combination of the two patterns should match almost every abbreviation but doesn't create too many false-positives.

Another tweak I made was to check only the closing verbatim-name which matches the first opening. With this a construct of several nested verbatim environments is evaluated correctly. Inline verbatim is still missing though as well as other exceptions like \url.

EDIT: I also wrote a function, that parses inline verbatim correctly, but only if there is only one kind of inline verbatim command per line and if it ends in the same line.

\documentclass{article}
\usepackage[ngerman]{babel}
\usepackage{fancyvrb} % for "Verbatim" environment
\usepackage{luacode}
\usepackage{url}

%% Lua-side code:
\begin{luacode}
local verbatim_env = { "verbatim" , "Verbatim" , "lstlisting" }
local verbatim_inl = { "verb" , "lstinline" }
-- by default, *not* in a verbatim-like env.:
local cur_verbatim_env = nil
local cur_verbatim_inl = nil
function replace_abbrs ( s )
    local rep_rep1 = 1
    local rep_rep2 = 1
    while rep_rep1 ~= 0 or rep_rep2 ~= 0 do
        s,rep_rep1 = unicode.utf8.gsub ( s , "(%l%.)(%a%.)(%a)", "%1\\,%2\\,%3" )
        s,rep_rep2 = unicode.utf8.gsub ( s , "(%l%.)(%a)",     "%1\\,%2" )
    end
    return(s)
end
function expand_inline_verb ( s , p )
    local r = ""
    while string.len(s) > 0 do
        local spos,epos = s:find( p.."%A" )
        if spos ~= nil then
            r = r .. replace_abbrs(s:sub(0,spos-1))
            r = r .. s:sub(spos,epos)
            local delim = s:sub(epos,epos)
            s  = s:sub(epos+1 , string.len(s))
            local verb_end = s:find( delim )
            r = r .. s:sub(0,verb_end)
            s = s:sub(verb_end+1 , string.len(s))
        else
            r = r .. replace_abbrs(s)
            break
        end
    end
    return(r)
end
function expandabbr ( s )
    if cur_verbatim_env == nil then
        for i,p in ipairs ( verbatim_env ) do
            if s:find( "\\begin{" .. p .. "}" ) then
                cur_verbatim_env = verbatim_env[i]
                break
            end
        end
    elseif s:find( "\\end{" .. cur_verbatim_env .. "}" ) then
        cur_verbatim_env = nil
    end
    if cur_verbatim_env == nil and cur_verbatim_inl == nil then
        for i,p in ipairs ( verbatim_inl ) do
            pos = s:find( "\\" .. p )
            if pos ~= nil then
                cur_verbatim_inl = s[pos+string.len(p)+1]
                break
            end
        end
    elseif cur_veratim_inl ~= nil then
        if s:find( cur_veratim_inl ) then
            cur_verbatim_inl = nil
        end
    end
    if cur_verbatim_env == nil then
        for i,p in ipairs ( verbatim_inl ) do
            if s:find( p ) then
                return(expand_inline_verb( s , p ))
            end
        end
        s = replace_abbrs ( s )
    end
    return(s)
end
\end{luacode}

%% LaTeX-side code:
\newcommand\ExpandAbbrOn{\directlua{%
  luatexbase.add_to_callback("process_input_buffer", 
  expandabbr, "expand_abbreviations")}}
\newcommand\ExpandAbbrOff{\directlua{%
  luatexbase.remove_from_callback("process_input_buffer", 
  "expand_abbreviations")}}
\AtBeginDocument{\ExpandAbbrOn} % enabled by default

%% Just for this example:
\setlength\parindent{0pt}
\obeylines

\begin{document}
Dies ist u.U. ein Test.
\begin{Verbatim}
Dies ist u.U. ein Test.
\end{Verbatim}

\begin{Verbatim}
\begin{verbatim}
\end{verbatim}
Dies ist u.U. ein schwierigerer Test.
\end{Verbatim}

z.B. u.a. u.Ä. u.ä. u.U. a.a.O. d.h. i.e. 
i.e.S. v.a. z.T. m.E. i.d.F. z.Z. u.v.m.
z.Zt. o.B.d.A. a.d.Gr. n.Chr. Anm.d.Red.
\verb|z.Zt. o.B.d.A. a.d.Gr. n.Chr. Anm.d.Red.|
\begin{verbatim}
z.B. u.a. u.Ä. u.ä. u.U. a.a.O. d.h. i.e. 
i.e.S. v.a. z.T. m.E. i.d.F. z.Z. u.v.m.
\end{verbatim}
U.S.A. U.K.
\ExpandAbbrOff % turn off abbreviation expansion
A tricky URL: \url{u.U.aaaa.z.T.bbb}
\end{document}