8

I am trying to output several megabytes of UTF8 text into a printable format. As enscript and friends don't like Unicode, this Unix SE question soon turned to LaTeX.

Summary of requirements: processing of several MB of Chinese, Latin and special chars (i.e. invalid LaTeX code), consisting of long and short lines, into a tightly packed, multi-column format. I know LaTeX has the ability to control font size, spacings, margins and so on, as well as headings and page numbers, so I am not too worried about that.

The problem I am having so far is getting both wrappable and UTF8-capable verbatim input of the file.

Is there a way to get both?

With the fancyvbr package, for example, I get good rendition of the Chinese text, but no line breaking:

\documentclass{article}
\usepackage{CJKutf8}
\usepackage{ucs}
\usepackage{multicol}
\usepackage{verbatim}
\usepackage[encapsulated]{CJK}
\newcommand{\myfont}{bsmi} % or {stheiti}, etc

\begin{document}

\begin{CJK}{UTF8}{\myfont}

    \begin{multicols}{2}
        \VerbatimInput[fontfamily=cmr]{file.txt}
    \end{multicols}
\end{CJK}
\end{document} 

I have so far been unable to get a listings-type environment to deal with the Unicode in the file.

Example of the type of thing I'm feeding it. This to to format thousands of lines of multi-lingual chat logs. Basically 100,000++ lines of this kind of thing:

###### 2013-10-26.223938+0000GMT.html ######
**** user@example.com/ (jabber) ****
(00:00:00) Lorem ipsum
(00:00:01) 車檢作畫病得星定局而是作的所由次園又此對這一馬的生故他試……外由懷黃客建時常嚴在位以說其配場戲回有部結一自法就生機,定的被各世皮全空!也地傳現他重城,書照展商直起眾家不思政國林年八計出地口早體故失離際們層氣,簡他廣集義,四便入的只了極。
(00:00:02) odesset ullamcorper quo. Cu adipisci assentior eam, debet definiebas eos ad. Te eos nihil populo vivendum, vix iusto noster peri
(00:00:02) odesset ullamcorper quo. Cu adip
(02:00:02) odesset ullamcorper quo. Cu adipisci assentior eam, debet d
(02:00:01) 車檢作畫病得星定局而是作的所由次園又此對這一馬的生故他試……外由懷黃客建時常嚴在位以說其配場戲回有部結一自法就生機,定的被各
(02:00:02) ok
(00:01:02) 病
(00:00:02) ok
(00:01:02) 

###### 2013-10-26.223938+0000GMT.html ######
**** user@example.com/ (jabber) ****
(00:00:01) 車檢作畫病得星定局而是作展商直起眾家不思政國林年八計出地口早體故失離際們層氣,簡他廣集義,四便入的只了極。
(00:00:02) odesset ullamcorper quo. Cu adipisci assentior eam, debet definiebas eos ad. Te eos nihil populo vivendum, vix iusto noster peri
(00:00:02) odesset ullamcorper quo. Cu adip
(02:00:02) odesset ullamcorper quo. 

I'm ideally looking for the most general solution possible, as the input text is not necessarily in a well-defined machine-readable format.

Edit also tried the following using the plain verbatim package:

\usepackage{verbatim}

\makeatletter
\def\@xobeysp{ }
\makeatother

with this in the document body:

\verbatiminput{file.txt}

This works for normal text and Chinese, but fails to break lines containing very long words, URLs or long strings of letters, all of which occur in the file.

  • Can you add an excerpt of the file.txt file? – egreg Nov 20 '13 at 16:19
  • Yet another possibility: instead of fancyvrb, try listings. See Verbatim environment and line-breaking. Don't forget columns=fullflexible. – Gilles 'SO- stop being evil' Nov 20 '13 at 16:49
  • I have tried, but have not been able to get listings to work with CJK. Workarounds like this one don't scale to several tens of thousands of Chinese characters! – Inductiveload Nov 20 '13 at 16:59
  • @Inductiveload: Latest version of xeCJK package (since v3.2.3) does support listings packages well, if you need it. – Leo Liu Nov 21 '13 at 11:18
6

As you have already used, verbatim package is easy to configure to get proper line breaking and escape all the special characters.

For CJK text, xeCJK is your friend. And there are some options of xeCJK to control the behavior for verbatim CJK text.

I hacked into xeCJK package to tune the linebreak. It seems better, but not perfect.

% -*- coding: utf-8 -*-
% Compile with XeLaTeX
\documentclass{article}

\usepackage{etoolbox}

\usepackage{verbatim}
\makeatletter
\def\@xobeysp{\ }% Or just a space, with a different result
\let\verbatim@nolig@list\empty
\appto\verbatim@font{\raggedright}
\makeatother

\usepackage{xeCJK}% You'd better use the latest version
\setCJKmonofont{SimSun}
\xeCJKsetup{Verb=false}
\normalspacedchars{}
\ExplSyntaxOn
% Hack into xeCJK package if you want to allow linebreaks after (almost) any character.
% Delete this if you don't want that.
\tex_chardef:D \c_fifty = 50 ~
\xeCJK_inter_class_toks:nnn { Default } { Default } { \tex_penalty:D \c_one_hundred }
\xeCJK_inter_class_toks:nnn { Default } { HalfLeft } { \tex_penalty:D \c_fifty }
\xeCJK_inter_class_toks:nnn { Default } { HalfRight } { \tex_penalty:D \c_one_thousand }
\xeCJK_inter_class_toks:nnn { HalfLeft } { Default } { \tex_penalty:D \c_one_thousand }
\xeCJK_inter_class_toks:nnn { HalfLeft } { HalfLeft } { \tex_penalty:D \c_one_thousand }
\xeCJK_inter_class_toks:nnn { HalfLeft } { HalfRight } { \tex_penalty:D \c_one_thousand }
\xeCJK_inter_class_toks:nnn { HalfRight } { Default } { \tex_penalty:D \c_fifty }
\xeCJK_inter_class_toks:nnn { HalfRight } { HalfLeft } { \tex_penalty:D \c_fifty }
\xeCJK_inter_class_toks:nnn { HalfRight } { HalfRight } { \tex_penalty:D \c_one_thousand }
\ExplSyntaxOff

\begin{document}

\verbatiminput{test.log}

\end{document}

enter image description here

The test log file with long lines:

###### 2013-10-26.223938+0000GMT.html ######
**** user@example.com/ (jabber) ****
(00:00:00) Lorem ipsum
(00:00:01) 車檢作畫病得星定局而是作的所由次園又此對這一馬的生故他試……外由懷黃客建時常嚴在位以說其配場戲回有部結一自法就生機,定的被各世皮全空!也地傳現他重城,書照展商直起眾家不思政國林年八計出地口早體故失離際們層氣,簡他廣集義,四便入的只了極。
(00:00:02) odesset ullamcorper quo. Cu adipisci assentior eam, debet definiebas eos ad. Te eos nihil populo vivendum, vix iusto noster peri
(00:00:02) `~!@#$%^&*()_+-={}[]|\:;"'.?/
(02:00:02) LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLONG LINE
  • This is the closest answer to what I want to happen, and it generates output that I can use right now. The only (minor) issue is that the line breaking doesn't prefer to break on a space or special char like, say, "?" in a URL. Is there a way to do this? – Inductiveload Nov 21 '13 at 17:31
  • @Inductiveload: You'd better specify how to break lines in detail. BTW, TeX don't understand the text, it can hardly recognize URLs. – Leo Liu Nov 22 '13 at 2:49
  • The way the script I posted does it is in two steps. There are a set of "breakable chars" that it looks for, so far "&;.?!", as this covers most URLs and file names. First it adds "\-" after the first of these chars in each run of chars. Then, it adds more soft-hyphens after every 12 sequences of characters that don't include an existing soft-hyphen or whitespace. This way, it will break on "nice" chars and spaces first, and then split long chunks up as best it can. Of course, there is no awareness of hyphenation correctness, but that is really not an issue, as 99% are done well. – Inductiveload Nov 22 '13 at 11:12
  • @Inductiveload: Maybe you have noticed, I edited my solution and tried my best to provide a reasonable linebreak routine. My solution exploited TeX's linebreaking algorithm via setting \penalty arguments, which may be smarter than your perl script. – Leo Liu Nov 22 '13 at 13:31
  • Wonderful! The breaking is indeed nice and very sensible looking! One final thing: any possibility of \raggedright in this kind of \verbatiminput environment? Currently, it is throwing a lot of underfull hboxes (when it can only fit one chunk on a line). – Inductiveload Nov 22 '13 at 17:57
6

enter image description here

You can tell xetex to linebreak Chinese and not to worry about characters that are normally special. (You probably need to make more characters catcode 12, this shows the basic idea)

\documentclass{article}

\usepackage{fontspec}

\setmainfont{SimSun}


\newcommand\showlog[1]{{%
\catcode`\#=12
\catcode`\^=12
\catcode`\$=12
\obeylines
\raggedright
\XeTeXlinebreaklocale "zh"
\input{#1}}}

\begin{document}


\showlog{\jobname.txt}

\end{document}
3

you do not have any special character except the # then you can use a simple \input. My example uses xelatex which makes live easier with CJK languages.

\documentclass{article}
\usepackage[margin=1cm]{geometry}
\usepackage{fontspec}
\setmainfont{Code2000}
\setmonofont{DejaVu Sans Mono}%% has no chines characters
\usepackage{pdflscape}
\newcommand\Listing[1]%
  {\begin{landscape}
   \catcode`\#=12%
   \obeylines
%   \ttfamily%%%    only possible with a mono type font with chinese characters
   \footnotesize
   \input{#1}%
  \end{landscape}}

\begin{document}
foo

\Listing{xxx.tex}%% THE EXTERNAL FILE

bar 
\end{document} 

enter image description here

  • 1
    I have no guarantees about what is and is not in the text. Other than #, I can see at least ^, \, $, / in the file, there could be anything else in there. – Inductiveload Nov 20 '13 at 18:39
  • no problem, it can also escaped. And, of course, you can use \VerbatimInput from package fancyvrb instead – user2478 Nov 20 '13 at 18:49
  • The problem is that doesn't wrap lines (though you can hack normal verbatim to break on spaces, long words, URLs, etc are unbroken and stick out of the right hand side of the column). – Inductiveload Nov 20 '13 at 19:14
  • @Inductiveload: So what do you want to break the lines? Is it allowed to break after any character? – Leo Liu Nov 21 '13 at 11:43
  • Yes, pretty much. So far I am using a bash script to insert a \- after special chars like \,.&?!, and then breaking any run of text without \- or whitespace with \-. – Inductiveload Nov 21 '13 at 11:49
1

My current best effort at this is based on the concept by @Gilles at Unix SE. It is not a pure *TeX implementation, and it is not perfect, but it is the one that has allowed me to compress my first 1.4MB of text onto 36 pages: around 40kB and 1400 lines per page.

I use perl to pre-process the input text to escape LaTeX special characters, and also add soft-breaks at certain special characters (like ?!/\$, etc), as well as artificially breaking up long words and meaningless strings into maximum-length runs with more soft-hyphens. Obviously, this won't respect "good" places to hyphenate, but that is an incredibly small detail, as in nearly all the cases, it either breaks on a space, special char, or the string itself doesn't have a good place to break anyway.

This is then processed with XeLaTeX and CJK to handle the Chinese using xeCJK. It is placed in a multicols with very narrow columns because there are a lot of very short lines, and I don't want to waste space.

#! /bin/bash

maxLength=12
breakableChars='\&\;\/\.\?\!\='

# break singly-occuring breakable chars, and the first one in runs with a soft hyphen
# break any un-broken runs of any ascii with a soft hyphen
# replace any backslashes with \textbackslash{}
# replace braces (except the ones in \textbackslash{})
# escape #$%&_ with backslashes
# escape ^ | and ~ with \textTHING{}
# line break after all non-empty lines
cat $1 | 
perl -pe 's/(?<![\s'$breakableChars'])(['$breakableChars'])(?!\s)/\1\\-/g;' | 
perl -pe 's/(((?<!-)[!-~]){'$maxLength'}(?!\s))/\1\\-/g;' |
perl -pe 's/\\(?!-)/\\textbackslash{}/g;' | 
perl -pe 's/(?<!\\textbackslash){/\\{/g;' | 
perl -pe 's/(?<!\\textbackslash\{)\}/\\}/g;' | 
perl -pe 's/([\#\%\$\&\_])/\\\1/g;' | 
perl  -pe 's/\^/\\textasciicircum{}/g;'| 
perl -pe 's/\|/\\textbar/g;' | 
perl -pe 's/~/\\textasciitilde{}/g;' | 
perl -pe 's/^(.*\S.*)$/\1\\\\/g;'  > $2

cat << EOF > driver.tex
\documentclass{article}

\usepackage{fontspec}
\usepackage{multicol}

\usepackage[margin=1cm]{geometry}
\usepackage{setspace} 
\usepackage{xeCJK}

\setstretch{0.5}
\setlength{\columnsep}{0.1cm}
\setmainfont{Nimbus Sans L Regular Condensed}
\setCJKmainfont{WenQuanYi Micro Hei}

\newcommand\showlog[1]{{%
\obeylines
\raggedright
\XeTeXlinebreaklocale "zh"
\input{#1}}}

\begin{document}
    \raggedright
    \fontsize{4}{5}\selectfont

    \begin{multicols}{9}
      \showlog{h.txt}
    \end{multicols}

\end{document}
EOF

/opt/texbin/xelatex driver.tex

The main remaining issue then is that non-Chinese, non-Latin-ish characters not in either the CJK font or the main font don't appear. Using ucharclasses and Code2000 works better, but it is horribly slow.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.