# How to create a table of random characters in XeTeX

I want to create a table of 1000 characters picked randomly from the unicode block "unified CJK ideograms" (4E00-9FFF). Each cell should contain exactly one huge character, no character should appear twice. It would be great too if one could specify a seed, so that one can create the same set of characters again.

Is there a way to create such a file automatically using (Xe)TeX? I am not very skilled in doing TeXnical programming, so any help is appreciated.

-
You can of course generate such test text with any programming language you like. It is usually easier to do it out of TeX. –  Leo Liu Jun 10 '11 at 15:49
@Leo Good idea either. –  FUZxxl Jun 10 '11 at 15:52
... why!? I'd love to know what use this could be put to... –  Seamus Jun 10 '11 at 16:26
@Seamus Basically, it is a friend of mine who learns Chinese. He recently asked me, how to find out how many characters he knows. So my idea is to take a sample of 1000 characters from Unicode and let him tick everyone he knows. Then I can calculate how many characters he knows by using statistics. –  FUZxxl Jun 10 '11 at 16:40
@FUZxxl: As a native Chinese speaker, I think I know no more than 6000 Chinese characters, but there are more than 60000 CJK characters in Unicode charset. For a Chinese learner, the first class 3755 characters (out of 6763) from GB2312-80 charset is quite enough. –  Leo Liu Jun 10 '11 at 16:53

I use a simple approach to generate the list. It may cost too much memory and can be improved using better algorithm.

\documentclass{article}
\usepackage[scale=0.8,centering]{geometry}
\usepackage[nofonts]{ctex}
\setCJKmainfont{SimSun}
\parindent=0pt

\usepackage{pgfcore,pgffor}
\pgfmathsetseed{2} % set seed if you wish
\begin{document}
\foreach \i in {1,...,1000}{%
\loop
\pgfmathrandominteger\randind{"4E00}{"9FA5}%
\ifcsname used\randind\endcsname\repeat
\expandafter\xdef\csname used\randind\endcsname{}%
\symbol{\randind}%
\ifnum\numexpr\i/25*25\relax=\i (\i)\par \fi
}
\end{document}


I tested myself, I know about 40% characters in the sheet. 8000 out of 20000 is more than expected, but there're 2103 simplified Hanzi with their unsimplified variants, say I know about 6000.

Well, due to some linguistics research, the most common 3800 characters can cover 99.9% general text, and 6600 cover 99.999%. Thus, 95% characters from Hongloumeng do not mean quite many. The most common 3500 Hanzi's from 《现代汉语常用字表》 cover 99.48% text, I think your friend can read all of them to make a proper test.

-
Note that there're some blanks in the document — some slots have no glyph in the font. –  Leo Liu Jun 10 '11 at 16:36
I checked Chap. 12 of Unicode 6.0 and my fonts, it seems the range can be 4E00--9FA5, while 9FA6--9FCB may contain some rare characters. –  Leo Liu Jun 10 '11 at 17:24
Okay. He finally did this test and knows about 200 characters out of the list, which means about 4000 characters at all. He knows both traditional and simplified characters. I only know about 100 out of the list, so about 2000 at all. –  FUZxxl Jun 11 '11 at 12:48
\documentclass[a4paper]{article}
\usepackage{geometry}
\geometry{margin=2cm,heightrounded}
\usepackage{fontspec}
\setmainfont{STFangsong}

\input{random}
\newcount\cjkcharcnt
%\random=3

\newif\ifshownumbers

\def\cjkchar{\setrannum{\cjkcharcnt}{"4E00}{"9FBB}%
\ifcsname CJK\the\cjkcharcnt\endcsname
\message{Recomputing (collision)}\let\next\cjkchar
\else
\expandafter\let\csname CJK\the\cjkcharcnt\endcsname\empty
\ifshownumbers{\footnotesize(\number\cjkcharcnt) }\fi
\ifnum\XeTeXcharglyph\cjkcharcnt=00
\message{Recomputing (missing character)}\let\next\cjkchar
\else
\char\cjkcharcnt\let\next\relax
\fi
\fi
\next}

\newcommand{\row}{\hbox to\hsize{%
\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil
\ifshownumbers\else
\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil
\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil
\fi
\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil\cjkchar\hfil\cjkchar}}

\begin{document}

%\shownumberstrue % uncomment to show (decimal) numbers

\row\row\row\row\row\row\row\row\row\row
\row\row\row\row\row\row\row\row\row\row
\row\row\row\row\row\row\row\row\row\row
\row\row\row\row\row\row\row\row\row\row
\row\row\row\row\row\row\row\row\row\row
\ifshownumbers
\row\row\row\row\row\row\row\row\row\row
\row\row\row\row\row\row\row\row\row\row
\row\row\row\row\row\row\row\row\row\row
\row\row\row\row\row\row\row\row\row\row
\row\row\row\row\row\row\row\row\row\row
\fi

\end{document}


Ten or twenty characters per row, fifty or a hundred rows; the (decimal) character number is attached if \shownumberstrue is uncommented. If by chance the same number is generated twice, it's discarded and recomputed. I get collisions very rarely (not more than one). The macros check also if the character exists in the font.

The response of time (without \shownumberstrue) is

real        0m2.740s
user        0m1.660s
sys         0m0.384s

-
FUZxxl needs 'no character should appear twice'. –  Leo Liu Jun 10 '11 at 16:22
@Leo: added the check, thanks. I've also restricted the range to "9FBB, as this seems the last CJK defined character. –  egreg Jun 10 '11 at 16:45
That's good. Hmm... how about adding a \iffontchar test for general purpose? And I still like the \loop \repeat approach. –  Leo Liu Jun 10 '11 at 17:02