%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % % A SMALL TUTORIAL ON THE MULTILINGUAL FEATURES OF PATGEN2 % % % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentstyle{article} \title{A small tutorial on the multilingual features of PatGen2} \author{Yannis Haralambous} \date{} \begin{document} \maketitle \def\XeT{X\kern-.1667em\lower.5ex\hbox{E}\kern-.125emT} I will very briefly discuss and illustrate by an example the features of PatGen2\footnote{the extension to PatGen by Peter Breitenlohner, author of many beautiful and extremely useful \TeX ware, such as \TeX--\XeT, {\tt DVIcopy} etc.}, related to the {\em translation} file. \section{Syntax of the translation file} \subsection{What is the problem} The problem is that in our (non-English, non-Latin, non-Indonesian\footnote{As far as I know, English, Latin and Indonesian are the only languages without diacritics or special characters...}) languages, we have more characters than just the~26 letters of the Latin alphabet. Not to mention that alphabetical orders can be quite different (remember: in Spanish, cucaracha comes before chacal), and that there can be many ways to express these additional characters: for example the French \oe{} (like in << Histoire de l'\oe il >> by Georges Bataille) can be given to \TeX\ as an 8-bit character, or as \verb*=\oe =, or as \verb=\oe{}=, or as \verb=^^f7= (since we are {\em all} now working with DC fonts) and so forth. Now let's make (or enhance the) hyphenation patterns for our language(s). Fred Liang has not provided PatGen with the extensibility feature: before PatGen2, if your language had less than~26 letters you could hack patterns by substituting letters, if it had more\ldots then the odds were against you. So the problem is to express additional characters in the patterns, in some understandable way (and if possible to get the results in the appropriate alphabetical order). \subsection{The solution} PatGen2 allows you to include more characters than the~26 letters of the alphabet. You can specify arbitrary many {\em input forms} for each of them (these forms will be identified internally), and you can specify the alphabetical order of your language. The latter feature is only interesting when you want to study the patterns file to correct bugs, it doesn't affect the result. These informations are transmitted to PatGen2 by means of a {\em translation} file (usually with the extension \verb=.tra=). The syntax of this file is not very user-friendly, but if you deal with PatGen you are supposed to be a hacker anyway, and hackers just {\bf love} weird-but-efficient-syntaxes [the proof: we all like \TeX!\footnote{\ldots just joking}]. On the first line you specify the \verb=\lefthyphenmin= and \verb=\righthyphenmin= values. The seven first positions of this line are used as follows: two positions for \verb=\lefthyphenmin=, two positions for \verb=\righthyphenmin=, and one position for substitute symbols of each of \verb=.=, \verb=-=, \verb=*= in the output file. This can be useful if you are going to use the dot the asterisk or the hyphen to describe your additional characters (for example one may use the dot to specify the dot accent on a letter). The next lines concern letters of the alphabet of our language. The first position of each line is the delimiter of our fields. You know from languages like C or {\tt awk} that it is very useful to define our own delimiter (instead of the blank space) if for any reason we want to {\em include} a blank space into our fields. This will be the case in the example: one usually leaves a blank space after a \TeX\ macro without argument. Otherwise just leave the first position blank. Next comes the standard representation of our character (preferably lowercase). This is how the character will appear in the patterns and in the hyphenated output file. Follows again a delimiter, and all the {\em equivalent} representations of your character: its uppercase form, and as many other input forms we wish, always followed by a delimiter. The order of lines specifies the alphabetical order of your characters. When PatGen sees two consecutive delimiters, it stops reading; so we can include comments after that. \section{An example} To illustrate this I have taken some Greek words (people who know $X\acute\alpha% \rho\rho\upsilon$ $K\lambda\acute\upsilon\nu\nu$ and his album $\Pi\alpha\tau% \acute\alpha\tau\epsilon\varsigma$ will recognize some of these words\ldots) and included an $\alpha$ with accent, and a variant representation of $\pi$, in form of a macro \verb*=\varpi =. Also I followed the Greek alphabetical order in the translation file. Here is my list of words: \begin{verbatim} A-NU-PO-TA-QTOS A-KA-TA-M'A-QH-TOS A-EI-MNH-STOS MA-NA TOUR-KO-GU-FTIS-SA NU-QO-KO-PTHS LI-ME-NO-FU-LA-KAS MPRE-LOK A-GA-\varpi A-EI PEI-RAI-'AS \end{verbatim} (file {\tt greek.dic}); and here is my translation file ({\tt greek.tra}): \begin{verbatim} 1 1 a A 'a 'A b B g G d D e E z Z h H j J i I k K l L m M n N o O #p#P#\varpi ## r R s S t T u U f F q Q y Y w W \end{verbatim} (yes, yes, that's not a joke; Greek really uses values~1 and~1 for minimal right and left hyphenations!). As you see, I have chosen \verb=#= as delimiter in the case of $\pi$, because \verb=\varpi= is supposed to be followed by a blank space. And then I wrote two delimiters (\verb=##=) to show PatGen2 that I finished talking about $\pi$. Here is what I got on my console: \begin{verbatim} PatGen -t greek.tra -o greek.out greek.dic This is PatGen, C Version 2.1 / Macintosh Version 2.0 Copyright (c) 1991-93 by Wilfried Ricken. All rights reserved. left_hyphen_min = 1, right_hyphen_min = 1, 24 letters 0 patterns read in pattern trie has 266 nodes, trie_max = 290, 0 outputs hyph_start: 1 hyph_finish: 2 pat_start: 2 pat_finish: 4 good weight: 1 bad weight: 1 threshold: 1 processing dictionary with pat_len = 2, pat_dot = 1 0 good, 0 bad, 31 missed 0.00 %, 0.00 %, 100.00 % 69 patterns, 325 nodes in count trie, triec_max = 440 25 good and 42 bad patterns added (more to come) finding 29 good and 0 bad hyphens, efficiency = 1.16 pattern trie has 333 nodes, trie_max = 349, 2 outputs processing dictionary with pat_len = 2, pat_dot = 0 29 good, 0 bad, 2 missed 93.55 %, 0.00 %, 6.45 % 45 patterns, 301 nodes in count trie, triec_max = 378 2 good and 43 bad patterns added finding 2 good and 0 bad hyphens, efficiency = 1.00 pattern trie has 339 nodes, trie_max = 386, 6 outputs processing dictionary with pat_len = 2, pat_dot = 2 31 good, 0 bad, 0 missed 100.00 %, 0.00 %, 0.00 % 47 patterns, 303 nodes in count trie, triec_max = 369 0 good and 47 bad patterns added finding 0 good and 0 bad hyphens pattern trie has 344 nodes, trie_max = 386, 13 outputs 51 nodes and 11 outputs deleted total of 27 patterns at hyph_level 1 pat_start: 2 pat_finish: 4 good weight: 1 bad weight: 1 threshold: 1 processing dictionary with pat_len = 2, pat_dot = 1 31 good, 0 bad, 0 missed 100.00 %, 0.00 %, 0.00 % 27 patterns, 283 nodes in count trie, triec_max = 315 0 good and 27 bad patterns added finding 0 good and 0 bad hyphens pattern trie has 295 nodes, trie_max = 386, 4 outputs processing dictionary with pat_len = 2, pat_dot = 0 31 good, 0 bad, 0 missed 100.00 %, 0.00 %, 0.00 % 27 patterns, 283 nodes in count trie, triec_max = 303 0 good and 27 bad patterns added finding 0 good and 0 bad hyphens pattern trie has 320 nodes, trie_max = 386, 6 outputs processing dictionary with pat_len = 2, pat_dot = 2 31 good, 0 bad, 0 missed 100.00 %, 0.00 %, 0.00 % 24 patterns, 280 nodes in count trie, triec_max = 286 0 good and 24 bad patterns added finding 0 good and 0 bad hyphens pattern trie has 328 nodes, trie_max = 386, 9 outputs 35 nodes and 7 outputs deleted total of 0 patterns at hyph_level 2 hyphenate word list? y writing PatTmp.2 31 good, 0 bad, 0 missed 100.00 %, 0.00 %, 0.00 % Time elapsed: 0:37:70 minutes. \end{verbatim} and here are the results: first of all the patterns (file {\tt greek.out}) \begin{verbatim} a1g a1e a1k a1m a1n a1p a1t a1q 'a1q e1l e1n h1t i1'a i1m i1r 1ko o1g o1p o1t o1f r1k s1s 1st u1l u1p u1f u1q \end{verbatim} As you see, {\tt o1t} comes before {\tt o1f}: (smile) simply because we are talking about $o1\tau$ and $o1\phi$. So the alphabetical order is well respected. Also you see that PatGen2 has read the word \verb*=A-GA-\varpi A-EI= exactly as if it were \verb*=A-GA-PA-EI= and has made out of it the pattern {\tt a1p} (if you look in the input words there is no other reason for this pattern to exist). And here is our result (file {\tt PatTmp.2}): \begin{verbatim} a*nu*po*ta*qtos a*ka*ta*m'a*qh*tos a*ei*mnh*stos ma*na tour*ko*gu*ftis*sa nu*qo*ko*pths li*me*no*fu*la*kas mpre*lok a*ga*pa*ei pei*rai*'as \end{verbatim} (Of course, it is correct; you think I would have shown it if it weren't correct?) As you see there is no \verb*=\varpi = anymore: PatGen2 has really replaced it by \verb=p=, and so \verb*=A-GA-\varpi A-EI= has become \verb=a*ga*pa*ei=. \section{Where do I find more information?} I voluntarily didn't discussed the various parameters used for pattern generation. For these there is very good litterature: \begin{itemize} \item the \TeX book, by the Grand Wizard of \TeX\ arcana, appendix H; \item \LaTeX\ Erweiterungsm\"oglichkeiten, by Helmut Kopka, Addison-Wesley, Pages 482--489; \item Swedish Hyphenation for \TeX, by Jan Michael Rynning, [sorry, I don't know where this paper is published]; \item Word Hy-phen-a-tion by Com-put-er, Stanford University Report {\tt STAN-CS-83-977}; \item forthcoming paper by Dominik Wujastyk, on British hyphenation; \item Hyphenation Patterns for Ancient Greek and Latin, TUGboat~13 (4), pages 457--469. \item and many others\ldots \end{itemize} Where to get PatGen2? probably everywhere, but certainly in Stuttgart ({\tt IP 129.69.1.12}), \verb=soft/tex/systems/pc/utilities/patgen.zip= (take a look also at \verb=soft/tex/systems/knuth/texware/patgen.version2.1/patgen.README=). \section{Go forth, etc etc} OK, once again {\sc Go Forth} and make masterpieces of hyphenation patterns\ldots {\bf but} please get in touch with the TWGMLC (Technical Working Group on Multiple Language Coordination) first, since people there are working on many languages: maybe they have already done what you need and are still testing it; or maybe they haven't and in that case you could help us a lot. \end{document}