		Preparing and maintaining Yacas documentation

	    Introduction

Yacas documentation in HTML and PS/PDF formats is generated by Yacas scripts from Yacas source files. However, it is cumbersome to write those source files in the Yacas language. The script {txt2yacasdoc.pl} makes it possible to create and maintain the documentation in an easy-to-read form.

The "source" form of all documentation is maintained in a special plain text format. The format is such that it is clearly readable without any processing and is easy to edit. To compile the documents, the system processes the plain text docs with a script to prepare Yacas-language files and then runs the Yacas scripts to produce the final documentation.

The source text must be formatted in a certain fashion to delimit sections, code examples, and so on, but the format is easy enough to enter in a plain text editor. Text is marked up mostly by TAB characters, spaces, and asterisks "{*}" at the beginning of a line. The script {txt2yacasdoc.pl} converts this markup into Yacas code.

There is also a utility {book2txt} that attempts to convert existing
documentation in the Yacas format into the plain text form suitable for
{txt2yacasdoc.pl}. After conversion, the plain text docs normally require
further editing.

	    Organization of the Yacas documentation

All documentation source files are kept in the subdirectory {manualmaker/in}.
During compilation, Yacas language files as well as HTML, $LaTeX$ and PS/PDF
files are automatically generated in the {manualmaker} subdirectory.
Contributors should only need to edit files in {manualmaker/in}.

Currently, documentation consists of five "books" (introductory tutorial,
programming tutorial, miscellaneous essays, user's reference manual, and
programmer's reference manual). The first three books are meant as stand-alone
texts, while the last two are "reference" books and are meant to be used
together. The Yacas {Help()} command will show a reference article from either
of the two reference books.

Books may be divided into "chapters", "sections" and "subsections". Reference
documentation contains each command description in a separate "section" which
is marked by a special {*CMD} label (see below). 

Stand-alone books are free-form, but reference books must be written with a
certain template that allows online hyperlinking. The file
{manualmaker/in/dummies} is an example template for a reference manual section.

At the beginning of each book there <i>must</i> be a book title and a short book description (labeled {*BLURB}) which does not appear in printed documentation but only in the HTML top-level book index.

At the beginning of each chapter there may be a "chapter introduction" labeled
{*INTRO} which is also a short description of the contents of that chapter. It
may be one paragraph only. Note that the chapter introduction is only really
useful for reference manual because it is the text that appears at the very top
of a reference manual section. In the HTML version of reference docs, each
chapter contains a list of all functions described in it, and this list goes
right after the first paragraph of "chapter introduction". For the printed (PS/PDF) documentation the "introduction" is indistinguishable from any other text.

	    Using the script {txt2yacasdoc.pl}

The script {txt2yacasdoc.pl} is used to transform plain text markup into the
Yacas language. The script acts as a stream filter:

	perl txt2yacasdoc.pl < file.txt \
	  > file.chapt

In this example, {file.txt} contains some formatted plain text (source text) and the resulting file {file.chapt} will be produced in Yacas-language documentation format.

There is a single option for {txt2yacasdoc}:

	perl txt2yacasdoc.pl -debug < file.txt \
	  > file.chapt
This option is to be used for debugging, i.e. when the resulting file does not
compile in Yacas. The effect of this option is to introduce many more breaks
between text strings in the generated file, so that the {Text()} function is
called more often. It is then easier to locate the source of the problem in the
Yacas-language file (Yacas will tell you the last line in the Yacas-language
file at which a syntax error occurred). This option is largely obsolete because
the {Text()} function is called frequently enough by default.

	    Formatting of source text files

Formatting of source text files uses TAB symbols; if your editor does not support them and converts them to spaces, you should convert the results back to contain real TAB symbols using the standard Unix {unexpand} utility or a custom perl script.

You may want to examine the source of this file ({manualmaker/} {in/}
{YacasDocs.chapt.txt}) to see how various features of the markup are used.
Currently the following markup is implemented:

*	Paragraphs are separated by blank lines (lines consisting of space or TAB characters or empty). Several blank lines next to each other are equivalent to one. However, TAB-indented blank lines inside a code example (see below) do not create another paragraph.

*	Book heading is quadruple TAB indented. Chapter heading is triple TAB indented. Section heading is double TAB indented. Subsection heading is indented by a TAB and 4 spaces. Chapter and section headings should be in one line (but that line can be as long as needed).

*	Sample code is single TAB indented, for example:
	In> 1+2;
	
	Out> 3;
Note that empty lines in a single sample code block must be also TAB indented
or else the sample code block will be split into several sample code
paragraphs. Text immediately following a sample code block will not be made
into a separate paragraph (this currently affects paragraph indentation in
printed docs). A sample code block may be separated from the text that follows
it by an empty line.

*	If a section or chapter heading immediately follows a sample code block
(i.e. when it is the last code sample in the previous section), they
<i>must</i> be separated from the headings by an empty (unindented) line. The
reason for this is that the script will assume that everything which is at
least single-TAB indented (up to a separation between paragraphs) belongs to
one sample code block. This makes it easier to enter multiply indented sample
code: a double-TAB indentation inside a sample code block will not start a new
section. For example:
	While(x<0) [
		x:=x+1;
		Write(x);
	];

*	Ordinary text must not be indented at all. Line length is arbitrary and
linebreaks
inside
a
paragraph
are of no significance to the resulting documentation. The first symbol on a
line should not be an asterisk ({*}) because it is reserved for markup
purposes. In most cases it is okay to have an asterisk in the first position,
though, as long as it does not conflict with other markup features.

*	Itemized text is marked by "*" in the first position, followed by TAB. For example:

	*	Item
	*	Another item

*	Enumerated text is marked by "*" followed by TAB, number and period. The number used in the source text is irrelevant because the enumerated environment of the final document will introduce its own counter. For example:
	*	0. First item
	*	0. Second item
This will produce:
*	0. First item
*	0. Second item

Note that the text of an item continues until the next itemized line is given or until end of paragraph (does not have to be all in one line).

Nesting of enumerated or itemized environments is not supported, except for a fringe case of nesting just one itemized list at the very end of an enumerated list or vice versa.

Enumerated environment is currently only implemented in $LaTeX$ docs; HTML docs render them as itemized.

*	Emphasized text (<i>italics</i>) should be surrounded by {<i>}
{</i>}. Note that the whole emphasized fragment of text must be located within a single line, or else it will not be emphasized.

*	{Typewriter font} text is surrounded by braces {{}}. The typewriter font
fragment must be within a single line and may contain no more than four nested
sets of {{}} inside. This is hopefully enough for our documentation. A
limitation of this markup is that there is no way to put a single brace in text
alone without a matching brace. This would be okay for HTML docs but it breaks
$LaTeX$ docs because braces are special in $TeX$ and because Serge was too lazy
to implement a real parser.

*	Web hyperlinks are surrounded by {<*}
{*>}. Text of the link must begin with {http://} or {ftp://}. Alternatively, a hyperlink with anchored text is made by the markup {<*}
<i>anchored text</i>{|}<i>Web URL</i>{*>}. For example:
	<*http://host.net/file.html#anchor*>
or
	<*click here|somewebpage.html*>


*	Mathematical expressions should be typed in Yacas syntax (not in $TeX$
notation) and surrounded by dollar signs. Both delimiting dollar signs
<i>must</i> be within one line of text. For example: {$}
{x^2 + y^2 != z^2 $}
produces $x^2+y^2 != z^2$. Double dollar signs will denote a displayed
equation, like in $TeX$; both pairs of dollar signs should still be within one
line. Any punctuation immediately following the second pair of dollar signs will be displayed on the same line. (This is used to get around the limitation of mathematical expressions that cannot end with a comma or a period or with another punctuation mark.) For example, the formula "{$}{$}{x^2/2}{$}{$,}" will produce $$x^2/2$$, with the comma on the same line. A formula such as "{$}x+1;{$}" will generate an error; the semicolon should be moved out of the dollar signs.

As special exceptions, one can enter the symbols "$TeX$" and "$LaTeX$"
as if they are Yacas symbols, i.e. "{$T}{eX$}" produces "$TeX$". One can also enter the name {Yacas} as "{{Yacas}}".

Mathematical expressions <i>must</i> be valid Yacas expressions, with balanced
parentheses, no hanging periods and so on, or else the Yacas script that
formats the docs will fail! (This limits the scope of mathematical formulae but
is hopefully not critical.) Currently, when creating online HTML documentation,
mathematics is kept in Yacas notation and set in boldface font. (This may
change in the future.) Of course, $LaTeX$ typesets printed documentation with
correct mathematical symbols.

Another feature of the $LaTeX$ exporter is that it will try to show all
functions and infix operators according to their mathematical meaning, and if
no such meaning is defined in Yacas, then it will show them exactly as they are
written in Yacas. For infix operators to work, they have to be declared in the
standard library, or else an error will occur when processing the manual. For
example, Yacas operators {=} and {==} are both represented by and equals sign
"=", the operator {:=} becomes "identically equal" ($a:=b$), and the cosmetic
operators {<>} and {<=>} become $a<>b$ and $a<=>b$. But you cannot use an
undefined infix operator such as "{:=*}".

*	To include contents of another document, use the {*INCLUDE} directive at
the beginning of a line, e.g.

	*INCLUDE ../essays/howto.chapt

Note that the included document must be a Yacas-language file. (This will become the {IncludeFile()} call in the Yacas document -- an alias to {Load()}.)

*	Comments may be introduced by the directive {*REM}. The line and the paragraph of text following {*REM} will be omitted from the documentation. An empty line should separate the {*REM} block from other text. For example,
	*REM this is
	a comment
	
	(documentation text continues)


	    Formatting text for the reference manual

The formatting explained in the previous section is enough to create most of the user guide and tutorial documentation. The script {txt2yacasdoc.pl} implements some additional markup features to help create reference manual sections.

The additional elements all start with an asterisk "{*}" in the first position on a line, followed by an uppercase keyword. A typical reference manual subsection documenting a certain function may look like this in plain text:

	*CMD PrintList --- print list with padding
	*STDLIB
	
	*CALL
	{PrintList}(list)
	{PrintList}(list, padding);
	
	*PARMS
	{list} -- a list to be printed
	{padding} -- (optional) a string
	
	*DESC
	Prints {list} and inserts the {padding} ...
	
	*E.G.
	
		In> PrintList({a,b,{c, d}}, " .. ")
		Out> " a ..  b .. { c ..  d}";
	
	*SEE Write, WriteString
Compare this with the reference manual section on the function {PrintList} to see how this plain text markup is rendered in the finished documentation.

Notes:
*	Some labels have parameters and some other labels do not; labels that do not have parameters must be put alone on a line.
*	The {*STDLIB} label is for functions defined in the standard library and the {*CORE} label is for built-in functions defined in the Yacas core engine. In addition, {*UNIX}, {*MSWIN} and {*MAC} can be used to label Unix, MS Windows and Macintosh-specific add-on functions.
*	Use of TAB characters is not mandatory between "{*LABEL}" and the following text; ordinary spaces work as well. There must be <i>some</i> whitespace separating the label and the text, however.
*	The comma-space combination "{, }" is mandatory in label argument lists (but can have several spaces after a comma). Lists of commands are used by the {*CMD} and {*SEE} labels.
*	Characters {<} and {>} are of special significance to both HTML and $TeX$ and should be always escaped by braces {{}} or by dollar signs.
*	In the "examples" section, there may be just one example, in which case the "{*EG}" could be used instead of "{*E.G.}". This will generate the word "Example:" instead of "Examples:" in the documentation.

In a subsection there may be either one function documented or several at once: for example, it may make sense to document {Sin}, {Cos} and {Tan} together. In this case, all function names should be simply listed in the {*CMD} header, for example:

	*CMD Sin, Cos, Tan --- Trigonometric ...

In addition, there is a tag {*INTRO} to denote a "reference chapter
introduction" corresponding to the {ChapterIntro()} function, {*BLURB} for book
introduction (only enters the HTML book index but not $LaTeX$ docs), {*A} which
creates an HTML link to a manual section corresponding to
{AddBody(HtmlAnchor()...)}, {*AA} which creates {AddAnchor()}, and {*HEAD}
which creates a small heading ({Topical()}). For instance,
	*PARMS
results in the same text as
	*HEAD Parameters:
Usage of the {*A}, {*AA}, {*HEAD} tags is for compatibility only and should not
be necessary. The {*CMD} tag generates all necessary HTML anchors.

The {*INTRO} and {*BLURB} tags only work for one paragraph. There must be no
empty line between {*INTRO}/{*BLURB} and that paragraph. Also, there must be no
empty line between the "blurb" and the book title (for technical reasons).
There must be one and only one "blurb" paragraph in a "book" and no more than
one "chapter intro" paragraph per chapter.

This markup should be sufficient for creating reference documentation in plain text.

If a manual book does not compile after you made some changes, examine the
Yacas-language files ({*.chapt}, {*.book}), find the line at which an error
occurred, and try to see what happened. Most probable causes of error are:
*	unbalanced brackets, parentheses, or dollar signs;
*	expressions which should not be split between lines but were accidentally split after reformatting paragraphs;
*	mathematical expressions containing an invalid Yacas operation, for example, a trailing period; e.g. you cannot say {$}{x+a.$} -- the period should go outside dollar signs.

	    {book2txt}: Conversion of existing documentation to plain text

(Note: as of version 1.0.49, all Yacas documentation is converted to plaintext format. This section is left for reference only.)

Currently, most but not all of the Yacas documentation markup functionality is
implemented in the simple plaintext filter; also, documentation includes some
extra HTML files. However, almost all of the reasonable markup needed to write
documentation is present. Therefore it is possible to maintain most of the
documentation in the plain text format described above. To convert existing
Yacas documentation back to the plain text format, a script
{book2txt.ys}/{book2txt.sh} can be used.

By using a command such as
	book2txt.sh file.chapt
one can create a source text file {file.chapt.txt} corresponding to the Yacas documentation file {file.chapt}. For example:

	12:51pm scriabin> book2txt.sh intro.book
	[editvi.ys] [gnuplot.ys] 
	True;
	Out> True;
	Quitting...
	File 'intro.book.txt' was created.
	12:51pm scriabin>

In the above example, the shell commands in {book2txt.sh} executed the following Yacas commands,
	Use("book2txt.ys");
	ToFile("file.chapt.txt")
	  Load("file.chapt");
This requires that the Yacas script {book2txt.ys} be available in the current directory. The shell script {book2txt.sh} assumes that {book2txt.ys} is stored in the same directory as {book2txt.sh}.

Of course, it is possible that some features of Yacas documentation were not implemented in the script and in that case the resulting file must be edited by hand. But the purpose of the {book2txt} script is exactly this: to make a plain text source file to be edited and maintained.

Several files can be converted at once, for example:
	book2txt.sh f1.chapt f2.chapt file3.book
Each file is processed by an independent Yacas session. Any errors of processing are printed on the screen.


	    {book2TeX}: preparing typeset documentation

The script {book2TeX.sh} prepares a $TeX$ file out of a Yacas-language documentation book. Usage is similar to {book2txt.sh}, except that only one file is processed at a time. For example:
	book2TeX.sh intro.book intro.book.tex
will create a $TeX$-formatted version of the introductory tutorial. It can be processed with standard tools, for example
	latex intro.book.tex
	latex intro.book.tex
	dvips -o intro.book.ps intro.book dvi
will prepare a Postscript version, while
	latex intro.book.tex
	pdflatex intro.book.tex
will prepare a PDF version.


The shell commands in {book2txt.sh} execute the following Yacas commands:
	Use("book2TeX.ys");
	ToFile("file.chapt.tex")
	  Load("file.chapt");
This requires that the Yacas script {book2TeX.ys} be available in the current directory. The shell script {book2TeX.sh} assumes that {book2TeX.ys} is stored in the same directory as {book2TeX.sh}.

Not all features of Yacas documentation are compatible with $TeX$ typesetting. To prevent errors, documentation should avoid certain things. In general, it is a good idea to check the typeset appearance of documentation, since it helps detect errors.

For example, the symbols {%}, {{ }}, {< >}, {#}, {\}, {_} and {&} are special to $TeX$. They should not normally be used in plain text; it is okay to use them in "typerwriter" text or code samples -- but <i>not</i> in section or chapter heads, because it makes it difficult to export to $TeX$ correctly. $TeX$ commands may be entered but will not be correctly rendered in HTML online documentation. Sometimes fixed-font text will hang over the right edge; a workaround is to break the fixed-font text into shorter fragments or to rephrase the text.

Another concern is that code examples are typeset in a fixed-width font and may not fit into the width of the page and will be hanging over it. To avoid this, the code in the code examples should not be longer than about 44 characters.

The current implementation uses a "report" style which allows chapters, sections, subsections and includes an automatically generated table of contents. ($LaTeX$ is automatically run twice to produce the TOC when executing {make texdocs}.)

Some complicated mathematical expressions may not correctly render in $TeX$.
This is because Yacas uses its internal function {TeXForm()} to transform Yacas
expressions to $TeX$.  Mathematical expressions are entered in the plain text
documentation source using Yacas syntax, then transformed to a special
non-evaluating call {TeXMath()} in the Yacas-language documentation, which
formats into HTML using a {Write()} call or into $TeX$ using a {TeXForm()}
call, as necessary. Testing should be performed on documentation before
releasing it. The most stringent limitation is that the expression between
dollar signs should evaluate in Yacas (perhaps to itself) and not cause syntax
errors. In case of doubt, try whether the expression evaluates without errors
and then try to use {TeXForm} on that expression and see if that evaluates
without errors.

	    {book2ys}: extracting Yacas code from books

{book2ys.sh} is a shell script that extracts Yacas code examples from a documentation chapter into a separate file. Usage is similar to {book2TeX.sh}. For example, the benchmarking test code {wester.yts} is automatically extracted from the corresponding essay chapter by the command
	sh ../manmake/book2ys.sh wester-1994.chapt
After this command, the file {wester-1994.ys} is created. Note that {wester-1994.chapt} is in Yacas language and is itself a generated file.

Using this script, one can write documentation and code together, a la "literate programming".

To prepare a documentation chapter in such a way that code extraction is possible, one only needs to make sure that all code examples in the chapter will become a correct sequence of Yacas expressions when cut out and written sequentially into a file. So, for instance, semicolons at the end of each statement are required. The script {book2ys} will <i>not</i> export example Yacas session code with "{In>}" and "{Out>}" prompts but it will export all other example code.

See the source file {wester-1994.chapt.txt} to get a feeling of how the source documentation is formatted to allow completely automatic code extraction. Note that the printed documentation will be in twocolumn format, and therefore it is necessary to split lines that are too long.

