# For Yacas developers

##

A crash course in Yacas maintenance for developers

This document intends to give a concise description of the way Yacas is maintained. There are a few parts to maintenance to take into account:

- The `autoconf`/`automake` part -- makefile maintenance over various systems.
- The `cvs` system -- enabling developers to work together.
- The back up repository -- storage of tarballs with versions of Yacas. This can be found at the following address:

http://www.xs4all.nl/~apinkus/backups/

###

The `autoconf`/`automake` system

The important thing to remember with the autoconf/automake system is that there are files enumerating the files and directories that are part of the distribution. The `Makefile.am` files contain the lists of files that are part of the distribution, and are in the respective directory. If a file is added by a developer, but it is not added to the list in the file `Makefile.am`, it will not be added to the tar ball which is uploaded to the back up repository. It has the nice side effect that you can have local files which don't automatically get added to the distribution.

Adding a directory requires adding the directory name to various places also, in configure scripts etcerera.

###

Maintaining Yacas through a `cvs` repository

`CVS` provides an efficient way for developers to work together, automatically merging changes various developers make, and at the same tile is a back up system (uploading your changes to another computer from which you can easily obtain it at a later time). After a little effort setting it up it becomes very easy to use. This section describes the few commands needed for keeping your version and the version in the Yacas repository up to date.

###

How does `cvs` work?

CVS has a copy of the files in the repository somewhere in a directory on some system, possibly your computer. Then there is such a thing as a cvs server which you can talk to to synchronize your version of the source code with the version on the server.

CVS uses a diff-like scheme for merging differences: it looks at two text files, determines the different lines, and merges accordingly. It discovers the changes you made by looking at the version you checked out last and the version you have now, to discover which lines changed (it maintains an automatic version number for each file).

If the version of a file on your system and the version in the cvs repository has a line that has been changed by both you and some one else, the cvs repository will obviously not know what to do with that, and it will signal a 'collision' which you will have to solve by hand (don't worry, this rarely happens). More on that later.

In commands to be described in this document are in short:

- `cvs checkout` will get you an initial version. You only need to call this once.
- `cvs update` will merge the two versions and put it on your computer, so you have the latest version.
- `cvs commit` will merge the two versions and put it in the cvs repository.
- `cvs add` to add a file or directory.
- `cvs remove` to remove a file.

###

Checking out an initial version of Yacas

There are two ways to check out a version of Yacas: as anonymous user and as maintainer. Anonymous users don't need to log in, but also have no right to commit changes. Maintainers first need to get an account (at sourceforge), and their account needs to be enabled so they are allowed by the maintainer to make changes. A maintainer needs to log in with every command. To be able to log in, you need ssh1 installed (ssh2 will not work). You can find this at http://www.ssh.org/download.html.

To check out Yacas as anonymous user, type:

    cvs -d:pserver:anonymous@cvs.yacas.
      sourceforge.net:/cvsroot/yacas login
    cvs -z3 -d:pserver:anonymous@cvs.yacas.
      sourceforge.net:/cvsroot/yacas co yacas

To check out as a maintainer, type:

    export CVS_RSH=ssh1

This will tell CVS to use ssh1 for communication. Then, in order to get yacas, type

    cvs -d:ext:loginname@cvs.yacas.sourceforge.
      net:/cvsroot/yacas co yacas

where `loginname` is your name on the sourceforge system. This then creates a directory `yacas/` with the full most recent distribution. You need to enter your password there, but other than that, that's it!

Those lines typed above are long and intimidating, but it is also the last time you need to type them. From now on, if you want to do anything with cvs, just go into the `yacas/` directory you just checked out, and type the cvs command without the `-d`... flag. This flag just tells cvs where to find the repository. But future cvs commands will know where to find them, which is why you don't need that flag.

###

Use case scenario 1 : getting the latest version of Yacas

You haven't looked at Yacas for a while (shame on you!) and want to check out the latest version. Just type

    cvs update -d

on the command line in the `yacas` directory, and that should essentially download the latest version for you in that directory (just the changes). The `-d` option here states that you are also interested in new directories that were added to the repository. Oddly enough, cvs will only get you changed and added files, not added directories, by default.

A command

    cvs -q update -d

will print messages only about changed files.

###

Use case scenario 2 : you made changes to Yacas

You got the latest version, but saw this huge, glaring omission in Yacas, and start hacking away to add it yourself. After a while, after playing with the code you wrote, and if you think you are finished with it, you decide you like to add it to the cvs repository. First, you added a few files, which you need to add:

`cvs add` [list of file names of ascii text files]

which adds ascii text files. If you added binary files (GIF images in the documentation directory, or something like that), you can add it with

`cvs add -kb` [list of file names of binary files]

Note that, when adding files to the CVS, you should normally also add them to the distribution list. This is done by adding the file name to the `EXTRA_DIST` variable in the file `Makefile.am` in the directory where you adding the file.

In case files need to be removed, there are two options:

- The file is still on your drive: call `cvs remove -f [filename]` to remove the file from both your computer and the cvs repository.
- You already removed the file from your system: call `cvs remove [filename]`

There seems to be no easy way to rename or move files; you would have to remove them at their old location and add them at a new location.

Now, when finished with that, you might want to 'commit' all changes with

    cvs commit

If the commit succeeds, an email is sent out to the maintainers, who can then scan the diff files for changes, to see if they agree with the changes, and perhaps fix mistakes made (if any).

If there is a collision, the commit fails (it will tell you so). In case of a collision, you need to invoke `cvs update` twice. The `cvs update` outputs a list of file names with a character in front of them. The important ones are the files with a 'C' before them. They have a collision. You can go into the file, and see the collision, which the cvs system conveniently marks as:

    <<<<<<
    old version
    ===========
    new version
    >>>>>>

You can edit the file by merging the two versions by hand. This happens very rarely, but it can happen. Use `cvs commit` afterwards to commit.

The `commit` and `update` commands can be performed in specific directories, and on specific files, if necessary, by stating them on the command line. Or you can go into a sub directory and do a `cvs commit` or `cvs update` there, if you are confident that is the only place that changed or whose changes you are interested in.

That is basically it, a quick crash course cvs. It is actually very convenient in that usually all that is needed is a `cvs commit` to fix small bugs. You type that in, and your version gets merged with the changes others made, and they get your changes, and you backed up your changes at the same time (all with that little command!).

You can find more information about cvs at http://cvsbook.red-bean.com/ .

##

Preparing and maintaining Yacas documentation

###

Introduction

Yacas documentation in HTML and PS/PDF formats is generated by Yacas scripts from Yacas source files. However, it is cumbersome to write those source files in the Yacas language. The script `txt2yacasdoc.pl` makes it possible to create and maintain the documentation in an easy-to-read form.

The "source" form of all documentation is maintained in a special plain text format. The format is such that it is clearly readable without any processing and is easy to edit. To compile the documents, the system processes the plain text docs with a script to prepare Yacas-language files and then runs the Yacas scripts to produce the final documentation.

The source text must be formatted in a certain fashion to delimit sections, code examples, and so on, but the format is easy enough to enter in a plain text editor. Text is marked up mostly by TAB characters, spaces, and asterisks "`*`" at the beginning of a line. The script `txt2yacasdoc.pl` converts this markup into Yacas code.

There is also a utility `book2txt` that attempts to convert existing documentation in the Yacas format into the plain text form suitable for `txt2yacasdoc.pl`. After conversion, the plain text docs normally require further editing.

###

Organization of the Yacas documentation

All documentation source files are kept in the subdirectory `manualmaker/in`. During compilation, Yacas language files as well as HTML, LaTeX and PS/PDF files are automatically generated in the `manualmaker` subdirectory. Contributors should only need to edit files in `manualmaker/in`.

Currently, documentation consists of five "books" (introductory tutorial, programming tutorial, miscellaneous essays, user's reference manual, and programmer's reference manual). The first three books are meant as stand-alone texts, while the last two are "reference" books and are meant to be used together. The Yacas `Help()` command will show a reference article from either of the two reference books.

Books may be divided into "chapters", "sections" and "subsections". Reference documentation contains each command description in a separate "section" which is marked by a special `*CMD` label (see below).

Stand-alone books are free-form, but reference books must be written with a certain template that allows online hyperlinking. The file `manualmaker/in/dummies` is an example template for a reference manual section.

At the beginning of each book there must be a book title and a short book description (labeled `*BLURB`) which does not appear in printed documentation but only in the HTML top-level book index.

At the beginning of each chapter there may be a "chapter introduction" labeled `*INTRO` which is also a short description of the contents of that chapter. It may be one paragraph only. Note that the chapter introduction is only really useful for reference manual because it is the text that appears at the very top of a reference manual section. In the HTML version of reference docs, each chapter contains a list of all functions described in it, and this list goes right after the first paragraph of "chapter introduction". For the printed (PS/PDF) documentation the "introduction" is indistinguishable from any other text.

###

Using the script `txt2yacasdoc.pl`

The script `txt2yacasdoc.pl` is used to transform plain text markup into the Yacas language. The script acts as a stream filter:

    perl txt2yacasdoc.pl < file.txt \
      > file.chapt

In this example, `file.txt` contains some formatted plain text (source text) and the resulting file `file.chapt` will be produced in Yacas-language documentation format.

There is a single option for `txt2yacasdoc`:

    perl txt2yacasdoc.pl -debug < file.txt \
      > file.chapt

This option is to be used for debugging, i.e. when the resulting file does not compile in Yacas. The effect of this option is to introduce many more breaks between text strings in the generated file, so that the `Text()` function is called more often. It is then easier to locate the source of the problem in the Yacas-language file (Yacas will tell you the last line in the Yacas-language file at which a syntax error occurred). This option is largely obsolete because the `Text()` function is called frequently enough by default.

###

Formatting of source text files

Formatting of source text files uses TAB symbols; if your editor does not support them and converts them to spaces, you should convert the results back to contain real TAB symbols using the standard Unix `unexpand` utility or a custom perl script.

You may want to examine the source of this file (`manualmaker/` `in/` `YacasDocs.chapt.txt`) to see how various features of the markup are used. Currently the following markup is implemented:

- Paragraphs are separated by blank lines (lines consisting of space or TAB characters or empty). Several blank lines next to each other are equivalent to one. However, TAB-indented blank lines inside a code example (see below) do not create another paragraph.

- Book heading is quadruple TAB indented. Chapter heading is triple TAB indented. Section heading is double TAB indented. Subsection heading is indented by a TAB and 4 spaces. Chapter and section headings should be in one line (but that line can be as long as needed).

- Sample code is single TAB indented, for example:

      In> 1+2;

      Out> 3;

  Note that empty lines in a single sample code block must be also TAB indented or else the sample code block will be split into several sample code paragraphs. Text immediately following a sample code block will not be made into a separate paragraph (this currently affects paragraph indentation in printed docs). A sample code block may be separated from the text that follows it by an empty line.

- If a section or chapter heading immediately follows a sample code block (i.e. when it is the last code sample in the previous section), they must be separated from the headings by an empty (unindented) line. The reason for this is that the script will assume that everything which is at least single-TAB indented (up to a separation between paragraphs) belongs to one sample code block. This makes it easier to enter multiply indented sample code: a double-TAB indentation inside a sample code block will not start a new section. For example:

      While(x<0) [
      	x:=x+1;
      	Write(x);
      ];

- Ordinary text must not be indented at all. Line length is arbitrary and linebreaks inside a paragraph are of no significance to the resulting documentation. The first symbol on a line should not be an asterisk (`*`) because it is reserved for markup purposes. In most cases it is okay to have an asterisk in the first position, though, as long as it does not conflict with other markup features.

- Itemized text is marked by "*" in the first position, followed by TAB. For example:

    *	Item
    *	Another item

- Enumerated text is marked by "*" followed by TAB, number and period. The number used in the source text is irrelevant because the enumerated environment of the final document will introduce its own counter. For example:

      *	0. First item
      *	0. Second item

  This will produce:
  - First item
  - Second item

Note that the text of an item continues until the next itemized line is given or until end of paragraph (does not have to be all in one line).

Nesting of enumerated or itemized environments is not supported, except for a fringe case of nesting just one itemized list at the very end of an enumerated list or vice versa.

Enumerated environment is currently only implemented in LaTeX docs; HTML docs render them as itemized.

- Emphasized text (italics) should be surrounded by `<i>` `</i>`. Note that the whole emphasized fragment of text must be located within a single line, or else it will not be emphasized.

- `Typewriter font` text is surrounded by braces `{}`. The typewriter font fragment must be within a single line and may contain no more than four nested sets of `{}` inside. This is hopefully enough for our documentation. A limitation of this markup is that there is no way to put a single brace in text alone without a matching brace. This would be okay for HTML docs but it breaks LaTeX docs because braces are special in TeX and because Serge was too lazy to implement a real parser.

- Web hyperlinks are surrounded by `<*` `*>`. Text of the link must begin with `http://` or `ftp://`. Alternatively, a hyperlink with anchored text is made by the markup `<*` anchored text`|`Web URL`*>`. For example:

      <*http://host.net/file.html#anchor*>

  or

      <*click here|somewebpage.html*>

- Mathematical expressions should be typed in Yacas syntax (not in TeX notation) and surrounded by dollar signs. Both delimiting dollar signs must be within one line of text. For example: `$` `x^2 + y^2 != z^2 $` produces x^2+y^2!=z^2. Double dollar signs will denote a displayed equation, like in TeX; both pairs of dollar signs should still be within one line. Any punctuation immediately following the second pair of dollar signs will be displayed on the same line. (This is used to get around the limitation of mathematical expressions that cannot end with a comma or a period or with another punctuation mark.) For example, the formula "`$``$``x^2/2``$``$,`" will produce

  x^2/2,
  with the comma on the same line. A formula such as "`$`x+1;`$`" will generate an error; the semicolon should be moved out of the dollar signs.

As special exceptions, one can enter the symbols " TeX" and " LaTeX" as if they are Yacas symbols, i.e. "`$T` `eX$`" produces " TeX". One can also enter the name `Yacas` as "`{Yacas}`".

Mathematical expressions must be valid Yacas expressions, with balanced parentheses, no hanging periods and so on, or else the Yacas script that formats the docs will fail! (This limits the scope of mathematical formulae but is hopefully not critical.) Currently, when creating online HTML documentation, mathematics is kept in Yacas notation and set in boldface font. (This may change in the future.) Of course, LaTeX typesets printed documentation with correct mathematical symbols.

Another feature of the LaTeX exporter is that it will try to show all functions and infix operators according to their mathematical meaning, and if no such meaning is defined in Yacas, then it will show them exactly as they are written in Yacas. For infix operators to work, they have to be declared in the standard library, or else an error will occur when processing the manual. For example, Yacas operators `=` and `==` are both represented by and equals sign "=", the operator `:=` becomes "identically equal" ( a:=b), and the cosmetic operators `<>` and `<=>` become a<>b and a<=>b. But you cannot use an undefined infix operator such as "`:=*`".

- To include contents of another document, use the `*INCLUDE` directive at the beginning of a line, e.g.

    *INCLUDE ../essays/howto.chapt

Note that the included document must be a Yacas-language file. (This will become the `IncludeFile()` call in the Yacas document -- an alias to `Load()`.)

- Comments may be introduced by the directive `*REM`. The line and the paragraph of text following `*REM` will be omitted from the documentation. An empty line should separate the `*REM` block from other text. For example,

      *REM this is
      a comment

      (documentation text continues)

###

Formatting text for the reference manual

The formatting explained in the previous section is enough to create most of the user guide and tutorial documentation. The script `txt2yacasdoc.pl` implements some additional markup features to help create reference manual sections.

The additional elements all start with an asterisk "`*`" in the first position on a line, followed by an uppercase keyword. A typical reference manual subsection documenting a certain function may look like this in plain text:

    *CMD PrintList --- print list with padding
    *STDLIB

    *CALL
    {PrintList}(list)
    {PrintList}(list, padding);

    *PARMS
    {list} -- a list to be printed
    {padding} -- (optional) a string

    *DESC
    Prints {list} and inserts the {padding} ...

    *E.G.

    	In> PrintList({a,b,{c, d}}, " .. ")
    	Out> " a ..  b .. { c ..  d}";

    *SEE Write, WriteString

Compare this with the reference manual section on the function `PrintList` to see how this plain text markup is rendered in the finished documentation.

Notes:

- Some labels have parameters and some other labels do not; labels that do not have parameters must be put alone on a line.
- The `*STDLIB` label is for functions defined in the standard library and the `*CORE` label is for built-in functions defined in the Yacas core engine. In addition, `*UNIX`, `*MSWIN` and `*MAC` can be used to label Unix, MS Windows and Macintosh-specific add-on functions.
- Use of TAB characters is not mandatory between "`*LABEL`" and the following text; ordinary spaces work as well. There must be some whitespace separating the label and the text, however.
- The comma-space combination "`,` " is mandatory in label argument lists (but can have several spaces after a comma). Lists of commands are used by the `*CMD` and `*SEE` labels.
- Characters `<` and `>` are of special significance to both HTML and TeX and should be always escaped by braces `{}` or by dollar signs.
- In the "examples" section, there may be just one example, in which case the "`*EG`" could be used instead of "`*E.G.`". This will generate the word "Example:" instead of "Examples:" in the documentation.

In a subsection there may be either one function documented or several at once: for example, it may make sense to document `Sin`, `Cos` and `Tan` together. In this case, all function names should be simply listed in the `*CMD` header, for example:

    *CMD Sin, Cos, Tan --- Trigonometric ...

In addition, there is a tag `*INTRO` to denote a "reference chapter introduction" corresponding to the `ChapterIntro()` function, `*BLURB` for book introduction (only enters the HTML book index but not LaTeX docs), `*A` which creates an HTML link to a manual section corresponding to `AddBody(HtmlAnchor()...)`, `*AA` which creates `AddAnchor()`, and `*HEAD` which creates a small heading (`Topical()`). For instance,

    *PARMS

results in the same text as

    *HEAD Parameters:

Usage of the `*A`, `*AA`, `*HEAD` tags is for compatibility only and should not be necessary. The `*CMD` tag generates all necessary HTML anchors.

The `*INTRO` and `*BLURB` tags only work for one paragraph. There must be no empty line between `*INTRO`/`*BLURB` and that paragraph. Also, there must be no empty line between the "blurb" and the book title (for technical reasons). There must be one and only one "blurb" paragraph in a "book" and no more than one "chapter intro" paragraph per chapter.

This markup should be sufficient for creating reference documentation in plain text.

If a manual book does not compile after you made some changes, examine the Yacas-language files (`*.chapt`, `*.book`), find the line at which an error occurred, and try to see what happened. Most probable causes of error are:

- unbalanced brackets, parentheses, or dollar signs;
- expressions which should not be split between lines but were accidentally split after reformatting paragraphs;
- mathematical expressions containing an invalid Yacas operation, for example, a trailing period; e.g. you cannot say `$``x+a.$` -- the period should go outside dollar signs.

###

`book2txt`: Conversion of existing documentation to plain text

(Note: as of version 1.0.49, all Yacas documentation is converted to plaintext format. This section is left for reference only.)

Currently, most but not all of the Yacas documentation markup functionality is implemented in the simple plaintext filter; also, documentation includes some extra HTML files. However, almost all of the reasonable markup needed to write documentation is present. Therefore it is possible to maintain most of the documentation in the plain text format described above. To convert existing Yacas documentation back to the plain text format, a script `book2txt.ys`/`book2txt.sh` can be used.

By using a command such as

    book2txt.sh file.chapt

one can create a source text file `file.chapt.txt` corresponding to the Yacas documentation file `file.chapt`. For example:

    12:51pm scriabin> book2txt.sh intro.book
    [editvi.ys] [gnuplot.ys]
    True;
    Out> True;
    Quitting...
    File 'intro.book.txt' was created.
    12:51pm scriabin>

In the above example, the shell commands in `book2txt.sh` executed the following Yacas commands,

    Use("book2txt.ys");
    ToFile("file.chapt.txt")
      Load("file.chapt");

This requires that the Yacas script `book2txt.ys` be available in the current directory. The shell script `book2txt.sh` assumes that `book2txt.ys` is stored in the same directory as `book2txt.sh`.

Of course, it is possible that some features of Yacas documentation were not implemented in the script and in that case the resulting file must be edited by hand. But the purpose of the `book2txt` script is exactly this: to make a plain text source file to be edited and maintained.

Several files can be converted at once, for example:

    book2txt.sh f1.chapt f2.chapt file3.book

Each file is processed by an independent Yacas session. Any errors of processing are printed on the screen.

###

`book2TeX`: preparing typeset documentation

The script `book2TeX.sh` prepares a TeX file out of a Yacas-language documentation book. Usage is similar to `book2txt.sh`, except that only one file is processed at a time. For example:

    book2TeX.sh intro.book intro.book.tex

will create a TeX-formatted version of the introductory tutorial. It can be processed with standard tools, for example

    latex intro.book.tex
    latex intro.book.tex
    dvips -o intro.book.ps intro.book dvi

will prepare a Postscript version, while

    latex intro.book.tex
    pdflatex intro.book.tex

will prepare a PDF version.

The shell commands in `book2txt.sh` execute the following Yacas commands:

    Use("book2TeX.ys");
    ToFile("file.chapt.tex")
      Load("file.chapt");

This requires that the Yacas script `book2TeX.ys` be available in the current directory. The shell script `book2TeX.sh` assumes that `book2TeX.ys` is stored in the same directory as `book2TeX.sh`.

Not all features of Yacas documentation are compatible with TeX typesetting. To prevent errors, documentation should avoid certain things. In general, it is a good idea to check the typeset appearance of documentation, since it helps detect errors.

For example, the symbols `%`, `{ }`, `< >`, `#`, `\`, `_` and `&` are special to TeX. They should not normally be used in plain text; it is okay to use them in "typerwriter" text or code samples -- but not in section or chapter heads, because it makes it difficult to export to TeX correctly. TeX commands may be entered but will not be correctly rendered in HTML online documentation. Sometimes fixed-font text will hang over the right edge; a workaround is to break the fixed-font text into shorter fragments or to rephrase the text.

Another concern is that code examples are typeset in a fixed-width font and may not fit into the width of the page and will be hanging over it. To avoid this, the code in the code examples should not be longer than about 44 characters.

The current implementation uses a "report" style which allows chapters, sections, subsections and includes an automatically generated table of contents. ( LaTeX is automatically run twice to produce the TOC when executing `make texdocs`.)

Some complicated mathematical expressions may not correctly render in TeX. This is because Yacas uses its internal function `TeXForm()` to transform Yacas expressions to TeX. Mathematical expressions are entered in the plain text documentation source using Yacas syntax, then transformed to a special non-evaluating call `TeXMath()` in the Yacas-language documentation, which formats into HTML using a `Write()` call or into TeX using a `TeXForm()` call, as necessary. Testing should be performed on documentation before releasing it. The most stringent limitation is that the expression between dollar signs should evaluate in Yacas (perhaps to itself) and not cause syntax errors. In case of doubt, try whether the expression evaluates without errors and then try to use `TeXForm` on that expression and see if that evaluates without errors.

###

`book2ys`: extracting Yacas code from books

`book2ys.sh` is a shell script that extracts Yacas code examples from a documentation chapter into a separate file. Usage is similar to `book2TeX.sh`. For example, the benchmarking test code `wester.yts` is automatically extracted from the corresponding essay chapter by the command

    sh ../manmake/book2ys.sh wester-1994.chapt

After this command, the file `wester-1994.ys` is created. Note that `wester-1994.chapt` is in Yacas language and is itself a generated file.

Using this script, one can write documentation and code together, a la "literate programming".

To prepare a documentation chapter in such a way that code extraction is possible, one only needs to make sure that all code examples in the chapter will become a correct sequence of Yacas expressions when cut out and written sequentially into a file. So, for instance, semicolons at the end of each statement are required. The script `book2ys` will not export example Yacas session code with "`In>`" and "`Out>`" prompts but it will export all other example code.

See the source file `wester-1994.chapt.txt` to get a feeling of how the source documentation is formatted to allow completely automatic code extraction. Note that the printed documentation will be in twocolumn format, and therefore it is necessary to split lines that are too long.