/ elang / tools

What's Updoc? by Terry Stanley & Mark Miller

Running Updoc & Elmer

This chapter is an example of an Updoc script that tests part of the java.util.Vector class. It is also an example of what makes Updoc such a useful utility. The test cases defined in this file ran successfully with some earlier version of the system. The question now is: Do the tests still succeed or do they fail because they no longer exercise the correct behavior?

Let's find out. Let's ask Updoc to process this file and tell us if the test cases (i.e., E code snippets) pass or fail. (Though please see "Security Considerations" below before running this or any other Updoc script.)

In bash or MSDOS shell (or most other shells), you can do this by entering the command updoc.e followed by the URL for this web page. Enter this after the shell's prompt, shown below as a "$".

$ updoc.e http://www.erights.org/elang/tools/updoc.html

http://www.erights.org/elang/tools/updoc.html:.............

The "$" represents your shell's prompt, whatever that may be. The rest of the first line is what you type into your shell. Depending on your configuration (see (*** need a link)), you may need to say "rune updoc.e" rather than just "updoc.e".

Following your command line, you see Updoc's output. First, there's a blank line. This is followed by the name of the file that was processed. Test cases that pass are shown as dots ('.'). (This is a simple visual indication that Updoc is still cranking away. When the input is very large and most tests succeed, the dots can be reassuring.) The report for failing test cases includes the expression that failed and failure details.

*** currently, Elmer no longer works ***

If you copy the contents of this chapter into a plain text file, let's say, "updoc.txt", you can interactively read it in Elmer. Elmer allows you to try out these examples -- or variants -- as dictated by your curiosity. You run Elmer by typing

$ elmer.e updoc.txt

Elmer is a simple, Notepad-like text editor with one extra feature: If you type the Enter key at the end of a line beginning with a question mark (question mark is the E prompt), this line is evaluated as a line of E script, and the resulting output shown. For example:

? pragma.syntax("0.8")

? 2 + 3
# value: 5

Try changing the example, putting the cursor at the end of the line, and typing the Enter key. When you want to stop interacting with E, just backspace over the new question mark. You are now just editing text again.

Example: Exploring Vectors

To begin with, we need a sample Vector. Vectors can be instantiated with the no-argument constructor, which we write in E as follows:

? def v := <unsafe:java.util.makeVector>()
# value: Vec[]

In E, a Vector prints by printing its elements separated by commas, enclosed by square brackets, and prefixed by "Vec". Therefore "Vec[]" is how an empty Vector prints. You can add elements to the end of a Vector like so:

? v.addElement("foo")
? v.addElement(3)
? v
# value: Vec["foo", 3]

A Vector has both a size and a capacity:

? v.size()
# value: 2

? v.capacity()
# value: 10

The size is how big it "really" is, ie, how many elements it holds. The capacity is a behind-the-scenes implementation concept: it says how big the size can get before the Vector implementation has to do an expensive reallocate-and-copy operation.

You can also place values at particular positions within the Vector:

? v.setElementAt("bar", 1)
? v
# value: Vec["foo", "bar"]

? v.setElementAt("baz", 4)
# problem: <ArrayIndexOutOfBoundsException: 4 >= 2>

? v.setSize(5)
? v.setElementAt("baz", 4)
? v
# value: Vec["foo", "bar", null, null, "baz"]

As you can see, you can only place values in the first "size" positions of the Vector, independent of capacity. However, you can always expand the Vector first using "setSize" to ensure that it contains the position of interest. "setSize" will initialize any new positions it creates with null.

Executable Documentation and More...

This vector example demonstrates the point of Updoc. Updoc enables this one chapter to simultaneously serve four purposes.

• The chapter serves as checkable documentation about the Vector class, where you can always check whether the embedded code examples are accurate. When they aren't, please complain!

• The chapter serves as a regression test. Brian Marick documents (*** need link) that regression tests normally degenerate over time as a result of maintenance. To keep the tests running as the software is changed, the purpose of the test is often lost, and the test is gradually changed into one that always passes. When the regression tests are embedded examples in readable documentation, the test-documentation combination would be rendered noticably incoherent if the test were to become meaningless.

• With this chapter, you can do experimental reading. When reading a code example in a programming manual, how often have you thought "But what if we try it this other way?" By bringing an Updoc chapter into Elmer, the text serves as a "user interface" for interactively poking at the API you're reading about, and trying out hypothetical variants of the examples shown.

• This chapter demonstrates interactive, or even adversarial writing. I wrote the first draft of this chapter, not by intending to write a chapter on Java's Vector class, but simply because I was curious about the meaning of Vector's "size()" vs "capacity()", so I brought up Elmer to try these out. By saving this Elmer transcript, I had my first draft. This is interactive writing. When writing test cases, we should be trying to creatively break the module being tested, by trying edge cases or whatever. A live Elmer session gives us a sense of a live adversary to break. The transcript of the resulting session will often be a great first draft of the needed chapter/regression-test of the module. This is adversarial writing.

The Command Line

updoc.e options... files...

The following options are planned, but not yet supported. Currently the output is always directed to stdout.

-o outputFileName	Write the test results to the output file instead of stdout
--help	Prints usage information
--quiet	Prints problems but not successes

Following "updoc.e" you give a list of URLs, file names, and directory names for it to check. If you give it a directory name, it will check all relevant files in that directory tree. Files not considered relevant are ignored. Updoc considers a URL or file name relevant if it ends in one of the extensions it recognizes. Updoc currently recognizes the following extensions:

filename.updoc filename.emaker filename.txt	Process the file as plain text
filename.html filename.htm	Extract text from HTML
directoryName	Process all relevant files in the directory tree.
url.updoc url.emaker url.txt	Process the web page as plain text
url.html url.htm	Extract text from HTML

Updoc Input Syntax

Updoc parses plain text files, looking for an expression to evaluate. A line whose first non-whitespace character is "?" is the first line of an expression. If the expression spans multiple lines, the first line starts with "? " and all subsequent lines start with "> ". Leading and trailing spaces are discarded.

Next, the parser looks for an answer: the result returned by the E interpreter when the test script was written. These are the original answers. An answer starts with "# keyword: value". If the answer spans multiple lines, all lines start with "# ". A blank line follows the last line of an answer. Some answers will contain run-dependent output, such as a stack trace. Updoc recognizes this output and ignores it. A single expression can have multiple answers.

Updoc Output

After parsing the input, we have a list of test cases. Each test case has an expression and its corresponding (original) answers. The expression is parsed and evaluated to produce new answers. The new answers are compared to the original, with five possible outcomes:

• New matches original -- success!
• New answer is missing
• Original answer is missing
• New answer and original answer differ
• Syntax error: the input expression didn't successfully parse.

The output starts with the file name. For each successful test case, a dot is shown. For the others, the original input expression and an explanation of the problem are shown. In the case of a syntax error, the rest of the test cases in this test script (this individual Updoc file) are skipped -- since they may not be meaningful in the absence of evaluating that expression -- and Updoc proceeds to the next test script.

Security Considerations

Danger Danger Warning Warning

Updoc and Elmer by default run Updoc scripts with all their user's authority -- just as *.e scripts are normally run. Actually, the situation is currently worse than that -- this default is the only option currently provided by these programs. This means that the decision to run an Updoc script must be taken exactly as seriously as the decision of run a *.e script, which is to say, as seriously as the decision to install a conventional program on your computer.

Future Directions

Bugs & Other Correctness Issues

Platform dependence. While Updoc itself is as platform independent as E (which, in turn, inherits Java's platform independence), Updoc scripts may be as platform dependent as E or Java programs. Worse, Updoc scripts-as-documentation tend toward platform dependence, since documentation (especially for beginners) needs concrete examples, that, for example, uses concrete file names on the file system. It would be hard to rewrite the chapter "Example: Finding Text" from the E tutorial to be platform independent while being just as readable. This issue is best addressed while also addressing the above security issue, since the solutions will share much mechanism. Until this issue is fixed, the existing Updoc test scripts in the E source tree and on the erights.org website make no claims of platform independence.

Internal modularity. As with many E programs, Updoc was prototyped as one long *.e file ("updoc.e"). Following this prototyping phase, most of the logic of an E program should be moved into *.emaker files, leaving the *.e file as a small readable script that does little but evaluate the EMakers and authorize (grant least authorities to) their instances. This allows one to understand what authority may be used (or abused) by a utility by reading the (hopefully small) *.e file, without needing to read the *.emaker files. (**Move this explanation somewhere else and link to it.) Updoc has not yet entered this post-prototyping phase -- it has not yet been separated into *.emaker files.

We need to implement the command line options explained above.

Which interpreter?. Currently, each test script is evaluated in an environment (scope) in which "interp", "stdout", and "stderr" are bound to the same values as for Updoc itself. Instead, each test script should be given a new interpreter, new output streams, and a new instance of the trace system. The output sent to the streams and to this trace instance by each test case should be treated as additional answers produced by that test case, to be compared along with the others.

Delaying test cases. The "interp" object seen by an Updoc test case does not yet support the messages "blockAtTop()" and "continueAtTop()". These are currently supported by the E interpreter, but need to be supported by Updoc as well, so that a test script can pause between a test case that is supposed to cause some effect to eventually happen and a test case written assuming the effect has happened (such as a promise being resolved).

Nesting exiting. The "interp" object seen by an Updoc test case also needs to support the message "exitAtTop(exitCode)", as does the normal E interpreter. E programs should call this, rather than "System.exit(exitCode)", in order to terminate early. By using "interp.exitAtTop(exitCode)", an E program being run by another E program (as with an Updoc test script) may exit its nested interpreter without causing the JVM to exit. (** Put this issue into documentation of the normal E interpreter.) ( Mostly fixed. )

Updoc's own exitCode. Updoc should remember whether any problems were encountered during an Updoc run, or if instead all the test cases passed. Only if all the cases passed should Updoc exit with an exitCode of 0. Otherwise it should exit with a non-zero exitCode, indicating a problem. This would allow shell scripts and makefiles to test or stop on a test failure. (Currently, Updoc always returns an exitCode of 0.) Of course, once the E interpreter supports exitAtTop(exitCode), Updoc should use it for this purpose.

An Updoc syntax error report doesn't yet conform to Updoc's output format. In addition, because of the order in which Updoc processes a test script, if a script contains a syntax error, no feedback from earlier test cases in that script will be seen.

Line continuation. Updoc should understand "\" at the end of a line in the original answer to mean, ignore the following newline. This would let answers in documentation (such as the ClassCastException in "Defining Variables") to be broken into shorter lines for readability.

Coming Attractions

Pattern matching. Sometimes the output from one run to the next differs in inconsequential ways. Sometimes the output contains more detail than needs to be shown, and would make the text script less readable as documentation. For both these purposes, Updoc should understand alternate keywords for the original answers, to signal that the text of the original answer is not literal text, but rather is a pattern to be matched against the corresponding new answer. For example, in "Iterating by Key-Value Pairs", the test case which defines the variable "capitals" shows an answer with keyword "match:". This should match against an actual "value:" answer by treating the "..." in the match-answer as a wildcard.

Elmer integration. For each test case that fails you probably want to find the file it's in, find the expression that failed, and fix the problem. Interactively locating and fixing problems in scripts will be supported when Updoc's functionality is integrated into Elmer.

Interface coverage. An individual Updoc file will often be an explanation of the API for a given abstraction, or a closely related set of abstractions. Chapters in programming manuals often begin with a broad statement of "What will be covered in this chapter", and end with a detailed "What was covered in this chapter". We expect Updoc to provide testable versions of these. For example, this chapter on the Vector class might begin with something like

"? def coverage := covering([<unsafe:java.util.Vector>])"

This both tells the reader what abstraction is being covered, and intitializes internal coverage flags, which are already present in ELib for this purpose. At the end of this chapter might be a corresponding

"? coverage.covered()"

whose value would say what portions of Vector's public interface was and wasn't exercised during the running of this test script. This both tells the user what's been demonstrated and left out by this chapter, and, because this coverage report would be shown as an Updoc answer, it is automatically rechecked for accuracy.