/ elang / tools

Causeway Message-oriented distributed debugging by Terry Stanley, Tyler Close, and Mark Miller, from an idea by E. Dean Tribble partially supported by the Critical Infrastructure Protection Project at George Mason University

Causeway, an open source distributed debugger written in E, lets you browse the causal graph of events in a distributed computation.

For current information on Causeway, see

wiki

paper

Screencast

The rest of this page is stale and appears here only for historical interest.

Causeway provides a post-mortem view, gathered from trace files written by the processes you wish to debug. (We explain below how to enable this tracing.) In the upper left pane, each tab represents the full order of events recorded by each process. This gives a "follow the process" view common to conventional distributed debuggers. In the upper right pane, we see an alternative "follow the conversation" outline view, in which each event expands to show the events it causes. Causeway assigns a unique color per vat. As we see, causality flows back and forth between the vats.

Causeway presents several different views of the causal relations. The views are coordinated such that, selecting an item in one view causes corresponding selections in other views.

• Causality Tree View This tree view contains the full graph built from the trace files.

• Pruned Causality Tree View This tree view is built by walking the full graph and applying a relevance function to each causal relation. The relations are either clipped, skipped, or kept, depending on how relevant, or interesting, they seem. Tree pruning is guided by relevance functions and filters. Currently, default options are selected, but we expect to support user-defined functions and filters in a future release.

• Full-order View The view to the left of the tree view lists the events from a single trace file, in full order. This is equivalent to the full order of events sent within a single process.

• Stack Frame View This view is not quite what you would expect. For a selected message in the tree view, the stack shows the state at the time the message was sent, not received. It's the stack of the cause, not the effect. Selecting an item in the stack view shows the corresponding source code.

• Source Code View This view shows the filename and E source code, if available.

Generating causality traces

E supports tracing and there are many options for turning on various kinds of traces, generated at different levels of detail. Tracing can be turned on using command line properties, or by explicit actions within the programs. The example programs "seller.e" and "buyer.e" turn causality tracing on and off in order to bracket just that portion of their computation that would be interesting to view in Causeway. We recommend this technique to avoid being swamped in irrelevant detail.

In the code below, the trace begins with the eventual send to x and ends eventually, sometime after answer is resolved. The trace includes the message sends (effects) caused by x <- question() (the cause) and the message sends caused by these effects, and so on.

# import tracing support
def tcr := <unsafe:org.erights.e.develop.trace.TraceController>

# ...

# turn causality tracing on
tcr.setProperty("TraceLog_causality", "debug")

def answer := x <- question()
when (answer) -> done(_) :any {
    # ...
} catch problem {
    # ...
} finally {
    # turn causality tracing off
    tcr.setProperty("TraceLog_causality", "warning")
}

So that the examples below will be more reusable, first set a shell variable EHOME to the location where you installed E. On MSWindows in the cygwin bash shell, this would typically look like

$ EHOME="c:/Program Files/erights.org"

In a bash shell on a Unixoid system, this would typically be

$ EHOME=/usr/local/e

The shell commands below generates a trace of seller.e in the current directory. (The tracing system is very finnicky about not overwriting files, so if you want to save a trace log to a specific name by using the TraceLog_dir and TraceLog_name properties, be sure to delete any previous trace of that name.)

$ rm -f seller.txt
$ rune -DTraceLog_dir=. -DTraceLog_name=seller.txt "$EHOME/scripts/test/causeway/seller.e"
waiting...

Wait for the seller to print "waiting...", as shown above. Then, in a separate shell window, generate a trace of buyer.e by saying

$ rm -f buyer.txt
$ rune -DTraceLog_dir=. -DTraceLog_name=buyer.txt "$EHOME/scripts/test/causeway/buyer.e"
***-----------------------***
*** Trace data written to ***
'/home/markm/buyer.txt'
***-----------------------***
$ 

After buyer.e has terminated, currently, you must manually kill seller.e.

The first time a process actually writes something to its trace file, it will also print a diagnostic to its stderr with the name of that trace file, as shown above. After the trace has completed, rename the trace files to shorter, more memorable names. For example, etrace.2004-11-20T01_17_58.600Z.txt could be renamed to seller.txt.

Causeway needs deep thread stacks to run. When using standard JDKs, you can tell rune to give its program deeper stacks by using, for example, the "-J-Xss8m" option, to give it 8 MBytes. Assuming your trace files are seller.txt and buyer.txt in the current directory, you can run Causeway as follows:

$ rune -J-Xss8m "$EHOME/scripts/causeway.e-swt" buyer.txt seller.txt

Causeway has a long startup time, so be prepared to wait a while until your window appears. If you have any problems, suggestions, or other reactions, please let us know.