Last updated: [1998/05/12 Bill Frantz]
Author(s): Bill Frantz (frantz-at-pwpconsult.com).
Subsystem originally written by Eric Messick.
This description matches the version of the comm system included in R167.
The comm system includes the following features:
For the design for the Neocosm data comm layer see: NewECommSystem.
Requirements curently implemented:
Requirements at best partially implemented:
Pure dumb things that need to be fixed:
The Comm system can be considered to consist of four components:
Current Architecture Overview
Each vat has one instance of the Registrar (Registrar.java) which performs the following functions:
Each vat has one instance of the VatTPMgr (VatTPMgr.java) which performs the following functions:
Each active connection is broken into three layers:
The Import/Export Tables
If, while encoding an object, an object is encountered which is not RtEncodeable, then RtStandardEncoder calls on MsgSender.encodeObject to encode the object. If the object is not an instance of RtDeflector, then a proxy is created for it on the remote vat. If it is an instance of RtDeflector, then it is a proxy for a remote object which is either hosted on the remote vat or on a third vat. All three cases are handled by the message type RtCodingSystem.kcManagerEncoded which is written by RtStandardEncoder. To manage proxys, the Msg layer has two tables, the ExportTable and the ImportTable.
The ExportTable holds information about objects which have been exported. Internally the ExportTable has Hashtables which map exportID to EObjectExport and the local object to exportID. The EObjectExport holds references to the local object and the EConnection, and a touch count. The touch count is used to avoid garbage collecting an object while a new reference is in-transit between the vats. N.B. The exportID is the Swiss number.
The ImportTable holds information about remote objects which have local proxys. It has a Hashtable which relates exportIDs to the local proxy objects.
If a proxy is to be sent to the remote vat, and the object has already been exported to that vat, the exportID is retrieved from ExportTable and the touch count for the object is incremented. If a new proxy must be created, the ExportTable creates a new EObjectExport places to appropriate entries in the two hash tables. In either case, the exportID is sent to the remote vat with an indication that this is an OUTBOUND_ID.
If the local object is a proxy and refers to an object in the remote vat, then the importID is sent with an indication that this is an INBOUND_ID. Otherwise a three vat handoff is performed by sending the exportID (which is the Swiss number), the registrarID and PLS search path for the vat which holds the object along with an indication that this is an HANDOFF_ID.
When RtStandardDecoder is decoding a serialized object, and it finds an encoding type of RtCodingSystem.kcManagerEncoded it calls MsgReceiver to decode the object. MsgReceiver uses the ID written by MsgSender to determine its course of action.
If the ID is INBOUND_ID, then the other vat has a proxy and this end has the object. The object is fetched from the local ExportTable and and used as the local decoded object.
If the ID is OUTBOUND_ID, then MsgReceiver checks the ImportTable to see if there is already a proxy for that object. If there is, it is used as the local decoded object. Otherwise a new proxy is created, entered into the ImportTable, and used as the local decoded object.
If the ID is HANDOFF_ID, then MsgReceiver gets an EConnection object for the third vat. (N.B. This EConnection object may not be actually connected to the third vat.) It checks the third vat's ImportTable to see if a proxy already exists for the object. If the proxy already exists, then it is used. Otherwise a new proxy is registered in the ImportTable for the third vat, and a EXPORT_OBJECT_REQUEST message is sent to the third vat. (N.B. This message is queued until the a connection is built if the third vat's EConnection is not already connected.
When the third vat receives the EXPORT_OBJECT_REQUEST message, it uses the registrarID of the vat initiating the handoff to locate the object in that EConnection's ExportTable and registers it in the current connection as an exported object. It then sends an EXPORT_OBJECT_REPLY message to the vat receiving the handoff.
The Listen Stack
The listen stack parallels the structure of the connections. The r167 system has two instances of the listen stack, one to listen for comm connections from other vats, and the other to listen for notifications that some local user action (e.g. clicking on a .mcm file in a browser) needs to be handled.
The listen stack has two threads, the ListenThread which runs as a daemon, and the UserThread. The ListenThread needs to be a daemon thread because as a user thread, it would be uninterruptable while it is waiting on an accept(). (This is due to a bug in Solaris.) However, as a daemon thread, the application can exit while it is still running. We don't want to exit while waiting for connections (and not doing anything else), so we have the UserThread which does nothing but wait. Since it is a user thread, it keeps the application from exiting, and since it is not waiting on an accept(), we can kill it. When we tell the ListenThread to shutdown, which we can do because it is a daemon thread, it sends a notify() to the UserThread which then shuts down. Hallelujah!
When a new connection arrives, the ListenThread (ListenThread.java) builds a new TCP interface layer and passes it to the MsgListener (MsgListener.java). The MsgListener builds a new Msg layer which engages in the connection startup protocol. The EListener (EListener.java) is only used to handle errors and during vat startup/shutdown.
Proposed Architecture Overview
Off the shelf alternatives
We may want to replace the encryption, and authentication in the TCP layer with SSL. See SSL vs. E Comm for an analysis.
My (wsf) own opinion is that we should only do this if there are significant advantages from the change, since the current code seems to work reasonably well and there are significant differences in the symantic model. Possible advantages of SSL are more supported crypto algorthms, and better crypto vetting because it is a published protocol. I don't think SSL will help with our Firewall problems.
It might be valuable to model our implementation on the Baltimore Technologies implementation. I (wsf) would be more convinced if several other implementations had substantially the same implementation structure.
Other Design Objectives, Constraints and Assumptions
This code uses multiple threads to overlap communication with computation, so it must be designed to be thread safe.
Lists any special objectives and assumptions of the code e.g. reusability, thread safety, security, performance, use of resources, compatibility with existing code etc. This section gives important context for reviewers
See current architecture above. See also External Interface for ec.e.net in r167 which shows which methods are referenced from outside the package.
Which directories on our tree does this subsystem cover?
ec4/javasrc/ec/e/net holds the source for this package.
Is it JavaDoc'ed?
The code is partially JavaDoc'ed. Since some classes use the foo <- blah form of Esend, javadoc barfs in processing them.
Testing and Debugging
(Optional) Lists any tests and debugging utilities which are to be developed to help test the design (e.g. test classes, trace categories, etc)
History of issues raised and resolved during initial design, or during design inspections. Can also include alternative designs, with the reasons why they were rejected
This section of the document is used by the authors and moderator to store any incomplete information - issues identified during a design inspection but not yet resolved (the task list), notes that aren't ready to be put into the main text, etc.
Does LDAP (Lightweight Directory Access Protocol) have any application to our PLS requirements?
Unless stated otherwise, all text on this page which is either unattributed or by Mark S. Miller is hereby placed in the public domain.