/ elib / distrib / vattp

VatTP Overview

Author(s): Bill Frantz (frantz-at-pwpconsult.com).

Subsystem originally written by Eric Messick.

Introduction

This description matches the version of the comm system included in R167.

The comm system includes the following features:

• Encrypted links.
• Unforgeable network object references with handoff logic.
• SturdyRefs and the Registrar.
• Message encode/decode.

Related Documents

See Comm system connections startup protocol.

For the design for the Neocosm data comm layer see: NewECommSystem.

Requirements

Requirements curently implemented:

• Authenticated, confidential links between E "vats" running on the same or different machines.
• Robust identities that persist over restarts of the application.
• Network location independence. An identity can move from one network address to another.
• E messages can be passed between vats.
• Three vat handoff: If vat A sends vat B a reference to an object in vat C, B will get a direct link to C for messages to that object.
• SturdyRefs: There is a type of inter-vat object reference which persists over restart of the referer, referee, or both.
• Allow a vat to build outbound connections and receive inbound connections. A vat can act as both a client and a server.

Requirements at best partially implemented:

• Firewall support: At best, a vat behind a firewall can only build outbound connections. It can not receive inbound connections.
• Connections are not torn down when they are no longer being used.
• The event logging and analysis policies log too many errors.
• Connections need to be periodically checked to ensure they are still working. (This is not being done in R167.)

Pure dumb things that need to be fixed:

• The byte and network layers need to be removed. They are left over from vat tethers.
• The number of threads per connection needs to be reduced to two.
• The Comm system needs to implement a "keep alive" function to ensure the other end is still alive.

Architecture

The Comm system can be considered to consist of four components:

• The Registrar which establishes the identity of the vat.
• The Connections Manager which manages the connections.
• The (multiple) connections to other vats.
• The listen stack which listens for new incoming connections.

Current Architecture Overview

Registrar

Each vat has one instance of the Registrar (Registrar.java) which performs the following functions:

• It holds the public/private key pair which define the identity of the vat. (When the vat is initialized, it generates the key pair.)
• It holds the list of Process Location Servers (PLSs) the vat will register with when it goes "onTheAir".
• It implements the "onTheAir" method which permits the vat to engage in inter-vat communications, and the "offTheAir" method which shuts down existing connections and terminates listening..
• It implements methods for the generation of SturdyRefs to local objects, following SturdyRefs to remote objects, and exporting transitory references to local objects.

VatTPMgr

Each vat has one instance of the VatTPMgr (VatTPMgr.java) which performs the following functions:

• Maintains lists of active and suspended/suspending connections.
• Accepts new connections from remote vats.
• Builds new connections to remote vats.
• Maintains usage information about connections so unused ones will be the first to be suspended.
• Handles and logs errors that occur while building connections, or while they are active.
• Provides connections statistics in string form for display in a UI.
• Holds a collection of string values that will be compactly encoded on the wire.

The Connections

Each active connection is broken into three layers:

1. The E layer (EConnection.java, ESender.java, and EReceiver.java) which:
• Maintains the import/export tables for the connection.
• Places incoming E messages <destObject, envelope> on the E Run Queue.
• Sends E messages <destObject, envelope> to the remote vat.
• Queue messages for connections which are not active.
• Handle the three vat handoff.
• Process the distributed garbage collection messages.
• Builds the necessary Msg layer for each location in the search path during vat location resolution. (PLS lookup)
2. The Msg layer (MsgConnection.java, MsgSender.java, and MsgReceiver.java) which:
• Represents one TCP connection.
• Handles the Connection startup protocol including E msg protocol version negoitation and crypto suite negoitation.
• Generates cryptographic keys for message privacy and authentication.
• Authenticates the remote end of the connection.
• Implements the RtDecodingManager interface. This interface helps in decoding references to local and remote objects which are referenced in the incoming message stream. It handles three vat handoffs.
• Handles the messages of the E protocol including envelopes, new classes, registering/unregistering objects, and distributed garbage collection.
• Uses the ec.e.db package to encode objects into envelopes and decode them from envelopes.
3. The TCP Interface layer (RawConnection.java, SendThread.java, and RecvThread.java) which:
• Maintains 3 threads for each connection, a send thread, a receive thread, and a control thread.
• Encodes the record oriented messages of the other layers for TCP's streem oriented transmission.
• Performs encryption/decryption of the messages.
• Generates a unique Initialization Value for use in encryption, and to ensure messages are not replayed.
• Generates and checks Message Authentication Codes (MACs) on each message to ensure the messages are not being inserted by an attacker. (It also ensures messages have not been altered in transmission.)
• Maintains a queue of messages for transmission.
• Collects multiple E protocol messages into one encryption block for transmission.
• Builds the TCP connection.
• Receives messages from the TCP connection and passes them into the vat.

The Import/Export Tables

Encoding

If, while encoding an object, an object is encountered which is not RtEncodeable, then RtStandardEncoder calls on MsgSender.encodeObject to encode the object. If the object is not an instance of RtDeflector, then a proxy is created for it on the remote vat. If it is an instance of RtDeflector, then it is a proxy for a remote object which is either hosted on the remote vat or on a third vat. All three cases are handled by the message type RtCodingSystem.kcManagerEncoded which is written by RtStandardEncoder. To manage proxys, the Msg layer has two tables, the ExportTable and the ImportTable.

The ExportTable holds information about objects which have been exported. Internally the ExportTable has Hashtables which map exportID to EObjectExport and the local object to exportID. The EObjectExport holds references to the local object and the EConnection, and a touch count. The touch count is used to avoid garbage collecting an object while a new reference is in-transit between the vats. N.B. The exportID is the Swiss number.

The ImportTable holds information about remote objects which have local proxys. It has a Hashtable which relates exportIDs to the local proxy objects.

If a proxy is to be sent to the remote vat, and the object has already been exported to that vat, the exportID is retrieved from ExportTable and the touch count for the object is incremented. If a new proxy must be created, the ExportTable creates a new EObjectExport places to appropriate entries in the two hash tables. In either case, the exportID is sent to the remote vat with an indication that this is an OUTBOUND_ID.

If the local object is a proxy and refers to an object in the remote vat, then the importID is sent with an indication that this is an INBOUND_ID. Otherwise a three vat handoff is performed by sending the exportID (which is the Swiss number), the registrarID and PLS search path for the vat which holds the object along with an indication that this is an HANDOFF_ID.

Decoding

When RtStandardDecoder is decoding a serialized object, and it finds an encoding type of RtCodingSystem.kcManagerEncoded it calls MsgReceiver to decode the object. MsgReceiver uses the ID written by MsgSender to determine its course of action.

If the ID is INBOUND_ID, then the other vat has a proxy and this end has the object. The object is fetched from the local ExportTable and and used as the local decoded object.

If the ID is OUTBOUND_ID, then MsgReceiver checks the ImportTable to see if there is already a proxy for that object. If there is, it is used as the local decoded object. Otherwise a new proxy is created, entered into the ImportTable, and used as the local decoded object.

If the ID is HANDOFF_ID, then MsgReceiver gets an EConnection object for the third vat. (N.B. This EConnection object may not be actually connected to the third vat.) It checks the third vat's ImportTable to see if a proxy already exists for the object. If the proxy already exists, then it is used. Otherwise a new proxy is registered in the ImportTable for the third vat, and a EXPORT_OBJECT_REQUEST message is sent to the third vat. (N.B. This message is queued until the a connection is built if the third vat's EConnection is not already connected.

When the third vat receives the EXPORT_OBJECT_REQUEST message, it uses the registrarID of the vat initiating the handoff to locate the object in that EConnection's ExportTable and registers it in the current connection as an exported object. It then sends an EXPORT_OBJECT_REPLY message to the vat receiving the handoff.

The Listen Stack

The listen stack parallels the structure of the connections. The r167 system has two instances of the listen stack, one to listen for comm connections from other vats, and the other to listen for notifications that some local user action (e.g. clicking on a .mcm file in a browser) needs to be handled.

The listen stack has two threads, the ListenThread which runs as a daemon, and the UserThread. The ListenThread needs to be a daemon thread because as a user thread, it would be uninterruptable while it is waiting on an accept(). (This is due to a bug in Solaris.) However, as a daemon thread, the application can exit while it is still running. We don't want to exit while waiting for connections (and not doing anything else), so we have the UserThread which does nothing but wait. Since it is a user thread, it keeps the application from exiting, and since it is not waiting on an accept(), we can kill it. When we tell the ListenThread to shutdown, which we can do because it is a daemon thread, it sends a notify() to the UserThread which then shuts down. Hallelujah!

When a new connection arrives, the ListenThread (ListenThread.java) builds a new TCP interface layer and passes it to the MsgListener (MsgListener.java). The MsgListener builds a new Msg layer which engages in the connection startup protocol. The EListener (EListener.java) is only used to handle errors and during vat startup/shutdown.

Proposed Architecture Overview

TBD

Off the shelf alternatives

We may want to replace the encryption, and authentication in the TCP layer with SSL. See SSL vs. E Comm for an analysis.

My (wsf) own opinion is that we should only do this if there are significant advantages from the change, since the current code seems to work reasonably well and there are significant differences in the symantic model. Possible advantages of SSL are more supported crypto algorthms, and better crypto vetting because it is a published protocol. I don't think SSL will help with our Firewall problems.

It might be valuable to model our implementation on the Baltimore Technologies implementation. I (wsf) would be more convinced if several other implementations had substantially the same implementation structure.

Other Design Objectives, Constraints and Assumptions

This code uses multiple threads to overlap communication with computation, so it must be designed to be thread safe.

Lists any special objectives and assumptions of the code e.g. reusability, thread safety, security, performance, use of resources, compatibility with existing code etc. This section gives important context for reviewers

Current implementation

See current architecture above.  See also External Interface for ec.e.net in r167 which shows which methods are referenced from outside the package.

Which directories on our tree does this subsystem cover?

ec4/javasrc/ec/e/net holds the source for this package.

Is it JavaDoc'ed?

The code is partially JavaDoc'ed. Since some classes use the foo <- blah form of Esend, javadoc barfs in processing them.

Examples

None

Testing and Debugging

(Optional) Lists any tests and debugging utilities which are to be developed to help test the design (e.g. test classes, trace categories, etc)

Design Issues

Resolved Issues

History of issues raised and resolved during initial design, or during design inspections. Can also include alternative designs, with the reasons why they were rejected

Open Issues

This section of the document is used by the authors and moderator to store any incomplete information - issues identified during a design inspection but not yet resolved (the task list), notes that aren't ready to be put into the main text, etc.

Does LDAP (Lightweight Directory Access Protocol) have any application to our PLS requirements?