Re: Clustering

Andy Valencia wrote:
>
> [Adrian Wong <atwong@nersc.gov> writes:]
>
> > 1. What are the current thoughts on clustering? Is it too
> > premature to start thinking about this?
>
> There's some stuff in the mail archives on this. I got transparent kernel
> messaging going; see vsta/src/srv/ka9q/cmds/proxyd.c and related ka9q source
> files.

Thanks for pointing me in the right direction on this. Actually
I found src/srv/ka9q/vsta.c and src/srv/ka9c/vsta_srv.c most
informative. I very much like the design that underlies the source.
It provides a mapping from ports to TCP/IP sockets with no
mods to the microkernel. I suppose the microkernel acts as the
switchboard and the user-space daemon does the job of repackaging
messages for a different protocol.

>
> > 4. Would port numbers be unique cluster-wide? How do you
> > "discover" the services "out there".
>
> If what you're talking about is sharing a single TCP/IP stack (so that all
> the machines appear behind a single IP address), then this is kind of a
> different path from the one I chose.

Actually, I wasnt even thinking of a heavyweight transport such
as TCP/IP. But rather some future (vaporware) such as InfiniBand
or VI Architecture. So here are my thoughts on the mechanism:

   1. Start an cluster service, mount point /sysnet. Daemon
      takes the single argument "address:port" of the cluster-wide
      master daemon. Connect to master daemon and obtain
      info of cluster network, available services etc. "ls /sysnet" does
      what you expect - list the cluster status.

   2. For each desired service, spawn a thread, open the port
      and register the mount point as if it were local, "/junk".
      For each incoming message, establish network connection
      to remote service, repackage message and send.

   3. On the server side, for each exported service, the cluster
      daemon opens a network endpoint and listens for incoming
      connections. Incoming messages are forwarded to the local
      service msg port from a spawned thread.

   4. The master daemon keeps the registry of participating
      nodes, the map of services to addresses, resolves name
      conflicts, etc. It also could handle failover.

So at a high level, the cluster daemon really just maps local
ports to remote ports and provides the necessary transport
between the two. The advantage is that no changes are required
to existing services so long as no assumptions are made about
the locality of the messages and the senders.

In this scenario, only the ports and network endpoints that
are being listened on are open. All other connections are
made "just-in-time". Obviously there is a case to be made
for open file descriptors and having a permanent connection
for the lifetime of the file descriptor.

>
> The other interesting aspect which comes to my mind is all the microkernel
> services which are *not* message based. Process creation, for instance.
> It'd be nice to find a way to move more of its raw services to messages, so
> that most of what can be done on one machine could be done just as well from
> a remote machine. QNX got this right. Interestingly, some of this was
> dropped from their newer Neutrino OS. At least in early releases (I haven't
> been keeping track lately).
>

I like the way Chorus handles remote processes with a control thread
and control port for asynchronous messages (aka signals) for each
process. That seems to retain the POSIX semantics for processes.
That being said, it is not clear to me what it means to "fork"
a remote process in the POSIX sense. "exec" I can get my head
around.

Adrian
Received on Fri May 19 11:16:22 2000