programmable window systems like NeWS; was Re: ponderings

># From: Gavin Thomas Nicol <nick@nsis.cl.nec.co.jp>
># To: vsta@cisco.com
># Subject: ponderings
>#
># Can anyone point me to and FTP site with papers on Sun NEWS?
>#
># Last night, I played with the idea of MADO messages being in some
># language (the choice of language would probably be one of lisp,
># postscript, forth, or a specially written one). Sun NEWS, and NeXT use
># a kind of postscript, and they seem to be very good.
>#
># At first I thought that it must be really bad for performance, but
># later, I was thinking of all the things people do with a window
># system. Basically, I thought of 3 things: display text, draw graphics,
># and use UI's. I figure that if some of the larger, but frequently used
># UI operations (for example menu's), could be "downloaded", it would
># probably reduce the amount of client->window system traffic a lot (ie.
># instead of sending perhaps 20 draw messages, we just send one "call
># function" message. It would seem that, provided we have a fast
># language intepreter, performance would *increase*, especially network
># performance. Also, client programs using UI's might become
># smaller/simpler. This idea could be extended to bitblt as well. For
># example, if a user downloads a menu into MADO, it could download the
># drawing parts to bitblt. I kind of like the idea. Does anyone have any
># comments?
>#
># I should note that even though I like the idea, I currently plan to do
># the messaging in a different way. VSTa *is* an experimental system
># though, so if I get positive feedback, I might just go ahead and do
># it.
>#

I don't have any FTP site adress for Sun NeWS. However, I did program
several thousands of lines of NeWS, and can summarize my experience and
critics. I also did some Xlib programming, thus knowing the essential
of X11. I am *not* a VSTa expert (or even a VSTa user yet). I just read
some Vandys' introductory papers on it (and also Mach3 papers).

[[practical reasons for not yet using VSTa::
   Actually my home 486DX2/66 PC (with 16Mb RAM and 240Mb IDE disk) is
   running Linux 0.99pl13; i have an ultrastor34f -an VLB scsi2
   controller- with which i still have problems (possibly a hardware
   problem realted to a cheap ET-4000 VLB graphic card). As soon as my
   ultrastor34f will be running reliably, i would be able to use at
   last my second scsi (240Mb) disk and, most importantly, my QIC150
   archiv viper tape streamer. Thus i would be able to backups. Once
   backups are possible, i will do some experience with VSTa (perhaps
   even, if time permit, write a VSTa file system server compatible
   with Linux Ext2 FS).
]]

=======================================================================
A point of view on the NeWS window system.
by Basile STARYNKEVITCH, october 1993,
 email= basile@soleil.serma.cea.fr

(you are permitted to distribute verbatim this text)

A. Introduction and overview.
-----------------------------

NeWS was a great thing from Sun (it is a dead Sun product; in
SunOS5.3, coming out FCS in december 1993, there is no more NeWS, just
an X11R5 server with Display Postscript) . NeWS means Network
extensible Window System (or Server). Actually, Sun's window server
was a mixture of NeWS and X11R4, but these can be almost viewed as 2
different servers sharing common devices (mouse, keyboard,
screen). More precisely, the X11 server was a special NeWS lightweight
process (see below). NeWS and X11 had different ports and different
protocols. I am talking of the latest NeWS server (in
Openwin3). Integrating NeWS and X11 was a difficult problem to Sun,
and they did not succeed perfectly on it.

The NeWS3 reference manual has part number 800-6736-10 from SunSoft
(september 1991). The NeWS Toolkit 3 reference manual has part number
800-6319-10.

The key idea of NeWS (and you already know it) was that the window server
was programmable. So a typical NeWS client:

1) initialized a connection to the NeWS server.

2) defined (or extended) his own protocol by defining specific NeWS procedures.
3) then, talked to and from the server only in this newly defined protocol.

Actually, a client did not talk naked NeWS but used a big library
(written in NeWS, more than 30000 lines of NeWS source code, and
loaded into the server at server initialization time) called The News
Toolkit (or TNT). TNT code handled connection initiation and
termination, global selection processing, keyboard maps, and provided
an object extension to NeWS. The initialization of the NeWS server
took more than 10 seconds. TNT provided an OpenLook compliant widget
toolkit (mostly running in the server).

NeWS is a superset of PostScript. I assume you are familiar of most
PostScript ideas (it is a stack language, and uses several almost
independent stacks, including a call stack, an operator stack, a
graphic state stack, a message sent stack -for object oriented
programming support- and a dictionnary stack. A dictionnary hold
variable bindings; it is a finite set of name (or key) to value
bindings). I also assume you know some basics of window systems such
as X11.

B. NeWS objects
---------------

NeWS data are atoms (numbers, booleans, strings, and names, quite like
in lisp, where names are called symbols), arrays, dictionnaries,
events, canvases, lightweight processes (or lwps) and others (fonts,
etc). Composite objects (arrays, dictionnaries, etc) contain
(reference to) any kind of objects (perhaps themselves). Executable
arrays are NeWS program fragments. All variable bindings (ie values)
are thru dictionnaries (in contrast to Scheme or Lisp), so a variable
is just a name, and a name has no internal content (except its
immutable string of characters designation), and the value of a
variable is obtained by scanning the dictionnary stack until a
dictionnary containing the variable as a key is found. NeWS is a
reference counting garbage collected system.

Window system specific objects such as canvases, events, lwps are
magic dictionnaries. They are magic in the sense that changing some
particular key value in such objects could have some important side
effect. Also, magic dictionnaries should have some fixed names (or
keys) and are created with them. All dictionnaries (magic or not) can
have any additional keys added into. Magic keys have some specific
predefined names.

Canvases are very similar to X11 windows. There is a hierarchy (tree)
of canvases, analog to the X11 window hierarchy. A canvas is a magic
dictionnary. Hence mapping a canvas is done by changing the /Mapped
key to true. Canvases are part of graphic contexts, so all graphic
operators are executed in a given canvas (much as on a PostScript
printer all graphics are executed on the paper). The top of the canvas
hierarchy is a virtual root canvas, containing several physical screen
canvases. Canvases, (like postscript printer devices) contains a
coordinate transformation matrix and an enclosing (clipping) path. In
effect, canvas can have arbitrary shapes (it is possible to have a
round canvas to implement a round clock). Canvases have cursors. Each
canvas has an interest list, which gives the event the canvas is
interested in. X11 windows are special kind of canvases.

Events are somelike analog to X11 events. The main difference is that
an event is not a protocol message; it is generated (by NeWS code or
by physical devices such as mouse and keyboard) and handled inside the
NeWS server thru appropriate client loaded code. Events are also magic
dictionnaries, containing /Name, /Action, /Canvas, /TimeStamp,
/Coordinates and several other important keys. The system generate
events. For instance, a mouse left button press generate an event
whose /Name is /LeftMouseButton, whose /Action is /DownTransition,
whose /Canvas is the canvas inside which the mouse cursor was, and
whose /TimeStamp and /Coordinates indicate unique time&space
measures. There is a global event queue (containing unhandled events)
and events are distributed (ie copied) to each lwp expressing an
interest into it. Interests are templates or patterns for events.
Syntactically, an interest is an event with the /Interest key set to
true. NeWS operators exist for event (or interest) creation, event
sending (ie adding an event to the global event queue, where system
initiated events are also automagically added), and interest
expression.

All events are put in the global event queue (by /TimeStamp order),
either because they are automagically generated (eg a keypress) or
because they are explicitly sent in NeWS code (thru the sendevent NeWS
primitive). So, events are also used as interclient communication
(such as selection processing). In practice, hardware generated events
are handled by existing NeWS code (ie the TNT library, see below)
which sometimes translate them into higher level events. Hence, event
can be used for agent style programming. Global selection processing
(eg cut&paste) was partly done in NeWS global code (inside TNT) using
such program generated (and not device generated) events.

Beside obvious hardware related events (mouse & keyboard press,
release, motion, etc) the NeWS server generates 2 important kinds of
events: /Damage & /Obsolete events. Damage events (whose /Name is
/Damage) are similar to X11 expose events and are generated when a
canvas (or part of it) is damaged (because it just became
visible). Damage events gives a damagepath, which enclose the canvas
region needed to be repainted. Obsolete events (whose /Name is
/Obsolete) are generated by the garbage collector for finalization
purposes. Each object reference in NeWS can be hard (the usual default
case) or soft (an object reference may be made soft thru the soften
NeWS primitive, and can be made hard by the harden NeWS primitive).
When all references to an object are soft, an Obsolete event is
generated (whose /Action jey is the obsoleted object). This permits
some cleanup. In particular, every cycle of references (and these are
very common in windowing systems) should contain at least one soft
reference.

A NeWS lightweight process (or lwp) is a virtual NeWS executing
machine inside the NeWS server. So it is a NeWS interpreter. Each lwp
is a magic dictionnary containing among others an /OperandStack, a
/DictionnaryStack, an /ExecutionStack, a /SendStack, an /EventQueue
(containing events distributed from the global event queue into this
lwp but not yet processed), an /Interests array which give the
template of events interesting for this lwp. Lwps are grouped in
process groups. Of course, all lwps are internal to the NeWS server;
as seen from the Unix kernel, there is only one Unix process which is
the NeWS server itself! Most lwps are of course asleep, usually
awaiting for input (on a socket from the client Unix process) or for
an incoming event thru the awaitevent NeWS primitive.

Synchronization primitives (in particular monitors) are provided for
serialization between lwps.

C. Using NeWS
-------------

In practice, NeWS was used with TNT. TNT contained code for NeWS
server initialization (including keyboard mapping) and for widget
programming. TNT is mostly made of a (multiinheritance) class
hierarchy. NeWS classes and instances are dictionnaries (magic or
normal). The send (object oriented message primitive) operator is a
NeWS primitive, but the class hierarchy was in TNT (and TNT was
delivered as 32000 lines of source NeWS code). The NeWS server
initialization was another 21000 lines of source NeWS code. In
practice most NeWS object creation (eg creation of interests, lwps,
canvases) was done by instantiation of (client specific subclasses of)
TNT classes. OpenLook widgets such as text fields, menus, buttons, etc
are provided by TNT classes (and subclasses of a ClassCanvas provided
in TNT). Window managment (in the sense of X11 wms) could be done
partly or completely within NeWS.

The NeWS initialization code started several NeWS system lwps,
including a NeWS listener (just for listen(2) and accept(2) socket
calls), an X11 listener, a GlobalEventMgr and a GlobalSystemEventMgr
(handling system wide NeWS events, including selection processing).

When a new Unix process make a successful connection to the NeWS
server, this connection (handled by the NeWS listener) made a new NeWS
lwp specific to this Unix NeWS client. This lwp just read NeWS stuff
from the wire (ie connection to Unix client) and interpreted it.

Data exchanges between NeWS and client is either in plain (printable)
ascii or in a more compressed (somehow binary encoded) format. Each
NeWS input token was in ascii (if it started with a printable ascii
character >=0x20 and <0x7f) or in binary encoded form (if starting
with a byte >=0x80, specifing the format and length of following
bytes). A NeWS object can be designed in ascii by its name or in
binary by its number in a connection specific array.

Conventionnally, a well behaved NeWS client has 2 NeWS lwps: the
client NeWS interpreter, created by the NeWS system listener lwp
(itself created at NeWS startup time), and a local event manager
(created by appropriate class instantiation at client initialization).
The local event manager expresses interests in all events pertinent
for the client (it was attached to activated canvases).

A typical client send specific definitions (class definition for new
widgets or new widgets variants; instanciation of classes for new
object creations). The loaded code either handled events locally (eg
keystrokes into a text widget are interpreted by TNT textwidget code
to fill or edit the text widget's value) and at some times send a
message back to the client. Hence, the expected wire bandwith was low;
for instance, a client recieved NeWS messages only when a text widget
was filled (by pressing a return key) and not at each keypress.
Similarily, most repainting (eg Damage events) is done inside TNT and
does not require client intervention.

Some simple applications (eg a calculator, or a window manager) could
even be written entirely in NeWS. A very simple client program, called
psh, was used as a NeWS shell. psh just opened a NeWS connection and
copied all its stdin into this connection.

A simple programming utility, called CPS, generated C code for sending
fixed form NeWS messages in binary form. Essentially, it generated
printf-like formats and statements.

NeWS3 also contains a package system.

D. Critics
----------

NeWS was in practice quite slow. Waiting half a minute before seeing
the first window is very annoying. I think that NeWS lacked of
easy extension capabilities (ie adding new NeWS primitives programmed
in compiled C code, in a dynamically loaded shared library).

Most of NeWS failure was marketing, not technology. I think (as several others)
that Sun should have put NeWS in the public domain.

In practice, i think that a fixed widget set could have been compiled
into a programmable window server. In other words, big parts of the
TNT widget code should have been rewritten in C and provided as NeWS
primitives. Perhaps a NeWS to C compiler could have been nice.
Ideally, an online, incremental, compiler would have been great: the
client could upload some code which could be compiled on the fly,
hence providing the best of 2 words: extensibility by code uploading
and speed by compilation.

Choosing PostScript as a programming langage was in my opinion a bad
idea. Debugging NeWS scripts and codes was really difficult (first
because of a lack of a good debugger, and mostly because stack
oriented languages such as PostScript or Forth are not enough
structured: passing an extra argument, or forgeting one, is a common
error, which is very difficult to detect, since there is no argument
list, but just an operator stack; the extra argument was kept on the
stack and incorrectly used later). I suggest using a small Scheme-like
dialect for programmable servers, because Scheme is a structured and
powerful, but yet small, language (of course, bignums should be
excluded since they are useless in window systems); and Scheme to C
compilers seems to exist somehow.

Having interpreters (lwps in NeWS) independent from client connections
is too general (and de facto unused in NeWS). I suggest attaching one
interpreter to each client connection. Also, I think that all messages
should be syntaxically alike (in NeWS terminology, even input from the
client connection should have been handled as an event).

I think that the imaging model (a complete transformation matrix with
a general clipping path) of PostScript (or NeWS) is too general (and
too expensive) for ordinary window systems. Plain rectangular window
with fixed pixel-based coordinates (ie the plain X11 imaging model)
should be enough for windowing system (and high multimedia technology
needs perhaps even more than PostScript like sophisticated coordinate
systems).

In my opinion, a programmable window server should already provide
some fixed low level Graphical User Interface (eg elementary widgets
like buttons, menus, multifont text fields and labels); having it
wired into the server langage would make more consistent look&feel.

Also, i strongly suggest writing (for prototyping and pedagogical
purposes) an X11 client emulating most of the proposed window server
programmable protocol. Perhaps looking into Tk would be a good idea,
although i think that tcl is not really appropriate as a server language.

Basile STARYNKEVITCH ---- Commissariat a l Energie Atomique
DRN/DMT/SERMA * C.E. Saclay bat.470 * 91191 GIF/YVETTE CEDEX * France
fax: (33) 1- 69.08.23.81; phone: (33) 1- 69.08.40.66
email: basile@soleil.serma.cea.fr; homephone: (33) 1- 46.65.45.53

Linuxing at home only.

PS. Forgive me my english; it is a foreign language to me

N.B. Any opinions expressed here are solely mine, and not of my organization.
N.B. Les opinions exprimees ici me sont personnelles et n engagent pas le CEA.
Received on Wed Oct 20 04:44:50 1993