The Tunes LLL Subproject


Contents of the LLL Subproject

Here are the current contents of this LLL project:
  • Firstly, the goals of the subproject
  • Second, the requirements for this LLL.
  • Then, a first sketch of the LLL semantics
  • A list of difficulties about the LLL
  • Some implementation ideas for the LLL
  • A list of modules to implement.
  • Our generic implementation of the LLL.
  • There is a page about the low-level problems of Storage: garbage collection, persistency, etc.
  • There are two sub-subprojects below the LLL: the i386 and the O'TOP subprojects.

    As you see, the LLL subproject is still in a draft process :(
    Feel free to contribute !

    You may also want to
    look at the what Review subproject said about implementations of other systems,
    go back to the main Tunes page, or
    jump back to the Tunes Subproject page.


    Goals for the Low Level Language

    The goal of this Subproject is a define and implement a low-level language that will fulfill all the requirement to be used as a basis for the Tunes system, including self-extending into the full Tunes HLL, etc.


    LLL Requirements

    • see the HLL Requirements...
    • portability accross different architecture
    • local (real-time ?) and distributed GC for higher-order persistent objects
    • annotations (typing is done through them)


    LLL Semantics

  • At the core of the LLL, all the computation words of ANS FORTH, excluding all that have to do with parsing, I/O and defining words; our version for these will be personal.
  • Above that, the LLL system provides some mechanism to do active annotations.

    Low-level objects

    • objects can be uniquely identified in some way.
    • Typically, objects in the local address space are identified by their 32 bit CPU address (perhaps with an implementation-dependent offset)
    • objects outside it are accessed using descriptors/handles inside it, so need not be considered as special objects wrt the LL memory system.
    • objects of a same address space must trust each other anyway, so let them have a cooperative GC and multithreading.
    • communication between address spaces happens directly through space-dependent drivers.

    Annotations

    • There are standard mechanisms for annotations and their resolution
    • Every annotation may have its own implementation, be it a hashtable of object to value association, or an array of values, or some executable code.
    • lazy evaluation and futures, typing, UI interface, scoping are done through annotations.
    • for example, an object's concrete type (relative to the GC mechanism) may be determined from bits in its address, whether statically or dynamically.


    LLL Difficulties

    See HLL difficulties...

    Hardware independence

    At what level shall word size be decided ?
    How can objects migrate back and forth between machines with different bytesize with no loss in information ?

    Garbage Collection

    Infix pointers (that do not point at some globally constant offset to the beginning of an allocation unit) greatly complicate the GC. They should be forbidden whenever possible.
    "C" like infix pointers can still be simulated with a segment-offset pair, with an aligned pointer and an offset inside the segment.
    The GC may have to accept infix pointers as for code return addresses, or else the calling convention may become grossly unefficient
    I propose that code "segments" should not cross, say 4K or 8K boundaries, so that finding the right code segment is just a matter of checking the list of segments in the segment obtained by masking the right bits.
    Big problem: how to efficiently differentiate pointers from numbers, etc ?
    structural differentiation is powerful, but may slow the GC considerably, unless descriptors are simple (can be just an integer for length of a pointer array, for most objects), and forbids dynamic differentiation, mixing integers and pointers in an array (e.g. simple stack), etc. That's why we'll use a simple bit pattern to differentiate integers (raw data) from pointers (structured data), and different kind of pointers from each other (that's a BIg Bunch Of Pages kind of GC).
    we must choose between integers having low bit set or low bit cleared. Having it set (and thus having bit cleared for pointers) may allow faster pointer access on RISC machines, but slows any arithmetics. Having bit set for pointers allow easier arithmetics, but forces the use of an offset for all memory accesses.
    The big question is: will integers be stripped of their low bit, which would simplify overflow testing code to naught, and make the implementation portable, but make a little harder doing pointer arithmetics and mixing of true integers with 31 bit ones. Or stripping them from their overflow bit, which makes integer overflows to generate GC-readjustable pointers, rather than providing flat modulo arithmetics, but allows easy pointer arithmetics and mixing of 31-bit integers and 32-bit ones ?
    We shall implement both ways, and compare actual execution time and code space measurements !!!
    A high-level page directory is used to determine the GC type of objects according to the page it is in. It is a multi-level hashed structure that may evolve with the GC code, so that it may allow to find quickly the type of objects. Typically a mix between arrays and balanced binary trees to recognize bit patterns.
    The GC type of an object, as determined by its address gives us routines to update the object during a GC, to destroy the object when it is not accessed anymore, etc.
    The GC type of a page chunk allows us to track down the beginning of individual objects pointed to on the page (in case infix pointers are used), also gives us the policy to follow when swapping out the page (which may be copying the page to disk, sending it to the network, or compressing it to memory for possible further actual swapping out of memory, etc).

    Persistence

    Be careful with distributed persistence: always remember previous states until all transactions using it are finished and confirmed.


    Implementation Ideas

    Implementation language

    We shall use the m4 preprocessor (or later our own HLL) to produce assembly source files for all our different target processors from mostly the same meta-source.
    We shall use C as though it was a mostly regular assembler, with labels, jumps, etc, so the same meta-source also produces the C source files. Hey, wasn't C called a portable assembler ? ;->

    Modules

    modules have some install/uninstall annotation fields explaining how to restore/resume the object from the state log as gotten from persistent store. In general, this will be a call to a standard trusted high-level module. However, this can be low-level code in the very first bootstrapping modules...
    This scheme can be used for migration in general: persistence, garbage collection, dynamic linking, etc.

    Mixed cooperative/preemptive multithreading:

    Preemption is necessary for real-time response as well as for unsecure tasks; but cooperation allows *far* better context switch time (on the i386, compare saving some 5 registers to saving the whole CPU+FPU+MMU state), not to talk about trivial implementation of mutual exclusion.
    In traditional systems (as well as in emulation boxes for them), all software is so completely and deeply unsecure that only preemption is possible for the system not to crash too often. Now, Tunes is a secure system, and we can and shall use cooperation whenever possible (=almost always): we shall require require standard compilers to generate cooperative code, that PAUSEs every now and then.
    Of course, real-time threads need preemption, as well as untrusted environments (which emulation boxes are); but as long as these use only statically allocated data (not managed by the shared garbage collector), everything is fine.
    Now, a common problem with cooperation is that PAUSEs are often too irregular, and while having them to far away for each other yields poor response time, having them to near yields too many context switches and related overhead (even in the best case, lots of cache misses). We can solve this problem by requiring some fine PAUSE resolution, but do actual context switch only when the timer says so (by modifying the code of a PAUSE code to just a RET/NEXT until it's time).
    This is perfect, but may slow tight loops significantly. One way is to unroll tight loops enough so that this cost is reduced, which is the way to use on harvard architectures with large separate code/data caches. The other way is to have dynamically modified code: before to enter a tight loop, some "interrupt recovery" routine is registered (which is unregistered afterwards); if an interrupt happens while in the tight loop, then the recovery routine is called, which has the responsibility to put the system in a state that follows the usual calling conventions; typically, the routine will continue normal execution of the loop until some modified point which will jump back into the remaining of the recovery routine, which can finish the work because it knows the exact point of execution in the loop (which is could have been at any point inside the loop). If modifying code is not possible, the recovery routine will have to find what to do using more traditional ways, according to the value of the interrupted program counter. Machines with lots of registers or register banks may have disjoint registers/banks for real-time and normal threads.


    Modules to implement

  • generic memory management modules
    • A generic generational garbage collector with full support for persistence (checkpointing and restarting of objects).
    • A generic module for inter-heap (perhaps distributed) synchronized persistent garbage collector
    • A generic module for back-reference unsynchronized persistent conservative garbage collector.
    • generic support for chunks of executable binary
    • generic full support for GOOSE modules.
    • mechanisms for the migration manager (from the Migration subproject) to use: swapping to disk, to a compressed RAM zone, to another host, etc.

  • low-level management of user interface hardware
    • A console adapter for text-mode
    • a text-mode adapter for serial consoles (perhaps just use the ncurses package)
    • A screen windowing multiplexer for text-mode screens
    • A generic output-synchronizing input multiplexer
    • A standard combination of the previous two.
    • A fast graphic library to attract game programmers

  • File formats (I guess these should presumably go to the HLL stdlib...)
    • A partition manager, that multiplexes hard disks according to standard partitioning methods.
    • Support for various existing file systems: MS-DOS FAT FS (and WindowsNT or Linux UMSDOS extensions), Linux EXT2 FS, etc.
    • Graphic file formats: GIF, JPEG, PCX, TIFF, etc
    • Various Audio file formats for sound samples


    To Do on this page

  • Find all LLL requirements.
  • Find all implementational problems.
  • Make first proposals.
  • Divide into subfiles.
  • Define separately semantics & representation
  • Find out how multiple architectures can be made interoperable...
  • Find some nice implementation for annotations...
  • Separate annotation mechanisms from annotation resolving policies...
  • Open a "Store" subproject about encodings and algorithms for the dynamically and actively annotable, distributed, persistent, garbage-collected store. Actually, distribution, persistence, and garbage-collection could be obtained by proper active annotations...


    Back to the Tunes Home page or Subprojects page


    Page Maintainer:
    Faré -- rideau@clipper.ens.fr