A CD database


Warning

All this is still an unfinished draft. Suggestions, enhancements, modifications, additions, criticizing, welcome.


Foreword

This is the one first example that came to mind, down in '91, when thinking about my own "New System". At the time, I didn't mean to revolution anything, but just tackle my growing dissatisfaction about existing computing systems.
The problem was to design a simple system which would help me manage a database of my own collection of classical music compact disks, that would allow me to do more than what I could do by just looking at the well-ordered shelf where the disks lie.
So I started to imagine what useful things the computer could (and thus eventually should) do to help me with my CD collection: of course, I should be able to make listings, to select entries and sort the database according to arbitrary criteria; but I'd also have to be able to communicate simply with other people, to enter names in whichever language they really are, to automatically link entries to external resources, to share my appreciations with others, or use it as the basis for biased random selection of a disk to listen; I'd like to ultimately know the main tune from each track, and perhaps even to retrieve a disk according to a tune in my mind.
After a while, I realized how such things were not just impossible, but unconceivable even at any reasonable cost in existing systems. Then, I understood that my project would be more than an enhanced toy project, but truely a revolution in the way computing is thought about. As Carl Sagan said much later,
"To make an apple pie from scratch, you must first create the universe", which fits perfectly the situation I found: to write my small piece of personally useful software, I'd have to first make a whole computing system.
When I had to choose a name for what this project became, I remembered this old CD database, and it was part of my choice for the name "Tunes". Therefore it could be said that the Tunes project began with this CD database idea...


Table of contents

  • Introduction: Goal of this Tunes session
  • Chapter I: A Simple Database
  • Chapter II: Dynamically extending the structure
  • Chapter III: More generic structure rewriting
  • Chapter IV: Optimizing the database
  • Chapter V: Communicating with the outside world
  • Chapter VI: Beyond any database
  • Conclusion: Why Tunes wins


    Introduction: Goal of this Tunes session

    In this document, we tell how a Tunes user, F, who never read the paper docs, but knows the general principles of Tunes computing, and how to use on-line help, will try to build a system to catalog all the CD's and tapes in his own personal collection.
    This database should eventually allow him to do just anything one does manually that could be automatized, and even things one couldn't do manually, that can be done automatically: classify and select disks according to multiple arbitrary criteria; compare the database to other databases; annotate disks, interpretations, and pieces with notes, comments, relative frequencies at which you want each CD to be selected when asking the system to randomly choose one, or whatever you want.
    This may seem an ambitious project, but hey, why shouldn't it be possible ? We'll prove that it is, and show how the techniques proposed in the Tunes project allow do it incrementally, just the way a human actually thinks about things when designing a database, through automatic rewrite of the database as the structure evolves...

    How to read this document

    This document describes an interaction between a Tunes user, who will be refered to as F, and the Tunes system.
    The Tunes system will dynamically divide into objects during the session. In this document, objects will be refered to by the list of names or code numbers that were successively attributed to subobjects to reach the object from the topmost user-visible one: () will be this original topmost object; aaa will be some object accessible as aaa from object (); 'aaa will be the context (or meta-) object for object aaa; aaa.bbb will be some object accessible as bbb from object aaa; aaa.bbb.ccc will be some object accessible as ccc from object aaa.bbb; xx.1 will be some object with code number 1 accessible from object xx.
    These objects communicate with the human user, and with each other, which will be denoted by a sequence with the message originator, followed by an arrow, followed by the message recipient, followed by a colon, followed by the message contents, e.g.
    () -> aa.1.xx : "how are you ?".
    For ease of reading and writing, the objects will more than often be replaced by some higher object in the accessibility hierarchy. Particularly, many objects will remain hidden behind the topmost () object.
    How those messages are actually passed is not in this paper's scope. The Tunes User Interface should manage the interaction between computer objects and the human, while other meta-protocols decide how objects communicate one to the other.


    Chapter I: A Simple Database

    Firstly, F creates an object, and refines its specifications until it looks like something usable as a database. For that, he'll access some global database of available software modules, which contains database management packages.

  • F -> (): I wanna create some new object.
  • () -> F: Easy, guy: here's an object 1. Initially, it'll have no particular attribute, that is, be considered in an empty context.
  • F -> '1: Hello ! Firstly, CDDB would be a nicer name than 1.
  • '1 -> F: Ok. This is now object CDDB instead of 1.
  • F -> 'CDDB: Then I'd like to extend this context.
  • 'CDDB -> (): Dear (), could you please extend me ?
  • () -> F: Ok, fellow; context object 'CDDB wants to extend itself. How do you want to do it ?
  • F -> (): Well, use the most reliable, yet simple, database management module you know.
  • () -> (): Let's look into the distributed database of available modules (well, considering only modules that I trust).
  • () -> (): Hey, here's the latest and greatest (db) module from the Tunes team.
  • () -> (db): Hi, there. Could you tell me what to do about this object CDDB ?
  • (db) -> 'CDDB: There, initialize those attributes. For the moment, you'll be just a raw list of data, currently empty.
  • () -> F: done.

    .....
    F enters data. He uses many methods for that, all of which are automatically and/or manually generated: forms that default w/ last entry, with some editable history


    .....

    In this chapter, we saw how Tunes allowed the user to create the database using the same tools as he uses everyday. He didn't have to learn a new procedure for just this case.
    You may have noticed that we didn't specify how exactly the messages were passed between the user and the system or between different subsystem. Actually, the user could have personalized his way to communicate with the system to fit his taste. If he didn't, he would have used the same standard way as was provided by the system developper and programmers of the tools he uses. Keyboard and Text screen, point-and-click interface and graphical screens, voice recognition and synthesis, or any conceivable means are acceptable.
    For example, the newly created object would not really be given a name 1, but on a graphical screen, a subwindow of a visible stack window may be its "name", that the user refers to by pointing with a mouse, while the system uses whatever ad-hoc pointing convention internally. Changing the object's name might be dragging the visual representation of the object into another persistent store than the default one, with would then prompt for a textual name, give a default one, or again have it be implicit until defined. All this is only a suggestion about how interaction could be done, for how it is really done constitutes a topic completely independant from managing a CD database. See the Interfaces subproject for more about it...
    
    
  • à tout moment, l'ordinateur permet à l'utilisateur de savoir ce qu'il fait; il donne même des explications détaillées (si celles-ci existent);
  • l'utilisateur peut exprimer son désaccord avec l'ordinateur;
  • l'ordinateur peut aussi exprimer son désaccord avec l'utilisateur;
  • l'ordinateur est prêt à discuter; mais c'est l'utilisateur qui tranchera en dernier ressort (!) plusieurs user
  • des fonctionnalités peuvent être débranchées (pour aller plus vite) et rebranchées;
  • on peut les débrancher dans l'optique de les rebrancher, et demander à l'ordinateur de collecter au fur et à mesure les infos que l'on sait qu'elles demanderont (au lieu de tout faire d'un coup), sans les traiter pour en tirer des conclusions. ...
  • 
    ****** Chapître III : une base de données commune se révèle insuffisante *****
    
  • structure hiérarchique de bases de données (l'une puisant dans les inférieures) lieu d'un grand tableau hyper redondant. structure hiérarchique.
  • multiples occurences d'une
  • plusieurs ensembles interliés au lieu d'un grand redondant ou même d'une structure hiérarchique.
  • malgré tous les autres ensembles construits, celui qui est le plus représentatif (car correspondant le plus à un objet physique) est celui des CD eux-mêmes, que l'on peut choisir, prendre, jouer, etc.
  • 
    « La métaclasse de (1) sera "base de données"; elle a de nombreux champs
    d'information. Pour l'instant, considérer la base comme ayant une seule liste
    d'éléments, cette liste étant vide, et ses éléments étant de type
    vide (2.1). »
     Remarque: on ouvre une liste tout de même pour que l'utilisateur n'ait pas à
    le faire, sachant que lorsqu'un utilisateur ouvre une base de données, c'est
    rarement pour ne lui intégrer aucune liste.
    
  • (1) à F. « J'ouvre une liste (1.1) correspondant à (2.1) dans la fenêtre de (1). »
  • F. à (2.1) « Donne le nom "titre" au type des éléments de la liste, ce type étant une chaîne de caractère de longueur fixe égale à 60, pour l'instant. (Je choisis des options aussi faciles que possible, qui me mènent à un fichier de même type que dBase). »
  • (2) à (1) « Mets-toi à jour avec cet ajoût dans ton type. »
  • F. à (1.1) « Voilà quelques titres de CD en ma possession »
  • F. à F. « Le titre ne suffit pas, entrons aussi le nom de l'auteur. »
  • F. à (2.1) « Inclus-moi "titre" dans un type enregistrement. »
  • (2.1) à (1) « Ok. Je donne le numéro (2.1) à l'enregistrement, et le numéro (2.1.1) à l'ancien champ "titre". A priori, l'enregistrement comme son sous-champ porteront tous deux le nom "titre". » Remarque: (1) sait que cette commande ne modifiera pas l'organisation de ses données, mais a intégré ce changement.
  • F. à (2.1) « Donne le nom "CD" à l'enregistrement ; oh puis non ! Donne lui le nom "Disque compact". » « Ajoute-moi le champ "Auteur" à (2.1); ce sera un champ de type chaîne de caractère. » Remarque: Il existe un type Nom de personne, mais F. ne l'a pas vu dans la liste (peut-être n'a-t-il pas pris la peine de consulter cette longue liste, ou n'a-t-il pas su restreindre cette liste aux types apparentés aux chaînes de caractères, et a-t-il été noyé sous la masse des types disponibles, choisissant le plus familier, mais aussi le plus primitif).
  • (2.1) à (1) « Mets-toi à jour en insérant la place pour le champ "Auteur" »
  • F. à (1) « Voilà les noms des auteurs des oeuvres déjà entrées » Remarque : F. n'a rentré que le nom des auteurs, pas leur prénom. « Voilà quelques oeuvres supplémentaires. (titre+auteur) »

  • Chapter II: Dynamically extending the structure


    .....
    As we see, a database is not a fixed structure; even it's "type" evolves. Database means that the computer integrates knowledge, and allows the human to access it. Knowledge is due to evolve unpredictably (see Karl Popper's works about this); no static structure can restrict it. That's why no static database structure can dynamically satisfy the human's needs.
    Now what of all the data already entered ? In traditional database systems, it would be lost, or the user would have to write some very complicated program to statically translate it from the old structure to the new one. Tunes won't let you down like that... It will automatically generate and apply the translating some function when needed, so the database is extended with all the existing data preserved.


    Chapter III: More generic structure rewriting

  • [The structure of the database can get completely rewritten, not just extended, so as to reflect deep changes while still conserving all the existing data.]
  • [Rewrite is done by default in a reversible way, so that errors can be undone trivially, and rewrite be done again in better ways.]


  • .....


    Chapter IV: Optimizing the database


  • Optimizing is a special case of directed rewrite that conserves the semantics.
  • Up to now, we focuses on logical structures; now that these are settled, we may think about their physical (well, lower level) representation...
  • to begin with, manual enhancements
  • then, automatical optimization:
  • based on profiling feedback from accumulated data, as well as on statistics on preceding modifications: what fields are more frequently modified, which have which kind of length.
  • compute redundancy in data;
  • refer to a library of well-known encoding methods
    .....
    In existing systems, you must give the low-level details of the database implementation right from the beginning, at a time when you have only a faint idea of what will be the actual manipulations users will do on the database, and the time constraints they will induce. Then, you cannot modify this implementation without having to rewrite a great lot of the database plus translators. It is impossible to test multiple implementations and choose the best one. Thus, however "optimizing" (according to completely arbitrary criteria), the existing databases are much slower than they could (hence should) be. Tunes solves this problem: optimizing is just some kind of performance-directed rewrite.


    Chapter V: Communicating with the outside world



    .....


    Chapter VI: Beyond any database


    The user can integrate music-playing and later music-recognition software to his database...
    .....
    In this chapter, we saw that integration in Tunes was not just a matter of a uniform interface, with some hack to have coarse-grained low-level applications communicate between each other, as in leading commercial operating systems, but simple and deep concept of fine-grained higher-order modularity. The consequences are that little technicity is required to customize a system, so large teams with lots of boring and stubborn docs are no longer needed to maintain and extend software, and people are no longer taken hostages by large corporations and specialized vendors: if one is not willing or able to learn the small amount of technicity required, one can easily find a qualified technician, and needs not abide by the diktats of monopolistic providers.
  • 1) list of CDs, each one music piece
  • 2) list of CDs, each list of music pieces
  • 3) music pieces:
  • Undo some badly designed structural change, using the old information.
  • Add first names.
  • Talk about how lazy evaluation and futures can help use of the database while still using structural rewrite with human interaction...
  • Use some shared implementation of names.
  • Split some database fields: name into first name+name, with intelligent, interactive lazy evaluation: the user has full control; queries will be blocked if necessary.
  • Restructure the database: instead of having author name, point to some author in an author database.
  • Now that there is an author database, and we have the author's nationality, use country-wise author name compression ?
  • Now CDs can contain more than one piece
  • CDs contain tracks, not pieces.
  • Tracks can contain part of a piece, or multiple pieces.
  • pieces may be divided into subdivisions.
  • According to the type of piece (concerto, symphony), have default lists of subdivisions.
  • Optimize the encoding: take advantage of the fact that most recordings are "nice", mapping subdivisions of a piece into consecutive tracks; encode interprets parallelly to the pieces in a list/interval of tracks.
  • There is a mechanism to select the subdivisions (default: all), to automatically map the subdivisions of a piece into consecutive tracks (default) or map them all into a single track.
  • Even with this trackwise description of CDs, be sure that the most important is the relation between pieces and CDs.
  • Tapes may be considered more or less as CDs with two tracks !
  • Now we differentiate interpretation from the piece.
  • Generate automatically the list of subdivisions of a piece from
  • Scores have write date (which may be unions of time intervals, with question quotes), and creation date.
  • cross-links with a database of partitions (paper or online texts for the music, exact, partial or adaptations).
  • All the entries in all databases can be annotated by text or anything: historical notes, textual or numerical judgement of the quality of the recording and/or the interpretation, etc.
  • Now, we can publish this fine piece of software on the net, so that all amateurs of music can benefit from it.
  • Add support for sharing/merging databases, cross-checking information, looking up critiques about CDs, etc... There would be some global CD database, everyone managing the CDs one has, with some system to merge/cross-reference diverging versions of entries. One's collection would just be a list of which CDs one has. CDs could also be let and the manager remember who borrowed what. One could publish his entries, but One has all control on the way his entries are updated from globally published sources (so that one won't be helpless victim of viruses), particularly when it comes to destructive updates and structural updates...
  • Now we can sell this piece of software to companies selling music recordings as the finest possible database system ;-)
  • At the end, add support for tune-directed browsing (e.g. you whistle the tune, and a tune-recognition system finds the piece, then the right tracks on the right CDs, and perhaps even the right second in the track !). This way, music lovers can walk their CD collection (or a CD library) easily, or play games about recognizing tunes, CD publishers can interactively advertise their products, copyright managers can check piracy more easily,
  • This is the miracle of Tunes: there is continuous progress between a personal hack and a globally useful publishing system. People can really reuse what they or other did, and really contribute to global welfare.


    Conclusion: Why Tunes wins

    Existing database system require your designing in advance the full structure of the database, with no possibility to correct, amend or enhance it whatsoever. That's why they are so lame.
    Again, all this is the same one problem that arises everywhere in computing: one has to choose between a static design or a dynamical system. Statism may be expedient if you have a one simple problem to solve immediately, but only dynamism is truely useful in the long run, where problems ever evolve and grow in complexity. All companies sadly seem to have chosen statism and expediency, as they live in the short term by their local interest. Tunes is a Useful, Not Expedient, System, hence chooses dynamism and utility.
    This is truely part of the moral fight of clairvoyance against narrow-mindedness. Tunes is an ethical project in the computing world, which is why it shall overcome eventually. To help it is to help you as well as the whole world. Please help yourself, please help us. The first and simplest way to help is to tell us how we are erring if you see that we are, so we can correct our views and be useful when we would have been harmful.


    To Do on this page

  • Annotate example points by the more generic issues they tackle.
  • Finish translating the old French part into English...
  • Complete the example.
  • A computer is to automate things. To manage things that can be done fully automatically (yet let me check and control what's being done); to help me choose, by providing me objective information; to remember things for me, and remind me when I need them; to teach me things I don't know yet, to help me create informationally consistent things. That is, it is to increase my informational self.

  • Back to the Examples page, or
    the Tunes Home page.


    Author:
    François-René "Faré" Rideau -- rideau@ens.fr