Browser based syntax highlighting code editor (part ||)

In a previous post I was moaning about the lack of a proper browser based (dhtml powered) syntax highlighting code editor.

I took the aproach descrived in that post of using a translucent textarea over the syntax highlighted code to achieve the result with the minimum of effort. Well, it hast turned out that it won’t work as fast as spected. The performance is horrible, at least in Firefox, so I’ve abandoned this aproach.

However, as I already have the code to highlight the code in real-time, I’m trying to implement the code editor following a more natural aproximation, which is to capture the keyboard input and process it. To speed up development time I’m focussing just in Firefox, but I guess it’ll be portable to IE, Opera 8.x and Safari quite easily.

To capture the input from the user we need to handle some events: onFocus, onBlur and onKeypress. Unfortunatly Gecko only allows onKeypress events on input and textarea fields (IE doesn’t have this limitation). To sort this problem out we just need to create a non-visible text input field (style.display: none) which receives the focus when the element containing the editor does. This way we can trick Gecko and make use of onKeypress evens (in IE this won’t work, since it doesn’t fire events on non-visible elements). The drawback with this is that the selection, if any, is lost when changing the focus to the input field, however it can be easily solved by recreating the selection range after switching focus.

The syntax highlighting algorithm uses regular expressions, since they should be much faster than implementing a traditional tokenizer/lexer in javascript. This means that, at least for now, the highlighting can’t be nested (“Hello $userName” is rendered in one color, as string, without applying another style to the variable $userName). Basicly, what the code does is create an internal structure divided in lines which in turn have N blocks. Each block defines an offset from the start of the line and a style (normal, comment, string …)

// comment
$text = "string test";

becomes

[
    {
        offset: 0,
        blocks : [
            {
                    offset: 0,
                    style: 'comment',
            }
        ]
    },
    {
        offset: 10,
        blocks: [
            {
                offset: 0,       // $text
                style: 'var',
            },
            {
                offset: 5,       //  =
                style: 'normal',
            },
            {
                offset: 8,       // "string text"
                style: 'string',
            },
            {
                offset: 22,      // ;
                style: 'normal',
            }
        ]
    },
]

This data structure is specially suited to parse source code, where most tokens can’t span over multiple lines. However, it’ll be very limited for languages like XML until we don’t support a stack based tokenizer (just a simple finite state machine).

That’s it for now, next installment will probably focus on how to handle the keyboard events :)

document.createElement('INPUT')

Just a quick post about yet another Internet Explorer glitch

When you want to add a new input element to a page, you must define the type attribute before including the new element in the document tree.

var inp = document.createElement('INPUT');
inp.setAttribute('type', 'checkbox');
document.body.appendChild(inp);

I was going crazy because I had the 2nd and 3rd lines swapped, which makes Internet Explorer display the input element as type text and gives an exception when trying to change the type.

In my coding style, even if it’s not the more efficient one, I try to append the new elements just after creating them so I don’t forget to do it.

Ideally, to optimize the execution speed, we should always add the element when it’s completely defined, to avoid browser redraws. Even if we need to add a bunch of elements, it’s better to create them in a document fragment, outside the document tree, and when we are done just insert the whole fragment.

My take on a flexible persmission system

Since I started programming using data sources and more specially web apps, I’ve always been looking for a flexible enough permission system to use in my projects. No matter how complicated the permission system implemented was that the client would ask for a given combination which isn’t supported by it >:(

Just recently a fellow developer told me about OpenLdap permissions system. I’ve been using a central repository (I call it registry) for all entities in my database schemas, this allows me to do a very light-weight pseudo object oriented data layer on top of a relational database like MySQL. This system is a bit similar to an LDAP datastore, after all it’s nothing more than a hierarchical object database. However, studying more in detail OpenLdap’s permission system I had all of the sudden the following idea.

Permission checks must be flexible and very fast, say you have a database with 10 thousend contacts which can be readed by the users in the same group as the owner and edited by the administration group and by the owner. Obviously you can’t waste 2 seconds just to query the database for the contacts an user is able to see in a listing. So I tried to simplify the whole concept, starting by what is a permission, when applied to an object/entity/resource a permission can be None, Read, Write or Delete. Things like Print or Export are in fact the Read permission, only with a different presentation layer. Since it’s only four possible values we can represent it with a simple integer like 0, 1, 2 and 4 respectively. So now, in this system, you have a registry with all the entities and another table with all the users. The only missing piece is to relate the entities with the users, for that the simplest thing to do (and the more flexible) is just to create a relation table which has a composite primary key of the entity id and the user id, as an attribute we put the access level we described as an int above. Now, we said we had 10.000 contacts or entities for this matter, and say we have 100 users, the resulting relation table could be upto 10.000×100 = 1 million rows! That is quite a lot of rows, even if it’s quite doubtfull that we will use the maximum of rows, it’s clear that this system has a drawback and this is that it won’t scale well with a huge number of users. I can live with it actually, at least it serves my purposes. If someone needs an application for some hundreds of users he may probably spend some bucks on a powerfull server :)

So now we have a system which allows for really flexible permissions but which only requires a simple table join to check them. Obviously it’s not operative right now, since we need a way to build the ACL table. In this task is where the OpenLdap method comes in handy. Basicly it allows to create rules based on the entity attributes (aka entity fields), so we can create a simple parser for this rules and populate the ACL table accordingly. Take the following example acl, it defines that all users can read any entity and the owner can write to it:

 access to *
       by self  write
       by users read
       by *     none

I’m not saying we need to implement OpenLdap’s syntax, it probably doesn’t fit too well a relational model, however it helps to ilustrate how we can achieve a very complex permissions system. One thing to note is that when the permission definition is modified, the whole ACL table needs to be recalculated. However, when we add a new entry we just need to calculate that new entry, so the overall performance is quite good.

In my opinion what this method offers compared to other commonly used permission systems in web applications is that it calculates the permission when creating/updating objects (or modifying the permission definitions) instead of performing complex (many times also slower) calculations when selecting data, which is most cases is the most frequent operation.

El uso de RAM del Exchange Store.EXE

Ya me he encontrado en varios servidores que el proceso Store.EXE del Microsoft Exchange usa cantidades desmesuradas de memoria.

En realidad esto lejos de ser un problema, es una característica del Exchange desde la versión 2000 creo. Para explicar porque hace esto de chupar memoria como si fuera un vampiro biteriano, primero hay que explicar como funciona el sistema de almacenamiento del Exchange.

Cuando recive un nuevo correo/contacto/fichero, vamos lo que sea que tenga que ser almacenado en los buzones, lo primero que hace es escribirlo en un fichero de logeado. De esta forma no tiene que estar esperando a que la base de datos este disponible para meter el mensaje, así que simplemente guarda la nueva petición en un fichero de log y vuelve a la espera de nuevas entradas. Por otro lado, un subproceso se encarga de ir procesando esos logs y va introduciendo las nuevas entidades en la base de datos. Como no tiene prisa por volver a escuchar nuevas peticiones, puede introducir los elementos en la base de datos de una forma adecuada. Adicionalmente, este sistema tiene la ventaja de que si se colgara el servidor (se va la luz, peta el exchange, vuelcas el café en el teclado…), no se perdería nada, ya que los datos se han guardado en los fichero de log, y aunque la base de datos quede corrupta por una operación no finalizada, Exchange puede restaurarla a un estado optimo haciendo uso de esos mismos logs.

Todo este rollo, nos viene al pelo para explicar el porque del alto consumo de RAM del proceso Store.EXE. Lo que hace es guardar los datos en el log y al mismo tiempo mantenerlos en memoria, a modo de cache, así que usará toda la memoria que tengamos disponible para crear un cache lo más eficiente posible. Por supuesto, los ingenieros de Microsoft no son tan tontos como para pensar que por ser quienes son tienen derecho a ‘secuestrar’ toda la RAM de un servidor. Lo que hace este proceso es usar una técnica llamada Dynamic Buffer Allocation, la cual va usando toda la memoria que quiere pero vigila el uso de la memoria virtual (cache de disco), si detecta que empieza a usarse la memoria virtual, el proceso comenzará a reducir su cache para dejar RAM libre a otras aplicaciones.

Como nota adicional, en mi busqueda de información he leido que para calcular la RAM necesaria para un Exchange (version 2003) se puede usar esta formula: 256Mb + 1Mb * Buzón Logicamente cuanto más RAM tenga mejor rendimiento, debido al uso del mencionado cache. Por cierto, para servidores con más de 1Gb se recomienda usar las opciones /3GB /USERVA=3030 en el boot.ini, y cambiar el valor del registro HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\HeapDeCommitFreeBlockThreshold a 0×00040000, para evitar la fragmentación de la memoria.