Last modification:
2005-2-28
2005-2-28
The key feature of RoKoFo is the ability to generate pictures on the fly. My design goal was to keep the quite expensive work under control. - Even under high load I don't want million processes running. It also makes no sense to share the work over multiple processes as the limiting factor is the CPU which gets used 100% no matter how many calculation get done. So I chose a queue design, where I enqueue the work which has to be executed. - It even has advantages: less task switches save resources and regularly gets a picture ready - and not one million at the same time after one hour...
Scheme of all over operation. - While the number of clients and application processes is under control of the web server, the number of worker threads is limited.
The graph reveals more details about the current implementation. - A number of clients sends requests to the HTTP server. Note that it is also possible of one client opening more connections to the server and so provoking multiple requests at the same time. The outline does not reveal the internals of the HTTP server (unblocking I/O, multi-tasking, multi-threading or mixture), as they don't change the Application.
At the moment the CGI interface gets used which boils down to one new process every request. Future enhancements include switching to a faster scheme; probably beginning with FastCGI. - A normal request is seen as an lightweight operation, only in case of picture generation the work gets enqueued. The first implementation used shared memory for passing the order. Now a client-server scheme is in use. I started actually with a TCP solution protected by SSL (basically for authentication reasons), which permits calculation getting done even on different machines (sharing the file system). The implementation distributed now is like the TCP/SSL one but reduced to unix domain sockets, where authentication are based on file permissions.
The worker is written entirely in C. For a possible adaption to multiple CPUs the worker daemon is multithreaded. At compile time one decides the number of threads executing the orders. One additional thread is created which accepts socket connections and enqueques orders. Later actually formed the TCP server or now the unix domain socket server. This thread is written with unblocking I/O making it accepting only memory limited number of connections.
The daemon is prepared to handle multiple queues with different priorities. Default is to work with two queues: one for orders in need and the other for orders with are expected to occur. An order consists of a command with gets executed unsighted and a target filename. While at first the target gets created with zero size, the commands output gets redirected to a temporary file. Once finished, the temporary file is moved atomically to the target.
This procedure permits:
The number of threads working on orders would practically find its upper limit in the available (virtual) CPUs. Less would leave some resources unused and more threads makes sense for debugging only. - The threads do nothing more than fork() and exec() the command of the order and waiting for it to terminate.
The working daemon could by design execute everything which generates output. The CGI script here orders the execution of "convert", the image processing tool from ImageMagick.
On our first contact with RoKoFos understanding of the name space in the http-URL path, lets just observe the last part. We consider the rest later on.
The "999x999" type restricts the outer box, "h999" limits the height, "w999" limits - guess - the width. Size arguments don't change the aspect ratio.
Now feel free to change the numbers... If you're the first one trying this size argument, the worker daemon kicks in, otherwise you get the prepared image lying on the server - until next cleanup.
Generally I don't want to discuss here the sense of "clean" URLs. There are numerous pages justifying my point of view (page from Simon Willison as example). So let's talk about the "name space" RoKoFo is using to achieve this goal.
The script could be hidden behind any kind of URL "root". I chose some Apache abilities to put it behind http://backsla.sh/photos. This is the base part of RoKoFo and remains currently unchanged during RoKoFos operation. Furture changes could involve e.g. parts of the server name to get interpreted for shortcuts or similar, provoking redirects. There will always be a alternative path-only way to get to the same resource.
After the "root" comes a directory-like part which represents the gallery somebody wants to see. A gallery is the abstraction of a collection of:
Please note that you have to finish the gallery with a slash ("/") to ensure the directory-like interpretation. Otherwise it could get mistaken as filename-like.
Just with the same syntax like the gallery choice, but after the gallery, comes the view the visitor wishes to see. Some easy heuristic is involved to distinguish between the gallery and view part. We understand under view a layout together with arguments. This is similar to the image resource URLs seen above.
Right now exist two layouts:
The view interpreter contains a very simple alias mechanism. It allows a nicer name for views to exist. This version of RoKoFo contains:
The "size" layout argument gets passed to every picture shown. Future layout arguments could also affect the layout only and not any individual image. E.g. a "style" argument that preselects a stylesheet or the XHTML template used by the layout (see below).
Please observe the nice optical effect of all images beeing in the same hight or width!
In the filename part of an URL (in one without parameter and fragment, it's the part from the last slash to the end) is the selection. It decides which portion of the gallery gets shown:
The first example shows the whole gallery like in the last tour example. The second selects one picture. As a single picture it apears a little small, so the third example kicks in with a changed view to see the selection bigger.
To get a path from one view to the next, we have to choose more views where one follows the other... For this simply chain one view after the other in the URL. Now it's known to RoKoFo which path you want to walk, but where do we store where in the path we are? - A single letter in the filename part will keep track of it (before a possible selection and separated by a dot in need).
Please look in the first example at the links for the images. They show a "b" as the view selector and the name of the corresponding image to select only this image in the following view.
The second example shows a configuration where is no view to follow on to, so there are no links provided for the pictures
The current implementation supports only to "zoom in" to the next view by clicking on an image. The range/selection for the following view is the picture chosen. Future releases should support a range selection behind a image link, depending on e.g. layout arguments.
Going from one gallery to a subgallery does neither "zoom in" nor "zoom out". We stay in the current view. - This is actually not a limit, but a clarification. (-: The limit is, here too, that there are no links provided for going "up" in the gallery hierarchy.
So, this first release needs a browser with back-button.
Because of the limited selection possiblities in this release the
around view pulls some stops to catch more effects... (-:
This supports no arguments and even the "all" selection does not what it
should. "Around" works as expected with a single image as
selection. It shows it with an default size and the two images before
and the two after that in a smaller size.
Selecting one of the smaller images changes only the selected image. This shows some intended modularity. The layout is free to show what it wants, giving it the whole gallery and the selection as "pointer".
The present default "nest" for a given gallery would look like:
/overview/around/fullsize/a
Reads: three views, beginning with the first (hehe :-). Since this is
the default, there is no need to carry this useless information in
every link. What actually is in need it the information in
which default view we actually are. I developed two procedures for the
default path, a third one still in mind. You can enter one of these
procedures by entering an URL which it could generate also on its
own...
The "default default" at this time is going to be removed, so I don't like to explain it here. You can catch the idea by entering http://backsla.sh/photos/ which of course will always be the entrance point of the "default default procedure".
Enter the second procedure at http://backsla.sh/photos/spanien/overview/ for the "Spanien" gallery as example. You see, "overview", the first default view is written as view. This gets recognized and "zoom in" links provide "around" as the only view seen in the link.
The unimplemented solution in my mind would be with an letter as an indicator like in manually selected views but without any view written in the URL.
One until here unmentioned shortcut comes up when entering views which don't collide with the default, but don't carry the letter indicator. In this case an "a" is asumed.
The image resource provides a clash point with the default entry. That restricts the default entry to the "all" selection. This means: While
Some sites try hard to disable people downloading images from their galleries but it keeps pointless. The browser downloads them anyway. Much cleverer would be to bind the visitor to the site by other means: like search, updates, flexibility and additional services based on these pictures.
My way is to make the relationship downloadable. A local copy of the pages completely in the flavour of my site sounds good to me. Anyway, the copyright considerations are beyond RoKoFo and I actually need a copy of my dynamically website on CD to give my father access to it.
Most web spiders, offline reader, network downloader, etc. do quite a good job on bad websites in changing absolute links in relative ones. But they have hard times finding good names for local copies of "files" without type and an endless tail of query parameter...
RoKoFo uses in any place fully relative links! Even the most dumb downloader will generate a working local copy!
Additionally it provides a way of having all links a typical type ending. Inspect this:
Look at the links and resources which get generated. Both of the latter change the mode of working by providing type information on all URLs.Conceptually the type enforcement allows resources to stay at least behind two different names. This lessens cache efficiency but I take the loss for the benefit of flexibility.
Providing the type once, does not only enable RoKoFo to follow this guide, it also enforces the selected type. For markup html and xhtml are supported, for the pictures only jpeg at the moment. 100% validate HTML and XHTML respectively are generated. The typeless mode does not only result in shorter URLs, it lets the way free for future implementations to drive more enhanced content negotiation. Imagine a new image format or just a switch to another. The typeless links could adapt easiely while it would be unexpected for users and some limited browsers like the Internet Explorer to receive an "image/png" with ending jpg.
A layout in RoKoFo is a code snippet. It's main task is to insert references and links in an XHTML DOM tree. Links and references are grepped from the chosen gallery and should reflect the selection desired by the user. A layout should not be bound to a specific DOM tree. - The DOM is generated beforehand based on any XHTML template file. With the help of XPath the place to insert is found by ID.
In reality many mentioned features are implemented in the "basic" layout. Because of this "coding mistake" the "around" layout is quite helpless. Moving this logic to common area is on the ToDo list.
Note: refer to the SQL initialisation script provided in the download area for detailed information
The biggest challenge is how to design the link between the data base and the file system. On one side there are the image files in the FS on the other side additional information in the DB. One could argue to put everthing in one place, the DB. While this seems theoretically extremely clean, it is a mess working with blobs where everybody except you is expecting files... - For God's sake, I'm not the first one thinking this way.
First, I use a relation which associates every file which has additional information with an unique id. An file is identified by the directory in which it is, a name, its type (file name ending) and its size. This split up of information helps in some cases of "desyncronization". Intended cases of losing information in the DB from its related file are:
| operation | what stays fixed | what could change |
|---|---|---|
| moving images around | name, type, size | directory |
| editing image or even changing type | directory, name | type, size |
| renaming files | directory, (type,) size | name |
Running a clean-up program after making changes of this type gets the DB in sync with the filesystem again. To support splitting a gallery in two I would like this program to support copying of attributes of unknown kind. So later written extentions would just work without adaption of the cleaner. In my first DB layout I chose table inheritance, but I dropped it because of quite limited support in PostgreSQL. - The cleaner program still needs to be written.
The beforementioned relation is called "files_base". "_base" is just an extention to show that access should be done throught a view called "files". The view would provide combined path names to fit the applications need. It should be updateable, while splitting up full path file names into corresponding parts fitting in the scheme of "files_base". - The view is not jet developed.
"files_base" has additonal entries like "parentId" and "targetId". They are going to be used in the future when virtual galleries need to get stored permanently. Virtual galleries are like normal galleries filled with pictures and subgalleries but have no backing in the file system.
Right now only one kind of attribute is implemented: persons. It has backup by the relation "persons" and the connecting table "filePersonRelations". Access to this attribute is provided by "filePerson", a view, masking the DB details.
The gallery title is still in files due to historical reasons (the beginning of my photo collection is older than any dynamic page wrapping it...)
Retrieving the information about persons works but still needs improvement. Look for the pictures "dscn5671" and "dscn5674" half way down in:
Stay with the mouse over these images to see the information (or look in the (x)html source).Searching for somebody and entering names: