Midgard follows Ubuntu's synchronized release schedule, and releases packages for that platform, but otherwise we have little to do with the distribution. Still, I found the following in Mark Shuttleworth's Jaunty Jackalope announcement interesting:

Another goal is the the blurring of web services and desktop applications. "Is it a deer? Is it a bunny? Or is it a weblication - a desktop application that seamlessly integrates the web!" This hare has legs - and horns - and we'll be exploring it in much more detail for Jaunty.

This echos quite well with our plans to take Midgard much further than just the web. As Everaldo often reminds me, Midgard may soon become very interesting to developers of desktop and mobile applications. What we provide to them is:

• A simple object-oriented persistent storage library
• Bindings for Python and Mono in addition to the C library
• Ability to use either SQLite or a "proper database" thanks to libgda
• Replication capability, with XMPP support on its way
• D-Bus signals for inter-process notifications of I/O events

And on top of all this, Midgard comes with a pretty efficient MVC framework for PHP. This means that the desktop applications can be coupled with a nice social web service, all built using same storage and replication mechanisms. Replicated, peer-to-peer applications could be a free software answer to the risk of cloud taking control of your data.

With beta2 out today, the new Midgard 8.09, or "Ragnaroek LTS" is finally coming. PHP4 has been dropped, many optimizations made, and now that Prototype has been dropped in favor of jQuery, I think we will finally be able to live with it as the Long Term Support release.

But before Ragnaroek is stable a lot of testing has to happen, as should polishing here and there. Piotras is fixing the site wizard, and I've been adding more information about the installed components.

Unlike eZComponents which are just libraries, components in Midgard are the building blocks of a website. Each of them is its own PHP application that the MidCOM MVC framework runs when a folder associated with that component is accessed. They all supply their own templates, localizations, and load whatever libraries needed.

Currently we have some 120 components in the MidCOM repository, and most of them have releases on the PEAR channel. The components range from simple things like a news listing to very specific applications like event registration or invoice processing.

One way to make components more visual was to add icons for them. MidCOM has had support for component icons for a while now, but they haven't really been used anywhere. So I made them used more widely:

Icons clarify component selection when creating a new folder:

Icons visualize how many components a developer has contributed to:

...and icons also make the localization tool more visual:

Finding good icons for many of the components is quite difficult. We're working with Kalle Persson from the Tango project to create new ones as needed.

The Free Software Foundation's GNU project turned 25 last week, and the English humorist Stephen Fry made a video to commemorate it:

Benjamin Mako Hill posted some thoughts on how the first generation of free software developers has grown:

Certainly, GNU has matured and accomplished wonderful things in last quarter-century. More importantly perhaps, it's produced wonderful progeny. It has spawned hundreds of thousands of free software projects, thousands of free or nearly-free operating systems, and an unbelievably vibrant global free and open source software community. Beyond the software realm, the free culture movement, most free licensing projects, and much of the access to knowledge movement can trace a connection back to GNU. We are living, and building, a new generation of the free software movement.

As computers are becoming more and more ubiquitous, and affect more and more every aspect of our lives, software freedom becomes a strong necessity. I personally have been involved with free software since late 90s. It gives me the operating system for our servers, our programming language, and our toolkit. It also gives me an amazing community to work and share ideas with. It is hard to imagine working in a world without them.

To support free software, write some code, use a free operating system, join the Fellowship, and celebrate the Software Freedom Day on September 20th!

I know, according to roadmap we all should be now focusing in hammering out bugs in Midgard 8.09, or as we call it, "Ragnaroek" instead of working on MidCOM 3. But Tero had a specific problem he needed to solve, and I wanted to ensure it was done right. And so I ended up adding support for context injectors into MidCOM 3.

What about contexts anyway?

Context in MidCOM is an object that contains all useful information related to the request at hand: arguments, URL, templating information. And the plan is to make it much more.

Context is the main way for controllers to find out about their environment, and for the templating system to decide which views to run and how. As we want to make the new MidCOM lean and efficient, we did not want to stick everything possible into the context. But in order to solve problems developers will need more stuff there. And so the idea of context injectors was born.

Context injectors are a new API components can use to add or modify information in the current context. The injectors will be run in two places:

• Process injectors will be called before dispatcher loads the component and calls the controller
• Template injectors will be called after controller has finished and before a template is collected and displayed

There are many things you can do there. Some immediate use cases are:

• Geopositioning injector could add user's current location to the context so that a controller can fetch nearby data
• Theming injector could switch the template entry point (in old Midgard term's the ROOT element) according to user's wishes
• Mobile optimization injector could also switch the template entry point if user is accessing the site with an iPhone or OpenMoko

Using context injectors

The way this works is relatively simple. To create an injector, you create a component and create an injector class to it. Depending which injector you want to provide, you either implement a inject_process or inject_template public method there. The method will have full access to the current context (in $_MIDCOM->context) and can add or change things there. The just declare that you have the class in the component's manifest by adding a key process_injector or template_injector with value being the class name.

MidCOM will take care of loading the component in question in the right time, and of calling the injector method. This will all happen automatically once the component is installed.

If your controller or template uses context information that is not part of the default context data, then make sure to first check that the data is available via an isset(). This way your component will be able to work around missing context information if applicable to your scenario, or throw an error.

Written outside in the Platonik waterpipe cafe in Beşiktaş, Istanbul, while vine leaves above keep the air cool enough for working. The picture is from the Freds House blog.

Google's Wikipedia competitor Knol launched a while back. It seeks to make writing informative entries more appealing to authors by offering a cut of advertisement revenue from the Knol pages. This can be a substantial thing as calculations say if Wikipedia had ads the revenue from them would be 42 million USD per year.

Having read Neal Stephenson's Snow Crash where the lead character Hiro, a fellow swordfighter and free hacker is making his living by collecting intel to upload onto the CIC library, the concept had some appeal. To test it, I created some pages:

Workstreaming
Workstreaming means collecting activities of geographically dispersed team members into a consistent news feed, enabling managers to track process and colleagues to stay up-to-date with the day-by-day happenings.

Developing P2P business applications
Moving business applications to the web has solved many issues like easier deployment and backups, but at same time introduced a single point of failure in the infrastructure. A group of open source frameworks seeks to solve the issue by helping developers to migrate their applications into resilient and scalable peer-to-peer networks.

Haedong Kumdo
Haedong Kumdo is a modern martial art based on ancient Korean sword techniques.

This should not detract from my open content activities, as the Knol pages are also Creative Commons-licensed.

Rasmus Lerdorf's excellent talk on PHP and simplicity in FrOSCon introduced me to concept of green programming. To quote the Green Programming blog:

Just like we should aspire to use renewable energy sources to help the health of the planet, we should also use reusable software elements to create robust, healthy code for our customers. Eco-friendly practices might be thought of as applicable to software devlopment. Just as we are concerned with the various biota and climate of the planet, we should be concerned with the over-all health of the software eco-system.

How does Midgard sit in with these ideas? Well, the current version of it does not. MidCOM 2 is a heavy, enterprise-oriented CMS framework that requires big machines, or even clusters to run.

But things are quite different in Midgard 2 land. We have rewritten the MidCOM framework from scratch, removing the need to have abstractions on top of abstractions to support the shift from classic Midgard APIs to the new ones, and from PHP4 to more object-oriented PHP versions. We have also focused on simplicity and adaptability, to make MidCOM an easier environment to work with for site developers.

While MidCOM3 is not due to be out before next March, Tero Heikkinen, an early adopter has been building some production sites with it already. And his experiences state that not only the performance is much better, but also it requires much less work from programmers.

I know Rasmus' argument is that frameworks are evil by requiring developers to adopt to their own concepts, and by introducing unnecessary overhead. But we should look beyond mere server-level productivity, and take programmer productivity into account. A framework makes things much easier for agile teams, enabling people with different competences to work in their isolated environments. Programmers can work on the code, and designers on the templates for instance.

While this makes the actual development and deployment process more efficient, how much performance is lost due to the framework?

Lies, damn lies and benchmarks

Like Rasmus, I used an APC-cached simple PHP "hello world" file as the baseline for my benchmarks. On my dog slow MacBook Air, the simple PHP file was able to serve 32.91 requests per second.

MidCOM3, when run on Midgard 8.09 with APC and memcached was able to serve 8.90 request per second, or in other words performed at 27% of the plain PHP file. Not too good, but considered that much SQL is run to serve a Midgard page, there are good explanations at this.

But compared to Drupal, which I did not install but instead calculated the performance based on the difference between my baseline (32.91 r/s) and Rasmus' (611.78 r/s), MidCOM3 was doing quite fine. Drupal would only be serving about 2.5 requests per second on my box. Zend Framework, again calculated from Rasmus' data would provide 6.5 request per second.

And this is not all. Piotr Pokora has done a stellar work with Midgard2, which will come out in spring numbered 9.03. With it, MidCOM's numbers were quite a lot better: 25.37 requests per second, or 77% of the simple PHP file's performance. This is actually faster than the same simple PHP file on my box without APC (24.84 r/s).

I believe the MidCOM3 performance story does not end here. Piotras has been doing experiments with caching Midgard's initial request data to reduce database queries, and MidCOM3 still misses the extensive caching infrastructure provided by MidCOM2. When these are in place, we may get pretty damn close to the plain PHP numbers.

So, with these numbers, I think we can say that Midgard2 will be quite a good framework for doing green programming while still retaining agile team efficiency.

Towards a greener future

This matters quite a lot, as then Midgard can be run with much smaller virtual machines consuming much less electricity to serve interesting information-rich applications.

When Ragnaroek gets out in end of September we will start focusing our efforts to the new platform that already is showing so much promise. A brighter, greener, more efficient future lies ahead!

Cross-posting this to Planet GNOME, as it is sure to be interesting to the community there, especially as Midgard is becoming much more than just CMS:

There is a project in works by me to rewrite Midgard’s core engine from plain C to Vala. Midgard 8.09’s code is taking huge use of GObject infrastructure and Vala is so good in expressing GObjects that actual development is much much easier.

It is not surprising that the project itself will be code-named ‘Váli’. It will include both streamlining of Midgard core and use of SpiderMonkey java script language embedded engine (courtesy of Mozilla project) to allow server-side extension of the core. Some might see these changes as an attempt to bring more web developers to Midgard’s Core but as this will anyway happen after Ragnaroek, I think its pure target is to simplify work for remaining gods on a new world for mere humans.

I think the Midgard Gathering in November shall be quite memorable, and not only because of the quantities of Polish vodka and sauna that will be present...

A thousand years ago, men from my lands traveled here in search of opportunities and adventure. Now, due to intercontinental relationships and working permits, I follow their footsteps.

While Istanbul didn't score perfectly in Monocle's Quality of Life index, I still find it a quite appealing place for a web worker. The lifestyle is more relaxed than in the north, WiFi-equipped cafes are everywhere, food is good, and the old city and Bosporus are absolutely beautiful.

Web working while running a consultancy in Helsinki is not the easiest thing to do, but with some new processes, and new tools I think we may be able to cope. The main thing is to learn using telepresence tools better than we do now.

At least the web working test run I did in London earlier this month felt very productive. Now the plan is to split my time between Helsinki and Istanbul, and of course the mandatory conference trips. This all will be challenging for regular project work, and so good web working tips will be appreciated!

But in any case, my reasons for being in Istanbul go far beyond mere working environment and productivity...

I'm again in the FrOSCon conference in Sankt Augustin, having found the conference last time reasonably big, but relaxed. I will be giving my GeoClue talk again on Sunday (16:30 in HS5).

Here are some notes from the conference:

• PHP 5.3 is going to have alpha2 next week. Namespaces could be nice in MidCOM, but I wonder how well they will work with the autoloader
• Turtle is a "friend-to-friend" framework to provide secure P2P data sharing in a trust network. Could be interesting with Midgard replication
• Apache and Lighttpd support the X-Sendfile header which we could utilize for simpler, fast file serving with Midgard
• The MINIX idea of automatically reincarnated user-space drivers and servers is pretty cool
• Lots of OpenMoko FreeRunner phones here!

Some interesting talks still left, including:

• Apache Mahout machine learning system
• dropr message queue for PHP
• Using PEAR installer for website releases

Unfortunately the Atom Publishing Protocol talk is only in German...

Yahoo! Fire Eagle, kind of "GeoClue for the Web" was released last week. It acts as a central hub collection position information from services like Plazes and Dopplr, and with a simple PHP script, Jaiku. Services needing user's location can then ask it from Fire Eagle instead of having to support all the services separately.

Midgard's positioning framework has been Fire Eagle compatible since March this year. Now that the service is open for a wider audience, I though it would be useful to tell how to activate it with your Midgard-powered site.

You need to authorize Midgard with your Fire Eagle account. To do this, go to:

http://www.example.net/midcom-exec-org.routamc.positioning/test-fireeagle.php

and follow the authorization instructions. This is due to Fire Eagle using the nice OAuth protocol.

Once this is done MidCOM cron will start tracking your Fire Eagle location. All objects you have created (photos, blogs, comments, etc) will then be automatically tagged with where you made them.

MidCOM is a PHP MVC framework where you create a site by building a tree structure and assigning components for the various folders. Each component is its own PHP application that can handle all URL requests under that folder using a set of configured routes.

In addition to making normal requests to the various routes available, routes provided by other folders can be loaded dynamically anywhere on the site. This is useful for instance for loading five latest news items to front page.

To make life easier, I now added a new feature to midcom.admin.help, the online help component. Each folder will automatically list their documentation and available routes when you access their URL __ais/help:

To get this feature you only need to upgrade the help component:

# pear upgrade midcom/midcom_admin_help

Today our ISP is having major power failures, and therefore many of our web services are down. For situations like this, it is good to establish an alternative channel of communications, and for us that is the micro-blogging platform Jaiku:

http://jaiku.com/channel/Nemein

If you are a Jaiku user, feel free to join the channel. Others can follow it via the RSS feed.

It is not only sauna, cold winters and Koskenkorva that sets us Finns apart, apparently:

The map also identifies the existence of two genetic barriers within Europe. One is between the Finns (light blue, upper right) and other Europeans. It arose because the Finnish population was at one time very small and then expanded, bearing the atypical genetics of its few founders.

New York Times: The Genetic Map of Europe (via Slashdot)

Just some pictures from happier days:

Best wishes to my friends in Tbilisi in these hard times.

The release synchronicity plan was accepted, and therefore Midgard will be switching from SVN to git, and the concept of feature branches.

With feature branches the idea is that each feature or bug fix is being developed in its own branch, and only landed to trunk (master in git terminology) when ready. This keeps the trunk clean and easy to test.

While the SVN migration will only happen in next developer meeting, you can already start using git for Midgard development thanks to git-svn. Once you have a working checkout, here is how you work with feature branches:

Ensure you have the latest stuff:

git svn rebase

Create a new branch:

git branch mynewfeature

Go to the new branch:

git checkout mynewfeature

Hack, add files, commit, test

Ensure the diff looks correct:

git diff master..HEAD

Go back to master:

git checkout master

Merge your feature branch:

git merge mynewfeature

Commit to SVN:

git svn dcommit

If you want to also work with other Midgard or MidCOM versions from SVN, check out the how to track multiple svn branches in git tutorial.

I'm currently in aKademy, the KDE conference, talking about adding the geographical context into the Linux desktop. GeoClue, our solution to this problem is built to be desktop-agnostic service, and therefore the same talk has been held in both GUADEC and aKademy.

Here are some notes from the conference:

• There was quite a good amount of interest in GeoClue. Both Plasma and Marble teams were talking with me, and I also gave an interview for Linux-Magazin Online
• Plasma is interesting tool for integrating widgets into the desktop. Being able to add widgets to the screensaver is also a great idea
• Another interesting Plasma idea is offering user widgets that are relevant for the current location coming from GeoClue. This means that when you land in London, you could be offered an Underground route planner app
• In general it seems the desktops are moving to a more service-oriented paradigm. With D-Bus your instant messaging functionality for instance can be part of any or all applications. This enables making much thinner and experimental applications that still integrate well with the desktop
• Marble is cool! With it, Qt developers can integrate a map widget into any application, making geo-aware tools much more visual
• openSUSE build service is about to get new features that will help generating virtual machine images
• The beer evening in the Het Anker brewery was a lot of fun. Thanks, Nokia!
• Speaking of Nokia, they are here really in force. They are the main event sponsor, and have a big bunch of people attending. Their acquisition of TrollTech has really changed the situation. Hopefully they'll do the right thing about GeoClue and Maemo

More stuff in my Jaiku notes. My GeoClue slides are also available.

At the moment the prevailing wisdom is that each CMS should have its own user interface, and that user interface should be web-based. But there is also another way: separating the user interface from the CMS using a CMS-neutral protocol called Neutron.

According to Sir Tim Berners-Lee, the earliest web browser was also an editor. And the late 90s Netscape Communicator followed this ideal by including a HTML editor that could publish changes to pages using the HTTP PUT method. But then the idea of editing via the web browser transformed to clunky forms and Java applet -based WYSIWYG editors, brought about by the rise of content management systems.

The problems of the user interface being part of the web page were multiple: it cluttered the produced HTML, it was possible to break by layout changes, sometimes the login and editing options were hard to find. It also meant that each and every CMS would have a completely different user interface. A problem made especially difficult if you had to use multiple systems. For example:

If you are Quim Gil, working for Nokia's maemo.org, you're living in a world of many, many CMSs. The marketing and community parts of the website are run by Midgard, the documentation wiki by MediaWiki, your own blog by WordPress, and the list goes on. While in this kind of corporate setup it has been possible to mostly unify usernames and passwords, it still means each part of your work runs with different UIs, and different usage logic. Back in 2003, I named this syndrome Frankenstein CMS.

In addition to consistency and usability, offline editing has been a big issue with most CMSs. In typical situation, it simply doesn't exist, making copy-and-paste way of taking documents with you to edit on a laptop while traveling.

There have been several initiatives in solving these problems. A quite limited, but so far successful example is the Universal Edit Button specification, making rounds earlier this summer. The idea is that CMSs would include metadata on where the editing view of a particular page was in the page itself, and then browsers would display a button leading to it. The approach has been adopted by several big players, including Wikipedia, and we also made it a part of the new CMS that will be built on Midgard 2:

Another, also somewhat limited example is the MetaWeblog API provided by most blogging platforms. It has enabled vendors to produce great offline blogging software like Ecto and MarsEdit that provide offline editing and work with almost any blogging system out there. As I write this, I'm using Ecto on my MacBook Air, and will later use it to publish this entry to my Midgard server. The Atom Publishing Protocol will likely be the successor of MetaWeblog, but appears to still keep the blog-only focus.

Back in the earlier days of OSCOM we had another initiative called Twingle. It was a XUL-based desktop CMS client that utilized WebDAV and some XML introspection files to edit and publish data on different CMSs. After a March 2003 OSCOM Sprint in Zurich we were able to demo the same client browsing and editing resources on three CMSs, including Zope and Midgard. Unfortunately, then CMS market became so hot that vendors were simply too busy to pick this up and the project died.

Luckily Michael Wechner, of OSCOM and Apache Lenya fame didn't let the idea die. He worked it onwards, naming the protocol used for conversation between the client and the server Neutron Publishing Protocol, a play on Atom. He also built a new client, Yulup, as a Firefox extension that could manage content on compatible servers.

In summer 2007 I saw a demo of Yulup and discussed Neutron with Michi and was impressed. But was back then too depressed by the Czech episode and other personal issues to press on with the idea. Having watched another year of the directions the community is taking Midgard CMS to, and how clients view CMS deployments, I'm starting to think the time would be ripe for going full forward with Neutron.

In nutshell Neutron is a metadata layer that allows a CMS to specify the actions user can take, and the methods provided for those actions. The methods can either be full-blown WebDAV, or simpler GET-and-POST that more limited CMSs can support. Neutron also provides its own authentication mechanism, though I would gladly see that overtaken by a more secure and widely supported standard like OAuth.

Widely-supported Neutron would provide huge advantages: letting the developers of a CMS to focus on the actual management and presentation layers of the system, and allowing more innovation to happen in the client end due to combined efforts from multiple CMS projects. It would allow building of different content management interfaces for different audiences or usage scenarios. Simple editing and publishing clients that could run fully in browser's AJAX space for casual bloggers or wiki editors, workflow tools for gatekeepers of corporate publishing processes, and full-blown desktop content management tools for site editors. Other special cases like mobile content management could also be provided for.

So what is needed for this future to become reality? First of all, we need to make CMS developers aware of the possibility. Then we need a killer client that every CMS provider would want to support. And then we can expand the protocol, and build additional clients to cater for special needs.

The big question is how this will happen. Industry groups in CMS space are loose or non-existing, and so they lack the muscle to push any standards. Effort by single CMS vendor would probably stay partisan, as developers of different systems are quite suspicious or ignorant of each other. But maybe a combination of the two would work: industry group, such as OSCOM organizing some compatibility hacking events for the developer community, and a company dedicated at building a killer client application. It warrants serious consideration if Nemein should be that company.

This manifesto on transforming how CMSs are used was written in an Eurostar London-Brussels train, sipping Dourthe's Bordeaux white and listening to the Karelia cycle by Sibelius.

ididwork.com, a simple workstreaming service has launched this week. Workstreaming means utilizing micro-blogging services to keep your colleagues aware of what you're doing. This helps with coordinating geographically dispersed teams of web workers.

As part of redesigning our processes, our company has been looking for a workstreaming solution. While private channels support in Jaiku would probably be the best option, we will now be testing ididwork.com for our needs.

Here is a quick overview of the first launched version of ididwork.com:

The good:

• Simple, pleasant user interface
• Tagging support (including machine tags), which means we can connect entries with Ajatus tags
• Email reminders for managerial review of entries
• Excel export of work reports

The bad:

• Users can be either managers or employees, not both. Managers don't have work to report?
• User's own log entries and log entries of colleagues are shown separately
• User can only belong to one organization. No multiple teams or projects

The ugly:

• No API
• No RSS feeds
• No XMPP support

ETech 2009, the O'Reilly conference on emerging technologies now has a call for papers. Here is my proposal for the "Nomadism & Shedworking" track:

The recent direction of business applications has been centralization to web-based systems, easing deployment, upgrades and management of application security. However, at same time centralization provides new risks like the introduction of a single point of failure for application usage and in most cases inability to work offline.

At the same time, working culture has started so shift more towards networked individuals, or "web workers" forming ad-hoc coalitions to work on various projects. For them, centralized applications might not be desirable due to data ownership and infrastructure requirements.

Moving the applications from centralized web servers to a peer-to-peer network allows web workers to be in control of their own data, stay productive even in unstable connectivity situations, and collaborate easily with their colleagues in an ad-hoc way.

This talk outlines some ways to move forward in building peer-to-peer networked business applications. There are many open source frameworks targeting the problem, including replicated databases like CouchDb and Prophet, and application-oriented P2P networks like Swallow/DBE. These will be discussed together with some real-world examples of business applications built with them:

Ajatus - a "Personal CRM" built on top of CouchDb, a RESTful replicated object database
OpenPsa - project management system doing resourcing over the DBE P2P network
Simple Defects - a P2P bug tracking system built on top of the Prophet replicated database

The first phase of P2P business applications will likely be services operating in closed networks of users' social networking or instant messaging contacts. But the P2P model can also provide opportunities for wider networking, making it possible to find new project partners or collaborators anywhere in the world. This wider-ranging business network will introduce new challenges like security and reputation management. Some ideas related to this will also be discussed.

Other interesting projects in this sphere I did not mention are Telepathy Tubes and F2F. We have also had some ideas for how Midgard could do this...

On Saturday I was in Ballina, Ireland, and a bit nervous: there was a Haedong Kumdo seminar in town, and I was scheduled to test for black belt. Behind this was years of different sword arts, like viking swordfighting, Iaido, Kendo, sabre fencing, and two years of quite intense Haedong Kumdo practice.

We tested together with two Swedes who already held black belt in the other federation. All three passed.

So what is Haedong Kumdo?

Haedong Kumdo is a Korean martial art based on the use of the Korean curved single edged sword. It is a modern system compiled from the ancient traditional sword techniques of Korea.

The movements are primarily circular and derive their power from centrifugal force of the body. Training develops strength, coordination, balance, focus, stamina, speed and flexibility.

Haedong in old Chinese language means Korea (Hae - sea or sun, Dong - east). Kumdo translates to way of life through the art of sword (kum - sword, do - way of life through martial art).

The Way of the Eastern Sword, Haedong Kumdo, is an exciting and dynamic martial art.

This important step done, the practice will intensify. Next I will need to focus on two-sword techniques and the Yedo series of forms. There will also be more cutting practice. In September we will go to Korea...