Site news

It has been a while since I've posted an update on localize.drupal.org services for various reasons. I greatly hoped that our Google Summer of Code project for activity tracking announced with big fanfare in May goes through well, but unfortunately my student dropped out from the program lacking time due to family reasons and did not make useful progress to roll out improvements.

Also, Dries selected me to lead an effort to dramatically improve language support in Drupal core for Drupal 8 and that has a scope way beyond just software translation. We did a pretty good job of providing tools for software translations so far with localize.drupal.org, l10n_update and l10n_client modules, but for Drupal 8, the focus is to bring complete core features and more usability to these and content and configuration translation alike. It is very exciting and going to bring Drupal to a whole new level where language support is needed! To follow that effort, see the video and mindmap at http://drupal.org/node/1260534. It is a huge undertaking and it required me to dramatically dial down my focus on localize.drupal.org improvements.

However, in the meantime, our misterious problems with some packages being partially parsed did not solve themselves, and we could not figure out the real reasons, so Zoltán Balogh got tired enough of the problem that he coded a workaround for every team manager to be able to request a release to reset itself and parsed again. That was just rolled out, so now every translation community manager should be able to see a Start over link on project release lists, with which you can request for a broken parsed release to be parsed again. Because our release parsing problems are transient, this should get you back a fully parsed release pretty soon afterwards.

There is also continued interest to set up new teams on the site, I've just added Oriya, Lao, Azerbaijani and Tamil (Sri Lanka) to the list of languages, which is now just one shy of a hundred (not counting our test language playground). That is an amazing number! Get ready to pop your champagnes! Who is going to be number 100?

Finally, Dashamir Hoxha did a stellar job to come in fresh to the Drupal community and update such a huge module as the Localization server module to Drupal 7! Yes! It passes all tests (although the test suite does not cover all of the functionality, it does cover all the basics and a sizable part of what localize.drupal.org needs). It would need more work, but we are a lot closer to be able to update localize.drupal.org to Drupal 7 in the future.

Now with all this while there are very nice prospects for localize.drupal.org, I will likely not be able to dedicate a good amount of time to the site, so I decided to post a public call for help instead, while I still have resources to help mentor people who would like to contribute and possibly eventually take over maintenance and improvement of the site. For a long time, localize.drupal.org was the posterchild of great improvements on drupal.org, now that all the awesome improvements are going on on drupal.org we should not leave localize.drupal.org behind either. Are you interested in working on gradual improvements? Porting the site to Drupal 7 (og, messaging and notifications and all that)? Comment and let's see how I can help you help Drupal!

At the end of March, when the Google Summer of Code 2011 Drupal group was in an idea collection phase, I've posted two suggestions: Expose Drupal translation activity and Gettext API for Drupal contrib and core. The first one involved tapping into the huge hidden activity that happens on localize.drupal.org. To quote from my suggestion:

Due to technical and scalability reasons the site uses custom code to provide translation of 7000 modules and about 20000 releases of these modules. Each release is parsed and ends up in the database of the site which results in almost 270000 strings to translate. That's a lot of stuff! Now add up that we have almost 100 languages and each string can have any number of suggestions for any language. Finally, each of these suggestions have their own "commit" history with data on who submitted them, when, via which method (multiple sources supported with a remote API in the mix), when was it approved, declined and approved again by whom and when, and so on. So you can consider this a 3 dimensional version control system in a database (version control per strings * languages * suggestions).

Now since this is all stored in this form, the activity happening is pretty "invisible". Moderators and fellow translators see little of this activity. Some teams use a wiki page to list things they work on, informing moderators that new stuff is available to review. Without that, reviewers can only look actively for stuff to review, which is tedious. Its a definite characteristic of communities online and offline that seeing the momentum, the activity and work being done energizes others and moves them to contribute more. So exposing all this information in natural and automated ways is crucial.

I'm happy to report that the immense task to expose all this activity with activity streams and graphs was taken on by a Hungarian student named Ádám Lippai, who is going to work on the project this summer. It should be an interesting mix of working on the drupal.org testing/staging environments, Hudson, our localization data storage and exposing the data changes in a meaningful way on the UI.

I'd like to thank both Ádám for signing up and the Google Summer of Code program for financing the project. This should really help take localize.drupal.org to another level.

Ps. Check out the other accepted projects at http://drupal.org/node/1139168 - note that we also have a project to work on activity exposure on drupal.org projects, so there'll be even more of this type of work on the site. Also, I'm still looking for people to work with on Gettext API for contrib and core (although no financing attached), as it should be part of efforts to improve multilingual support even further in Drupal 8.

There has been tremendous growth on localize.drupal.org lately thanks to new teams launching from all over the world. The latest ten teams launched are: Maltese, Uighur, Occitan, Bengali, Malagasy, Scots Gaelic, Sinhala, Tigrinya, Kurdish and Kyrgyz. We are approaching 100 teams fast with 94 translations projects in the system at the moment (including our testing group). There are also many more in the queue being discussed.

Since our last usability update in January, there have been a few subtle but hopefully helpful changes on the site, including the following:

1. Direct linking to source strings is now even easier. The # marks that were previously hidden by default are now displayed as "list item markers" in the filtering list results, so they are a more natural part of the page. Just clicking them gets you the permalink for translations of the given source string.
2. There were also some improvements in the backend. We've prepared the file packaging system to be able to provide progress status of their translation readiness at a later point. This also helped debugging some Drupal project release parsing issues.
3. Partly as a result of those backend improvements, we now have a new release listing page for each project. See http://localize.drupal.org/translate/projects/views/releases for example. This exposes some previously internal data such as when did we pull in the release for parsing, how many files and actual source strings did we find in the release, and whether we found warnings in the releases. You can also look at details of a release, review the files we parsed and the warnings we found.
4. The front page got some great new possibilities to order the Drupal core project status by different criteria. It is still ordered by language by default but it can be ordered by progress or contributor count. The contributor count only includes people who actually contributed to translations in that language (vs. members of the group). The language overview page got the same treatment and both pages got some better introductory help.
5. The project listing page got alphanumeric filtering possibilities and is on its way to be simplified and get actual sorting support at a later point. It is not made up from one SQL query, so not exactly trivial to dress up with sorting features (much like the language listing pages), but it is going to happen.
6. Projects removed from drupal.org or those which never had releases are not displayed in the project listings from now on. This was a long overdue bugfix and should make project browsing more meaningful.
7. Translation export pages got lots of attention and are much more simplified. Archaic export options for CVS commits, or Drupal 6 or 5 structured packaging are gone. Remember to use l10n_update module for best experience downloading translations for your sites.
8. Finally, you can actually trust the pager on the translation pages now, it will count the number of source strings to display not the number of translations and suggestions matched by your filter.

I hope these usability improvements make it easier for all of you to use the site day by day. As always, see the issues page to submit bugs and requests.

At the time of this writing the Drupal.org git migration is still fresh, and I expect project maintainers and especially translation maintainers to possibly wonder about one thing. Sure, CVS ID strings were removed from all files with the git migration, but one more thing happened: all translation files were removed from drupal.org projects. As it was announced in the Drupal.org Maintainers newsletter over two months ago, translations are now to be contributed to localize.drupal.org. The drupal.org projects themselves should not have .po files committed and maintained, as that would duplicate the work going on at localize.drupal.org. The files were removed in an automated commit on each project branch by the Drupal Git User with the comment Removing translation directories. (Therefore the files are available for reference in the history).

Project maintainers should not accept .po files anymore for inclusion with modules or themes, localize.drupal.org builds these for download, and Localization update can be used to automate the download and even the update of these for releases of modules and themes. There are various advantages to this including the possibility to release and fix translations after a module release goes out, making downloads smaller and only grabbing the languages you actually need.

Translation projects themselves are kept (such as the 6.x-1.x branch of the Hungarian project). These projects should also move to localize.drupal.org entirely, but in the interest of them being available for reference (not just in the CVS read-only backup), the projects are migrated to git. These are however also to be considered as reference and not to be worked on in git.

Any questions about this transition? Let me know in the comments.

One of the goals of localize.drupal.org is to provide a simple UI on top of a complicated process: translation submission and approval. The system handles strings in three states: suggestion (pending approval), translations (which are approved) and declined strings (which were deemed to have bad quality). For the sake of approaching this as simple as possible, the backend focused on the submission and approval details so far, and neglected other transitions. However, in practice many limitations became evident.

Drupal 7.0 just hit the streets earlier today. This is the first time many people will face that the localization setup and update process is entirely changed (although it is not just changed for Drupal 7, also for older versions). Now translations are managed on localize.drupal.org, not as drupal.org projects, and localize.drupal.org generates its own translation distribution files for ftp.drupal.org, which are not included with drupal.org project releases (and are not going to be so). While this makes it easier to install Drupal 7 out of the box localized (you only need to copy one .po file instead of unpackaging multiple .po files in a funky structure), it makes it much more complicated to get and keep your contributed module localizations up to date on your websites.

Fortunately there is a great solution for this. Localization update module works similar to Drupal's built-in Update module in that it identifies the version of projects you are running with. But it uses that information to let you know if you have the latest version of translations for these projects, and downloads translation updates for you automatically. (It also supports custom modifications to community translation, and keeps track of those).

What's even better is that we have all built into one Drupal distribution now! The Localized Drupal distribution is built with Localization update module (as well as Localization client) to make all the automation around the new translation system available for you out of the box. I must say the easiest way to get started with Drupal 7.0 now if you'll localize your site right away is to go with Localized Drupal. It is still in beta for Drupal 7 due to the state of advancement of the two contributed modules bundled (which do need more testing and feedback), but in our testing, it works pretty well.

Don't miss the wave, get going with Drupal 7.0 today, localized out of the box! Please provide any issues you find in the issue queue for the distribution. Thanks!

Ps. This distribution also makes it very easy to contribute to the translations as well!

We are happy to report that the session I (Gábor Hojtsy) and Amir Helzer submitted for Drupalcon Chicago on Drupal 7's multilingual system is going well with the votes. We'd love to present you all the new stuff that was and is being developed and how the new pieces map together, how does the new Drupal version as well as rethought contributed modules solve multilingual issues. I'll of course keep you posted on localization improvements on this blog, but there is nothing like a well thought out overview where you can even ask your own questions. Therefore we'd love if you could add your support as well, so we can be sure to get this topic on the map for Drupalcon. (Make sure you buy a ticket to be able to vote). Thank you!

We rolled out the entirely end user oriented translation downloads functionality almost three months ago. This time was enough to gather some feedback, look at the new functionality from a more objective standpoint and tweak it to user needs. With localize.drupal.org neatly integrated into drupal.org, it is clear we should put our download functionality first and foremost (as drupal.org projects do). Also pointing into this direction lots of feedback that I got from users and translation teams alike that the duplication of localize.drupal.org teams and drupal.org translation projects is confusing. Therefore I set out to implement some improvements in this area.

Translation teams are moving to localize.drupal.org

There are not many teams not yet migrated to localize.drupal.org, and new teams are only accepted on localize.drupal.org (not as drupal.org projects). We are phasing out the use of drupal.org projects for translations. Why? People found it confusing that some teams also still use the CVS space on drupal.org for translations, and when you look at the Drupal 7 translation status on drupal.org for example, it really is not representative of the work of all the teams. If you look on localize.drupal.org, numerous teams are close to completion, and their downloads are available and ready to use. Doing away with the need to commit .po files to CVS was in fact one of the driving goals of localize.drupal.org. Once we don't do that, all is left for the drupal.org projects is the issue queue. However, that is highly disconnected from the translations themselves, and we could maybe do that better on localize.drupal.org (see below). All-in-all, we are moving all teams to localize.drupal.org now, and you should not keep committing files to CVS anymore (a detailed blog post about this detail coming up).

Related issues on drupal.org: http://drupal.org/node/980658, http://drupal.org/node/980682 and http://drupal.org/node/980686

Usability improvements deployed

There are various interesting changes to the organization of the site, mostly on the welcome and overview pages. Localize.drupal.org got an all new and shinier front page. It now starts off explaining what this site is about and suggests some Drupal software to help integrate yourself with these services. We are back to using the Drupal translation progress stats here, as was requested by many people insted of a giant download table. However, once you pick a language, you get to experience the new drupal.org project like team frontpages.

In an effort to make translations more familiar to people browsing drupal.org projects, we made the translation pages more like drupal.org project pages. The team welcome message is displayed prominently (like drupal.org project descriptions). Team managers can edit this text by editing their group. Then we have Drupal translation downloads and downloads for further 20 top Drupal projects. Localize.drupal.org now knows about the usage stats for projects and picks the top projects based on that. We plan to use this data for other usability improvements, but only introduced it here for now.

The posts that are targeted at contributors now moved to their own tab, called "Board". This is quite a bland listing of posts in the group now, and we plan to come back to this soon and improve on it immensely. Your input is welcome there! We enabled all group managers recently to administer posts and comments in their group, so you can now make posts sticky, remove posts and comments, etc. This was a much requested feature from team managers. However, we still only have "story" posts and wiki pages for teams. I think introducing two new ways to organize your communication would be in order:

• Books! Using books, you could present documentation for newcomers, post your guidelines (even as wiki pages), and structure that in a hierarchical way, if you prefer this organization to wiki networks.
• Discussions (or issues)! Ways to organize your team work on a day to day basis. Think things like assigning work to certain contributors, discussing wording changes, etc. I'm thinking issues would have a status, a person assigned and maybe priority. However, most importantly it would be ideal if we could link issues to strings or sets of strings. Requirements and ideas on how should this be best implemented are welcome.

So the board is supposed to house these four types content for you and let you jump out to review and manage assignments, documentation, etc. That is why we called it a board up front, even though it needs some work to become true to its name.

Finally, there are various smaller usability improvements, like the "join" links on groups now prominently highlighted to drive in contributors, the top contributors becoming a table instead of a bland list, etc.

Help shape the future of localize.drupal.org. Tell us what you think!

We started localize.drupal.org back in August 2009 with the intent to provide a more user friendly tool for Drupal interface translators. Many people were frustrated in the CVS account application queue, in the module issue queues trying to get .po files committed, on their own system attempting to generate new translation templates, and so on. Therefore we attempted to automate that all in localize.drupal.org's service.

Initial uptick of translation teams was definitely impressive, and the growth in number of teams slowed down a bit. This is not surprising, the team that were active before are almost all on the central server now and we even enabled lots of new languages to start off. We are at 74 teams now. Let's see some of the details!

One of the great advantages of Drupal translation is that the modules and themes share the same text space. So if a module has 'Home' translated, it does not need to be translated separately in the other 23 modules you use on your site. One of the disadvantages of that is that Drupal does not know/care about the module the string is coming from, so if you have short strings which can mean different things based on their use, Drupal is not capable of translating them differently. At least not until Drupal 7!

Let's take a simple example: "View". This can mean a lot of different things. A view as in something in views module, a view as in an SQL database view, a view as in "how nice is the view from your window", or view as in "click to view this post". One long standing example in Drupal core is "May". Drupal has strings for both long and short month names, like January, Jan, February, Feb, etc. But May is the same for short and long. No difference.

Well, Drupal 7 introduces string contexts to solve this problem. For "May", it adds the "Long month name" context, so we know if its not the normal short month name (which lacks context), but the explicitly told long month name. The way this looks like in code is as follows: $long_may = t('May', array(), array('context' => 'Long month name'));

You can read more on the API page for t() and friends (this syntax can be used with other localization functions too), but the general concept is really simple: you elaborate on the context of the short string being used. Contexts are to be used for places where the string itself is too short to give enough context and could lead to mistranslation, or can have multiple translation in different uses. It is not to be abused wholesale for module strings, but only to be used after consideration.

Now contexts are something developers need to introduce in their modules to help translators. Only a few contributed module authors jumped on the bandwagon so far, and maybe not all of them use the API correctly yet. Good news is that Hilde Austlid (zirvap from the Norvegian teams) started a great practice of tagging related issues with "string context", so we can track issues across projects which deal with string contexts.

I'm hopeful that contexts will be a great way to solve translation conflicts, when short strings are impossible to translate in different ways otherwise. We should try to do our best to use it well and not to abuse the great powers we got though. Establishing guidelines as to when to use contexts (and what kind of short strings to leave context-less - such a short month names in core) is still to be done.