Localize.drupal.org growing, easier import screen and some bugs fixed

I'm happy to report that on http://localize.drupal.org, the past month have seen British English, Estonian, Filipino, Finnish, Lolspeak (yes, lolspeak), Persian (Farsi), Punjabi, Romanian, Slovak, Slovenian and Tamil added; growing the list from 33 languages to 44. We've also grown to almost 500 contributors which means about 11 contributors per language on average.

Focus on the past few weeks was on improving the user interface and submission feedback on imports as well as adding test coverage for imports and fix some odd bugs found by active users. The freshly deployed new import screen is now much more usable, showing what will exactly happen when either option is chosen:

Improved import screen

We've seen people having problems with the attribution and status options before, given that they were checkboxes assuming a default behavior from where one could divert. Now all paths are clearly explained, so those importing .po files should know exactly the results to expect.

We've also worked on fixing bugs. Claudiu Cristea was very active submitting suggested fixes which got refined, committed and deployed. Thanks to those efforts, the translation view screen now handles plurals properly, languages with more then 2 plural variants got the plural string copied to those further fields as well and the moderation screen now properly remembers the selected filters on form submissions. I've also worked on signaling errors for modules which have empty strings to translate.

Work is still ongoing towards an improved translation interface, first breaking out more granular permissions for moderation. Help and testing is - as always - welcome!

Your feedback requested on a proposed new translation user interface

While Gerhard Killesreiter is working out solutions for queued .po file imports (to work around the relatively big resource needs we have on an import), and Jose Reyero is working on building translations to actual .po files which people (and automated clients) can download, Konstantin Kaefer was busy with the user interface of the module.

Let me focus now on Konstantin's ongoing work, because it is in need of your feedback. A major update to the translation user interface like this should be done with your feedback heard. So I've studied the new features (and Konstantin's previous video) and made a demonstration video with the up to date functionality.

The new approach introduced treats translations and suggestions more on the same level. So when a string has a translation and outstanding suggestions, you can review them on the same level. The advantage for translators is that they can see previous suggestions and take them into account when composing their suggestion. Duplicates (which are catched by the server) would not take time to write anymore. Also, the active translation and the outstanding suggestions can be viewed in a diff view (when hovering over one of the strings, differences to the others are shown). This helps reviewers to figure out actual changes in relatively long strings. There are also some other niceties with the diff coloring, since it includes marking HTML tags and their closing pairs as well as placeholders. So you can match the placeholders from the English original text to the translation or between the different suggestions. Treating outstanding suggestions and translations on the same level allows you to filter for all of them at once (with the contains and author filters).

The approval workflow also changes with the patch. When you approve a suggestion to be the active translation, you don't need to disapprove all other outstanding suggestions for the same source. You can pick declined suggestions one by one or actually disapprove all other strings by approving the best translation with a double click. After saving your changes on the form, you cannot get back the declined suggestions, but before saving your changes, you can still undo your decisions.

Check out the video here (or as hosted on vimeo) and provide your UI feedback on the issue at http://drupal.org/node/563128

Opened up, added wiki & more formatting, fixing imports and better remote submission errors

Through the previous days, several small to big changes happened on localize.drupal.org, so I thought it would be a good idea to write a quick post summarizing the changes.

How did you implement your wordlists, what are the requirements?

Many people brought up the need for "wordlists" on localize.drupal.org in the past. These would be (maybe not so) mini dictionaries per translation team, so people coming in would be able to quickly accommodate themselves with the terminology used, without the need to infer that from other translations. The basic form of this would be a table with English and suggested translation words. While this can just be a full HTML input format node (to let you put in a table), ideally we'd reuse this data to actually help translators.

Deployed performance optimizations and more caching, long way to go still

Thanks to many hours of work from Jakub Suchy and myself, I've integrated and deployed many performance optimization and caching improvements on localize.drupal.org tonight. The front page and the main explore pages (explore languages and explore projects) are now using fully cached data, so will load way faster then before. This is on the expense of absolute accuracy.

A little look behind the scenes of the localize.drupal.org team creation process

We have seen certain teams eagerly pushing for their groups to be created on localize.drupal.org as soon as possible, no matter what. I've just received a personal contact email which even included this remark about the process, so I decided to repost the reasoning here for the public to read and hopefully understand.

[...] I want to know why this process is so ugly - we need to make some bureaucracy for adding active projects into drupal growing process

I don't think it is ugly, but I agree it might not be clear. We need to ensure that other random people speaking your language are not taking over your existing fine work. (Ie. they do not open a new team without consulting the existing project/team maintainers). So we need to ensure that whatever previous work was done in the given language is cleared in terms of cooperation, so we are not opening some new parallel effort. This is hopefully ensured both by looking for existing teams and asking for their feedback and by letting the issues sit for some time for others to find out about what is going to happen.

Also, similar issues apply to some people having tensions inside language teams and wanting to fork translations. While this happened before with multiple servers per language with overlapping projects or with one server for core and another for contrib translations. We are actively discouraging that and trying to get people under the same roof. We are providing a (hopefully) good infrastructure for people to just concentrate on translation, so that they don't need to suffer from the maintenance overhead.

Those creating a team should understand that this will be the primary source of their translations and they should not do work in CVS without adding to the web service too and they should not operate a localization server elsewhere with the same team for Drupal.org projects. Many of the teams run localization servers before and we need to migrate that data, which takes some time given the accumulated suggestions on those sites, for which we have no tool to import. The issues are used to make sure that those, who open the teams understand these goals and directions and go with them.

Finally, we are beta testing, so the less people experience big issues the better; but we need people to experience those issues for them to come up and get fixed. Those starting early on the localize.drupal.org site naturally experience more issues then those coming later, since the base issues should be resolved by then. For example, we expect to roll out some performance improvements later as performance issues are starting to show and could get painful if we open the gates freely to any number of translation teams.

As with new software, it is not about secrecy or exclusiveness, it is about doing things right, so that you'll suffer less. You should not feel shut out if your team is not created within hours of your initial issue being submitted, there are teams waiting for up to a week already. Please be patient as we go through the process, it is to ensure that your existing contributions are valued and that you can effectively work on the localize.drupal.org interface when you get there.

Some of the reasons above will apply later as well, so I'm not expecting that the moderation based group creation workflow will turn into a first come first served free-for-all process in the long term either.

"Here comes localize.drupal.org" session videos

The fine folks, many of whom also did the Drupalcon Szeged video recordings were in Paris to do the video recording for that conference too. Finally, my session video is online, so you can watch most of my session about localize.drupal.org hosted on archive.org. It was uploaded in two parts, so to watch from (almost) the start to end, watch them in succession. To download in different formats, go to http://www.archive.org/details/Herecomeslocalizedrupalorg

I've posted my slides previously at http://acquia.com/blog/drupalorg-steps-helping-localized-drupal-installs... and http://localize.drupal.org/node/119

New filtering options and moderation functionality on localize.drupal.org

Thanks to the diligent work of Jose A Reyero and myself, I'm happy to announce that starting from about an hour ago, we launched new filtering options and a moderation screen for easier translation management.

New and more granular filters

Over Drupalcon Paris, we added and launched more granular status filters, which allow you to separately filter for translated/untranslated strings and ones with suggestions or lacking suggestions. These were in one bizarrely integrated filter before, and now are separated.

New filtering functionality

Today, we also launched two new filters on the Browse and Translate screens. First up is the author filter. That allows people to look up translations submitted by a given user on the site. This also works for suggestions which were approved unedited, because that is directly attributed to the submitter as well. Note that since the Browse and Translate screens filter by the translations themselves, the added filter does not allow filtering for strings which have suggestions submitted by a given user. Just like the "Contains" filter only looks at the source string and the active translation, but not any of the suggestions.

The other added filter is a new limitation capability. Now you can limit your workspace to less or more strings, so if you are unsure about your internet connection, you can work in smaller chunks, or if you are a translator hero, you can churn through more strings at once. In the plans are a more specialized UI for the case when limiting to 1 string at once (not among the limit options yet), which would help you achieve better translations faster.

New moderation functionality

Thanks to previous work done by Jose A Reyero, we have new moderation functionality for suggestions right at your fingertips. While the same filters apply, these actually look at the suggestions and not at the translations, so using the "submitted by" filter will look at the authors of the suggestions in this case, or using the "contains" filter will search in the original string and the suggestion. So these filters complement the lookup functionality of the other screens and also provide bulk actions to work with suggestions. If a user accidentally imported .po file with different language translations, you can filter those out quickly here and decline those to clean up your database.

New moderation functionality

This was among the top feature requests from people, and many translation teams just solved these mass-moderation issues with SQL queries on their own localization servers. Not having such functionality was a blocker for established teams to move on the central server both due to not having a tool to clean up their suggestion queue to import fresh to the central service and to manage new suggestions onwards on the server. I'm hopeful that this new functionality will both help people to migrate and existing teams to handle the amazing number of suggestions cropping up for the existing teams.

Tested backend

Because this is the first huge public install of the localization server, it was important for the sustainability of the development to have automated tests. There are many interesting improvement waiting in the issue queue, but sometimes it is scary to commit changes which modify the database or the UI in big ways, since there are not many people testing these patches. Therefore I went ahead and took a lot of time to write automated tests for the underlying code.

In fact, I went to write tests for the Translation template extractor powering the source code parsing functionality of the backend, now covering Drupal 5, 6 and 7 source code parsing. The module supports parsing Drupal code on all three APIs, but there was no automated tests to verify that. Now we have 70 passing asserts to verify parsing for all three major APIs.

Testing of the localization server itself was a much bigger feat to be honest. The test written reuse the source code from the Translation template extractor tests, package them up into tar.gz packages disguised as Drupal modules ready to be parsed, even with different releases. Then all of them are parsed up and checked for their appearance on the database/output. The tests add suggestions, verify the individual and mass moderation screens and filtering by different criteria, including all the new features mentioned above. Direct translation submission is also covered. The tests now have 634 passing asserts, and do not include testing of imports, exports and translatability error listings yet.

All-in-all, the new testing setup helps us ensure that big changes like the above get in well-tested. I've found one bug in our existing code and four bugs in the patch before being committed due to test coverage for its functionality. After all users of the online service benefit by getting tested new features to be more effective in their work.

A shout-out for those of you waiting in the l.d.o issue queue

We already have 18 translation teams set up on localize.drupal.org and more then that waiting in the issue queues. I'd love if those waiting in the issue queue understand that our current phase allows us to get the existing teams find issues so teams coming later will suffer less. We also need to clear off previous contribution histories, so teams starting off on localize.drupal.org inherit the translations from CVS and previous translation servers and not start over from scratch. We believe it is important to build on previous work and ensure that we have one definitive source for translations, so users do not need to hunt around the internet to find the latest and greatest translations for their modules and themes.

Localize.drupal.org launches beta test

I'm happy to announce that a new service launched beta testing on Drupal.org under http://localize.drupal.org/ to help translation teams involve more people and reach new heights in the level of Drupal.org project translations. This central web based translation service is about to step in place of generating translation templates manually, committing them to CVS and releasing them with projects waiting for teams to picking them up and then committing translations on their behalf from the perspective of the module and theme maintainers.


Subscribe with RSS Subscribe to RSS - Drupal planet