Neil Drumm

One of the Jenkins jobs is much faster,

The total time is still up to 45 minutes; this runs every 30 and Drupal cron every 15. This can all be more flexible, more standard, and probably faster with borrowing some code from localize.drupal.org to use tgz downloads instead of version control.

We have a new Drupal.org Redesign initiative page, a good way to organize issues. I still have a lot of work to do filling out issues. This should be best place to track progress and get started.

The infrastructure team and myself have been working to improve the staging site, redesign.drupal.org, access to code in BZR, access to a database, and even complete development environments.

There is still plenty of organizing to do; I expect volunteers will be able to be immediately productive next week.

We have a new staging site at redesign.drupal.org. There are a lot of rough edges. It will be a good way to see our progress. As much as possible, it is automated; theme changes every minute and database changes from drupal.org every day. More on the infrastructure is in the handbook.

I’ve been busy collecting everything that has been done so far; making sure everything is either committed to the correct place or has an issue on Drupal.org, including work from 10 SVN sandboxes. Issue queues still need review. If you would like to hear an audio update, Lullabot did a podcast with Git migrator Sam Boyer and myself.

I’ll be looking for help with:

• Theme
• Drupalists
• Dashboard content and implementation
• Home page, including the map
• And much more

The staging site for API.drupal.org is ready for testing. We have added the theme from Drupal.org’s redesign and documentation from classes. Behind the scenes there is a new PHP parser, project support, and automation with Hudson. This is a good test case for the latest Drupal.org infrastructure work, and has been great to work with. Please note problems in this issue queue.

I’m pleased to announce that, starting today, I am starting as the Drupal.org Redesign Architect, joining Git migration lead Sam Boyer and 3 more forthcoming positions. My responsibilities are:

• Work with the Drupal.org redesign team, including infrastructure team, theme maintainers, project module maintainers, implementers and project managers.
• Create an implementation plan for each section with functional descriptions and user stories of the work to be implemented so teams can complete them. Includes priorities and rough time estimates.
• Manage the deployment of each redesigned site.
• Provide regular updates to the community.

I’ve been volunteering on the redesign implementation since the start, but I haven't been responsible for high-level planning until now. This week I’ll be jumping in to see where everything is and what I can get launched.

• Get the Bluecheese theme running on subsites. The Drupal Association launched shortly after DrupalCon SF, and Localize last week. API and Groups are next. These sites are a good testing ground for the new theme before it goes to Drupal.org.
• Meet with everyone and see where everything is; especially, content, search, project, and dashboard.
• Familiarize myself with recent infrastructure developments; we now manage deployments with BZR and Hudson.
• Go through issue queues for all components.

I’m excited to get this project done. Drupal.org is a large Drupal deployment with a lot of moving parts. It is a bit daunting and will take a few weeks. With a new Drupal.org we can provide a better home for contributors to work and new users to learn.

The API module is almost ready to show classes and many more improvements on api.drupal.org. It is not quite ready yet; everything is in the database, but menu callbacks and template are needed to show the classes.

I’m looking for all bugs and criticals to be fixed and then, after a bit of testing, api.drupal.org can finally be upgraded. The configuration I’m using is

There is new support for projects. We will start off small, just Devel and Examples. The goal is all of contrib, but the module needs some changes to easily handle that volume.

The PHP parser was separated from documentation parsing and replaced with Grammar Parser, which goes toward fixing many bugs and the class & interface support.

There is a growing set of unit tests, which were unfortunately written after these refactorings. API is well-suited to testing since it is a parser with well-defined input and output. There are plenty of possibilities and testing will help make those happen. But not until the next release cycle, this one is already too long.

Thanks to

• Jennifer Hodgdon for helping out in the issue queue.
• Jim Berry for Grammar Parser.
• Randy Fay for pulling together examples.

• Features on the bottom, bugs and tasks on the top.
• Critical issues within the black lines.
• Colored as usual: grey open, red needs work, tan/yellow-ish needs review, green reviewed and tested by community.
• We used to track the development version as x.y.z.

I think we should have better tools, user interfaces, and visualizations for developers. This is one way to steer Drupal’s large developer community. For example, query cost is now more accessible; I think that will lead to less slow queries on Drupal 7 sites. Regressions are tested with SimpleTest; so there will be relatively less bugs.

This graph shows how many issues are open. I wanted to show something we haven’t seen before and maximize the amount of data on the screen. Project*’s database isn’t designed for this, it is good at saying where we are right now and how a single issue has progressed.

1. Query in SQL via Drush.
2. Translate results to JSON, including PHP unserializing.
3. Process to usable data in JS.
4. Graph with Protoviz.

This process is non-ideal, but was good for rapid iteration. Now that there is an algorithm, it can be written into a module to store the usable data in SQL. The data & code are available to hack on. This should make its way to project pages on Drupal.org; we need more ways to quickly evaluate modules.

I am working on API.drupal.org, but have been bogged down with parser rewriting. There is now a solid parser backed by Grammar Parser and the beginning of real project support. What UI improvements will increase developer efficiency and understanding?

Too often I’m stuck with ugly, circuitous tools on an underpowered development server. We can do better.

I am doing multi-server Aegir work and deployments were taking 10 seconds to get code to the servers. I have a hostmaster server and a web-only server, both need provision. I edit code on my laptop with my usual shell and editor. Now pushing 3 codebases takes less than 1 second. Use SSH master connections by adding

Host *
ControlMaster auto
ControlPath ~/.ssh/master-%r@%h:%p

to ~/.ssh/config. When another SSH connection is open, new connections piggyback on it, saving connection time and getting straight to the rsync.

#!/bin/bash
( rsync -azC hostmaster/ aegir@aegir-dev:hostmaster-HEAD/profiles/hostmaster; echo aegir-dev hostmaster ) &
( rsync -azC provision/ aegir@aegir-dev:.drush/provision; echo aegir-dev provision ) &
( rsync -azC provision/ aegir@web:.drush/provision; echo web provision ) &
wait

The -C skips .git files. The & backgrounds each task and wait waits, so all three are done in parallel.

Now I have my familiar shell and editor with localhost speed. No more jumping around servers and forgetting to update one of the provisions. And updating code is fast enough to not slow me down.

A work in progress for my Core Developer Summit presentation at DrupalCon San Francisco.

MAPLight.org is a Drupal site I’ve been working with for quite awhile. Their upgrade to Drupal 6 and redesign launched last week. We worked with Chapter Three for design and theming; and MAPLight’s excellent team for the idea, research, and quite detail-oriented QA. It is already getting some mentions in the media.

Both money and legislative data sources have modules to pull the same source data into your database. OpenSecrets Open Data for donations to U.S. Congress; Legislature for votes and more from GovTrack.us. MAPLight’s research is available in an API.

This marks the end of my direct responsibility for Drupal 5 sites. That leaves Drupal 5 itself to maintain, where the issue queue has been quiet. With no real sites to test and no SimpleTest coverage, core releases will be a little more stressful, but not much is changing. Remember, Drupal 7’s release is Drupal 5’s end of life.

I am presenting JavaScript best practices at Design 4 Drupal Stanford on Sunday. This is an updated version of the talk I gave at Bay Area Drupal Camp. New developments include

• jQuery’s new API site
• Profiling with Firebug
• jQuery Lint is an interesting project

The slides are available here.

At MAPLight I occasionally have to convert CSV files to SQL. The Table Wizard module handles the import well, but the columns are all long text.

I made a quick module to resize the columns. It uses Table Wizard’s built-in analysis and Drupal’s Schema API to do some basic conversions. It is nowhere near complete functionality or UI-wise. Just barely good enough to get me through 12 tables with 165 columns.

I put it on GitHub since it is incomplete and I don’t have time to do more work. And I wanted to try out GitHub.

As a “Drupal Genius*” for Chapter 3, I occasionally write for their website. My inaugural post is about JavaScript best practices. The “200% more AJAX” starburst was actually revived from an April fools joke years ago. Read the post and don’t actually use 200% more AJAX.

I will be presenting material from this post this weekend at Bay Area Drupal Camp. This should be a good camp; Saturday is packed with excellent sessions.

Update: the slides are available online.

An important part of the Drupal.org redesign is the dashboard. Power users have a towers of navigation, contributor, and/or documentation links on every page. That is clutter when you don’t need to jump to another section of the site. And one size does not fit all; there are a few options now, but either you turn on a block of links or you don’t.

The redesigned pages clean up the administrative clutter. You will be able to build your own, or keep the good defaults, dashboard to watch and get to what you care about. However, the content in the prototype is mostly positional. We need to design and build the widgets that are most useful for the Drupal community. We need to know what will help you be more productive on Drupal.org.

I started building out a dashboard module at the previous Drupal.org redesign sprints. But, it needs a lot more work. In addition to the issue queue, I made a wiki page to track priorities. I’ll be at DrupalCon Paris looking for people to help work on this important functionality.

By the way, the API.drupal.org work I wrote about, launched a few days ago. Thanks for all the testing and feedback. There are already a a few comments. There are still a lot of improvements needed, especially rewriting the parser. Find me if you want to help make more improvements.

One of the common requests for API.drupal.org is comments, like PHP has. With some help from chx, this is finally almost a reality. But I need your help; this uses a new single sign-on system and has a number of changes throughout the module.

I have been focusing on making the site faster. The auto-complete works at the speed of JavaScript, not AJAX. Short URLs, like http://api.drupal.org/hook_menu will work. OpenSearch plugins work with your browser.

If you have a few minutes, take an early look at the changes and make sure everything works before this all goes live.

Here is Drupal’s doc comment coverage:

Except for the lack of objects, doing relatively well, but there are a few undocumented files: Read more

• Josh finished the cross site activity client and server. It is now ready for modules to implement activities, such as commenting, project issue followups, and CVS commits.
• Karoly continued working on the documentation parser and API commenting.
• Matt finished the Apache Solr search facets implementing Mark Boulton’s prototype and implemented a jQuery drop down results sorter.
• Todd continued work on the Bluecheese theme.
• Chris finished mapping out Drupal.org’s current content and worked on mapping the prototype’s content. This will show what existing content we need to fit in and reveals any content we need to write.
• I worked on the dashboard with Dmitri and Ken.

Single sign on, comments for the API reference, search, and project pages all had significant progress during this sprint and will be deployed in the next 2 or 3 weeks.

This was the 3rd multi-day sprint, and there have been 2 one-day sprints. 57 people have SVN accounts and 144 people have joined the redesign implementers group. We are building a flexible and well-engineered theme and a suite of modules for single-sign-on, activity, and dashboard widgets shared among multiple sites. Drupal.org and its subsites have a lot of moving parts supporting the Drupal community. There is still a lot of work to do, especially in the modules we are building and doing editorial work on the content.

Thanks again to the Drupal community making this possible via the Drupal Association.

During a lunch at the Paris sprint, Dries suggested doing a sprint in San Francisco. I ran with the idea and now we are working on implementing the Drupal.org redesign at Chapter 3’s freshly-refurnished office. Mark Burdett, Matt Cheney, Erik Hopp, Josh Koenig, Kieran Lal, Courtney Miller, Todd Ross Nienkerk, Colin Sagan, David Strauss, and I are busy implementing the new Drupal.org. Chris Bryant, Dmitri Gaskin, Karoly Negyesi, Stephanie Pakrul, and Derek Wright are joining us later this week. I am working on the dashboard and helping organize everything. Kieran has a full list of what we are working on.

Thanks to Chapter 3 for expediting their new desks making room for 12 more people. AF83 and Advomatic are each buying lunches for us. The Drupal Association provided funding for some flights and hotels, we are keeping costs low by couch surfing and using local talent as much as possible.

If you want to help, join the group and #drupal-infrastructure. We are filing many issues to track the progress. We will be working through the weekend. Drupal.org is a big site with many moving parts, with more being added, all being built and maintained by volunteers. The effort in Cologne, Cambridge, Paris, and Washington D.C. has been amazing and shows what this community can accomplish.

I talked with a few people interested in API.drupal.org improvements at DrupalCon D.C. This post outlines the details of what I want to see done. Some tasks, like commenting on API pages, seem simple, however there are time-consuming details.

This comprehensive list is a lot of work. It will not get done quickly without help. I am organizing a Drupal.org redesign sprint April 8 to 11 or 12 in San Francisco, some API module work will likely be done April 11 and 12. Email me if you are interested in attending. I will be attending the documentation planning sprint in Toronto to work with the larger documentation project.

I would like to see more people, particularly Drupal shops, using API module. More users means more contributors. For example, Development Seed is trying out API with Spaces on their Code site. I talked with Young about the implementation and the scalability to 4000 modules. Read more

Yesterday I presented about security at FOSDEM. The slides are now available. At Drupalcon D.C. I will be joining Greg, author of Cracking Drupal, and Matt to give a more broad security presentation with live demos.

I am in Paris for the week continuing work on the Drupal.org upgrade and redesign implementation. We split into two teams, triaged issues and are set to make a lot of progress this week.

We have a new design for Drupal.org, now we need to implement it. To get this done quickly, Dries is organizing 4 sprints. I will be taking 2 weeks off of client work to help out in Boston, Paris and Washington, D.C., joining Gábor, Derek & Chad, Gerhard, Damien and others. We have a lot of work to do: upgrading project module and the existing site to Drupal 6, implementing the new design and new features.

I have been working on

• infrastructure with Gerhard, Narayan and Damien,
• the current Bluebeach theme with Gábor,
• association.drupal.org with John Forsythe
• and api.drupal.org.

The Drupal community is great at working online, but this type of project will greatly benefit from concentrated, in-person collaboration. Getting people in a room is not free; please chip in to help make this happen. At the time of writing, 48 people have contributed an average of $37.35 each. More convincing is available in English, Français, Magyar and 中文.