AEC Media Day: Time Spent with the Development Team

So, I did promise to get back to this topic, and I have at least a few more posts about 2011 to come to. Steve's review of the whole day was good, so I wanted to focus in particular on the time we spent with a development team.

No, I'm not going to give you any insights into what is coming in terms of features, instead we had a fascinating opportunity to get an idea of how the software we use every day is developed.

The Cast

As Steve said we got to visit with Greg's development team. So we met (most of) them in a conference room. Hopefully I get the titles right: Greg the Product Designer was there, then there was a gent from the Quality Assurance part of the Factory, Erik was there for a few minutes, but had to pop out (he is product design too) then we had Lev Lipkin (long time Revit developer), two more developers (whose name's escape me) and one last developer on the phone.

So what is everyone's roles in creating a new feature? Obviously the developers write the code that makes the software run. I'm assuming there is some further breakdown in terms of responsibilities, but we did not get into that level of detail.

Product Designers provide the specification of what the new feature is, what are the goals it is supposed to achieve and what is the required functionality.

This then leaves you to wonder, what is QA doing there? Well QA in the Factory is an interesting role. They not only test the software as it is developed, and test again, and retest, and test some more, but they also regularly use the software, and they typically have a design background, so they're familiar with our collective industries. QA and can offer input from a user background, their interaction with clients, and their experience troubleshooting bugs and other problems that users do run into. Lastly, QA is valuable because they are the ones who have to test and effectively approve new features for release. Therefore when considering the time for a development cycle (effectively about 9 months, when you take out 3 months for project scoping and research) you have to consider, is it realistic to test the proposed new feature set in that length of time, when you consider all the other things that have to happen to develop new code or modifying existing.

This is a critical part in the software development process that I think a great number of people underestimate the complexity and time involved. The QA team has a "huge" server room dedicated to their work where they run, and re-run thousands (if not hundreds of thousands) of tests on the software (every night). Everyday a new build of Revit is released in the factory and every night those builds are tested. Furthermore, as the software grows, the number of tests continues to increase. These are all tests to make sure that:

1. New features don't break old features.
2. Modified features continue to work.
3. New features aren't broken (this can't be done until you have a test for the new feature).

So a new feature is a big deal, you have to test is against what you already have, then you have to design a test for it, to make sure it continues to work, and that takes a great deal of time. Even with what is a pretty darn good QA process, and really, really good QA team, bugs still find their way through to us the end users, and we don't even catch them all in Alpha or Beta either.

The Process

In the discussion we were part of, Greg began by going through an outline of what the end goals were for the feature, also recognizing that this feature would only be one small step towards more features in the future. From there the discussion commenced....

QA was concerned about scope. In particular there was concern about enough time to fully the test the new feature and there was concern that the Revit team, would start to take on and own something that was not previously "theirs" in terms of development responsibility and support. This is actually a big deal, its kinda like the architect saying to the Mechanical engineer, "oh don't worry, we'll make sure to put all the HVAC diffusers and returns in the all the right spots". In many cases we might do pretty well, but we're not experts, and do we really want that responsibility in the first place? So the Factory has the same issues to contend with in their world.

So, this issue of ownership then led to a discussion about, what could be done internally in Revit, to support the proposed workflow, without taking on scope that Revit did not have previously. This then led to a discussion about what did users really, need, what could they get by with, and what might have to just wait (no matter how much the users might want it).

Essentially then feature development becomes a process of risk analysis. What is the minimum required to keep users happy? What is required for the feature to be useful? Can the code be written and tested to meet those needs? This is not to say that it is only a discussion of numbers, but no matter what, the bottom line is the bottom line, code cannot be changed until the 9th hour, its just not how it works.

So the discussion then really became focused on what would meet the user needs, and really focusing in on what the core goals were that Greg had in mind, and what could be "stripped" away to meet them, or what was the best way to meet them. This is where development starts to speak up, because they have an idea of what they might be able to do with existing code, and they also (mostly) know where the the Jimmy Hoffas of the Revit world are buried ("we" learned about at least one which I think took a few people by surprise....) The ideas of the developers are interesting, when compared to how an end user might consider something, and it this mix of Product Design, QA and Developers that eventually leads to a finalized feature.

As "flies on the wall" we were able to "speak up" from a user's point of view and hopefully provide a little bit of insight (Product Designers also have the responsibility of interviewing and researching real users, like us). Steve even got up an drew on the white board!

There is more I wish I could share, but it would reveal too much about the feature being discussed. In any event, it really was quite a unique experience, and educational. As we all left, the comment from the Factory staff was "this is what we do all day every day", a stretch perhaps, but quite telling. Not a single bit of code has been written for this feature yet, and won't be for probably several more months, instead there will be more meetings, more discussions, analysis of user interview data, and debate how best to achieve the goal. That way, when the developers do sit down to write the code, they can focus on writing code that immediately produces the desired results, rather then writing and re-writing code, because it does not do what "we" thought it would.