Redmine Implementation Notes

We recently decided to use Redmine + Backlogs to manage a series of projects and I have been knee-deep implementing it to suit our needs and provide information to the various project stakeholders.

Redmine provides a bunch of features out-of-the-box (basic task/sub-task management, Gantt chart, wiki, etc) but like many packages there are a few features that either just aren't right, or are plain missing. The good news is that Redmine supports a plugin architecture with many freely available plugins provided by the Redmine community. As with all things free: buyer beware; and be prepared to wade through some wacky features (why would you subject yourself to advertisements on an internal project management tool? Why?).  The next bit of good news is that it is open source, so if you don't like something you can always roll-up your sleeves, embrace the Ruby and change it.

My Redmine MVPs (Most Valuable Plugins):

• Better Gantt Chart - provides a mildly better Gantt UI, but the big benefit is for task-rescheduling. Redmine's default task management works for pushing dates out but is hopeless for pulling dates back in on a task (and subsequent linked tasks).
• Redmine Auto Percent - When a task is moved to 'Done', automatically set the percentage complete to 100%. I have plans to extend this plugin so that when a task moves from the Backlog to "In Progress" it will automatically set the percentage complete to 10% (ie. not zero).
• Redmine Wiki UNC - Provide the ability to create UNC links in the Redmine wiki.

My Redmine MVCs (Most Valuable Code-Changes):

• Change Field Labels - Redmine's default unit of time is hours. We normally estimate effort in days, so it made sense to change the field labels to prevent any confusion from the various stakeholders who use the tool. The quickest way to do this is to edit the appropriate language .yml file (eg. en-GB.yml) and change any references within that. A nicer way that will survive upgrades is to make a copy of the language file you are using and set both the language code and identifier to a unique name. Remember to re-start the web server and choose your new language file in the administration console. Language files are located at:
• \Redmine\config\locales
• \Redmine\vendor\plugins\redmine_backlogs\config\locales
• Change Issues query to display Parent Story Name instead of Parent Story ID in results - The version of Redmine we are using unhelpfully displays the ID of the parent story which makes it hard to develop meaningful reports. The following code changes in the \Redmine\app\helpers directory allow the Parent Story Name to be shown:
• queries_helper.rb - Modify method "column_content", specifically case statement option "Issue" to modify "link_to_issue" parameter "Subject" from false to true. This change by itself will make the Parent Task column display both the ID and the Name together.
• application_helper.rb - Modify method "link_to_issue" so that only the parent task subject is displayed. Code changes highlighted below.

  def link_to_issue(issue, options={})
    title = nil
    subject = nil
    if options[:subject] == false
      title = truncate(issue.subject, :length => 60)
    else
      subject = issue.subject
      if options[:truncate]
        subject = truncate(subject, :length => options[:truncate])
      end
    end
 ####################################################
 # Modified code follows
 # Purpose: Display parent task name instead of ID in query results.
 ####################################################
 if options[:subject] == true
  s = link_to "#{h subject}", {:controller => "issues", :action => "show", :id => issue},
   :class => issue.css_classes,
   :title => title
 else
  s = link_to "#{issue.tracker} ##{issue.id}", {:controller => "issues", :action => "show", :id => issue},
   :class => issue.css_classes,
   :title => title
  s << ": #{h subject}" if subject
  s = "#{h issue.project} - " + s if options[:project]
 end
 #
 ####################################################
 # Original Code follows (Replaced by changes above)
 ####################################################
 #s = link_to "#{issue.tracker} ##{issue.id}", {:controller => "issues", :action => "show", :id => issue},
 #                                             :class => issue.css_classes,
 #                                             :title => title
 #s << ": #{h subject}" if subject
 #s = "#{h issue.project} - " + s if options[:project]
 ####################################################
    s
  end

These changes have resulted in a much more useful and usable project management tool. It's still not perfect (linked task date re-scheduling is a pain), but vastly improved for how we need to use the tool. Hopefully this information is useful to anyone else going down a similar path.

From Waterfal to Agile - A Journey (Pt 2)

Our first attempt at agile shook a few nerves. So with our confidence rebuilt after getting a number of retrospectives under our collective belt (analysing what worked, what didn't and what we could do better from our first foray into agile) a new development period brought with it a new opportunity to Go Agile.

This time, instead of changing processes for one test project the decision was "let's change process for all of them". After I'd unclenched my buttocks, I saw the wisdom in this approach - following different processes for different projects will just confuse people. The twist however is that rather than going all-out agile, we would use this development period to iterate towards agile in a manner that doesn't require the team to change everything that they've become accustomed to when developing software.

So instead of a methodology big-bang, we're dipping the toe in the water with Rational Unified Process (RUP). Generally speaking, it couples big up front design to an iterative development approach. To me RUP isn't Agile, but if you structure things just-so you can introduce the team to elements of Agile so that taking the next step of adopting a process like Scrum (hopefully) becomes a little easier. You can also start to get pointy-haired boss types used to various tools and artefacts that will come with a more complete Agile adoption.

Whilst tools do not define the success (or otherwise) of an agile project, the right tool can certainly make life easier. Initially, we had planned on using Telerik's TeamPulse instead of the more traditional mix of in-house tools + MS Project + Excel. However, for various reasons we have instead gone with the open-source Redmine project management tool along with the Backlogs plugin that provides sprint management and card-wall functionality.

It can be a little odd using an agile tool in a non-agile manner. One of the key differences for how we are using the tool is that sprints are filled with "stories that are being worked on" as opposed to "stories that will be completed" during the sprint. As a consequence, stories roll over from sprint-to-sprint. To me, this is fine for where we are on the agile journey, and is an improvement over how we previously managed projects. We get the agile-ish benefits of early feedback, iterative planning and an ability to change directions and priorities as required, married with the waterfall-ish aspects that the project stakeholders know & expect (design up-front and a somewhat detailed schedule with distinct sign-off phases).

So while attempt #2 was never intended to teleport the team into some kind of agile utopia, the approach of taking baby-steps towards that vision appears to be paying dividends. I'm still setting my sites on that utopia - it just might take a few more iterations to get there.

Root Cause Analysis

Root Cause Analysis is one of those terms that can get thrown around to indicate that your gunna do some Serious Work. Analysis will be done, lists will be made, names will be taken. But what does Root Cause Analysis really involve in the software world. And can you do Root Cause Analysis on your Root Cause Analysis?

It sounds simple - Root Cause Analysis is about finding a fundamental underlying issue that is causing one or more problems. However, as Bob Nelms writes 'Plenty of organizations are addressing the "physical" causes of failure. Even more seem to enjoy finding out "who did it" so that disciplinary action can occur'.

The implications of this is that it's not enough to resolve the physical cause of the problem (eg. an unhandled exception); nor is it enough to ensure that the problem never happens again (fix the bug that caused the exception and add an automated test to cover the case). What's needed is to go deeper - we need to imitate a 3 year old child and ask why ALL THE TIME. Why did the programming practices that are (or are not) being employed allow the problem to occur? Why did the various layers of the architecture not prevent the problem? This technique even has a name - the Five Whys. Doesn't sound hard right - just keep asking why... Still, if you love you some processes, the Six Sigma folks have got you covered.

The one question I don't think you should ask is "who?". Why? Because you'll probably find out "who" if you keep asking "why". If you're interested in finding out who caused a problem then you're probably not really performing a Root Cause Analysis, your doing something else entirely.

So can you do Root Cause Analysis on your Root Cause Analysis? Of course you can! It probably just means that the first time you did it, you didn't get to the root cause. You just need to find out why.

Performance Fundamentals

There is no magic bullet or one-size-fits all architecture for creating a high-performance enterprise application, but there are some pretty basic fundamentals that will certainly assist you along the way.

General Performance Fundamentals:

The following points are focused around avoiding what is usually the slowest part of the deployment topology - the network infrastructure.

• Minimise network traffic - every machine-to-machine "hop" that your code traverses while fulfilling a request (eg. browser to web-server to app-server to database) can add enormously to the time to complete an operation. Aside from the physical network hop, there are often other overheads involved like serialisation and compression. All of this time adds up and can affect the responsiveness of your application making it feel somewhat sluggish (even if every operation executed on each tier was lightning fast).
• Avoid SQL calls within loops - if we're trying to minimize network traffic, we sure as heck don't want to add extra calls to the DB for every record being processed in a collection. It's oh so easy to fall into this trap by accident. Wherever possible, get all the data you need in one hit from the database rather than issuing separate calls to get an additional related value or description. This same advice holds for other remote method calls like web services.
• Reduce unnecessary serialisation - the point of serialisation is (usually) to copy data or state from point A to point B so that some action can occur. However, there's no point lugging a bunch of data over to point B if the processing there isn't going to look at it.

Database Performance Fundamentals:

• Use bind variables - A bind variable is a place-holder variable in a SQL statement that must be replaced with a valid value before the statement can successfully execute.  Using bind variables in your sql statements can improve performance as the database doesn't have to parse the execution plan for every sql statement that comes along with a different value in the where clause.
• Use appropriate indexes - A database index is a data structure that improves the speed of data retrieval operations on a database table at the cost of slower writes and increased storage space. You could write a book about database index design. Knowing how and where to apply indexes can sometimes be more of an art than a science and is very particular to the specific application and environment you are working with. My advice is to read-up as much as possible to understand what is going on under the covers.
• Be pragmatic about schema normalisation - Much has been written about database design strategies, and there is no one-size-fits-all "best" approach for all situations. However, in my experience with business applications, some form of redundancy is preferred over dogmatic adherence to a particular higher-level normalisation scheme. The reason for this is that the number and expense of joins on data retrieval (for complex systems) can easily outweigh the cost of redundant data storage. The level of redundancy you choose will no doubt be unique for your situation.

.Net Rules of Thumb

• Use database connection pooling - Establishing and cleaning up connections to the database can be very time-consuming. Instead of creating new connections as needed, it is better to use an existing pool of connections that are held open and available at all times. Most database flavours (eg. Oracle and SqlServer) support connection pooling via their respective .Net connection libraries.
• DirectCast outperforms CType -Especially useful to know if you need to serialise to/from XML. Note that this only applies to VB.Net.
• Compiled Lambdas outperform Activator.CreateInstance - my experience has largely encountered problems with Activator.CreateInstance in framework code that I'm unable to (easily) modify, but it's still useful to note as the benefits can be huge.
• Use StringBuilder for concatenating strings - a well known tip that is still worth remembering. StringBuilder almost always outperforms a bog-standard string object when concatenating.

Summary
Diagnosing and improving performance problems can be extremely difficult. Hopefully the above tips and fundamentals will help, but inevitably you'll need to go deeper. There are a bunch of performance profilers available that can assist with this. I also strongly recommend building in some level of telemetry directly into the application (that can be turned on/off when needed). It's easy to go overboard, so start with the best bang-for-buck areas and go from there. Alternatively, rather than build it yourself, use a service such as New Relic or Google Analytics to get info quickly. These initiatives will allow you to gather performance data from production systems and avoid "works on my machine"-esque scenarios.

So, if in doubt, MEASURE IT! Building a culture whereby developers know and understand the performance metrics of the code pays dividends, and if the performance metrics can be automated, so much the better.

Free .Net Developer Resources

For when you need a dev environment at home, but don't have access to all the toys that work has paid for.

Visit asp.net/get-started and download the Visual Web Developer Express package. This also includes MVC3 libraries and SqlServer Express.

And to sharpen your skills, Microsoft has released an updated training kit for .Net Framework 4 and framework technologies including:

• C# 4
• Visual Basic 10
• F#
• Parallel Extensions
• Windows Communication Foundation (WCF)
• ASP.NET 4
• Entity Framework
• ADO.NET Data Services
• Managed Extensibility Framework
• Application Lifecycle Management
• Windows Azure

From Waterfal to Agile - A Journey (Pt 1)

The company I work for tends to be somewhat conservative. It's fantastic in that it's a company who's primary reason for being is the creation of awesome software; it's been pushing out software for many years with a lot of success. Which is where the conservative culture comes in... We didn't need/use/want shiny X back then, why do you need it now?

As an internal observer, it pretty much seems like the prototypical organisation distrustful of Agile referred to in books and blogs (though we probably only fit four of the six signs in the link - as one colleague commented "it's like they're watching us"). However, to the company's credit, things are slowly starting to change. Granted, sometimes it can feel like two steps forward, one back; but I think the momentum might be starting to push things ahead.

The kind of development my team is traditionally involved with falls roughly into two categories:

1. Large Enhancements - Traditional Waterfall delivering new features with big up-front design; massive coding efforts; then a concentrated testing effort trying to mop up all the bugs before eventually shipping out the door.
2. Bug Fixes - Code & Fix on shipped software with priorities driven by whichever fire is burning brightest & no-one being particularly happy about the lack of strategy behind the fixing effort.

The deficiencies of these approaches are well understood and Agile (any Agile) seemed like it would be a better place to be in. However the trick always seemed to be finding the right project to trial a new methodology in order to demonstrate the benefits that could be achieved. Well, if you can't choose, then one will be thrust upon you. So, we got our project... but it sure wasn't the right one. The project didn't fail (we still managed to ship), however there were some things that definitely didn't contribute to a smooth-running project.

Lessons Learned:

• Ensure everyone on your shiny new agile project has a shared understanding of what agile is, and the nuances of the agile flavour you are adopting. There were some big assumptions made that people across all disciplines were talking the same language.
• Don't try to maintain the same artefacts from your old process. Accept & embrace the change.
• Avoid cross-team dependencies for your first ever agile project like the plague. Different teams have competing priorities and deadlines that may not be compatible with your project.
• Give people on the project the power to be 100% dedicated to the project. Don't saddle them up with multiple competing projects to be delivered within the same timeframe.
• Ensure the team sits together - the daily standup shouldn't be the only time everyone talks with each other throughout the day.
• Ensure that you customer is able to engage with you in the manner & frequency that Agile methods require.

The pain and frustrations that this project brought about lead to a brief period where Agile was only talked about in hushed tones with averted eyes. Thankfully, we didn't get to the point where Agile was completely dismissed (we all knew the mistakes that had been made). However, this again put us in the position of trying to find the "right" project to Go Agile.

To be continued in Pt 2...

Technorati Claim Post

This post is to poke & prod the Technorati bots so that they know this blog is mine. Please ignore :-)

Technorati Claim Token: 5NAWTUQFRG4H 

We now resume regular programming...