Monday 16 January 2012

Root Cause Analysis

Root Cause Analysis is one of those terms that can get thrown around to indicate that your gunna do some Serious Work. Analysis will be done, lists will be made, names will be taken. But what does Root Cause Analysis really involve in the software world. And can you do Root Cause Analysis on your Root Cause Analysis?

It sounds simple - Root Cause Analysis is about finding a fundamental underlying issue that is causing one or more problems. However, as Bob Nelms writes 'Plenty of organizations are addressing the "physical" causes of failure. Even more seem to enjoy finding out "who did it" so that disciplinary action can occur'.

The implications of this is that it's not enough to resolve the physical cause of the problem (eg. an unhandled exception); nor is it enough to ensure that the problem never happens again (fix the bug that caused the exception and add an automated test to cover the case). What's needed is to go deeper - we need to imitate a 3 year old child and ask why ALL THE TIME. Why did the programming practices that are (or are not) being employed allow the problem to occur? Why did the various layers of the architecture not prevent the problem? This technique even has a name - the Five Whys. Doesn't sound hard right - just keep asking why... Still, if you love you some processes, the Six Sigma folks have got you covered.

The one question I don't think you should ask is "who?". Why? Because you'll probably find out "who" if you keep asking "why". If you're interested in finding out who caused a problem then you're probably not really performing a Root Cause Analysis, your doing something else entirely.

So can you do Root Cause Analysis on your Root Cause Analysis? Of course you can! It probably just means that the first time you did it, you didn't get to the root cause. You just need to find out why.

Thursday 5 January 2012

Performance Fundamentals

There is no magic bullet or one-size-fits all architecture for creating a high-performance enterprise application, but there are some pretty basic fundamentals that will certainly assist you along the way.

General Performance Fundamentals:

The following points are focused around avoiding what is usually the slowest part of the deployment topology - the network infrastructure.
  • Minimise network traffic - every machine-to-machine "hop" that your code traverses while fulfilling a request (eg. browser to web-server to app-server to database) can add enormously to the time to complete an operation. Aside from the physical network hop, there are often other overheads involved like serialisation and compression. All of this time adds up and can affect the responsiveness of your application making it feel somewhat sluggish (even if every operation executed on each tier was lightning fast).
  • Avoid SQL calls within loops - if we're trying to minimize network traffic, we sure as heck don't want to add extra calls to the DB for every record being processed in a collection. It's oh so easy to fall into this trap by accident. Wherever possible, get all the data you need in one hit from the database rather than issuing separate calls to get an additional related value or description. This same advice holds for other remote method calls like web services.
  • Reduce unnecessary serialisation - the point of serialisation is (usually) to copy data or state from point A to point B so that some action can occur. However, there's no point lugging a bunch of data over to point B if the processing there isn't going to look at it.

Database Performance Fundamentals:
  • Use bind variables - A bind variable is a place-holder variable in a SQL statement that must be replaced with a valid value before the statement can successfully execute.  Using bind variables in your sql statements can improve performance as the database doesn't have to parse the execution plan for every sql statement that comes along with a different value in the where clause.
  • Use appropriate indexes - A database index is a data structure that improves the speed of data retrieval operations on a database table at the cost of slower writes and increased storage space. You could write a book about database index design. Knowing how and where to apply indexes can sometimes be more of an art than a science and is very particular to the specific application and environment you are working with. My advice is to read-up as much as possible to understand what is going on under the covers.
  • Be pragmatic about schema normalisation - Much has been written about database design strategies, and there is no one-size-fits-all "best" approach for all situations. However, in my experience with business applications, some form of redundancy is preferred over dogmatic adherence to a particular higher-level normalisation scheme. The reason for this is that the number and expense of joins on data retrieval (for complex systems) can easily outweigh the cost of redundant data storage. The level of redundancy you choose will no doubt be unique for your situation.

.Net Rules of Thumb
  • Use database connection pooling - Establishing and cleaning up connections to the database can be very time-consuming. Instead of creating new connections as needed, it is better to use an existing pool of connections that are held open and available at all times. Most database flavours (eg. Oracle and SqlServer) support connection pooling via their respective .Net connection libraries.
  • DirectCast outperforms CType -Especially useful to know if you need to serialise to/from XML. Note that this only applies to VB.Net.
  • Compiled Lambdas outperform Activator.CreateInstance - my experience has largely encountered problems with Activator.CreateInstance in framework code that I'm unable to (easily) modify, but it's still useful to note as the benefits can be huge.
  • Use StringBuilder for concatenating strings - a well known tip that is still worth remembering. StringBuilder almost always outperforms a bog-standard string object when concatenating.
Summary
Diagnosing and improving performance problems can be extremely difficult. Hopefully the above tips and fundamentals will help, but inevitably you'll need to go deeper. There are a bunch of performance profilers available that can assist with this. I also strongly recommend building in some level of telemetry directly into the application (that can be turned on/off when needed). It's easy to go overboard, so start with the best bang-for-buck areas and go from there. Alternatively, rather than build it yourself, use a service such as New Relic or Google Analytics to get info quickly. These initiatives will allow you to gather performance data from production systems and avoid "works on my machine"-esque scenarios.

So, if in doubt, MEASURE IT! Building a culture whereby developers know and understand the performance metrics of the code pays dividends, and if the performance metrics can be automated, so much the better.

Tuesday 3 January 2012

Free .Net Developer Resources

For when you need a dev environment at home, but don't have access to all the toys that work has paid for.

Visit asp.net/get-started and download the Visual Web Developer Express package. This also includes MVC3 libraries and SqlServer Express.

And to sharpen your skills, Microsoft has released an updated training kit for .Net Framework 4 and framework technologies including:
  • C# 4
  • Visual Basic 10
  • F#
  • Parallel Extensions
  • Windows Communication Foundation (WCF)
  • ASP.NET 4
  • Entity Framework
  • ADO.NET Data Services
  • Managed Extensibility Framework
  • Application Lifecycle Management
  • Windows Azure

Monday 2 January 2012

From Waterfal to Agile - A Journey (Pt 1)

The company I work for tends to be somewhat conservative. It's fantastic in that it's a company who's primary reason for being is the creation of awesome software; it's been pushing out software for many years with a lot of success. Which is where the conservative culture comes in... We didn't need/use/want shiny X back then, why do you need it now?

As an internal observer, it pretty much seems like the prototypical organisation distrustful of Agile referred to in books and blogs (though we probably only fit four of the six signs in the link - as one colleague commented "it's like they're watching us"). However, to the company's credit, things are slowly starting to change. Granted, sometimes it can feel like two steps forward, one back; but I think the momentum might be starting to push things ahead.

The kind of development my team is traditionally involved with falls roughly into two categories:
  1. Large Enhancements - Traditional Waterfall delivering new features with big up-front design; massive coding efforts; then a concentrated testing effort trying to mop up all the bugs before eventually shipping out the door.
  2. Bug Fixes - Code & Fix on shipped software with priorities driven by whichever fire is burning brightest & no-one being particularly happy about the lack of strategy behind the fixing effort.
The deficiencies of these approaches are well understood and Agile (any Agile) seemed like it would be a better place to be in. However the trick always seemed to be finding the right project to trial a new methodology in order to demonstrate the benefits that could be achieved. Well, if you can't choose, then one will be thrust upon you. So, we got our project... but it sure wasn't the right one. The project didn't fail (we still managed to ship), however there were some things that definitely didn't contribute to a smooth-running project.

Lessons Learned:
  • Ensure everyone on your shiny new agile project has a shared understanding of what agile is, and the nuances of the agile flavour you are adopting. There were some big assumptions made that people across all disciplines were talking the same language.
  • Don't try to maintain the same artefacts from your old process. Accept & embrace the change.
  • Avoid cross-team dependencies for your first ever agile project like the plague. Different teams have competing priorities and deadlines that may not be compatible with your project.
  • Give people on the project the power to be 100% dedicated to the project. Don't saddle them up with multiple competing projects to be delivered within the same timeframe.
  • Ensure the team sits together - the daily standup shouldn't be the only time everyone talks with each other throughout the day.
  • Ensure that you customer is able to engage with you in the manner & frequency that Agile methods require.
The pain and frustrations that this project brought about lead to a brief period where Agile was only talked about in hushed tones with averted eyes. Thankfully, we didn't get to the point where Agile was completely dismissed (we all knew the mistakes that had been made). However, this again put us in the position of trying to find the "right" project to Go Agile.

To be continued in Pt 2...