we know our users are complaining, we know we are losing business, we have been trying to identify the issue for days, I am losing credibility, I missed several friends dinner, I work every weekends, I have to leave the office now because I have to jump on a change management conference call while driving with the kids screaming in the back of the car. I have other things on my plate, like launching our new private banking services, budgeting for new servers to address our merger with ABC company, I need to grow my business, we can’t even have a feel on how our services behave nor identifying simple problem such as one out of five times the browser hangs when entering employee badge number. The assumption I made last week on where the problem might have been are now wrong, the change management team applied a patch against that specific application and the problem didn’t go away. I am stress and tired…. I am stress and tired…. I am stress and tired…. I am stress and tired….

IT experts would say: “I have tools several, several, several, several tools, and it is true after triaging all the alerts, the tools were able to isolate issues but I really just care about what impacted my users in company ABC. What is the behavior of my most revenue generating transactions today and what will it be after we merge the two companies’ systems next week, how would I know if it improves or degrades the overall business service?”Familiar with THIS?  What if you would take a peek at introducing Business Transaction Management (BTM) into your IT process?

You would finally see at this moment the IT consumers and IT producers of business transaction information, knowing whom and what is impacted, focusing only on the most important services. What if you knew the exact flow of the information and the behavior of your special revenue generating credit card application transactions? BTM is a source of rich IT information.  It is much more than incident management, you can not only understand the current behavior and plan for growing your business you can see the impact on your services of an unplanned or planned change.

This is the cure to resolve the “THIS”, today, tomorrow, next week, on a constantly changing fluid IT environment. Really who could have predicted that you would transact business via text messages?  With this information on hand feel free to use those specialized tools and apply them appropriately to isolate granular application components issues but change the way you think about managing IT,  It is not always about technical components. Now, I won’t cure all your stress and fatigue as there always be screaming kids, traffic, lines at the coffee shop but one less thing to worry about, getting a little more of your personal life back, one more thing to proudly walk to your management and really feeling good that you know the “THIS” at every moment of the day and I guarantee you will be grooving along a Disco song….

When discussing BTM a lot of the conversation centers on the technical complexity of business transactions. So for instance we were talking the other week about a simple “bill payment” transaction. Seems such a “simple” payment transaction flows in many cases across as many as ten IT environments: a client service portal (web front ends, security appliances, front end portal servers, portal personalization databases), then crosses over to the payment application (payment system app servers, master payment database and the interfaces to external payment services). There is huge value in exposing this technical flow across the two apps and the eight technical components.

But if we look beyond the technical complexity we’ll see that each and every one of the steps in this flow has a person behind it: from the business user behind the payment itself to the IT people behind the apps and infrastructure pieces. Everyone is focused on making their part of the overall story work well – but they all have very little to go by in terms of aligning their efforts with the overall goal. And the overall goal that really matters is getting that payment thru, without glitches, without holdups, time after time.

Well, transactions, when made visible, provide us the ultimate facility to connect these people, business and IT alike. Transactions expose the “lines between the dots” and provide a meaningful context to the people who participate in planning the systems, creating the apps and operating them. They allow them to focus on their jobs while effectively communicating with their peers. They provide a shared context around which they can all connect.

Think about it :  “Transactions – connecting people”.

When people in IT learn about Business Transaction Management for the first time, their reaction is often a similar mix of awe and disbelief.  They are awed, because gaining true visibility into the behavior and flow of all business transactions has been an industry holy grail for quite some time now, and they also express disbelief, well, for the exact same reason.  How is it possible to auto-discover and track all business transactions, when transactions are not tangible, manageable configuration elements like servers and routers?  At best, such technology might work in a simple, straightforward environment, but our IT environment is huge, complex, distributed, heterogeneous… can this BTM work for us?

Like other emerging technologies, BTM is climbing up a maturity curve.  From just being able to show round-trip response times and infer some latency breakdown of network versus data center time, superior BTM solutions are now capable of tracking transactions deep into the data center, providing rich, granular topology views that cross hundreds of web, application, authentication, messaging, and database servers.  The ability to show simple request-response sequences has expanded to cover complex, asynchronous flows using pub/sub and “send and forget” messaging protocols.  In addition to showing all transactions, some BTM solutions will now auto-discover and show entire business processes, aggregating many discrete transactions into a “short list” of real business flows.  The ability to measure service quality is also maturing from application and server uptime SLAs (remember five nines?) to transaction-specific response time SLAs and Apdex ratings that represent end-user satisfaction in real-time.

Despite being one of the privileged few who witnessed BTM grow from a mere idea into a full-blown enterprise solution, I still find myself sometimes awed by this technology.  It is, after all, an ambitious attempt to visualize complex, abstract business ideas, and manage them like any other assets of the organization.  How is it possible?  Does it really work?  As my friend and colleague Andy previously noted, the “aha moment” for BTM typically doesn’t arrive until after the customer has already seen it live in their own environment.  And that’s when the fun begins.

For those of you who read my blog you may perceive me as just another product manager of an IT company. One of my interests outside of work is motorsport and generally driving a car as fast as is physically possible. I’m not the type of guy who drives to work cruising on the motorway in 6th gear doing 50mph (no offense meant for people who do this btw). For me its about getting from point A to point B as quickly as possible whilst maintaining strict adherence to government speed limits…or something along those lines.

Anyway, I was thinking  the other day just how complex a car is underneath the glossy paint and metal shell that most people perceive a car to be. You’ve got the engine for starters (literally), then you’ve got things like air filters, radiators, oil tank, fuel tank, catalytic converters, spark plugs, exhausts, gearbox, clutch and so on (I won’t bore you with the other 1842 parts).  The car also has hundreds of sensors to detect failure, tolerance levels of components and even stupid people who don’t wear their seatbelts (again no offense intended for people who don’t wear seatbelts).  It’s actually an engineering miracle that so many pieces can work together without failure for so long (unless you happen to own a TVR of course). And the great thing is that when something does go wrong your car dashboard lights up like a Christmas tree and tells you what’s wrong – how cool is that?. The monitoring and operation of all those car components is simplified through a lovely glowing dashboard. The oil light comes on when you need  more oil, the tyre light comes on when you need new tyres or more pressure. If you drive a BMW then the onboard computer even tells you that your not driving close enough to the car in front like other BMW drivers. In the unfortunate case of an engine light, the problem normally involves a trip to your car garage where some guy in white overalls plugs in a computer to your cars ECU. Usually within 2 minutes he’s detected that your car is broken and needs £2000 worth of work to fix it. Needless to say 95% of issues can be fixed in no more than a few hours (which is why Audi, Mercedes and BMW garages charge £150 per hour labour) .

Simple to drive but Complex to engineer

My point with the car is that it’s a simple bit of kit to use and monitor despite its hidden complexities. Car manufacturers have done a stellar job of simplifying complexity so that our cars don’t have 101 dashboards to report status or issues. You turn the key to start, turn the wheel to steer and plant your foot firmly to the floor to go fast. Your car’s dashboard does all the rest to inform you of what you need to know. When the car needs an update the garage simply remaps the car’s ECU at the next service rather than letting than the owner do it himself with a laptop, OBC connection and a hotfix off the internet.

If monitoring cars can be so simple then why can’t monitoring applications? Applications have just as many components and complexity, they are even built by engineers who use keyboards rather than spanners. They even have a nice pretty appearance (unless they’ve been built in the 1990′s with visual basic or something). I know what your thinking “Applications are more complex, nothing can be as complex as coding an EJB or writing some complex SQL”. Try telling that to the folks at Ferrari or Porsche that spend millions each year optimising their traction control and stability systems that stop people like me from ending up in a hedge.

As a product manager working for a software company in the monitoring space I feel a sense of responsibility for putting an end to this complexity of monitoring business transactions, applications, SOA environments, end users, networks, JVM’s, databases, servers, enterprises buses and pretty much everything else that requires several million products, agents , appliances, dashboards and user interfaces. Software vendors should do what car manufacturers have been doing for the last 20 years. They should provide simple usable solutions that abstract over all complexity and make it as straight forward as possible to manage business transactions and the IT infrastructure with which they flow.