Archive for July, 2009
Manage IT with Business Impact not with Traffic Lights
I’ve been using the phrase “If everything is important then nothing is important” quite a lot in the last week. In my desperate attempts as a product manager to respond to every email, enhancement request, PRD, conference call and tweet it’s becoming quite challenging to say the least. I’m constantly fighting the battle of email and have even tried sending less email recently in the vain attempt that I’ll receive less…which didn’t seem to work at all. I even tried setting filters up on my inbox but still the emails keep getting through, it’s actually a novelty these days when someone picks up the phone and has the audacity to speak to me.
A typical day for me starts with a latte (and more often than not a chocolate chunk cookie) from Starbucks followed by a quick prioritization session. What things am I going to do today that will have the biggest impact on the company I work for? I could attempt each day to deal with email and tasks as they arrive on my desk in the vain attempt that I’ll keep everyone happy which normally requires working till 2am in the morning each day. Alternatively, I can be smart with how I work and push back of things that are less of a priority or have no tangible impact on the business.
Traffic lights don't always reflect the true business impact
What I go through daily as a product manager is pretty much identical to what operations and application support teams go through each day. Most support teams get email, in fact they get several hundred email or even several thousand emails as a result of the enterprise monitoring solutions they have hooked up to every component of their infrastructure. They have alerts and traffic lights configured for their OS, networks, storage, middleware, messaging, databases and users across hundreds of applications and thousands of physical servers. Customer’s enterprise dashboards turn red and stay red because they simply cannot deal with the volume they receive daily. It’s a monumental task to browse through alerts and put all the pieces together in the attempt that you can identify and isolate an issue before the business picks up the phone and starts asking questions.
More importantly, 99% of these alerts have no business context. The alerts contain technical information based on KPI metrics for a given threshold breach or state, they do not provide any visibility into how the alert is impacting the business. If an enterprise monitoring team receives 5,000 alerts a day how can they make sure they deal with the 3 or 4 alerts that are impacting the business vs. the 4,997 alerts that are just noise?
The answer is Business Transaction Management. When you can manage all business transactions across all tiers all of the time you have total visibility into how your business runs on IT. More importantly you can quantify business impact in real-time by seeing with your own eyes which business transactions, users and applications are experiencing service level breaches. You manage IT with business impact so that you can truly prioritise your teams and resources to deal with the incidents that are most detrimental your business. Gone are the days when your IT support department manages IT with traffic lights based on infrastructure alerts or by investigating each alert as it arrives in the inbox that is running out of disk quota.
Not all business transactions, users and applications are equal. Just like not all emails, enhancement requests and PRD’s are equal for a product manager. If you can’t prioritize and focus on the things that have an impact on your business then the amount of value you’re providing to that business is pretty questionable. In many organizations the business is IT and without IT the business would fail. It’s therefore essential that IT is aligned to the needs and priorities of the business.
2 comments July 28, 2009
Business Transaction Management has Disco Fever
Life is dull when you can predict everything that is going to happen. For instance, I was driving home last week in rush hour on the M4 in the fast lane and in my mirrors I could see a black car approaching quickly. A few seconds later this black TVR Tuscan with big yellow stripes was behind me, pretty cool and a pretty rare sight on a motorway. As the traffic ground to a halt the owner of the TVR pulled into the middle lane next to me and rev’d his engine to prove a point whilst looking at me with a smug grin. The first thing that entered my mind was “Your car’s not going to last long sitting in this traffic mate”. Guess what? A few minutes later smoke started pouring out the front of this TVR with the owner looking pretty stressed. I was laughing and feeling smug also but not surprised in the slightest as the TVR pulled into the hard shoulder in a cloud of white smoke. For those not familiar with TVR sports cars, they are about as reliable as the Windows Operating System with no firewall or anti-virus protection – you leave them to idle and your in trouble.
Today, I can’t help thinking that enterprise monitoring is largely predictable, or even somewhat dull despite servers and applications going up in flames occasionally. For the people who manage helpdesks or application support, monitoring software is about as interesting as watching a set of traffic lights for 8 hours a day. The lights turn red, all hell breaks loose and the blame finger comes out. The lights stay green and you can kick back on Face Book or Twitter and see whose updated his/her status (only joking) or read blogs that describe just how your feeling
Enterprise monitoring needs an adrenaline boost, it needs mojo, it needs to shock and deliver answers to problems that you would never have predicted or guessed. If you can predict or assume why outages might have occurred then it becomes quite boring blaming the same DBA or network administrator every week. When the database is slow everybody assumes it’s a missing index or the DBA hasn’t updates the table statistics for several years. If the JVM is firing OutOfMemory exceptions everybody assumes it’s a memory leak and gets paranoid about finding the irresponsible code without checking JVM memory parameters first like MaxPermSize which will often resolve 90% of memory issues. Another classic example is where a JMX metric shows connections to the database are being exhausted so the first thought is to increase the database connection pool size in the JVM without actually figuring out what’s holding onto the exhausted connections (like slow SQL) in the first place.
Imagine if your enterprise monitoring software provided you with answers that shocked you. Imagine if you were in denial for a split second or even freaked out at the prospect that the solution to your problem is something which you’ve never even considered before. To be shocked you and your enterprise monitoring software first needs to be able to discover new things. The traditional way to deploy enterprise monitoring software is to ask the customer “Which servers/tiers do we need to put an agent on or monitor?”. This approach means customers get visibility into the server/tiers they are expecting their application and business transactions to flow through. The data provided is therefore predictable and somewhat unexciting.
Forget about monitoring servers/tiers for a moment (or a few years). Imagine if you monitored business transactions instead and their respective flows – things start to get interesting very quickly. Wherever the business transaction goes so does your monitoring capabilities and visibility. You begin to discover servers and tiers that you never imagined your business transactions or applications utilised. You begin to learn new things about how your applications and business transactions behave, you learn their dependencies, their interactions and more importantly their contributions in managing your service levels and end user experience. Welcome to the world of Business Transaction Management (BTM).
Being shocked is a good thing
I’ve seen many customers shocked, in denial and more importantly buzzed about what BTM can do for them and their organisation. Seeing a customers face is priceless when you tell them that their business transactions flow from their production application servers to a UAT test database. It’s even more priceless when you here them pick up the phone and describe it to other people in their organisation that real users business transactions are executing against a UAT test database. Its also impressive to show customers their real application topology based on business transaction flow than to keep referencing the partial diagram they think their application actually uses. It was only last week where BTM pointed out four application tiers to a customer that had no idea the tiers actually existed. Shock, denial and then amazement would be how I described that customer.
Apologies to those reading this blog who were expecting references to Disco Funk, big hair, big flairs and the king of pop. All I can say is that Business Transaction Management discovers lot of things that make life a bit more exciting and unpredictable. If everything was predictable then managing IT wouldn’t be fun each day.
Add comment July 20, 2009
There is more than one way to skin a cat
I admit this idiom made me cringe the first time I heard it. Hebrew doesn’t have a similar idiom and maybe the fact that I was in Texas at the time also added to my shuddering. but the concept stuck with me and the broader context of the conversation I was having was actually about the value of BTM and the reality that there is more than one way to manage applications, handle new releases, plan for capacity and resolve performance issues. OpTier was competing with a traditional Application Performance Management (APM) vendor for some new customer business. The customer was particularly interested in improving the speed in which they are able to fix performance problems and recover from intermittent service interruptions. My Texan partner was worried about the fact that the competitor has shown the customer some impressive details about how their application server code was performing including method names, connection pool metrics and lots more techie details. They did not show transaction tracking capabilities but this customer has an established relationship with our competitor and my partner worried that it will only be too easy for them to sign a contract with an existing vendor instead of forming a new relationship with OpTier. “But what about the problem they have?” I asked how the relationship or level of detail is relevant to their objective of fixing performance issues quickly. “Well, there is more than one way to skin a cat” was the answer. This got me thinking immediately. after all it’s true, and I don’t know about the cat thing, but I accept that there are ways to find and even resolve performance issues without knowing much about the transactions and the users that execute them (all hands calls, log digging, profiling and correlating data from multiple places etc..) so the question begs: if it’s easier and a little cheaper for the customer to get an APM tool (aka “deep dive” solution) from an existing vendor why should they invest in a BTM solution when their sole purpose is fixing performance issues?
The answer is efficiency; it is simply much more effective to deal with performance problems from the transaction perspective. it’s the best way to know what your end users were doing and where exactly things went bad, It completely eliminates the need for all hands calls and it is a lot simpler than digging through logs or correlating a ton of sophisticated metrics from multiple sources.
We decided to work together with them to see how we can translate this BTM efficiency into real gains that would justify BTM over the alternative. they had 6 critical applications that wanted to deploy on and the results they came up with were truly impressive.
They did the numbers, not us and their calculation pointed to a 70% reduction in expenses directly tied to time and material relating to IT work on performance issues (using data from the last 12 months) – yes there are more ways than BTM to achieve their tasks but here was a clear indication of what the best way was, indubitably.
Yesterday I searched the web for an idiom that describes the best way to do something and couldn’t find it…. My colleague and fellow blogger Stephen Burton suggested “kill two badgers with one boomerang”… he’s not from Texas and I still don’t know of an idiom to describe the best way to do something and doesn’t include butchering anything but I know that come hell or high water BTM is the most effective way to deal with performance issues quickly.
1 comment July 15, 2009
Connecting people
Like many other cell phone users I see the message “Connecting people” every time I turn on my phone. More than just giving away my manufacturer preference, I wanted to share in this post why that message strongly resonates for me as I think about BTM.
When discussing BTM a lot of the conversation centers on the technical complexity of business transactions. So for instance we were talking the other week about a simple “bill payment” transaction. Seems such a “simple” payment transaction flows in many cases across as many as ten IT environments: a client service portal (web front ends, security appliances, front end portal servers, portal personalization databases), then crosses over to the payment application (payment system app servers, master payment database and the interfaces to external payment services). There is huge value in exposing this technical flow across the two apps and the eight technical components.
But if we look beyond the technical complexity we’ll see that each and every one of the steps in this flow has a person behind it: from the business user behind the payment itself to the IT people behind the apps and infrastructure pieces. Everyone is focused on making their part of the overall story work well – but they all have very little to go by in terms of aligning their efforts with the overall goal. And the overall goal that really matters is getting that payment thru, without glitches, without holdups, time after time.
Well, transactions, when made visible, provide us the ultimate facility to connect these people, business and IT alike. Transactions expose the “lines between the dots” and provide a meaningful context to the people who participate in planning the systems, creating the apps and operating them. They allow them to focus on their jobs while effectively communicating with their peers. They provide a shared context around which they can all connect.
Think about it : “Transactions – connecting people”.
Add comment July 9, 2009
Why Business Transaction Management is not average
Sometimes in life the truth hurts. I remember the first time I went to parent’s evening at school and listened to my teachers telling my parents what I was actually like in the classroom. I was seven at the time and it was quite possibly the worst experience of my life… bar the occasion I fell down an escalator in Covent Garden tube station in front of about 200 people in rush hour. School teachers would often say “Stephen is a pleasant pupil for most of the time….however sometimes he loses his concentration and his work suffers as a result.” It was a polite way of basically saying “Stephen can be a total liability at times when he’s not paying attention in class.” The truth was hard for me to take but over time it stopped me being a liability in the classroom and eventually turned me into the nice, pleasant and text book professional I am today (I excelled at creative writing, BTW).
A more composed photo of me at School.
Business Transactions are just like school kids; for most of the time they behave themselves until one or two cause chaos and the next thing you know it’s all kicking off and people start shouting. When it comes to finding out who was responsible for the chaos it’s often an impossible task unless you know what every school kid or business transaction was actually doing at the time. The fact that the majority of school kids or business transactions were behaving is irrelevant if the class or application is currently in turmoil.
Being able to pin point the culprits is often an impossible task when you have to work with data that has been summarised and averaged. For example, if you’re managing a trading application and a single trade worth millions takes 50 seconds to execute and you’re looking at data that is telling you that your average trade response time is milliseconds then you my friend have had it.
This summarization of data is often a key reason why organisations continue to have issues despite investing several million in Enterprise monitoring technology. The reason for this is that many traditional monitoring vendors focus on tiers. They collect every KPI and metric available for a given tier to ensure they have a wealth of historical data available for later analysis. The problem is that these vendors can’t collect every metric from every tier all of the time, so they resort to sampling and summarization of data to reduce data storage and agent overhead. The net result of all of this is that the data the user looks at is averaged.
For example, the average response time of a Servlet, API call or SQL statement in a tier could be 2.3 seconds. If the average is low then everything looks great – the problem is that it’s impossible to spot exceptions or differentiate response times that are specific to a single user or single business transaction. A problematic business transaction will often get masked or unnoticed in the mix of similar business transactions that ran fine with no issue.
Today many IT vendors are providing Business Transaction Management (BTM) capabilities that focus on monitoring business transactions rather than tiers. The guiding principle is to track all business transactions across all tiers all of the time providing customers with complete visibility and definitive data. In the same way that a school teacher finds out Billy started the class riot by super gluing Stephanie’s hair to the table you can find out why an individual business transaction ran slow because you know exactly how long that business transaction spent in each tier it flowed through. You can therefore identify and isolate business impact in seconds.
Hundreds of IT professionals week in week out are being blamed for application outages or sev 1 issues. Why? A code red generally means twenty people get on a call and start investigating the network, web servers, app server, database, storage and so on. They do this without considering what the business impact actually is and what tier is actually causing the business impact in the first place. People still fly blind and make decisions on data that isn’t entirely accurate like average response times. Personal assumptions are therefore made which can often add fuel to a fire that is already raging.
The good news is that these BTM solutions allow you to manage with fact because the data you look at is real, accurate and definitive. It hasn’t been summarised, averaged or lost. BTM provides you with total visibility into who executed what where and for how long. This basically means you have all the facts before you make that code red call and bother 20 people. You therefore slash Mean Time to Resolution (MTTR) and in the process improve the quality of life for the people who you work with. You’ll blame them less and they’ll spent most of their time on work they enjoy rather than fire-fighting or being in denial every time they look at subjective data from other people.
Stop guessing and start knowing with Business Transaction Management.
1 comment July 9, 2009
Google wants to make the web faster. Good news for the enterprise?
Raise your hand if you’re nostalgic for the days of waiting for web pages to load. That’s right, the days of going to make coffee while you wait for Netscape to load pictures of your five-year old niece’s birthday party on Geocities. We’ve come a long way since the “World Wide Wait” and thankfully today we take for granted instant access to high bandwidth services such as video and browser-based SaaS.
Well, the good folks at Google want to speed up the web even more, and are using their bully pulpit to get the industry to make the web faster. Their focus on improving web performance centers around four areas for improvement:
- Web pages are slow due to suboptimal use of Javascript, CSS, compression, etc.
- Web servers are often not optimized for speed
- TCP, HTML and other protocols, designed 10+ years ago, do not meet today’s requirements
- Users are not using the latest set of speed-enhanced browsers
While addressing these four issues will go a long way towards improving performance of access to rich media and on-demand applications, it misses the mark for what’s needed to improve performance for web based services that are built upon a multi-tier architecture. Business applications, e-commerce applications, and enterprise portals are typically deployed with a complex set of tiers behind the web server through which transactions flow.
Google’s primary focus on streamlining the flow between the browser and the web server may serve the needs for many consumer web applications (think faster Youtube and Google Docs), but ignores the needs of businesses that are working to improve the performance of their transaction flows end to end. As we’ve said here before, the value of true Business Transaction Management (BTM) is the visibility it provides into all transactions across all tiers–from the clients and web servers, to the application servers, load balancers, databases, mainframes, authentication servers, message busses, ginsu knives, and any other tiers in the environment—in order to optimize the flow of these transactions.
While we laud Google’s efforts to speed up the web, it seems to us another example of how Google’s approach to software doesn’t work for enterprises. We’ve got our own ideas for how to “make the business service faster” through standards, enhanced vendor collaboration, and a business-centric approach to IT management. More on that in future posts.
Add comment July 2, 2009
Lindsay Diamond
Assaf Amit
Andy Wetzel
Stephen Burton
Russell Rothstein
Motti Tal
Marie-Pierre Belanger
cpennoptier
Linh C. Ho