User Acceptance Testing – Dealing with Clever Users

I have never been particularly fond of the ubiquitous phrase, “where the rubber meets the road.” However, if ever there were a situation when that phrase applies, it is user acceptance testing (UAT). After all the planning, designing, meetings, reviews and internal testing, there is one final act before the buyer (or internal client) accepts the software or solution. Yes, your team and you have worked months to perfect it but the real gate for saying it is ready is when it is tested by real users on their system.

Bad User Acceptance Testing Strategies
Waiting until the end of development to think about user testing: NO-Start early. Plan the user testing from the beginning of the project. Build user testing into the schedule. Do NOT fall victim to the strategy of, “we’ll rely on the results of system testing and skip user testing to catch up with a delayed schedule.”  Those of us with lots of years experience can tell you that you build in testing and most surely with actual users if you want the system to work in the production environment.

Not allowing plenty of time for user testing: In a report from Sentry, they found that (and I paraphrase from their report: “assume that you have 120 test items such as producing a specific report or scanning 10 batches of 30 documents each into a database or generating an export file of 600,000 records. Further, assume it takes about 3 hours to setup each test cycle. Do the math. The time to setup, test, collect data and analyze approximates 360 hours.)  So how many projects build that into their schedule?  The ones that do I will bet find lots of real world issues and work them out before deployment.

Your project’s testing requirements will differ from any other projects because your product and situation are unique. However, the point is that meaningful and useful user testing takes time and planning.  You might even do something novel like include users up front in planning what testing and acceptance criteria they will use!

Here’s another failing user testing strategy from our friend, Dilbert.

OK- so enough bad strategies, how about a few tips on what to do right:

Good Advice on User Acceptance Testing:

  1. Include user-testing requirements in the original proposal, plan, cost and schedule. Use input from your business analyst or project team – that include customer representatives – to generate typical scenarios and scripts for user testing.
  2. Create a detailed plan of what the UAT is testing. You do not want a bug report that says, “The system failed to turn straw into gold.” User testing should be done based on agreed requirements, not everything a user may want the system to do.  In other words, bound the scope of the testing to be in line with the solution.
  3. Select test subjects (let’s call them users) with a range of experience and diversity of backgrounds following pre-defined scripts to get from task beginning to task complete.  Do not pick all junior or all experienced users, a mix is the best.
  4. Train your users (test subjects) how to report problems. “User Acceptance Testing War Stories – Writing a Perfect Bug” suggests that testers write down the steps they followed that led to failure. Then wait a few hours and try to follow the steps as they wrote them to see if the bug appears again. You are teaching testers how to find bugs and report them with sufficient detail to developers to repair.  Use some online tool if possible – it makes it easier.  Even a SharePoint list can be used to track the items that come out of the UAT.
  5. Maintain detailed reports of testing including procedures and outcomes. Communicate findings with the team, management and clients. Depending on the timing of the UAT, you may want to submit status reports daily or weekly. Besides a cursory pass/fail, add some analytical information about causes or commonalities.

Remember that user testing is about making sure the system does what the customer claimed was needed to help them do their job. If the specifications and earlier plans, all agreed to by the parties involved, resulted in a product that does not support user’s needs, document the revised needs and begin the process of negotiating the follow on contract.

Best Practices and Lessons Learned:
This is one of my favorite topics and I have made almost every mistake from cutting out user testing to letting users test things not included in the original scope and requirements.  A couple of Lessons learned that I hope you can avoid:

  1. Get the client/stakeholders involved at the beginning to define what the testing will need to be.  It is like having the students help write the test – they can’t say it was a bad test.
  2. Balance the formality of the UAT with the scope and length of the project and solution.  I have seen too much formality for a small system acceptance and not enough rigor for a large scale solution being deployed for thousands of users.
  3. Don’t start out user testing or UAT with negative points – in other words don’t sound like a lawyer!  If you have done everything right, including getting the users to help up front, then this is an exciting time where they get to try out the system before everyone else.  Kind of like test driving the new model car before everyone else.

So what are your tips or experiences in UAT?  I am sure others would love to hve you share.

Thanks for sharing.

Dealing with Angry Clients

Recently, while sharing a quiet dinner at a nice restaurant (in lovely Regina Canada), I could not help but overhear the conversation at the next table. The patron was angry and demanding that the cook re-do the meat because it was not to his liking. I paid attention to this because the customer had already returned his wine as, “inferior” and the atmosphere around that table was toxic.

Now, this person may have just been a jerk or perhaps he had a very, very bad day. However, not all anger is unjustified. People get angry for reasons. Sometimes those people are clients and sometimes they are right.

To be honest, whether the client’s anger is justified is not really the issue. The situation is the issue. You, the project manager or program manager have to deal with it. By “deal with it,” I mean you need to diffuse the situation and move forward. Angry clients or employees often do not keep their disappointment or anger to themselves. They tell friends, co-workers and even competitors.

Reasons Clients Become Angry

  • They can’t get scope changes for free. (Do not be surprised if this anger involves some posturing, since agreeing on cost and schedule is a negotiation.)
  • Your project did not deliver what was promised.
  • The project manager or key personnel changed without notice.
  • Your project missed a milestone or cost bogey (“target” for those non-military types).
  • Some “user requested changes” were over-ridden by the client’s senior staff or the project team.
  • What they want is not possible given project cost and schedule constraints. (the old project triangle theory)

Reasons Staff Become Angry

  • Scope creep: Workers agree to do a certain amount of work within a specific time. Then, customers, clients, senior management – someone – keeps adding tasks without changing the schedule or adding hours to do the work.
  • Organizational change: Events outside of the project alter the organization’s usual way of doing business, which eventually trickles down to the project staff as added training, forms and procedures. (See: “Don’t Take Organizational Change for Granted – Manage it”)
  • Perceived lack of appreciation or respect. (Don’t under-estimate this one!!)

Dealing with Anger

  • Accept that the person is angry. They may be angry with you, your company or your team. They could be showing displaced anger from situations in their lives outside of the project. Alternatively, they may display an angry pose as their way of intersecting with the world. Therefore, acknowledging the anger is a place to begin repairing the situation, if it can be repaired.
  • Clarify the situation. What is the client angry about. If there are several points of anger, write them down on paper or on a whiteboard and address each concern.
  • Be careful of your body language. Relax. A red face, clenched jaw or fist pounding does not help.
  • Some clients express anger or disappointment passively. If a client is consistently slow to return emails or voice messages, misses meetings, stops contributing to discussions, check with them about their state of comfort with the project and ask them if there is a problem. Do not assume they are angry; just give them an opening to discuss their perceptions or concerns.
  • Provide realistic feedback. If you or the project created the problem, acknowledge it and see if there is an acceptable resolution. If the problem itself is not solvable, acknowledge both the concern and the reality of the situation. “I realize you wanted the entire system to be compatible with your legacy software, John, unfortunately that cannot be done because the systems handle data differently. To write a translator would cause the performance to degrade below your minimum expectations …” etc.

Best Practices

  • Whenever possible meet face-to-face.
  • Deal with the situation quickly – do not let anger fester.
  • Stay calm, speak quietly and do not escalate the situation.
  • Invite a senior member of your organization to the meeting to demonstrate your organization’s concern and commitment to improving the situation.
  • Follow up a confrontational meeting with a call or email that describes the resolution and your commitment to meet the new schedule, do the task or just check into current perceptions of the project.
  • When dealing with an angry project staff, follow many of steps suggested above – listen, accept the anger, be realistic and if changes need to be made either make them or make it clear you are working with decision-makers to improve the situation.
  • Keep the end goal in mind, you want to leave the meeting with the anger reduced or hopefully replaced with positive feelings.
  • Monitor yourself – stop breathe and take a pause to think.

Seth Godin: How to Deal with an Angry Customer

Steven Flannes, Ph.D.: Working Effectively with the Angry, Critical Client: Real World Solutions to Help You Get the Job Done

Click to access 107-30.pdf

Bloomberg Business Week: Dealing with Angry Customers

The Lazy Project Manager's Blog

The Home of Productive Laziness Thoughts

Thoughts, experience, tips and tricks on issues affecting managers and project management

A Girl's Guide to Project Management

Project Management musings for one and all

LeadingAnswers: Leadership and Agile Project Management Blog

Thoughts, experience, tips and tricks on issues affecting managers and project management

Project Management Hut

Thoughts, experience, tips and tricks on issues affecting managers and project management

Herding Cats

Thoughts, experience, tips and tricks on issues affecting managers and project management


Pushing the Edges Out ...


Just another site