How big is a test case?

I run an automated testing team, and I've noticed something interesting lately. As we start adding more test automation, and looking at the test automation numbers used by other teams, it's impossible to tell at glance what the scope of a test case is. What does it mean to say that you have 10 automated tests for a particular feature area? How does that compare to 10 manual tests? Some of the tests that we run in our automation system are pretty much direct ports of manual test case. Often the manual test case was defined by another tester at some point in the past. Many of the tests that are currently being automated are tests of the rules engine. In the test you create an rule, apply it to the device under test, pass traffic of various types through the device, and look at the results. The test may set 5-20 parameters on the device, and check the results of a similar number of distinct pieces of traffic sent through the DUT. This all counts as one test. In contrast, some of our automated tests contain an enormous number of setting changes and traffic checks per reported test case. The SSL test suite that we've built around the codenomic does 7200 discrete pieces of traffic for each of the 144 tests that we report in automation summary. Our compression test suite tests 432 discrete settings for each of its test cases. We have 480 test cases in our compression suite, for a total of 207,360 checks. If we were to say that a manual test case averages 15 checks, then the compression test suite would be the equivalent of nearly 14,000 manual tests. Obviously, it isn't the equivalent. The manual test suite for the entire project only contains about 2000 tests. So how big should a test case be? Is it possible to define tests in such a way that you can meaningfully compare manual tests to automated tests? Would that be worth doing?

Got called by one of my employees on Avoiding Productive Conflict

I had a one-on-one meeting this morning with one of my reports. He was expressing some frustration about working with one of the other team members. He also was concerned that his ideas were being ignored. As we discussed it he said that I tend to side with another team member most of the time, and that has discouraged him from raising issues.

I thanked him for calling me out on it. I said I was really pleased to see him do that--it's tough to tell your boss that he's doing something wrong. I complimented his courage to bring it up directly with me. This is a good example of my own tendency to prefer harmony over productive conflict, the fourth of the Five Temptations of the CEO.

He is pretty conflict averse himself, and not very verbally articulate (he needs time to think before he can express himself clearly), so he is not naturally inclined to push his ideas in meetings. We talked about various ways to handle that. I suggested that he write up his ideas and come talk with me directly. Then after he and I have had a chance to discuss the ideas and flesh them out, he could raise them in the larger group.

I was really pleased, both with him for being willing to take the risk he took, and with myself for creating a team environment in which he felt safe enough to take the risk.

Cooking Lesson

The ten-year-old has developed an interest in cooking. Unfortunately, he only wants to fix elaborate, time-consuming dishes that we don't have the time and energy to help him with. This is a very common issue for the ten-year-old. He always wants to start at the top, by doing something advanced and complicated. We have not yet managed to persuade him of the value of starting with something simple and working his way up.

This weekend we found a way to accommodate his interest. He found a recipe that he liked, scanned it, and emailed it to Grandpa. Then we went over to Grandma and Grandpa and Grandpa and the ten-year-old made dinner. I was very impressed with how Grandpa worked with him. When we got there are the ingredients were laid out on the counter. The recipe called for two chopped onions, and Grandpa had done most of the chopping ahead of time so that he would be able to keep things moving quickly during the actual cooking. He gave not only clear instructions, but very clear explanations of the reasons behind the instructions.

It was really nice to see them both enjoying doing something together.

What's the difference between a waste basket and a laundry hamper?

There is no difference. Not if you a seven-year-old with autism. Yesterday I emptied the waste basket in the bathroom. In addition to trash, I found 3 pairs of pants, 1 shirt, a water bottle, and a half-dozen plastic farm animals. This morning I was taking stuff out of the hamper to do laundry. In addition to the dirty cloths I found 5 shoes, 1 drinking glass, 3 dirty diapers, 5 candy wrappers, and a partially eaten bagel. Apparently the seven-year-old has been very busy.

The Five Temptations

I stumbled across a book at the library last week called The Five Temptations of a CEO: A Leadership Fable, by Patrick Lencioni. Intrigued by the title, I checked it out and took it home. It turned out to be both a good book and a quick read.

The ideas in the book are neatly encapsulated in “The Model”, a short section towards the end of the book. The model lists the five temptations:

• Choosing status choosing results
• Choosing popularity over accountability
• Choosing certainty over clarity
• Choosing harmony over productive conflict
• Choosing invulnerability over trust

I’ve been thinking about my own strengths and weaknesses and how they manifest in this particular model.

Status over results

I don’t think that this is particularly a problem for me. My primary motivation for work these days is paying for my children’s autism therapy, not moving ahead in the world. I have worked with (and for) people for whom this was a primary motivation. The big problem that I saw coming out of this was risk-aversion. They didn’t want to do anything that might endanger their status.

Popularity over accountability

It is tempting to try and be friends with the people who work for you. When everything is going well, it isn’t a problem. But when there are problems, it usually makes the problem harder. I have, and do, struggle with holding people accountable, but popularity isn’t the core of the problem. The core of the problem for me lies in temptation number three.

Certainty over clarity

I am fairly risk-averse by nature. In my early years as a manager, I put off a lot of decisions that I should have made because I wanted to be absolutely sure that I was making the right decisions. With the help of a good therapist and some good managers, I’ve gotten a lot better at this in the past five or six years. The desire for certainty made it harder for me to hold people accountable. Because I was waiting to be certain, I didn’t commit to a decision and make it clear to the team. Since I hadn’t given them clarity, it didn’t seem fair to hold them accountable. That’s a point that the author calls out explicitly, and it really rings true for me. While I have improved here, this is still something that I need to watch out for.

Harmony over productive conflict

This is another one that poses a challenge for me. I spent 2002 working on a Masters degree in counseling (then my kids were diagnosed with autism and I realized this wasn’t a viable career change, I never went back for the second year of the program). One of the more interesting things that we did was something called the Thomas-Kilman Conflict Mode instrument. It rates you in various styles of handling conflicts. It took me a while to make sense of my results. My two high areas (much higher than the others) were Avoiding and Competing. In fact, they tied each other for #1. After thinking about it for a while, I realized that it did fit. I preferred to avoid conflict if possible, but once it was clearly unavoidable, I became very competitive and tried to “win”. In fact, I tended to come out swinging for blood. Neither of these are terribly effective ways of handling conflict, and now I know that I need to watch that carefully. As a manager, I need to make sure that I don’t squash the conflicts that a team needs to go through in order to evaluate alternatives and reach good decisions. At a previous company I watched one of the senior executives squash any conflicts that involved his staff. It prevented some serious personnel problems from getting solved. I work hard now to distinguish between unproductive conflict that needs to be squashed and productive conflict that needs to be fostered.

Invulnerability over trust

I’ve made deliberate choices to be vulnerable. As I see it, in order to be invulnerable, I would have to behave as though I didn’t trust anyone. That’s just too depressing of an assumption for me. Assuming that people aren’t trustworthy, and then acting accordingly makes me unhappy.

Lessons learned from Civ IV

I've been playing Civilization IV again recently, and realized that there are lessons in Civ IV that are applicable to a problem I've been wrestling with at work.

I run a team that builds test automation tools for a large company in Seattle. There are several other groups within the company that have built similar tools. Recently I was charged leading a team to look at all the different tools in use within the company and figuring out how to reduce the amount of duplicated work, as well as increasing the total amount of our testing that is automated.

After much conversation we came up with a kind of intellectual framework describing 6 tasks of test automation and 4 contexts in which automation is used within the company. Some time soon I'll post about that in more detail. The tool that my team has built addresses the most complex types of test automation, but is more cumbersome than some of the other tools that address only the simpler automation contexts. We've spent a lot of time trying to figure out how to make it easier to use our tool for less complex automation tasks, without a lot of success.

In my recent Civ IV games (I bet you were starting to think I'd forgotten about that...) I've been focusing on winning Cultural victories rather than military or scientific victories. In a cultural victory one of the most powerful things you do is assimilate the cities of other players. It occured to me that we could assimilate the other automation tools in the company--we can wrap our tool around them, use it to handle the most complex tasks, and let the testers and developers who don't need that complexity continue to develop their automation using the simpler tools.

Crazymaking

I'm experiencing what life would be like if we lived in 600 square feet. It would NOT be good.

Our house, built in 1971, has popcorn ceilings that contain asbestos. Recently, Younger Son, who has low-functioning autism, has started scraping it off the ceiling and eating it. We decided have it removed in the rooms where he can access the ceiling (he loves to be up high and climbs furniture; his favorite spot to watch TV is atop the wardrobe in the living room).  This is quite a project. Before the remediation company came we had to take everything out of the rooms where they were working, including the light fixtures.

The living room and two bedrooms had to be emptied, and the things in that 800 square feet of house stuffed into the remaining 600 square feet (plus the garage). All the rooms are overflowing. Younger Son, who hates change, is very distressed. 

He wants lights on all the time, and the light fixtures are gone. I ended up getting my shop lights from the garage. It was the only way to calm him down. Last night, after waking around 1 a.m. and realizing everything was different, he spent 30 minutes walking around the house carrying his laundry hamper. He set it down, then picked it up and moved it again. I think he was trying to find someplace to put it that would make everything seem normal. He had trouble sleeping and awoke twice more.

Tonight he and I are sleeping at my in-laws (they are out of town). It took him about an hour longer than usual to calm down for sleep. One of the last things he did was walk around the house holding his shoes and socks in his hand. He dropped them in waste baskets then took them out.  The second time, I realized what he was doing and led him to the laundry hamper. He put his shoes and socks in the hamper, he was able to lie down and sleep.