Pyzdek Institute » statistical technique

Jumping to Statistical Conclusions

Thomas Pyzdek — Tue, 08 Sep 2009 07:00:08 +0000

Have you attributed your results to the right base data?

It may come as a surprise that the biggest challenge facing black belts and master black belts is usually not in selecting the best statistical technique for analyzing a particular data set. Most statistical techniques work fairly well even if the underlying assumptions are not precisely correct. If a black belt supplements the numerical analysis with graphical evaluation, the chance of making grossly erroneous decisions is negligible.

A mistake that is far more serious–but far more common–is comparing the results of a study to the wrong base data. These “apples to oranges” comparisons often lead to poor decisions and, worse still, to inaccurate beliefs that can derail faith in the Six Sigma approach itself. A recent incident with a client brought this point home for me.

The situation involved a project in the sales organization of a software company. The company had several sales teams and wanted to know if a new approach to closing the sale would improve the rate of closing sales. The company didn’t have a Six Sigma program, and the project was planned and carried out without black belts. The results were presented to management in a classic form: a bar chart (see Figure 1). The team had declared victory, and management–convinced by the “data”–prepared to revamp the sales training to incorporate the new approach companywide. All of the leaders looked forward to the bottom-line improvement they’d see from a 29-percent improvement in the sales closing rate.

Figure 1: Sales Closing Rate Improved by New Approach

All of the leaders, that is, except Lorraine. She’d received green belt training from her previous employer, and she’d seen enough black belt presentations to know that the analysis of the sales team was seriously flawed. It was undeniable that the project team’s sales close rate was 2.53 percent higher than the sales close rate for the rest of the sales department during the 16 weeks of the test, and, yes, the 2.53 percent did represent a 29-percent improvement over the 8.83-percent rate for the rest of the team. Despite these “facts” and the air of scientific objectivity surrounding the analysis, Lorraine had many unanswered questions. She asked management to delay any decision until she could explore these questions with a Six Sigma consultant. That’s where things stood when I entered the picture.

Table 1: Old vs.
New Closing Rates

Lorraine viewed the analysis as important because it would demonstrate that the Six Sigma approach could be applied in this service company, something that skeptical managers didn’t believe. In a meeting with the sales team leader, I was presented with the data shown in Table 1. As often happens, this summary data was all that was available; for a variety of reasons (but chiefly due to a time constraint) the number of sales calls used to compute these rates could not be obtained.

If you are a black belt or master black belt, or just statistically inclined, please take a couple of minutes before reading the remainder of this column to think about the data and jot down how you’d proceed from here.

When dealing with the data in Table 1, it’s tempting to apply a statistical technique such as a paired t-test to it. Using Microsoft Excel, it’s a simple matter to compute the t-statistic, which is 4.55, a highly significant result. Statistical purists would ask if the data are approximately normal and an endless variety of other technical questions about the data. I would argue, however, that all of this is premature and, ultimately, beside the point. The first order of business is to determine if we are comparing apples to apples.

Table 2: Apples-to-Apples Comparison

Further discussion revealed that the company had not two but nine sales teams, all of the same size. A further complication was that the teams sold different products. More probing uncovered the fact that four of the eight other teams sold a product mix similar to that of the team using the new closing method. At this point it appeared that, to make an apples-to-apples comparison, you would assess the results of these five teams for the 16-week project. Descriptive statistics are shown in Table 2.

Table 3: Data Groups

Further analysis using nonparametric methods indicated that there are three distinct groups in these data (see Table 3).

Table 3 presents a decidedly different picture than was originally given to management. The new closing method now appears to be no better than normal. Still, there are bright spots. Assuming that teams 5 and 8 aren’t oranges being compared to apples, potential gains should be possible from discovering why team 5 performs under the norm, and why team 8 outperforms the norm. More information might also be obtained by plotting the 16 weeks over time to identify trends and other patterns. Using the Six Sigma approach, the information can be converted to knowledge, the knowledge to action, and the action to an improved bottom line. It’s more work than the old standby, the bar chart, but it’s worth it.

The complete data file used in this article is posted at www.pyzdek.com/2000-05.xls . The challenge is to analyze the data in a number of different ways to determine how the different analyses would affect management decisions. Send your results to me for inclusion in a future column.

Thinking Outside the Box

Thomas Pyzdek — Sun, 07 Jun 2009 07:54:16 +0000

The “P” in SPC stands for process, not product.

A common problem with SPC is that the world appears too complicated for a statistical approach to work. In complex electronics products, for example, circuit boards may have thousands of holes and microchips may have millions of transistors. Plotting control charts of each and every dimension is clearly not feasible. What can be done?

To answer this question, consider a simple product: the box in Figure 1. How many things could we measure on this box? It turns out, a great many. Length, width and height are obvious choices. But we could also measure the diagonals on all six sides, interior diagonals front-to-back and back-to-front, linear combinations of these measurements and a great many more. We could conceivably come up with dozens of measurements on this simple box.

But–and this is critical–we don’t need these measurements to control the box process. The “P” in SPC stands for process, not product. When we focus on the product, we lose sight of the fact that we’re not trying to control the product. Control of the box process may be a great deal more simple than controlling the product. And if we control the process properly, the product will take care of itself.

The statistical technique known as principle components analysis can help us determine just what is important and what is not. Most statistical software packages can perform PCA. To illustrate the approach, I measured an assortment of boxes (see figure 2). The measurements I obtained are shown (in inches) in Table 1.

When these data are crunched through PCA, we find that three principle components explain 99 percent of the variation in the data set: Component No. 1 explains 76.9 percent of the variation, component No. 2 explains 14.1 percent, and component No. 3 explains 8 percent. The PCA clearly shows that these three components are associated with A, B and C respectively. Thus, the “box process” can be characterized almost entirely by controlling these three characteristics. If we do that, the other dimensions will be OK, too.

An example of using this approach in the real world involves CNC machining. A defense plant machined parts for use in guided missiles. The parts were extremely complex, with thousands of holes, cutouts, etc. on each. However, when the data were analyzed using PCA, it was determined that four principle components accounted for nearly all of the process variation. Further study showed which measurements were correlated with each principle component.

From this, the engineers determined that, for all the apparent complexity, the machining process was, in fact, quite simple. The four principle components corresponded with the machining center’s four axes of movement: X, Y and Z movement of the bed, and the rotation of the table on which the parts were mounted. SPC could be accomplished by selecting those features most difficult to position in each axis of movement. Often, a single feature could measure more than one axis; for example, a hole furthest from the “home” position in both the X and Y axes. The result: One or two control charts suffice for the control of a process placing thousands of features.

Note that the features selected for SPC may be of little or no importance to the product itself. In fact, some parts were designed with “process control features” that were later removed from the part entirely. This makes sense when remembering that P stands for process, not product. If you keep that in mind, the complexity you face might just evaporate before your eyes.