Normally when the bottom axis represents time periods, the more recent time periods are on the right side. Not so with this graphic of Schwab U.S. Treasury Money Fund dated April 30, 2010. On this chart the most recent period is on the left, not the right. Nice try Schwab!

When using graphics in quality and process improvement, care must be taken not to inadvertently “lie” with statistics. Sometimes even the best get it wrong.

Two researchers at the University of Arizona performed a study to determine why some Wikipedia articles rate high in terms of quality, while others score lower. Eller College of Management Professor Sudha Ram and Jun Liu, a graduate student, have found that entries on Wikipedia – the world’s largest open-access online encyclopedia – gain greater quality with contributions from people in many different roles. Sudha Ram, a UA’s Eller College of Management professor, co-authored the article with Jun Liu, a graduate student in the management information systems department (MIS). Their work in this area received a “Best Paper Award” at the Workshop on Information Technology and Systems held in conjunction with the International Conference on Information Systems, or ICIS.

Wikipedia has an internal quality rating system for entries, with featured articles at the top, followed by A, B, and C-level entries. Ram and Liu randomly collected 400 articles at each quality level and applied a data provenance model they developed in an earlier paper. “What was missing was an explanation for why some articles are of high quality and others are not,” Ram said. “We investigated the relationship between collaboration and data quality.”

To generate the best-quality entries, she says, people in many different roles must collaborate. Ram and Liu suggest that the results of this study should spark the design of software tools that can help improve quality. “A software tool could prompt contributors to justify their insertions by adding links,” she said, “and down the line, other software tools could encourage specific role setting and collaboration patterns to improve overall quality.”

In a letter published in the journal Nature (Nature 463, 527-530 (28 January 2010)) entitled “Ensemble reconstruction constraints on the global carbon cycle sensitivity to climate” the authors discuss the processes controlling the carbon flux and carbon storage of the atmosphere, ocean and terrestrial biosphere. These processes are likely to provide a positive feedback leading to amplified anthropogenic (i.e., human caused) warming. But the magnitude of the climate sensitivity of the global carbon cycle and thus of its positive feedback strength, is under debate, giving rise to large uncertainties in global warming projections. The paper describes a study designed to quantitatively estimate the feedback parameter, γ, based on pre-industrial CO₂ estimates based on “proxies” such as ice cores.

The authors conclusion:

“We find that γ is about twice as likely to fall in the lowermost than in the uppermost quartile of their range. Our results are incompatibly lower (P < 0.05) than recent pre-industrial empirical estimates of ~40 p.p.m.v. CO2 per °C, and correspondingly suggest ~80% less potential amplification of ongoing global warming.” (italics added.)

In short, the cabon cycle feedback effect is weaker than formerly thought by climate researchers. This will require a revision of the simulation models used to forecast climate change and will, in all likelihood, lower the projected impact of human activity on the climate. An amplification reduction in the 80% range could result in dramatically lower projected impact.

All models are wrong, some models are useful. Corollary: apply models with care and always temper their interpretation with sound judgment.

A reader asks

“I want practice  to SPC method to know whether my production process is in control, in case of all data available is from batch to batch, is it rational to construct the sub-group based on batch to batch data? What conclusion can I get from batch to batch? Any suggestion? Thank you very much.”

The answer is, maybe. I’d need a more complete description of your process so I can figure out what you mean. For example, I don’t know if your process is chemical, mechanical, or electrical. I don’t know if batches are arbitrarily created by filling a container from a larger container. Et cetera.

The guiding principle is called rational subgrouping. Your control limits should compare long-term variability to limits based on short-term variability. The underlying premise is that in a stable process there won’t be any long-term variability unless something substantial changed in the process, i.e., a special cause. Usually this would mean basing your control limits on within-batch variation and plotting batch-to-batch results against these limits. However, for some processes this doesn’t work because there’s too little variation within a batch compared to between batches. For example, in a homogenous chemical solution the within batch variation may be miniscule. The solution in these cases is to use individuals control charts and base control limits on moving ranges from subgroups formed by consecutive observations. And if your data are autocorrelated (i.e., observations taken at close to the same time are correlated), then the sampling interval of the individuals chart will need to be adjusted.

Take a look at your process and see if this works for you.

PS: You may also wish to look at the article by John David Kendrick.

Have you attributed your results to the right base data?

It may come as a surprise that the biggest challenge facing black belts and master black belts is usually not in selecting the best statistical technique for analyzing a particular data set. Most statistical techniques work fairly well even if the underlying assumptions are not precisely correct. If a black belt supplements the numerical analysis with graphical evaluation, the chance of making grossly erroneous decisions is negligible. A mistake that is far more serious–but far more common–is comparing the results of a study to the wrong base data. These “apples to oranges” comparisons often lead to poor decisions and, worse still, to inaccurate beliefs that can derail faith in the Six Sigma approach itself. A recent incident with a client brought this point home for me. The situation involved a project in the sales organization of a software company. The company had several sales teams and wanted to know if a new approach to closing the sale would improve the rate of closing sales. The company didn’t have a Six Sigma program, and the project was planned and carried out without black belts. The results were presented to management in a classic form: a bar chart (see Figure 1). The team had declared victory, and management–convinced by the “data”–prepared to revamp the sales training to incorporate the new approach companywide. All of the leaders looked forward to the bottom-line improvement they’d see from a 29-percent improvement in the sales closing rate. Figure 1: Sales Closing Rate Improved by New Approach All of the leaders, that is, except Lorraine. She’d received green belt training from her previous employer, and she’d seen enough black belt presentations to know that the analysis of the sales team was seriously flawed. It was undeniable that the project team’s sales close rate was 2.53 percent higher than the sales close rate for the rest of the sales department during the 16 weeks of the test, and, yes, the 2.53 percent did represent a 29-percent improvement over the 8.83-percent rate for the rest of the team. Despite these “facts” and the air of scientific objectivity surrounding the analysis, Lorraine had many unanswered questions. She asked management to delay any decision until she could explore these questions with a Six Sigma consultant. That’s where things stood when I entered the picture. Table 1: Old vs. New Closing Rates Lorraine viewed the analysis as important because it would demonstrate that the Six Sigma approach could be applied in this service company, something that skeptical managers didn’t believe. In a meeting with the sales team leader, I was presented with the data shown in Table 1. As often happens, this summary data was all that was available; for a variety of reasons (but chiefly due to a time constraint) the number of sales calls used to compute these rates could not be obtained. If you are a black belt or master black belt, or just statistically inclined, please take a couple of minutes before reading the remainder of this column to think about the data and jot down how you’d proceed from here. When dealing with the data in Table 1, it’s tempting to apply a statistical technique such as a paired t-test to it. Using Microsoft Excel, it’s a simple matter to compute the t-statistic, which is 4.55, a highly significant result. Statistical purists would ask if the data are approximately normal and an endless variety of other technical questions about the data. I would argue, however, that all of this is premature and, ultimately, beside the point. The first order of business is to determine if we are comparing apples to apples. Table 2: Apples-to-Apples Comparison Further discussion revealed that the company had not two but nine sales teams, all of the same size. A further complication was that the teams sold different products. More probing uncovered the fact that four of the eight other teams sold a product mix similar to that of the team using the new closing method. At this point it appeared that, to make an apples-to-apples comparison, you would assess the results of these five teams for the 16-week project. Descriptive statistics are shown in Table 2. Table 3: Data Groups Further analysis using nonparametric methods indicated that there are three distinct groups in these data (see Table 3). Table 3 presents a decidedly different picture than was originally given to management. The new closing method now appears to be no better than normal. Still, there are bright spots. Assuming that teams 5 and 8 aren’t oranges being compared to apples, potential gains should be possible from discovering why team 5 performs under the norm, and why team 8 outperforms the norm. More information might also be obtained by plotting the 16 weeks over time to identify trends and other patterns. Using the Six Sigma approach, the information can be converted to knowledge, the knowledge to action, and the action to an improved bottom line. It’s more work than the old standby, the bar chart, but it’s worth it. The complete data file used in this article is posted at www.pyzdek.com/2000-05.xls . The challenge is to analyze the data in a number of different ways to determine how the different analyses would affect management decisions. Send your results to me for inclusion in a future column.

A Black Belt steps up to the plate with Six Sigma confidence.

Bill had a problem. His company’s baseball team wasn’t doing that well, and he was part of the reason. Bill was in a long slump. Frankly, he stunk at the plate. But Bill is a Six Sigma Black Belt. He decided to approach his batting problem just like he would approach any process problem at work–by conducting a designed experiment. First, Bill determined which factors are important. He wrote up a lengthy list and then winnowed it down to four experimental variables (see Table 1). Table 1: Experimental Variables for Hitting Bill decided to spend a few evenings and weekends on the practice field swinging at 100 pitches for each of the 16 combinations of the four variables needed to conduct a full-factorial experiment. The field was equipped with a pitching machine that could be programmed to throw pitches at either 60 mph or 80 mph. Bill decided to count any ball that went past the infield in fair territory as a hit. Over a two-week period Bill was able to complete the experiment, producing the results shown in Table 2. Table 2: Bill’s Batting Experiment The analysis indicates that factors B and D, and especially the C-D interaction, make big differences in Bill’s performance. Factors A and C do not have a significant effect on Bill’s batting average. The analysis in Table 3 shows the details. Table 3: Significant Factor Effects The 95-percent confidence interval for C (position in the batter’s box) includes zero, meaning that C is not statistically significant as a main effect. (C is included because the significant C-D interaction term requires it for statistical reasons.) However, the other factors in the table–B (choke on the bat) and D (speed of the pitch)–are statistically significant. The most important factor is the C-D interaction, which has an impressive effect of more than 9 percent. The coefficient estimate tells us what happens to Bill’s batting average as we go from one level of the variable to another. For example, when B is at the high level (choke up on the bat two inches), Bill’s batting average improves by about four percentage points. The analysis indicates that when Bill is facing a pitcher with real heat (80 mph isn’t too bad for an amateur pitcher), he can improve his batting average from 8 percent to 28.75 percent by standing near the back of the batter’s box (see Table 4). Conversely, when Bill is up against a 60-mph hurler, he’s better off in the front of the batter’s box (38.75 percent in front hits vs. 15 percent in back). Combining all of these results, Bill’s strategy is to always choke up on the bat and position himself in the batter’s box depending on the expected speed of the pitch. Table 4: Bill’s Results Bill may not be ready for the majors with this strategy, but he’s hitting a lot better than the .206 (20.6%) he’d been getting without a strategy. In the meantime, Bill, work on hitting that fast ball!

The Obama administration has proposed, and the House of Representative has recently passed, sweeping legislation to deal with the problem of climate change or, more specifically, global warming. The problem is supposedly that human activity is releasing CO2 and this in turn is causing catastrophic increases in global temperatures. This hypothesis is supported in part by data from ground-based temperature recording stations situated around the world. The United States has 1,221 stations scattered across the continent and it is believed that the data obtained from these stations is among the most reliable and accurate available.

A recent report by the Surface Stations Project casts serious doubt on this assumption. Volunteers for the project have to date examined, photographed 819 of the stations and rated 807. Ratings are based on he Climate Reference Network Rating Guide – adopted from NCDC Climate Reference Network Handbook, 2002, specifications for siting (section 2.2.1) of NOAA’s new Climate Reference Network:

Class 1 (CRN1)- Flat and horizontal ground surrounded by a clear surface with a slope below 1/3 (<19deg). Grass/low vegetation ground cover <10 centimeters high. Sensors located at least 100 meters from artificial heating or reflecting surfaces, such as buildings, concrete surfaces, and parking lots. Far from large bodies of water, except if it is representative of the area, and then located at least 100 meters away. No shading when the sun elevation >3 degrees.

Class 2 (CRN2) – Same as Class 1 with the following differences. Surrounding Vegetation <25 centimeters. No artificial heating sources within 30m. No shading for a sun elevation >5deg.

Class 3 (CRN3) (error >=1C) – Same as Class 2, except no artificial heating sources within 10 meters.

Class 4 (CRN4) (error >= 2C) – Artificial heating sources <10 meters. Class 5 (CRN5) (error >= 5C) – Temperature sensor located next to/above an artificial heating source, such a building, roof top, parking lot, or concrete surface.”

The bottom line is that 89% of the sites examined to date are in categories 3, 4, or 5. In other words, they fail to meet established NOAA requirements.

A logical question would be, does it matter? Evidence indicates that it does. For example, here’s a photograph of a site that meets the standard:

A site that meets NOAA specifications

The chart shows that the temperatures recorded by this station have actually declined over time. Compare this to the results from this poorly sited recording station:

A site that fails NOAA siting requirements

The chart clearly shows warming. However, the cause of the warming is most likely the changes in the quality of the site over time, not due to human-induced increases in CO2.
In the business world we use quality and Lean Six Sigma to help us make good decisions based on facts and data. The process used is called DMAIC, or Define-Measure-Analyze-Improve-Control. In the Measure phase we assure that we have valid, reliable data before moving on to analysis and improvement. This is a lesson that policy makers in Washington and elsewhere should learn before embarking on policies that will cost trillions of dollars.

Unit yields are a misunderstood tradition.

Sam handed Peter a computer printout and asked, “If the yields are so high, why is my efficiency so low?” Peter studied the report for a moment and then nodded. “Let me show you what’s going on,” he said as he picked up a marker and drew a diagram (see Figure 1). Figure 1: Process with 10 Steps “This process has 10 separate steps,” Peter began. “Each step has a yield of about 90 percent. This is the unit yield for that process step.” “Right,” Sam interjected. “And all of them are about 90 percent, so the average yield for the whole process should be about 90 percent.” “Yes, but that isn’t the number you need if you’re trying to determine the final yield for the process,” Peter responded. “Final yield is the proportion of defect-free units out of the final process step relative to what you started with at the first process step.” Sam nodded. “Yeah, but even though the average yield is nearly 90 percent, our final yield is nowhere near that high.” Peter turned back to the board. “Here’s a mathematical model of what happens when all process steps have the same unit yield.” He wrote an equation: Yoverall = (Ystep)number of steps “The unit yield at every step is about 0.9, but you have to multiply the step unit yields together to get the final unit yield. You can’t just average them,” Peter explained. “Think of a simple two-stage process. You start 100 units at the first step and 90 pass. These 90 start the second step and 90 percent of them pass, leaving 81. The average unit yield is 90 percent, but the final unit yield is only 81 percent.” “So for our 10-step process,” Sam began. Peter punched his calculator keys. “0.9 raised to the 10th power is about 0.35. So 35 percent is your predicted final yield.” “And that’s pretty close to what we’re getting,” Sam said. Peter knew that misunderstandings on yields lead to a variety of poor management decisions. He was pleased that Sam had asked for clarification. But, Peter knew, Sam still didn’t know the whole picture. Six sigma requires an entirely different mental model of yields. “That’s not all,” Peter said. “So far we’ve been talking about unit yields. That’s the customary way of doing it around here, but there’s a better way.” “Unit yields often have very little to do with costs,” Peter continued. “Who knows how we got those 350 good units? Maybe they were reworked several times. There can be a lot of cost hidden in the numbers. If you want an accurate picture of process performance, use rolled throughput yields.” Peter sketched another picture on the board (see Figure 2). Figure 2: Unit Yields vs. Rolled-Throughput Yield “Let’s assume that we have two lines making the same product. If we only look at unit yields, they look much different. One process has a 50-percent yield, the other a 90-percent yield. But assume that each unit had 10 critical-to-quality characteristics. If we look at characteristics, we see that both have produced five defects in 100 defect opportunities. In terms of the ability to produce defect-free quality characteristics, they’re actually the same.” “So if it costs $100 to fix a defect, the two processes have about the same rework cost, even though the unit yields would make the first process look a lot better,” Sam replied, nodding. “This is exactly why we use rolled throughput yields in six sigma,” Peter responded. “They correlate much more closely with labor, cycle time, rework costs and other important management metrics.” Sam frowned. “That means that our efficiency reports are worse than useless–they’re misleading!” Peter smiled. “Thanks, Peter!” Sam exclaimed. “I think you’re just the man to head a project to fix them!” Yields: A Glossary Yield, First-time Yield (unit-based)–the number of units that pass a particular inspection compared to the total number of units that pass through that point in the process. Final Yield (unit-based)–the number of units that pass the last step in a series of steps in a process compared to the number of units the entire process started with. Throughput Yield (defect-based)–the probability that all defect opportunities produced at a particular step in the process will conform to their respective performance standards. Rolled Throughput Yield (defect-based)–the probability of being able to pass a unit of product or service through the entire process defect-free. Normalized Yield (defect-based)–the geometric average throughput yield one would expect at any given step in the process. Analogous to the “typical” yield. For a k -step process, the normalized yield would be the kth root of the rolled throughput yield. A note of caution: This metric can be misleading if the throughput yields of the process steps vary a great deal.

Statistical probability should be used only when we lack knowledge of the situation and cannot obtain it at a reasonable cost.

I recently attended a presentation using a control chart. The control chart showed a process in statistical control at about an 8-percent reject rate. The presenter noted that the process was stable and went on with her presentation. I barely avoided shouting that, while stability is nice, an 8-percent reject rate is not acceptable. The 8-percent level represents a certain amount of ignorance about the process; a level I find unacceptable. The problem is that the presenter didn’t think of it that way at all. To her, 8 percent represented a considerable accomplishment.

This blog is for those of you who, like me, want to scream that, as long as improvement is economically justified, “It’s never good enough!” I will present a way of measuring ignorance; a simple-to-compute statistic which highlights the fact that there is always something to learn about how to improve a given process.

First, let’s take a look at the philosophy that underlies statistics. In his book, The Art of Thinking, philosopher Leonard Peikoff wrote, “Statistics are applicable only when: 1. You are unavoidably ignorant about a given concrete; 2. Some action is necessary and cannot be deferred.”

In other words, if you’re trying to determine a course of action, your best bet is to acquire knowledge, not to blindly use statistics to guide you. While it’s true that we don’t want to tamper with a stable process, it’s also true that we don’t want to settle for anything other than the best level of quality we can provide. Control charts guide us away from tampering, but they don’t tell us how we can improve the process. Only new knowledge can do that.

Statistical probability should be used only when we lack knowledge of the situation and cannot obtain it at a reasonable cost. If we have direct knowledge about a situation, or can get it through a bit of research or by consulting someone who has it, then we should not blindly follow the statistical probabilities. In other words, if you know something about the situation, you should act on what you know.

Statistics are an expression of ignorance. They should only be used when ignorance is unavoidable, i.e., when knowledge is absent and unobtainable. Statistics are not knowledge. They are a calculation that permits action in the face of ignorance. This is the critical point missed by the presenter. She assumed that if she simply stated the level of ignorance, further improvement was not necessary.

Properly used, statistics measure ignorance or, conversely, knowledge. For example, assume that you want to buy a new piece of production machinery. Think of the important variables in the process as a list of 100 items, all of them unknown. You begin by creating a list of those items you believe to be important and prepare a plan to control as many of these items as possible. Let’s say you start with 75 items. Assuming that every item on your list is actually an important variable, these 75 items are special causes–things that affect your process and must be controlled. The remaining 25 items are common causes of variation, unknown to you but also important causes of process variability even though any one of these causes will have only a small effect.

From this starting point, you conduct a process capability study and, using statistics, quantify your knowledge as explaining all but +/-0.003″ of variation in the process. There are some out-of-control data points. After investigating these, you identify five more important variables. The process stabilizes, i.e., all of the remaining points on the control chart fall within the control limits.

Let’s assume that the control limits for the X-chart are now +/-0.002″. In philosophical terms, this means that you acquired +/-0.001″ of new knowledge, but +/-0.002″ of ignorance still remains. As time goes by and you learn more, the control limits will measure the amount of your learning. If in a year the control limits are at +/-0.001″, then you’ve learned enough to reduce the process variation by 50 percent.

As soon as you acquire this knowledge, the previous statistics become irrelevant. Gaining knowledge is the equivalent of converting special causes into common causes. This is like discovering more and more items on the list of things that cause your process to vary. You may never discover every item on the list, but with statistics to help you keep score, it’s fun to try. One way to make it even more fun is to plot a “knowledge chart.” Here’s how it works:

Record the process standard deviation from your most recent process control chart, for example, S₀ = 10.

For each subsequent complete control chart, compute the process standard deviation, for example, s₁ = 9.

Compute your relative knowledge,

k, as K=100% x (S₀-S₁)/S₀

For our example, K= 100% x (10-9)/10 = 10%

As you reduce your ignorance to zero, the knowledge measure will go to 100 percent. It’s a fun way to keep track of your quality progress!

To many quality engineers and managers, process capability is a jumbled confusion of ideas expressed in jargon that only the anointed can understand.

Imagine the following scene. The boss rushes into the quality director’s office. He’s obviously distraught.

(Boss enters, walking quickly from stage right.)

Boss: “Jane, we’ve got a serious problem. Our biggest customer just called. Their assembly line is shut down because the last batch of XYZ-50’s that we shipped won’t fit into their assembly fixtures. What happened?”

(Jane, sitting at her desk, puts down her pen and looks up at her boss. She shakes her head in dismay.)

Jane: “I knew this would happen sooner or later, boss. The problem is that our customer requires us to provide a Cpk of 1.33 or higher. But the formula they make us use assumes normality, and the XYZ-50 has a skewed distribution. If we center the process to maximize Cpk, then the tail area extends beyond the specification limit .”

(Boss exits, stage right, shaking his head and wearing a puzzled expression.)

I fear that when the quality profession talks about process capability, this is how we sound to others. To many quality engineers and managers, process capability is a jumbled confusion of ideas expressed in jargon that only the anointed can understand. Let me try to clear the air on the subject.

Process capability is about one thing, and one thing only: quality. It answers the simple question, “Can you meet my requirements?” Ideally the customer would like a simple answer, yes or no. Unfortunately, this is not possible due to one or more of the following:

Inspection is not perfect; even 100-percent inspection won’t guarantee 100-percent quality. Explaining this becomes complicated.

All processes vary, and the variation must be analyzed using statistical methods that always predict at least an occasional failure. The statistics virtually always get complicated.

Measurement isn’t perfect, so even if a process did have zero variation, our measurements would still vary. This means that we might accidentally ship a defective item even if we measure it carefully. Not only that, our measurements of a particular item might be somewhat different from our customer’s measurements. Explaining how two trained people using the same type of instruments can check the same item and get different results can get complicated.

We or our customer might not properly understand the requirements. Human communication is always complicated.

Yet it’s really not complicated at all. In fact, the customer’s question can be answered easily, and the answer is: no. For all of the reasons listed, and many more, we cannot guarantee that we will always deliver a product or service that meets the customer’s requirements as understood by the customer.

So, now what? The best approach is also the most radical: Be honest. Tell customers about how many items they are likely to receive, on average, that will not meet the requirements. This cuts right to the heart of the matter. It tells customers what they want to know. It works for variables data and attributes data. If control charts are being used, the estimate can be obtained directly from the process average (for attributes data) or the process average and standard deviation (for variables data). The count can be adjusted to include sorting operations, inspection error, measurement error and all of the other factors that influence what the customer receives.

If our process is extremely good, we can tell the customer that, while we can’t guarantee perfection, we can provide quality in the near-perfect range. One good way of quantifying this is to use parts-per-million quality statements. For example: “Our return rate on this item is three returns per million items in service per year.” Most people can easily understand this statement. A customer ordering up to several thousand items will probably, and accurately, interpret this to mean “zero defects.”

If our process is less capable, stating the expected number of defective items that the customer will receive might result in a shock to both the employees and the customer. This may provide the incentive needed to improve quality.

High-volume production is another area where stating process capability as expected defectives can provide insights. A defect rate of 1/10 percent sounds pretty good. But a can line may produce in excess of 1,000 cans per minute, so a reject rate of 1/10 percent would result in the production of 1,440 defective cans per day. If the defect is major, say a leaking can that could damage many cases of product in a warehouse or truck, even a defective rate of one in a million might not be acceptable; it would result in several serious problems each week. For such processes, parts per billion quality may be required.

If a process is not in statistical control for unknown reasons, there is no way to state the process capability with any degree of precision. The best option is to tell the customer what the expected defectives will be (based on the historical data) and hope for the best.

The key to good customer relations is clear communication. The easiest way to get the point across is to tell the customer what level of product or service quality to expect, using plain language.