Design of Experiments - DoE

Cookies in the Kitchen

You've put Sunday afternoon aside to bake some cookies but you discover you have run out of eggs. Your partner in marital bliss has gone out and taken the car. You call a couple of mates and they tell you to try bananas, vegetable oil or apple sauce, as egg substitutes. You decide to have some fun and mix it up on a few batches to see what's best. Woohoo, you are on the road to DoE, Design of Experiments.

Cookies in the Research Lab

Transpose the above to a business situation and you have a food technologist in a research laboratory. He has plenty of eggs but he wants to cut the cost of using eggs. He doesn't have any mates to call, and has thousands of options such as mangoes, guava and mashed potatoes instead of the banana; olive oil, peanut oil, sesame oil and many others. He needs some way of measuring things. DoE gives a structured way to do experiments with a great number of known variables, by adjusting the many known factors in groups, with the minimum number of trials. It gives some clever sums to tell him which is best, with what interactions.

DoE - a Research Tool

There's plenty of DoE examples. Everyone has heard of stainless steel. Anyone who has bought it has heard of 18/8, 304 and 316. These give exact proportions of iron, nickel and chromium and molybdenum, that a research metallurgist 100 years ago, found were best for certain properties. He probably used DoE.

There's nothing new or magical about DoE. DoE dates back at least 270 years when it was used for research on scurvy. Almost every one of the new 3,000,000 new engineers and scientists each year, learn DoE as part of their uni course. Very few ever use it, with the main exception of those with PhDs heading for research.

Quel Reality

The words 'design of experiments' give an important clue. It is a way to carry out experiments. DoE is a research tool. However in recent years, peddlers of certificates have claimed DoE as a production nirvana. Why haven't millions of engineers already been using their university education and applying DoE everywhere? How can a certificate supposedly provide what a university education cannot? What is the reality?

Like most engineers, I learned DoE at uni. I have worked in research, development and in production. In research, I worked on 2 major projects at different times, with a co-worker, in a research laboratory. Both processes were re-works of discarded old methods.

Research Labs

The first was a new process for extraction of CO2 from flue gas. It used a solid, rather than the more common liquid absorber. Now you can DoE flow rates and temperatures till you are blue in the face, but the problem is the low contact area of a solid adsorbent. A bit of thinking was required. My innovation was a new type of substrate and a new way of loading it, to dramatically increase surface area. Once the idea happened, the magic happened. Setting up testing was then trivial.

The second project was a new process for clarification of raw sugar washings. Again, you could DoE compositions, temperatures and flow rates till you were blue in the face, but the clarification flock clagged the filters. The floc worked great but you couldn't get rid of the stuff. In this case, the key was my thought to examine the floc under a microscope. Again, another bit of thinking. It revealed that some microscopic crystals were also being filtered along with the floc. I reasoned that if the crystals could be grown, they might act as a lattice, to act as a filter aid. Again, the idea allowed the magic to happen. Again, testing was then routine.

I have closely examined other research projects such as the progress of construction of an ore sorter, where individual rocks are heated, and the radiation spectrum analysed to determine whether to keep or eject. Again DoE played an insignificant role. The key was thinking and clever ideas.

However, there are situations such as creating new cookies or new forms of steel, where it is simply about the best combination of temperatures and compositions of components. In such cases, research is done on as small a scale as is practical, in a research facility and it is simply a matter of DoE.

Production

Once the research is finished, products roll into production. The rule is look but don't touch. I cannot imagine any factory manager in his right mind allowing some fellow clutching a certificate, to start experimenting on his process. The aim is to keep product rolling out the door. If it stops, money is lost and the factory manager has to answer for it. The process should already be understood from the research phase.

While experimental tools are for research, the key tool in production is the observational tool, the control chart. As Dr Wheeler points out "Observational studies are like studying lions in the wild. Experimental studies are like studying lions in a zoo."

In the research laboratory, the major variables are known and may be changed in experiments. However this is not the case in production. There are a myriad of unknown factors that may effect a product. The way to find them is by observation. This is the role of the control chart.

Production Experiments

In one production job, I worked as shift foreman in a large factory making plaster, wallboard and cornice. On one occasion, the cornice factory started making reject. I was told to stay away. For 6 weeks, 3 shifts, the plant swarmed with head office engineers, while a warehouse filled with reject. There was no chance of rework. The stuff barely even looked like cornice.

The cornice setting belt had a dozen shaped rollers down the line, that forced the wet cornice to the correct shape in the setting belt. The engineers spent weeks changing the positions of the rollers and changing roller pressures. This might have been great DoE potential! Dozens of variables and potentially hundreds of experiments! However they could have DoE'd till they were blue in the face. It would have had zero benefit.

Suddenly, the problem vanished of its own accord.

Two months later, at around 2am, I was on shift, and suddenly the cornice problem raised its ugly face again. It was just me and a bunch of Greeks and Turks, with whom I had become very close. I respected them and they all respected and looked up to me. It was essential to have their help, trust and support. My first step was to find the cause. I checked back through the records and noticed that a new batch of PLB board had a much higher porosity. I reasoned it might be stiffer. I could imagine how this might create the strange, flattened, defective shape.

Next step was to carry out simple experiments. I was on my own, so I could do what I pleased. The boys had my back. The scoring wheels didn't have adjustments, so I had the operator find a heavy bolt and we did our own. Sparks flew. My aim was to research the effect of every possible thing that we could change. No maths. No statistics. No clever experimental structure. We'd make a change and I'd yell, or run down to the end of the line to see what effect it had. One change at a time. We kept it simple. Just common sense, thinking and dedicated operators.

In 4 hours, I showed that production was an exercise in origami. By adjusting the relative depths and separation of the scoring wheels, I could control every parameter.

I also showed that the forming rollers, that the engineers spent 6 weeks adjusting, did absolutely nothing. Too much pressure on a roller and the wet plaster is forced out causing a plant shutdown. I removed every roller.

DoE was not a solution for anything. The key was again clear thinking, not tools.

The Aftermath

At 9 am when staff arrived, they were shocked to see the changes I had made. Much screaming. All the rollers were promptly put back into place, despite my protestations. I explained that the forming rollers were more superstition than science, and contributed greatly to plant stoppages. Superstition ruled.

My changes actually produced a slightly different shape on the hidden side, that made the cornice much stronger when stacked in storage. This was also ignored.

I had also developed new operating procedures. There was no reward, no praise. There was only admonishment. You don't carry out experiments on the process. You don't touch the process. If something goes wrong, you call staff. Things went back to normal. Pompous staff with noses in the air, trying to avoid eye contact with operators, continued to do their walks (long before the pretentious term 'gemba' existed). Staff never paused to ask about my hand written description of new control procedures, sticky taped to the former machine. Nor did they question the bolt hanging on the string next to my instructions. The operators understood. They had watched everything. It was passed on from one operator to the next in languages I didn't understand.

In 4 hours I had carried simple experiments that should have been carried out in a research facility, before the factory was built. It should not have been needed in production. If research had been done properly, the former station would have been designed very differently with adjustable scoring wheels.

Process Improvement

You have optimised your process, perhaps with DoE in the research labs and/or pilot plant and you roll it out to production. Your workers are well trained in quality improvement and they work with management to reduce special causes. Operating processes predictably is by far the most important step.

The next step is to reduce common cause variation, or variation inherent in the system. Again this is the role of research. This might also apply where cheaper raw materials are to be investigated. If improvements are found, they are fed into production carefully, one at a time. There must be minimum risk of disrupting production and minimum risk of production of waste.

In some circumstances even a pilot plant does not mimic the process sufficiently well. An example is a glass furnace in insulation manufacture. A full DoE trial however, is usually far too risky. It could easily disrupt production. Single variable changes may be made very carefully, to observe their effect on the process. One-factor-at-a-time may be less efficient than simultaneous multi variable changes but it is far less disruptive and far less likely to produce waste. One-factor-at-a-time does not detect potential variable interactions but these are usually of secondary importance. Experimentation is conservative if essential in production.

In some situations, where research is needed on a large scale rather than in the laboratory, experimentation in the factory may be possible if production is not continuous. In sugar mills, for example, the factory is shut down during an off season, allowing experimental trials to be carried out with minimal risk.

A major factor is whether production is batch or continuous. The loss of a single item or batch may be of far less consequence that disruption to continuous production. Regardless, DoE is not a magic button and is no substitute for insight and thinking.

In all these situations, experimentation is best carried out by experienced engineers or researchers, not certificate holders. It comes down to common sense. We would never experiment with the production version of our software, and we'd certainly never let a neophyte mess with it. We continually do research off line and feed in any improvements gradually, to production. We always aim for happy customers, without risk.

EVOP

George Box suggested not only simultaneous changes to multiple variables via DoE on a production process, but to do so continuously, in what he called EVOP, Evolutionary Operation. In situations where a process is being operated predictably, that is, in control; where cost of reject is low; where plant shutdowns are brief and painless, where start-ups are quick, easy and produce minimal waste, or where operations are batch wise; where there are large returns from small improvements; where operators are enlightened; and where management is unafraid of complexity, and is prepared to boldly go forth without fear of disruption nor split infinitives, this may be appropriate.

Note that Box's EVOP is different to EVolutionary OPtimisizer hill climbing control systems.

Summary

In summary, DoE is plain toast with the crust cut off, for PhD's in research labs. Almost every engineer and scientist has learned DoE, but most have never used it. For non engineers, or non scientists, DoE certification is a total waste of time and money. DoE is greatly over hyped and of virtually no benefit in production.

The essential production tool is the observational tool, the control chart, not DoE. The real key to process improvement is not tools, but being able to think logically, often under extreme pressure, and having the support of the operators.

   by Dr Tony Burns BE (Hon 1) PhD (Chem Eng)

Resources