5.
Process Improvement
5.4.
Analysis of DOE data
5.4.7.
Examples of DOE's
5.4.7.2.
|
Fractional factorial example
|
|
|
|
A "Catapult" Fractional Factorial Experiment
|
|
A step-by-step analysis of a fractional factorial "catapult"
experiment
|
This experiment was conducted by a team of students on a catapult
– a table-top wooden device used to teach design of experiments and
statistical process control. The catapult has several controllable
factors and a response easily measured in a classroom setting. It has
been used for over 10 years in hundreds of classes. Below is a small
picture of a catapult that can be opened to view a larger version.
Catapult
|
|
|
Description of Experiment: Response and Factors
|
|
The experiment has five factors that might affect the distance the
golf ball travels
|
Purpose: To determine the significant factors that affect the
distance the ball is thrown by the catapult, and to determine the
settings required to reach 3 different distances (30, 60 and 90 inches).
Response Variable: The distance in inches from the front of the
catapult to the spot where the ball lands. The ball is a plastic golf
ball.
Number of observations: 20 (a 25-1 resolution
V design with 4 center points).
Variables:
- Response Variable Y = distance
- Factor 1 = band height (height of the pivot point for the rubber
bands – levels were 2.25 and 4.75 inches with a centerpoint
level of 3.5)
- Factor 2 = start angle (location of the arm when the operator
releases– starts the forward motion of the arm – levels
were 0 and 20 degrees with a centerpoint level of 10 degrees)
- Factor 3 = rubber bands (number of rubber bands used on the
catapult– levels were 1 and 2 bands)
- Factor 4 = arm length (distance the arm is extended – levels
were 0 and 4 inches with a centerpoint level of 2 inches)
- Factor 5 = stop angle (location of the arm where the forward
motion of the arm is stopped and the ball starts flying –
levels were 45 and 80 degrees with a centerpoint level of 62
degrees)
|
|
Design matrix and responses (in run order)
|
The design matrix appears below in (randomized) run order.
|
|
You can download the data in a spreadsheet
|
Readers who want to analyze this experiment may download an Excel
spreadsheet catapult.xls or a
JMP spreadsheet capapult.jmp.
|
|
One discrete factor
|
Note that 4 of the factors are continuous, and one – number of
rubber bands – is discrete. Due to the presence of this
discrete factor, we actually have two different centerpoints, each with
two runs. Runs 7 and 19 are with one rubber band, and the center of
the other factors, while runs 2 and 13 are with two rubber bands and
the center of the other factors.
|
|
5 confirmatory runs
|
After analyzing the 20 runs and determining factor settings needed to
achieve predicted distances of 30, 60 and 90 inches, the team was asked
to conduct 5 confirmatory runs at each of the derived settings.
|
|
|
Analysis of the Experiment
|
|
Analyze with JMP software
|
The experimental data will be analyzed using SAS JMP 3.2.6 software.
|
|
|
Step 1: Look at the data
|
|
Histogram,
box plot, and
normal probability plot
of the response
|
We start by plotting the data several ways to see if any trends or
anomalies appear that would not be accounted for by the models.
The distribution of the response is given below:
We can see the large spread of the data and a pattern to the data that
should be explained by the analysis.
|
|
Plot of response versus run order
|
Next we look at the responses versus the run order to see if there might
be a time sequence component. The four highlighted points are the center
points in the design. Recall that runs 2 and 13 had 2 rubber bands and
runs 7 and 19 had 1 rubber band. There may be a slight aging of the
rubber bands in that the second center point resulted in a distance
that was a little shorter than the first for each pair.
|
|
Plots of responses versus factor columns
|
Next look at the plots of responses sorted by factor columns.
Several factors appear to change the average response level and most
have a large spread at each of the levels.
|
|
|
Step 2: Create the theoretical model
|
|
The resolution V design can estimate main effects and all 2-factor
interactions
|
With a resolution V design we are able to estimate all the main effects
and all two-factor interactions cleanly – without worrying about
confounding. Therefore, the initial model will have 16 terms – the
intercept term, the 5 main effects, and the 10 two-factor interactions.
|
|
|
Step 3: Create the actual model from the data
|
|
Variable coding
|
Note we have used the orthogonally coded columns for the analysis, and
have abbreviated the factor names as follows:
Bheight = band height
Start = start angle
Bands = number of rubber bands
Stop = stop angle
Arm = arm length.
|
|
JMP output after fitting the trial model (all main factors and
2-factor interactions)
|
The following is the JMP output after fitting the trial model (all
main factors and 2-factor interactions).
|
|
Use p-values to help select significant effects, and also use
a normal plot
|
The model has a good R2 value, but the fact that R2
adjusted is considerably smaller indicates that we undoubtedly have some
terms in our model that are not significant. Scanning the column of
p-values (labeled Prob>|t| in the JMP output) for small values
shows 5 significant effects at the 0.05 level and another one at the
0.10 level.
The normal plot of effects is a useful graphical tool to determine
significant effects. The graph below shows that there are 9 terms
in the model that can be assumed to be noise. That would leave 6 terms
to be included in the model. Whereas the output above shows a
p-value of 0.0836 for the interaction of bands and arm, the
normal plot suggests we treat this interaction as significant.
|
|
A refit using just the effects that appear to matter
|
Remove the non-significant terms from the model and refit to produce the
following output:
|
|
R2 is OK and there is no significant model
"lack of fit"
|
The R2 and R2 adjusted values are acceptable. The
ANOVA table shows us that the model is significant, and the Lack of Fit
table shows that there is no significant lack of fit.
The Parameter estimates table is below.
|
|
|
Step 4: Test the model assumptions using residual graphs
(adjust and simplify as needed)
|
|
Histogram of the residuals to test the model assumptions
|
We should test that the residuals are approximately normally distributed,
are independent, and have equal variances. First we create a
histogram
of the residual values.
The residuals do appear to have, at least approximately, a normal
distributed.
|
|
Plot of residuals versus predicted values
|
Next we plot the residuals versus the predicted values.
There does not appear to be a pattern to the residuals. One observation
about the graph, from a single point, is that the model performs poorly
in predicting a short distance. In fact, run number 10 had a measured
distance of 8 inches, but the model predicts -11 inches, giving a
residual of 19. The fact that the model predicts an impossible
negative distance is an obvious shortcoming of the model. We may not be
successful at predicting the catapult settings required to hit a distance
less than 25 inches. This is not surprising since there is only
one data value less than 28 inches. Recall that the objective is for
distances of 30, 60, and 90 inches.
|
|
Plot of residuals versus run order
|
Next we plot the residual values versus the run order of the design.
The highlighted points are the centerpoint values. Recall that run
numbers 2 and 13 had two rubber bands while run numbers 7 and 19
had only one rubber band.
|
|
Plots of residuals versus the factor variables
|
Next we look at the residual values versus each of the factors.
|
|
The residual graphs are not ideal, although the model passes "lack
of fit" quantitative tests
|
Most of the residual graphs versus the factors appear to have a slight
"frown" on the graph (higher residuals in the center). This may indicate
a lack of fit, or sign of curvature at the centerpoint values. The Lack
of Fit table, however, indicates that the lack of fit is not significant.
|
Consider a transformation of the response variable to see if we can
obtain a better model
|
At this point, since there are several unsatisfactory features of the
model we have fit and the resultant residuals, we should consider whether
a simple transformation of the response variable (Y = "Distance")
might improve the situation.
There are at least two good reasons to suspect that using the logarithm
of distance as the response might lead to a better model.
- A linear model fit to LN Y will always predict a positive
distance when converted back to the original scale for
any possible combination of X factor values.
- Physical considerations suggest that a realistic model for
distance might require quadratic terms since gravity plays a
key role - taking logarithms often reduces the impact of
non-linear terms.
To see whether using LN Y as the response leads to a more
satisfactory model, we return to step 3.
|
|
|
Step 3a: Fit the full model using LN Y as the response
|
|
First a main effects and 2-factor interaction model is fit
to the log distance responses
|
Proceeding as before, using the coded columns of the matrix
for the factor levels and Y = the natural logarithm of
distance as the response, we initially obtain:
|
|
A simpler model with just main effects has a satisfactory fit
|
Examining the p-values of the 16 model coefficients, only the
intercept and the 5 main effect terms appear significant. Refitting the
model with just these terms yields the following results.
This is a simpler model than previously obtained in Step 3 (no
interaction term). All the terms are highly significant and there
is no quantitative indication of "lack of fit".
We next look at the residuals for this new model fit.
|
|
|
Step 4a: Test the (new) model assumptions using residual
graphs (adjust and simplify as needed)
|
|
Normal probability plot, box plot, and histogram of the residuals
|
The following normal plot,
box plot, and
histogram of the
residuals shows no problems.
|
|
Plot of residuals versus predicted LN Y values
|
A plot of the residuals versus the predicted LN Y values looks
reasonable, although there might be a tendency for the model to
overestimate slightly for high predicted values.
|
|
Plot of residuals versus run order
|
Residuals plotted versus run order again show a possible slight
decreasing trend (rubber band fatigue?).
|
|
Plot of residuals versus the factor variables
|
Next we look at the residual values versus each of the factors.
|
|
The residuals for the main effects model (fit to natural log
distance) are reasonably well behaved
|
These plots still appear to have a slight "frown" on the graph (higher
residuals in the center). However, the model is generally an improvement
over the previous model and will be accepted as possibly the best that
can be done without conducting a new experiment designed to fit a
quadratic model.
|
|
|
Step 5: Use the results to answer the questions in your
experimental objectives
|
|
Final step: quantify the influence of all the significant effects
and predict what settings should be used to obtain desired distances
|
The software used for this analysis (JMP 3.2.6) has an option called
the "Prediction Profiler" that can be used to
derive settings that will yield a desired predicted natural log distance
value. The top graph in the figure below shows the direction and
strength of each of the main effects in the model. Using
natural log 30 = 3.401 as the target value, the Profiler allows us to
set up a "Desirability" function that gives 3.401 a maximum desirability
value of 1 and values above or below 3.401 have desirabilities that
rapidly decrease to 0. This is shown by the desirability graph on the
right (see the figure below).
The next step is to set "bands" to either -1 or +1 (this is a discrete
factor) and move the values of the other factors interactively until a
desirability as close as possible to 1 is obtained. In the figure below,
a desirability of .989218 was obtained, yielding a predicted natural
log Y of 3.399351 (or a distance of 29.94). The corresponding
(coded) factor settings are: bheight = 0.17, start = -1, bands = -1,
arm = -1 and stop = 0.
|
|
Prediction profile plots for Y = 30
|
|
|
Prediction profile plots for Y = 60
|
Repeating the profiler search for a Y value of 60 (or
LN Y = 4.094) yielded the figure below for which a natural log
distance value of 4.094121 is predicted (a distance of 59.99) for coded
factor settings of bheight = 1, start = 0, bands = -1, arm = .5 and
stop = .5.
|
|
Prediction profile plots for Y = 90
|
Finally, we set LN Y = LN 90 = 4.4998 and obtain (see the
figure below) a predicted log distance of 90.20 when bheight = -0.87,
start = -0.52, bands = 1, arm = 1, and stop = 0.
|
|
"Confirmation" runs were successful
|
In the confirmatory runs that followed the experiment, the team was
successful at hitting all 3 targets, but did not hit them all 5 times.
NOTE: The model discovery and fitting process, as illustrated
in this analysis, is often an iterative process.
|
