Empirical Study - Sensitivity analysis
The parameters of the model,α,β,γ,
μ, ν, can be learned
from a training set using a hill-climbing search algorithm.
To study how different values of those parameters affect
the effectiveness score E of the technique, we conducted a
sensitivity analysis experiment. We varied α and γ
within the range from 0 to 1 with increments of 0.1 such that
α + γ <= 1. The other parameters are computed as follows:
- β = 1 - α - γ
- μ = α/(α+γ)
- ν = γ/(α+γ)
For each localized bug, we took the average of E
high and E
low to obtain
E
avg and measured the average of E
avg on all the subject
systems that were previously used. The results are shown
in the chart below. E
avg slightly rises as α increases from 0.1 to
1 and peaks at 89% with α = 1;
γ= 0. In contrast, when
α = 0, γ = 0, Eavg drops significantly to only 55%. The
similarity of the lines suggests that E
avg tends to follow a
pattern, and the parameters can be learned from a large
data set to obtain optimum results. Thus, in general use of
MkFault, one can train the model's parameters based on the
history of previous bugs, and run it for the current fault.
