Selection of Fixer Candidates
In this experiment, we tuned different options for the selection of fixer
candidates (i.e. developer cache). Bugzie allows to choose x% of top
fixers having most recent fixing activities. We ran it with various values of
x%, from 1-100% (at x=100%, all developers in the project’s history
were chosen). For each value of x, we measured prediction accuracy and total
processing time (for training and recommending). The same process was applied
for all datasets in Table 1. Figures 3 and 4
show the graphs for the top-1 and top-5 prediction accuracy for different values
of x for all datasets.
As seen, all graphs exhibit the same behavior. The accuracy peaks at some value
x that is quite smaller than 100%. In all 7 projects, accuracy reaches
its peak at x < 40%. This implies that selecting a suitable portion of
recent fixers as candidates actually does not lessen much the accuracy. In some
cases, it improves the prediction accuracy. For example, in FireFox, at x = 20%,
Bugzie has top-5 accuracy of 72.4%, while top-5 accuracy at x = 100% is only
70.7%, i.e. when considering all available fixers as candidates.
Definitely, selecting only a portion of available fixers as candidates also
significantly improves time efficiency. Figure 5 displays the total processing
time for all systems, which includes training and prediction time.
Since in prediction/recommendation phase, Bugzie just needs to compute
membership scores based on the stored counting values, prediction time is just a
few tens of seconds for all cases. As seen, the processing time for FireFox and
Eclipse is higher than that for other projects due to their large datasets.
This result suggests that the selection of fixer candidates (i.e. developer
cache) significantly improve time efficiency because Bugzie just needs to
process a smaller number of developers. In some cases, it even helps improve
prediction accuracy. We examined those cases and found that Bugzie fits well
with the nature of the locality in fixing activity: the appropriate cache was
able to capture the majority of actual fixers.
