Usefulness in Code Completion - Results
Table 1. Controlled Experiment Results
Table 1 shows the result. There are four cells for each task in
the table because there are two subjects using GraPacc and two subjects
Meanings of columns:
- Ratio of Filled Code: ratio of tokens filled
by Grapacc (when using Grapacc) or GCE and total tokens of code.
- Programming Errors: number of programming
errors in the submitted code.
- Mean and Std Dev: the mean and the standard
deviation of the measured data - used to compare the variations of two
- The subjects used GraPacc and submitted code with 2 errors, while code from the subjects using GCE had in total
programming errors. We analyzed those for 6 programming errors and
found that, two of them are errors due to the subjects'
misunderstanding of the requirement, and two others are actually
related to the usage of the tools.
- The result suggests that GraPacc is more useful than GCE in
programming assistance. In practice, it could help improve developers'
productivity by correctly autofilling in the details of API usages,
thus, reducing human errors and saving more time for them to design
the algorithms or to test the code. It could also help improve code
quality, because the recommended usage patterns could help developers
design more efficient/reliable algorithms with better reusing of
software libraries. In addition, the usage patterns also reduce the
errors due to API mis-usages.
- Some of the comments from our subjects illustrate those
Subject 3: “GraPacc's recommendation reminds me to use
many usage patterns that I forgot for a long time”.
Subject 4: “I intended to use FileInputStream, but the
tool GraPacc recommended me Scanner, which is easier to use...”.
Subject 4: “... I (would have) missed a close call, but
the tool recommended me to do so...”.