How Developers Stay in Control While Using Copilot

Table of Links

Abstract and 1. Introduction

2. Methodology and 2.1. Research Questions

2.2. Data Collection

2.3. Data Labelling

2.4. Data Extraction

2.5. Data Analysis

3. Results and Interpretation and 3.1. Type of Problems (RQ1)

3.2. Type of Causes (RQ2)

3.3. Type of Solutions (RQ3)

4. Implications

4.1. Implications for the Copilot Users

4.2. Implications for the Copilot Team

4.3. Implications for Researchers

5. Threats to Validity

6. Related Work

6.1. Evaluating the Quality of Code Generated by Copilot

6.2. Copilot’s Impact on Practical Development and 6.3. Conclusive Summary

7. Conclusions, Data availability, Acknowledgments, CRediT authorship contribution statement and References

4. Implications

4.1. Implications for the Copilot Users

Seeking inspiration from Copilot’s code suggestions rather than relying on them can lead to better control over your coding tasks. In Feature Request, we have observed that users frequently present their wish to accept code suggestions word-by-word or line-by-line when reviewing them. As a user complained in an SO post, “Copilot inputs tons of code when people just want a tiny part” (SO #74091857). Users’ desire for such requests may reflect their concerns about losing control over their coding tasks because Copilot sometimes is too “productive” for them. In the experiment conducted by Vaithilingam et al. (2022), five participants perceived the same worry when they failed to understand the code suggested by Copilot. Thus, finding a right balance between leveraging Copilot’s productivity and maintaining developers’ command over their code is important for Copilot users. As pointed out by Ziegler et al. (2024), the value of Copilot lies in whether the suggestions serve as a useful starting point for further development. Therefore, a useful position of Copilot would be a tool that provides coding insights, rather than a complete replacement for developers in coding tasks. Before using Copilot for coding, users should ideally have a clear understanding of their code structure. Then, by guiding Copilot to provide code suggestions, they can further optimize the code implementation. This way ensures that users maintain control over their code while incorporating the inspiring suggestions provided by Copilot, keeping the role of users as a real “pilot” of the coding tasks.

Review the code suggestions provided by Copilot before accepting them to prevent introducing quality issues into your project. According to the results of RQ1, the emergence of INSECURE SUGGESTION, LESS EFFICIENT SUGGESTION, and SUGGESTION WITH INVALID SYNTAX indicates potential quality issues in the code suggested by Copilot. The code quality of Copilot generated code has been a critical topic for the Copilot team and many researchers (e.g., Yetistiren et al. (2022), Siddiq et al. (2022)). Since the outputs of LLMs are unpredictable, it is unrealistic to expect Copilot to consistently produce quality-assured code under all circumstances. Additionally, we identified only a few issues related to the quality of Copilot generated code in our study. One possible reason is that developers may lack experience or may not be attentive enough to identifying quality issues such as code vulnerabilities (Fu et al., 2024) and code smells (Zhang et al., 2024) in Copilot generated code. We also noticed that users may find it challenging to fully understand the code suggestions provided by Copilot in INCOMPREHENSIBLE SUGGESTION, leading to difficulties in recognizing the quality issues introduced by Copilot. Therefore, it is essential to carefully examine whether there are any quality issues within the code suggestions from Copilot before accepting them.

It is advisable to utilize IDEs or code editors that are officially supported by Copilot. According to our study, discussions regarding Copilot usage problems are predominantly reported on mainstream coding development platforms such as VSCode, Visual Studio, and PyCharm. The Copilot team also actively reacts to the queries arising on the platforms officially supported by Copilot. Consequently, users utilizing Copilot on these IDE or code editors are more likely to find solutions for problems like Operation Issues and Compatibility Issuesthrough public Q&A platforms. We found that Unsupported Platform ranks as the fourth most frequent causes leading to Copilot usage problems. Based on the results of RQ2 (see Table 3), it is also one of the primary reasons causing various types of Operation Issue, which suggests that users may encounter unforeseen problems when attempting to use Copilot on the platforms that are not supported yet. These problems are often challenging to resolve. In our study, none of the 31 issues associated with Unsupported Platform has been resolved effectively. Therefore, users should prioritize the use of Copilot on officially supported platforms to ensure an optimal experience.