Software Development Bots

bot-based automation in software projects

Automation is a key enabler for effective software development. It can help developers to focus on the more creative aspects of software development, while repetitive tasks are taken care of by machines. Our key research interest in this area is how developers interact with highly automated tools (“DevBots”).

We identify six key characteristics of DevBots (Erlenhov et al., 2020). On a high level, all DevBots are automation tools with exhibit some human traits. Further, DevBots commonly have a human-like identity, such as a name.

Other characteristics depend on the type of bot. In our research, we distinguish between >Charlie bots (where the defining characteristic is a human-language interface), Alex bots (which are defined by doing things autonomously, independent of a human developer), and Sam bots (which exhibit a "smartness" that goes beyond executing a well-defined script).

DevBot Characteristics

DevBot Characteristics

Process to Decide if an Automation Tool is a DevBot

Process to Decide if an Automation Tool is a DevBot

We also propose an process to identify if a specific automation tool is indeed a DevBot, and if so, which type (Erlenhov et al., 2020).

In our ongoing work, we are also interested in bots for continuous performance assessment (Markusse et al., 2022), connecting the lab's interest in performance engineering and measurement with software development automation. We are currently working with researchers at Concordia Univeersity (Canada), Eindhoven University of Technology (The Netherlands), and Mozilla on better bots for monitoring software performance.

A DevBot for Continuous Performance Assessment

A DevBot for Continuous Performance Assessment

Obviously, the rise of Large Language Models (LLMs) has transformed this area of research. ICET-lab members are actively involved in research on how LLMs can support developers (Khojah et al., 2024), and for which use cases more traditional DevBots and automation pipelines are more suited.

Contacts:

Dr. Francisco Gomes

Dr. Philipp Leitner

An Empirical Study of Bots in Software Development: Characteristics and Challenges from a Practitioner’s Perspective

Linda Erlenhov, Francisco Gomes de Oliveira Neto, and Philipp Leitner

In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , Virtual Event, USA, Feb 2020

@inproceedings{erlenhov:20,
  author = {Erlenhov, Linda and Neto, Francisco Gomes de Oliveira and Leitner, Philipp},
  title = {An Empirical Study of Bots in Software Development: Characteristics and Challenges from a Practitioner’s Perspective},
  year = {2020},
  isbn = {9781450370431},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  doi = {10.1145/3368089.3409680},
  booktitle = {Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
  pages = {445–455},
  numpages = {11},
  keywords = {Software engineering, Software bot, Empirical study},
  location = {Virtual Event, USA},
  series = {ESEC/FSE 2020},
}

Using Benchmarking Bots for Continuous Performance Assessment

Florian Markusse, Philipp Leitner, and Alexander Serebrenik

IEEE Software, Feb 2022

@article{markusse:22,
  author = {Markusse, Florian and Leitner, Philipp and Serebrenik, Alexander},
  journal = {IEEE Software},
  title = {Using Benchmarking Bots for Continuous Performance Assessment},
  year = {2022},
  volume = {39},
  number = {5},
  pages = {50-55},
  keywords = {Benchmark testing;Bot (Internet);Software;Codes;Software development management;Manuals;Open source software},
  doi = {10.1109/MS.2022.3184430},
}

Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice

Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes Oliveira Neto

In Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024) , Feb 2024

@inproceedings{khojah:24,
  title = {{Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice}},
  booktitle = {{Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024)}},
  author = {Khojah, Ranim and Mohamad, Mazen and Leitner, Philipp and de Oliveira Neto, Francisco Gomes},
  year = {2024},
  issue_date = {July 2024},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  volume = {1},
  doi = {10.1145/3660788},
}