Large Language Model-Based Agents for Software Engineering: A Survey

Original Paper: https://arxiv.org/abs/2409.02977

By: Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, Yiling Lou

Abstract:

The recent advance in Large Language Models (LLMs) has shaped a new paradigm of AI agents, i.e., LLM-based agents. Compared to standalone LLMs, LLM-based agents substantially extend the versatility and expertise of LLMs by enhancing LLMs with the capabilities of perceiving and utilizing external resources and tools. To date, LLM-based agents have been applied and shown remarkable effectiveness in Software Engineering (SE). The synergy between multiple agents and human interaction brings further promise in tackling complex real-world SE problems. In this work, we present a comprehensive and systematic survey on LLM-based agents for SE. We collect 106 papers and categorize them from two perspectives, i.e., the SE and agent perspectives. In addition, we discuss open challenges and future directions in this critical domain. The repository of this survey is at this https URL

Summary Notes

Figure: Structure of This Survey

Introduction

The advent of Large Language Models (LLMs) has revolutionized various domains, including Software Engineering (SE). Traditional LLMs, although powerful, face limitations when dealing with complex, multifaceted SE tasks. Enter LLM-based agents—sophisticated systems that extend LLMs by incorporating external tools and resources, enabling them to tackle real-world SE challenges more effectively. This blog post presents a comprehensive survey of 106 research papers on LLM-based agents for SE, highlighting their methodologies, findings, and promising future directions.

Key Methodologies

Planning Strategies

LLM-based agents use diverse planning strategies to break down complex tasks into manageable sub-tasks.

These strategies include single-path planning, where tasks are planned linearly, and multi-path planning, which generates multiple plans to select the optimal path.

Some agents employ iterative planning, adjusting their plans based on feedback from previous steps, enhancing flexibility and accuracy.

Memory Mechanisms

Effective memory management is crucial for LLM-based agents, enabling them to store and utilize historical data.

Short-term memory retains task-specific information, while long-term memory stores valuable experiences for future tasks.

Agents can use various memory formats, including natural languages, programming languages, structured messages, and key-value pairs.

Perception Capabilities

LLM-based agents primarily rely on textual and visual input for perception.

Textual input includes natural language instructions and programming code, while visual input involves processing images such as UML diagrams and UI screenshots.

These capabilities enable agents to understand and interact with their environment more effectively.

Action Components

To extend their capabilities beyond textual interaction, LLM-based agents integrate various external tools. These include:

• Searching Tools: For retrieving relevant information from web searches or knowledge bases.
• File Operations: For managing code repositories and documentation.
• GUI Operations: For automating interactions with graphical user interfaces.
• Static and Dynamic Program Analysis: For collecting code features and runtime information.
• Testing Tools: For validating software behavior through test execution frameworks.

Main Findings and Results

Application in SE Tasks

LLM-based agents have been successfully applied across various SE tasks, including:

• Requirements Engineering: Multi-agent systems automate phases like elicitation, specification, and validation, reducing manual effort and improving accuracy.
• Code Generation: Agents use advanced planning and iterative refinement to enhance code generation, addressing issues like hallucination and coverage.
• Static Code Checking: Agents improve static bug detection by combining traditional techniques with multi-agent collaboration and tool integration.
• Testing: Agents generate high-quality tests with iterative refinement, leveraging feedback from execution results and coverage analysis.
• Debugging: Unified debugging approaches integrate fault localization and program repair, enhancing debugging efficiency and accuracy.
• End-to-End Software Development: Agents follow traditional software process models (e.g., waterfall, agile) to cover the entire development life cycle, from requirements to deployment.
• End-to-End Software Maintenance: Agents automate the maintenance process, including issue localization, patch generation, and verification, by leveraging tools and multi-agent collaboration.

Evaluation and Benchmarks

Current evaluations of LLM-based agents focus on task-specific success rates, but future research should develop more fine-grained metrics to assess intermediate states and overall robustness. Existing benchmarks often feature simplified tasks, highlighting the need for more realistic and complex benchmarks to better reflect real-world SE challenges.

Implications and Potential Applications

LLM-based agents hold significant promise for automating and enhancing various SE tasks.

Their ability to incorporate external tools and resources allows them to tackle complex problems more effectively than standalone LLMs. Potential applications include:

• Automated Code Review and Quality Assurance: Agents can automate code reviews, detect vulnerabilities, and ensure code quality through static analysis and dynamic testing.
• Collaborative Software Development: Multi-agent systems can simulate real-world development teams, improving efficiency and reducing manual effort.
• Efficient Debugging and Maintenance: Agents can localize faults, generate patches, and verify fixes, streamlining the maintenance process and reducing downtime.

Conclusion and Future Directions

LLM-based agents represent a significant advancement in SE, offering enhanced capabilities for complex tasks through planning, memory management, perception, and action components. Future research should focus on developing more realistic benchmarks, fine-grained evaluation metrics, and exploring diverse perception modalities. Integrating well-established SE techniques and domain knowledge into agent systems will further enhance their effectiveness and applicability.

As the field continues to evolve, LLM-based agents are poised to become indispensable tools in the software development and maintenance landscape, driving innovation and efficiency.