Original Paper: https://arxiv.org/abs/2403.11446

By: Clint Morris, Michael Jurado, Jason Zutty

Abstract:

In the realm of machine learning, traditional model development and automated approaches like AutoML typically rely on layers of abstraction, such as tree-based or Cartesian genetic programming. Our study introduces "Guided Evolution" (GE), a novel framework that diverges from these methods by utilizing Large Language Models (LLMs) to directly modify code. GE leverages LLMs for a more intelligent, supervised evolutionary process, guiding mutations and crossovers. Our unique "Evolution of Thought" (EoT) technique further enhances GE by enabling LLMs to reflect on and learn from the outcomes of previous mutations. This results in a self-sustaining feedback loop that augments decision-making in model evolution. GE maintains genetic diversity, crucial for evolutionary algorithms, by leveraging LLMs' capability to generate diverse responses from expertly crafted prompts and modulate model temperature. This not only accelerates the evolution process but also injects expert like creativity and insight into the process. Our application of GE in evolving the ExquisiteNetV2 model demonstrates its efficacy: the LLM-driven GE autonomously produced variants with improved accuracy, increasing from 92.52% to 93.34%, without compromising model compactness. This underscores the potential of LLMs to accelerate the traditional model design pipeline, enabling models to autonomously evolve and enhance their own designs.

Summary Notes

Simplifying Model Development with LLM Guided Evolution

In the fast-evolving world of artificial intelligence (AI), combining human-like thinking with automated processes is key to advancing model development.

This blog post introduces Guided Evolution (GE), a new approach that brings together Large Language Models (LLMs) and genetic algorithms to innovate Neural Architecture Search (NAS).

We aim to explain the basics of Guided Evolution and its role in advancing AI design, offering insights for AI Engineers in large companies.

Exploring Neural Architecture Search (NAS)

Neural Architecture Search (NAS) is all about automating the design of high-performing neural networks. Traditional NAS methods include:

• Reinforcement Learning: Using agents to find and refine architecture setups.
• Evolutionary Algorithms: Evolving network designs through a process similar to natural selection.
• Surrogate Model-Based Optimization: Using predictions to steer the search toward better architectures.

Despite advancements, integrating LLMs into NAS through Guided Evolution introduces a new level of adaptability and insight.

Unpacking Guided Evolution

Guided Evolution focuses on enhancing a starting model, dubbed ExquisiteNetV2, with the help of LLMs. The process includes:

• Decomposing the Seed Model: Breaking down ExquisiteNetV2 into smaller, evolvable code segments.
• Guided Mutations and Crossovers: Using LLMs to direct the mutation and merging of these segments, adding creativity and improving the process.
• Key Techniques:
• Character Role Play (CRP): Boosts mutation diversity for more innovative model variants.
• Evolution of Thought (EoT): LLMs learn from previous mutations to enhance their guidance over time.

The Benefits: Better Performance and Efficiency

Applying GE to ExquisiteNetV2 has led to models with higher accuracy and more compact designs. These evolved models excel in balancing performance with computational efficiency.

The Future of Model Development

The fusion of Guided Evolution with NAS represents a significant advancement in automating machine learning. This approach opens up possibilities for creating models that are efficient, creative, and insightful, qualities once thought unique to humans.

The success seen with ExquisiteNetV2 highlights the potential of LLM-guided evolution in model development.

In summary, Guided Evolution offers an exciting route to improving the model development process, combining LLMs with genetic algorithms.

As this methodology evolves, it promises to unlock new potentials for building smarter, more efficient models. AI Engineers at large companies should consider exploring this innovative approach to push the boundaries of neural architecture design further.