Publication

Language Model Crossover

Variation through Few-Shot Prompting

Fig. 1. Language Model Crossover (LMX). New candidate solutions are generated by concatenating parents into a prompt, feeding the prompt through any large pre-trained large language model (LLM), and collecting offspring from the output. Such an operator can be created through very few lines of code. The enormity and breadth of the dataset on which the LLM was trained, along with its ability to perform in-context learning, enables LMX to generate high-quality offspring across a broad range of domains. Domains demonstrated in this paper include (a) binary strings, (b) mathematical expressions, (c) English sentences, (d) image generation prompts, and (e) Python code; many more are possible. When integrated into an optimization loop, LMX serves as a general and effective engine of text-representation evolution.

Abstract

Language Model Crossover: Variation through Few-Shot Prompting

Elliot Meyerson, Mark J. Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K Hoover, Joel Lehman

This paper pursues the insight that language models naturally enable an intelligent variation operator similar in spirit to evolutionary crossover. In particular, language models of sufficient scale demonstrate in-context learning, i.e., they can learn from associations between a small number of input patterns to generate outputs incorporating such associations (also called few-shot prompting). This ability can be leveraged to form a simple but powerful variation operator, i.e., to prompt a language model with a few text-based genotypes (such as code, plain-text sentences, or equations), and to parse its corresponding output as those genotypes’ offspring. The promise of such language model crossover (which is simple to implement and can leverage many different open-source language models) is that it enables a simple mechanism to evolve semantically-rich text representations (with few domain-specific tweaks), and naturally benefits from current progress in language models. Experiments in this paper highlight the versatility of language-model crossover, through evolving binary bit-strings, sentences, equations, text-to-image prompts, and Python code. The conclusion is that language model crossover is a flexible and effective method for evolving genomes representable as text.

Download publication

Associated Researchers

Elliot Meyerson

Cognizant AI Labs

Mark J. Nelson

American University

Herbie Bradley

University of Cambridge & CarperAI

Arash Moradi

New Jersey Institute of Technology

Amy K. Hoover

New Jersey Institute of Technology

Joel Lehman

Carper AI

View all researchers

Related Resources

Publication

2022

T-Domino: Exploring Multiple Criteria with Quality-Diversity and the Tournament Dominance Objective

A new ranking system for Multi-Criteria Exploration (MCX) that uses…

Publication

2023

CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

Generating shapes using natural language can enable new ways of…

Publication

2022

Reconstructing editable prismatic CAD from rounded voxel models

Reverse Engineering a CAD shape from other representations is an…

Publication

2021

Collective Transport of Unconstrained Objects via Implicit Coordination and Adaptive Compliance

We present a decentralized control algorithm for robots to aid in…

Get in touch

Something pique your interest? Get in touch if you’d like to learn more about Autodesk Research, our projects, people, and potential collaboration opportunities.

Contact us