ReLLM: Exact Structure for Large Language Model Completions

What if you could ensure that an LLM only returned specific syntactic structure (e.g. JSON or XML), specific semantic structure (e.g. a date or a number), or even complete templates (e.g. a sentence with a blank to fill in)? What if the model never deviated from that structure?

ReLLM is a small Python library I open-sourced (MIT) to get deterministic structure out of any large language model completion. This makes improves the quality (and parsability) of even the smallest models. How does it work? 

First, you input both a prompt and a regular expression (regex) that determines the pattern for the generated tokens. 

Prompt: ReLLM, the best way to get structured data out of LLMs, is an acronym for

Pattern: Re[a-z]+ L[a-z]+ L[a-z]+ M[a-z]+

A quick refresher — [a-z]+ is a regex that matches any character between ‘a’ and ‘z’ one or more times (the ‘+’). The rest of the letters (and whitespace) is matched directly.

ReLLM then iterates over the entire token vocab and tests each potential new token against the regex. If there’s not a partial match, it masks that token’s logit so that it will not get generated (a partial match is when a pattern did not match due to the end of input, but could have matched if more input had been available).

Logit bias lets us modify the likelihood of tokens appearing in the completion. For anything that potentially doesn’t match the regex, we set the bias to negative infinity so it will not be generated.

Here’s the output in a very small model (GPT2, 124 million parameters):

ReLLM: Realized Logistic Logistics Model

Without ReLLM: Largest Largest Address Space (MELSP), which has its roots in the Internet network, at least when compared

Some more interesting examples:

—

Prompt: Return the first three letters of the alphabet in a json array:

Pattern ["[a-z]", "[a-z]", "[a-z]"]

ReLLM: ["a", "b", "c"]

Without ReLLM: { "index": 0, "id":"1", "description":"", "text": "[{ "id": 0, "name":

—

Prompt: Fill in the sentence with an interesting story about the dentist:

Pattern: Today I'm going to the [a-z]+ to [a-z]+ because ([a-z]+ )*.

ReLLM: Today I'm going to the dentist to see because it is a very important day for me

Without ReLLM: 'My family bought me an appointment with a dentist when I was 15. The dentist gave me one a year and then I was told on

—

Prompt: Is this a good demo?

Pattern: (Yes|No)

ReLLM: No.

Without ReLLM: I don't know, but this is amazing! Even more amazing is how the design can take place on a small stage that uses LEDs. As

—

Prompt: Convert the date May 4, 2023 to the format mm/dd/yyyy:

Pattern: [0-9]{2}/[0-9]{2}/[0-9]{4}

ReLLM: 00/00/0045

Without ReLLM: mm:ss

A-Z, Z-A, W-H (0-9:9:19)

Z-R

—

Prompt: Jeff Dean is a

Pattern (Programmer|Computer Scientist|AGI)

ReLLM: Computer Scientist

Without ReLLM: former national basketball champion and a former professional basketball player. He currently serves as general counsel for the NCAA Office of the Vice President for Academic Affairs.

—

Prompt: I can eat

Pattern: [0-9]{1,10} [a-z]* of [a-z]*

ReLLM: 800 calories of coffee

Without ReLLM: iced coffee here on the west side and do this, so can you?"

"Why, I don't understand. What did you mean by