Matt Rickard

Share this post

llm.ts

blog.matt-rickard.com

Discover more from Matt Rickard

Thoughts on engineering, startups, and AI.
Continue reading
Sign in

llm.ts

May 4, 2023
9
Share this post

llm.ts

blog.matt-rickard.com
4
Share

There are over 100 different LLMs, with more shipping every day. They differ slightly in their architectures, and the data they were trained on, but all of them do text completion. It’s the APIs that are fragmented — OpenAI uses a “completions” endpoint with parameters like “top_p” and “stop,” Cohere uses a “generate” endpoint with parameters “p” and “stop_sequences,” HuggingFace uses “max_new_tokens” instead of “max_tokens.” 

How do you test prompts across different models? How to call different models without managing 10 different client libraries? Why deal with slightly different but logically the same APIs?

I just published llm.ts, an open-source and MIT-licensed TypeScript library that lets you call over 30 different LLMs from a single API. Send multiple prompts to multiple LLMs and get the results back in a single response. It has zero dependencies and clocks in under 10kB minified. Bring your own API keys and call the models directly over HTTPS. Supports most LLM parameters (presence_penalty, stop_sequences, top_p, top_k, temperature, max_tokens, frequency_penalty, …).

If you’re interested in adding new models or hosting providers, don’t hesitate to send a pull request. See a list of the models supported and installation instructions on GitHub.

How does it work? Here’s an example that multiplexes 2 prompts over 3 different models. 

import { LLM, MODEL } from 'llm.ts';
(async function () {
    await new LLM({
        apiKeys: {
            openAI: process.env.OPENAI_API_KEY ?? '',
            cohere: process.env.COHERE_API_KEY ?? '',
            huggingface: process.env.HF_API_TOKEN ?? '',
        }
    }).completion({
        prompt: [
            'Repeat the following sentence: "I am a robot."',
            'Repeat the following sentence: "I am a human."',
        ],
        model: [
            // use the model name
            'text-ada-001',
            // or specify a specific provider
            'cohere/command-nightly',
            // or use enums to avoid typos
            MODEL.HF_GPT2,
        ],
    }).then(resp => {
        console.log(resp);
    })
})()

The results are returned in an OpenAI-compatible JSON response.

{
  "created": 1683079463217,
  "choices": [
    {
      "text": "\n\nI am a robot.",
      "index": 0,
      "model": "text-ada-001",
      "promptIndex": 0,
      "created": 1683079462
    },
    {
      "text": "\n\nI am a human.",
      "index": 1,
      "model": "text-ada-001",
      "promptIndex": 1,
      "created": 1683079462
    },
    {
      "text": "\nI am a robot.",
      "index": 2,
      "model": "command-nightly",
      "promptIndex": 0,
      "created": 1683079463217
    },
    {
      "text": "\nI am a human.",
      "index": 3,
      "model": "command-nightly",
      "promptIndex": 1,
      "created": 1683079463216
    },
    {
      "text": " \"Is that your question? I was expecting the answer.\" \"Then why do you think you are being asked!\" 1. \"What are you?\" \"What are you?\" \"Why are you",
      "index": 4,
      "model": "gpt2",
      "promptIndex": 0,
      "created": 1683079463088
    },
    {
      "text": " — this quote is most often cited in reference to the Qur'an. (e.g. Ibn `Allaahu `udayyyih, Al-Rai`an, Al",
      "index": 5,
      "model": "gpt2",
      "promptIndex": 1,
      "created": 1683079463091
    }
  ]
}
9
Share this post

llm.ts

blog.matt-rickard.com
4
Share
Previous
Next
4 Comments
Share this discussion

llm.ts

blog.matt-rickard.com
Bruno
May 4Liked by Matt Rickard

this is great!

Expand full comment
Reply
Share
1 reply by Matt Rickard
Ankesh Bharti
Writes The Westworld Dream
May 12

Great stuff. This also reminded me of Vercel AI Playground: https://play.vercel.ai/

Expand full comment
Reply
Share
1 reply by Matt Rickard
2 more comments...
Top
New
Community

No posts

Ready for more?

© 2023 Matt Rickard
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing