ChatGPT Plugins Don't Have PMF
In a now-taken-down blog post summarizing an event with Sam Altman, Altman revealed that he doesn’t believe that ChatGPT plugins have product-market fit (outside of the browsing plugin) and won’t be coming to the API soon. Why? A few hypotheses (not mutually exclusive).
Correct concept but not good enough yet:
GPT-4 picks the wrong plugins or fails to chain together multiple calls reliably. This is the major problem with most agent or plugin frameworks — they don’t work. They might be able to initiate a call to an external API but are so brittle that they often break or misbehave quickly. Whether or not we need bigger models or more specific ones (i.e., fine-tuned), I’m not sure.
The killer-app plugins have yet to be developed.
Larger context windows mean more plugins can be called simultaneously, unlocking more powerful workflows.
The concept is not correct:
Altman alludes to this in the post (paraphrased by the author) — a lot of people thought they wanted their apps to be inside ChatGPT, but what they really wanted was ChatGPT in their apps.
LLMs will have “horizontal” extensions, such as connecting them to a web search or a database, but they will not call generic APIs through an App Store-like interface. Each use case will need a specific interface.
Correct concept, but not the right implementation:
Chat is not the right UX for plugins. If you know what you want to do, it’s often easier to just do a few clicks on the website. If you don’t, just a chat interface makes it hard to steer the model toward your goal.
Too expensive to serve at the current price — GPT-4 has a quota of 25 messages every 3 hours. This might not be enough for users to reach the “aha moment.”
Not the right UX in some other way (e.g., having users choose plugins ahead of time, OpenAPI specification not the correct interface).
Can’t aggregate enough demand with a plugin system that only works with a single model and needs broader adoption (potentially open-source). Building a successful app store is hard — and often doesn’t lead to the monopolies observed by Apple’s iOS App Store (see necessary conditions for an app store monopoly).