Mechanical Turks

The Mechanical Turk was a chess-playing AI constructed in 1770. For 84 years, the machine toured and beat most human opponents. It could also do tricks like the knight’s tour (moving a knight to land on every chessboard square exactly once). It was originally made to impress the Empress of Austria. It supposedly defeated Napoleon and Benjamin Franklin in chess games.

But the Mechanical Turk wasn’t actually an AI — it was just a machine that cleverly concealed a human inside. The interior was intentionally misleading. It had a series of cabinets that opened and gave the illusion of moving gears and open compartments (in fact, the operator had a sliding seat so that they could move back and forth as the viewers inspected the insides). The pieces moved with strong magnets (although the inventor carefully ensured external magnets didn’t affect the board). The board was numbered inside the box.

The idea of the Mechanical Turk was revived in 2005 when Amazon launched its Amazon Mechanical Turk service. It’s a simple interface that lets requesters post “human intelligence tasks” (“HITS”) for humans to complete for a small amount of cash. These small tasks included transcribing audio, rating products, image tagging, or surveys. The requester operates with an API and doesn’t have to worry about scheduling or distributing the tasks among workers. Workers select whatever jobs they want.

Mechanical Turk has been especially useful for researchers collecting data and for companies to build labeled training sets for training machine learning models. Companies like Scale AI have evolved the idea and built specialized tagging tools for workers.

But what happens in the world of LLMs? Most “Turkers” use ChatGPT or a similar tool behind the scenes. Labeled data (still important, but not as important as in the last wave) can now be contaminated by other models. Pre-LLM labeled data might become the low-background steel of AI.

Mechanical Turk was once called “artificial artificial intelligence” by Jeff Bezos. It will be interesting to see what the Mechanical Turk of LLMs will be.