Where AI Fits in Engineering Organizations

Suspend disbelief and assume there will be an “AI Engineer” in the future. Where does this role fit in? What organization does it become part of?

• R&D (Competition centers on training the best or biggest model.) Traditionally this is where AI has been in the organization. Labs experimenting with foundational model research. Examples are Google Brain/DeepMind and some of the earlier foundational model companies — OpenAI and Anthropic. Of course, the number of qualified PhDs in this position is small (but growing).

• Data Science (Your data matters the most.) Getting customer data is hard, but sometimes getting that same data internally is even more challenging. Siloed databases, uncleaned data, or misaligned incentives make it hard for the right teams to get the correct data. AI engineers embedded in the data team make the most sense. While data scientists might be a natural fit for using foundational models, we also might see a more specialized data engineer-type role (which was essentially DevOps for Data).

• Product (Everyone uses the same models, but it’s how you use them.) While companies raise large sums of venture money to train models from scratch or build out large GPU farms, others are standing on the shoulders of giants and delivering new experiences straight to the customer. And it’s not just boring vertical SaaS, but more general-purpose productivity tools like Notion. 

• DevOps (Everyone needs to run their own version of X.) Open-source models continue to get better. Running them in your own data center or AWS account ensures that your sensitive data never goes through a third party. Running these models doesn’t require PhD-level knowledge of the model, but it does require some DevOps knowledge. How to configure and set up distributed systems, how to plumb through cloud GPUs, and how to monitor inference and training endpoints.

• Analyst (Prompting is the primary way that users interact with models.) Data analysts have a relatively narrow skillset in the data stack — they might know how to write SQL and some configuration files. They usually aren’t expected to know how to use general programming languages. AI may evolve to support another one of these technical-but-not-a-programmer-type roles through prompting.