Matt Rickard

Share this post

Local AI: Part 1

blog.matt-rickard.com

Discover more from Matt Rickard

Thoughts on engineering, startups, and AI.
Continue reading
Sign in

Local AI: Part 1

Dec 29, 2022
Share this post

Local AI: Part 1

blog.matt-rickard.com
Share

What happens when parts of the AI stack go on-device? It’s already fairly easy to run Stable Diffusion locally, although these solutions won’t be the ones that reach the majority of consumers. You can use DiffusionBee, Automatic1111’s web UI, or just run it yourself via Python or a Jupyter Notebook.

So what could be run locally, and where? Local AI has interesting properties: low latency, data sovereignty, works offline, cheaper.

Training from scratch is probably out of the question. Anything other than fine-tuning needs distributed computing.

But fine-tuning is a different question. You can fine-tune Stable Diffusion (via Dreambooth) on an NVIDIA T4 with 15GB of RAM (from my own experience).

Inference is relatively cheap but still unfeasible on most end-user devices. However, the open-sourcing of models has outsourced a lot of the optimization — already, some folks have found ways to reduce the memory footprint needed. You can even run Stable Diffusion with only 3.2 GB of VRAM, at the cost of time (huggingface).

Something like GPT-J  needs closer to 64GB of GPU RAM. GPT-3 (175 billion parameters) probably needs somewhere in the order of 300 GB (an uneducated guess).

For reference, iPhones have about 6GB of RAM. Some Samsung phones have up to 16GB.

But raw memory size is the wrong way to look at the problem. Things could change quickly:

  • Optimizations to the model (code, architecture, framework, etc.)

  • Specialized hardware on-device (e.g., tensor processing units)

  • Specialized models (fine-tuned, smaller context models, higher compression, tuned hyperparameters)

  • Optimizations to the hardware (e.g., Moore’s Law, etc.)

Share this post

Local AI: Part 1

blog.matt-rickard.com
Share
Previous
Next
Comments
Top
New
Community

No posts

Ready for more?

© 2023 Matt Rickard
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing