Should Your Developers Work in the Cloud?
The more your development environment looks like what you're ultimately deploying, the fewer headaches your team is going to have.
When using Kubernetes, you have a few different options of how you could have your developers work. I've built developer tools across the whole spectrum and here are some benefits and drawbacks I've seen to each.
Build Local, Run Local [without Docker or Kubernetes]
Benefits:
No upfront migration: Continue to develop your applications exactly how you did before
No learning curve: Developers don't have to learn new tooling
Usually quick: Once development environment is set up and dependencies are downloaded, builds are usually quick. Can leverage native tools: compiler level caching, running on local ports, and instant changes (no networked filesystem, ssh tunnel, etc.)
Downfalls
Parity between environments: Significant departure from how services are actually ran. Many places that things can go wrong.
"Works on my machine": Setting up a developer environment needs to be done on a per-user basis.
Single platform development: Development OS cannot be different from runtime environment (e.g. can't develop on a pixelbook or MacBook and deploy to a linux environment.)
Build Local, Run Local [with Docker and Kubernetes]
Benefits
Closer to Production: Fewer differences with higher environments. Developers can catch issues in development rather than waiting for CI or QA or catch them.
Portable: You can run Docker on Kubernetes on every major OS.
Declarative environment: Setup and teardown development environments easily. No need for long developer environment setup documents. Applying the configuration for a cluster can be as easy as
kubectl apply -f folder/.Reproducible: Alongside declarative environments, bugs and other issues are easier to reproduce because Docker and Kubernetes manage the immediately dependencies for an application.
Full Control: Developers manage the entire stack and therefore have few limitations when developing.
Drawbacks
Limited: Environment may be too large to run on your workstation. Istio suggests 8GB and 4 vCPUs on minikube. Won't work for users with high data or compute requirements (e.g. ML workloads)
Ops work for the Developer: Developers have to manage a local cluster. Minikube and Docker for Desktop provide one-click cluster setup, but what happens when your cluster goes down? Networking issues, OOM errors, and more can require developer intervention.
Build Local, Deploy Remote [with Kubernetes]
Benefits
Closest to Production: While it doesn't really matter what guest OS Docker uses, Kubernetes still has many host dependencies with the kubelet, which doesn't run containerized. A Kubernetes feature might work on Docker for Desktop or minikube's custom VM image but not the one your production cluster.
More Portable: You can run Docker on every major OS.
Managed Declarative environment: Have your ops team manage the cluster, instead of the developers. Manage O(orgs) clusters, not O(developers).
Can support arbitrarily large environments
Can be shared by multiple users
Can utilize ops-managed resources (dashboard, logging, monitoring, specialized hardware like TPUs)
Drawbacks
Cost: You have to buy hardware for your developers anyways
Speed: Build artifacts can be large, and it takes time to move large objects across a network.
New Development Tools: Apps aren't deploy to localhost by default like they might be locally.