The pipeline is the control
In a regulated environment a CI pipeline is not automation that happens to be convenient. It is the evidence that software reached production legitimately, and evidence has to outlive a log retention window.
Writing
Things worth writing down
Longer arguments, field notes, and the operational details that are easy to lose once the incident is over.
InfrastructureDeliveryOperationsMachine LearningSecurityField Notes#AWS#MLOps#Terraform#Architecture#Docker#Kubernetes#Security#CI/CD#Compliance#Automation
Notebooks may explore, they may not deploy
The line that made model provenance tractable was not banning notebooks. It was deciding that nothing which reaches an endpoint may have originated in one.
A model can be healthy, fast and wrong
Standard service monitoring tells you an inference endpoint is up and responsive. It cannot tell you the predictions stopped making sense three days ago.
Security groups should describe relationships, not addresses
A rule written against a CIDR block encodes an accident of your current network. A rule written against another security group encodes what you actually meant.
Least privilege is a scheduling problem
Nobody argues against least privilege. It loses anyway, because a broad policy works in thirty seconds and a narrow one takes an afternoon, and the afternoon is never available.
What teaching Git taught me about Git
Running hands-on Git workshops for students at LBEF exposed which parts of my own understanding were cargo cult. Explaining a command is a much harder test than using it.
K3s is small, not simple
Choosing K3s over managed Kubernetes cut the operational surface substantially and cut the conceptual surface not at all. That distinction is worth understanding before you make the same choice.
Multi-AZ RDS is not automatically correct
It roughly doubles your database cost to buy a synchronous standby. Sometimes that is exactly the right purchase. Deciding it by default is how architecture reviews stop being reviews.