Foreword by NGINX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Foreword by Ihor Dvoretskyi. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
1. Revolution in the Cloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
The Creation of the Cloud 2
Buying Time 3
Infrastructure as a Service 3
The Dawn of DevOps 3
Nobody Understands DevOps 5
The Business Advantage 5
Infrastructure as Code 6
Learning Together 6
The Coming of Containers 7
The State of the Art 7
Thinking Inside the Box 8
Putting Software in Containers 8
Plug and Play Applications 9
Conducting the Container Orchestra 10
Kubernetes 11
From Borg to Kubernetes 11
What Makes Kubernetes So Valuable? 11
Will Kubernetes Disappear? 13
Kubernetes Doesn’t Do It All 13
Cloud Native 14
The Future of Operations 16
Distributed DevOps 17
Some Things Will Remain Centralized 17
Developer Productivity Engineering 17
You Are the Future 18
Summary 19
2. First Steps with Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Running Your First Container 21
Installing Docker Desktop 22
What Is Docker? 22
Running a Container Image 22
The Demo Application 23
Looking at the Source Code 23
Introducing Go 24
How the Demo App Works 24
Building a Container 25
Understanding Dockerfiles 25
Minimal Container Images 26
Running docker image build 26
Naming Your Images 27
Port Forwarding 27
Container Registries 28
Authenticating to the Registry 28
Naming and Pushing Your Image 28
Running Your Image 29
Hello, Kubernetes 29
Running the Demo App 30
If the Container Doesn’t Start 30
Minikube 31
Summary 31
3. Getting Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Cluster Architecture 33
The Control Plane 34
Node Components 35
High Availability 35
The Costs of Self-Hosting Kubernetes 37
It’s More Work Than You Think 37
It’s Not Just About the Initial Setup 38
Tools Don’t Do All the Work for You 39
Kubernetes Is Hard 39
Administration Overhead 39
Start with Managed Services 40
Managed Kubernetes Services 41
Google Kubernetes Engine (GKE) 41
Cluster Autoscaling 42
Amazon Elastic Container Service for Kubernetes (EKS) 42
Azure Kubernetes Service (AKS) 43
OpenShift 43
IBM Cloud Kubernetes Service 43
Heptio Kubernetes Subscription (HKS) 43
Turnkey Kubernetes Solutions 44
Stackpoint 44
Containership Kubernetes Engine (CKE) 44
Kubernetes Installers 44
kops 45
Kubespray 45
TK8 45
Kubernetes The Hard Way 45
kubeadm 46
Tarmak 46
Rancher Kubernetes Engine (RKE) 46
Puppet Kubernetes Module 46
Kubeformation 46
Buy or Build: Our Recommendations 47
Run Less Software 47
Use Managed Kubernetes if You Can 48
But What About Vendor Lock-in? 48
Use Standard Kubernetes Self-Hosting Tools if You Must 49
When Your Choices Are Limited 49
Bare-Metal and On-Prem 49
Clusterless Container Services 50
Amazon Fargate 50
Azure Container Instances (ACI) 51
Summary 51
4. Working with Kubernetes Objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Deployments 53
Supervising and Scheduling 54
Restarting Containers 54
Querying Deployments 55
Pods 55
ReplicaSets 56
Maintaining Desired State 57
The Kubernetes Scheduler 58
Resource Manifests in YAML Format 59
Resources Are Data 59
Deployment Manifests 59
Using kubectl apply 60
Service Resources 60
Querying the Cluster with kubectl 63
Taking Resources to the Next Level 64
Helm: A Kubernetes Package Manager 64
Installing Helm 65
Installing a Helm Chart 65
Charts, Repositories, and Releases 66
Listing Helm Releases 67
Summary 67
5. Managing Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Understanding Resources 69
Resource Units 70
Resource Requests 70
Resource Limits 71
Keep Your Containers Small 72
Managing the Container Life Cycle 72
Liveness Probes 72
Probe Delay and Frequency 73
Other Types of Probes 73
gRPC Probes 74
Readiness Probes 74
File-Based Readiness Probes 75
minReadySeconds 75
Pod Disruption Budgets 76
Using Namespaces 77
Working with Namespaces 78
What Namespaces Should I Use? 78
Service Addresses 79
Resource Quotas 79
Default Resource Requests and Limits 80
Optimizing Cluster Costs 81
Optimizing Deployments 82
Optimizing Pods 83
Vertical Pod Autoscaler 84
Optimizing Nodes 84
Optimizing Storage 85
Cleaning Up Unused Resources 86
Checking Spare Capacity 88
Using Reserved Instances 88
Using Preemptible (Spot) Instances 89
Keeping Your Workloads Balanced 91
Summary 92
6. Operating Clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Cluster Sizing and Scaling 95
Capacity Planning 96
Nodes and Instances 98
Scaling the Cluster 101
Conformance Checking 102
CNCF Certification 103
Conformance Testing with Sonobuoy 104
Validation and Auditing 105
K8Guard 106
Copper 106
kube-bench 107
Kubernetes Audit Logging 107
Chaos Testing 107
Only Production Is Production 108
chaoskube 108
kube-monkey 109
PowerfulSeal 109
Summary 110
7. Kubernetes Power Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Mastering kubectl 111
Shell Aliases 111
Using Short Flags 112
Abbreviating Resource Types 112
Auto-Completing kubectl Commands 113
Getting Help 113
Getting Help on Kubernetes Resources 114
Showing More Detailed Output 114
Working with JSON Data and jq 114
Watching Objects 115
Describing Objects 116
Working with Resources 116
Imperative kubectl Commands 116
When Not to Use Imperative Commands 117
Generating Resource Manifests 118
Exporting Resources 118
Diffing Resources 119
Working with Containers 119
Viewing a Container’s Logs 119
Attaching to a Container 121
Watching Kubernetes Resources with kubespy 121
Forwarding a Container Port 121
Executing Commands on Containers 122
Running Containers for Troubleshooting 122
Using BusyBox Commands 124
Adding BusyBox to Your Containers 124
Installing Programs on a Container 125
Live Debugging with kubesquash 125
Contexts and Namespaces 126
kubectx and kubens 128
kube-ps1 128
Kubernetes Shells and Tools 129
kube-shell 129
Click 129
kubed-sh 130
Stern 130
Building Your Own Kubernetes Tools 130
Summary 131
8. Running Containers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Containers and Pods 133
What Is a Container? 134
What Belongs in a Container? 135
What Belongs in a Pod? 136
Container Manifests 136
Image Identifiers 137
The latest Tag 138
Container Digests 138
Base Image Tags 139
Ports 139
Resource Requests and Limits 139
Image Pull Policy 140
Environment Variables 140
Container Security 141
Running Containers as a Non-Root User 142
Blocking Root Containers 142
Setting a Read-Only Filesystem 143
Disabling Privilege Escalation 143
Capabilities 144
Pod Security Contexts 145
Pod Security Policies 145
Pod Service Accounts 146
Volumes 147
emptyDir Volumes 147
Persistent Volumes 148
Restart Policies 149
Image Pull Secrets 149
Summary 150
9. Managing Pods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Labels 153
What Are Labels? 153
Selectors 154
More Advanced Selectors 155
Other Uses for Labels 156
Labels and Annotations 156
Node Affinities 157
Hard Affinities 158
Soft Affinities 158
Pod Affinities and Anti-Affinities 159
Keeping Pods Together 159
Keeping Pods Apart 160
Soft Anti-Affinities 161
When to Use Pod Affinities 161
Taints and Tolerations 162
Pod Controllers 163
DaemonSets 164
StatefulSets 165
Jobs 166
Cronjobs 167
Horizontal Pod Autoscalers 167
PodPresets 169
Operators and Custom Resource Definitions (CRDs) 170
Ingress Resources 171
Ingress Rules 172
Terminating TLS with Ingress 172
Ingress Controllers 173
Istio 174
Envoy 175
Summary 175
10. Configuration and Secrets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
ConfigMaps 179
Creating ConfigMaps 180
Setting Environment Variables from ConfigMaps 181
Setting the Whole Environment from a ConfigMap 183
Using Environment Variables in Command Arguments 184
Creating Config Files from ConfigMaps 184
Updating Pods on a Config Change 186
Kubernetes Secrets 187
Using Secrets as Environment Variables 187
Writing Secrets to Files 188
Reading Secrets 189
Access to Secrets 190
Encryption at Rest 190
Keeping Secrets 191
Secrets Management Strategies 191
Encrypt Secrets in Version Control 191
Store Secrets Remotely 192
Use a Dedicated Secrets Management Tool 193
Recommendations 193
Encrypting Secrets with Sops 194
Introducing Sops 194
Encrypting a File with Sops 195
Using a KMS Backend 196
Summary 197
11. Security and Backups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Access Control and Permissions 199
Managing Access by Cluster 199
Introducing Role-Based Access Control (RBAC) 200
Understanding Roles 201
Binding Roles to Users 201
What Roles Do I Need? 202
Guard Access to Cluster-Admin 202
Applications and Deployment 203
RBAC Troubleshooting 204
Security Scanning 204
Clair 204
Aqua 205
Anchore Engine 205
Backups 206
Do I Need to Back Up Kubernetes? 206
Backing Up etcd 206
Backing Up Resource State 207
Backing Up Cluster State 207
Large and Small Disasters 208
Velero 208
Monitoring Cluster Status 211
kubectl 211
CPU and Memory Utilization 213
Cloud Provider Console 213
Kubernetes Dashboard 214
Weave Scope 216
kube-ops-view 216
node-problem-detector 216
Further Reading 217
Summary 217
12. Deploying Kubernetes Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Building Manifests with Helm 219
What’s Inside a Helm Chart? 220
Helm Templates 221
Interpolating Variables 222
Quoting Values in Templates 223
Specifying Dependencies 223
Deploying Helm Charts 223
Setting Variables 223
Specifying Values in a Helm Release 224
Updating an App with Helm 225
Rolling Back to Previous Versions 225
Creating a Helm Chart Repo 226
Managing Helm Chart Secrets with Sops 226
Managing Multiple Charts with Helmfile 228
What’s in a Helmfile? 228
Chart Metadata 229
Applying the Helmfile 230
Advanced Manifest Management Tools 231
ksonnet 231
Kapitan 232
kustomize 233
kompose 233
Ansible 234
kubeval 234
Summary 235
13. Development Workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Development Tools 237
Skaffold 237
Draft 238
Telepresence 238
Knative 238
Deployment Strategies 239
Rolling Updates 239
Recreate 240
maxSurge and maxUnavailable 240
Blue/Green Deployments 241
Rainbow Deployments 242
Canary Deployments 242
Handling Migrations with Helm 243
Helm Hooks 243
Handling Failed Hooks 244
Other Hooks 244
Chaining Hooks 244
Summary 245
14. Continuous Deployment in Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
What Is Continuous Deployment? 247
Which CD Tool Should I Use? 248
Jenkins 248
Drone 249
Google Cloud Build 249
Concourse 249
Spinnaker 249
GitLab CI 250
Codefresh 250
Azure Pipelines 250
CD Components 250
Docker Hub 250
Gitkube 250
Flux 251
Keel 251
A CD Pipeline with Cloud Build 251
Setting Up Google Cloud and GKE 251
Forking the Demo Repository 252
Introducing Cloud Build 252
Building the Test Container 252
Running the Tests 253
Building the Application Container 253
Validating the Kubernetes Manifests 254
Publishing the Image 254
Git SHA Tags 254
Creating the First Build Trigger 255
Testing the Trigger 255
Deploying from a CD Pipeline 257
Creating a Deploy Trigger 259
Optimizing Your Build Pipeline 259
Adapting the Example Pipeline 260
Summary 260
15. Observability and Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
What Is Observability? 263
What Is Monitoring? 263
Black-Box Monitoring 263
What Does “Up” Mean? 265
Logging 266
Introducing Metrics 268
Tracing 269
Observability 270
The Observability Pipeline 271
Monitoring in Kubernetes 272
External Black-Box Checks 272
Internal Health Checks 274
Summary 276
16. Metrics in Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
What Are Metrics, Really? 277
Time Series Data 278
Counters and Gauges 278
What Can Metrics Tell Us? 279
Choosing Good Metrics 279
Services: The RED Pattern 280
Resources: The USE Pattern 281
Business Metrics 282
Kubernetes Metrics 283
Analyzing Metrics 286
What’s Wrong with a Simple Average? 287
Means, Medians, and Outliers 287
Discovering Percentiles 288
Applying Percentiles to Metrics Data 288
We Usually Want to Know the Worst 290
Beyond Percentiles 290
Graphing Metrics with Dashboards 291
Use a Standard Layout for All Services 291
Build an Information Radiator with Master Dashboards 292
Dashboard Things That Break 293
Alerting on Metrics 295
What’s Wrong with Alerts? 295
On-call Should Not Be Hell 296
Urgent, Important, and Actionable Alerts 296
Track Your Alerts, Out-of-Hours Pages, and Wake-ups 297
Metrics Tools and Services 298
Prometheus 298
Google Stackdriver 300
AWS Cloudwatch 301
Azure Monitor 301
Datadog 301
New Relic 302
Summary 303
Afterword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307