Borg, Omega and Kubernetes

Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. Commun. ACM, 59(5), 50—57. https://doi.org/10.1145/2890784

Summary

In this article five Google engineers discuss lessons learned from over a decade of container management at Google such as

1. the transformation from machine-oriented to application oriented (machine and operation system agnostic) deployment
   • containers are the only entity supported by Google's infrastructure
   • log collection, telemetry (cAdvisor) and configuration is done on a per application level
2. bundling of application containers into pods. A pod may, for instance, contain containers for (i) the main application, (ii) log rotation, etc. This design increases modularity and allows multiple teams to develop different pieces of the service.
3. Reduction of complexity by standardizing APIs and metadata. Kubernetes for instance provide the following three fields for every object:
   • ObjectMetadata: information on objects (name, UID, etc.)
   • Specification: the object's desired state
   • Status: the object's current state
4. recommendations for best practices
   • don't let the container system manage port numbers
   • provide labels and metadata with the containers
   • don't expose raw state (e.g. force service access through a centralized API that hides implementation details and provides services for object validation, defaulting and versioning).