Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. Commun. ACM, 59(5), 50—57. https://doi.org/10.1145/2890784
In this article five Google engineers discuss lessons learned from over a decade of container management at Google such as
- the transformation from machine-oriented to application oriented (machine and operation system agnostic) deployment
- containers are the only entity supported by Google's infrastructure
- log collection, telemetry (cAdvisor) and configuration is done on a per application level
- bundling of application containers into pods. A pod may, for instance, contain containers for (i) the main application, (ii) log rotation, etc. This design increases modularity and allows multiple teams to develop different pieces of the service.
- Reduction of complexity by standardizing APIs and metadata. Kubernetes for instance provide the following three fields for every object:
- ObjectMetadata: information on objects (name, UID, etc.)
- Specification: the object's desired state
- Status: the object's current state
- recommendations for best practices
- don't let the container system manage port numbers
- provide labels and metadata with the containers
- don't expose raw state (e.g. force service access through a centralized API that hides implementation details and provides services for object validation, defaulting and versioning).