Blog

Hot-Reload Certificates and Safely Rollout Envoy with Contour 1.2

Steve Sloka

Mar 3, 2020

Contour continues to add new features to help you better manage Contour operations in a cluster. Our latest feature release, Contour 1.2.0, now includes support for certificate rotation for xDS gRPC interface between Contour and Envoy as well as a new subcommand which assists in Envoy rollouts to minimize the number of connection errors. Additionally, Contour v1.2.1 is a security release which upgrades the version of Envoy to v1.13.1 which includes many CVE fixes.

Hot-Reload Certificates

A few releases ago, Contour enabled secure communication between Contour and Envoy. This new feature ensured that any communication between Contour and Envoy over its gRPC connection would be secure, mainly securing the communication using TLS certificate keys.

This was just the first step, however, and we understood that it wouldn’t solve all of our users’ problems. Thanks to @tsaarni, we now have support for Contour to rotate its certificates without the need to restart the Contour process.

Future work includes enabling this same functionality for Envoy. This currently has some open issues that need to be solved in Envoy first.

Big thanks to Tero on all your effort to send these PRs as well as driving the issues upstream!

Envoy Shutdown Manager

The Envoy process, the data path component of Contour, at times needs to be re-deployed. This could be due to an upgrade, a change in configuration, or a node-failure forcing a redeployment.

As with any application rollout strategy, we want a way to implement the rollout while minimizing the effect on users. If the Envoy pods are terminated while there are still open connections, then users will receive errors.

Contour implements a new envoy sub-command which has a shutdown-manager whose job is to manage a single Envoy instance’s lifecycle for Kubernetes. The shutdown-manager runs as a new container alongside the Envoy container in the same pod. It exposes two HTTP endpoints that are used for livenessProbe as well as to handle the Kubernetes preStop event hook.

• livenessProbe: This is used to validate the shutdown manager is still running properly. If requests to /healthz fail, the container will be restarted
• preStop: This is used to keep the container running while waiting for Envoy to drain connections. The /shutdown endpoint blocks until the connections are drained

The Envoy container also has some configuration to implement the shutdown manager. First the preStop hook is configured to use the /shutdown endpoint which blocks the container from exiting. Finally, the pod’s terminationGracePeriodSeconds is customized to extend the time in which Kubernetes will allow the pod to be in the Terminating state. The termination grace period defines an upper bound for long-lived sessions. If during shutdown, the connections aren’t drained to the configured amount, the terminationGracePeriodSeconds will send a SIGTERM to the pod killing it.

{: .center-image }

{% youtube oO52CV-EAkw %}{: .center-image }

For more information on this feature, check out the docs

Thank you!

We’re immensely grateful for all the community contributions that help make Contour even better! For version 1.2, special thanks go out to the following people:

@awprice
@alex1989hu
@bgagnon
@danehans
@dhxgit
@SDBrett
@uablrek
@rohandvora
@tsaarni @shyaamsn
@idealhack
@dbason

Future Plans

The Contour team would love to hear your feedback! Many of the features in this release were driven by users who needed a better way to solve their problems. We’re working hard to add features to Contour, especially in expanding how we approach routing.

We recommend reading the full release notes for Contour 1.2 as well as digging into the upgrade guide, which outlines the changes to be aware of when moving to version 1.2.

If you are interested in contributing, a great place to start is to comment on one of the issues labeled with Help Wanted and work with the team on how to resolve them.