Introduction:

Often times, when you are using Kubernetes (K8s) to manage your microservices, you would want to know in realtime if an error occurred in your K8s cluster, such as a container restarting.

To achieve this, you can set up Prometheus alerts (among other solutions), however if you don’t want to manage Prometheus, a simpler solution can be to deploy a monitoring pod in your cluster that can emit alerts directly to a Microsoft Teams channel.

This functionality can be achieved with less than 25 lines of C# code. This article will show you a C-sharp (C#) implementation of this functionality using the C# K8s client library.

Quick Demo:

In the demo below, we deploy the monitoring application as a K8s pod, we then create another pod that exits in 5 seconds, and see how a Microsoft Teams channel receives a notification of that event.

Note: the full code of the solution can be found on Github here

Solution Steps:

1. Get a Teams webhook URL
2. Watch Event of K8s Pods
3. Emit container restart events to Teams

1) Getting Teams Webhook URL:

The instructions to get a webhook URL from Teams are straightforward. Simply follow the instructions found here.

2) Watch Events for K8s Pods:

The C# K8s client library, makes it extremely easy to watch for changes to the pods and return them as a stream of add, update and remove notifications. We just need to call ListNamespacedPodWithHttpMessagesAsync; with the watch argument set to true.

Once the event is received, we check the ContainerStatuses inside the pod and determine wether or not a restart happened, if so, we post a message to Teams.

var podlistResp = client.ListNamespacedPodWithHttpMessagesAsync(KubeNamespace, watch: true);
using (podlistResp.Watch<V1Pod, V1PodList>((type, item) =>
{
    Console.WriteLine("Event Received From K8s");
    // the type is: Added, Deleted, or Modified
    // the item is a V1Pod
    Monitor.Verify(type, item);
}))

3) Emit Event to Microsoft Teams:

Once we decide to post a message to Teams. The process of doing so is straightforward, it is a simple HTTP post request with the desired content and correct format.

string json = JsonConvert.SerializeObject(message);
StringContent data = new StringContent(json, Encoding.UTF8, "application/json");

using HttpClient httpClient = new HttpClient();

HttpResponseMessage response = await httpClient.PostAsync(webhookUrl, data);

Deployment:

To install this feature in your K8s cluster. You can either run it directly as a C# application on your machine (provided that you have a correctly placed kube config). Or, better yet, you can deploy it as a K8s pod within the cluster.

For the latter solution, you would need to give the pod the correct RBAC permissions (in this example we used the default service account). We pass the Teams webhook as an environment variable to the pod. All the yaml manifests and the command to deploy, can be found here.

Conclusion:

In this article we showed an example of posting Team messages based on K8s events. The code-base can be easily modified to accommodate other events such as pod removal or security sensitive events like modifying a network policy or RBAC roles.

For more great content, click here

About the Author:

Ali is a Senior Software Engineer at Microsoft, He previously worked for IBM Research and Ericsson Research. He holds a Ph.D. in computer engineering.

Reference: