How to add the Shell into Chiseled Image for Troubleshooting

[AbdulShaikh]

Describe the Problem

Currently, we are utilizing standard Docker images for .NET Core and are considering transitioning to chiseled images. In our production environment, there are instances where we may need to get inside the container for troubleshooting purposes. How can we achieve similar capabilities with the distroless image? We are seeking the best approach to address this. Alternatively, is it feasible to incorporate a shell into the distroless image as needed?

Describe the Solution

We could employ separate Dockerfiles, one utilizing the chiseled image and the other the standard image, and adjust the Dockerfile based on the environment variable when troubleshooting is required. However, a drawback of this approach is the lack of consistency in using different images. Our preference is to maintain uniformity by continuing to use the same images rather than switching between different ones.

Additional Context

Current Image: mcr.microsoft.com/dotnet/aspnet:8.0
Move to Image: mcr.microsoft.com/dotnet/aspnet:8.0-jammy-chiseled-extra

[MichaelSimons]

What are the troubleshooting scenarios? Would the dotnet/monitor address any of your scenarios?

It is possible to add Bash on top of the .NET chiseled images. The UX is not great nor is the documentation. This issue tracks improving the documentation. This comment illustrates the pattern that can be used to add additional package.

[AbdulShaikh]

The scenarios could involve verifying/checking the DB connectivity inside the container, ensuring that the domain is correctly resolved related to Docker networking, or checking application logs, among other tasks. Up to this point, we have been comfortably handling these tasks or similar ones.

[EugeneKrapivin]

What are the troubleshooting scenarios? Would the dotnet/monitor address any of your scenarios?

It is possible to add Bash on top of the .NET chiseled images. The UX is not great nor is the documentation. This issue tracks improving the documentation. This comment illustrates the pattern that can be used to add additional package.

I'm actually battling with this matter at the moment, we have a 150+ net6 (soon migrating) services in production, however we find it impossible to move a dump from a pod running the service to some machine to analyze it. Is there any specific documentation on how to use the monitor image? I've expected to use it as a debug image, but after looking into the dockerfile (and actually attempting to use it with kubectl debug) I understood that that's probably not the intended use-case. on the topic of kubectl debug and ephemeral images, is there a microsoft suggested base image for the troubleshooting use case?

[lbussell]

@EugeneKrapivin you will want to reference the dotnet-monitor documentation. dotnet monitor provides an HTTP API for collecting diagnostic info. The /dump endpoint should let you capture and download a dump.

[EugeneKrapivin]

@lbussell Yeah I looking at it the whole day... :D

trying to understand is there a way to deploy it using kubectl debug or something. the documentation seems to expect me to setup the monitor container beforehand as a sidecar.

when running kubectl debug it seems that I can't port-forward from an ephemeral container (or maybe I don't know how). I really feel that a "real world" production troubleshooting is missing, especially when dealing with distroless containers from which I can't use kubectl cp to copy a dump, or even take a dump without first wget ing the dotnet-dump binary.

I'd be really happy if you could point me in the direction of some example/documentation that would help

[jander-msft]

Unfortunately, I do not have a sample or a walkthrough to demonstrate this at the moment, but I'll take this as feedback that we should have one.

There's already an existing discussion in dotnet/dotnet-monitor that describes some of what needs to be done to make it work: dotnet/dotnet-monitor#7017 (comment). It looks like kubectl debug --custom argument in addition to needing a preconfigure volume mount may work.

I would encourage you to file a discussion item at https://github.com/dotnet/dotnet-monitor/discussions if you need further assistance in using .NET Monitor or participate in the existing ones.

[akoken]

You cannot shell into the distroless container as it lacks both a shell and a package manager. This aligns with the core principle of distroless images, aimed at minimising the attack surface. They contain only the essential system libraries necessary for running the application.

For troubleshooting purposes, you might consider utilising ephemeral containers, a special container type that runs temporarily within an active pod. This can be achieved through the kubectl debug command. If you use Docker instead of K8s, there is a similar command named docker debug.

If you prefer to continue using regular images, you can install tools during the build stage and then copy them to your final stage:

FROM mcr.microsoft.com/dotnet/sdk:8.0 as build
...
RUN dotnet tool install gcdump --tool-path /tools
...

FROM mcr.microsoft.com/dotnet/aspnet:8.0 as final
...
COPY --from=build /tools . 
...