State of the art on using .NET + Docker + Apple M1 #3848

richlander · 2022-06-08T21:58:30Z

richlander
Jun 8, 2022
Maintainer

Update: See https://devblogs.microsoft.com/dotnet/improving-multiplatform-container-support/

A friend of mine asked for an update on what the story was on using .NET container images on Apple M1, assuming your goal was to deploy to x64 services in Azure (or pick your cloud).

There are various challenges, some specific to .NET and others more general. Windows Arm64 (with x64 emulation) is likely identical to this.

Here's what you want:

High-fidelity dev experience with prod.
High performance dev experience.
Intuitive docker gestures.

Today, it is very easy to fall into an Arm64-centric experience on M1 (for any app plat, not just .NET) and then push an Arm64 image to your registry, and then realize it won't work on your x64 cloud hardware. And then you are not sure what happened. Ughh.

Docker desktop defaults to Arm64

Docker on Apple M1 defaults to native architecture, which is Arm64. If the images you are pulling are multi-arch, you'll pull Arm64. If they are single-arch (x64 or Arm64), you'll pull whatever that arch is. That's the right design choice, but also confusing if your deployment target in x64.

All .NET images are multi-arch. We want to enable using the same tags in a variety of environments. That's all good. It however means that you'll always get Arm64 images by default, on Apple M1 machines. Other app platforms will be the same.

I'll show you, using dotnetapp:

% docker build --pull -t dotnetapp .
% docker run --rm dotnetapp
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.5
Debian GNU/Linux 11 (bullseye)

OSArchitecture: Arm64
ProcessorCount: 4
TotalAvailableMemoryBytes: 3.84 GiB

That's clearly Arm64. We can force building as x64.

% docker build --pull --platform linux/amd64 -t dotnetapp .
% docker run --rm dotnetapp
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.5
Debian GNU/Linux 11 (bullseye)

OSArchitecture: X64
ProcessorCount: 4
TotalAvailableMemoryBytes: 3.84 GiB

That's x64. That's what we wanted for this scenario.

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

That error is coming from Docker. It's there to tell that you're using emulation.

If you want to live the x64 lifestyle on your Apple M1 machine, you need to use the --platform linux/amd64 flag. Perhaps there is a way to enable amd64 as the default. I don't know.

.NET isn't supported in QEMU

The bigger issue is that .NET isn't supported in QEMU. QEMU is the emulator that Docker Desktop uses for emulation, on both x64 (to emulate Arm64) and Arm64 (to emulate x64) machines.

I'll show you the problem, using aspnetapp (building and running as x64):

% docker build --pull --platform linux/amd64 -t aspnetapp .
% docker run --rm -p 8000:80 aspnetapp
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Unhandled exception. System.IO.IOException: Function not implemented
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationExtensions.Add[TSource](IConfigurationBuilder builder, Action`1 configureSource)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, IFileProvider provider, String path, Boolean optional, Boolean reloadOnChange)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, String path, Boolean optional, Boolean reloadOnChange)
   at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.<>c__DisplayClass11_0.<ConfigureDefaults>b__1(HostBuilderContext hostingContext, IConfigurationBuilder config)
   at Microsoft.AspNetCore.Hosting.BootstrapHostBuilder.RunDefaultCallbacks(ConfigurationManager configuration, HostBuilder innerBuilder)
   at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(WebApplicationOptions options, Action`1 configureDefaults)
   at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder(String[] args)
   at Program.<Main>$(String[] args) in /source/aspnetapp/Program.cs:line 1
qemu: uncaught target signal 6 (Aborted) - core dumped

That's not pretty. We're hoping this QEMU issue gets resolved.

Workaround 1

Use the multi-arch tags as intended, just like in the aspnetapp Dockerfile. Run .NET in containers as Arm64 in Apple M1. .NET has excellent fidelity across architectures so you will get an experience. If you want to deploy images to a registry, either build with --platform linux/amd64 or build in a CI service like GitHub Actions. They will naturally build as x64.

This approach 100% works. You get the best performance on Apple M1 and straightforward gestures. It's the no-compromises option, since emulated QEMU will always be a lot slower than native architecture.

Workaround 2

This workaround forces your Dockerfile to produce x64 assets by default. The SDK runs in whatever arch is chosen for docker build. It enables you to pivot the final image pretty easily with a --build-arg. The intent of this is that docker build will produce an asset that runs in your x64 cloud by default.

I modified the aspnetapp Dockerfile a bit:

ARG ARCH=amd64
ARG TAG=6.0-bullseye-slim-$ARCH
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /source

# copy csproj and restore as distinct layers
COPY *.sln .
COPY aspnetapp/*.csproj ./aspnetapp/
RUN dotnet restore

# copy everything else and build app
COPY aspnetapp/. ./aspnetapp/
WORKDIR /source/aspnetapp
RUN dotnet publish -c release -o /app --no-restore

# final stage/image
FROM mcr.microsoft.com/dotnet/aspnet:$TAG
WORKDIR /app
COPY --from=build /app ./
ENTRYPOINT ["dotnet", "aspnetapp.dll"]

I added the two ARG lines at the start and added $TAG in the last FROM statement. By default, this Dockerfile will produce x64 images by default, even when built on Apple M1 machines.

However, if you build that image and docker run it, it will still fail, as I demonstrated earlier. If you want a good dev experience, you can opt into building and running it as Arm64.

Like this:

% docker build --pull --build-arg ARCH=arm64v8 -t aspnetapp .
% docker run --rm -p 8000:80 aspnetapp

That approach allows you to build an image on your M1 machine -- x64 by default -- and then push to a registry. For some folks, that's might be what they are looking for.

woachk · 2022-06-09T05:27:55Z

woachk
Jun 9, 2022

.NET isn't supported in QEMU

Note: On the Rosetta for Linux implementation present on macOS Ventura, .NET is functional.

$ sudo docker run --rm --platform linux/amd64  -it mcr.microsoft.com/dotnet/samples:aspnetapp
Unable to find image 'mcr.microsoft.com/dotnet/samples:aspnetapp' locally
aspnetapp: Pulling from dotnet/samples
42c077c10790: Already exists 
87ed72daf751: Already exists 
5873b5a7554b: Already exists 
016563de4cce: Already exists 
3ce4f4cca166: Pull complete 
f9a0382f79fd: Pull complete 
d98dcc714b37: Pull complete 
Digest: sha256:25c74bf653c681ab74d72d5d712405cf26778e079f66744037d212c83d2863e9
Status: Downloaded newer image for mcr.microsoft.com/dotnet/samples:aspnetapp
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {0d4fcd8a-bc32-4a7b-b857-bc03304e7d48} may be persisted to storage in unencrypted form.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app/

I'd vote for approach 1. Servers on the cloud are not always x86 either. :)

And then override via --platform when required, that's a task to be done by the user.

0 replies

richlander · 2022-06-09T05:39:40Z

richlander
Jun 9, 2022
Maintainer Author

Very nice! This is great to see. I specifically posted this here in the hope that I'd learn something. This is better than I was expecting.

100% agree on all your points.

I guess I need to upgrade to Ventura!

0 replies

richlander · 2022-06-10T23:20:30Z

richlander
Jun 10, 2022
Maintainer Author

Got a cheatsheet on that @woachk? I installed Ventura but don't see a difference in behavior. I'm sure there is more to it. I looked around and didn't see any instructions anywhere.

Also, why do you use sudo?

0 replies

woachk · 2022-06-10T23:41:02Z

woachk
Jun 10, 2022

@richlander the cheat sheet is at https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta?language=objc and https://developer.apple.com/documentation/virtualization/running_gui_linux_in_a_virtual_machine_on_a_mac?language=objc for the overall VM test project. (Adding Rosetta tidbits to it is quite trivial, as explained on the Rosetta page)

Docker Desktop didn’t pick that up yet (and i’m currently writing an alternative to it anyway, because of licensing reasons notably, but not only that.)

Why I use sudo is because the account that I’m calling Docker from isn’t part of the docker group, and as such can’t communicate with the dockerd daemon otherwise.

0 replies

richlander · 2022-06-10T23:57:57Z

richlander
Jun 10, 2022
Maintainer Author

OK. I saw all that. I was wondering if there was something more productized from Docker. Good to know that I wasn't missing anything.

OK, the usual reason for using sudo. Just checking.

Do you just install Docker on Linux (in the VM) and then re-direct your docker daemon to it?

0 replies

woachk · 2022-06-11T00:05:38Z

woachk
Jun 11, 2022

Yeah, I just ssh or execute directly from the VM.

0 replies

richlander · 2022-06-11T00:14:14Z

richlander
Jun 11, 2022
Maintainer Author

Oh. I see. I was fixated on docker desktop (for macOS). Thanks!

0 replies

dgilsonAfelio · 2023-08-18T07:31:40Z

dgilsonAfelio
Aug 18, 2023

I'm not able to make it works using Rancher Desktop.
Tried both QEmu and VZ

8 replies

gilsdav Aug 23, 2023

FROM mcr.microsoft.com/dotnet/sdk:6.0 I tried with the sdk 8 too but same error.
I'm on M1 and I'm not using multi-arch.
If I'm understand well, RancherDesktop run everything into lima VM on Qemu or VZ.
I tried with platform arm and amd but it always freezes.

richlander Aug 23, 2023
Maintainer Author

Can you build this Dockerfile and share the results? It will validate the environment.

https://gist.github.com/richlander/70cde3f0176d36862af80c41722acd47

.NET images will only run if they are run natively or in Rosetta. They don't run well in QEMU.

gilsdav Aug 23, 2023

I got the same logs for QEMU and VZ + Rosetta.

With QEMU:

With VZ + Rosetta support:

Maybe Apple's VZ is not supported better than QEMU?

richlander Aug 23, 2023
Maintainer Author

Super odd. That's not virtualization but running natively, which is what I wanted to validate.

Can you share the error you are seeing with your actual Dockerfile?

gilsdav Oct 16, 2023

I tried with podman and it works. I Will officialy switch from Docker to podman. And... Podman runs on Qemu 😅

dhduvall · 2023-09-08T18:24:18Z

dhduvall
Sep 8, 2023

I'm surprised not to see any mention of using TARGETPLATFORM and BUILDPLATFORM. It's a little more setup, but it works nicely, as long as you remember to use --platform linux/amd64 when building for x64 on M1. It also requires cross-compilation, which IIRC didn't work in all combinations until somewhat recently.

The complication is that the platform names .NET uses differ from the ones Go uses, so you have to use an annoying translation step.

My Dockerfile looks like this:

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:7.0 as build

ARG BUILDPLATFORM
ARG TARGETPLATFORM

RUN mkdir /src /app
WORKDIR /src

COPY xlate-platform.sh .
RUN sh -x xlate-platform.sh

COPY . /src

RUN . ./.targetplatform; dotnet build -r $RUNTIME --no-self-contained -o /app

FROM mcr.microsoft.com/dotnet/aspnet:7.0

WORKDIR /app

COPY --from=build /app /app

xlate-platform.sh:

#!/bin/sh

# .NET platform names are different from the ones Docker (i.e., Go) uses, so we
# can't just say dotnet build -r $TARGETPLATFORM.  We have to transform the name
# and use the new one.
#
# This script does the mapping and creates a shell-sourceable file setting the
# variable $RUNTIME, for use in later RUN invocations; e.g.,
#
#     RUN sh /xlate-platform.sh
#     RUN . ./.targetplatform; dotnet publish -r $RUNTIME

# If $TARGETPLATFORM isn't defined, just assume amd64 linux.
case $TARGETPLATFORM in
    "" | "linux/amd64")
        RUNTIME=linux-x64 ;;
    "linux/arm64")
        RUNTIME=linux-arm64 ;;
    *)
        echo "Unsupported target platform \"$TARGETPLATFORM\""
        exit 1
esac

echo "RUNTIME=$RUNTIME" > .targetplatform

If there's a cleaner way of introducing $RUNTIME into the environment, I'd love to know. Since RUN commands are neither login nor interactive shells, there doesn't seem to be a way to configure those shell executions. Perhaps a better way is to configure MSBuild instead, but I haven't explored that yet.

2 replies

lbussell Sep 8, 2023
Maintainer

Since .NET SDK 7.0.400, you can use -a $TARGETARCH instead of your -r $RUNTIME. We have an open PR to update our samples to use the new format.

dhduvall Sep 8, 2023

Nice. Once we're off of 6.0, I'll have to update to use that. I guess the OS is going to be linux even on MacOS, so only the architecture is interesting?

kurt-mueller-osumc · 2023-10-16T15:36:47Z

kurt-mueller-osumc
Oct 16, 2023

https://github.com/kurt-mueller-osumc/dotnet-amd64

This is a simple repository that demonstrates that commands like docker build --platform linux/amd64 . stall when running commands like dotnet restore. I have a Mac with docker desktop installed and rosetta emulation enabled.

This is frustrating because I'd love to develop dotnet-isolated Azure Functions in docker on my Mac.

1 reply

shad0w-jo4n Jan 30, 2024

Does that have any solution or opened issue? I have the same problem

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

State of the art on using .NET + Docker + Apple M1 #3848

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 10 comments 11 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

State of the art on using .NET + Docker + Apple M1 #3848

richlander Jun 8, 2022 Maintainer

Docker desktop defaults to Arm64

.NET isn't supported in QEMU

Workaround 1

Workaround 2

Replies: 10 comments · 11 replies

richlander Jun 9, 2022 Maintainer Author

richlander Jun 10, 2022 Maintainer Author

richlander Jun 10, 2022 Maintainer Author

richlander Jun 11, 2022 Maintainer Author

richlander Aug 23, 2023 Maintainer Author

richlander Aug 23, 2023 Maintainer Author

lbussell Sep 8, 2023 Maintainer

richlander
Jun 8, 2022
Maintainer

Replies: 10 comments 11 replies

richlander
Jun 9, 2022
Maintainer Author

richlander
Jun 10, 2022
Maintainer Author

richlander
Jun 10, 2022
Maintainer Author

richlander
Jun 11, 2022
Maintainer Author

richlander Aug 23, 2023
Maintainer Author

richlander Aug 23, 2023
Maintainer Author

lbussell Sep 8, 2023
Maintainer