Debugging CoreFX on macOS

A few weeks ago I was trying to set up a small iOS application when I came across an error in .NET Core when attempting to connect to the new HTTP/2-based Apple Push Notification System gateway.

It appears that the issue is two-fold:

The version of curl and libcurl that Apple ship with macOS does not support HTTP/2. (rdar://problem/29891359). This causes a crash when Apple's servers terminates the connection.
Even when forcing .NET Core to use a version of libcurl that does support HTTP/2, it crashes when parsing the response.

The exceptions here occurred in System.Net.Http which is part of the .NET Framework base class libraries.

In the .NET Core world, .NET Core is split into two main parts - the Common Language Runtime is called CoreCLR, and the foundational libraries are called CoreFX.

Armed with this information and a handful of markdown pages from the CoreFX wiki, I set out to debug the exception in the second point.

For this post, I'll be using $COREFX to refer to the directory where the CoreFX source tree lives. Wherever you see this, substitute the full path to wherever you put the source code.

First, to get the CoreFX source code, run:

 git clone https://github.com/dotnet/corefx

On macOS there are some build-times and run-time prerequisites, so also run:

 brew install cmake pkgconfig openssl
 
 ln -s /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib /usr/local/lib/
 ln -s /usr/local/opt/openssl/lib/libssl.1.0.0.dylib /usr/local/lib/
    
 mkdir -p /usr/local/lib/pkgconfig

 ln -s /usr/local/opt/openssl/lib/pkgconfig/libcrypto.pc /usr/local/lib/pkgconfig/
 ln -s /usr/local/opt/openssl/lib/pkgconfig/libssl.pc /usr/local/lib/pkgconfig/
 ln -s /usr/local/opt/openssl/lib/pkgconfig/openssl.pc /usr/local/lib/pkgconfig/

Next, to ensure the workspace is clean, all build tools are available, and build the code, run the following:
```
 cd $COREFX
 ./clean.sh -all 
 ./sync.sh
 ./build.sh
 ./build-tests.sh 
```
This will build both the native and managed components of CoreFX, and will also run all unit tests.

If, like me, you also have an older Mac, this will make it sound like a small jetliner for a few minutes.

Since I'm trying to debug an issue with HTTP/2, I was also quite pleased to spot the following in the build output:
```
 -- Performing Test HAVE_CURL_HTTP_VERSION_2_0
 -- Performing Test HAVE_CURL_HTTP_VERSION_2_0 - Success
```
All-up, that took 55 minutes to compile and test. I also had 3 failing unit tests out of the box, which really isn't a good look.
Next, we need to write a repro case and run it. In my case, I already had Visual Studio Code installed with the C# extension. If you don't have this already set up, set it up.

Once Visual Studio Code is set up, open it to $COREFX/src.

Next, find somewhere to plonk a unit test that reproduces the issue. Since I already had a repro case in the issue I filed, it was easy enough to adapt it into the following Xunit test case, which I added to the top of System.Net.Http/tests/FunctionalTests/HttpClientTest.cs.
```
 [Fact]
 public async Task Yaakov_Test_Http2_DoNotCommit()
 {
     HttpResponseMessage response;            
     using (var client = new HttpClient())
     {
         var request = new HttpRequestMessage(HttpMethod.Post, "https://api.push.apple.com/3/device/AAA");
         request.Version = new Version(2, 0);
         response = await client.SendAsync(request);
     }
     Assert.Equal(HttpStatusCode.Forbidden, response.StatusCode);
     var content = await response.Content.ReadAsStringAsync();
     Assert.Equal(content, @"{""reason"":""MissingProviderToken""}");
 }
```
I don't usually write my code that compact, but the Markdown parser on this blog misbehaved when I had blank lines in that snippet.
Recompile the test assembly and run it. .NET code is typically compiled using MSBuild, and in CoreFX you can find a wrapper script for this for macOs and Linux, in $COREFX/Tools/msbuild.sh.

To recompile the System.Net.Http FunctionalTests assembly, run:
```
 $COREFX/Tools/msbuild.sh src/System.Net.Http/tests/FunctionalTests/System.Net.Http.Functional.Tests.csproj /t:RebuildAndTest
```
Once complete, you'll get a warning that one unit test failed. Time to debug it.
To debug the unit test, click the fourth icon in Visual Studio Code's left menu bar, the one that looks like a no-bug sign.

In the Debug menu just to the right, click on the gear, then select '.NET Core'.

In launch.json, which is now open, modify the ".NET Core Launch (console)" section as follows. Again, substitute $COREFX for the full path to CoreFX.
1. Delete the line "preLaunchTask": "build",.
2. Modify the value of "program" to be :
```
 "$COREFX/bin/runtime/netcoreapp-OSX-Debug-x64/corerun"
```
3. Modify the value of "args" to be:
```
 ["$COREFX/bin/Unix.AnyCPU.Debug/System.Net.Http.Functional.Tests/netstandard/xunit.console.netcore.exe", "System.Net.Http.Functional.Tests.dll", "-method", "System.Net.Http.Functional.Tests.HttpClientTest.Yaakov_Test_Http2_DoNotCommit"]
```
  This will launch the Xunit runner, and run just the single unit test that we defined earlier.
4. Modify the value of "cwd" to be:
```
 "$COREFX/bin/Unix.AnyCPU.Debug/System.Net.Http.Functional.Tests/netstandard/"
```
Switch back to the unit test source code. In the margin, just to the left of the line numbers, click to set a breakpoint at the start of the test.

In the dropdown to the left of the gear menu we clicked on previously, select ".NET Core Launch (console)", then click on the green play button.
Now that we're debugging the unit test, we're looking for an exception that caused the request to fail. In the Breakpoints section in the lower-left of the screen, check 'All Exceptions', then click the green play button in the upper-centre of the screen to continue.

Our first exception occurs in CurlHandler.ThrowIfCURLError, and if we mouse over the variable error we can see that it has the value 56.

This matches the exception from the issue that was thrown when using the Apple version of libcurl, but not the one thrown when using the Homebrew version. Let's try that instead.
Click the red stop button in the top of the screen to stop debugging. Open launch.json again (or just switch tabs), and add the following block to the configuration we were editing previously:
```
 "env": {
     "DYLD_LIBRARY_PATH": "/usr/local/opt/curl/lib"
 }
```
This will set the DYLD_LIBRARY_PATH environment variable before launching corerun, which will cause dyld to look in the path we have specified before looking in other paths for dynamic libraries. Since libcurl.dylib exists there, it will be the one that is loaded.
Click the green play button again to start debugging, then click the one in the top-centre of the screen to continue past the first breakpoint.

The IDE has stopped at an exception thrown with an error value of SR.net_http_unix_invalid_response.

If we step up the stack one frame we can see what's going on here. In the next screenshot, I've added a few expressions to the watch window so that we can see what's going on.

Over here, we're parsing the first line of the HTTP response, which is HTTP/2 403. The code in .NET Core is expecting the string HTTP/ followed by a number, a '.' character, and then another number.

Since HTTP/2 is not HTTP/2.0, there is no '.' to be seen, and CheckResponseMsgFormat throws an exception.

I'll leave actually fixing the bug as an exercise for the reader, or you can just go and have a look at my Pull Request.

All in all, although this started off seeming quite complex and daunting, it's surprisingly simple and straightforward once I understand how CoreFX is set out, how it's build toolchain works (roughly), and how to run and debug it.

I managed to go from absolute scratch to filing a Pull Request in an hour (plus the time it took to compile CoreFX initially), knowing nothing about the internals of the platform or the development environment required to run and debug it.

I'm quite impressed with how well it's pulled together and how well-documented the guides are on the CoreFX wiki.

I only hope I'll actually be able to use HTTP/2 with .NET Core in the near future. 😁

Debugging CoreFX on macOS

Stack traces from async methods in .NET

Using HTTP/2 in .NET client applications