[BUG] APM profiler-based auto instrumentation causes applications to crash when accessing satellite assemblies for resources

[stevejgordon]

APM Agent version

1.27.0

Operating system and version:
Windows 11

.NET Framework/Core name and version: .NET 4.8

Describe the bug

If the APM automatic instrumentation is enabled against an ASP.NET (.NET Framework) application that uses resources in satellite assemblies, the app crashes multiple times. Eventually, it will usually start sending trace and metric data. This is reproducible for me locally. It seems the trigger is the load of the satellite assembly containing resources. As soon as code runs which accesses a resource, the crash occurs.

To Reproduce

1. Create a ASP.NET WebApi project.
2. Add MyResources.resx and MyResources.en-US.resx files

3. Add a single string and value

4. Consumbe the resource in code

public class HomeController : Controller
{
    public ActionResult Index()
    {
        CultureInfo originalCulture = Thread.CurrentThread.CurrentCulture;

        try
        {
            CultureInfo newCulture = new CultureInfo("en-US");
            Thread.CurrentThread.CurrentCulture = newCulture;
            Thread.CurrentThread.CurrentUICulture = newCulture;

            ViewBag.Title = MyResources.Test;
        }
        catch (CultureNotFoundException e)
        {
            Console.WriteLine("Unable to instantiate culture {0}", e.InvalidCultureName);
        }
        finally
        {
            Thread.CurrentThread.CurrentCulture = originalCulture;
            Thread.CurrentThread.CurrentUICulture = originalCulture;
        }

        return View();
    }
}

5. Deploy the app to IIS and observe it starting without issue
6. Enable profiler auto instrumentation per this doc
7. After recycling the app pool, as soon as the first code accesses a resource (i.e. a request hits the home controller above), the app will often crash at least once. It generally eventually recovers after several crashes.

The crash can be captured using tools such as DebugDiag. When a rule is active against the app pool, one will often observe at least one crash.

...Second_Chance_Exception_C0000374.dmp

Expected behavior

The application starts without crashing.

Actual behavior

The application often crashes.

Native call stack at the point of the crash:

 # Child-SP          RetAddr               Call Site
00 0000009a`d927c350 00007ff9`e195c133     ntdll!RtlReportFatalFailure+0x9
01 0000009a`d927c3a0 00007ff9`e19651ca     ntdll!RtlReportCriticalFailure+0x97
02 0000009a`d927c490 00007ff9`e19654aa     ntdll!RtlpHeapHandleError+0x12
03 0000009a`d927c4c0 00007ff9`e19714a5     ntdll!RtlpHpHeapHandleError+0x7a
04 0000009a`d927c4f0 00007ff9`e18fc418     ntdll!RtlpLogHeapFailure+0x45
05 0000009a`d927c520 00007ff9`e188ab11     ntdll!RtlpFreeHeapInternal+0x70d98
06 0000009a`d927c5e0 00007ff9`89b6b357     ntdll!RtlFreeHeap+0x51
07 0000009a`d927c620 00007ff9`b7bdce76     elastic_apm_profiler!DllGetClassObject+0x12487
08 0000009a`d927c810 00007ff9`b7a900dc     clr!EEToProfInterfaceImpl::AssemblyLoadFinished+0x72
09 0000009a`d927c850 00007ff9`b77d1d30     clr!Module::NotifyProfilerLoadFinished+0x26f9f8
0a 0000009a`d927c8e0 00007ff9`b77d18a4     clr!DomainFile::DoIncrementalLoad+0x148
0b 0000009a`d927c920 00007ff9`b77d15e4     clr!AppDomain::TryIncrementalLoad+0x104
0c 0000009a`d927ccc0 00007ff9`b782ab59     clr!AppDomain::LoadDomainFile+0x154
0d 0000009a`d927cd90 00007ff9`b782a8db     clr!AppDomain::LoadDomainAssemblyInternal+0x221
0e 0000009a`d927cea0 00007ff9`b782c74e     clr!AppDomain::LoadDomainAssembly+0xa3
0f 0000009a`d927d1c0 00007ff9`b78b1f24     clr!AssemblySpec::LoadDomainAssembly+0x21a
10 0000009a`d927d790 00007ff9`59022a07     clr!AssemblyNative::Load+0x3c4
11 0000009a`d927db20 00007ff9`59022056     0x00007ff9`59022a07

CLR call stack, which triggers the issue:

0000009ad927da18 00007ff9e195c169 [HelperMethodFrame_PROTECTOBJ: 0000009ad927da18] System.Reflection.RuntimeAssembly._nLoad(System.Reflection.AssemblyName, System.String, System.Security.Policy.Evidence, System.Reflection.RuntimeAssembly, System.Threading.StackCrawlMark ByRef, IntPtr, Boolean, Boolean, Boolean)
0000009ad927db20 00007ff959022a07 System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(System.String, System.Globalization.CultureInfo, System.Version, Boolean, System.Threading.StackCrawlMark ByRef)
0000009ad927dbc0 00007ff959022056 System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(System.Globalization.CultureInfo, System.Threading.StackCrawlMark ByRef)
0000009ad927dc30 00007ff9587c2d65 System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(System.Globalization.CultureInfo, System.Collections.Generic.Dictionary`2<System.String,System.Resources.ResourceSet>, Boolean, Boolean, System.Threading.StackCrawlMark ByRef)
0000009ad927dcd0 00007ff9587c29f6 System.Resources.ResourceManager.InternalGetResourceSet(System.Globalization.CultureInfo, Boolean, Boolean, System.Threading.StackCrawlMark ByRef)
0000009ad927dd90 00007ff9587c270a System.Resources.ResourceManager.InternalGetResourceSet(System.Globalization.CultureInfo, Boolean, Boolean)
0000009ad927ddd0 00007ff9587c1c73 System.Resources.ResourceManager.GetString(System.String, System.Globalization.CultureInfo)
0000009ad927de50 00007ff959491a75 WebApplication8.Controllers.HomeController.Index()

[MobaLinux]

@stevejgordon there isn't anything to resolve this issue? any workaround?

[stevejgordon]

@MobaLinux Unfortunately, there's no workaround for the profiler-based instrumentation. The manual NuGet installation is unaffected and is the current workaround. We spent a fair bit of time trying to find a suitable solution but have yet to solve the issue reliably. Another potential alternative is using the OpenTelemetry auto-instrumentation and exporting to the APM server, although I've yet to test if that runs into similar limitations.