System.InvalidProgramException: Common Language Runtime detected an invalid program

Question

I have a .net 6 solution that has two Service Fabric stateless services. The first one "ServiceA" is a web service that receives incoming API requests and then it creates a proxy to second one "ServiceB" that is deployed in the same cluster. The problem is, the when the proxy is created and I try to access the method that ServiceB implements, I get this exception:

"message": "System.AggregateException: One or more errors occurred. (Common Language Runtime detected an invalid program.)
 ---> System.InvalidProgramException: Common Language Runtime detected an invalid program.
at ServiceB.Contracts.Infrastructure.IServiceB_.service.disp.IServiceBMethodDispatcher.OnDispatchAsync(Int32 , Object , IServiceRemotingRequestMessageBody , IServiceRemotingMessageBodyFactory , CancellationToken )
at Microsoft.ServiceFabric.Services.Remoting.V2.Builder.MethodDispatcherBase.DispatchAsync(Object objectImplementation, Int32 methodId, IServiceRemotingRequestMessageBody requestBody, IServiceRemotingMessageBodyFactory remotingMessageBodyFactory, CancellationToken cancellationToken)
at Microsoft.ServiceFabric.Services.Remoting.V2.Runtime.ServiceRemotingMessageDispatcher.OnDispatch(IServiceRemotingRequestMessageHeader requestMessageHeaders, IServiceRemotingRequestMessageBody requestBody, CancellationToken cancellationToken)
--- End of stack trace from previous location ---
at Microsoft.ServiceFabric.Services.Remoting.V2.Runtime.ServiceRemotingMessageDispatcher.OnDispatch(IServiceRemotingRequestMessageHeader requestMessageHeaders, IServiceRemotingRequestMessageBody requestBody, CancellationToken cancellationToken)
at Microsoft.ServiceFabric.Services.Remoting.V2.Runtime.ServiceRemotingMessageDispatcher.<>c__DisplayClass8_1.<HandleRequestResponseAsync>
    b__0(CancellationToken cancellationToken)
at Microsoft.ServiceFabric.Services.Remoting.Runtime.ServiceRemotingCancellationHelper.DispatchRequest[T](Int32 interfaceId, Int32 methodId, String callContext, Func`2 dispatchFunc)
at Microsoft.ServiceFabric.Services.Remoting.V2.Runtime.ServiceRemotingMessageDispatcher.HandleRequestResponseAsync(IServiceRemotingRequestContext requestContext, IServiceRemotingRequestMessage requestMessage)\r\n   at Microsoft.ServiceFabric.Services.Remoting.V2.FabricTransport.Runtime.FabricTransportMessageHandler.RequestResponseAsync(FabricTransportRequestContext requestContext, FabricTransportMessage fabricTransportMessage)
--- End of inner exception stack trace ---
at Microsoft.ServiceFabric.Services.Communication.Client.ServicePartitionClient`1.InvokeWithRetryAsync[TResult](Func`2 func, CancellationToken cancellationToken, Type[] doNotRetryExceptionTypes)\r\n   at Microsoft.ServiceFabric.Services.Remoting.V2.Client.ServiceRemotingPartitionClient.InvokeAsync(IServiceRemotingRequestMessage remotingRequestMessage, String methodName, CancellationToken cancellationToken)
at Microsoft.ServiceFabric.Services.Remoting.Builder.ProxyBase.InvokeAsyncV2(Int32 interfaceId, Int32 methodId, String methodName, IServiceRemotingRequestMessageBody requestMsgBodyValue, CancellationToken cancellationToken)
at Microsoft.ServiceFabric.Services.Remoting.Builder.ProxyBase.ContinueWithResultV2[TRetval](Int32 interfaceId, Int32 methodId, Task`1 task)\r\n   at MyApp.WebService.Proxies.ServiceBProxy.IsSomeStuffInProgressAsync() in C:\\dev\\repos\my-sf-app\\MyApp.WebService\\Proxies\\ServiceBProxy.cs:line 21   
at MyApp.WebService.Controllers.FileController.CheckSomeStuffState() in C:\\dev\\repos\my-sf-app\\MyApp.WebService\\Controllers\\FileController.cs:line 44

... and some other bits"

This is how the proxy is created in ServiceA:

public class ServiceBServiceProxy : IServiceB
    {
        private const string ServiceBUri = "fabric:/MyApp.ServiceB/ServiceBService";


        public async Task<bool> IsSomeStuffInProgressAsync()
        {
            var proxy = ServiceProxy.Create<IServiceB>(new Uri(ServiceBUri));
            return await proxy.IsSomeStuffInProgressAsync().ConfigureAwait(false); -- error thrown at this poit
        }

    }

The interface that is implemented in both services ServiceA and ServiceB

public interface IServiceB : IService
    {
        Task<bool> IsSomeStuffInProgressAsync();
    }

IService implementation is ServiceB:

internal sealed class ServiceB : StatelessService, IServiceB
    {
        private readonly Container _container;

        public ServiceB(StatelessServiceContext context, Container container)
            : base(context)
        {
            _container = container;
        }

    protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
    {
        return this.CreateServiceRemotingInstanceListeners();
    }

        public async Task<bool> IsSomeStuffInProgressAsync()
        {
            await using (AsyncScopedLifestyle.BeginScope(_container))
            {
                var someService = _container.GetService<SomeService>(); 

                return await someService.CheckSomeStuff().ConfigureAwait(false);
            }
        }

    }

Please note, this error is thrown in local environment and on real Test server. Any ideas are appreciated, thanks! Let me know if I need to populate this question with more details.

SF package versions match in both services.

<PackageReference Include="Microsoft.ServiceFabric.Services" Version="3.3.638" />
<PackageReference Include="Microsoft.ServiceFabric.Services.Remoting" Version="3.3.638" />
<PackageReference Include="Microsoft.ServiceFabric.AspNetCore.Kestrel" Version="3.3.638" /> - in WebService (ServiceA)

Both services are successfully deployed to SF cluster. Currently I use the remoting version 1. I did try updating it to version 2 but I go exactly same error in the end. I tried disabling code optimization and enabling support for 32-bit apps but that didn't help as well.

Originally, the ServiceB was stateful but now I converted it to stateless. There were some net framework 4.8 class libraries but all of them were converted to .net 6 and all dependencies were updated/resolved as well. Not sure what else needs to be done to make this work. All the code looks pretty much the same as in this MS documentation https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-communication-remoting

Answer 1

Ok so to fix this I just needed to update ServiceFabric packages to the latest version 3.3.638 -> 6.0.1048.