I have an ASP.NET Core 8.0 Web API that is deployed in k8s.
What is happening is that after some time (random might range from 30 minutes to a day or more), the following exception is printed in console and application terminates i.e. k8s restarts it.
Unhandled exception in Main: System.NullReferenceException: Object reference not set to an instance of an object.
at System.Collections.Concurrent.ConcurrentDictionary`2.Enumerator.MoveNext()
at System.Net.Http.HttpConnectionPoolManager.RemoveStalePools()
at System.Net.Http.HttpConnectionPoolManager.<>c.<.ctor>b__11_0(Object s)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
Also there are some exception related to Kestrel:
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Object reference not set to an instance of an object.)
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.RemoveConnection(Int64 id)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.RemoveConnection(Int64 id)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1.ExecuteAsync()
--- End of inner exception stack trace ---
I have made sure that all async are being awaited, all IDisposable are in using blocks.
Taskthat is not awaited and is throwing exception. I would look for that. Or try to subscribe to event TaskScheduler.UnobservedTaskExceptionSystem.AggregateException: A Task's exception(s) were not observed either by ..is being logged in TaskScheduler.UnobservedException and also set as observed. Also I have gone through all the web api routes from starting of server to crash to make sure that all tasks are awaited.