Backend sometimes freezes #7598

kasem created 4 years ago

Backend sometimes freezes in production without any response until restarting IIS. No related logs found to tell anything about this problem. I only catched ony exception in logs "There is already an open DataReader associated with this Command which must be closed first" and I'm not sure if such a bug can freeze the whole system? or such type of exceptions should be silently happening without stucking the system

Any idea how to find and solve this annoying critical problem?

Thanks

13 Answer(s)

0

alper created 4 years ago
Support Team

check the event viewer logs for unhandled exceptions. sometimes this can occur if an async task is not called with await keyword. and "already open datareader" may be the result of this issue.
0

kasem created 4 years ago

The problem that exception I got lacks details that help me identify source of problem. No method, file, line, nothing. I have quite a large codebase and finding such a problem in my codebase is like shooting in the dark.
0

kasem created 4 years ago

A possible fix for Open DataReader problem is enabling MARS https://docs.microsoft.com/en-us/sql/relational-databases/native-client/features/using-multiple-active-result-sets-mars?view=sql-server-2017

Do you recommend doing so?
0
kasem created 4 years ago
Sorry another question. I feel it's hard to find unhandled exception and I really don't want the app in production to crash. I need your help in the following:

Is there a way to auto restart service when app crashes as a result of an unhandled exception?

How can I debug/find details about unhandled exceptions? Can't you log unhandled exceptions before app crashes so I get more info about the bug in order to fix it? https://docs.microsoft.com/en-us/dotnet/api/system.windows.forms.application.setunhandledexceptionmode?view=netframework-4.8

alper created 4 years ago

Support Team

hi,

To catch unhandled exceptions and also exceptions in different threads you can use the below code block in your startup of the web project.

  AppDomain.CurrentDomain.UnhandledException += delegate (object sender, UnhandledExceptionEventArgs args)
            {
                //write log...
                Console.WriteLine(args.ExceptionObject as Exception);
            };

            System.Threading.Tasks.TaskScheduler.UnobservedTaskException +=
                delegate (object sender, UnobservedTaskExceptionEventArgs args)
                {
                    //write log...
                    Console.WriteLine(args.Exception);
                    args.SetObserved();
                };

0

ismcagdas created 4 years ago
Support Team

Hi @kasem

Do you have any "async void" method in your app ?
0

kasem created 4 years ago

Thank you @alper and @ismcagdas

I don't have async void.

Everytime system crashes I get the following two errors in windows event viewer:

Application: MyApp.Web.Host.exe CoreCLR Version: 4.6.27817.3 Description: The process was terminated due to an unhandled exception. Exception Info: System.InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first. at System.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand command) at System.Data.SqlClient.SqlInternalTransaction.Rollback() at System.Data.SqlClient.SqlTransaction.Dispose(Boolean disposing) at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Dispose() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ClearTransactions() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Dispose() at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Microsoft.EntityFrameworkCore.DbContext.Dispose() at Castle.MicroKernel.ComponentActivator.AbstractComponentActivator.ApplyConcerns(IEnumerable`1 steps, Object instance) at Castle.MicroKernel.ComponentActivator.AbstractComponentActivator.Destroy(Object instance) at Castle.MicroKernel.Lifestyle.AbstractLifestyleManager.Release(Object instance) at Castle.MicroKernel.Burden.Release() at Castle.MicroKernel.Releasers.LifecycledComponentsReleasePolicy.Dispose() at Castle.MicroKernel.DefaultKernel.Dispose() at Microsoft.AspNetCore.Hosting.Internal.WebHost.Dispose() at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token, String shutdownMessage) at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token) at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host) at MyApp.Web.Startup.Program.Main(String[] args) in C:\code\MyApp\aspnet-core\src\MyApp.Web.Host\Startup\Program.cs:line 11

Faulting application name: MyApp.Web.Host.exe, version: 0.0.0.0, time stamp: 0x5c414d46 Faulting module name: KERNELBASE.dll, version: 6.3.9600.19425, time stamp: 0x5d26b6e9 Exception code: 0xe0434352 Fault offset: 0x000000000000908c Faulting process id: 0x17e0 Faulting application start time: 0x01d56396ebf75622

Can you please tell me where exactly in Startup.cs I should add the code posted by @Alper ? Are you referring to Startup.cs in web.host?

alper created 4 years ago

Support Team

If you're using Angular, you can put that code in Startup.cs of Web.Host project.
If you're using MVC, you can put it in Startup.cs of Web.MVC project

See the updated version:


//CATCH UNHANDLED EXCEPTIONS
AppDomain.CurrentDomain.UnhandledException += delegate (object sender, System.UnhandledExceptionEventArgs args)
{
	if (args == null)
	{
		return;
	}

	if (args.ExceptionObject is Exception unhandledEx)
	{
		System.IO.File.WriteAllText(@"C:\logs\unhandled-exceptions.txt", unhandledEx.ToString());
	}
};

//CATCH UNOBSERVED TASK EXCEPTIONS
System.Threading.Tasks.TaskScheduler.UnobservedTaskException += delegate (object sender, System.Threading.Tasks.UnobservedTaskExceptionEventArgs args)
{
	if (args == null)
	{
		return;
	}

	var unobservedTaskEx = args.Exception;
	if (unobservedTaskEx != null)
	{
		System.IO.File.WriteAllText(@"C:\logs\unobserved-exceptions.txt", unobservedTaskEx.ToString());
	}

	args.SetObserved();
};

Create C:\logs folder manually, in case File.WriteAllText doesn't create it.

1

kasem created 4 years ago

Thanks a lot Alper! I added your code to Startup(IHostingEnvironment env) method and deployed to production. I've created Logs folder in c:\ with IUSR permissions. I'm now waiting the next crash to check logs!
0

kasem created 4 years ago

Hi!

I got the expected crash again. Unfortunately I got in c:\logs exactly same message I got in windows event viewer. There's nothing that helps me idenfity source of problem!

The weird thing that I get logs only when I restart IIS. System crashed and was stucked for hours without any logs in c:\logs nor in event viewer.

Your urgent assistance is needed please since this problem has been negatively impacting the user experience of my users in production!
0

alper created 4 years ago
Support Team

observe the audit logs just before the exception happens. which application service was called before this issue. or if you have background jobs check them.

I guess it's related with one of your queries. Is there any query that you're sending to view without ToList function. Or are you sending an entity in your application services?
0
kasem created 4 years ago
I researched logs already and I think I'm able to locate the 1-2 services that produce that exception. I already optimized my queries to avoid executing a query while iterating over the results from another query. Also, I stopped all background workers and jobs. All the mentioned didn't help yet and I still need to find the exact place so I better know how to fix it.

I'm still aiming for two things:

Background jobs and workers are handled through Hangfire. I temporarily stopped them but I still need them so I wonder if I can follow your Migrator project to do something similar for an isolated service running my background workers independently. Problem in those workers fails this particular service without affecting the whole application.

Gather more details about the unhandled exception. Wonder if I placed your code in the right place Startup(IHostingEnvironment env) ? please let me know how I can fix this

Thanks

alper created 4 years ago

Support Team

I think you've put my catcher code in the right place, because it logs an exception. one thing that you can do is; add logs to the beginning and the ending of your app service methods. like

public class MyAppService
{
    public async Task GetReportAsync()
    {
            Logger.Debug("MyAppService - GetReportAsync - Start");
            //do work...
            Logger.Debug("MyAppService - GetReportAsync - End");
      }
}

Have an answer to this question? Log in and write your answer.