Base solution for your next web application
Open Closed

Azure - Problem solving why API went down and needed to be restarted #8200

User avatar
0
jtallon created

Hi, this morning our app went down on production and needed to be restarted from Azure. Looking at the logs the last thing to happen was:

DEBUG 2019-12-14 02:59:18,666 [4    ] Quartz.Core.QuartzSchedulerThread        - Batch acquisition of 0 triggers
DEBUG 2019-12-14 02:59:46,115 [4    ] Quartz.Core.QuartzSchedulerThread        - Batch acquisition of 1 triggers
DEBUG 2019-12-14 03:00:00,032 [4    ] Quartz.Core.QuartzSchedulerThread        - Batch acquisition of 0 triggers
DEBUG 2019-12-14 03:00:00,082 [13   ] Abp.Quartz.AbpQuartzJobListener          - Job TempFilesCleanUpJob executing...
DEBUG 2019-12-14 03:00:00,083 [13   ] Quartz.Core.JobRunShell                  - Calling Execute on job FileJobs.TemporaryFilesCleanUpJob
INFO  2019-12-14 03:00:00,085 [13   ] work.Portal.FileJobs.TempFilesCleanUpJob - Executing TemporaryFilesCleanUpJob
DEBUG 2019-12-14 03:00:06,238 [31   ] Abp.Quartz.AbpQuartzJobListener          - Job TempFilesCleanUpJob sucessfully executed.
DEBUG 2019-12-14 03:00:06,334 [31   ] Quartz.Core.JobRunShell                  - Trigger instruction : NoInstruction
DEBUG 2019-12-**14** 03:00:26,903 [4    ] Quartz.Core.QuartzSchedulerThread        - Batch acquisition of 0 triggers
DEBUG 2019-12-**16** 07:01:48,206 [1    ] Abp.Modules.AbpModuleManager             - Loading Abp modules...

To me it looks like it completed our job "TempFilesCleanUpJob" successfully and then 20 seconds later it looks for another trigger and then everything shuts down. No Fatal, No errors.

Looking at the Azure app insights it shows: "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full" - this seems to happen 250 times then nothing else happens after that.

A sample call stack of an exception is

_System.Net.Http.HttpRequestException:   at System.Net.Http.ConnectHelper+<ConnectAsync>d2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.ValueTask`1.getResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<CreateConnectionAsync>d44.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.ValueTask`1.getResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<WaitForCreatedConnectionAsync>d49.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.ValueTask`1.getResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d39.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.RedirectHandler+&lt;SendAsync&gt;d4.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.DiagnosticsHandler+<SendAsync>d2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Http.ConnectHelper+<ConnectAsync>d2.MoveNext:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.ConnectHelper+<ConnectAsync>d2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)__
Go to accepted answer
2 Answer(s)
  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @jtallon

    Is it possible to shar eyour TempFilesCleanUpJob privately ? You can sen it to [email protected].

  • User Avatar
    0
    ismcagdas created
    Support Team

    This issue is closed because it has not had recent activity for a long time.

Assignee
No one
Labels
Non yet