Base solution for your next web application
Open Closed

unwanted system restart? #7788


User avatar
0
BobIngham created

HI guys, Last night my Azure CPU pegged high for no apparent reason at 03:32: There is nothing to suggest any errors in any logs but it appears that Zero decided to restart for some reason, here is my log:

INFO  2019-10-04 02:31:51,228 [178  ] ore.Mvc.Internal.ControllerActionInvoker - Executed action Nuagecare.NcForm.NcFormAppService.SubmitForms (Nuagecare.Application) in 283.3617ms
INFO  2019-10-04 02:31:51,228 [178  ] ft.AspNetCore.Routing.EndpointMiddleware - Executed endpoint 'Nuagecare.NcForm.NcFormAppService.SubmitForms (Nuagecare.Application)'
INFO  2019-10-04 02:31:51,228 [178  ] soft.AspNetCore.Hosting.Internal.WebHost - Request finished in 300.4157ms 200 application/json; charset=utf-8
**DEBUG 2019-10-04 02:32:47,023 [1    ] Abp.Modules.AbpModuleManager             - Loading Abp modules...**
DEBUG 2019-10-04 02:32:47,289 [1    ] Abp.Modules.AbpModuleManager             - Found 24 ABP modules in total.
DEBUG 2019-10-04 02:32:47,320 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Nuagecare.Web.Startup.NuagecareWebHostModule, Nuagecare.Web.Host, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,320 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Nuagecare.Web.NuagecareWebCoreModule, Nuagecare.Web.Core, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,320 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Nuagecare.NuagecareApplicationModule, Nuagecare.Application, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Nuagecare.NuagecareCoreModule, Nuagecare.Core, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.Zero.AbpZeroCoreModule, Abp.ZeroCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.Zero.AbpZeroCommonModule, Abp.Zero.Common, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.AbpKernelModule, Abp, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.Zero.Ldap.AbpZeroLdapModule, Abp.Zero.Ldap, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.AutoMapper.AbpAutoMapperModule, Abp.AutoMapper, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.AspNetZeroCore.AbpAspNetZeroCoreModule, Abp.AspNetZeroCore, Version=1.2.2.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.MailKit.AbpMailKitModule, Abp.MailKit, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Nuagecare.EntityFrameworkCore.NuagecareEntityFrameworkCoreModule, Nuagecare.EntityFrameworkCore, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.Zero.EntityFrameworkCore.AbpZeroCoreEntityFrameworkCoreModule, Abp.ZeroCore.EntityFrameworkCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.EntityFrameworkCore.AbpEntityFrameworkCoreModule, Abp.EntityFrameworkCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.EntityFramework.AbpEntityFrameworkCommonModule, Abp.EntityFramework.Common, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.IdentityServer4.AbpZeroCoreIdentityServerEntityFrameworkCoreModule, Abp.ZeroCore.IdentityServer4.EntityFrameworkCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.IdentityServer4.AbpZeroCoreIdentityServerModule, Abp.ZeroCore.IdentityServer4, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.AspNetZeroCore.Web.AbpAspNetZeroCoreWebModule, Abp.AspNetZeroCore.Web, Version=1.2.2.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.AspNetCore.AbpAspNetCoreModule, Abp.AspNetCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.Web.AbpWebCommonModule, Abp.Web.Common, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,382 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.AspNetCore.SignalR.AbpAspNetCoreSignalRModule, Abp.AspNetCore.SignalR, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,382 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Nuagecare.Startup.NuagecareGraphQLModule, Nuagecare.GraphQL, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,382 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.Runtime.Caching.Redis.AbpRedisCacheModule, Abp.RedisCache, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,382 [1    ] Abp.Modules.AbpModuleManager             - Loaded module: Abp.Hangfire.AbpHangfireAspNetCoreModule, Abp.HangFire.AspNetCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,398 [1    ] Abp.Modules.AbpModuleManager             - 24 modules loaded.
DEBUG 2019-10-04 02:32:47,539 [1    ] o.Configuration.LanguageManagementConfig - Converted Abp (Abp.Localization.Dictionaries.DictionaryBasedLocalizationSource) to MultiTenantLocalizationSource
DEBUG 2019-10-04 02:32:47,539 [1    ] o.Configuration.LanguageManagementConfig - Converted AbpZero (Abp.Localization.Dictionaries.DictionaryBasedLocalizationSource) to MultiTenantLocalizationSource
DEBUG 2019-10-04 02:32:47,539 [1    ] o.Configuration.LanguageManagementConfig - Converted Nuagecare (Abp.Localization.Dictionaries.DictionaryBasedLocalizationSource) to MultiTenantLocalizationSource
DEBUG 2019-10-04 02:32:47,554 [1    ] o.Configuration.LanguageManagementConfig - Converted AbpWeb (Abp.Localization.Dictionaries.DictionaryBasedLocalizationSource) to MultiTenantLocalizationSource
DEBUG 2019-10-04 02:32:48,539 [1    ] ameworkCore.AbpEntityFrameworkCoreModule - Registering DbContext: Nuagecare.EntityFrameworkCore.NuagecareDbContext, Nuagecare.EntityFrameworkCore, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:50,179 [1    ] Abp.Localization.LocalizationManager     - Initializing 4 localization sources.
DEBUG 2019-10-04 02:32:50,539 [1    ] Abp.Localization.LocalizationManager     - Initialized localization source: Abp
DEBUG 2019-10-04 02:32:50,742 [1    ] Abp.Localization.LocalizationManager     - Initialized localization source: AbpZero
DEBUG 2019-10-04 02:32:50,820 [1    ] Abp.Localization.LocalizationManager     - Initialized localization source: Nuagecare
DEBUG 2019-10-04 02:32:50,836 [1    ] Abp.Localization.LocalizationManager     - Initialized localization source: AbpWeb
DEBUG 2019-10-04 02:32:50,851 [1    ] bp.Hangfire.HangfireBackgroundJobManager - Start background worker: Abp.Hangfire.HangfireBackgroundJobManager
DEBUG 2019-10-04 02:32:50,945 [1    ] Abp.AutoMapper.AbpAutoMapperModule       - Found 6 classes define auto mapping attributes
DEBUG 2019-10-04 02:32:50,945 [1    ] Abp.AutoMapper.AbpAutoMapperModule       - Nuagecare.Web.Models.TokenAuth.ExternalLoginProviderInfoModel
DEBUG 2019-10-04 02:32:50,945 [1    ] Abp.AutoMapper.AbpAutoMapperModule       - Nuagecare.Friendships.Cache.FriendCacheItem
DEBUG 2019-10-04 02:32:50,961 [1    ] Abp.AutoMapper.AbpAutoMapperModule       - Nuagecare.Dto.OrganizationUnitDto
DEBUG 2019-10-04 02:32:50,961 [1    ] Abp.AutoMapper.AbpAutoMapperModule       - Nuagecare.Dto.RoleDto
DEBUG 2019-10-04 02:32:50,961 [1    ] Abp.AutoMapper.AbpAutoMapperModule       - Nuagecare.Dto.UserDto+RoleDto
DEBUG 2019-10-04 02:32:50,961 [1    ] Abp.AutoMapper.AbpAutoMapperModule       - Nuagecare.Dto.UserDto+OrganizationUnitDto
DEBUG 2019-10-04 02:33:00,507 [1    ] enancy.SubscriptionExpirationCheckWorker - Start background worker: Nuagecare.MultiTenancy.SubscriptionExpirationCheckWorker
DEBUG 2019-10-04 02:33:00,507 [1    ] cy.SubscriptionExpireEmailNotifierWorker - Start background worker: Nuagecare.MultiTenancy.SubscriptionExpireEmailNotifierWorker
INFO  2019-10-04 02:33:01,179 [1    ] fire.SqlServer.SqlServerObjectsInstaller - Start installing Hangfire SQL objects...
INFO  2019-10-04 02:33:01,289 [1    ] fire.SqlServer.SqlServerObjectsInstaller - Hangfire SQL objects installed.
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             - Starting Hangfire Server
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             - Using job storage: 'SQL Server: tcp:nuagecare-sqlserver.database.windows.net,1433@Nuagecare'
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             - Using the following options for SQL Server job storage:
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             -     Queue poll interval: 00:00:15.
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             - Using the following options for Hangfire Server:
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             -     Worker count: 10
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             -     Listening queues: 'default'
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             -     Shutdown timeout: 00:00:15
INFO  2019-10-04 02:33:01,695 [1    ] Hangfire.BackgroundJobServer             -     Schedule polling interval: 00:00:15
FATAL 2019-10-04 02:33:03,679 [1    ] soft.AspNetCore.Hosting.Internal.WebHost - Hosting startup assembly exception
System.InvalidOperationException: Startup assembly StartupBootstrapper failed to execute. See the inner exception for more details. ---> System.IO.FileNotFoundException: Could not load file or assembly 'StartupBootstrapper' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   --- End of inner exception stack trace ---
FATAL 2019-10-04 02:33:03,695 [1    ] soft.AspNetCore.Hosting.Internal.WebHost - Hosting startup assembly exception
System.InvalidOperationException: Startup assembly DiagnosticServices.HostingStartup failed to execute. See the inner exception for more details. ---> System.IO.FileNotFoundException: Could not load file or assembly 'DiagnosticServices.HostingStartup' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   --- End of inner exception stack trace ---
FATAL 2019-10-04 02:33:03,695 [1    ] soft.AspNetCore.Hosting.Internal.WebHost - Hosting startup assembly exception
System.InvalidOperationException: Startup assembly Microsoft.AspNetCore.AzureAppServices.HostingStartup failed to execute. See the inner exception for more details. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.AzureAppServices.HostingStartup' or one of its dependencies. The system cannot find the file specified.
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   --- End of inner exception stack trace ---

Note the three FATAL messages which we have been told to ignore because "Zero continues to operate normally so just ignore these messages". Hardly professional but hey-ho there we go. My question is why would this happen? I would attach my log file but there's no way to do it in this forum.


11 Answer(s)
  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @bobingham

    Note the three FATAL messages which we have been told to ignore because "Zero continues to operate normally so just ignore these messages". Hardly professional but hey-ho there we go.

    Sorry I couldn't understand this. Who told this ?

    This seems like a problem related to Azure. Have you contacted with them ?

  • User Avatar
    0
    BobIngham created

    Hi @ismcagdas, I am currently working with datadog to try find the root cause, I am up to my ears in logs! I have UK Fast (the UK's largest Azure consultancy) also trying to track the problem. Too early to say whether it's my code or Azure, I'm sure it's not Zero. As for

    Note the three FATAL messages which we have been told to ignore because "Zero continues to operate normally so just ignore these messages". Hardly professional but hey-ho there we go.

    See Microsoft.AspNetCore.AzureAppServices.HostingStartup problem. Comments welcome.

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @bobingham

    This might be related to application insights (related issue https://github.com/aspnet/AspNetCore/issues/14547). I didn't want to blame Azure :), the logs you have shared seems to be related to Azure.

    If you can get more error log information, we can dig deeper.

  • User Avatar
    0
    BobIngham created

    Hi @ismcagdas, I may be taking you up on that offer to dig deeper.

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @bobingham

    Do you have any extra logs or is it same with the one in your original post ?

  • User Avatar
    0
    BobIngham created

    Hi @ismcagdas, The system diagnostic setting to collect the dump was implemented yesterday. The issue only occurs every few days so as soon as I have a dump I will analyse and if I think it's a Zero problem I will forward the dump file.

  • User Avatar
    0
    BobIngham created

    Hi @ismcagdas,

    See the problem I had at 22:23 on 22nd November (Saturday night). I have a full system dump taken from system diagnostics. You said you would like to take a look. Where do I send them to?

    Cheers, Bob

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @bobingham

    We got your email. We will reply to you via email first.

    Thanks

  • User Avatar
    0
    BobIngham created

    Thanks.

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @bobingham, We can continue on https://github.com/aspnetzero/aspnet-zero-core/issues/2796. Latest version of ABP might help to solve this issue.

    What do you think ?

  • User Avatar
    1
    BobIngham created

    Hi @ismcagdas, yep I have no problem in continuing over there. Makes sense to close the issue and keep the forum tidy.