0
BobIngham created
HI guys,
Last night my Azure CPU pegged high for no apparent reason at 03:32:
There is nothing to suggest any errors in any logs but it appears that Zero decided to restart for some reason, here is my log:
INFO 2019-10-04 02:31:51,228 [178 ] ore.Mvc.Internal.ControllerActionInvoker - Executed action Nuagecare.NcForm.NcFormAppService.SubmitForms (Nuagecare.Application) in 283.3617ms
INFO 2019-10-04 02:31:51,228 [178 ] ft.AspNetCore.Routing.EndpointMiddleware - Executed endpoint 'Nuagecare.NcForm.NcFormAppService.SubmitForms (Nuagecare.Application)'
INFO 2019-10-04 02:31:51,228 [178 ] soft.AspNetCore.Hosting.Internal.WebHost - Request finished in 300.4157ms 200 application/json; charset=utf-8
**DEBUG 2019-10-04 02:32:47,023 [1 ] Abp.Modules.AbpModuleManager - Loading Abp modules...**
DEBUG 2019-10-04 02:32:47,289 [1 ] Abp.Modules.AbpModuleManager - Found 24 ABP modules in total.
DEBUG 2019-10-04 02:32:47,320 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Nuagecare.Web.Startup.NuagecareWebHostModule, Nuagecare.Web.Host, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,320 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Nuagecare.Web.NuagecareWebCoreModule, Nuagecare.Web.Core, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,320 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Nuagecare.NuagecareApplicationModule, Nuagecare.Application, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Nuagecare.NuagecareCoreModule, Nuagecare.Core, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.Zero.AbpZeroCoreModule, Abp.ZeroCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.Zero.AbpZeroCommonModule, Abp.Zero.Common, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.AbpKernelModule, Abp, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.Zero.Ldap.AbpZeroLdapModule, Abp.Zero.Ldap, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.AutoMapper.AbpAutoMapperModule, Abp.AutoMapper, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.AspNetZeroCore.AbpAspNetZeroCoreModule, Abp.AspNetZeroCore, Version=1.2.2.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.MailKit.AbpMailKitModule, Abp.MailKit, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Nuagecare.EntityFrameworkCore.NuagecareEntityFrameworkCoreModule, Nuagecare.EntityFrameworkCore, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.Zero.EntityFrameworkCore.AbpZeroCoreEntityFrameworkCoreModule, Abp.ZeroCore.EntityFrameworkCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,336 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.EntityFrameworkCore.AbpEntityFrameworkCoreModule, Abp.EntityFrameworkCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.EntityFramework.AbpEntityFrameworkCommonModule, Abp.EntityFramework.Common, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.IdentityServer4.AbpZeroCoreIdentityServerEntityFrameworkCoreModule, Abp.ZeroCore.IdentityServer4.EntityFrameworkCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.IdentityServer4.AbpZeroCoreIdentityServerModule, Abp.ZeroCore.IdentityServer4, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.AspNetZeroCore.Web.AbpAspNetZeroCoreWebModule, Abp.AspNetZeroCore.Web, Version=1.2.2.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.AspNetCore.AbpAspNetCoreModule, Abp.AspNetCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,351 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.Web.AbpWebCommonModule, Abp.Web.Common, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,382 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.AspNetCore.SignalR.AbpAspNetCoreSignalRModule, Abp.AspNetCore.SignalR, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,382 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Nuagecare.Startup.NuagecareGraphQLModule, Nuagecare.GraphQL, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,382 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.Runtime.Caching.Redis.AbpRedisCacheModule, Abp.RedisCache, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,382 [1 ] Abp.Modules.AbpModuleManager - Loaded module: Abp.Hangfire.AbpHangfireAspNetCoreModule, Abp.HangFire.AspNetCore, Version=4.4.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:47,398 [1 ] Abp.Modules.AbpModuleManager - 24 modules loaded.
DEBUG 2019-10-04 02:32:47,539 [1 ] o.Configuration.LanguageManagementConfig - Converted Abp (Abp.Localization.Dictionaries.DictionaryBasedLocalizationSource) to MultiTenantLocalizationSource
DEBUG 2019-10-04 02:32:47,539 [1 ] o.Configuration.LanguageManagementConfig - Converted AbpZero (Abp.Localization.Dictionaries.DictionaryBasedLocalizationSource) to MultiTenantLocalizationSource
DEBUG 2019-10-04 02:32:47,539 [1 ] o.Configuration.LanguageManagementConfig - Converted Nuagecare (Abp.Localization.Dictionaries.DictionaryBasedLocalizationSource) to MultiTenantLocalizationSource
DEBUG 2019-10-04 02:32:47,554 [1 ] o.Configuration.LanguageManagementConfig - Converted AbpWeb (Abp.Localization.Dictionaries.DictionaryBasedLocalizationSource) to MultiTenantLocalizationSource
DEBUG 2019-10-04 02:32:48,539 [1 ] ameworkCore.AbpEntityFrameworkCoreModule - Registering DbContext: Nuagecare.EntityFrameworkCore.NuagecareDbContext, Nuagecare.EntityFrameworkCore, Version=6.5.0.0, Culture=neutral, PublicKeyToken=null
DEBUG 2019-10-04 02:32:50,179 [1 ] Abp.Localization.LocalizationManager - Initializing 4 localization sources.
DEBUG 2019-10-04 02:32:50,539 [1 ] Abp.Localization.LocalizationManager - Initialized localization source: Abp
DEBUG 2019-10-04 02:32:50,742 [1 ] Abp.Localization.LocalizationManager - Initialized localization source: AbpZero
DEBUG 2019-10-04 02:32:50,820 [1 ] Abp.Localization.LocalizationManager - Initialized localization source: Nuagecare
DEBUG 2019-10-04 02:32:50,836 [1 ] Abp.Localization.LocalizationManager - Initialized localization source: AbpWeb
DEBUG 2019-10-04 02:32:50,851 [1 ] bp.Hangfire.HangfireBackgroundJobManager - Start background worker: Abp.Hangfire.HangfireBackgroundJobManager
DEBUG 2019-10-04 02:32:50,945 [1 ] Abp.AutoMapper.AbpAutoMapperModule - Found 6 classes define auto mapping attributes
DEBUG 2019-10-04 02:32:50,945 [1 ] Abp.AutoMapper.AbpAutoMapperModule - Nuagecare.Web.Models.TokenAuth.ExternalLoginProviderInfoModel
DEBUG 2019-10-04 02:32:50,945 [1 ] Abp.AutoMapper.AbpAutoMapperModule - Nuagecare.Friendships.Cache.FriendCacheItem
DEBUG 2019-10-04 02:32:50,961 [1 ] Abp.AutoMapper.AbpAutoMapperModule - Nuagecare.Dto.OrganizationUnitDto
DEBUG 2019-10-04 02:32:50,961 [1 ] Abp.AutoMapper.AbpAutoMapperModule - Nuagecare.Dto.RoleDto
DEBUG 2019-10-04 02:32:50,961 [1 ] Abp.AutoMapper.AbpAutoMapperModule - Nuagecare.Dto.UserDto+RoleDto
DEBUG 2019-10-04 02:32:50,961 [1 ] Abp.AutoMapper.AbpAutoMapperModule - Nuagecare.Dto.UserDto+OrganizationUnitDto
DEBUG 2019-10-04 02:33:00,507 [1 ] enancy.SubscriptionExpirationCheckWorker - Start background worker: Nuagecare.MultiTenancy.SubscriptionExpirationCheckWorker
DEBUG 2019-10-04 02:33:00,507 [1 ] cy.SubscriptionExpireEmailNotifierWorker - Start background worker: Nuagecare.MultiTenancy.SubscriptionExpireEmailNotifierWorker
INFO 2019-10-04 02:33:01,179 [1 ] fire.SqlServer.SqlServerObjectsInstaller - Start installing Hangfire SQL objects...
INFO 2019-10-04 02:33:01,289 [1 ] fire.SqlServer.SqlServerObjectsInstaller - Hangfire SQL objects installed.
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Starting Hangfire Server
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Using job storage: 'SQL Server: tcp:nuagecare-sqlserver.database.windows.net,1433@Nuagecare'
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Using the following options for SQL Server job storage:
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Queue poll interval: 00:00:15.
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Using the following options for Hangfire Server:
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Worker count: 10
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Listening queues: 'default'
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Shutdown timeout: 00:00:15
INFO 2019-10-04 02:33:01,695 [1 ] Hangfire.BackgroundJobServer - Schedule polling interval: 00:00:15
FATAL 2019-10-04 02:33:03,679 [1 ] soft.AspNetCore.Hosting.Internal.WebHost - Hosting startup assembly exception
System.InvalidOperationException: Startup assembly StartupBootstrapper failed to execute. See the inner exception for more details. ---> System.IO.FileNotFoundException: Could not load file or assembly 'StartupBootstrapper' or one of its dependencies. The system cannot find the file specified.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
--- End of inner exception stack trace ---
FATAL 2019-10-04 02:33:03,695 [1 ] soft.AspNetCore.Hosting.Internal.WebHost - Hosting startup assembly exception
System.InvalidOperationException: Startup assembly DiagnosticServices.HostingStartup failed to execute. See the inner exception for more details. ---> System.IO.FileNotFoundException: Could not load file or assembly 'DiagnosticServices.HostingStartup' or one of its dependencies. The system cannot find the file specified.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
--- End of inner exception stack trace ---
FATAL 2019-10-04 02:33:03,695 [1 ] soft.AspNetCore.Hosting.Internal.WebHost - Hosting startup assembly exception
System.InvalidOperationException: Startup assembly Microsoft.AspNetCore.AzureAppServices.HostingStartup failed to execute. See the inner exception for more details. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.AzureAppServices.HostingStartup' or one of its dependencies. The system cannot find the file specified.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
--- End of inner exception stack trace ---
Note the three FATAL messages which we have been told to ignore because "Zero continues to operate normally so just ignore these messages". Hardly professional but hey-ho there we go. My question is why would this happen? I would attach my log file but there's no way to do it in this forum.
11 Answer(s)
-
0
Hi @bobingham
Note the three FATAL messages which we have been told to ignore because "Zero continues to operate normally so just ignore these messages". Hardly professional but hey-ho there we go.
Sorry I couldn't understand this. Who told this ?
This seems like a problem related to Azure. Have you contacted with them ?
-
0
Hi @ismcagdas, I am currently working with datadog to try find the root cause, I am up to my ears in logs! I have UK Fast (the UK's largest Azure consultancy) also trying to track the problem. Too early to say whether it's my code or Azure, I'm sure it's not Zero. As for
Note the three FATAL messages which we have been told to ignore because "Zero continues to operate normally so just ignore these messages". Hardly professional but hey-ho there we go.
See Microsoft.AspNetCore.AzureAppServices.HostingStartup problem. Comments welcome.
-
0
Hi @bobingham
This might be related to application insights (related issue https://github.com/aspnet/AspNetCore/issues/14547). I didn't want to blame Azure :), the logs you have shared seems to be related to Azure.
If you can get more error log information, we can dig deeper.
-
0
Hi @ismcagdas, I may be taking you up on that offer to dig deeper.
-
0
Hi @bobingham
Do you have any extra logs or is it same with the one in your original post ?
-
0
Hi @ismcagdas, The system diagnostic setting to collect the dump was implemented yesterday. The issue only occurs every few days so as soon as I have a dump I will analyse and if I think it's a Zero problem I will forward the dump file.
-
0
Hi @ismcagdas,
See the problem I had at 22:23 on 22nd November (Saturday night).
I have a full system dump taken from system diagnostics.
You said you would like to take a look.
Where do I send them to?Cheers, Bob
-
0
Hi @bobingham
We got your email. We will reply to you via email first.
Thanks
-
0
Thanks.
-
0
Hi @bobingham, We can continue on https://github.com/aspnetzero/aspnet-zero-core/issues/2796. Latest version of ABP might help to solve this issue.
What do you think ?
-
1
Hi @ismcagdas, yep I have no problem in continuing over there. Makes sense to close the issue and keep the forum tidy.
