Base solution for your next web application
Open Closed

NotificationDistributionJob failing #11235

User avatar
0
rickfrankel created

Prerequisites

  • What is your product version? 11.2.1

  • What is your product type (Angular or MVC)? Angular

  • What is product framework type (.net framework or .net core)?

.Net Core

  • What is ABP Framework version? 7.3.0

Since the last few days I've started noticing that the NotificationDistributionJob is not completeing. I'm running Hangfire for the background processors and have found that they are timing out. I'm also running Azure Signal R and noticing these error messages appearing in my logs.

Microsoft.Azure.SignalR.ServiceConnection - The connection H28v6o2EbbQ7jtNpaY2W5g7f285e9c1 has a long running application logic that prevents the connection from complete. Microsoft.Azure.SignalR.ServiceConnection - The connection pj_2095KeEu-AkzJXqgBegdac83bff1 has a long running application logic that prevents the connection from complete.

Its not super consistent as to what happens. Sometimes the job finishes in a few seconds. Othertimes it might finish in 30 seconds. But more often than not at the moment it is timing out and failing after

Eventually this is the error I get in the logs. (Yes I'm using MySQL as my DB. I've rebooted everything. Including all app services, DB's, Redis, and SignalR). I've looked over all recent commits and nothing in this space.

DB load is almost zero. The number of users the notifications are being distributed to is less than 5. I cleared the AbpNotifications table. The AbpTenantNotifications and AbpUserNotifications tables aren't huge. I'm not sure what I'm missing here.

Thanks Rick

System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException (104): Connection reset by peer at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken) at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory1 buffer, CancellationToken cancellationToken) at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory1 buffer) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory1 buffer) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory1 buffer, CancellationToken cancellationToken) at MySqlConnector.Protocol.Serialization.StreamByteHandler.<WriteBytesAsync>g__DoWriteBytesAsync|7_0(ReadOnlyMemory1 data) in /_/src/MySqlConnector/Protocol/Serialization/StreamByteHandler.cs:line 105 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at MySqlConnector.Protocol.Serialization.StreamByteHandler.<WriteBytesAsync>g__DoWriteBytesAsync|7_0(ReadOnlyMemory1 data) at MySqlConnector.Protocol.Serialization.StreamByteHandler.WriteBytesAsync(ReadOnlyMemory1 data, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/StreamByteHandler.cs:line 90 at MySqlConnector.Protocol.Serialization.ProtocolUtility.WritePacketAsync(IByteHandler byteHandler, Int32 sequenceNumber, ReadOnlyMemory1 contents, IOBehavior ioBehavior) in //src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 533 at MySqlConnector.Protocol.Serialization.StandardPayloadHandler.WritePayloadAsync(ReadOnlyMemory1 payload, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/StandardPayloadHandler.cs:line 44 at MySqlConnector.Core.ServerSession.SendReplyAsync(PayloadData payload, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 965 at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(IReadOnlyList1 commands, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/Core/CommandExecutor.cs:line 52 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in //src/MySqlConnector/MySqlCommand.cs:line 282 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken) in //src/MySqlConnector/MySqlCommand.cs:line 275 at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.CreateSavepointAsync(String name, CancellationToken cancellationToken) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.CreateSavepointAsync(String name, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChangesAsync(IList1 entries, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList1 entriesToSave, CancellationToken cancellationToken) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.<>c.<SaveChangesAsync>b__106_0(DbContext _, ValueTuple2 t, CancellationToken cancellationToken) at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Abp.EntityFrameworkCore.AbpDbContext.SaveChangesAsync(CancellationToken cancellationToken) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Abp.EntityFrameworkCore.AbpDbContext.SaveChangesAsync(CancellationToken cancellationToken) at Abp.Zero.EntityFrameworkCore.AbpZeroCommonDbContext3.SaveChangesAsync(CancellationToken cancellationToken) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Abp.Zero.EntityFrameworkCore.AbpZeroCommonDbContext3.SaveChangesAsync(CancellationToken cancellationToken) at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.SaveChangesInDbContextAsync(DbContext dbContext) at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.SaveChangesAsync() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.SaveChangesAsync() at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.CompleteUowAsync() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.CompleteUowAsync() at Abp.Domain.Uow.UnitOfWorkBase.CompleteAsync() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at Abp.Domain.Uow.UnitOfWorkBase.CompleteAsync() at Abp.Domain.Uow.UnitOfWorkManagerExtensions.WithUnitOfWorkAsync(IUnitOfWorkManager manager, Func1 action, UnitOfWorkOptions options) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread) at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetExistingTaskResult(Task1 task, TResult result) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() at Abp.Notifications.DefaultNotificationDistributer.<>c__DisplayClass7_0.<<DistributeAsync>b__0>d.MoveNext() at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetExistingTaskResult(Task1 task, TResult result) at Abp.Notifications.DefaultNotificationDistributer.SaveUserNotificationsAsync(UserIdentifier[] users, NotificationInfo notificationInfo) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetExistingTaskResult(Task1 task, TResult result) at Abp.Domain.Uow.UnitOfWorkManagerExtensions.WithUnitOfWorkAsync[TResult](IUnitOfWorkManager manager, Func1 action, UnitOfWorkOptions options) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread) at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) at System.Threading.Tasks.Task1.TrySetResult(TResult result) at Abp.Notifications.DefaultNotificationDistributer.<>c__DisplayClass11_0.<<SaveUserNotificationsAsync>b__0>d.MoveNext() at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetExistingTaskResult(Task1 task, TResult result) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() at Abp.Notifications.NotificationStore.InsertUserNotificationAsync(UserNotificationInfo userNotification) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

Go to accepted answer
4 Answer(s)
  • User Avatar
    0
    rickfrankel created

    Ok. I think I've narrowed it down a bit.

    The issue appears to come from me implementing this solution. https://support.aspnetzero.com/QA/Questions/11027/Azure-Signalr-Service-issue-with-Azure-App-Service-when-the-App-service-is-scaled-out#answer-b7792b56-a93d-8ac9-099a-3a033b15dc28

    I had many thousands or rows in the online client cache which I believe is the cause of the issue. It wasn't optimal in trying to find out who was connected and must have been timing out.

    As a temporary solution clearing that table and letting all clients repopulate has worked. Clearly this table is not removing entries when they are disconnected.

    Will look into a better solution going forward.

    Is there any official solution to this from AspNetZero on the OnlineClientCache and making it work in a load balanced environment.

    Thanks

  • User Avatar
    0
    rickfrankel created

    For anyone who gets here. Don't follow the links above. Note that a proper solution has been done using Redis.

    https://github.com/aspnetboilerplate/aspnetboilerplate/commit/b58d50ad5796da2cf1bf060fe791d33652700ba9

    Doesn't look like its in the ABP 7.3 thats the current release I can see, but hoping it will be very shortly in the next one.

  • User Avatar
    0
    sedulen created

    Hi @rickfrankel ,

    I realize that I'm very late in responding to this topic, but I was curious if your solution using the ABP RedisOnlineClientStore was working for you.

    While I was working on the initial AzureTablesOnlineClientStore implementation, I had also written my own RedisOnlineClientStore implementation but ran into challenges with Network I/O and concurrent request load in my Azure Redis cache, which is why I decided to move towards Azure Tables instead. Have you found any network I/O or load issues connecting to your Redis cache using the RedisOnlineClientStore ?

    I do recognize that in my AzureTablesOnlineClientStore implementation, I did not add a timestamp or any mechanism to clean out the table for old or stale connections.

    If the ABP RedisOnlineClientStore implementation works for you, that's awesome. I'll have to look into switching to that store instead.

    Cheers and Happy Coding! -Brian

  • User Avatar
    0
    rickfrankel created

    Yeah the RedisOnlineClientStore from ABP works well for us. Since we've switched over to it we've had no issues or problems at all.

    1
Assignee
No one
Labels
None yet