Hello. We use aspnet zero version 11.4.0 right now. Our application is deployed to azure. So we have Redis Cache (6gb memory), Azure Signalr (2x instance), Backend App Service(Single instance), FrontEnd App Service(Single), Azure Functions App Service(Single), Azure Blob and Sql Server
When it is under load, sometimes we get a timeout from redis server. I think this problem is related with Signalr. here is the stack trace.
ERROR 2022-12-15 16:07:46,533 [149 ] .AspNetCore.SignalR.HubConnectionHandler - Error when dispatching 'OnConnectedAsync' on hub. StackExchange.Redis.RedisTimeoutException: Timeout performing HMSET (5000ms), inst: 0, qu: 75, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: dc-redis-live.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: WN0LDWK00003O, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=13,Free=1010,Min=8,Max=1023), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor
1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2884 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.HashSet(RedisKey key, HashEntry[] hashFields, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 445 at Abp.Runtime.Caching.Redis.RealTime.RedisOnlineClientStore.Add(IOnlineClient client) at Abp.RealTime.OnlineClientManager.Add(IOnlineClient client) at Abp.AspNetCore.SignalR.Hubs.OnlineClientHubBase.OnConnectedAsync() at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.HubConnectionHandler1.RunHubAsync(HubConnectionContext connection) ERROR 2022-12-15 16:07:46,704 [98 ] .AspNetCore.SignalR.HubConnectionHandler - Error when dispatching 'OnConnectedAsync' on hub. StackExchange.Redis.RedisTimeoutException: Timeout performing HMSET (5000ms), inst: 0, qu: 75, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: dc-redis-live.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: WN0LDWK00003O, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=12,Free=1011,Min=8,Max=1023), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2884 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.HashSet(RedisKey key, HashEntry[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 445 at Abp.Runtime.Caching.Redis.RealTime.RedisOnlineClientStore.Add(IOnlineClient client) at Abp.RealTime.OnlineClientManager.Add(IOnlineClient client) at Abp.AspNetCore.SignalR.Hubs.OnlineClientHubBase.OnConnectedAsync() at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.HubConnectionHandler1.RunHubAsync(HubConnectionContext connection) ERROR 2022-12-15 16:07:52,394 [166 ] icrosoft.Azure.SignalR.ServiceConnection - Error processing the connection 7xEIgc7k3ngOq0ATq8Wxmw5a1ef6cd1. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.AggregateException.GetBaseException() at Microsoft.Azure.SignalR.ServiceConnection.ProcessClientConnectionAsync(ClientConnectionContext connection) ERROR 2022-12-15 16:07:52,923 [130 ] .AspNetCore.SignalR.HubConnectionHandler - Error when dispatching 'OnConnectedAsync' on hub. StackExchange.Redis.RedisTimeoutException: Timeout performing HMSET (5000ms), inst: 0, qu: 86, qs: 0, aw: False, bw: Activating, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: dc-redis-live.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: WN0LDWK00003O, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=22,Free=1001,Min=8,Max=1023), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2884 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.HashSet(RedisKey key, HashEntry[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 445 at Abp.Runtime.Caching.Redis.RealTime.RedisOnlineClientStore.Add(IOnlineClient client) at Abp.RealTime.OnlineClientManager.Add(IOnlineClient client) at Abp.AspNetCore.SignalR.Hubs.OnlineClientHubBase.OnConnectedAsync() at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher1.OnConnectedAsync(HubConnectionContext connection) at Microsoft.AspNetCore.SignalR.HubConnectionHandler`1.RunHubAsync(HubConnectionContext connection)
Do you have any idea how this can be fixed? is this related with Abp 7.4 upgrade since i have seen some issues with multiple signalr instances?
5 Answer(s)
-
0
Also i need to say before this exception, i am getting out of memory exception. But i don't see any memory issue on the metrics not in backend or redis cache. Maybe after this exception signalr also goes down since it uses redis?
ERROR 2022-12-15 16:06:08,713 [129 ] Abp.Runtime.Caching.Redis.AbpRedisCache - StackExchange.Redis.RedisConnectionException: Failed to write ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, Char* pThreadName) at System.Threading.Thread.StartCore() at System.Threading.Thread.Start(Object parameter, Boolean captureContext) at StackExchange.Redis.PhysicalBridge.WriteMessageTakingWriteLockAsync(PhysicalConnection physical, Message message) in //src/StackExchange.Redis/PhysicalBridge.cs:line 1012 --- End of inner exception stack trace --- at Abp.Runtime.Caching.Redis.AbpRedisCache.TryGetValueAsync(String key) at Abp.Runtime.Caching.AbpCacheBase
2.GetAsync(TKey key, Func2 factory) StackExchange.Redis.RedisConnectionException: Failed to write ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Threading.Thread.StartInternal(ThreadHandle t, Int32 stackSize, Int32 priority, Char* pThreadName) at System.Threading.Thread.StartCore() at System.Threading.Thread.Start(Object parameter, Boolean captureContext) at StackExchange.Redis.PhysicalBridge.WriteMessageTakingWriteLockAsync(PhysicalConnection physical, Message message) in //src/StackExchange.Redis/PhysicalBridge.cs:line 1012 --- End of inner exception stack trace --- at Abp.Runtime.Caching.Redis.AbpRedisCache.TryGetValueAsync(String key) at Abp.Runtime.Caching.AbpCacheBase2.GetAsync(TKey key, Func2 factory) -
0
Hi,
Do you use https://aspnetboilerplate.com/Pages/Documents/PerRequestRedisCache ? If not, could you try using it and see if it helps ?
-
0
i will try it @ismcagdas, i am not using it.
-
0
Thanks. I will be waiting for your reply.
-
0
Hello @ismcagdas it is difficult to say but we didn't get any exception until now. i will close the issue for now. if sth happens i can open a new thread thank you for your help.
