Hi,
As you know from my previous questions, I'm facing many performance issues while adding new users to the app. I'm dealing with multi instance on Azure, Redis cache, Azure SignalR...
The problem here is that some requests are very long to execute. Please look at following example :
As you can see, the request is blocked for 10s on token validation... On the left side of the screenshot you can see thaat the same request can be very fast. I suspect a lock somewhere in the app because all Azure resources are good (CPU, RAM...).
It is very hard to find the root cause from my side.
Can you please provide some help to find a solution ?
Yes, this table is cleared every day by a background job. If this is what you want to check I confirm that it is working fine on my side. My concern is more about the design, taking into account that a refreshoken as an expiration time of 1 year, it means that this table can grow in a exponential way
Hi @ismcagdas,
I'm currently trying on my dev machine. Everything works great ! I notice a huge performance improvement between PerRequestRedisCache activated and deactivated. I will deploy it on Azure today and give a feedback if something goes wrong. Thanks for your 5* support @ismcagdas :)
Hi @ismcagdas,
Since my first message I decided to delete all records and the app performance is better. Azure was suggesting to create an index for AbpUserTokens table. I will try to add it and see differences.
But the main question is : don't you think that this table could be really big with time ? For example, if you have 500 users that connects every day, this means arround 200k records in that table at the end of one year... something is wrong here.
Hi support team,
I'm facing performance issues on Azure. While investigating requests performances, I noticed that a lot of requests where slow due to token validation. The AbpUserTokens table has arround 50k records which is a lot and very bad for performance. All records are still valid : so that means that the job in charge of deleting expired tokens is working.
I had a look to similar issues in the past : https://support.aspnetzero.com/QA/Questions/9009/Bad-performance-on-Azure-App-Service https://support.aspnetzero.com/QA/Questions/10545/Token-Authenticate-time-out-after-long-time-usage-with-same-user https://github.com/aspnetzero/aspnet-zero-core/issues/2925 https://github.com/aspnetzero/aspnet-zero-core/issues/2926
For you information, my refreshtoken expriration is fixed to one year... the angular web app AND the mobile app are not using it (these apps have been updated from an old Zero version that was not using refresh tokens at this time). Therefore, only few API clients are using it.
Right now, I plan to delete the whole table and get some performance back. What do you think ? Is it bad to do it ?
I plan also to change refresh tokens expiration to one month.
Even so, if I let it like that, I will have the same issue after some time... what is your advice to avoid this issue ? I think that a lot of Abp apps are facing the same.
Hi @ismcagdas, thank you for this. I will try it and provide feedback on all related issues.
Hi @ismcagdas,
I just tried PerRequestRedisCache with a fresh new redis server.
I'm getting same exception as explained here : Unable to cast object of type 'Abp.MultiTenancy.TenantCacheItem
Here is the status of the redis database before running PerRequestRedisCache and after exceptions occurs :
Hi,
I'm getting some logged errors due to Redis cache timeout. The UI shows a message "InternalServerError". Here is the log :
WARN 2021-12-08 08:21:43,756 [102 ] Abp.MultiTenancy.TenantResolver - StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), inst: 0, qu: 19, qs: 0, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: , IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=17,Free=32750,Min=1,Max=32767), v: 2.2.62.27853 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2851
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54
at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 2409
at Abp.Runtime.Caching.Redis.AbpRedisCache.TryGetValue(String key, Object& value)
at Abp.Runtime.Caching.AbpCacheBase`2.Get(TKey key, Func`2 factory)
at Abp.Runtime.Caching.TypedCacheWrapper`2.Get(TKey key, Func`2 factory)
at Abp.MultiTenancy.TenantCache`2.GetOrNull(String tenancyName)
at Abp.MultiTenancy.TenantStore.Find(String tenancyName)
at Abp.AspNetCore.MultiTenancy.DomainTenantResolveContributor.ResolveTenantId()
at Abp.MultiTenancy.TenantResolver.GetTenantIdFromContributors()
StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), inst: 0, qu: 19, qs: 0, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: , IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=17,Free=32750,Min=1,Max=32767), v: 2.2.62.27853 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2851
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54
at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 2409
at Abp.Runtime.Caching.Redis.AbpRedisCache.TryGetValue(String key, Object& value)
at Abp.Runtime.Caching.AbpCacheBase`2.Get(TKey key, Func`2 factory)
at Abp.Runtime.Caching.TypedCacheWrapper`2.Get(TKey key, Func`2 factory)
at Abp.MultiTenancy.TenantCache`2.GetOrNull(String tenancyName)
at Abp.MultiTenancy.TenantStore.Find(String tenancyName)
at Abp.AspNetCore.MultiTenancy.DomainTenantResolveContributor.ResolveTenantId()
at Abp.MultiTenancy.TenantResolver.GetTenantIdFromContributors()
Do you think that this can be linked with multiple requests to Redis due to GetAll requests ? Is "Per Request Redis Cache" implementation will solve that ?
Thank you @ismcagdas
Hi,
I'm getting very long request issues with GetProfilePictureByUserId endpoint. Sometimes more than 1 min ! Here is an example from Azure insights :
As you can see, a lot of time is "waste" before accessing database (which is very short time access).
I slightly change GetProfilePictureByUserId to implement a cache mechanism. Here is the code :
public async Task<GetProfilePictureOutput> GetProfilePictureByUserId(long userId)
{
//Get from cache first
var outputFromCache = await _cacheManager.GetCache(UserPictureByUserIdCacheName).GetOrDefaultAsync(userId.ToString()) as GetProfilePictureOutput;
if (outputFromCache != null)
return outputFromCache;
else
{
var user = await UserManager.GetUserByIdAsync(userId);
if (user.ProfilePictureId == null)
{
return new GetProfilePictureOutput(string.Empty);
}
var output = await GetProfilePictureById(user.ProfilePictureId.Value);
//Set cache
await _cacheManager.GetCache(UserPictureByUserIdCacheName).SetAsync(userId.ToString(), output);
return output;
}
}
When testing this code, it runs very fast. But on production, a lot of requests gets stuck sometimes. I'm using Redis cache with azure app service in 2 instances.
I have the same issues with other endpoints like :
All this seems to be linked with cache management. This is frustrating because the app can be very slow time to time and frustrating lots of users.
I tried to use AbpPerRequestRedisCache but it is throwing exceptions at the moment...
How can I fix this ?