Open Closed

Token validation very long #10772


0
ricavir created
  • Abp 6.6.2
  • Angular
  • .NET 5

Hi,

As you know from my previous questions, I'm facing many performance issues while adding new users to the app. I'm dealing with multi instance on Azure, Redis cache, Azure SignalR...

The problem here is that some requests are very long to execute. Please look at following example :

As you can see, the request is blocked for 10s on token validation... On the left side of the screenshot you can see thaat the same request can be very fast. I suspect a lock somewhere in the app because all Azure resources are good (CPU, RAM...).

It is very hard to find the root cause from my side.

Can you please provide some help to find a solution ?


10 Answer(s)
  • 0
    ricavir created

    Here is more information from profiler. ValidateToken method is waiting most of the time from another thread :

  • 0
    ricavir created

    The problem should came from AsyncHelper.RunSync. All user requests are blocked when applicaation is in waiting state... maybe when updating cache data. I saw on abp source code that some locks and semaphore are being done.

    I should not be the only customer facing that issues I guess.

    We are pushing this issue in our TOP priorities ; I will continue my findings with dump files and provide feedback.

    Your help will be much appreciated

  • 0
    ismcagdas created
    Support Team

    Hi @ricavir

    We tried to reduce usages of AsyncHelper and removed it from many places. However, for security token validation, we are waiting for https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/issues/468.

    Could you create an issue on GitHub, we will try to find a different solution to this problem.

  • 0
    ricavir created

    Hi @ismcagdas,

    Thank you for your suggestion. I will create an issue on Github.

    The problem is happening on a regular basis : many requests are delayed from 10s and more ! When that happens everything seems to be locked. Something else must be wrong in the project. I can't imagine that other abp apps are having the same issue without doing anything.

    We are actively working on this issue and analysing memory dumps right now.

    Don't you have a clue to guide us ? We need to find a workarround in a very short time.

  • 0
    ismcagdas created
    Support Team

    Hi @ricavir

    The problem is ValidateToken is not async in *JwtSecurityTokenHandler. I think using a custom ASP.NET Core middleware instead of this class would be an option. In a middleware, you can totally validate the token asynchronously.

  • 0
    ricavir created

    Thank you for your feedback @ismcagdas. I saw that you planned to work on it for v11.1, this is great. While waiting, do you have an idea or workarround to improve this ?

    We mitigate this issue and we found some similar issues (request queued) due to azure app service thread pool mechanism. It seems that adding threads to the pool can take 500ms. Thus, some guys are using this code :

    ThreadPool.SetMinThreads(50, 50); // worker and IO threads
    

    This allows to define a minimum thread count for the instance... this seems to improve a lot performance but we also found a lot of warnings. We are going to test it anyway in our staging slot but we would like to have your advice about it.

  • 0
    ricavir created

    Finally, we decided to implement ThreadPool.SetMinThreads on staging slot. The first results were very good ! It is now in production and we improved by 2 request performance. It is still under monitoring for the next days.

    Our finding is that a sync method that is in an hot path (like ValidateToken) can turn very bad on azure since the allocation of threads is also taking some time to be done (500ms per allocation). Then, with increasing requests incoming, we had lots of requests queued wainting for available threads.

    This has been misleading us for a while because we were just adding more Azure resources ; but this problem is not depending on the server CPU or RAM, it is depending on thread pool behavior. Therefore, the only workarround is to set a minimum thread count and carefuly tune the value (we started to a value of 100 minimun threads).

    Hope this can help other abp users.

    We are then waiting for an async implementation of JWT token validation by ABP team to avoid definitely these side effects :)

  • 0
    ismcagdas created
    Support Team

    @ricavir thank you for your feedback. We will try to provide a solution with v11.1. I hope Microsoft Team cna provide the async support before we release v11.1 because it would be the best solution :).

  • 0
    ricavir created

    Hi @ismcagdas, I also hope that Microsoft will fix that quickly but the issue is opened since 2016... I'm not confident for a fix until some months ! We are still getting big waiting times (sometimes more than 20s) because of this. It would be welcome to have a work arround for the next version 11.1 as you plan it on the milestone https://github.com/aspnetzero/aspnet-zero-core/milestone/93

  • 0
    ismcagdas created
    Support Team

    Hi @ricavir

    Thanks, yes, even if Microsoft doesn't provide a fix, we will at least provide a workaround for this.