Open Closed

Hundreds of Redis commands in GetLanguageTexts and GetRoleForEdit caused bad performance #8948


0
OriAssurant created

Hi,

Our product is based on Core2.1 + Angular8, and we planned to turn on Redis for cache.

Before we turn on Redis, the default memory cache was being used.

Take GetRoleForEdit as an example,

Before Redis was turned on, The EditRole modal was rendered in 1-2 seconds.

After Redis was turned on: Whenever we click on GetRoleForEdit, it takes long time (more than 50s+, sometimes timeout) to render the modal.

Same thing happens for my local redis setup -- it takes 6.08 seconds for the result to come back when I use redis on my local workstation.

For GetLanguageTexts endpoint, the longest request duration was even 19min today.

One of assumptions is that Redis is reflecting what's happening under the hood -- GetRoleForEdit is visiting cache hundreds of time, and the performance issue emerges after we turned on Redis. Another assumption is the some logic in AbpRedisCacheManager causes the 'infinite' calls to Redis cache.

Please advise why this performance issue happens after turning on Redis.

Thank you,


14 Answer(s)
  • 0
    maliming created
    Support Team

    hi

    I will check this problem, thanks for your feedback.

  • 0
    maliming created
    Support Team

    hi

    Try to override the GetGrantedPermissionsAsync method of AbpRoleManager. To see if performance has improved.

    https://github.com/aspnetboilerplate/aspnetboilerplate/pull/5546/files

    You should also check your redis, I think its performance is very poor. Of course zero code needs to be enhanced, but redis should not be as slow as you mentioned.

  • 0
    OriAssurant created

    Thank you so much for quick turnaround, maliming!

    The performance improved a lot as it doesn't have duplicate calls to retrieve 'abpzerorolepermissions'. But there are still some duplicate calls (50+, based on permissions we have I guess) to "AbpZeroMultiTenantLocalizationDictionaryCache" and "AbpZeroLanguages".

    Also there is also similar issue with GetLanguageTexts (These two places might be caused by the same code). Could you please help look into that one as well? When I click on ChangeTexts Then in redis:

    Update:

    Seems issue comes from MultiTenantLocalizationSource.GetStringOrNull and then MultiTenantLocalizationDictionary.GetOrNull. https://github.com/aspnetboilerplate/aspnetboilerplate/blob/dev/src/Abp.Zero.Common/Localization/MultiTenantLocalizationSource.cs https://github.com/aspnetboilerplate/aspnetboilerplate/blob/dev/src/Abp.Zero.Common/Localization/MultiTenantLocalizationDictionary.cs

    This line is reading from cache every time.

  • 0
    OriAssurant created

    I think I get that every localization is calling redis cache, even following simple line in GetRoleForEdit method (could you give some hint how this happens? I don't get how it reaches out the cache in following line):

    ObjectMapper.Map<List<FlatPermissionDto>>(permissions).OrderBy(p => p.DisplayName).ToList();
    

    Thank you

  • 0
    maliming created
    Support Team

    hi

    I know the above code is reading the cache multiple times, I will investigate and try to solve this problem, before that, You should also check your redis, I think its performance is very poor.

    https://github.com/aspnetboilerplate/aspnetboilerplate/issues/5551

  • 0
    OriAssurant created

    Thank you very much maliming! Is that because of the implementation of IMultiTenantLocalizationSource causing multiple trips to cache data store?

    Yeah, the Azure Redis cluster performance is 10 times worse than my local as it's cross data center. The cluster's IOPS is not bad but network performance causes the delay.. Will investigate more on that.

    Best regards,

  • 0
    ismcagdas created
    Support Team

    Hi,

    Is that because of the implementation of IMultiTenantLocalizationSource causing multiple trips to cache data store?

    Yes,

  • 0
    OriAssurant created

    Thank you ismcagdas!

    @maliming, Do you know when would the pull request be raised? Not trying to push sorry. I might choose to implment my own solution as our timeline is approaching to the release date.

    Thank you, Robin

  • 0
    maliming created
    Support Team

    Do you know when would the pull request be raised?

    It is planned in abp v5.8(Due by May 21, 2020).

  • 0
    OriAssurant created

    Hi Maliming,

    Thank you for making this issue as priority. But seems it's only applied to Abp 5.8 (.net core 3.1) and we're using Abp 4.10 (.net core 2.2). I'm afraid there are some reference conflict if I change the Abp reference directly to version 5.8 in the project. Is there any suggestion to work around this issue? I'm thinking of overwriting all localization files, but that's also quite a big change.

  • 0
    maliming created
    Support Team

    hi

    I will apply the same changes in abp 4.x version. https://github.com/aspnetboilerplate/aspnetboilerplate/issues/5583

  • 0
    OriAssurant created

    Thank you so much maliming!! You're the savior. Would it be abp version 4.11?

    Best regards,

  • 0
    maliming created
    Support Team

    It should be abp v4.16

  • 0
    OriAssurant created

    Thank you again maliming. Let me close this thread:)