AbpPerRequestRedisCache concurrency exception #11992

We use ASP.NET Zero 9.1.1, Redis cache is used, with AbpPerRequestRedisCache enabled.

Recently our live system has experienced large amount of exceptions - sometimes 70,000 exceptions in a short period, with hundreds exceptions generated per second. All exceptions have 2 patterns like these:

2024-05-06 12:25:09,133 ERROR [orker] me.Caching.Redis.AbpPerRequestRedisCache - System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value) at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.SetAsync(String key, Object value, Nullable`1 slidingExpireTime, Nullable`1 absoluteExpireTime) at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory) System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value) at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.SetAsync(String key, Object value, Nullable`1 slidingExpireTime, Nullable`1 absoluteExpireTime) at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory)

2024-05-07 02:23:44,996 ERROR [orker] me.Caching.Redis.AbpPerRequestRedisCache - System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. at System.Collections.Generic.Dictionary`2.FindValue(TKey key) at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value) at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.TryGetValueAsync(String key) at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory) System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. at System.Collections.Generic.Dictionary`2.FindValue(TKey key) at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value) at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.TryGetValueAsync(String key) at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory)

It seems the error happens at https://github.com/aspnetboilerplate/aspnetboilerplate/blob/v9.1/src/Abp.AspNetCore.PerRequestRedisCache/Runtime/Caching/Redis/AbpPerRequestRedisCache.cs when it is accessing httpContext.Items dictionary.

Our system use async / await extensively.

Should AbpPerRequestRedisCache be updated to support concurrent access within the same request? This seems the cause. Is there any other possible cause?