We use ASP.NET Zero 9.1.1, Redis cache is used, with AbpPerRequestRedisCache enabled.
Recently our live system has experienced large amount of exceptions - sometimes 70,000 exceptions in a short period, with hundreds exceptions generated per second. All exceptions have 2 patterns like these:
2024-05-06 12:25:09,133 ERROR [orker] me.Caching.Redis.AbpPerRequestRedisCache - System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value) at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.SetAsync(String key, Object value, Nullable`1 slidingExpireTime, Nullable`1 absoluteExpireTime) at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory) System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value) at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.SetAsync(String key, Object value, Nullable`1 slidingExpireTime, Nullable`1 absoluteExpireTime) at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory)
2024-05-07 02:23:44,996 ERROR [orker] me.Caching.Redis.AbpPerRequestRedisCache - System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. at System.Collections.Generic.Dictionary`2.FindValue(TKey key) at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value) at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.TryGetValueAsync(String key) at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory) System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. at System.Collections.Generic.Dictionary`2.FindValue(TKey key) at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value) at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.TryGetValueAsync(String key) at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory)
It seems the error happens at https://github.com/aspnetboilerplate/aspnetboilerplate/blob/v9.1/src/Abp.AspNetCore.PerRequestRedisCache/Runtime/Caching/Redis/AbpPerRequestRedisCache.cs when it is accessing httpContext.Items dictionary.
Our system use async / await extensively.
Should AbpPerRequestRedisCache be updated to support concurrent access within the same request? This seems the cause. Is there any other possible cause?
We use the latest ASPNET Zero genrated solution. We are adding a new integration test project. Most is working fine.
However, the generated class MyCompanyJwtSecurityTokenHandler under MyCompany.Web.Authentication.JwtBearer uses lots of IocManager.Instance to resolve dependencies. IocManager.Instance is fine normally, but in a test project, it doesn't resolve dependencies. IIocManager should be used in my reading, that is designed for test projects.
How should we fix this class? It is hard constructed by AuthConfigurer.Configure() method.
Thanks for your response.
Dear Support,
We are using ASPNET Zero subscription, that has a maximum developers count. We have a support personnel - he reads the source code from Visual Studio, and runs it locally to play with. But he doesn't write or change any source code - read only access. Does he count to the developers in the subscription? We just reach out limit, and not sure whether we need to buy an extra developer licence.
Thanks! Richard
In the Abp.DynamicEntityProperties.DynamicPropertyManager class, the cache is defined as:
private ITypedCache<int, DynamicProperty> DynamicPropertyCache => _cacheManager.GetCache<int, DynamicProperty>(CacheName);
TypedCache is not tenant safe. We use a multi-tenant configuration. When we get the DynamicProperty by Id, the cache returns the DynamicProperty with that Id, from a different tenant.
The same bug occurs for DynamicEntityPropertyManager class too.
Suggest to review all ITypedCache usage.
Dear Support,
Aspnet Zero provides a language text customistion at host level. Can it be adpated to tenant level? So each tenant can customise languiage text by them self. What should we be aware to do that?
Are language texts cached?
Thanks! Richard
This is just a suggestion / feedback.
PagedAndSortedResultRequestDto is the DTO used Power Tool generated code. PagedAndSortedInputDto is the code used in other places by Zero.
They are really the same thing at the server side. However, nswag generated Angular code has different sequence on sorting/paging fields. It causes confusion, and causes strange error if not following the field sequence exactly.
It is better to just use one of them.
Dear support,
We have 1 host database, and multiple separate tenant databases for those tenants with large amount of own data.
Host database stores some common host data, such as tenants / editions tables. Because of primary / foreign key enforcement, we have to replicate these host data to every separate tenant database.
Do you have a way to maintain these? For example, when creating a new separate tenant database, we need to copy host data; when tenant information changed in the host database, we have to replicate manually to each separate tenant database. Would you recommend a better way?
Thanks Richard
Thank you very much!
Thank you!
We got all these working.
My question is to enhance the DynamicParameter entities themself. We have an ETL process to produce a data warehouse, relies on LastModificationTime. We need to transfer dynamic values to the data warehouse, but DynamicParameter entities don't have any time stamps. How can we add?