Abp 6.6.2 Angular .NET 5
Hi,
I'm getting a timeout exception when trying to migrate our Azure prod database. We never had this exception before for more than 100 migrations in the past years. This exception is only happening on production. We notice a DB consumption spike on azure every time we run the migrator : therefore, we upgraded our plan from 20DTU to 50DTU. Still same error. We have tried to add a connection timeout of 900s in the connection string but it also fails.
Changing the connection string to local DB works correctly.
Here is the exception :
2022-05-31 07:40:18 | Host database: Server=tcp:xxx.database.windows.net,1433;Initial Catalog=xxxx;User ID=xxxx;Password=xxxx;
2022-05-31 07:40:25 | Continue to migration for this host database and all tenants..? (Y/N):
Y
2022-05-31 07:40:37 | HOST database migration started...
2022-05-31 07:41:37 | An error occured during migration of host database:
2022-05-31 07:41:37 | Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Dépassement du délai d'attente.
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean isAsync, Int32 timeout, Boolean asyncWrite)
at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteNonQuery(RelationalCommandParameterObject parameterObject)
at Microsoft.EntityFrameworkCore.Migrations.MigrationCommand.ExecuteNonQuery(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationCommandExecutor.ExecuteNonQuery(IEnumerable`1 migrationCommands, IRelationalConnection connection)
at Microsoft.EntityFrameworkCore.Migrations.Internal.Migrator.Migrate(String targetMigration)
at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.Migrate(DatabaseFacade databaseFacade)
at Abp.Zero.EntityFrameworkCore.AbpZeroDbMigrator`1.CreateOrMigrate(AbpTenantBase tenant, Action`1 seedAction)
at Abp.Zero.EntityFrameworkCore.AbpZeroDbMigrator`1.CreateOrMigrateForHost(Action`1 seedAction)
at App.Migrator.MultiTenantMigrateExecuter.Run(Boolean skipConnVerification) in C:\Users\User\source\repos\App_AspnetZero\aspnet-core\src\App.Migrator\MultiTenantMigrateExecuter.cs:line 62
ClientConnectionId:f8435bcc-7872-4566-aac5-57cf9941e750
Error Number:-2,State:0,Class:11
2022-05-31 07:41:37 | Canceled migrations.
Press ENTER to exit...
Our production code is currently blocked because of this exception. Any help would be much apreciated :)
Hi @sedulen,
No, not using Appinsights anymore as the app is running well now. On prod environment, our logging level is set to WARN. I don't have any information to provide about serialization, I'm using default ANZ implementation. Didn't see any issue with it.
Glad to help if I can :)
Hi @sedulen,
I know this is not an easy task to do... be strong :) From our side, we are running on Azure App Service (Windows). App Insights has been very useful to find our main issues, but be careful with its usage, we found that it "costs" on app performance. We heard that it can consume 20 to 25% of performance down ! When we stoped AppInsights, we noticed a gain of more than 50 ms on request time average. Your AWAIT_TIMEs can be caused by the usage of sync calls instead of async. I know that the lastest ABP releases have fixed a lot of that issues. Try to have a look at it !
Cheers
Hi @sedulen
Yes, I have implemented asynchronous JWT token validation. It is better from my side, the app is faster and I'm not having these AWAIT_TIME anymore.
You are pointing out AbpUserTokens, try to reduce the refresh token expiration date et check if this table has not too many records, this can affect you performance.
I also tuned the thread pool count as explained upper on this thread ; this has also improved the app performance.
I don't have used a loading test tool as you did.... but I should :)
Hi @ismcagdas,
Tks for pointing out this issue. I already noticed better performance with projectoin before querrying so that means it will be even better after this resolution.
Hi,
After some research on ABP github repo, I found that automapper projection is already available by using :
var dtos = await ObjectMapper.ProjectTo<EntityDto>(entities)
.OrderBy(input.Sorting)
.PageBy(input)
.ToListAsync();
The generated SQL is better ; only needed properties are available. This seems to be better also in performance but as fast as my first tests.
@support_team : do you agree that object projection before querying is better than mapping objects after querying ? If so, you should add a note in the docs.
Hi support team,
I'm currently focusing on EF queries optimization, specially for read only queries. I realized that EF queries like this one :
await _entityRepository.getAll().Include(e => e.NestedEntity).OrderBy(input.Sorting).PageBy(input).ToListAsync()
is causing database delays because SQL engine tries to get every properties from entity (event more if a nested entity is included in the query).
After some tests, I achieve a query 4x times faster by selecting only needed entity properties like that :
await _entityRepository.getAll().Include(e => e.NestedEntity).Select(e => new EntityDto { Id = e.Id, Description = e.Description}).OrderBy(input.Sorting).PageBy(input).ToListAsync()
As you can see, I'm using a DTO (EntityDto) to map from Entity object (this is called "projection"). My goal is to reuse all existing DTO's to optimize all select queries ; BUT I'm doing the mapping manually, which can cause errors in the future (ex : if a new property is added to the DTO). I need this to be more maintainable and optimized.
Do you have a solution to automatically map the DTO in the SELECT statement of the query ?
Hi @ismcagdas : I've sent you the files by mail
Hi @sedulen : Thank you for your help. I'm not using hangfire, only using all ABP / ANZ native capabilities. My app has started some years ago and the update process according to current ZERO version is specific. Some time ago, we took the decision to update ABP packages, .NET and Angular only. We only take new ABP features if needed on our side. We stopped updating Metronic theme by version 5 also (as it looks better from our opinion). About background jobs and workers, we are using latests updates from ABP. We always look at the current implementation of ABP team before changing something. All that stuff has been changed from sync to async some months ago. Doing sync stuff is not recommended with latest versions of .NET. This issue is not systematic and is running well most of the time... but sometimes, we are having these logs for background jobs and background workers. I asked ABP team to write some guidelines about multi instance management specialy for theses cases (jobs, workers, cache management...) as it seems to be causing lots of issues...
I can confirm that this error is logged even ef a single instance is running
Hi abp team,
I'm getting following error several times per day :
ERROR 2022-02-11 10:24:53,841 [86 ] me.Caching.Redis.AbpPerRequestRedisCache - System.ObjectDisposedException: IFeatureCollection has been disposed.
Object name: 'Collection'.
at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed()
at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Items()
at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.TryGetValueAsync(String key)
at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory)
System.ObjectDisposedException: IFeatureCollection has been disposed.
Object name: 'Collection'.
at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed()
at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Items()
at Abp.Runtime.Caching.Redis.AbpPerRequestRedisCache.TryGetValueAsync(String key)
at Abp.Runtime.Caching.AbpCacheBase`2.GetAsync(TKey key, Func`2 factory)
Is it a problem we can fix ? Is it due to redis data expiration policy ?