PeriodicBackgroundWorkers and Load Balanced Environments #12000

Hi,

We run in a load balanced environment for our app servers in production. PeriodicBackgroundWorkers don't appear to play nicely in this environment.

As they are in memory scheduled jobs they clash with each other quite often.

For example we get this error in our production logs a lot. We also re-wrote the ExpiredAuditLogDeleterWorker to be a hangfirejob.

Is there any recommendation or plans on how we can easily remove/convert these background jobs into hangfire jobs which will only run on one server and not have concurrency issues like most of the non hangfire run periodic workers do?

Thanks Rick

2024-05-15T08:35:27.585+10:00 - (null) - (null) - b1c4f22152a4 - Abp.Authorization.Users.UserTokenExpirationWorker2[[XXXX.MultiTenancy.Tenant, XXXX.Core, Version=13.1.0.1574, Culture=neutral, PublicKeyToken=null],[XXXX.Authorization.Users.User, XXXX.Core, Version=13.1.0.1574, Culture=neutral, PublicKeyToken=null]] - Abp.Domain.Uow.AbpDbConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions. ---> Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions. at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyException(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected) at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.Consume(RelationalDataReader reader) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(IEnumerable1 commandBatches, IRelationalConnection connection) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IList1 entriesToSave) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(StateManager stateManager, Boolean acceptAllChangesOnSuccess) at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlExecutionStrategy.Execute[TState,TResult](TState state, Func3 operation, Func3 verifySucceeded) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(Boolean acceptAllChangesOnSuccess) at Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess) at Abp.EntityFrameworkCore.AbpDbContext.SaveChanges() --- End of inner exception stack trace --- at Abp.EntityFrameworkCore.AbpDbContext.SaveChanges() at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.SaveChanges() at Abp.Domain.Uow.UnitOfWorkBase.Complete() at Abp.Authorization.Users.UserTokenExpirationWorker2.DoWork() at Abp.Threading.BackgroundWorkers.PeriodicBackgroundWorkerBase.Timer_Elapsed(Object sender, EventArgs e) Abp.Domain.Uow.AbpDbConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions. ---> Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions. at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyException(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected) at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.Consume(RelationalDataReader reader) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.Execute(IRelationalConnection connection) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(IEnumerable1 commandBatches, IRelationalConnection connection) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IList1 entriesToSave) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(StateManager stateManager, Boolean acceptAllChangesOnSuccess) at Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlExecutionStrategy.Execute[TState,TResult](TState state, Func3 operation, Func3 verifySucceeded) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(Boolean acceptAllChangesOnSuccess) at Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess) at Abp.EntityFrameworkCore.AbpDbContext.SaveChanges() --- End of inner exception stack trace --- at Abp.EntityFrameworkCore.AbpDbContext.SaveChanges() at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.SaveChanges() at Abp.Domain.Uow.UnitOfWorkBase.Complete() at Abp.Authorization.Users.UserTokenExpirationWorker`2.DoWork() at Abp.Threading.BackgroundWorkers.PeriodicBackgroundWorkerBase.Timer_Elapsed(Object sender, EventArgs e)