Base solution for your next web application
BLACK FRIDAY
From Nov 21
To Dec 1
XXDAYS : XXHRS : XXMIN : XXSEC
SAVE UP TO$3K
Also valid for the
existing customers!
Open Closed

Background worker database exceptions #10672


0
ricavir created
  • Abp 6.4
  • Angular 12
  • .NET 5

Hi,

I'm getting some warning exceptions with background workers. Can you please advise ?

Here is the exception :

WARN 2021-11-09 13:33:48,021 [76 ] Abp.BackgroundJobs.BackgroundJobManager - Abp.Domain.Uow.AbpDbConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
---> Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyException(Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected)
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithoutPropagationAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(DbContext _, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Abp.EntityFrameworkCore.AbpDbContext.SaveChangesAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Abp.EntityFrameworkCore.AbpDbContext.SaveChangesAsync(CancellationToken cancellationToken)
at Abp.Zero.EntityFrameworkCore.AbpZeroCommonDbContext`3.SaveChangesAsync(CancellationToken cancellationToken)
at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.SaveChangesAsync()
at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.CompleteUowAsync()
at Abp.Domain.Uow.UnitOfWorkBase.CompleteAsync()
at Abp.Domain.Uow.UnitOfWorkManagerExtensions.WithUnitOfWorkAsync[TResult](IUnitOfWorkManager manager, Func`1 action, UnitOfWorkOptions options)
at Abp.BackgroundJobs.BackgroundJobStore.UpdateAsync(BackgroundJobInfo jobInfo)
at Abp.BackgroundJobs.BackgroundJobManager.TryUpdateAsync(BackgroundJobInfo jobInfo)
Abp.Domain.Uow.AbpDbConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
---> Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyException(Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected)
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithoutPropagationAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(DbContext _, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Abp.EntityFrameworkCore.AbpDbContext.SaveChangesAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Abp.EntityFrameworkCore.AbpDbContext.SaveChangesAsync(CancellationToken cancellationToken)
at Abp.Zero.EntityFrameworkCore.AbpZeroCommonDbContext`3.SaveChangesAsync(CancellationToken cancellationToken)
at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.SaveChangesAsync()
at Abp.EntityFrameworkCore.Uow.EfCoreUnitOfWork.CompleteUowAsync()
at Abp.Domain.Uow.UnitOfWorkBase.CompleteAsync()
at Abp.Domain.Uow.UnitOfWorkManagerExtensions.WithUnitOfWorkAsync[TResult](IUnitOfWorkManager manager, Func`1 action, UnitOfWorkOptions options)
at Abp.BackgroundJobs.BackgroundJobStore.UpdateAsync(BackgroundJobInfo jobInfo)
at Abp.BackgroundJobs.BackgroundJobManager.TryUpdateAsync(BackgroundJobInfo jobInfo)

14 Answer(s)
  • 0
    ismcagdas created
    Support Team

    Hi @ricavir

    Could you share your BackgroundJob code ? There might be an async over sync usage.

  • 0
    ricavir created

    Hi @ismcagdas,

    I don't see async with sync usage. Here is the backgroundjob code :

    public class FileGeneratorEventJob : AsyncBackgroundJob<FileGeneratorEventJobArgs>, ITransientDependency
        {
            private readonly IRepository<DocumentTemplate> _documentTemplateRepository;
            private readonly IFileGenerator _fileGenerator;
            private readonly IUnitOfWorkManager _unitOfWorkManager;
            private readonly IAppNotifier _appNotifier;
            private readonly IObjectMapper _objectMapper;
            private readonly IEventQueryManager _eventQueryManager;
    
            public FileGeneratorEventJob(IRepository<DocumentTemplate> documentTemplateRepository,
                                                    IFileGenerator fileGenerator,
                                                    IUnitOfWorkManager unitOfWorkManager,
                                                    IObjectMapper objectMapper,
                                                    IAppNotifier appNotifier,
                                                    IEventQueryManager eventQueryManager)
            {
                _documentTemplateRepository = documentTemplateRepository;
                _fileGenerator = fileGenerator;
                _appNotifier = appNotifier;
                _unitOfWorkManager = unitOfWorkManager;
                _objectMapper = objectMapper;
                _eventQueryManager = eventQueryManager;
            }
    
            public override async Task ExecuteAsync(FileGeneratorEventJobArgs args)
            {
                await _unitOfWorkManager.WithUnitOfWorkAsync(async () =>
                {
                    using (UnitOfWorkManager.Current.SetTenantId(args.TenantId))
                    {
                        using (CultureInfoHelper.Use(await SettingManager.GetSettingValueForTenantAsync(LocalizationSettingNames.DefaultLanguage, args.TenantId)))
                        {
                            try
                            {
                                var documentTemplate = await _documentTemplateRepository.FirstOrDefaultAsync(args.DocumentTemplateId);
                                if (documentTemplate == null)
                                    return;
                                if (documentTemplate.Extension != TemplateExtension.Word)
                                    return;
    
                                var filteredEvents = _eventQueryManager.GetEventsQueryable(args.EventForecastsInput);
    
                                var events = await filteredEvents.Take(AppConsts.MaxExportSize).OrderBy(args.EventsInput.Sorting).ToListAsync();
    
                                if (events.IsNullOrEmpty())
                                    return;
    
                                var dynamicFields = new List<DocumentTemplateField>();
                                if (args.DynamicFields != null)
                                {
                                    dynamicFields = _objectMapper.Map<List<DocumentTemplateField>>(args.DynamicFields);
                                }
    
                                DataExporting.FileObject file = null;
                                file = await _fileGenerator.GenerateFileAsync(events, args.DocumentTemplatePath, args.DocumentTemplatePrefixName, args.DocumentTemplateId, dynamicFields, args.ConvertToPdf, args.MergeFields, args.SaveAsFileReference, args.TenantId);
    
                                await _appNotifier.FileGenerationJobSucceedAsync(file, "FileGenerationJobForEventSucceed", args.UserIdentifier);
                            }
                            catch (Exception ex)
                            {
                                await _appNotifier.FileGenerationJobFailedAsync("FileGenerationJobForEventFailed", args.UserIdentifier);
                            }
                        }
                    }
                });
            }
    

    and it is called like this :

    
    ` await _backgroundJobManager.EnqueueAsync&lt;FileGeneratorEventJob, FileGeneratorEventJobArgs&gt;(new FileGeneratorEventJobArgs
                    {
                        EventsInput = input.eventsInput,
                        ConvertToPdf = input.templateInput.ConvertToPdf,
                        DocumentTemplateId = input.templateInput.DocumentTemplateId,
                        DocumentTemplatePath = documentPath,
                        DocumentTemplatePrefixName = documentTemplate.Code,
                        DynamicFields = input.templateInput.DynamicFields,
                        MergeFields = input.templateInput.MergeFields,
                        SaveAsFileReference = input.templateInput.SaveAsFileReference,
                        TenantId = AbpSession.TenantId.Value,
                        UserIdentifier = user.ToUserIdentifier()
                    });`
    
  • 0
    ismcagdas created
    Support Team

    Hi @ricavir

    Yes, this seems fine. I have two questions;

    • Do you run one instance for your web app ?
    • How often await _backgroundJobManager.EnqueueAsync is called ? Is it possible that, more than 1 instance of this job is executed at the same time ?

    Thanks

  • 0
    ricavir created

    Hi @ismcagdas,

    In general the app is running in one instance, but can be two according to CPU load. The job is fired manually by users from the web app, so the frequency depends. It can be done twice in short period. I'm not updating any data on EF on this jobs, therefore I suspect an issue while updating backgoundjob status in the database.

  • 0
    ismcagdas created
    Support Team

    Thanks @ricavir, please follow https://github.com/aspnetzero/aspnet-zero-core/issues/4103. We will work on this in the next version.

  • 0
    musa.demir created

    Hi @ricavir

    Sorry for late response. I could not reproduce the problem. Can you please share a project that contains that problem?

  • 0
    ricavir created

    Hi @musa.demir

    Unfortunately, I'm not authorized to send the project. Would it be possible to organize a Teams meeting to share my screen with you ?

    The problem can be linked with backgroundworkers our backgroundjobs running with over multiple instances.

    I also noticed that this occurs on app restart or after swaping slots in Azure app service.

  • 0
    musa.demir created

    Hi @ricavir

    Can you please run your app with single instance and check if it happens?

  • 0
    ricavir created

    Currently, the app runs with a single instance. When app is restarted, only one instance is running. We have a scale out rule that switch the app service to 2 or 3 instances according to CPU load but this is not happening frequently.

  • 0
    ricavir created

    I can confirm that this error is logged even ef a single instance is running

  • 0
    ismcagdas created
    Support Team

    Hi @ricavir

    Is it possible to share your BackgroundJob (FileGeneratorEventJob) and entities used in it privately with info@aspnetzero.com ?

    Thanks,

  • 0
    sedulen created

    Hi @ricavir ,

    I've been following this thread for a few days now. Just curious, for your BackgroundJob manager, are you using the ABP / ANZ native capability, or are you using Hangfire?

    Looking at the code you provided, I do something similar where a user submits a requests through the web app that a document is generated, and we offload the document generation to a background job, and use a Notification to notify the user that the job is done.

    As as you noted in a previous response, if your job isn't modifying an data in the database, then the only likelihood would be the Background Job manager keeping track of the internal state of it's record. We see this in the initial call stack that you provided

    at Abp.Domain.Uow.UnitOfWorkManagerExtensions.WithUnitOfWorkAsync[TResult](IUnitOfWorkManager manager, Func1 action, UnitOfWorkOptions options)
    **at Abp.BackgroundJobs.BackgroundJobStore.UpdateAsync(BackgroundJobInfo jobInfo)**
    at Abp.BackgroundJobs.BackgroundJobManager.TryUpdateAsync(BackgroundJobInfo jobInfo)
    

    Looking at the code you provided, I see you are explicitly starting a new UnitOfWork

    await _unitOfWorkManager.WithUnitOfWorkAsync(async () =>
                {
                ...
    

    I haven't used that function before, so I don't know if that's added in a newer version of ABP or ANZ that I'm just not on yet. (I'm still on .NET Core 3.1). I'm definitely not an expert in Unit Of Work here, but what I'm thinking is that you may have a conflict between your UOW and the UOW that is handled by BackgroundJobManager.

    Comparing your code to mine, I start my ExecuteAsync function with this:

                using (var uow = _unitOfWorkManager.Begin())
                {
                    using (_unitOfWorkManager.Current.SetTenantId(args.TenantId))
                    {
                        using (AbpSession.Use(args.TenantId, args.UserId))
                        {
    

    So I explicitly create my own new UnitOfWork within my function execution.

    Additionally, you have await _unitOfWorkManager.WithUnitOfWorkAsync(async () => where I don't. I'm curious if you changed that _unitOfWorkManager call from async to sync, if that could possibly have any impacts.

    Another observation, I don't know if it's required or not, but I call AbpSession.Use so that my UserId & TenantId are set in AbpSession within the scope of my transaction, just in case anything else downstream attempts to rely on AbpSession. I don't know if that's required or not.

    Another observation, I don't see any logging in your class, so I'd be curious to inject an instance of ILogger into your background job class and to write it to a logfile, just to try and get further evidence if this background job is possibly being executed multiple times concurrently.

    And lastly, if you aren't using Hangfire, I'd be curious to know if you have this same issue if you switched your BackgroundJob processing to leverage Hangfire instead of the ABP Native BackgroungJob Manager.

    Hope any of those ideas or insights might help, -Brian

  • 0
    ricavir created

    Hi @ismcagdas : I've sent you the files by mail

    Hi @sedulen : Thank you for your help. I'm not using hangfire, only using all ABP / ANZ native capabilities. My app has started some years ago and the update process according to current ZERO version is specific. Some time ago, we took the decision to update ABP packages, .NET and Angular only. We only take new ABP features if needed on our side. We stopped updating Metronic theme by version 5 also (as it looks better from our opinion). About background jobs and workers, we are using latests updates from ABP. We always look at the current implementation of ABP team before changing something. All that stuff has been changed from sync to async some months ago. Doing sync stuff is not recommended with latest versions of .NET. This issue is not systematic and is running well most of the time... but sometimes, we are having these logs for background jobs and background workers. I asked ABP team to write some guidelines about multi instance management specialy for theses cases (jobs, workers, cache management...) as it seems to be causing lots of issues...

  • 0
    ismcagdas created
    Support Team

    Hi @ricavir

    Thanks, I have received the email.

    I asked ABP team to write some guidelines about multi instance management specialy for theses cases (jobs, workers, cache management...) as it seems to be causing lots of issues...

    We will create a guideline for this, thank you for pointing it out.