Background worker database exceptions #10672

14 Answer(s)

• 

  0

  ismcagdas created 3 years ago

  Support Team

  Hi @ricavir

  Could you share your BackgroundJob code ? There might be an async over sync usage.

  Save Cancel
• 

  0

  Ricavir created 3 years ago

  Hi @ismcagdas,

  I don't see async with sync usage. Here is the backgroundjob code :

  public class FileGeneratorEventJob : AsyncBackgroundJob<FileGeneratorEventJobArgs>, ITransientDependency
      {
          private readonly IRepository<DocumentTemplate> _documentTemplateRepository;
          private readonly IFileGenerator _fileGenerator;
          private readonly IUnitOfWorkManager _unitOfWorkManager;
          private readonly IAppNotifier _appNotifier;
          private readonly IObjectMapper _objectMapper;
          private readonly IEventQueryManager _eventQueryManager;
  
          public FileGeneratorEventJob(IRepository<DocumentTemplate> documentTemplateRepository,
                                                  IFileGenerator fileGenerator,
                                                  IUnitOfWorkManager unitOfWorkManager,
                                                  IObjectMapper objectMapper,
                                                  IAppNotifier appNotifier,
                                                  IEventQueryManager eventQueryManager)
          {
              _documentTemplateRepository = documentTemplateRepository;
              _fileGenerator = fileGenerator;
              _appNotifier = appNotifier;
              _unitOfWorkManager = unitOfWorkManager;
              _objectMapper = objectMapper;
              _eventQueryManager = eventQueryManager;
          }
  
          public override async Task ExecuteAsync(FileGeneratorEventJobArgs args)
          {
              await _unitOfWorkManager.WithUnitOfWorkAsync(async () =>
              {
                  using (UnitOfWorkManager.Current.SetTenantId(args.TenantId))
                  {
                      using (CultureInfoHelper.Use(await SettingManager.GetSettingValueForTenantAsync(LocalizationSettingNames.DefaultLanguage, args.TenantId)))
                      {
                          try
                          {
                              var documentTemplate = await _documentTemplateRepository.FirstOrDefaultAsync(args.DocumentTemplateId);
                              if (documentTemplate == null)
                                  return;
                              if (documentTemplate.Extension != TemplateExtension.Word)
                                  return;
  
                              var filteredEvents = _eventQueryManager.GetEventsQueryable(args.EventForecastsInput);
  
                              var events = await filteredEvents.Take(AppConsts.MaxExportSize).OrderBy(args.EventsInput.Sorting).ToListAsync();
  
                              if (events.IsNullOrEmpty())
                                  return;
  
                              var dynamicFields = new List<DocumentTemplateField>();
                              if (args.DynamicFields != null)
                              {
                                  dynamicFields = _objectMapper.Map<List<DocumentTemplateField>>(args.DynamicFields);
                              }
  
                              DataExporting.FileObject file = null;
                              file = await _fileGenerator.GenerateFileAsync(events, args.DocumentTemplatePath, args.DocumentTemplatePrefixName, args.DocumentTemplateId, dynamicFields, args.ConvertToPdf, args.MergeFields, args.SaveAsFileReference, args.TenantId);
  
                              await _appNotifier.FileGenerationJobSucceedAsync(file, "FileGenerationJobForEventSucceed", args.UserIdentifier);
                          }
                          catch (Exception ex)
                          {
                              await _appNotifier.FileGenerationJobFailedAsync("FileGenerationJobForEventFailed", args.UserIdentifier);
                          }
                      }
                  }
              });
          }
  

  and it is called like this :

  
  ` await _backgroundJobManager.EnqueueAsync<FileGeneratorEventJob, FileGeneratorEventJobArgs>(new FileGeneratorEventJobArgs
                  {
                      EventsInput = input.eventsInput,
                      ConvertToPdf = input.templateInput.ConvertToPdf,
                      DocumentTemplateId = input.templateInput.DocumentTemplateId,
                      DocumentTemplatePath = documentPath,
                      DocumentTemplatePrefixName = documentTemplate.Code,
                      DynamicFields = input.templateInput.DynamicFields,
                      MergeFields = input.templateInput.MergeFields,
                      SaveAsFileReference = input.templateInput.SaveAsFileReference,
                      TenantId = AbpSession.TenantId.Value,
                      UserIdentifier = user.ToUserIdentifier()
                  });`
  

  Save Cancel
• 

  0

  ismcagdas created 3 years ago

  Support Team

  Hi @ricavir

  Yes, this seems fine. I have two questions;

  • Do you run one instance for your web app ?
  • How often await _backgroundJobManager.EnqueueAsync is called ? Is it possible that, more than 1 instance of this job is executed at the same time ?

  Thanks

  Save Cancel
• 

  0

  Ricavir created 3 years ago

  Hi @ismcagdas,

  In general the app is running in one instance, but can be two according to CPU load. The job is fired manually by users from the web app, so the frequency depends. It can be done twice in short period. I'm not updating any data on EF on this jobs, therefore I suspect an issue while updating backgoundjob status in the database.

  Save Cancel
• 

  0

  ismcagdas created 3 years ago

  Support Team

  Thanks @ricavir, please follow https://github.com/aspnetzero/aspnet-zero-core/issues/4103. We will work on this in the next version.

  Save Cancel
• 

  0

  musa.demir created 3 years ago

  Hi @ricavir

  Sorry for late response. I could not reproduce the problem. Can you please share a project that contains that problem?

  Save Cancel
• 

  0

  Ricavir created 3 years ago

  Hi @musa.demir

  Unfortunately, I'm not authorized to send the project. Would it be possible to organize a Teams meeting to share my screen with you ?

  The problem can be linked with backgroundworkers our backgroundjobs running with over multiple instances.

  I also noticed that this occurs on app restart or after swaping slots in Azure app service.

  Save Cancel
• 

  0

  musa.demir created 3 years ago

  Hi @ricavir

  Can you please run your app with single instance and check if it happens?

  Save Cancel
• 

  0

  Ricavir created 3 years ago

  Currently, the app runs with a single instance. When app is restarted, only one instance is running. We have a scale out rule that switch the app service to 2 or 3 instances according to CPU load but this is not happening frequently.

  Save Cancel
• 

  0

  Ricavir created 3 years ago

  I can confirm that this error is logged even ef a single instance is running

  Save Cancel
• 

  0

  ismcagdas created 3 years ago

  Support Team

  Hi @ricavir

  Is it possible to share your BackgroundJob (FileGeneratorEventJob) and entities used in it privately with [email protected] ?

  Thanks,

  Save Cancel
• 

  0

  sedulen created 3 years ago

  Hi @ricavir ,

  I've been following this thread for a few days now. Just curious, for your BackgroundJob manager, are you using the ABP / ANZ native capability, or are you using Hangfire?

  Looking at the code you provided, I do something similar where a user submits a requests through the web app that a document is generated, and we offload the document generation to a background job, and use a Notification to notify the user that the job is done.

  As as you noted in a previous response, if your job isn't modifying an data in the database, then the only likelihood would be the Background Job manager keeping track of the internal state of it's record. We see this in the initial call stack that you provided

  at Abp.Domain.Uow.UnitOfWorkManagerExtensions.WithUnitOfWorkAsync[TResult](IUnitOfWorkManager manager, Func1 action, UnitOfWorkOptions options)
  **at Abp.BackgroundJobs.BackgroundJobStore.UpdateAsync(BackgroundJobInfo jobInfo)**
  at Abp.BackgroundJobs.BackgroundJobManager.TryUpdateAsync(BackgroundJobInfo jobInfo)
  

  Looking at the code you provided, I see you are explicitly starting a new UnitOfWork

  await _unitOfWorkManager.WithUnitOfWorkAsync(async () =>
              {
              ...
  

  I haven't used that function before, so I don't know if that's added in a newer version of ABP or ANZ that I'm just not on yet. (I'm still on .NET Core 3.1). I'm definitely not an expert in Unit Of Work here, but what I'm thinking is that you may have a conflict between your UOW and the UOW that is handled by BackgroundJobManager.

  Comparing your code to mine, I start my ExecuteAsync function with this:

              using (var uow = _unitOfWorkManager.Begin())
              {
                  using (_unitOfWorkManager.Current.SetTenantId(args.TenantId))
                  {
                      using (AbpSession.Use(args.TenantId, args.UserId))
                      {
  

  So I explicitly create my own new UnitOfWork within my function execution.

  Additionally, you have await _unitOfWorkManager.WithUnitOfWorkAsync(async () => where I don't. I'm curious if you changed that _unitOfWorkManager call from async to sync, if that could possibly have any impacts.

  Another observation, I don't know if it's required or not, but I call AbpSession.Use so that my UserId & TenantId are set in AbpSession within the scope of my transaction, just in case anything else downstream attempts to rely on AbpSession. I don't know if that's required or not.

  Another observation, I don't see any logging in your class, so I'd be curious to inject an instance of ILogger into your background job class and to write it to a logfile, just to try and get further evidence if this background job is possibly being executed multiple times concurrently.

  And lastly, if you aren't using Hangfire, I'd be curious to know if you have this same issue if you switched your BackgroundJob processing to leverage Hangfire instead of the ABP Native BackgroungJob Manager.

  Hope any of those ideas or insights might help, -Brian

  Save Cancel
• 

  0

  Ricavir created 3 years ago

  Hi @ismcagdas : I've sent you the files by mail

  Hi @sedulen : Thank you for your help. I'm not using hangfire, only using all ABP / ANZ native capabilities. My app has started some years ago and the update process according to current ZERO version is specific. Some time ago, we took the decision to update ABP packages, .NET and Angular only. We only take new ABP features if needed on our side. We stopped updating Metronic theme by version 5 also (as it looks better from our opinion). About background jobs and workers, we are using latests updates from ABP. We always look at the current implementation of ABP team before changing something. All that stuff has been changed from sync to async some months ago. Doing sync stuff is not recommended with latest versions of .NET. This issue is not systematic and is running well most of the time... but sometimes, we are having these logs for background jobs and background workers. I asked ABP team to write some guidelines about multi instance management specialy for theses cases (jobs, workers, cache management...) as it seems to be causing lots of issues...

  Save Cancel
• 

  0

  ismcagdas created 3 years ago

  Support Team

  Hi @ricavir

  Thanks, I have received the email.

  I asked ABP team to write some guidelines about multi instance management specialy for theses cases (jobs, workers, cache management...) as it seems to be causing lots of issues...

  We will create a guideline for this, thank you for pointing it out.

  Save Cancel