Hi Team, Our application has been experiencing the following Hangfire error: Microsoft.Data.SqlClient.SqlException (0x80131904): The INSERT statement conflicted with the FOREIGN KEY constraint "FK_HangFire_State_Job". The conflict occurred in database "PortalSystemConfig", table "HangFire.Job", column 'Id'. Prior to September 29, this issue occurred sporadically. However, beginning September 29, the frequency increased significantly, with a large number of Hangfire background-job creation attempts failing. Internal investigation confirmed that the failures were occurring while Hangfire was attempting to create jobs and insert their associated state records. The current leading hypothesis is a race condition in which the corresponding HangFire.Job row is deleted or expired before the HangFire.State insert completes. High job-processing volume and Hangfire cleanup/expiration activity are being investigated as possible contributors; the root cause has not yet been conclusively established. As an immediate mitigation to restore application functionality, we temporarily disabled the FK_HangFire_State_Job foreign key constraint. Following this change, the errors stopped and job processing resumed. This is strictly a temporary mitigation and not the intended permanent solution. We need assistance determining the underlying cause and the appropriate permanent remediation before re-enabling the foreign key constraint. In particular, we would like to understand whether this could be related to Hangfire version/schema behavior, job expiration and cleanup, SQL Server configuration/isolation, or another concurrency issue. The Version of aspnetzero we are running is v13.1.0
7 Answer(s)
-
0
Hi @OriAssurant
Could you share your Hangfire version as well ? By the way, is your app running as 1 instance or multiple instances ?
Thanks,
Markdown is supportedCopy & paste or drag & drop images (max 30 MB per image) -
0
The current Hangfire version is => • Abp.HangFire.AspNetCore = 9.1.2 • Hangfire.SqlServer = 1.8.9
Our app is running on 3 instances in production.
Markdown is supportedCopy & paste or drag & drop images (max 30 MB per image) -
0
Hi @OriAssurant
Could you upgrade Hangfire to 1.8.18 (or to latest is better) and see if it fixes the problem. There are severel concurrency fixes on Hangfire side after 1.8.9 .
Markdown is supportedCopy & paste or drag & drop images (max 30 MB per image) -
0
[ismcagdas] said: Hi @OriAssurant
Could you upgrade Hangfire to 1.8.18 (or to latest is better) and see if it fixes the problem.
There are severel concurrency fixes on Hangfire side after 1.8.9 .
Markdown is supportedCopy & paste or drag & drop images (max 30 MB per image) -
0
[ismcagdas] said: Hi @OriAssurant
Could you upgrade Hangfire to 1.8.18 (or to latest is better) and see if it fixes the problem.
There are severel concurrency fixes on Hangfire side after 1.8.9 .
Markdown is supportedCopy & paste or drag & drop images (max 30 MB per image) -
0
Hi @ismcagdas what version of aspnetzero has the target version of Hangfire? i don't see Hangfire version mentioned in change-logs https://docs.aspnetzero.com/aspnet-core-angular/latest/Change-Logs
Markdown is supportedCopy & paste or drag & drop images (max 30 MB per image) -
0
Hi,
We regularly upgrade Hangfire, you can check https://www.nuget.org/packages/Abp.Hangfire package to see which Hangfire version is used by which ABP version.
For version > 10.5, you can check your private ABP NuGet feed.
Markdown is supportedCopy & paste or drag & drop images (max 30 MB per image)
