INotificationPublisher PublishAsync hanging & not completing #11604

Good morning everyone,

A new issue has come up recently in my production environment that I'm struggling to understand. I have a LOT of Hangfire background jobs, which have been running very smoothly for a very long time.

In the last ~5-6 days, I have been experiencing an issue where 1 specific background job hangs and never completes.
This background job builds a zip file, stores it in blob storage, sends an email to the intended recipient, and then publishes a ABP notification to the internal user who initiated the job.

I see the zip file being created and I see the email being sent. However, in the Hangfire dashboard, the background job never completes. When I look at my applicaiton logs, it looks like my code reaches the call to INotificationPublisher PublishAsync, but never continues.

I am using Azure SQL for my RDBMS, and when I look at my SQL DB, and I look at the transaction locks on the AbpNotifications table, I see that a table lock exists:

SELECT * FROM sys.dm_tran_locks
  WHERE resource_database_id = DB_ID()
  AND resource_associated_entity_id = OBJECT_ID(N'dbo.AbpNotifications');

I'm not seeing any evidence of deadlocks in my Azure SQL database, so it's unclear to me what would be causing this to hang.

I'm also finding that this seems to be somewhat intermittent & inconsistent, where some instances of this job complete just fine, while others hang like this and never complete.

This seems to be specific to my Azure SQL database, and I will continue to investigate, but I was curious if anyone else has encountered this kind of behavior.

Thanks! -Brian