Good afternoon,
I'm on an older version of ABP (v4.10), and while I am working on upgrading to a newer version, I'm running into a new and rather challenging issue.
I have been using Hangfire for my background jobs for years now, and only very recently, I have 1 job in particular that seems to be getting stuck.
The background job is extremely simple. It takes a list of files that a user has selected, packages them up into a zipfile, and then sends a user a link to download the zipfile and sends the user who submitted the request a notification that the zipfile was sent.
In an effort to troubleshoot the issue and ensure I don't send emails with the zipfile being constructed, I am explicitly beginning and completing UoW using UnitOfWorkManager
Here is my code (logging removed for readability)
public class BuildZipfileBackgroundJob : AsyncBackgroundJob<BuildZipfileArgs>, ITransientDependency
{
// ... properties & constructor removed to condense the code...
protected override async Task ExecuteAsync(BuildZipfileArgs args)
{
if (args.CreateNewPackage)
{
using (var uow = _unitOfWorkManager.Begin())
{
using (_unitOfWorkManager.Current.SetTenantId(args.TenantId))
{
using (AbpSession.Use(args.TenantId, null))
{
await _zipFileManager.BuildZipFileAsync(args.RequestId);
await uow.CompleteAsync();
}
}
}
}
if (args.Send)
{
using (var uow = _unitOfWorkManager.Begin())
{
using (_unitOfWorkManager.Current.SetTenantId(args.TenantId))
{
using (AbpSession.Use(args.TenantId, null))
{
await _zipFileManager.SendZipFileAsync(args.RequestId);
await uow.CompleteAsync();
}
}
}
}
}
}
As I've been trying to troubleshoot what is happening, I have added Logger statements, and what I am seeing is that the first await uow.CompleteAsync() is never completing. My logging stops and I never see any more logging after that for this job.
Additionally, if other Users queue up more requests for this background job, the issue continues to group, and more Hangfire jobs become clogged.
I have read some about the Configuration.UnitOfWork.Timeout setting, and right now I'm not changing that value in my startup configuration.
What I'm seeing is that the job hangs for hours, so Hangfire thinks the job is an orphaned job, and queues another instance of the same job, which further compounds the issue. Ultimately, Hangfire queues up the same job ~5-6x which then causes problems with Hangfire polling it's JobQueue table, and that breaks the entire Hangfire queue processing, leaving jobs enqueued but never processing.
What I'm struggling with is why is await uow.CompleteAsync() getting stuck and never completing. It seems like there is a transaction lock or deadlock that could be causing the problem, but I've been really struggling to figure out the root cause (and resolution).
Given my version of ABP (v4.10.0), I'm thinking perhaps it's the use of AsyncHelper. Since my class inherits from AsyncBackgroundJob, the job execution is:
public override void Execute(TArgs args)
{
AsyncHelper.RunSync(() => ExecuteAsync(args));
}
I don't think I have a way around this, and I have many other background jobs that inherit from AsyncBackgroundJob that do not have this problem.
I've thought about using UnitOfWorkOptions when calling .Begin(), and adjusting the Timeout and the Scope attributes, but I feel like I'm just swatting blindly.
My environment is deployed in Azure, using Azure Storage for files and Azure SQL for the RDBMS.
Any ideas or suggestions you can offer would be greatly appreciated! Thanks, -Brian
Good morning everyone,
A new issue has come up recently in my production environment that I'm struggling to understand. I have a LOT of Hangfire background jobs, which have been running very smoothly for a very long time.
In the last ~5-6 days, I have been experiencing an issue where 1 specific background job hangs and never completes.
This background job builds a zip file, stores it in blob storage, sends an email to the intended recipient, and then publishes a ABP notification to the internal user who initiated the job.
I see the zip file being created and I see the email being sent.
However, in the Hangfire dashboard, the background job never completes.
When I look at my applicaiton logs, it looks like my code reaches the call to INotificationPublisher PublishAsync, but never continues.
I am using Azure SQL for my RDBMS, and when I look at my SQL DB, and I look at the transaction locks on the AbpNotifications table, I see that a table lock exists:
SELECT * FROM sys.dm_tran_locks
WHERE resource_database_id = DB_ID()
AND resource_associated_entity_id = OBJECT_ID(N'dbo.AbpNotifications');
I'm not seeing any evidence of deadlocks in my Azure SQL database, so it's unclear to me what would be causing this to hang.
I'm also finding that this seems to be somewhat intermittent & inconsistent, where some instances of this job complete just fine, while others hang like this and never complete.
This seems to be specific to my Azure SQL database, and I will continue to investigate, but I was curious if anyone else has encountered this kind of behavior.
Thanks! -Brian
Hello! It's been a very long time since I've posted (or answered any question). Apologies for that.
I did have a question I was hoping that someone could help me with I have a feature in my application UI to "open in a new window" for a document viewer. The new window that opens doesn't need the header / footer / notifications / signalr / or any of the platform UI wireframe. It truly just needs to be this 1 component.
I feel like I should know how to do this, but I'm just drawing a blank.
My angular chops aren't where they should be I guess.
Has anyone else had this need and had a solution?
There are definitely some pieces that I'll need like Features, Permissions, Localized Text, etc...
my "Stand Alone" component still inherits from AppComponentBase and right now I'm just using CSS to hide the header, footer, and to maximize the body of the page to be full window.
In my browser's Network traffic, I still see a LOT of communications that I don't need, like websockets (signalr), Profile Picture, Notifications, Linked Users, etc, ...
Any recommendations on how to cut that out of the page load would be incredibly helpful
Thanks! -Brian
In performing a security audit of an application I manage, I observed that in the AbpUserConfiguration/GetAll endpoint, in the response json, under "setting.values", all of the application settings are returned. In reviewing the custom AppSettings that I have added to this platform, there are several that I do not want returned to any public / anonymous external client. Additionally, I have some settings that I do not want returned to authorized / authenticated clients either. These settings are strictly accessible under the Administration UI for setting management, which are then used only server-side for processing and execution.
As an idea, I was thinking either in the AppSettings or in SettingDefinition, adding the ability to add an authorization boundary, similar to MultiTenancySides, and then a set of optional required permissions.
The authorization boundary would drive AbpUserConfiguration for retrieving settings, and would only return those settings that were appropriately defined (or attributed).
So when loading the login page, I would only load those settings that were made available to anonymous visitors.
Then when loading an authenticated page, I would only load those settings that were available anonymous and authorized visitors, and that matched the current user's permissions set.
I do recognize for AbpUserConfiguration that I can define my own Controller to customize this behavior. That being said, I thought that this could be a valuable feature to include in the base ABP or ANZ frameworks.
Thoughts?
Thanks! -Brian
Good morning all, and Happy Monday! I was interested to know if anyone had implemented a Rate Limiting strategy for AppServices or Controllers?
I had found this library: https://github.com/stefanprodan/AspNetCoreRateLimit But looking over the documentation, it appears to be heavily driven by appSettings.json. Comparing that to how the dynamic endpoints are generated through ANZ's AppService architecture, I wasn't sure that this was a good fit.
I also saw that it used IDistributedCache and wasn't sure how that would work in parallel with ANZ's CacheManager.
I am mainly interested in exploring a Rate Limiting implementation against the public endpoints. I know I can prevent DDoS attacks and other
Thanks! -Brian
Please answer the following questions before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.
I am worrking with a potential partner for my platform, and they are asking about delivering an SDK. I would like to do this using my Azure CI/CD pipelines and build automation.
Since ANZ uses Swashbuckle to generate the Swagger / OpenAPI specification, I had found Swashbuckle.AspNetCore.Cli https://www.nuget.org/packages/Swashbuckle.AspNetCore.Cli https://github.com/domaindrivendev/Swashbuckle.AspNetCore
the following instructions that I have read in other articles, I have done the following:
dotnet new tool-manifest
dotnet tool install --version 6.3.1 Swashbuckle.AspNetCore.Cli --ignore-failed-sources
dotnet swagger tofile --output api.json bin/Debug/net6.0/MyProject.Web.Host.dll v1
unfortunately I receive the following error:
Unhandled exception. Castle.MicroKernel.ComponentNotFoundException: No component for supporting the service Abp.AspNetCore.Configuration.AbpAspNetCoreConfiguration was found
at Castle.MicroKernel.DefaultKernel.Castle.MicroKernel.IKernelInternal.Resolve(Type service, Arguments arguments, IReleasePolicy policy, Boolean ignoreParentContext)
at Castle.MicroKernel.DefaultKernel.Resolve(Type service, Arguments arguments)
at Castle.Windsor.WindsorContainer.Resolve[T]()
at Abp.Dependency.IocManager.Resolve[T]()
at Abp.AspNetCore.Mvc.Providers.AbpAppServiceControllerFeatureProvider.IsController(TypeInfo typeInfo)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerFeatureProvider.PopulateFeature(IEnumerable1 parts, ControllerFeature feature)
at Microsoft.AspNetCore.Mvc.ApplicationParts.ApplicationPartManager.PopulateFeature[TFeature](TFeature feature)
at Microsoft.AspNetCore.Mvc.ApplicationModels.ControllerActionDescriptorProvider.GetControllerTypes()
at Microsoft.AspNetCore.Mvc.ApplicationModels.ControllerActionDescriptorProvider.GetDescriptors()
at Microsoft.AspNetCore.Mvc.ApplicationModels.ControllerActionDescriptorProvider.OnProvidersExecuting(ActionDescriptorProviderContext context)
at Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider.UpdateCollection()
at Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider.Initialize()
at Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider.get_ActionDescriptors()
at Microsoft.AspNetCore.Mvc.ApiExplorer.ApiDescriptionGroupCollectionProvider.get_ApiDescriptionGroups()
at Swashbuckle.AspNetCore.SwaggerGen.SwaggerGenerator.GetSwagger(String documentName, String host, String basePath)
at Swashbuckle.AspNetCore.Cli.Program.<>c.<Main>b__0_4(IDictionary2 namedArgs) in C:\projects\ahoy\src\Swashbuckle.AspNetCore.Cli\Program.cs:line 82
at Swashbuckle.AspNetCore.Cli.CommandRunner.Run(IEnumerable1 args) in C:\projects\ahoy\src\Swashbuckle.AspNetCore.Cli\CommandRunner.cs:line 68
at Swashbuckle.AspNetCore.Cli.CommandRunner.Run(IEnumerable1 args) in C:\projects\ahoy\src\Swashbuckle.AspNetCore.Cli\CommandRunner.cs:line 59
at Swashbuckle.AspNetCore.Cli.Program.Main(String[] args) in C:\projects\ahoy\src\Swashbuckle.AspNetCore.Cli\Program.cs:line 121
I have pulled the source code for version 6.3.1 of Swashbuckle.AspNetCore.Cli, and Program.cs does some interesting things with CommandRunner and SubCommands, which I'm not very familiar with.
Ultimately it looks like it's trying to run the dotnet exec process on the assembly that contains the Program & Startup for the web app.
Has anyone tried using this Swashbuckle Cli before? Is it possible to get this to work, or am I running into a wall?
Thanks! -Brian
Please answer the following questions before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.
Hello everyone! I realize I've been silent for a little while. Life has kept me pretty busy this past month. I know I owe a few people some outstanding action items, and I will work to get those done next wek.
In the meantime, I have a question. In working with a partner / reseller on my product, their security team did an audit of my database schema, and per their security requirements, they need the AbpUsers.Username field to be stored encrypted.
I'm comfortable doing the work to support that, but I was curious if: a.) this was already implemented in a later ABP framework version OR b.) if anyone else had already considered this, and if so, how they approached it. I see in UserManager and LoginManager, that they inherit from the AbpUserManager and AbpLogInManager classes, where the methods are defined as virtual. so I should be able to simply override the methods that I need, add my encryption / decryption where need it, and move on.
Before I started down this path I wanted to check and ask.
Thoughts?
Thanks! -Brian
Please answer the following questions before submitting an issue.
I'm troubleshooting a real weird issue I'm having that seems to be environment related, but I'm not sure why it would be. I deploy using linux docker container images, and I've got a reseller now who's also deploying my application. I have a built docker image that I'm running in ~4 different environments (locally, hosted DEV, hosted UAT, and finally the reseller's hosted UAT)
Locally on my laptop, and in my 2 hosted environments, my app works great. In my reseller's hosted UAT environment, I'm seeing a really strange behavior where the UnitOfWork for the AppService method is Disposing before the EntityCreatingEventHandlers are done. I have logic in my AppService method that tries to optimize how a file is handled. The AppService medthod does some data manipulation using a Stream. When the EntityCreatingEventHandler fires, that file is posted to Azure Blob Storage. But to help keep the filesystem clean, I have attached to the UnitOfWork.Disposed event, where I delete the file from the filesystem.
Looking at my logfiles for where there are issues, the UnitOfWork is Disposing and deleting the file from the filesystem before my EntityCreatingEventHandler completes. I've been very careful to make sure all of my async/await calls are correct, so that I'm not forgetting to await an async call.
What's also very bizarre to me is that the same docker image w/ the same runtime ENV Var configuration would behave correctly in 3 environments, but incorrectly in this 4th environment.
Just to make sure I'm not misunderstanding how this is supposed to work, if I have IEventHandler or IAsyncEventHandler classes that attach to the EntityCreatingEventData<T>, that should execute before my UnitOfWork completes? Is this correct, or am I wrong here?
Thanks! -Brian
Please answer the following questions before submitting an issue.
Would anyone benefit from expanding the AbpTenants object and UI customization to include a per-tenant custom favicon? I have a use-case where a tenant would like to have a custom favicon.
I'd be happy to do the work and submit it for PR
Thanks! -Brian
In continuing to investigate the performance issues I'm seeing, I've been capturing .dump files of my running system under load. In the latest .dump file I captured, the Diagnostics Analysis revealed somthing very eye-catching
In that screenshot, where it provides the hyperlinks in the Analysis summary, if you click on one of those hyperlinks, it opens the Parallel Tasks view, where I see:
is there anything specific in ABP or ANZ that requires using NewtonSoft for JSON serialization & deserialization at the WebAPI level? I was reading this post on the NewtonSoft.Json github repo https://github.com/JamesNK/Newtonsoft.Json/issues/1795
Curious what everyone's thoughts are. Thanks! -Brian