I am familiar with this problem. This is not specific to the Azure SignalR service, but rather now the internal ANZ OnlineClientStore works.
When a client registers to your SignalR Hub, evening when using the Azure SignalR service, the ANZ framework stores the ClientId for that connected client in an OnlineClientStore class. By default, this class uses an InMemory Dictionary, which is not shared across your instances.
So even though, technically, the Client is connected through the Azure SignalR service, the sending of the message goes from Server to Client, so the Server needs to know who is getting which message when a new message is published. (typical Pub / Sub model). When ANZ Notification Publication classes try to publish a new message to a user via Notifications, the OnlineClientStore on instance B isn't shared with the OnlineClientStore on instance A.
To solve this, you need to replace the IOnlineClientStore service with something that supports a distributed store. I have developed code that supports 3 possible distributed stores:
SQL Pro - it's transactional and it's super easy to implement Con - it's in your db, so it's the slowest possible implementation, plus it's potentially transient data being stored in your db*
Redis Pro - also super simple to implement Con - could encounter timeouts due to high network I/O, and doesn't really fit with the ANZ PerRequestRedisCache design. Another con is that even if you try to use KeySpace events within Redis and keep the cache sync'd locally across multiple instances as local in-memory cache, you could still possibly encounter timing issues. So IMHO this is the least reliable.
Azure Storage Account (Tables) Pro - also transactional and super easy to implement. No timing issues, like Redis and no transient data bloat in your db Con - it's 1 more 3rd party service that you have to manage (connectionStrings, security, ...)
Are you familiar with replacing ANZ core classes on Module initialization? If not, I can try to share some of my code with you.
@ismcagdas - do you think this is something that the ABP / ANZ framework could benefit from? I'm happy to contribute.
I hope this helps! -Brian
Here is a scrubbed version of my DEV CI pipeline. I have a separate release pipeline that handles the CD aspect of my infrastructure.
As I noted before, I use a lot of variables.
# Variable 'BROWSERSLIST_IGNORE_OLD_DATA' was defined in the Variables tab
# Variable 'BuildAngularCliVersion' was defined in the Variables tab
# Variable 'BuildConfiguration' was defined in the Variables tab
# Variable 'BuildDotNetCoreVersion' was defined in the Variables tab
# Variable 'BuildEntityFrameworkVersion' was defined in the Variables tab
# Variable 'BuildNodeVersion' was defined in the Variables tab
# Variable 'BuildNugetVersion' was defined in the Variables tab
# Variable 'BuildUseGulp' was defined in the Variables tab
# Variable 'BuildUseNuGet' was defined in the Variables tab
# Variable 'ProjectName' was defined in the Variables tab
# Variable 'yarnSetVersionEnabled' was defined in the Variables tab
# Variable 'yarnVersion' was defined in the Variables tab
name: $(date:yyyyMMdd)$(rev:.r)
resources:
repositories:
- repository: self
type: git
ref: refs/heads/develop
jobs:
- job: Phase_1
displayName: Build and Package
cancelTimeoutInMinutes: 1
pool:
vmImage: ubuntu-20.04
steps:
- checkout: self
- task: YarnInstaller@3
displayName: Install Yarn Version
condition: and(succeeded(), eq(variables.yarnSetVersionEnabled, true))
inputs:
versionSpec: $(yarnVersion)
- task: CmdLine@2
displayName: Check Yarn Version
condition: and(succeeded(), eq(variables.yarnSetVersionEnabled, true))
inputs:
script: yarn --version
- task: UseDotNet@2
displayName: Use .NET Core
inputs:
version: $(BuildDotNetCoreVersion)
- task: NuGetToolInstaller@1
displayName: Use NuGet
condition: and(succeeded(), eq(variables.BuildUseNuGet, true))
inputs:
versionSpec: $(BuildNugetVersion)
- task: NuGetCommand@2
displayName: NuGet restore
inputs:
solution: $(ProjectName).Web.sln
- task: NodeTool@0
displayName: Use Node
inputs:
versionSpec: $(BuildNodeVersion)
- task: DotNetCoreCLI@2
displayName: Initialize EntityFrameworkCore
inputs:
command: custom
custom: tool
arguments: install dotnet-ef --global --version $(BuildEntityFrameworkVersion) --ignore-failed-sources
- task: DotNetCoreCLI@2
displayName: Build
inputs:
projects: $(ProjectName).Web.sln
arguments: --configuration $(BuildConfiguration) --no-restore
- task: Npm@1
displayName: Install Angular CLI
inputs:
command: custom
workingDir: src/$(ProjectName).Web.Host
verbose: false
customCommand: install -g @angular/cli@$(BuildAngularCliVersion)
- task: Yarn@3
displayName: Yarn Install
inputs:
projectDirectory: src/$(ProjectName).Web.Host
arguments: install --verbose
- task: Yarn@3
displayName: Yarn Add Gulp
condition: and(succeeded(), eq(variables.BuildUseGulp, true))
inputs:
projectDirectory: src/$(ProjectName).Web.Host
arguments: add gulp --dev
- task: CmdLine@2
displayName: Check Node & Angular/CLI versions
inputs:
script: node ./node_modules/@angular/cli/bin/ng --version
workingDirectory: src/$(ProjectName).Web.Host
failOnStderr: true
- task: CmdLine@2
displayName: Check Gulp version
condition: and(succeeded(), eq(variables.BuildUseGulp, true))
inputs:
script: node ./node_modules/gulp/bin/gulp --version
workingDirectory: src/$(ProjectName).Web.Host
failOnStderr: true
- task: CmdLine@2
displayName: Build Gulp
condition: and(succeeded(), eq(variables.BuildUseGulp, true))
inputs:
script: node ./node_modules/gulp/bin/gulp build
workingDirectory: src/$(ProjectName).Web.Host
failOnStderr: true
- task: CmdLine@2
displayName: Build Angular
inputs:
script: node --max-old-space-size=8192 ./node_modules/@angular/cli/bin/ng build --progress=false --configuration=$(BuildConfiguration) --output-path=$(Build.ArtifactStagingDirectory)/temp/wwwroot --source-map=false
workingDirectory: src/$(ProjectName).Web.Host
- task: DotNetCoreCLI@2
displayName: Publish Website
inputs:
command: publish
publishWebProjects: false
projects: src/$(ProjectName).Web.Host/$(ProjectName).Web.Host.csproj
arguments: -c $(BuildConfiguration) -o $(Build.ArtifactStagingDirectory)/temp /p:PublishProfile=$(BuildConfiguration) --no-restore
zipAfterPublish: false
modifyOutputPath: false
- task: ArchiveFiles@2
displayName: Zip API
inputs:
rootFolderOrFile: $(Build.ArtifactStagingDirectory)/temp
includeRootFolder: false
sevenZipCompression: 5
archiveFile: $(Build.ArtifactStagingDirectory)/$(ProjectName)_$(BuildConfiguration)_$(Build.BuildNumber).zip
- task: DeleteFiles@1
displayName: Delete Temp Folder
inputs:
SourceFolder: $(Build.ArtifactStagingDirectory)
Contents: temp
- task: PublishBuildArtifacts@1
displayName: Publish Web Artifacts
inputs:
ArtifactName: Web
- task: DotNetCoreCLI@2
displayName: Create SQL Migration Scripts
inputs:
command: custom
custom: ef
arguments: migrations script -i -p src/$(ProjectName).EntityFrameworkCore/$(ProjectName).EntityFrameworkCore.csproj -o $(Build.ArtifactStagingDirectory)/Migrations/$(ProjectName).Core_$(BuildConfiguration)_migrations_$(Build.BuildNumber).sql
- task: CopyFiles@2
displayName: Copy SQL files into Sql Artifact
inputs:
SourceFolder: sql
TargetFolder: $(Build.ArtifactStagingDirectory)/Migrations/
- task: PublishBuildArtifacts@1
displayName: Publish Sql Artifact
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)/Migrations/
ArtifactName: SQL
...
The only thing that might not work out of the box for you would be the NuGet Restore command, if you happen to use a private Artifact Feed instead of all public NuGet repositories. It's fairly simple to wire those up as well, if you are using private Artifact Feeds.
This framework has worked beautifully for me years. I have added to it to include building docker images and tagging + pushing docker images to an Azure ACR. I have also added 3rd party code scanning services, such as WhiteSource and Snyk.
I can share more of those additions later if you would like, but those start to get into AZDO service connections and 3rd party service accounts, so it made sense to keep those out for now.
I also do not build or publish the .Public or .Migrator projects. I could, I just decided not to use them for this project. Instead we generate the .sql files using the ef migrations script command.
Lastly, I've been working on upgrading from v6.9.0 to a more current version for a while now. Sadly, I'm still on dotnet core 2.2. That is why you see many "Version" parameters and even executing the "gulp" command all being controlled by conditional variables.
This same pipeline should work for you for both dev branch and your master branch. I haven't worked with 1 .yml file as a template for multiple pipelines, so I basically got this .yml file as I wanted it, and then uploaded it twice, once to my DEV pipeline and once to my PROD pipeline, and configured the variables and source repo.
I hope this helps! Let me know if you have any question.
-Brian
@4Matrix - I'm wrapping up another feature request this morning so I should have time to get you a copy of my Azure pipeline shortly.
To note - I use a lot of Variables in my pipelines, so that they can be used across my DEV, UAT, and PROD environments. Are you comfortable working with Azure Pipelines and variables?
I should have something posted for you tomorrow. -Brian
Hi @omkarchoudhari,
A quick observation - double-check the path to the impersonation endpoint: https://devmovescoutproapi.sirva.com**//**api/TokenAuth/ImpersonatedAuthenticate
that "//" between your domain and the api path segment might be interfering. I've encountered that before.
If that doesn't solve it, what Azure service are you using to host your application? AKS? AppServices? Container Instances? And how are you configured for logging?
You should be able to see what the Internal Server Error is in your application logfiles.
Let me know if that helps, -Brian
Hi @demo.solaru,
I am curious if you tried either of my recommendations in my response. What do you see if you disable Auditing?
In your .Core project, in the PreInitialize() method of your <Project>CoreModule
Configuration.Auditing.IsEnabled = auditLogEnabled; Configuration.Auditing.Selectors.Clear();
I'm mainly curious just to see what the observed behavior is, if that could help the ASPNZ team. I am also curious, if you are running these on your local environment, where all of your resources are.
For the build you are running, is this running the code using the Visual Studio debugger with IIS Express? Or have you built and deployed the code somewhere else?
In a production-like environment, you wouldn't have the overhead of running Visual Studio and JMeter locally on the same machine that is also running your webserver and your database server . Also if you are seeing SQL timeout errors, is your database local as well? Or is the database somewhere else?
My thought is that performance testing can be a bit of a black hole (I know from experience over the last ~2-3 months). You want to establish a reliable baseline, but running everything locally is potentially a skewed test, because everything is running locally on your machine, including programs that wouldn't be involved in a production environment (like Visual Studio and JMeter). The combination of all of those programs all running on 1 machine could yield resource contention which could impact performance.
Lastly - are you running Azure Application Insights? If you are, and you have an Azure subscription, if you could report your telemetry metrics to an Azure Application Insights hosted dashboard that could give you some more insights into exactly what these requests are doing. Azure Application Insights has been a huge help for me for troubleshooting performance-related issues.
Trying to offer ideas. I hope my input helps, -Brian
Hi @pliaspzero
I will be providing my work to the ASPNZ team in the github ticket that @ismcagdas referenced. My project last week took longer than expected, but I finished it up over the weekend, so I should be able to get caught up on your question this week.
-Brian
@Hammer & @ismcagdas
I'd be willing to help with this. I've been using Azure CICD pipelines for several years now and they work great. My project last week took longer than anticipated, but I finished up over the weekend.
I have a few things that I've offered to provide to others, so it may take me a day or two to get back to this, but I will try to get you a generic version of my pipeline that should work for you.
-Brian
Hi @ismcagdas,
Sure - let me see if I can get you a redacted copy of the code that I can share.
I'll follow up in a day or two. -Brian
Hi @pliaspzero,
I have actually implemented something like this in my application. My implementation isn't ideal where I'm using a polling task in the UI to a "GetVersion" endpoint, where I compare the version to a value that is stored in localStorage. If those values don't match, I show an alert that a new version is available, and when they click the OK button, it reloads the window.
The main dependency is that any browser cacheable assets include a cache buster ?d=<value>
I can look at my implementation and see if this is something generic I can offer back to the ANZ community.
Ideally my implementation would have used SignalR & websockets rather than an explicit polling task from the browser, but perhaps the team would be able to make it "right".
Give me a day or two to wrap up something I'm working on now, and I should be able to pull you an example implementation I check for a new version. -Brian
Hi @ismcagdas ,
Just to keep everyone informed, I tried to upgrade our production environment again this weekend, and I think we have run into issues once again and we will be forced to rollback.
At this point I think we can isolate our issue to the EF Core 2 to EF Core 3 breaking changes, specifically the "Cartesean Explosion" https://github.com/dotnet/efcore/issues/18022#issuecomment-542397085 https://docs.microsoft.com/en-us/ef/core/what-is-new/ef-core-3.x/breaking-changes#eager-loading-single-query
I am looking at our code now, and specifically considering the effort required to upgrade just our dotnet libraries to dotnet core 6 and the latest ABP / ANZ dotnet code.
I will let you know how it goes. -Brian