Dashboard Exceptions with MultiNode Deployment #11673

JapNolt created one year ago

Our project is hosted in azure and is replicated across 3 nodes. There seems to be an issue with the Dashboard where we receive a "page not found" exception on a brand new dashboard(or after resetting to default) when clicking add new widget. The default dashboard definition creates a page with a unique id (guid), which seems to be cached locally but not in our redis cache, and so wont be the same id when the next api call is fetched and it hits a different node.

This video shows the issue: https://landiscomputer-my.sharepoint.com/:v:/g/personal/gavin_landistechnologies_com/EViEe61wZJlCvFYme4gIsdUBEknxr4_v-JVvViMdSOD4KQ?e=xg52jc

Also this shows every third request, works as expected, when the api hits the first node that generated the dashboard definition: https://landiscomputer-my.sharepoint.com/:v:/g/personal/gavin_landistechnologies_com/EVek59wkW5FHg5bQryj1vGIByoxZBx8qe5DQ5-ul63Zhpw?e=LwXmkL

What is the recommend solution to this?

15 Answer(s)

0

ismcagdas created one year ago
Support Team

Hi @japnolt

Sorry for the late reply. Are you using Redis on your app ?
0

JapNolt created one year ago

Hi @japnolt

Sorry for the late reply. Are you using Redis on your app ?

Yes we are using redis on our app as well as using the per request redis.
0

JapNolt created one year ago

Any update on this?
0

ismcagdas created one year ago
Support Team

Hi @japnolt

Could you also share the related error message from the Log file ? If you can't share it here, you can send the log fiel to [email protected].

Thanks,
0

JapNolt created one year ago

Here is the related error logs:

Audit Logs

Input Prams: { "input": { "dashboardName": "TenantDashboard", "application": "Angular", "pageId": "Page3e4759efcafa4ed5a09096a59fa2a175" } }

Error state: Abp.UI.UserFriendlyException: [Unknown page] at DashboardCustomization.DashboardCustomizationAppService.GetAllAvailableWidgetDefinitionsForPage(GetAvailableWidgetDefinitionsForPageInput input) in C:**\2**\src***.Application\DashboardCustomization\DashboardCustomizationAppService.cs:line 205 at lambda_method10582(Closure , Object ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker invoker) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

UserFriendlyException.Code:0 UserFriendlyException.Details:

App Service(Kudu)

WARN 2023-08-14 13:44:42,518 [62 ] Mvc.ExceptionHandling.AbpExceptionFilter - [Unknown page] Abp.UI.UserFriendlyException: [Unknown page] at ***.DashboardCustomization.DashboardCustomizationAppService.GetAllAvailableWidgetDefinitionsForPage(GetAvailableWidgetDefinitionsForPageInput input) in C:\**\**\2\s\src\***.Application\DashboardCustomization\DashboardCustomizationAppService.cs:line 205 at lambda_method10582(Closure , Object ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
0

ismcagdas created one year ago
Support Team

Hi @japnolt

Sorry for this problem. It seems like some parts of the Dashboard Management is not desiged to work on multi instance environments. We will fix this problem, please follow https://github.com/aspnetzero/aspnet-zero-core/issues/4968
0

JapNolt created one year ago

Alright, thanks. Is there any workaround we can use for now for this?
0

ismcagdas created one year ago
Support Team

Hi @japnolt

At the moment, I'm not sure but we will try to work on this issue as soon as possible.
0

ismcagdas created one year ago
Support Team

Hi @japnolt

Could you check if this PR solves your problem https://github.com/aspnetzero/aspnet-zero-core/pull/4973
0
JapNolt created one year ago
I forgot to mention this earlier, but we are currently on ANZ 11.3. But I think the problem we're facing still exists in the latest version of ANZ although we haven't tested it.

To work around the issue, we added a database migration to insert a row into the AbpSettings table so that a default setting is supplied by the database instead of from code. Here's our migration code that adds an Application level setting (not tenant or user). migrationBuilder.Sql(@"INSERT INTO AbpSettings(CreationTime, Name, Value) SELECT GETUTCDATE(), 'App.DashboardCustomization.Configuration.Angular.TenantDashboard', '<dashboard_definition_as_JSON_string' ");

I think this problem could still exist even if in your PR fix, but I didn't test. I will try to explain what I think the problem is:

in AppSettingsProvider, settings are defined for the Dashboard in GetDashboardSettings

in GetDefaultAngularTenantDashboardView, for example, a new Dashboard object is created, with a new Page.

In the constructor of Page, a new GUID is generated..

This default setting is generated on each node in the cluster, but each node has a different value for the default setting.

This creates a problem in the DashboardCustomizationAppService in GetDashboardWithAuthorizedWidgets because await SettingManager.GetSettingValueAsync(GetSettingName(application, dashboardName)); will return a different value, depending on which node the request gets routed to.

Let me know if you have any more questions.
0

ismcagdas created one year ago
Support Team

Hi,

Settings are cached, so it shouldn't be a problem. I think the problem is keeping dashboard definitions in an in-memory collection. If you have time, please try my PR and see if it works.
0

JapNolt created one year ago

@ismcagdas I disagree. The work around that I presented above actually works BECAUSE we've inserted a Setting into the database. BUT if there is NO user level setting, NO tenant level setting, and NO application level setting saved in the database, then ANZ uses the default setting value defined in AppSettingProvider. And, as I described above, the default setting is different on each node because you are generating a unique page id for the default setting.

The error we're receiving is throw new UserFriendlyException(L("UnknownPage"));. From my understanding, this error has nothing to do with DashboardDefinition. This error only happens if the page id that is being requested by the user in GetAllAvailableWidgetDefinitionsForPage is not available in the response from GetUserDashboard. And this can happen in the scenario I described in the paragraph above.
0

JapNolt created one year ago

Any update?
0

ismcagdas created one year ago
Support Team

Hi @japnolt

Sorry, we couldn't reproduce this yet on our end. We will update here once we re-examine this problem.
0

ismcagdas created one year ago
Support Team

Hi @japnolt,

Could you also try this on your project https://github.com/aspnetzero/aspnet-zero-core/pull/4973/commits/b875efc56185b08c2650d1146c9abb9a012fd486

Thanks,

Have an answer to this question? Log in and write your answer.

15 Answer(s)

Audit Logs

App Service(Kudu)