Base solution for your next web application
Open Closed

Dashboard Exceptions with MultiNode Deployment #11673

User avatar
0
JapNolt created

Our project is hosted in azure and is replicated across 3 nodes. There seems to be an issue with the Dashboard where we receive a "page not found" exception on a brand new dashboard(or after resetting to default) when clicking add new widget. The default dashboard definition creates a page with a unique id (guid), which seems to be cached locally but not in our redis cache, and so wont be the same id when the next api call is fetched and it hits a different node.

This video shows the issue: https://landiscomputer-my.sharepoint.com/:v:/g/personal/gavin_landistechnologies_com/EViEe61wZJlCvFYme4gIsdUBEknxr4_v-JVvViMdSOD4KQ?e=xg52jc

Also this shows every third request, works as expected, when the api hits the first node that generated the dashboard definition: https://landiscomputer-my.sharepoint.com/:v:/g/personal/gavin_landistechnologies_com/EVek59wkW5FHg5bQryj1vGIByoxZBx8qe5DQ5-ul63Zhpw?e=LwXmkL

What is the recommend solution to this?

Go to accepted answer
15 Answer(s)
  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @japnolt

    Sorry for the late reply. Are you using Redis on your app ?

  • User Avatar
    0
    JapNolt created

    Hi @japnolt

    Sorry for the late reply. Are you using Redis on your app ?

    Yes we are using redis on our app as well as using the per request redis.

  • User Avatar
    0
    JapNolt created

    Any update on this?

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @japnolt

    Could you also share the related error message from the Log file ? If you can't share it here, you can send the log fiel to [email protected].

    Thanks,

  • User Avatar
    0
    JapNolt created

    Here is the related error logs:

    Audit Logs

    Input Prams: { "input": { "dashboardName": "TenantDashboard", "application": "Angular", "pageId": "Page3e4759efcafa4ed5a09096a59fa2a175" } }

    Error state: Abp.UI.UserFriendlyException: [Unknown page] at DashboardCustomization.DashboardCustomizationAppService.GetAllAvailableWidgetDefinitionsForPage(GetAvailableWidgetDefinitionsForPageInput input) in C:**\2**\src***.Application\DashboardCustomization\DashboardCustomizationAppService.cs:line 205 at lambda_method10582(Closure , Object ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker invoker) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

    UserFriendlyException.Code:0 UserFriendlyException.Details:

    App Service(Kudu)

    WARN 2023-08-14 13:44:42,518 [62 ] Mvc.ExceptionHandling.AbpExceptionFilter - [Unknown page] Abp.UI.UserFriendlyException: [Unknown page] at ***.DashboardCustomization.DashboardCustomizationAppService.GetAllAvailableWidgetDefinitionsForPage(GetAvailableWidgetDefinitionsForPageInput input) in C:\**\**\2\s\src\***.Application\DashboardCustomization\DashboardCustomizationAppService.cs:line 205 at lambda_method10582(Closure , Object ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @japnolt

    Sorry for this problem. It seems like some parts of the Dashboard Management is not desiged to work on multi instance environments. We will fix this problem, please follow https://github.com/aspnetzero/aspnet-zero-core/issues/4968

  • User Avatar
    0
    JapNolt created

    Alright, thanks. Is there any workaround we can use for now for this?

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @japnolt

    At the moment, I'm not sure but we will try to work on this issue as soon as possible.

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @japnolt

    Could you check if this PR solves your problem https://github.com/aspnetzero/aspnet-zero-core/pull/4973

  • User Avatar
    0
    JapNolt created

    I forgot to mention this earlier, but we are currently on ANZ 11.3. But I think the problem we're facing still exists in the latest version of ANZ although we haven't tested it.

    To work around the issue, we added a database migration to insert a row into the AbpSettings table so that a default setting is supplied by the database instead of from code. Here's our migration code that adds an Application level setting (not tenant or user). migrationBuilder.Sql(@"INSERT INTO AbpSettings(CreationTime, Name, Value) SELECT GETUTCDATE(), 'App.DashboardCustomization.Configuration.Angular.TenantDashboard', '<dashboard_definition_as_JSON_string' ");

    I think this problem could still exist even if in your PR fix, but I didn't test. I will try to explain what I think the problem is:

    • in AppSettingsProvider, settings are defined for the Dashboard in GetDashboardSettings
    • in GetDefaultAngularTenantDashboardView, for example, a new Dashboard object is created, with a new Page.
    • In the constructor of Page, a new GUID is generated..
    • This default setting is generated on each node in the cluster, but each node has a different value for the default setting.
    • This creates a problem in the DashboardCustomizationAppService in GetDashboardWithAuthorizedWidgets because await SettingManager.GetSettingValueAsync(GetSettingName(application, dashboardName)); will return a different value, depending on which node the request gets routed to.

    Let me know if you have any more questions.

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi,

    Settings are cached, so it shouldn't be a problem. I think the problem is keeping dashboard definitions in an in-memory collection. If you have time, please try my PR and see if it works.

  • User Avatar
    0
    JapNolt created

    @ismcagdas I disagree. The work around that I presented above actually works BECAUSE we've inserted a Setting into the database. BUT if there is NO user level setting, NO tenant level setting, and NO application level setting saved in the database, then ANZ uses the default setting value defined in AppSettingProvider. And, as I described above, the default setting is different on each node because you are generating a unique page id for the default setting.

    The error we're receiving is throw new UserFriendlyException(L("UnknownPage"));. From my understanding, this error has nothing to do with DashboardDefinition. This error only happens if the page id that is being requested by the user in GetAllAvailableWidgetDefinitionsForPage is not available in the response from GetUserDashboard. And this can happen in the scenario I described in the paragraph above.

  • User Avatar
    0
    JapNolt created

    Any update?

  • User Avatar
    0
    ismcagdas created
    Support Team

    Hi @japnolt

    Sorry, we couldn't reproduce this yet on our end. We will update here once we re-examine this problem.

  • User Avatar
    0
    ismcagdas created
    Support Team
Assignee
No one
Labels
Non yet