AbpUserConfigurationController/GetAll getting slower #5466 | Support Center

adevell created 6 years ago

Hi All,

As always, great framework! And very helpful, informative, and instructional coding patterns.

For the past year+ we have been developing a number of mobile apps that leverage an ASP.Net Zero based platform, operating on several server farms, as the backend. For the most part it works brilliantly. However, mobile app launch and user logins are steadily getting slower. As the platform grows, mobile app launch and logins require an increasing amount of time. Our largest mobile app now requires upwards of 60 seconds to launch. We eventually determined the primary cause for the lengthy launch and login times are the 1-2 GetAll calls per launch and/or login. GetAll calls are now ranging from 4 seconds to 39 seconds (screenshot of example calls attached). The more recent 30+ second calls have required a bump in the AbpClient FlurlClient/HttpClient TimeOut parameter, increased from 30 seconds to 60 seconds.

Have others experienced this phenomena? Is there a known quick fix? We are considering two solutions; 1) Caching GetAll results locally on mobile apps for immediate use while executing a background update call; 2) Rewrite GetAll method and supporting dependencies to better leverage cached data.

Any ideas?

Thanks in advance.

11 Answer(s)

0

strix20 created 6 years ago

Do you have the ABP pdb files?

The AbpUserConfigurationBuilder does an awful lot of work. It would be interesting to see which part of it is creating this impact. If I had to guess, my guess would be the permission and localization building.

If you don't have the ABP pdb files to debug through the ABP source code, you could always create your own implementation of the userConfigBuilder, and implement your own GetAll method exactly the same as ABP does (calling the base class methods), and then user a performance profiler to see how long each of the individual calls is taking.
0

alper created 6 years ago
Support Team

hi,

GetAll() should be completed in a second. See the <a class="postlink" href="http://demo.aspnetzero.com">http://demo.aspnetzero.com</a> 's GetAll() performance. [attachment=0:11xuu9fc]getall-performance.jpg[/attachment:11xuu9fc]

You don't need to cache GetAll() because it caches and invalidates cache when needed. So it's already developed performance focused. Maybe you can replace AbpUserConfigurationBuilder with your custom builder to see which line is time consuming. <a class="postlink" href="https://github.com/aspnetboilerplate/aspnetboilerplate/blob/dev/src/Abp.Web.Common/Web/Configuration/AbpUserConfigurationBuilder.cs">https://github.com/aspnetboilerplate/as ... Builder.cs</a>

PS: The reason why the client (mobile/browser) requests the GetAll() twice is; the first one is for non-authenticated settings like application's default language, WebApi URLs, multi-tenancy settings etc... the second GetAll() is for user-specific which returns user's settings, localizations etc...
0

adevell created 6 years ago

Thanks @strix20.

I agree, AbpUserConfigurationBuilder does a lot of work, producing a rich collection of data. And your guess is consistent with my initial guess. We have substantially increased the number of permissions and localization entries compared to the out-of-the-box numbers.

Our concern is our platform may have grown beyond the original intent of Abp. The AppPermissions file is approaching 250 lines. The primary localization xml file is well over 1,000 lines and growing rapidly. Additionally, the latest version has more than 150 entities, upwards of 500 Dtos, more than 250 Apis, and more than 100 backend bots managing data, primarily stored in Redis cache and SQL dbs, pushed to/received from service bus topics, or pushed to/received from 3rd party Apis. However, I suspect there are dozens, if not hundreds, of Abp implementations substantially larger than ours.
0

adevell created 6 years ago

Thanks @alper.

After a considerable number tracing exercises, we noticed the reason for multiple GetAll calls. And I agree with your comment regarding GetAll typically completing in less than one second. Our out-of-the-box implementation seemed to consistently fall within that duration timeframe. However, after bolting on a year and half worth of custom entities, dtos, permissions, localization entries, core services, app services, and apis, GetAll call durations have steadily increased. This became abundantly clear after troubleshooting recent client mobile app failures, which identified GetAll calls exceeding the default 30 second HttpClient time out parameter.

I definitely agree with your recommendation. Adding a few diagnostic StopWatch timers in the AbpUserConfigurationBuilder class is a good starting point. We’ll look into replacing the NuGet package reference with Abp source files and implementing microscopic duration monitoring with logging.

Thanks again for the response, insight, and suggestions.

adevell created 6 years ago

BTW – One coding pattern we have applied in our custom code that has resulted in considerable execution duration time reductions leverages Task.WhenAll(). Below is an example of before and after. In one case, we reduced execution duration from > 300ms to < 30ms.

In addition to inserting StopWatch timers, we will evaluate leveraging the multithreading pattern below.

Before:

async Task&lt;SeveralThingsDto&gt; GetSeveralThingsAsync()
        {
            return new SeveralThingsDto
            {
                things_1 = await GetThingsAsync_1(),
                things_2 = await GetThingsAsync_2(),
                things_3 = await GetThingsAsync_3(),
                things_4 = await GetThingsAsync_4(),
            };
        }

After:

async Task<SeveralThingsDto> GetSeveralThingsAsync()
        {
            var tasks = new List<Task>();

            var taskThings_1 = GetThingsAsync_1();
            tasks.Add(taskThings_1);

            var taskThings_2 = GetThingsAsync_2();
            tasks.Add(taskThings_2);

            var taskThings_3 = GetThingsAsync_3();
            tasks.Add(taskThings_3);

            var taskThings_4 = GetThingsAsync_4();
            tasks.Add(taskThings_4);

            await Task.WhenAll(tasks);

            return new SeveralThingsDto
            {
                things_1 = taskThings_1.Result,
                things_2 = taskThings_2.Result,
                things_3 = taskThings_3.Result,
                things_4 = taskThings_4.Result,
            };
        };

0

alper created 6 years ago
Support Team

Yeap Task.WhenAll() is definitely more performance. And let us know on which line your GetAll() waits so that we can think a solution for long lasting logins.

adevell created 6 years ago

Brief update:

After a bit of testing last Friday, we quickly identified a few processing bottlenecks and opportunities for improvement. All AbpUserConfigurationBuilder methods are extremely serial in nature. Although async/await statements exist, concurrent processing is not leveraged. As the list of permissions, settings, and localization options increase, loop sequence counts increase, as do execution times. Further investigation confirmed methods subsequently called by AbpUserConfigurationBuild methods are also serial in nature, such as; AbpUserManager.IsGrantedAsync.

Starting with AbpUserConfigurationBuild, we began implementing coding patterns that introduce concurrent processing as the thread pool has available threads. The resulting AbpUserConfigurationBuilder methods are listed below.

This week we plan to continue testing for possible adverse impact to thread pool, cache server, and database server operations as a result of the new concurrent processing logic. Additionally, we plan to implement additional concurrent processing logic in the methods called by AbpUserConfigurationBuilder methods, such as; AbpUserManager.IsGrantedAsync.

protected virtual async Task<AbpUserAuthConfigDto> GetUserAuthConfig()
        {
            var config = new AbpUserAuthConfigDto();

            var allPermissionNames = PermissionManager.GetAllPermissions(false).Select(p => p.Name).ToList();
            var grantedPermissionNames = new List<string>();

            config.AllPermissions = allPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");

            if (AbpSession.UserId.HasValue)
            {
                var tasks = allPermissionNames.Select(async permissionName => new { permissionName, filter = await PermissionChecker.IsGrantedAsync(permissionName) });
                var results = await Task.WhenAll(tasks);
                grantedPermissionNames = results.Where(x => x.filter).Select(x => x.permissionName).ToList();

                config.GrantedPermissions = grantedPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
            }

            return config;
        }

protected virtual async Task<AbpUserSettingConfigDto> GetUserSettingConfig()
        {
            var config = new AbpUserSettingConfigDto
            {
                Values = new Dictionary<string, string>()
            };

            var settingDefinitions = SettingDefinitionManager
                .GetAllSettingDefinitions()
                .Where(sd => sd.IsVisibleToClients);

            if (settingDefinitions != null && settingDefinitions.Count() > 0)
            {
                var tasks = settingDefinitions.Select(async settingDefinition => new { settingName = settingDefinition.Name, settingValue = await SettingManager.GetSettingValueAsync(settingDefinition.Name) });
                var results = await Task.WhenAll(tasks);
                config.Values = results.ToDictionary(k => k.settingName, v => v.settingValue);
            }

            return config;
        }

protected virtual async Task<AbpUserFeatureConfigDto> GetUserFeaturesConfig()
        {
            var config = new AbpUserFeatureConfigDto()
            {
                AllFeatures = new Dictionary<string, AbpStringValueDto>()
            };

            var allFeatures = FeatureManager.GetAll().ToList();

            if (AbpSession.TenantId.HasValue)
            {
                var currentTenantId = AbpSession.GetTenantId();

                if (allFeatures != null && allFeatures.Count() > 0)
                {
                    var tasks = allFeatures.Select(async feature => new { featureName = feature.Name, value = await FeatureChecker.GetValueAsync(currentTenantId, feature.Name) });
                    var results = await Task.WhenAll(tasks);
                    config.AllFeatures = results.ToDictionary(x => x.featureName, y => new AbpStringValueDto
                    {
                        Value = y.value
                    });
                }
            }
            else
            {
                foreach (var feature in allFeatures)
                {
                    config.AllFeatures.Add(feature.Name, new AbpStringValueDto
                    {
                        Value = feature.DefaultValue
                    });
                }
            }

            return config;
        }

adevell created 6 years ago

Another update:

We recommend not using the aggressive multitasking coding pattern previously shared. Although elegant and efficient, test results suggest introducing a couple hundred thread requests virtually simultaneously can adversely impact the server’s thread pool, slowing the server to a crawl for several seconds. After testing several different approaches, we have opted for a coding pattern that leverages a variable number of threads, but maximizes the potential thread count request to 20.

Suggested coding changes are listed below.

Our primary focus was on the GetUserAuthConfig method. Diagnostics identified this method as the slowest method by a significant margin. This method also proved to benefit greatly with the introduction of multiple concurrent threads. Since implementing the multithreading coding pattern, execution times have remained well below 10 seconds, far from the 30+ seconds previously encountered with the serial coding pattern.

The second slowest method is GetUserSettingConfig. Surprisingly, a multithreaded solution resulted in very little improvement in performance. We ran out of time to investigate further and opted to back out all multithreading coding patterns applied within GetUserSettingConfig. However, a few test runs quickly determined executing GetUerSettingConfig’s original serial coding pattern in parallel with GetUserAuthConfig’s multithreaded coding pattern shaved an extra 1-2 seconds off execution times.

Let me know what you think.

Enhanced GetAll method:

public virtual async Task<AbpUserConfigurationDto> GetAll()
        {
            var taskGetUserAuthConfig = GetUserAuthConfig();
            var taskGetUserSettingConfig = GetUserSettingConfig();

            await Task.WhenAll(taskGetUserAuthConfig, taskGetUserSettingConfig);

            return new AbpUserConfigurationDto
            {
                MultiTenancy = GetUserMultiTenancyConfig(),
                Session = GetUserSessionConfig(),
                Localization = GetUserLocalizationConfig(),
                Features = await GetUserFeaturesConfig(),
                Auth = taskGetUserAuthConfig.Result,
                Nav = await GetUserNavConfig(),
                Setting = taskGetUserSettingConfig.Result,
                Clock = GetUserClockConfig(),
                Timing = await GetUserTimingConfig(),
                Security = GetUserSecurityConfig()
            };
        }

Enhanced GetUserAuthConfig method:

protected virtual async Task<AbpUserAuthConfigDto> GetUserAuthConfig()
        {
            var config = new AbpUserAuthConfigDto();

            var allPermissionNames = PermissionManager.GetAllPermissions(false).Select(p => p.Name).ToList();
            var grantedPermissionNames = new List<string>();

            if (AbpSession.UserId.HasValue)
            {
                grantedPermissionNames = await SplitAndMergePermissionsAsync(allPermissionNames);
            }

            config.AllPermissions = allPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
            config.GrantedPermissions = grantedPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");

            return config;
        }

        async Task<List<string>> SplitAndMergePermissionsAsync(List<string> allPermissionNames)
        {
            const int maxLoopCount = 15;
            const int maxThreadCount = 20;

            var grantedPermissionNames = new List<string>();

            var tasks = new List<Task<List<string>>>();

            var totalPermissionCount = allPermissionNames.Count();

            // Split AllPermissionNames list for distribution to multiple threads.
            int threadCount;
            if (totalPermissionCount < 300)
                threadCount = (totalPermissionCount / maxLoopCount) + (totalPermissionCount % maxLoopCount > 0 ? 1 : 0);
            else
                threadCount = maxThreadCount;

            int takeCount = totalPermissionCount / threadCount;
            int takeOverage = totalPermissionCount % threadCount;
            var skipCount = 0;

            for (var i = 0; i < threadCount; i++)
            {
                // Spread overage evenly over threads until overage is exhausted.
                int takeCountPlus = takeCount;
                if (takeOverage > 0)
                {
                    takeCountPlus++;
                    takeOverage--;
                }

                var permissionNames = allPermissionNames.GetRange(skipCount, takeCountPlus);

                // Generate granted permission lists in parallel.
                tasks.Add(GetGrantedPermissionsAsync(permissionNames));

                skipCount = skipCount + takeCountPlus;
            }

            await Task.WhenAll(tasks);

            // Merge grantedPermissionNames lists to output a single list.
            foreach (var task in tasks)
            {
                if (task.Result != null)
                    grantedPermissionNames.AddRange(task.Result);
            }

            return grantedPermissionNames;
        }

        async Task<List<string>> GetGrantedPermissionsAsync(List<string> allPermissionNames)
        {
            var grantedPermissionNames = new List<string>();

            foreach (var permissionName in allPermissionNames)
            {
                if (await PermissionChecker.IsGrantedAsync(permissionName))
                {
                    grantedPermissionNames.Add(permissionName);
                }
            }

            return grantedPermissionNames;
        }

0

alper created 6 years ago
Support Team

could you send a Pull Request to the Aspnet Boilerplate repository?
0

adevell created 6 years ago

Will do.
0
VuCA created 3 years ago
Is the enhance of @advell is apply. I still have a issuess whit GetAll() too. When increase number of connection (concurrent user) => It extreamly slowly. more than 3 minute even 5 minute

Info: Azure host

Aspnet zero: 8.5.0.0

Have an answer to this question? Log in and write your answer.