AbpUserConfigurationController/GetAll getting slower #5466

11 Answer(s)

• 

  0

  strix20 created 5 years ago

  Do you have the ABP pdb files?

  The AbpUserConfigurationBuilder does an awful lot of work. It would be interesting to see which part of it is creating this impact. If I had to guess, my guess would be the permission and localization building.

  If you don't have the ABP pdb files to debug through the ABP source code, you could always create your own implementation of the userConfigBuilder, and implement your own GetAll method exactly the same as ABP does (calling the base class methods), and then user a performance profiler to see how long each of the individual calls is taking.

  Save Cancel
• 

  0

  alper created 5 years ago

  Support Team

  hi,

  GetAll() should be completed in a second. See the <a class="postlink" href="http://demo.aspnetzero.com">http://demo.aspnetzero.com</a> 's GetAll() performance. [attachment=0:11xuu9fc]getall-performance.jpg[/attachment:11xuu9fc]

  You don't need to cache GetAll() because it caches and invalidates cache when needed. So it's already developed performance focused. Maybe you can replace AbpUserConfigurationBuilder with your custom builder to see which line is time consuming. <a class="postlink" href="https://github.com/aspnetboilerplate/aspnetboilerplate/blob/dev/src/Abp.Web.Common/Web/Configuration/AbpUserConfigurationBuilder.cs">https://github.com/aspnetboilerplate/as ... Builder.cs</a>

  PS: The reason why the client (mobile/browser) requests the GetAll() twice is; the first one is for non-authenticated settings like application's default language, WebApi URLs, multi-tenancy settings etc... the second GetAll() is for user-specific which returns user's settings, localizations etc...

  Save Cancel
• 

  0

  adevell created 5 years ago

  Thanks @strix20.

  I agree, AbpUserConfigurationBuilder does a lot of work, producing a rich collection of data. And your guess is consistent with my initial guess. We have substantially increased the number of permissions and localization entries compared to the out-of-the-box numbers.

  Our concern is our platform may have grown beyond the original intent of Abp. The AppPermissions file is approaching 250 lines. The primary localization xml file is well over 1,000 lines and growing rapidly. Additionally, the latest version has more than 150 entities, upwards of 500 Dtos, more than 250 Apis, and more than 100 backend bots managing data, primarily stored in Redis cache and SQL dbs, pushed to/received from service bus topics, or pushed to/received from 3rd party Apis. However, I suspect there are dozens, if not hundreds, of Abp implementations substantially larger than ours.

  Save Cancel
• 

  0

  adevell created 5 years ago

  Thanks @alper.

  After a considerable number tracing exercises, we noticed the reason for multiple GetAll calls. And I agree with your comment regarding GetAll typically completing in less than one second. Our out-of-the-box implementation seemed to consistently fall within that duration timeframe. However, after bolting on a year and half worth of custom entities, dtos, permissions, localization entries, core services, app services, and apis, GetAll call durations have steadily increased. This became abundantly clear after troubleshooting recent client mobile app failures, which identified GetAll calls exceeding the default 30 second HttpClient time out parameter.

  I definitely agree with your recommendation. Adding a few diagnostic StopWatch timers in the AbpUserConfigurationBuilder class is a good starting point. We’ll look into replacing the NuGet package reference with Abp source files and implementing microscopic duration monitoring with logging.

  Thanks again for the response, insight, and suggestions.

  Save Cancel
• 

  0

  adevell created 5 years ago

  BTW – One coding pattern we have applied in our custom code that has resulted in considerable execution duration time reductions leverages Task.WhenAll(). Below is an example of before and after. In one case, we reduced execution duration from > 300ms to < 30ms.

  In addition to inserting StopWatch timers, we will evaluate leveraging the multithreading pattern below.

  Before:

  async Task<SeveralThingsDto> GetSeveralThingsAsync()
          {
              return new SeveralThingsDto
              {
                  things_1 = await GetThingsAsync_1(),
                  things_2 = await GetThingsAsync_2(),
                  things_3 = await GetThingsAsync_3(),
                  things_4 = await GetThingsAsync_4(),
              };
          }
  

  After:

  async Task<SeveralThingsDto> GetSeveralThingsAsync()
          {
              var tasks = new List<Task>();
  
              var taskThings_1 = GetThingsAsync_1();
              tasks.Add(taskThings_1);
  
              var taskThings_2 = GetThingsAsync_2();
              tasks.Add(taskThings_2);
  
              var taskThings_3 = GetThingsAsync_3();
              tasks.Add(taskThings_3);
  
              var taskThings_4 = GetThingsAsync_4();
              tasks.Add(taskThings_4);
  
              await Task.WhenAll(tasks);
  
              return new SeveralThingsDto
              {
                  things_1 = taskThings_1.Result,
                  things_2 = taskThings_2.Result,
                  things_3 = taskThings_3.Result,
                  things_4 = taskThings_4.Result,
              };
          };
  

  Save Cancel
• 

  0

  alper created 5 years ago

  Support Team

  Yeap Task.WhenAll() is definitely more performance. And let us know on which line your GetAll() waits so that we can think a solution for long lasting logins.

  Save Cancel
• 

  0

  adevell created 5 years ago

  Brief update:

  After a bit of testing last Friday, we quickly identified a few processing bottlenecks and opportunities for improvement. All AbpUserConfigurationBuilder methods are extremely serial in nature. Although async/await statements exist, concurrent processing is not leveraged. As the list of permissions, settings, and localization options increase, loop sequence counts increase, as do execution times. Further investigation confirmed methods subsequently called by AbpUserConfigurationBuild methods are also serial in nature, such as; AbpUserManager.IsGrantedAsync.

  Starting with AbpUserConfigurationBuild, we began implementing coding patterns that introduce concurrent processing as the thread pool has available threads. The resulting AbpUserConfigurationBuilder methods are listed below.

  This week we plan to continue testing for possible adverse impact to thread pool, cache server, and database server operations as a result of the new concurrent processing logic. Additionally, we plan to implement additional concurrent processing logic in the methods called by AbpUserConfigurationBuilder methods, such as; AbpUserManager.IsGrantedAsync.

  protected virtual async Task<AbpUserAuthConfigDto> GetUserAuthConfig()
          {
              var config = new AbpUserAuthConfigDto();
  
              var allPermissionNames = PermissionManager.GetAllPermissions(false).Select(p => p.Name).ToList();
              var grantedPermissionNames = new List<string>();
  
              config.AllPermissions = allPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
  
              if (AbpSession.UserId.HasValue)
              {
                  var tasks = allPermissionNames.Select(async permissionName => new { permissionName, filter = await PermissionChecker.IsGrantedAsync(permissionName) });
                  var results = await Task.WhenAll(tasks);
                  grantedPermissionNames = results.Where(x => x.filter).Select(x => x.permissionName).ToList();
  
                  config.GrantedPermissions = grantedPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
              }
  
              return config;
          }
  

  protected virtual async Task<AbpUserSettingConfigDto> GetUserSettingConfig()
          {
              var config = new AbpUserSettingConfigDto
              {
                  Values = new Dictionary<string, string>()
              };
  
              var settingDefinitions = SettingDefinitionManager
                  .GetAllSettingDefinitions()
                  .Where(sd => sd.IsVisibleToClients);
  
              if (settingDefinitions != null && settingDefinitions.Count() > 0)
              {
                  var tasks = settingDefinitions.Select(async settingDefinition => new { settingName = settingDefinition.Name, settingValue = await SettingManager.GetSettingValueAsync(settingDefinition.Name) });
                  var results = await Task.WhenAll(tasks);
                  config.Values = results.ToDictionary(k => k.settingName, v => v.settingValue);
              }
  
              return config;
          }
  

  protected virtual async Task<AbpUserFeatureConfigDto> GetUserFeaturesConfig()
          {
              var config = new AbpUserFeatureConfigDto()
              {
                  AllFeatures = new Dictionary<string, AbpStringValueDto>()
              };
  
              var allFeatures = FeatureManager.GetAll().ToList();
  
              if (AbpSession.TenantId.HasValue)
              {
                  var currentTenantId = AbpSession.GetTenantId();
  
                  if (allFeatures != null && allFeatures.Count() > 0)
                  {
                      var tasks = allFeatures.Select(async feature => new { featureName = feature.Name, value = await FeatureChecker.GetValueAsync(currentTenantId, feature.Name) });
                      var results = await Task.WhenAll(tasks);
                      config.AllFeatures = results.ToDictionary(x => x.featureName, y => new AbpStringValueDto
                      {
                          Value = y.value
                      });
                  }
              }
              else
              {
                  foreach (var feature in allFeatures)
                  {
                      config.AllFeatures.Add(feature.Name, new AbpStringValueDto
                      {
                          Value = feature.DefaultValue
                      });
                  }
              }
  
              return config;
          }
  

  Save Cancel
• 

  0

  adevell created 5 years ago

  Another update:

  We recommend not using the aggressive multitasking coding pattern previously shared. Although elegant and efficient, test results suggest introducing a couple hundred thread requests virtually simultaneously can adversely impact the server’s thread pool, slowing the server to a crawl for several seconds. After testing several different approaches, we have opted for a coding pattern that leverages a variable number of threads, but maximizes the potential thread count request to 20.

  Suggested coding changes are listed below.

  Our primary focus was on the GetUserAuthConfig method. Diagnostics identified this method as the slowest method by a significant margin. This method also proved to benefit greatly with the introduction of multiple concurrent threads. Since implementing the multithreading coding pattern, execution times have remained well below 10 seconds, far from the 30+ seconds previously encountered with the serial coding pattern.

  The second slowest method is GetUserSettingConfig. Surprisingly, a multithreaded solution resulted in very little improvement in performance. We ran out of time to investigate further and opted to back out all multithreading coding patterns applied within GetUserSettingConfig. However, a few test runs quickly determined executing GetUerSettingConfig’s original serial coding pattern in parallel with GetUserAuthConfig’s multithreaded coding pattern shaved an extra 1-2 seconds off execution times.

  Let me know what you think.

  Enhanced GetAll method:

  public virtual async Task<AbpUserConfigurationDto> GetAll()
          {
              var taskGetUserAuthConfig = GetUserAuthConfig();
              var taskGetUserSettingConfig = GetUserSettingConfig();
  
              await Task.WhenAll(taskGetUserAuthConfig, taskGetUserSettingConfig);
  
              return new AbpUserConfigurationDto
              {
                  MultiTenancy = GetUserMultiTenancyConfig(),
                  Session = GetUserSessionConfig(),
                  Localization = GetUserLocalizationConfig(),
                  Features = await GetUserFeaturesConfig(),
                  Auth = taskGetUserAuthConfig.Result,
                  Nav = await GetUserNavConfig(),
                  Setting = taskGetUserSettingConfig.Result,
                  Clock = GetUserClockConfig(),
                  Timing = await GetUserTimingConfig(),
                  Security = GetUserSecurityConfig()
              };
          }
  

  Enhanced GetUserAuthConfig method:

  protected virtual async Task<AbpUserAuthConfigDto> GetUserAuthConfig()
          {
              var config = new AbpUserAuthConfigDto();
  
              var allPermissionNames = PermissionManager.GetAllPermissions(false).Select(p => p.Name).ToList();
              var grantedPermissionNames = new List<string>();
  
              if (AbpSession.UserId.HasValue)
              {
                  grantedPermissionNames = await SplitAndMergePermissionsAsync(allPermissionNames);
              }
  
              config.AllPermissions = allPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
              config.GrantedPermissions = grantedPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
  
              return config;
          }
  
          async Task<List<string>> SplitAndMergePermissionsAsync(List<string> allPermissionNames)
          {
              const int maxLoopCount = 15;
              const int maxThreadCount = 20;
  
              var grantedPermissionNames = new List<string>();
  
              var tasks = new List<Task<List<string>>>();
  
              var totalPermissionCount = allPermissionNames.Count();
  
              // Split AllPermissionNames list for distribution to multiple threads.
              int threadCount;
              if (totalPermissionCount < 300)
                  threadCount = (totalPermissionCount / maxLoopCount) + (totalPermissionCount % maxLoopCount > 0 ? 1 : 0);
              else
                  threadCount = maxThreadCount;
  
              int takeCount = totalPermissionCount / threadCount;
              int takeOverage = totalPermissionCount % threadCount;
              var skipCount = 0;
  
              for (var i = 0; i < threadCount; i++)
              {
                  // Spread overage evenly over threads until overage is exhausted.
                  int takeCountPlus = takeCount;
                  if (takeOverage > 0)
                  {
                      takeCountPlus++;
                      takeOverage--;
                  }
  
                  var permissionNames = allPermissionNames.GetRange(skipCount, takeCountPlus);
  
                  // Generate granted permission lists in parallel.
                  tasks.Add(GetGrantedPermissionsAsync(permissionNames));
  
                  skipCount = skipCount + takeCountPlus;
              }
  
              await Task.WhenAll(tasks);
  
              // Merge grantedPermissionNames lists to output a single list.
              foreach (var task in tasks)
              {
                  if (task.Result != null)
                      grantedPermissionNames.AddRange(task.Result);
              }
  
              return grantedPermissionNames;
          }
  
          async Task<List<string>> GetGrantedPermissionsAsync(List<string> allPermissionNames)
          {
              var grantedPermissionNames = new List<string>();
  
              foreach (var permissionName in allPermissionNames)
              {
                  if (await PermissionChecker.IsGrantedAsync(permissionName))
                  {
                      grantedPermissionNames.Add(permissionName);
                  }
              }
  
              return grantedPermissionNames;
          }
  

  Save Cancel
• 

  0

  alper created 5 years ago

  Support Team

  could you send a Pull Request to the Aspnet Boilerplate repository?

  Save Cancel
• 

  0

  adevell created 5 years ago

  Will do.

  Save Cancel
• 

  0

  VuCA created 3 years ago

  Is the enhance of @advell is apply. I still have a issuess whit GetAll() too. When increase number of connection (concurrent user) => It extreamly slowly. more than 3 minute even 5 minute

  • Info: Azure host
  • Aspnet zero: 8.5.0.0

  Save Cancel