Will do.
Another update:
We recommend not using the aggressive multitasking coding pattern previously shared. Although elegant and efficient, test results suggest introducing a couple hundred thread requests virtually simultaneously can adversely impact the server’s thread pool, slowing the server to a crawl for several seconds. After testing several different approaches, we have opted for a coding pattern that leverages a variable number of threads, but maximizes the potential thread count request to 20.
Suggested coding changes are listed below.
Our primary focus was on the GetUserAuthConfig method. Diagnostics identified this method as the slowest method by a significant margin. This method also proved to benefit greatly with the introduction of multiple concurrent threads. Since implementing the multithreading coding pattern, execution times have remained well below 10 seconds, far from the 30+ seconds previously encountered with the serial coding pattern.
The second slowest method is GetUserSettingConfig. Surprisingly, a multithreaded solution resulted in very little improvement in performance. We ran out of time to investigate further and opted to back out all multithreading coding patterns applied within GetUserSettingConfig. However, a few test runs quickly determined executing GetUerSettingConfig’s original serial coding pattern in parallel with GetUserAuthConfig’s multithreaded coding pattern shaved an extra 1-2 seconds off execution times.
Let me know what you think.
Enhanced GetAll method:
public virtual async Task<AbpUserConfigurationDto> GetAll()
{
var taskGetUserAuthConfig = GetUserAuthConfig();
var taskGetUserSettingConfig = GetUserSettingConfig();
await Task.WhenAll(taskGetUserAuthConfig, taskGetUserSettingConfig);
return new AbpUserConfigurationDto
{
MultiTenancy = GetUserMultiTenancyConfig(),
Session = GetUserSessionConfig(),
Localization = GetUserLocalizationConfig(),
Features = await GetUserFeaturesConfig(),
Auth = taskGetUserAuthConfig.Result,
Nav = await GetUserNavConfig(),
Setting = taskGetUserSettingConfig.Result,
Clock = GetUserClockConfig(),
Timing = await GetUserTimingConfig(),
Security = GetUserSecurityConfig()
};
}
Enhanced GetUserAuthConfig method:
protected virtual async Task<AbpUserAuthConfigDto> GetUserAuthConfig()
{
var config = new AbpUserAuthConfigDto();
var allPermissionNames = PermissionManager.GetAllPermissions(false).Select(p => p.Name).ToList();
var grantedPermissionNames = new List<string>();
if (AbpSession.UserId.HasValue)
{
grantedPermissionNames = await SplitAndMergePermissionsAsync(allPermissionNames);
}
config.AllPermissions = allPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
config.GrantedPermissions = grantedPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
return config;
}
async Task<List<string>> SplitAndMergePermissionsAsync(List<string> allPermissionNames)
{
const int maxLoopCount = 15;
const int maxThreadCount = 20;
var grantedPermissionNames = new List<string>();
var tasks = new List<Task<List<string>>>();
var totalPermissionCount = allPermissionNames.Count();
// Split AllPermissionNames list for distribution to multiple threads.
int threadCount;
if (totalPermissionCount < 300)
threadCount = (totalPermissionCount / maxLoopCount) + (totalPermissionCount % maxLoopCount > 0 ? 1 : 0);
else
threadCount = maxThreadCount;
int takeCount = totalPermissionCount / threadCount;
int takeOverage = totalPermissionCount % threadCount;
var skipCount = 0;
for (var i = 0; i < threadCount; i++)
{
// Spread overage evenly over threads until overage is exhausted.
int takeCountPlus = takeCount;
if (takeOverage > 0)
{
takeCountPlus++;
takeOverage--;
}
var permissionNames = allPermissionNames.GetRange(skipCount, takeCountPlus);
// Generate granted permission lists in parallel.
tasks.Add(GetGrantedPermissionsAsync(permissionNames));
skipCount = skipCount + takeCountPlus;
}
await Task.WhenAll(tasks);
// Merge grantedPermissionNames lists to output a single list.
foreach (var task in tasks)
{
if (task.Result != null)
grantedPermissionNames.AddRange(task.Result);
}
return grantedPermissionNames;
}
async Task<List<string>> GetGrantedPermissionsAsync(List<string> allPermissionNames)
{
var grantedPermissionNames = new List<string>();
foreach (var permissionName in allPermissionNames)
{
if (await PermissionChecker.IsGrantedAsync(permissionName))
{
grantedPermissionNames.Add(permissionName);
}
}
return grantedPermissionNames;
}
Brief update:
After a bit of testing last Friday, we quickly identified a few processing bottlenecks and opportunities for improvement. All AbpUserConfigurationBuilder methods are extremely serial in nature. Although async/await statements exist, concurrent processing is not leveraged. As the list of permissions, settings, and localization options increase, loop sequence counts increase, as do execution times. Further investigation confirmed methods subsequently called by AbpUserConfigurationBuild methods are also serial in nature, such as; AbpUserManager.IsGrantedAsync.
Starting with AbpUserConfigurationBuild, we began implementing coding patterns that introduce concurrent processing as the thread pool has available threads. The resulting AbpUserConfigurationBuilder methods are listed below.
This week we plan to continue testing for possible adverse impact to thread pool, cache server, and database server operations as a result of the new concurrent processing logic. Additionally, we plan to implement additional concurrent processing logic in the methods called by AbpUserConfigurationBuilder methods, such as; AbpUserManager.IsGrantedAsync.
protected virtual async Task<AbpUserAuthConfigDto> GetUserAuthConfig()
{
var config = new AbpUserAuthConfigDto();
var allPermissionNames = PermissionManager.GetAllPermissions(false).Select(p => p.Name).ToList();
var grantedPermissionNames = new List<string>();
config.AllPermissions = allPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
if (AbpSession.UserId.HasValue)
{
var tasks = allPermissionNames.Select(async permissionName => new { permissionName, filter = await PermissionChecker.IsGrantedAsync(permissionName) });
var results = await Task.WhenAll(tasks);
grantedPermissionNames = results.Where(x => x.filter).Select(x => x.permissionName).ToList();
config.GrantedPermissions = grantedPermissionNames.ToDictionary(permissionName => permissionName, permissionName => "true");
}
return config;
}
protected virtual async Task<AbpUserSettingConfigDto> GetUserSettingConfig()
{
var config = new AbpUserSettingConfigDto
{
Values = new Dictionary<string, string>()
};
var settingDefinitions = SettingDefinitionManager
.GetAllSettingDefinitions()
.Where(sd => sd.IsVisibleToClients);
if (settingDefinitions != null && settingDefinitions.Count() > 0)
{
var tasks = settingDefinitions.Select(async settingDefinition => new { settingName = settingDefinition.Name, settingValue = await SettingManager.GetSettingValueAsync(settingDefinition.Name) });
var results = await Task.WhenAll(tasks);
config.Values = results.ToDictionary(k => k.settingName, v => v.settingValue);
}
return config;
}
protected virtual async Task<AbpUserFeatureConfigDto> GetUserFeaturesConfig()
{
var config = new AbpUserFeatureConfigDto()
{
AllFeatures = new Dictionary<string, AbpStringValueDto>()
};
var allFeatures = FeatureManager.GetAll().ToList();
if (AbpSession.TenantId.HasValue)
{
var currentTenantId = AbpSession.GetTenantId();
if (allFeatures != null && allFeatures.Count() > 0)
{
var tasks = allFeatures.Select(async feature => new { featureName = feature.Name, value = await FeatureChecker.GetValueAsync(currentTenantId, feature.Name) });
var results = await Task.WhenAll(tasks);
config.AllFeatures = results.ToDictionary(x => x.featureName, y => new AbpStringValueDto
{
Value = y.value
});
}
}
else
{
foreach (var feature in allFeatures)
{
config.AllFeatures.Add(feature.Name, new AbpStringValueDto
{
Value = feature.DefaultValue
});
}
}
return config;
}
BTW – One coding pattern we have applied in our custom code that has resulted in considerable execution duration time reductions leverages Task.WhenAll(). Below is an example of before and after. In one case, we reduced execution duration from > 300ms to < 30ms.
In addition to inserting StopWatch timers, we will evaluate leveraging the multithreading pattern below.
Before:
async Task<SeveralThingsDto> GetSeveralThingsAsync()
{
return new SeveralThingsDto
{
things_1 = await GetThingsAsync_1(),
things_2 = await GetThingsAsync_2(),
things_3 = await GetThingsAsync_3(),
things_4 = await GetThingsAsync_4(),
};
}
After:
async Task<SeveralThingsDto> GetSeveralThingsAsync()
{
var tasks = new List<Task>();
var taskThings_1 = GetThingsAsync_1();
tasks.Add(taskThings_1);
var taskThings_2 = GetThingsAsync_2();
tasks.Add(taskThings_2);
var taskThings_3 = GetThingsAsync_3();
tasks.Add(taskThings_3);
var taskThings_4 = GetThingsAsync_4();
tasks.Add(taskThings_4);
await Task.WhenAll(tasks);
return new SeveralThingsDto
{
things_1 = taskThings_1.Result,
things_2 = taskThings_2.Result,
things_3 = taskThings_3.Result,
things_4 = taskThings_4.Result,
};
};
Thanks @alper.
After a considerable number tracing exercises, we noticed the reason for multiple GetAll calls. And I agree with your comment regarding GetAll typically completing in less than one second. Our out-of-the-box implementation seemed to consistently fall within that duration timeframe. However, after bolting on a year and half worth of custom entities, dtos, permissions, localization entries, core services, app services, and apis, GetAll call durations have steadily increased. This became abundantly clear after troubleshooting recent client mobile app failures, which identified GetAll calls exceeding the default 30 second HttpClient time out parameter.
I definitely agree with your recommendation. Adding a few diagnostic StopWatch timers in the AbpUserConfigurationBuilder class is a good starting point. We’ll look into replacing the NuGet package reference with Abp source files and implementing microscopic duration monitoring with logging.
Thanks again for the response, insight, and suggestions.
Thanks @strix20.
I agree, AbpUserConfigurationBuilder does a lot of work, producing a rich collection of data. And your guess is consistent with my initial guess. We have substantially increased the number of permissions and localization entries compared to the out-of-the-box numbers.
Our concern is our platform may have grown beyond the original intent of Abp. The AppPermissions file is approaching 250 lines. The primary localization xml file is well over 1,000 lines and growing rapidly. Additionally, the latest version has more than 150 entities, upwards of 500 Dtos, more than 250 Apis, and more than 100 backend bots managing data, primarily stored in Redis cache and SQL dbs, pushed to/received from service bus topics, or pushed to/received from 3rd party Apis. However, I suspect there are dozens, if not hundreds, of Abp implementations substantially larger than ours.
Hi All,
As always, great framework! And very helpful, informative, and instructional coding patterns.
For the past year+ we have been developing a number of mobile apps that leverage an ASP.Net Zero based platform, operating on several server farms, as the backend. For the most part it works brilliantly. However, mobile app launch and user logins are steadily getting slower. As the platform grows, mobile app launch and logins require an increasing amount of time. Our largest mobile app now requires upwards of 60 seconds to launch. We eventually determined the primary cause for the lengthy launch and login times are the 1-2 GetAll calls per launch and/or login. GetAll calls are now ranging from 4 seconds to 39 seconds (screenshot of example calls attached). The more recent 30+ second calls have required a bump in the AbpClient FlurlClient/HttpClient TimeOut parameter, increased from 30 seconds to 60 seconds.
Have others experienced this phenomena? Is there a known quick fix? We are considering two solutions; 1) Caching GetAll results locally on mobile apps for immediate use while executing a background update call; 2) Rewrite GetAll method and supporting dependencies to better leverage cached data.
Any ideas?
Thanks in advance.
Thanks. It's on my to-do list to begin the effort within the next few weeks--most likely early July. I'll keep you posted on progress.
Is there an easy way to configure and support OAuth refresh tokens in Asp.Net Boilerplate and/or Module Zero?
Currently, within the WebAPI Account Controller, the access token expiration is set for 20 minutes. I would like to leave the access token expiration at 20 minutes (or less), while configuring refresh token expiration for 2-3 months.