Activities of "Web2workNL"

Hello,

We are currently working on a project that uses IdentityServer and ASP.NET Zero. This is the first time I am using these techniques and I am not sure whether I understand these concepts correctly.

Our goal is to provide an authentication/authorization platform for clients in which users can use their authentication cookie of an external identity provider to check whether they have a valid license. These licenses are managed in our portal. The workflow would look like something like this:

• At the (external) client website, a user wants to open a part of the website that requires a valid license.
• The client website contacts our platform and navigates to the IdentityServer.
• The IdentityServer redirects to the external identity provider (uses SAML2).
• The external IDP authenticates the user and returns a cookie that contains the (external) user identifier.
• Using the user identifier in this cookie, the user gets authenticated in ABP (this user already exists, as they are imported using a background job and are assigned to licenses. This import also sets the AbpUserLogin with the correct provider and provider key).
• With the authenticated user, it is possible to retrieve the assigned licenses. These are added to the result as claims.
• The clients retrieves a response from the IdentityServer including the license claims.

I have started experimenting with the startup template of ASP.NET Zero but my lack of experience with these concepts makes it difficult to determine whether I am on the right track. I am able to use a MVC client to contact the IdentityServer of the ABP project and am successfully getting authenticated using the external Idp (using the SAML2 package of Sustainsys) but am unable to authenticate in Abp. Also, I am not sure how this authentication is related to the External Login Providers that Abp already provides (the social logins for Google, Facebook etc.). Should I somehow add a new provider? I have tried to build an ExternalAuthenticateModel and use the ExternalAuthenticate method in the TokenAuthController to authenticate but am not sure if this is the way to go.

I hope that there is someone here who has a little more experience with this and can tell me if I'm on the right track.

Kind regards, Nick

We are running into caching issues with our application. The problem seems equal to the problem mentioned by another customer https://support.aspnetzero.com/QA/Questions/8897.

AbpZeroTenantFeatures returnzs the wrong entries for a given tenant (Mismatched from what is in the database). Clearning the cache updates a tenants features to the proper list again. Then it will be fine for a period of time and then start returning incorrect features for a tenant until we clear the cache again.

We were running ABP 4.8.1 and have already upgraded to 4.12.0 but this did not solve the problem. Also, we experience this problem with in memory caching and Redis caching.

When we check the cache contents of AbpTenantFeatures for tenant 1 this normally contains the following:

{
    "EditionId": 1,
    "FeatureValues": {
        "App.NarrowcastingFeature": "true",
        "App.NarrowcastingMaxScreenCount": "10"
    }
}

This corresponds to the TenantFeatureSettings stored in the database for this tenant:

At the moment the cache is “corrupted” for tenant 1 it contains te following:

{ 
    "EditionId": 1, 
    "FeatureValues": { 
        "App.AbsenceReporter": "false", 
        "App.Calendar": "false", 
        "App.MessagingFeature": "false", 
        "App.NarrowcastingFeature": "true", 
        "App.NarrowcastingMaxFeedCount": "2", 
        "App.NarrowcastingMaxScreenCount": "3", 
        "App.PrivacyFeature": "false", 
    } 
}

We checked AbpFeatureValueStore.GetTenantFeatureCacheItemAsyncto see if something goes wrong there. The code that gets the features from the database looks ok:

using (var uow = _unitOfWorkManager.Begin())
{
    using (_unitOfWorkManager.Current.SetTenantId(tenantId))
    {
        var featureSettings = await _tenantFeatureRepository.GetAllListAsync();
        foreach (var featureSetting in featureSettings)
        {
            newCacheItem.FeatureValues[featureSetting.Name] = featureSetting.Value;
        }

        await uow.CompleteAsync();
    }
}

However, it seems that the tenant filter is not applied at the moment the problem occurs. We think this is the case because when all TenantFeatureSettings are returned from the database, the last value returned from the table for that setting will be set in the FeatureValues dictionary.

A small LinqPad script that does the same:

// Get tenant features for all tenants
var features = AbpFeatures.Where(f => f.Discriminator == "TenantFeatureSetting").OrderBy(f => f.Id).ToList();

// populate dictionary
var d = new Dictionary<string,string>();

foreach (var feature in features)
{
	d[feature.Name] = feature.Value;
}

// show results
d.Dump();

Result: The result is exactly the same as the items found in the "corrupted" cache.

Now the big question is: why would the tenant filter not be applied? Is there any circumstance that _unitOfWorkManager.Current.SetTenantId(tenantId) does not work? E.g. when we disable the tenant filter? Or when we set the tenant Id ourselves earlier?

We use the default FeatureChecker implementation to check if features are enabled for tenants. Next to this we use the RequiresFeature Attribute on almost all our AppServices. We don't know how this Attribute is implemented behind the scenes.

We don't know where this behavior comes from so it is hard to answer your second question. We only know the end result as we pointed out in the original post. We did suspect using DisableFilter for MayHaveTenant/MustHaveTenant or when using SetTenantId on the CurrentUnitOfWork but were unable to reproduce it using these scenarios.

We also have the assumption that the data gets corrupted only when the cache has expired and data is re-added through either GetTenantFeatureCacheItemAsync or GetTenantFeatureCacheItem.

In summary, we know the end result but not what leads to this.

We know about the possibility to clear the cache by hand, but have been unable to reproduce the corrupted cache after doing this.

It doesn't seem to happen everytime, which is why we assume there is a specific scenario somewhere that leads to the tenant filter not being applied in AbpFeatureValueStore.GetTenantFeatureCacheItem(Async)

We believe that when this scenario presents itself just as the cache has been expired, these two situations together lead to the cache being corrupted. Me and my team have been going through our code to find this scenario, but have not had any luck.

We would be very grateful if you could brainstorm with us on what could possibility lead to this.

Hi JakeE,

We would love to hear from you wether you managed to find a solution to the problem other than expiring the caches after 1 second. We are experiencing the same issues (https://support.aspnetzero.com/QA/Questions/9015)

We managed to reproduce the behaviour. Please see the GitHub issue.

Hai,

My project has a situation where I want to get a Permission from the PermissionManager inside a backgroundjob. The permission is defined as follows:

administration.CreateChildPermission(AppPermissions.Pages_Administration_TeacherGroups, L("GroupsForTeachers"), multiTenancySides: MultiTenancySides.Tenant);

inside the AppAuthorizationProvider. When the following code is run

using (CurrentUnitOfWork.SetTenantId(tenantId))
                {                    
                    using (_session.Use(tenantId, null))
                    {
                      
                        var permission = _permissionManager.GetPermissionOrNull(AppPermissions.Pages_Administration_TeacherGroups);
                     }
                }
                

the permission is always NULL. What am I missing?

We are currently using ABP 4.12.0

Yes, the PermissionManager is created through dependency injection. So is the _session btw private readonly IAbpSession _session;

The background job is not a timed job but is handled like the notifications are (through AbpBackgoundJobs)

  await _backgroundJobManager.EnqueueAsync<SyncWithMooJob, SyncWithMooJobArgs>(new SyncWithMooJobArgs(AbpSession.GetTenantId()));

And

 public class SyncWithMooJob : BackgroundJob<SyncWithMooJobArgs>, ITransientDependency
        {
            private readonly IMooManager _mooSyncManager;
            
            public SyncWithMooJob(IMooManager mooSyncManager)
            {
                _mooSyncManager = mooSyncManager;
            }

            public override void Execute(SyncWithMooJobArgs args)
            {
                AsyncHelper.RunSync(() => _mooSyncManager.AutomaticSyncWithMoo(args.TenantId));
            }
        }
    }

If I strip the method down to just

        public async Task AutomaticSyncWithMoo(int tenantId)
        {
            try
            {
                using (CurrentUnitOfWork.SetTenantId(tenantId))
                {
                   
                    using (_session.Use(tenantId, null))
                    { 
                    var permission = _permissionManager.GetPermissionOrNull(AppPermissions.Pages_Administration_TeacherGroups);
                     }
                } 
            }
            catch (Exception ex)
            {
                Logger.Error(ex.Message + ex.StackTrace);
            }
        }
                    

I still get a NULL result. Just for clarity, when I add _permissionManager.GetPermissionOrNull(AppPermissions.Pages_Administration_TeacherGroups); to for instance an AppService there are no issues

• What is your product version? ABP Nuget packages 6.6.1
• What is your product type (Angular or MVC)? ANGULAR
• What is product framework type (.net framework or .net core)? .NET 5

Yesterday we rolled out a new version of our application where we updated from using version 4.12 of the ABP packages to 6.6.1 We found an unexpected issue with our caching. We implemented our own hybrid solution, where we try to get values from the memory cache which has a small expiry and if it's not there, we get it from Redis and put in the Memory cache for a next time. Previously we had a small amount of redis cache hits and most of the work was done by memory cache. We made some alterations to make it possible to upgrade from 4.12 to 6.6.1, but unfortunately the colleague that made these changes for the caching has since left the company.

Can you help us understand what might have changed in ABP that could cause this, or how our solution might be responsible for this. If you need any further info, please let me know!

`namespace xxx.xxx.Web.Caching { using System; using System.Collections.Generic; using System.Threading.Tasks; using Abp.Runtime.Caching; using Abp.Runtime.Caching.Memory; using Abp.Runtime.Caching.Redis; using Abp.Timing;

/// <summary>
/// Create a cache existing of two caching layers, the first being a memory cache, the second being a Redis cache.
/// The memory cache expires quickly, so a change on another AppService instance is -through the use of the Redis cache- propagated quickly to the memory cache of this instance
/// Create, update and delete actions are always applied to the memory cache AND the Redis cache
/// Getting an item from cache will try to get it from the memory cache. If this doesn't succeed, it will try to get it from the Redis cache.
/// </summary>
public class MemoryAndRedisCache : CacheBase
{
    private readonly AbpMemoryCache _memoryCache;
    private readonly AbpRedisCache _redisCache;

    /// <summary>
    /// Constructor.
    /// </summary>
    public MemoryAndRedisCache(string name, AbpMemoryCacheManager memoryCacheManager, AbpRedisCacheManager redisCacheManager) : base(name)
    {
        // Set expiry to 1 minute for the memory cache, so a change on another AppService instance is propagated quickly to the memory cache of this instance
        _memoryCache = (AbpMemoryCache)memoryCacheManager.GetCache(name + "_memory");
        _memoryCache.DefaultAbsoluteExpireTime = Clock.Now.AddMinutes(1);

        // Expiry of the Redis cache is done through normal configuration in ABP
        _redisCache = (AbpRedisCache)redisCacheManager.GetCache(name);
    }

    public override bool TryGetValue(string key, out object value)
    {
        // Try to get the value from memory
        var memorySuccess = _memoryCache.TryGetValue(key, out value);
        if (memorySuccess)
        {
            var expiryInfo = _memoryCache.DefaultAbsoluteExpireTime != null ? _memoryCache.DefaultAbsoluteExpireTime.ToString() : _memoryCache.DefaultSlidingExpireTime.ToString();
            Logger.Debug($"Get from {this.Name}: {key} [memory] [{expiryInfo}]");
            return true;
        }

        // Getting from memory failed, try to get it from Redis
        var redisSuccess = _redisCache.TryGetValue(key, out value);
        if (redisSuccess)
        {
            var expiryInfo = _redisCache.DefaultAbsoluteExpireTime != null ? _redisCache.DefaultAbsoluteExpireTime.ToString() : _redisCache.DefaultSlidingExpireTime.ToString();
            Logger.Debug($"Get from {this.Name}: {key} [redis] [{expiryInfo}]");
            // We got the value from Redis, now store it in memory cache for repeated use
            _memoryCache.Set(key, value, null, null);
            return true;
        }

        return false;
    }

    public override object GetOrDefault(string key)
    {
        // Try to get the value from memory
        var value = _memoryCache.GetOrDefault(key);
        if (value != null)
        {
            var expiryInfo = _memoryCache.DefaultAbsoluteExpireTime != null ? _memoryCache.DefaultAbsoluteExpireTime.ToString() : _memoryCache.DefaultSlidingExpireTime.ToString();
            Logger.Debug($"Get from {this.Name}: {key} [memory] [{expiryInfo}]");
            return value;
        }

        // Getting from memory failed, try to get it from Redis
        value = _redisCache.GetOrDefault(key);
        if (value != null)
        {
            var expiryInfo = _redisCache.DefaultAbsoluteExpireTime != null ? _redisCache.DefaultAbsoluteExpireTime.ToString() : _redisCache.DefaultSlidingExpireTime.ToString();
            Logger.Debug($"Get from {this.Name}: {key} [redis] [{expiryInfo}]");
            // We got the value from Redis, now store it in memory cache for repeated use
            _memoryCache.Set(key, value, null, null);
        }
        else
        {
            Logger.Debug($"Get from {this.Name}: {key} [source]");
            // The caching logic of ABP will call the function to get the data from the source - no need to do anything
        }

        return value;
    }

    public override object[] GetOrDefault(string[] keys)
    {
        var values = _memoryCache.GetOrDefault(keys);

        for (var t = 0; t < keys.Length; t++)
        {
            if (values[t] == null)
            {
                values[t] = _redisCache.GetOrDefault(keys[t]);
            }
        }

        return values;
    }

    public override async Task<object> GetOrDefaultAsync(string key)
    {
        // Try to get the value from memory
        var value = await _memoryCache.GetOrDefaultAsync(key);
        if (value != null)
        {
            var expiryInfo = _memoryCache.DefaultAbsoluteExpireTime != null ? _memoryCache.DefaultAbsoluteExpireTime.ToString() : _memoryCache.DefaultSlidingExpireTime.ToString();
            Logger.Debug($"Get from {this.Name}: {key} [memory] [{expiryInfo}]");
            return value;
        }

        // Getting from memory failed, try to get it from Redis
        value = await _redisCache.GetOrDefaultAsync(key);
        if (value != null)
        {
            var expiryInfo = _redisCache.DefaultAbsoluteExpireTime != null ? _redisCache.DefaultAbsoluteExpireTime.ToString() : _redisCache.DefaultSlidingExpireTime.ToString();
            Logger.Debug($"Get from {this.Name}: {key} [redis] [{expiryInfo}]");
            // We got the value from Redis, now store it in memory cache for repeated use
            await _memoryCache.SetAsync(key, value, null, null);
        }
        else
        {
            Logger.Debug($"Get from {this.Name}: {key} [source]");
            // The caching logic of ABP will call the function to get the data from the source - no need to do anything
        }

        return value;
    }

    public override async Task<object[]> GetOrDefaultAsync(string[] keys)
    {
        var values = await _memoryCache.GetOrDefaultAsync(keys);

        for (var t = 0; t < keys.Length; t++)
        {
            if (values[t] == null)
            {
                values[t] = await _redisCache.GetOrDefaultAsync(keys[t]);
            }
        }

        return values;
    }

    public override void Set(string key, object value, TimeSpan? slidingExpireTime = null, DateTimeOffset? absoluteExpireTime = null)
    {
        _memoryCache.Set(key, value, null, null);
        _redisCache.Set(key, value, null, null);
    }

    public override async Task SetAsync(string key, object value, TimeSpan? slidingExpireTime = null, DateTimeOffset? absoluteExpireTime = null)
    {
        await _memoryCache.SetAsync(key, value, null, null);
        await _redisCache.SetAsync(key, value, null, null);
    }

    public override void Set(KeyValuePair<string, object>[] pairs, TimeSpan? slidingExpireTime = null, DateTimeOffset? absoluteExpireTime = null)
    {
        _memoryCache.Set(pairs, null, null);
        _redisCache.Set(pairs, null, null);
    }

    public override async Task SetAsync(KeyValuePair<string, object>[] pairs, TimeSpan? slidingExpireTime = null, DateTimeOffset? absoluteExpireTime = null)
    {
        await _memoryCache.SetAsync(pairs, null);
        await _redisCache.SetAsync(pairs, null, null);
    }

    public override void Remove(string key)
    {
        _memoryCache.Remove(key);
        _redisCache.Remove(key);
    }

    public override async Task RemoveAsync(string key)
    {
        await _memoryCache.RemoveAsync(key);
        await _redisCache.RemoveAsync(key);
    }

    public override void Remove(string[] keys)
    {

        _memoryCache.Remove(keys);
        _redisCache.Remove(keys);
    }

    public override async Task RemoveAsync(string[] keys)
    {
        await _memoryCache.RemoveAsync(keys);
        await _redisCache.RemoveAsync(keys);

    }

    public override void Clear()
    {
        _memoryCache.Clear();
        _redisCache.Clear();
    }

    public override void Dispose()
    {
        _memoryCache.Dispose();
        _redisCache.Dispose();
    }
}

} `

Thanks for the suggestion! I was unaware that ABP had something similar out of the box 👍.

I have a few questions about the implementation of the PerRequestRedisCache though

• Does this cache stuff to memory like the MemoryCache implementation does, or does it cache for the specific current connection so THIS user doesn't have to go to redis if they need the same value again
• We have a scale out solution with multiple instances, so we've set the expiry on the memory cache really low so it expires quickly and is updated from Redis frequently. Is it possible to set the expiry of the memorycache with PerRequestRedisCache?