Activities of "adamphones"

We would like to create a module or use app service) to import large data into the system. We noticed is that when current services are used or domains are used to import the data, the process takes ages or sometimes never to finish. The issue due to the fact that EF tracks changes to all entities. As the loop continues to increase, memory gets smashed with all tracking an as a result process gets very slow.

You can read the issue here: https://weblog.west-wind.com/posts/2014/dec/21/gotcha-entity-framework-gets-slow-in-long-iteration-loops#:~:text=There%20are%20a%20few%20simple,tracking%20for%20the%20dbContext%20instance

The suggestion is to get a new instance of the db context or stop tracking to speed up the process.

I read through your documation and disabling UnitOfWork speeds up the process a bit (basically using a single unit of work for each loop and commiting changes at the end of the loop). However we believe that creating nearly 10 000 records ( in various tables) should not take around 15 minutes.

Somehow we need to access the DB context and disable the following to gain more speed while importing data as it suggested in this link : https://stackoverflow.com/questions/6107206/improving-bulk-insert-performance-in-entity-framework

yourContext.Configuration.AutoDetectChangesEnabled = false; yourContext.Configuration.ValidateOnSaveEnabled = false;

My questions are:

1- What is your suggestion about importing large data(not just into single table but multiple tables) with best performance? 2- How can we access the DB context to disable those options while we import the data? 3- Would we be able to still use Repository injections? For example : IRepository<Product>

Hi

apb : v8.6.0.0 angular 9 asp.net core 3.1

This is an intermittent bug.

Steps to produce:
1- Change the language to en-GB from dropddown. (this is are default language). 2- Go to >Administration> Languages> and disable en (United States). Leave only en-GB (United Kingdom) enabled and set United Kingdom as default language. 3- Log out and clear all caches. Clear all cookies. and restart app pool. and re-run ng serve if possbile. (Basically completely clean start) 4-Log back in 5- Remove a page permisison for a user( it doesn't matter which page) 6- Login as that user and try to access the page that the user does not have permission. 7- An error message pops up similar to attached screenshot.

Problem: The error popup does not display required permissions as it does in en(United States) language.

Please make sure clear all caches otherwise when you change language from en to en-GB then the error message shows the permisisons fine. The key thing is when en language is disabled a completely clean start then the issue appears.

When we change the language to en-GB we cannot see the required permisisons in the error message.

<text name="AtLeastOneOfThesePermissionsMustBeGranted">Required permissions are not granted. At least one of these permissions must be granted: {0}</text> exists in en-GB.xml file.

Any idea why we do not see the required permissions in the error message?

Thanks,

Hi Musa,

I appriciate your help. I am reading the info from a csv file where there are over 10000 records. I assume you wouldnt want to me to load all data in the memory and deal with that in the memory? The process needs to do the job row by row. Read > check > write and commit. That is the requirement.

In your example If I have entries repeated in the source file then I would have to distinct those too. But Sometimes the row product name can be the same and I don't need to create a new product but I will need to use other columns in that row to add new child properties (ProductItems) in the product.

Can you please let me know how I would initiate the context in the method? That is all I need to know.

Thanks

Hi,

I knew you would ask that.. :)

It is nothing to do with the code but I will put an example here.

public Task ImportData()
{

 var productsToImport =  await anotherDataSourcerepository.GetAll().AsNoTracking().ToListAsync();
 
 foreach(var productImport in productsToImport)
    {
         var existingProduct = _productRepository.GetAll().AsNoTracking().FirstOrDefaultAsync(x=>x.Name == productImport.Name);

        if(existingProduct==null)
        {
             var product = new Product(){Name= productImport.Name};
             await _productRepository.InsertAsync(product)
             await CurrentUnitOfWork.SaveChangesAsync();
         }
     }
}

If I have 10 products to import finishes in less than a second(instantly). If I have 100 products then this takes over 3 minutes! It doesn't add up. Clearly db context getting bloated as the link suggests with the records and it gets slower and slower.

As the link suggests I would need to re-create the db context in the iteration instead of using single context.

But how to do that?

The key is to import data as fast as possible.

is there a way to create new db context in the loop? something like : var productRepostotry = new DbContext()..

Regards,

Hi,

We would like to import data into system from CSV file. We created a process read cv file each row one by one and create necessary Entities and sub entities by using repository pattern and save changes by calling await CurrentUnitOfWork.SaveChangesAsync(); or await _productRepository.InsertAndGetIdAsync(product);

However, the process run very fast in the first 10-20 iterations then gets slower and slower and It takes over 10 minutes to complete around 150 rows!

I looked up on the internet and found this link where @Rick explained the problem: https://weblog.west-wind.com/posts/2014/dec/21/gotcha-entity-framework-gets-slow-in-long-iteration-loops.

What I tried: 1- call AsNoTracking() in all repository access but I still see that process very slow.

2- Used using (var unitOfWork = _unitOfWorkManager.Begin()) {...} in the iteration thinking that this would refresh the context in each iteration as the link suggests but no luck with this either.

Do you have any idea how to overcome this issue? How do you import large users into the system?

Regards,

Hi @alwefaq,

Can you have a look at this ticket, I raised this issue and I think this is exact same issue you are facing. No clear answer to the isuee yet.

Hi Ismail,

Yes it is the latest angular.

Where exactly? Do I need to install Sticky? or it is part of metronic already.

I can't execute var sticky = new Sticky('[data-sticky="true"]'); line anywhere in any component.

Id appriciate if you could send an example.

Thanks,

Hi

I would like to enable stick head and sticky portlet [https://keenthemes.com/metronic/preview/demo1/components/portlets/sticky-head.html](see here) for metronic template.

I tried to add data-sticky="true" as it described in the doc but no success.

Any idea how this would be achieved?

Regards,

Hi Ismail,

I am not sure what I am missing here and also wondering whether no one else expereineced this issue...

Sure you can agree with me that time is crucial for systems.

I will put it in this way: Here are what we are trying to achive:

Rule 1 : Capture any datetime (for audit purposes creation time modification time etc) in UTC time regardless of users timezone. (I believe this is achieved by setting the Clock.Provider = ClockProviders.Utc; in core module. Rule 2 : If user selects a Date ( no time just the date or even a month selection) that date should be sent back to the server as it is without converting the date to utc. If user selects a date as 2020-05-01 and the user in British Summer Time zone that date must NOT be sent back to the server as 2020-04-30T11:00. This is wrong!

What we tried:

As we want the capture the times in UTC we anable Clock.Provider = ClockProviders.Utc;. Then you have a selection appears under Settings. What that option should be set to? default is UTC. But do you think we should set that to Servers time zone? I tried both GMT and UTC. Both gives me the same result. What is the purpose of this option and how it should be used?

I tried your code in AppPreBootstrap.ts and nothing has changed.

The only way I could make the code work as I wanted is as below (I have to say that I think there must be a better way for step 3)

1- Set: Clock.Provider = ClockProviders.Utc; in core mode 2- Change default time zone under "Settings" to GMT time zone 3- Change client side code before saving as this.product.dateTo = moment(moment(this.product.dateTo).format(moment.defaultFormatUtc)); (Not sure why I have to wrap this.product.dateTo in a 'moment' even though the type of it is moment.Moment. Without wrapping you can't call format..).

I just don't want to move on because I found a solution that works for me. I'd better understand why it needs the step 3? and how can we overcome that part.