Performance decays exponentially when inserting bulk data into grails app -

Before we run the demonstration test, we need to apply with 3 million institutions to simulate 3 million entities 3 years of real data simulation Should be loaded through the application.

We are putting 1-5000 units at a time. Initial reaction is very good in time, but after some time they rapidly decay.

We use on Groovy script to kill a URL to begin each round of the inclusion.

1. Restarting the application resets the response time - that means it temporarily fixes the problem.
2. The script's rearrange, without restarting the app, has no effect.

We use the following to increase performance

1) After every 100 entries, GORM:

  def Session = sessionfineCR session session Full () Session.clear () DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP.get (). Clear ()  

(Old Ted Nellied trick :)

2) We use GPars for parallel insertion:

  GParsPool.withPool {(0 .. .. lt; 1000) .eachParallel {def unit = new unit (...) insertion service. Institution (unit)}}  

Notes

• Looking at the log output, I have found that the processing time for each unit is the same, There is a break between the frequency now and longer.
• The correct number of entities involved is not important, just about 3 milli, so if some f

help

I'm really doing

There is a good idea about fixing a person's problem.

Environment

• Grails: 2.4.2 (GRAILS_OPTS = -Xmx2G-Xms512m -XX: MaxPermSize = 512m)
• Java: 1.7.0_55
• MBP: OS X 10.9.5 (2,6 GHz Intel Core i7, 16 GB 1600 MHz DDR3)

Due to Pausing I think this JVM is collecting garbage. Have you used visual profiles like a profiler to see if the time is spent for collecting the cubebase? Generally speaking, this would be the best way to understand what is happening in your app within JVM.

In addition, loading data directly into the database is better, if you execute according to the "seed" application course.

(added as a comment)