we have situation have grails 2.3.11 based application uses quartz (version 2.2.1 / grails quartz plugin version 1.0.2) jobs long running processes (1-5 minutes) in background polling service allows browser fetch progress. used import , export of data application. example, when application first starts, export 200,000+ rows takes approx 2 minutes. following day export takes 3+ minutes. third day export takes more 6 minutes.
we have narrowed problem down quartz jobs. when system in degraded state other web pages respond identical response times when system in optimal condition. appears quartz jobs tend slowdown linearly or incrementally on period of 2 3 days. may usage related or time, uncertain.
we familiar memory leak bug reported burt beckwith , added fix our code. experiencing memory leak before memory management appears health, when job performance 5-10x slower
the jobs use gorm of queries. we've optimized use criterias projects light weight haven't been able change logic on there number of gorm objects. in case of exports we've changed queries read-only. logic clears out hibernate session appropriately limit number of objects in memory.
here few additional details:
- the level-2 cache disabled
- running on tomcat 7
- using mysql 5.6
- using java jdk 1.7.0_72
- linux
- system i/o, swapping , cpu within reason
- heap , permgen memory usage reasonable
- garbage collection stable , reasonably consistent based on usage levels
- the issue occurs when there single user
- we have done period stack/thread dump analysis
- we have been profiling application xrebel (development) , appdynamics (production) have java melody installed application
- we had problem grails 1.3.8 upgraded 2.3.11 may have exasperated problem.
any suggestions appreciated.
thanks,
john
