Search Indexing OOM

[dpwrussell]

I was attempting to reindex my search index and ran into this problem a lot. Good news is that a lot of reindexing did happen and search is beginning to work. However, the indexing process is hitting an OutOfMemory error quite frequently. The reindexing process may eventually get past these, but the thing with search is that to make it useful, it needs to be reaching 100% coverage or it can't be relied upon.

Code: Select all

2014-02-28 05:39:14,873 WARN  [        ome.services.util.ServiceHandler] (3-thread-3) Unknown exception thrown.

java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-02-28 05:39:14,873 WARN  [        ome.services.util.ServiceHandler] (3-thread-3) Method interface ome.services.util.Executor$Work.doWork invocation took 5428
2014-02-28 05:39:14,874 ERROR [             org.quartz.core.JobRunShell] (3-thread-3) Job DEFAULT.fullTextThreadRun threw an unhandled Exception:
org.springframework.scheduling.quartz.JobMethodInvocationFailedException: Invocation of method 'run' on target class [class ome.services.fulltext.FullTextThread] failed; nested exception is ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
   at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:273) ~[org.springframework.context.support.jar:3.0.1.RELEASE-A]
   at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86) ~[org.springframework.context.support.jar:3.0.1.RELEASE-A]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:203) ~[quartz.jar:1.5.2]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
   at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
2014-02-28 05:39:14,874 ERROR [             org.quartz.core.ErrorLogger] (3-thread-3) Job (DEFAULT.fullTextThreadRun threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception.
   at org.quartz.core.JobRunShell.run(JobRunShell.java:214) ~[quartz.jar:1.5.2]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
   at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]


My bug report: http://trac.openmicroscopy.org.uk/ome/ticket/12034
JB's ticket with new search fixes: https://trac.openmicroscopy.org.uk/ome/ticket/11948

Cheers

[jballanc]

I believe I've discovered a potential cause for the continued OOM conditions. It seems that our ServiceHandler is keeping BufferedReader objects around even though we've already closed them. I've opened a new ticket to track the work on fixing this, though if the fix is sufficient hopefully we will be able to close two tickets with one patch.

If you would like to follow along with this new issue the ticket is: http://trac.openmicroscopy.org.uk/ome/ticket/12049