Open Source Document Management

Knowledge in Action
https://forums.logicaldoc.com/

Too many open files

https://forums.logicaldoc.com/viewtopic.php?f=6&t=599

Page 1 of 1

Too many open files

Posted: Wed Aug 01, 2012 2:57 pm
by Rabbit80
Hi,

I'm using Tomcat6 + Ubuntu 10.04 +LD 6.4

Anyway - the search service stops working frequently. In the logs, it appears that we are running into a "Too many open files" error.

Do you have a solution?

Code: Select all

2012-08-01 13:38:35.373	ERROR	com.logicaldoc.core.searchengine.Search	/home/LDRepository/index/en/_6y8.fdx (Too many open files)
java.io.FileNotFoundException:/home/LDRepository/index/en/_6y8.fdx (Too many open files)
    	at java.io.RandomAccessFile.open(Native Method) 
    	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) 
    	at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:70) 
    	at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:97) 
    	at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:92) 
    	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) 
    	at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:133) 
    	at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:212) 
    	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117) 
    	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93) 
    	at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:113) 
    	at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:29) 
    	at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81) 
    	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754) 
    	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75) 
    	at org.apache.lucene.index.IndexReader.open(IndexReader.java:462) 
    	at org.apache.lucene.index.IndexReader.open(IndexReader.java:308) 
    	at com.logicaldoc.core.searchengine.FulltextSearch.search(FulltextSearch.java:92) 
    	at com.logicaldoc.core.searchengine.FulltextSearch.internalSearch(FulltextSearch.java:68) 
    	at com.logicaldoc.core.searchengine.Search.search(Search.java:124) 
    	at com.logicaldoc.webservice.search.SearchServiceImpl.find(SearchServiceImpl.java:44) 
    	at sun.reflect.GeneratedMethodAccessor388.invoke(Unknown Source) 
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    	at java.lang.reflect.Method.invoke(Method.java:616) 
    	at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173) 
    	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) 
    	at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:60) 
    	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:75) 
    	at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
    	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) 
    	at java.util.concurrent.FutureTask.run(FutureTask.java:166) 
    	at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) 
    	at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106) 
    	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243) 
    	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111) 
    	at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99) 
    	at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431) 
    	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:186) 
    	at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148) 
    	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179) 
    	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:103) 
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) 
    	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159) 
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) 
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    	at com.logicaldoc.web.CharsetFilter.doFilter(CharsetFilter.java:26) 
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    	at com.logicaldoc.web.SessionFilter.doFilter(SessionFilter.java:61) 
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    	at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:164) 
    	at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141) 
    	at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:90) 
    	at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:417) 
    	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
    	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) 
    	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) 
    	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465) 
    	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
    	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
    	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) 
    	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) 
    	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) 
    	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) 
    	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) 
    	at java.lang.Thread.run(Thread.java:636)
Your help is appreciated.

Cheers

Shane

Re: Too many open files

Posted: Wed Aug 01, 2012 3:53 pm
by Rabbit80
OK - I think I have found the problem - a ulimit -a command shows :open files (-n) 1024

It seems to be a bit of a pain in Ubuntu to set this higher - but I got there eventually by modifying:
/etc/security/limits.conf
adding the lines:
* soft nofile 65535
* hard nofile 65535

and also:
/etc/pam.d/su
uncommenting the line:
session required pam_limits.so

The internets also mentioned the same line as above in the /etc/pam.d/common-session file, however I found this made no difference to the limit.

What values should I use? And why does Logicaldoc appear to leave these files open?

Re: Too many open files

Posted: Wed Aug 01, 2012 4:16 pm
by car031
LogicalDOC doesn't leave opened files, but the search engine some times needs to unpack the index and it will create huge set of files.

In logicaldoc.sh you have:
ulimit -Hn 6000
ulimit -Sn 6000
ulimit -v unlimited

these seems to be good in most cases, but this may depends on the environment, this is what we call the Linux *

Re: Too many open files

Posted: Wed Aug 01, 2012 4:23 pm
by Rabbit80
OK, thanks for that.

Not too sure that I understand this correctly though - would this mean that Logicaldoc was simply hitting the 1024 open file limit imposed by Ubuntu expecting to be able to open 6000 files? What impact does changing the values in logicaldoc.sh actually have?

Cheers

Shane

Edit: Just noticed I do not have logicaldoc.sh - is this the installer for the commercial version?

Re: Too many open files

Posted: Wed Aug 08, 2012 10:05 am
by car031
the logicaldoc.sh is in the commercial edition, yes.
All times are UTC+01:00
Page 1 of 1
Powered by phpBB® Forum Software © phpBB Limited
https://www.phpbb.com/