Issue:
After upgrade to Domino 11.0.1, there are Java crash files.
Each set of dump files (2 DMP, 1 TRC, and 1TXT) files are 650+MB each, most in the one of the .dmp files. The crashes are intermittent and can happen a few a day to a few every 5 to 15 minutes.
The end result is that you can fill up a hard disk overnight.
In the Domino systemctl service logs, messages similar to the following are listed, one for each set of crash files:
Aug 06 09:13:14 appserver.mindwatering.com java[7853]: IBM Java[7853]: JVMDUMP032I JVM requested Snap dump using '/local/notesdata/Snap.20200806.091231.7853.0003.trc' in ...o an event
Aug 06 09:13:14 appserver.mindwatering.com java[7830]: IBM Java[7830]: JVMDUMP032I JVM requested Snap dump using '/local/notesdata/Snap.20200806.091225.7830.0003.trc' in ...o an event
Aug 06 09:13:14 appserver.mindwatering.com java[7919]: IBM Java[7919]: JVMDUMP032I JVM requested Snap dump using '/local/notesdata/Snap.20200806.091250.7919.0003.trc' in ...o an event
In the console.log file, we found the following message at the same times:
"... could not connect to Tika process..."
$ sudo netstat -tunlp
Does not include a Tika service at port 9998. No process is started by the Domino server. In addition, checking other app servers running Sametime and Traveler, and mail services not having this issue, also do not have a Tika service, but no crashing, either.
OS/Domino Version:
Cent OS 7.7 and 8.0 running Domino 11.0.1 and Domino 11.0.1 FP1. (Upgrading to FP1 does not fix this issue.)
Workaround:
Create cron script and add to cron.d to purge the crash files on a 15 minute schedule.
Solution:
The issue caused by the Indexer task. An open-source Tika task is started and typically is supposed to stay resident to process attachments.
We added to the Domino server the following notes.ini updates:
Switch/login as the notes user and update the notes.ini:
# su notes
$ cd /local/notesdata
$ vi notes.ini
...
TIKA_PORT=9998
JavaMaxHeapsize=1024M
<esc :wq>
Other Notes:
- We tried JavaMaxHeapsize=512M, but the crashes persisted.
- The JavaMaxHeapsize is not related to the JVM entries for HTTP for XPages and HTTP JVM agents.
(HTTPJVMMaxHeapSizeSet=1 and HTTPJVMMaxHeapSize=2048M)
The Tika process is started by the Indexer process, which makes sense. Once the Tika service is kicked off the Tika server should show up in netstat and stay running/persist until the service is shutdown. The process won't say indexer, but java.
tcp6 0 0 127.0.0.1:9998 :::* LISTEN 20860/java
previous page
|