Conversation
…er.tika.timeout The parse ran on the executor thread with no time limit, so a document which kept a parser busy held that thread for as long as it took, and the tuple timed out and was replayed into the same parse. The new key parser.tika.timeout sets the maximum time in milliseconds a document may take to parse. When it is set, the parse runs on a separate thread and the bolt waits for it up to that time. A document which takes longer is sent to the status stream as an ERROR with the message "parse timeout" and counted under error_parse_timeout. The parsing thread is interrupted, and the content handler stops the parse at its next SAX event; a parser which neither produces output nor checks for interrupts keeps its thread until it returns, so the bolt moves on to a new thread for the following documents. Parse errors and the text limit behave as before. The default stays -1 so the behaviour does not change unless the key is set.
|
I may misunderstand how Tika is used in StormCrawler, but interrupting a Tika thread has no guarantees that it will stop parsing unless you're using Tika pipes parser? |
|
If you want to test this rigorously, a) I can send you a fun PDF or b) you can use our MockParser from tika-core test jar to instrument a parser that takes up a lot of CPUs and never stops. |
|
Ref: https://tika.apache.org/docs/4.0.x/using-tika/java-api/index.html I'm drafting a new page on how to call tika pipes within Java. Have your 🤖 see if the PipesForkParser is a good fit. |
|
Thanks Tim. Yes, the interrupt only works if the parser still writes to the handler, no? If it loops without output the thread keeps running, the bolt just stops waiting for it. The docs do overclaim :D Please send me the PDF, I'll try it against this branch. |
|
@tballison As my go-to-TIKA expert: would you recommend PipesFork for SC? ;-) |
|
Yes! Please! Then you'd actually be in Tika's security model! There are caveats about sizing/cpus etc. That's something that stormcrawler has to figure out. I can help. |
|
I convert this to draft and have my agents take a look at the PDF you send me + PipesFork for SC. Once this is in and tested, I will re-open this for ready to review ;-) - happy to get your feedback on it. |
|
I'm having an agent draft a PR. |
Would be great :) closing here ! |
|
Thx @tballison |
2 participants
Fixes #2097 (the remaining part after #2174).
Adds
parser.tika.timeoutin milliseconds, default -1 (no limit). When set, the parse runs on a separate thread and a document which takes longer is sent to the status stream as an ERROR withparse timeout. The thread is interrupted and the parse stops at its next SAX event; a parser stuck without output keeps its thread and the following documents use a new one.