At 14:02 UTC on a Tuesday, our search cluster stopped responding. Here is how an innocent thread pool config change killed node 3.
We had modified search thread pool queue size from 1000 to unbounded in an attempt to prevent request rejection under traffic spikes.
Founder of developerOS & CloudPrep. Former Infrastructure Engineer and Systems Architect.