IssueRaiser

LLRC should detect I/O Reactor failure and log the halting exception

LLRC makes use of the Apache HTTPComponents Async Client. The client is powered by an internal I/O Reactor that reacts to and dispatches IO events as they happen. In some cases, this I/O Reactor can encounter exceptions from higher up in the call stack or even from the Java NIO library. If these exceptions are not handled, the I/O Reactor shuts down and leaves the RestClient instance stuck. It can be difficult to discern what causes the reactor to shut down, as the offending exceptions are not always written to the log. Despite exceptions not being logged to a logger, they are collected in an internal audit log on the IOReactor implementation. We should find a way to: 1) Detect that the I/O Reactor has stopped. 2) Log the contents of the I/O Reactor to the client's logger. This would simplify the process of troubleshooting why the rest client has died and allow for a better exception to be thrown rather than the arcane `I/O reactor status: STOPPED` message.

Posted by jbaiera on Nov 14, 2019, 10:53 PM

Tags

Related issues

Raise real-world problems, find people facing the same issue, and discuss solutions.3,500 issues0 comments

reliability (Software)

LLRC should detect I/O Reactor failure and log the halting exception

jbaieraNov 14, 2019, 10:53 PM

LLRC makes use of the Apache HTTPComponents Async Client. The client is powered by an internal I/O Reactor that reacts to and dispatches IO events as they happen. In some cases, this I/O Reactor can encounter exceptions from higher up in the call stack or even from the Java NIO library. If these exceptions are not handled, the I/O Reactor shuts down and leaves the RestClient instance stuck. It can be difficult to discern what causes the reactor to shut down, as the offending exceptions are not always written to the log. Despite exceptions not being logged to a logger, they are collected in an internal audit log on the IOReactor implementation. We should find a way to: 1) Detect that the I/O Reactor has stopped. 2) Log the contents of the I/O Reactor to the client's logger. This would simplify the process of troubleshooting why the rest client has died and allow for a better exception to be thrown rather than the arcane `I/O reactor status: STOPPED` message.

Comments (0)

No comments yet. Start the discussion.

Comment on this issue