Several calls just came in, asking why our ordering system was unavailable. Users couldn't login anymore on the initial web page. The web-based J2EE application is hosted on a JBoss-3.2.7 server, and normally sends out error mail when fatal errors occur. Accessing the JBoss JMX console works fine, and the log did not display an obvious problem.
In the end, it was "netstat" coming to the rescue: while the application server keeps growing open connection, there was one, single connection which stayed in half open mode. That was a TCP connection to a remote SMTP server. A quick call to the operations department revealed a new problem with the mail server, and after a reboot, the ordering system came back on track easily.
The bottom line: we must make our application less dependable on external services. Asynchronous mailing in both application and logging, and possible stricter timeout settings on network connections, like SMTP. This will probably require further investigation in the log4j framework with AsyncAppender and SMTPAppender.
No comments:
Post a Comment