mod_security woes

26 Feb 2008

A customer had an unfortunate experience with mod_security recently. They were getting occasional HTTP 500 responses from their Rails app on certain large pages. ExceptionNotifier wasn't reporting any stacktraces, and a check of the actual log file didn't show any problems either. Even connecting to the production machines and running the same request using script/console's app.get worked fine!

Finally they took an entire slice out of their production architecture and made the request while watching the Apache logs. And lo and behold - mod_security was seeing a large response and returning a 500 code. This was a surprise since mod_security had been (we thought) configured in "logging-only" mode.

Lessons learned are 1) load up the staging environment with lots of data to shake out any such issues, 2) study the mod_security settings to ensure it'll do what's expected, and 3) if ExceptionNotifier and app.get tells you no exception is happening in the Rails app, widen your search.