It took me [insert large number here] years of Rails and Ruby but I finally saw this in my logs:

Invalid or incomplete POST parameters

But the parameters were fine! It was just an innocuous XML document coming in through the API! After some flailing - basically bisecting a request payload - I reproduced it with:

$ curl -si -X POST -d '%' http://localhost/ | head -1
HTTP/1.1 400 Bad Request

To cut to the chase, adding a content type header solves it. With text/xml, we get a garden-variety 404 error since there's no route for a POST to /:

$ curl -si -H 'content-type: text/xml' -X POST -d '%' http://localhost/ | head -1
HTTP/1.1 404 Not Found

So the root cause is that if there's no explicit content type on a request, Rack attempts to parse it as if it were application/x-www-form-urlencoded and % is invalid input in that case.

There are a couple of interesting elements to this one. As far as the exception is concerned, most blog posts and Stack Overflow questions around this error involve users mistyping URLs and putting in consecutive percent signs or something. So those are GET requests, not POSTs, and thus not immediately relevant. Also, the exception comes from Rack. So the logs won't have a stracktrace and the usual error reporting tools won't get a chance to show a good post-mortem for this.

Poking around in Rack gets us moving though. Here's a comment from lib/rack/request.rb:

# This method support both application/x-www-form-urlencoded and
# multipart/form-data.

So there's a reference to x-www-form-urlencoded, which refers to RFC 1738 for encoding, which explains why a percent sign, or a string like 15% of net would be invalid input.

Supposing you can't add a content-type header to the code that's making the requests? Perhaps the POST is coming from a third party. In that case, Rack middleware to the rescue. Nothing fancy, just:

class Rack::AddTheHeader
  def initialize(app)
    @app = app
  end
  def call(env)
    if env['PATH_INFO'] == "/some/path"
      env['CONTENT_TYPE'] = 'text/xml'
      Rack::Request.new(env).body.rewind
    end
    @app.call(env)
  end
end

This scopes the header addition to a specific URL space, which seems prudent. That's about it, hope this saves someone a few minutes!