Invalid or incomplete POST parameters

But the parameters were fine! It was just an innocuous XML document coming in through the API! After some flailing - basically bisecting a request payload - I reproduced it with:

$ curl -si -X POST -d '%' http://localhost/ | head -1
HTTP/1.1 400 Bad Request

To cut to the chase, adding a content type header solves it. With text/xml, we get a garden-variety 404 error since there's no route for a POST to /:

$ curl -si -H 'content-type: text/xml' -X POST -d '%' http://localhost/ | head -1
HTTP/1.1 404 Not Found

So the root cause is that if there's no explicit content type on a request, Rack attempts to parse it as if it were application/x-www-form-urlencoded and % is invalid input in that case.

There are a couple of interesting elements to this one. As far as the exception is concerned, most blog posts and Stack Overflow questions around this error involve users mistyping URLs and putting in consecutive percent signs or something. So those are GET requests, not POSTs, and thus not immediately relevant. Also, the exception comes from Rack. So the logs won't have a stracktrace and the usual error reporting tools won't get a chance to show a good post-mortem for this.

Poking around in Rack gets us moving though. Here's a comment from lib/rack/request.rb:

# This method support both application/x-www-form-urlencoded and
# multipart/form-data.

So there's a reference to x-www-form-urlencoded, which refers to RFC 1738 for encoding, which explains why a percent sign, or a string like 15% of net would be invalid input.

Supposing you can't add a content-type header to the code that's making the requests? Perhaps the POST is coming from a third party. In that case, Rack middleware to the rescue. Nothing fancy, just:

class Rack::AddTheHeader
  def initialize(app)
    @app = app
  end
  def call(env)
    if env['PATH_INFO'] == "/some/path"
      env['CONTENT_TYPE'] = 'text/xml'
      Rack::Request.new(env).body.rewind
    end
    @app.call(env)
  end
end

This scopes the header addition to a specific URL space, which seems prudent. That's about it, hope this saves someone a few minutes!

Initially I was writing little one-off scripts to do this stuff. For each record, grab it, munge the JSON, save it. But that was kind of stressful since I was always one typo away from messing up the data and having to re-import or restore from the audit records. Unit tests help, of course, but having to write a new script each time was tedious too.

Thinking more about the problem, it felt less like an imperative "if key x is present then do y" and more like a series of instructions, or transformations, that we were running on the data. This felt nicer, more functional; send in a hash, get back another hash with a slight modification. Also, why spend 15 minutes writing a script when you can automate it in a day? This line of thinking resulted in a little flurry of classes like this:

  # h = {a: 2, b: 3}
  # CopyInst.new(:a, :z).call(h)
  # {:a => 2, :b=>3, :z=>2} # new h
  class CopyInst
    attr_reader :from, :to
    def initialize(from, to)
      @from = from
      @to = to
    end
    def call(hash)
      hash[to] = hash[from]
    end
  end

It's not quite functional because it modifies the hash argument rather than dup'ing. But each little instruction (CopyInst, RenameInst, and a couple of other business-specific ones) was easy to understand and test. And after instantiating a series of instructions you just iterate over them and call each one in turn:

def munge(hash)
  instructions.each do |inst|
    inst.call(hash)
  end
  hash
end

And out the other side comes a cleaned up hash suitable for to_json-ing.

Expressing these transformations as a series of instructions - a 'program' - suggests some interesting possibilities for the thing that is executing them - the 'virtual machine'. We could validate those instructions so that each one verifies it can be run on the target hash; check for missing keys for RenameInst, checks for already existing keys for the CopyInst, and so on. We could coalesce instructions to reduce the number of method calls. We could detect no-op sequences (copy 'a' to 'b' followed by copy 'b' to 'a') or common subexpressions. If there were non-technical users you could imagine defining a little user interface to let end users express whatever changes they wanted in terms of these instructions, and then save off that 'program' and run it. And there's probably an undo stack in there somewhere as well. All kinds of nifty compiler-ish things!

This experiment turned what was drudgery into an interesting and fun exercise. Always a win!

... lots of build process output ....
linking miniruby
Undefined symbols for architecture x86_64:
  "_je_calloc", referenced from:
      _rb_objspace_alloc in gc.o
... and much more ....

One fix is a small change to the autoconf script, configure.in. Ruby 2.4.1's configure.in script had this but 2.4.2 didn't:

4211a4212,4214
> AS_CASE(["$with_jemalloc: $LIBS "], [no:* | *' -ljemalloc '*], [],
> 	[LIBS="-ljemalloc $LIBS"])
>

This kind of makes sense. Per the docs, AS_CASE is a macro that generates a case statement and adding -ljemalloc to a list of libraries seems like what's needed here. It results in this change to the generated Makefile, which also seems reasonable:

\79c79
< LIBS = -lpthread -ldl -lobjc $(EXTLIBS)
---
> LIBS = -ljemalloc -lpthread -ldl -lobjc $(EXTLIBS)

And then you can verify jemalloc is linked with:

$ ruby -e "p RbConfig::CONFIG['LIBS']"
"-ljemalloc -lpthread -ldl -lobjc"

Another way to verify it is to run a script under dtruss and the process will attempt to read a jemalloc config file:

$ sudo dtruss -t readlink ruby -e "42"
... some output ...
readlink("/etc/je_malloc.conf\0", 0x7FFF58DAFB80, 0x400)		 = -1 Err#2
... more output ...

If jemalloc isn't linked in, that readlink system call won't happen.

Side note - at first I attempted to verify that jemalloc was included using FFI. I figured I'd just call a function that was declared in jemalloc.c. So I did:

require 'ffi'
module Foo
  extend FFI::Library
  ffi_lib 'libjemalloc.dylib'
  attach_function :je_malloc_usable_size, [ ], :int
end
puts "Usable size: #{Foo.je_malloc_usable_size}"

But then I realized that doesn't make sense. ffi_lib will load up the specified library regardless of whether it's linked into the Ruby process, so that would never fail.

Finally, if you're using Ruby 2.5.1, it's the same change except since this commit the build process uses configure.ac rather than configure.in. But just paste in that same snippet on line 4045, bam done.

ber db:migrate:down VERSION=20180319191352

I mean! Those are precious seconds wasted. Well, no more. Just put this snippet in your .bash_profile:

_rake_migrate() {
  COMPREPLY=()
  if [ $3 != "db:migrate:down" ]; then
    return 0
  fi
  local cur="${COMP_WORDS[COMP_CWORD]}"
  IFS='=' read -r -a array <<< "$cur"
  local version_number_part=${array[1]}
  if [ "$COMP_CWORD" -eq 2 ]; then
    local res=$(find db/migrate/ -type f -exec basename {} \; | grep ${version_number_part})
    COMPREPLY=($(compgen -W "$res" ))
  fi
}

complete -F _rake_migrate rake

Now when you type rake db:migrate:down VERSION=2018 and hit the tab key Bash will list out all the migration version files from 2018. And when you select one, even though we're placing the entire filename to the right of the VERSION= it still works. That's because the db:migrate:down Rake task uses String#to_i to process the VERSION environment variable, and that handles it. Per the documentation, String#to_i "Returns the result of interpreting leading characters in str as an integer base base (between 2 and 36). Extraneous characters past the end of a valid number are ignored."

>> "42_asdf".to_i
=> 42

Hats off to Mike English for his very helpful Bash completion article!

For example, you might have an Order that has_one CustomLogo, but rather than a foreign key it relies on matching values for several other fields: origin and client_id. Here's how you could manage that:

class Order < ApplicationRecord
  has_one :custom_logo,
    ->(order) {
      unscope(:where).where(
        origin_value: order.origin,
        client_id:    order.client_id)
    }
end

I wish I had thought of this technique, but credit goes to Dwight on Stack Overflow. It's a neat way to handle this situation, but it does have a few pitfalls. One issue is that since the scope uses attributes from a specific object instance, we can't eagerly load all the logos for a collection of orders:

>> Order.all.eager_load(:custom_logo)
Traceback (most recent call last):
ArgumentError (The association scope 'custom_logo' is instance dependent (the scope block takes an argument). Preloading instance dependent scopes is not supported.)

The same thing happens with an includes relation. Active Record will raise this exception regardless of whether the association scope actually uses the argument. That's a pretty reasonable design choice since the alternative is to attempt to do a data flow analysis inside the scope's Proc.

You'll also run into problems when using some of the methods generated by a has_one declaration. If you attempt to assign a record via the association accessor you'll get an exception around the missing foreign key:

>> Order.first.custom_logo = CustomLogo.last
Traceback (most recent call last):
        1: from (irb):4
ActiveModel::MissingAttributeError (can't write unknown attribute `order_id`)

Incidentally, to me the unscope call looks like a complicated part of this scope. But all it does is replace any existing where clause objects in the relation with nil. And it returns the relation, so we can then define a new where clause like we do in this example. Notice that this is different than ActiveRecord::Relation#rewhere, which lets you redefine the where clause for a particular attribute.

Even more incidentally, how does the association scope only return one record even though the where clause doesn't constrain the resulting relation? This is a more general has_one question, and the answer is in ActiveRecord::Associations::SingularAssociation#find_target:

def find_target
  # ... some code that's interesting but not relevant ...
  sc = reflection.association_scope_cache(conn, owner) do
    StatementCache.create(conn) { |params|
      as = AssociationScope.create { params.bind }
      target_scope.merge(as.scope(self, conn)).limit(1)
    }
end

So limit(1) explains that.

Finally, there's some interesting behavior around creation helper methods. Using the association helper with no arguments to create a record results in only one CustomLogo record being created, no matter how many invocations:

$ be rails runner "CustomLogo.destroy_all ; \
Order.first.create_custom_logo ; \
Order.first.create_custom_logo ; \
Order.first.create_custom_logo rescue nil ; \
puts CustomLogo.count"
1

This makes sense because Active Record is using the equality Arel nodes in the association scope to create the record. So if the first Order has an origin_value of "facebook" and a client_id of 42, so will the one associated CustomLogo. The rescue nil is there because when we call create_custom_logo after the first record is in place, ActiveRecord::Associations::HasOneAssociation#replace calls ActiveRecord::Associations::Association#set_owner_attributes with an order_id value in an attempt to set a foreign key. That raises the same MissingAttributeError exception that we saw earlier, so I'm rescuing that to prevent the script from exiting early.

The interesting thing is what happens if you pass arguments to the creation helper; calling that repeatedly will create a bunch of records:

$ be rails runner "CustomLogo.destroy_all ; \
Order.first.create_custom_logo(origin_value: 'twitter', client_id: 42) rescue nil; \
Order.first.create_custom_logo(origin_value: 'twitter', client_id: 42) rescue nil; \
Order.first.create_custom_logo(origin_value: 'twitter', client_id: 42) rescue nil; \
puts CustomLogo.count"
3

This goes through a different code path. All record creations, even the first one, call set_owner_attributes and thus we need to rescue the MissingAttributeError every time.

Pulling back a little on this, if origin_value and client_id form effectively a composite unique key, a good way to prevent odd data would be to add a partial uniqueness constraint on those two columns.

To sum up, you can define associations without foreign keys - but don't be surprised when some things don't work the way they normally do.