Sharing the RubyForge bandwidth load

27 Sep 2005

Robby Russell and Scott Braynard both signed up to host RubyForge file mirrors this weekend, they're serving files even now. You can see all the RubyForge mirrors listed here. Props especially to Dennis Oelkers who volunteered his server to handle 50% of the load - about 110 GB per month!

Just to clarify - only the file releases over a certain size (currently 1 MB) are distributed to the mirrors. The entire site doesn't get mirrored; that'd be tricky since all the pages are dynamically created from the data in the PostgreSQL database.

The way the mirror redirection works is simple; when someone tries to download a large file, the PHP page does a rand() and selects one of the mirrors and does an HTTP redirect (just sends down a new Location header). This is nice since we can adjust it to give more load to one mirror; for example, if rand() > .5 redirect to Dennis' machine. Right now we're not doing any attempt at geographic lookups, although that might not be a bad idea.

Technically, all a mirror provider has to do is have about 1.2 GB of space on a server somewhere and be able to rsync down the files directory once every half hour or so - and of course handle the bandwidth load whch is currently around 30 GB per month.

Thanks much to all the folks who have offered additional mirrors! We're probably OK with current mirrors at the moment, although we may need more in six months or so. But thanks for the offers!