Recently I moved my internet stuff (this blog, some apps) over to a new virtual private server (VPS) on Digital Ocean. Here are some notes for myself in 2023 - well, hopefully sooner - when I go to move everything again.
First thing is that 6 years is a pretty long time to go between VPS rebuilds. I had forgotten a bunch of things about how I set the box up, and I found databases hanging around from projects that I had shut down years ago. Probably moving everything once every year or two would be a better interval, even if it is just moving from one instance to another on the same provider. And that would be good motivation for keeping the provisioning scripts up to date; more on that later.
I run my own mail transport agent (Postfix) for thomasleecopeland.com. This is a tremendous pain, but it gives me an appreciation for the far more difficult work done by a real email server administrator or deliverability person. I hd configure SPF on the old box and it was kind of a hassle - I vaguely recall having to manually install some Python packages. I was pleasantly surprised at how much easier it was on the new box. It was basically just install the pypolicyd-spf RPM and configure it in main.cf and master.cf and away it goes. The only tricky thing was that I didn't quite understand the connection between the entry in main.cf - i.e., the check_policy_service entry in main.cf - and the label in master.cf. Also, so far I've been a little disappointed at how little spam it has stopped. But, seems worth doing anyhow.
Along the same lines, I had DKIM set up on the old server and thus set it up on the new one as well. This was also easier than I remember; just install the opendkim RPM, copy over the keys from the old VPS, set up a few configs, and away we go. And there are plenty of testing sites out there that let you verify things. I did have to change my TXT record's name to match the new one, but no big deal there. I haven't set Postfix up to reject email that doesn't have a DKIM signature; from Googling around it seems like that could result in mailing list emails getting rejected. It's good to know that email I'm sending has a better chance of getting delivered, though; the testing sites are showing negative SpamAssassin scores.
More generally for email, I did have to rename my VPS to match my domain name to establish a reverse PTR record. There might be another way to do that, but renaming it worked fine. And a few days after I initially set things up I remembered that I had forgotten to configure greylisting - once I did that (with postgrey), spam volume dropped way down.
The old VPS had a bunch of HTTPS-enabled sites, but when I set those up the Apache RPM that came with it didn't support SNI and thus I had replaced it with a locally compiled Apache 2.2. This meant I had a bunch of files in /usr/local/httpd-2.2 and had hand-edited the init script. Of course I didn't clean up things properly and so had some stuff left over in /etc/httpd. So the new server was a nice cleanup; the default Apache version was 2.4 and everything just worked. I copied over my SSL certificates and my virtual host definitions, but just used the default httpd.conf figuring I can tweak connection counts and pipelining settings and whatnot if it becomes a problem. I've never really had Apache-related load issues since this blog is just static files generated by Jeykll, and my side projects aren't popular enough to cause problems. Hopefully someday that'll be something I have to deal with!
I had also built PostgreSQL from scratch on the old VPS, although thinking about it more I bet I could have used an RPM from PGDG by adding the appropriate Yum repo. But on the gripping hand, the nice thing about building it from scratch is that I was able to more easily upgrade. Anyhow, this time around I installed the latest RPM and so the new VPS has a shiny new 9.6.2 installation. I set up iptables to block all ports except the usual 22/80/443/587, so I configured pg_hba.conf with pretty permissive settings. In my copious spare time maybe I can go back and lock that down and learn a little more about the various PostgreSQL authorization methods.
A big change was shifting from sysv-style init scripts and chkconfig to systemd. It's been a challenge to modify my muscle memory from service httpd restart to systemctl restart httpd. It's all worked pretty smoothly so far, although I need to read up on it more since it still feels really opaque. I'm not sure of best practices around running Rails apps under unicorn with systemd either - I need a config file that starts them on boot, or to use monit or something - so that's still on my list to figure out.
Way back when, I was doing lots of devops stuff and so I wrote a bunch of puppet manifests for the old VPS. I was just running puppet standalone, not using puppetmaster, so nothing fancy. But I failed to maintain those scripts and they bitrotted. I'm kind of conflicted as to whether to put in the effort to document things via manifests on the new box. It certainly would be nice to have configuration files versioned in source control, though, and the manifests wouldn't have forgotten about Postgrey like I did. But I wouldn't have been able to copy over the configuration files because those all changed quite a bit due to the version upgrades. But anyhow, maybe I'll make the effort. Convenience and documentation aside, the sense of superiority I would have with up-to-date provisioning manifests, can you put a price on that? I THINK NOT.
On the old box I had an app running on Ruby 1.8.7 and Passenger. I'm hoping that that app can get moved forward to 1.9.3 because I couldn't even get 1.8.7 to compile on the new box due to openssl incompatibilities. If not, perhaps that might be a chance to do something with docker; seems like an ideal scenario for a self-contained image.
Finally, I've been really happy with everything on Digital Ocean so far. I'm getting twice the RAM for half the cost, the tools have been responsive, the dashboard graphs and whatnot look great, and generally everything has Just Worked. Thanks to Dan Rabinowitz for pointing me this way!
