RubyForge Mailman cleanup

12 Sep 2005

Andrew Dunstan noticed that lots of mailing lists on PGFoundry had pending messages. I checked RubyForge to see if we had the same problem, and yup, we've got 313 MB of pending messages.

Andrew posted this series of shell commands to tot up the worst offenders:

ls -1 /var/mailman/data | grep heldmsg \
| sed -e 's/heldmsg-//' -e 's/-[0-9]*\.pck//' \
| sort | uniq -c | sort -rn

So I wrapped it in a little Ruby script that creates the data on one run, puts it in a file, and then sends emails to the list admins on the next run:

class Job
  attr_accessor :listname, :adminaddress, :msgcoun
    def Job.load
    jobs = []
    File.read('rpt.txt').each_line do |line|
      jobs << Job.new
      jobs.last.listname, jobs.last.adminaddress, jobs.last.msgcount = line.split(',')
    end
    jobs
  end
  def to_s
    '#{listname},#{adminaddress},#{msgcount}'
  end
end
def raw_data
  msgs = `ls -1 /var/mailman/data | grep heldmsg \
    | sed -e 's/heldmsg-//' -e 's/-[0-9]*\.pck//' \
    | sort | uniq -c | sort -rn`
  f = File.open('rpt.txt', 'w')
  msgs.each_line {|line|
    next if line.strip.split(' ')[0].to_i < 20
    j = Job.new
    j.msgcount, j.listname = line.strip.split(' ')
    j.adminaddress = `/var/mailman/bin/list_owners #{j.listname}`.strip.gsub(/\n/,'|')
    f.write('#{j}\n')
  }
  f.close
end

Job.load.each do |j|
  begin
    #next if j.adminaddress != 'tom@infoether.com'
    txt = File.read('message.txt').gsub(/listname/, j.listname).gsub(/msgcount/, j.msgcount)
    File.open('message.tmp.txt', 'w') {|f| f.write(txt)}
    cmd = 'mail #{j.adminaddress.gsub(/\|/,',')} -s \'RubyForge mailing list cleanup\' < message.tmp.txt'
    puts cmd
    `#{cmd}`
  rescue Exception=>e
    puts e
    puts 'Failed to send to #{j}'
  end
end

Note the little gizmo to run the Mailman list_owners script and get multiple owners... fun stuff! And as I type this up, folks are cleaning up their lists - that directory's gone from 313 MB to 298 MB already!

Update: Also, Lyle Johnson asked about discard non-member postings. To do that, go to your admin page, click 'Privacy Options' and click on 'Sender filters'. At the bottom of the page there's a 'generic_nonmember_action' option at the bottom of the page. Just select 'discard' to, well, discard non-member postings. Alternatively there's the DEFAULT_MAX_DAYS_TO_HOLD option which can discard postings after 10 days or whatever; that's available now that we've upgraded to 2.1.6.