Search tool crash and burn

Sigh. My little search tool experiment is feeling growing pains already. The search indexes grew to over 230MB and it seems I am klogging my server memory or something. I added a few more heavy sites that people requested and the URL database stopped building.

Anyway, I tried crawling SIGIA-L only and got "Rejected: URL not in the limits!" errors. I may have to work on this more before it's ready for use. Will keep you updated. I tried another perl-based search tool which also can't find the links on archive's index.php file. Ugh. I don't have the patience for dealing with this. I thought it was working, alas it is nearly dead. I successfully indexed March for the time being.

Apologies for prematurely announcing.

I wish Google would index this url daily already: http://info-arch.org/lists/sigia-l/