09//08/2003: Mail Servers - Spam Filters

[Brangwyn]

Thanks for the heads up on this, but I need to let off a little steam here so appologies in advance if anyone takes this the wrong way, I'm trying to be constructive in my criticisim here and nothing else.

It's rather annoying to tell customers one day we've installed xyz to filter spam and you can do abc to filter it using your client, only to have to tell them 1-2 weeks later oh sorry we are changing it all again, I've already got enough angry customers with the attachment size limit being reduced this isn't really going to help any. I know for many of my clients having reliable email is much more important to them than having reliable webservers, email is so critical to todays business environment, outages of only a few hours can make a difference to their business.

Almost any body/content filtering method is going to be resource hungry perhaps not so much as SA but at least in SA's defense it seems to be doing what it's supposed to based on the reject figures Brad posted a few days ago. Instead of removing it how about seeing if theres a way to load balance the accounts over the servers a little better even if it means adding not 1 but 2 or 3 new mailservers to accomplish it. I think Brad or someone mentioned the load on the servers is around 60-70 or something, I don't see thats a problem sounds like theres still headroom there, whats the disk "trashing" like ? defragged recently ?

I do honestly appreciate your trying to improve the situation here for everyone and the time and effort your putting into it I know we all appreciate, but lets face it the mail service here has been degraded for months now and nothing anyone seems to do does any good for more than a few days or a week or two at best. Maybe it's time to think about alternatives to qmail ?

Consder taking mail.hsphere.cc off ns.hsphere.cc both mail and DNS require as much memory as possible, having "high volume" mail and DNS on the same box isn't a good setup.

Put the absolute maximum Memory in the mailservers that the motherboard will take this is a must

Consider a separate DNS Caching Server just for the mailservers to use so that its separated from client/customer DNS request, again this box should have the absolute maximum memory possible in it.

Perhaps consider LocalDirector for load balancing the work across a few machines ?

What about products like Senmail Mailstream Manager ?

Ive heard good things about Merak Mail Server, but thats windows of course so probably not really an option, it claims to be able to virus scan over 2000 emails per second, sounds fast even if it is just propaganda.

How about testing these changes outside of the live environment and only putting something into production when your certain it's going to work and stay implemented, spend some time building a test environment for high volume mail and play with that a little until your happy with it, you can't just keep changing things every week or two it gets to a point after a while where people start using the "incompetent" word to describe our services.

I'm sorry for the rant, and its not directed at anyone in particular so please Matt don't take anything personal from this as I said I know your trying, I'd just like to see things tried somewhere other than live servers first, you don't like our customers testing scripts on live servers and causing problems so why should we be happy with you guys testing scripts on the live mail servers either ?

I'm sure I've made some huge assumptions here which are probably totally incorrect so please correct me and perhaps we can all work together on this to help find a good reliable working solution that remains scaleable

<flameproof suit on ... or should that be Vixenproof suit?>

[vonbrocklin]

I hear you Brangwyn and hope that Vortech takes a look at some of your ideas. Email is absolutely critical. This is NOT a knock against Vortech, but just an observation:

I have had an account at pair.com for years, and I have NEVER had lost emails, unexplained bounce backs, SPAM downtime, or any real problem. This includes the recent Sobig virus. Completely trouble free. I know that Pair has many more times the clients that Vortech has, and they surely must be facing the same exact issues.

The difference is that they are doing SOMETHING different, and it works. Maybe they have more expertise, or better servers, or less traffic per server, or a better infrastructure. Whatever it is I hope that Vortech takes action now. I do not want to leave here, but I will be FORCED to leave if email does not work. It is a complete deal breaker for me, and it has been one problem after another for weeks on end now. May the force be with you, Vortech, in your efforts to solve this once and for all.

[resell01]

I feel the pain also...I need stable email...

[mresell]

I am concerned the mail servers may be multitasking a bit much as well. Email is one of those services that is a huge priority for businesses and should be treated that way,whatever that takes. I also agree that using realworld servers for "testing" may not be the greatest idea. Constantly, changing something that only results in bad performance, unfortunately, just looks bad to most people and makes them unsettled. Don't want to sound ungrateful...really not the case, but this issue seems to be going on for quite awhile with no real resolution and it is disconcerting. In fact, I am sure vortech in general, is stressed about the situation, but the same approach is obviously, not working.

[MEELAN]

I too with Brangwyn in this case and hope Vortech will do something that will benefit all the parties concerned.

[admin]

Merak Mail Server you have to be kidding me.. LoL That thing would fall over and die in less then 5 min.

We have tried many things 1/2 of them you guys came back and cryied there to tight thats not going to work by mail bounces because my server is black listed.. We install SA on the servers it drives the load sky high.. Its not the mail servers that are under powered or over loaded..

Mail has over 4,000 domains we cut it off at 4,000 because of NS and mysql running on the same system.

mail1 has about 10,000 domains, mail2 has about 8,000 domains, mail3 has close to 4,000 domains. But mail1 gets about 50x more mail then any other server 90% of it spam coming in..

Most of the things you talk about could or maybe could be done with out H-Sphere but because of H-Sphere it can't be done.

I have really put up with this mail issue all I can, tomorrow there will be some REAL changes to stop all this spam from coming in. If it blocks your real mail because its an open realy. Your going to get one answer from us CLOSE THE DAMN relay or ask the RBL to fix it.

I am sick of all the bitching about the mail getting backed up, our spam filters are too tight, it took 30 min for mail to come in ect ect, the mail would not be backed up if there was not over 1,000,000 messages a day coming in to the server with over 90% being spam. But when we stop the spam we have ever body bitching about there mail can't come in because jim bob's server is black listed. TOO BAD GET OFF THE LIST THEN.

Tomorrow this will all change, I am picking 3 of the RBL's to use to block open relays, known spamers, and open proxies, along with SA, and spamguard..

At this point you will have stable mail servers but don't come and crying to me when your uncle bob's mail servers are blocked because there an open relay.. That will be something for you to deal with NOT us, because we are not on that list they are.

It was turned off due to users getting mail bombed, sobig virus wanting to make it last 2 days known. SA will be turned back on..

There will be no more testing, SA will be turned back on with the 3 RBLs we pick, along with spam guard and a few other good spam filters to stop the spam...

It will ether stop it or not.. Yes there may be some real emails lost because of black listing but then again whos fault would that be. The bad admin that got on the list or the good admin keeping the spam out of our mail servers to keep them stable.


**Don't take any thing personal from my post its only a rant and to let you know what I have coming to fix this issue..

[Brangwyn]

Is spam really the issue Brad, if your servers were getting as much email that was legitimate then what will you do then if it doesn't work, you'd be left in exactly the same situation you are today with unreliable mail services.

1,000,000 messages per day, your kidding .. thats all ?? that's barely 12 messages per second almost anything should be ok with that load.

Dont know when you last checked out Merak but I've been hearing some good things from a few friends who own a medium sized ISP's here in NZ about V6 or whatever it is they're up to now.


** Nothing personal taken, I totally understand your frustration, you can understand ours as well I really wanted to try and get us talking about the problem, finding out more about your end of it and perhaps collectively putting our heads together to help find a good solution, but I guess you'll do what you think is best without further suggestions from us.

[Brangwyn]

If it blocks your real mail because its an open realy. Your going to get one answer from us CLOSE THE DAMN relay or ask the RBL to fix it.

Now see theres why people started moaning before, how the hell do you know the emails getting blocked if your not receiving it in the first place ? pretty sure no potential customer whos email to me just bounced is going to ring NZ to let me know about it.

Well over half the people I talk to about spam would rather receive it and deal with it themselves, I guess this is becuase prefer to feel like they are in control of the mail rather than someone else.

Brangwyn,
You make some points, but I think your a little misinformed. Load is based on CPU usage, it has nothing to do with memory. A THREE is high, let alone a 60.

They are all running dual 2.4GHZ Xeons and STILL have 60 loads... a load of 2-4 is nominal for a dual xeon, each load of "1" represents "1" cpu or the number of processes a process has to wait to be processed... so when you have a load of 60, that would require 60 cpus to get done, or make a process have to wait for 60 other processes to finish before being completed. This just stacks and stacks and eventually, we end up with a huge mail queue.

Adding more machines is also something we have been doing. But adding mail3,mail4,mail5,mail6 wont clear up mail1. If half the domains moved off to another mail server, then perhaps it could handle the sheer volume of mail that flows through in a day. Hsphere doesn't make it easy to move mail domains, and people wont like the thought of their mail being deferred because their DNS MX record has to repropagate.

Stats for Monday (mail1)
119925 messages were successfully delivered
35629 messages failed delivery for normal reasons (mailbox doesn't exist, etc.)
175105 messages were identified as spam
163506 messages were identified as spam with a rating >8.

(I believe these stats are pretty accurate, I grepped the logs, but may have forgotten a few strings to grep for. Also, this isn't the statistics of the busiest day in the past week, just monday.)

Thats 319060 total messages that were processed by the machine, of those 55% were spam and 11% were undeliverable. Thats means 66% of messages that the machine processes are junk. This doesn't include the relay attempts, executable attachments blocked, etc.

Thats an average of about 4 messages processed each SECOND, but the mail peaks from 1200-1600EST and slows down greatly in the earlier and later hours. So, the average for the middle of the day is probably more like 10 or 15 per second, and 1 or 2 at night. SpamAssassin just isn't optimized to handle that quantity of mail, maybe if it weren't perl, but the machine would still have to work hard. With an RBL in place, it should reduce the amount of mail requiring processing.

Other stats:
93% of the spam messages scored a rating > 8. Only 7% of the messages identified as spam get sent to mailboxes.

37250 messages with executable attachments were stopped.

SpamAssassin is as far as I know the BEST mail filter. Its downfall is that it is written a perl and even with the spamd/spamc combo, requires massive amounts of CPU power, not memory nor disk space, just pure processing power.

And I did test SpamAssassin, it worked great, but I couldn't estimate that it would throw a huge load on a server with as much traffic as mail1. In fact, MOST of the time, mail1 runs with a 1-2 load, but once it goes up, thats it, filters off , and 2 hours for the queue to clean out.

PS - If I had found another solution, I would have worked in the spam header I announced, and your clients would be none the wiser...

Also, I'm sorry if some of our changes seem "drastic" but with the state of the internet now, everybody is making these quick changes... How many ISPs have blocked 25 in the past 3 weeks due to Sobig - Alot! Hotmail even put a 1MB limit on the same day we put a 5MB limit!

If you think the changes are drastic, then think of it this way... If we didn't put the past couple changes in (blocking EXEs, size limit, etc.) you wouldn't have mail service at all by this point. Sobig's spoofed headers not only generate the original message to someone, but generate thousands of bounces all over the place... At least it supposidly stops spreading tomorrow or wednesday...

[Brangwyn]

Thank you Matt, thats the kind of info and feedback I was hoping for.

sobig d day is the 11th I think I remember reading, and yes thank heck for that being over with finally hopefully (though theres no less than 3 and counting variations I think out there too)

I realise load is based on CPU usage (though I thought Brad had given %'s rather standard nix load figures but probably my mistake). Memory is I believe quite critical as I've always found best practice is to throw absolutely as much memory at any mail machine as possible and I've seen machines go from hardly coping with 50 to 60k messages a day to absolutely flying through them. With RAM as cheap as it is today even for ECC Registered it's probably one of the quickest and cheapest things you can try.

If they're totally CPU bound which sounds like could be the case here then I guess it won't help

I'm really still quite surprised at that mail count though, thats truly a lot lower than I would have expected even just for inbound stats.

My point wasn't that things weren't tested becuase I'm sure they are, but I doubted whether they were tested in an environment similar to a production server, i.e we have scripts we use to generate several thousand messages at a time to try and replicate our prod mail server so we can load test changes before they went out which hopefully lessens our chances of "guessing" how something will perform once it's live.

Hsphere doesn't make it easy to move mail domains, and people wont like the thought of their mail being deferred because their DNS MX record has to repropagate.

Would it have to though ? couldn't some magic with secondary MX's be done ?

PS - If I had found another solution, I would have worked in the spam header I announced, and your clients would be none the wiser...

sweet, that would at least stop us from looking quite so stupid to our customers

SpamAssassin is as far as I know the BEST mail filter

Agree with you there, its certainly the best I've had the plesure of using anyway.

Also, I'm sorry if some of our changes seem "drastic" but with the state of the internet now, everybody is making these quick changes... How many ISPs have blocked 25 in the past 3 weeks due to Sobig - Alot! Hotmail even put a 1MB limit on the same day we put a 5MB limit!

Actually some very well known RBL's have gone off the air this last week or so as well

"....In a recent post from Slashdot, ariehk writes "As of today, Osirusoft, distributer of the SPEWS and open relay blocklists, among others, is no longer operational. Servers using these lists (including the FTC) are currently rejecting ALL email. This shutdown seems to be in response to a several-week-long DDoS attack on Osirusoft, SPEWS and others, resulting in both sites being down. This has caused much discussion on n.a.n-a.e, including the suggestion that the attack is somehow related to the SoBig worm. The spammers must be hurting if they can devote these kinds of resources to attacking blacklists......"

-- taken from W2K News this week http://www.w2knews.com/?id=442 the article on sp@m blocking methods.

[dlrmartin]

Sure, I can understand all reason from matrix.... I can be patient, I can explain to my customers about virus, spam, bla bla bla bla...
They are furious with me! and I cannot says anything, nor more excuses.... I will be lost them

But, this issue never finish!

I cannot understand, as somebody says above, how others biggest and smollers providers (I verified) not SUFFER mail issues, and nor one mail was missed. I'm sure ALL the providers have spamm, virus and troubles. or not ?

I can see you are putting more y more patchs to mails servers, but....

I think that something more deep is wrong here, maybe mail software server.... may be other infraestructure. But I realize that put more patchs and restrictions is not the way.

Please, I like Matrix, I want to keep here!!!!!!!!!

I cannot lost more mails!

ISPs can do a lot of things we cant. They know who their customers are for one, and probably only host one or two mail domains (As opposed to 10,000)... You get one or 5 or 10 email boxes, and most of them have pretty set limits.

We're trying to let you keep the freedom you have with your mail choices, and at the same time provide a great quality of mail service. It takes some work, but we're doin it to serve you better.

PS - Everyone had troubles with Sobig. I lost my ROADRUNNER mail for a few DAYS because of it.

[dlrmartin]

Yes no questions about your work, Im sure you are doing the best. Im very happy with matrix team and hardware (except mail servers )
But you must understand that e-mail is crucial in our bussines and clients bussines, and we are from many time ago with numerous no minor issues.
I mean, may be you should see more deep on troubles, and be sure about software/hardware infraestructure to drive 10,000 domains. May be (I don't know mail software servers so far Ms exchange), software are outrange or is inappropriate for manage big queues, or so many domains....

May be, you can evaluate infraestructure.
Im sure that I'm no win Nobel award for this comment, but I try to help in my benefit.!
continuos working hard! I appreciate it, but We need good results

Email is crucial to us, but when 80% of what goes through a day is junk, it costs us a lot of money on the bandwidth, disk space, and cpu power to deal with that much crap.

I'll be installing two patches later that should help qmail handle high volumes of mail better.

[electricfox]

I don't know much about server administration, but why is it that NS.hsphere and mail are the same server? In my primitive understanding of hosting servers, these seem like the two most performance expensive items in one's lineup, so it behooves me to hear that they are both on one overworked server? Is there some sort of advantage or h-sphere requirement that keeps them the same server?