Distributed mail filters/relays

Hi,

I currently have multiple backup mailservers, located in
different physical locations for redundancy purposes, which
currently run spamassassin and clamav with a static ruleset.
I would like to switch these over to running maia, so that
users can manage their own spam filtering rules, however i
understand the database needs to be stored centrally, which
results in the mailservers being unable to queue mail if the
database is unavailable.

I could change the mail backups to be dumb backups, with no
filtering and just forward direct to filtering machines, but
this is far from optimal...

MySQL replication won't work, since each filtering box
needs to be able to write to the database, and mysql
replication is really just one-way...

Is there any way to have a distributed system like this,
which perhaps just holds mail if the database is
unavailable, and filters it if the database becomes
available again...
Or perhaps keep a local cache of the database, and queue up
the database updates until it becomes available again?

What have other people done in situations like this? Or is
it best to have all my filtering machines in one place close
to the database?


_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

> I currently have multiple backup mailservers, located
in different
> physical locations for redundancy purposes, which
currently run

This is a difficult problem no doubt.  Would that Mysql
allowed
distributed multimaster replication. LOL

WRT the idea of having the database at one location:  Sure,
the remote MX
could queue up and send it through once the database is
available.  Since
there is probably a lot more database traffic between
amavisd-maia and the
database than between the MTA and amavisd-maia, I'd also
have the
amavisd-maia process near the database.

This brings to the point of questioning why a remote MX is
necessary.  I
know it has been standard policy for a lot of organizations,
but I
question how often it is really a necessity.  One site
properly designed
and multihomed can be quite sufficient and redundant, aside
from major
catatrophic events.   One site also allows for easier
clustering, backups,
and maintenance.

In case of a major catastrophe, like a data center fire,
earthquake,
tornado... etc, you could use drdb or something over a
dedicated link to a
remote site to keep a remote site up to date... and in case
of major
problems, cut over to the backup site.

Obvously, maintaining a redundant remote site is more
expensive, so I'd do
a cost analysis to be sure it is really needed. I'm just
saying this
because I think a lot of people are worrying about it when
it's not that
important costwise. 

I'd love to hear any ways to distribute the database, as
it's the only
part of Maia that doesn't cluster well.
_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

On Fri, 2006-06-23 at 10:02 -0500, mortondadgrmm.net
wrote:

> Obvously, maintaining a redundant remote site is more
expensive, so I'd do
> a cost analysis to be sure it is really needed. I'm
just saying this
> because I think a lot of people are worrying about it
when it's not that
> important costwise. 

I can't speak for every organization, but we have the same
sorts of
things going here, and we just have multiple campuses in the
same city
linked by high-speed networks. What it comes down to is that
this is
often more of a political issue than a technical one. I can
make the
technical argument that we have a reliable network, a
wireless backup
system in case the fiber links fail, generators and UPS in
the machine
room to guard against power failures, etc. And I can point
out that we
don't have the 24x7 operations staff at the other sites
either, that
maintaining machines located far from our offices is more
consuming of
staff time, we can't respond as quickly when problems do
come up, etc.
But there are still people who constantly push for us to
duplicate
services such as DNS and mail at all three campuses. To
them, it's just
a few thousand dollars for another set of servers and a
perception (not
at all based on reality) that cross-city links are somehow
less reliable
than the ones inside the buildings. Yes, a cross-city link
is more
likely to fail due to circumstances beyond our control (e.g.
a fiber cut
caused by road construction equipment), but I would bet that
the overall
failure rate is not much worse than our inter-building
networks, and
those are also extremely reliable. 

So I completely agree with David here, but some of us have
to deal with
distributed services anyway.


> I'd love to hear any ways to distribute the database

I'm using DRBD to do it. But I have a dedicated GigE link
between the
two servers. Now that I have some experience with DRBD, I
can say that
it works very well. In three months, I have only had one
failure that
was related to DRBD (some kind of I/O error caused the file
system to
get remounted read-only, which brought everything to a halt
until the
primary server was rebooted). DRBD has two problems though.
The most
obvious one is performance. If you have an application that
really
pushes disk I/O, the necessity to replicate every write
across the
network is going to slow things down. I have no idea what
the
performance of DRBD would be over a slower offsite link. The
second
problem is that only one of the two servers can have the
replicated file
system(s) mounted at a time. This means you couldn't use
DRBD to
replicate the database between two servers if you want to
have them both
serving mail at the same time. The second server can only be
a hot spare
which takes over as primary (and mounts the replicated file
system) only
when the primary fails.

In my case, there is no doubt that the necessity to store
every message
in a single database is the biggest problem with Maia;
almost every
problem I've had with the mail system since installing Maia
has ended up
having something to do with the database. So I too would be
interested
in any solutions to that problem.

--Greg


_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

Le vendredi 23 juin 2006 à 10:02 -0500, mortondadgrmm.net
a écrit :

> I'd love to hear any ways to distribute the database,
as it's the only
> part of Maia that doesn't cluster well.

This is a generic Database problem, not maia specific.

We dont use MySQL since this is not a "real"
database engine, like
PostgreSQL or some other commercials are.
I know that MySQL has put (since 3.23.15) transactional and
some real
production needed code (heh, from BerkleyDB ..), but at the
moment, it
still a step behind for a production database (and lack of
SQL
compliance doesnt help)

You can find commercial products that support off-site load
balancing
and / or high avail., but as you said, this has a real
impact on your
wallet or the performances you can achieve with.

For example, there is a master-slave replica system on Pg
called Slony
(http://gborg.postgresql.org/project/slony1/projdisplay
.php). This is a
great product, that fit some of the maia needs :

o) You can setup master / replica env, but read only on
replica
o) Replicas can be elected has master if the link
master-replica goes
down (on site survivability if you loose the main link)
o) The new master can be master for other slave (mean that a
elected
master can control other lost replicas)
o) If old master come online again, it will still offline,
until human
can clean up replicas, to keep the most up-to-date datas.


As you can read, this is a very powerful system, that let
you control
how your database can be shared between multiple hosts, for
redundancy
and load sharing (move a replica near massive reads users).

The downside is the quite complex system that you have to
take care, and
the performance loss (each write has to be replicated on all
replicas).

So I really agree, you really need to check pros / cons,
since those
solutions are always complex.

-- 
        Alexandre

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

On Fri, 2006-06-23 at 17:47 +0200, Alexandre Ghisoli wrote:

> 
> o) You can setup master / replica env, but read only on
replica

Unless I've missed something, this isn't going to help
with Maia,
because amavisd absolutely needs write access to the
database. A
read-only replica is useless except as a standby hot spare
that could
become writable if the master went down. So this is about
the same as
having DRBD. On the other hand, I don't know what the
performance over a
remote link is of DRBD vs. Slony.

--Greg


_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users