Re: DBmail webmail app

Tom Allison wrote:
> 
> So if I add one line at the end of a word doc, this
will be considered
> to be the same file unless the file format includes
something unique
> within the first 8MB of the file?

My point also. Doing sha1 on the whole file seems to make
the most sense. People
who want to do bigger files faster can simply add more lmtpd
hosts with bigger
cpus and more ram.

The proposed schema has some merit, but I would rather setup
two new tables
(partslists and mimeparts) which would be functionally like
Jonathan's two
tables. Main difference is I want to leave the current
tables as they are: new
messages get inserted into the new setup, and old messages
are left in the old
format until converted by a lo-pri run of dbmail-util. The
message retrieval
code would then first try to reconstruct the message from
the new tables and
failing that use the 'old' current setup.


-- 
 
____________________________________________________________
____
  Paul Stevens                                      paul at
nfg.nl
  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
  The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

Paul J Stevens wrote:
> Tom Allison wrote:
> 
> My point also. Doing sha1 on the whole file seems to
make the most sense. People
> who want to do bigger files faster can simply add more
lmtpd hosts with bigger
> cpus and more ram.
> 

Why is the hashing performance penalty considered such an
issue? All of 
the hashing algorithms are streaming, hence all daemons
capable of 
accepting a message (smtp/lmtp/imap) can call a utility
function on the 
data that is unloaded from the socket. Or no?
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

>
> The proposed schema has some merit, but I would rather
setup two new tables
> (partslists and mimeparts) which would be functionally
like Jonathan's two
> tables. Main difference is I want to leave the current
tables as they are: new
> messages get inserted into the new setup, and old
messages are left in the old
> format until converted by a lo-pri run of dbmail-util.
The message retrieval
> code would then first try to reconstruct the message
from the new tables and
> failing that use the 'old' current setup.
>   
While we are talking about adding tables has anybody thought
of using
archive type tables?
As our data is stuff that generally compresses well and
doesn't change
much compressed type tables could also lead to some
significant storage
savings. Obviously its not for everybody but if we are
talking about
scanning multiple tables for records anyway ;->
Perhaps its something that could be done with flags? If you
flag an imap
folder as "archive" then stuff in it will get
shuffled into a different
table by the util? with any luck this should avoid the
performance hit
of having to scan 2 tables all the time for the data.

Its probably not a good idea to compress the data outside of
the storage
engine (ie in the dbmail SW itself) as you would probably
loose indexing
and the like (and full text indexing/searching would
probably be the
greatest use of the archive)
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

Sounds migratory...
But it might be the best approach.

I think at the current state of the art the following is
reasonably true:
SHA1/MD5 is fast enough for the typically allowed file sizes
in a mail
system (10MB) given the frequency with which they occur.

If someone has a future need of 100MB or 10GB file sizes as
the norm then
we can also expect to be looking a quad-core also being the
norm --
making all things equal.

If they have a current need for 100MB files, then they
probably already
have a really big server at their disposal.

One thing I do like about dbmail is the scalability.  I
interpret this to
mean both small and large mail environments can use with
application
with existing or reasonably hardware.  To require super huge
machines
for a personal domain isn't very scalable.

On 6/4/2007, "Paul J Stevens" <paulnfg.nl> wrote:

>
>Tom Allison wrote:
>>
>> So if I add one line at the end of a word doc, this
will be considered
>> to be the same file unless the file format includes
something unique
>> within the first 8MB of the file?
>
>My point also. Doing sha1 on the whole file seems to
make the most sense. People
>who want to do bigger files faster can simply add more
lmtpd hosts with bigger
>cpus and more ram.
>
>The proposed schema has some merit, but I would rather
setup two new tables
>(partslists and mimeparts) which would be functionally
like Jonathan's two
>tables. Main difference is I want to leave the current
tables as they are: new
>messages get inserted into the new setup, and old
messages are left in the old
>format until converted by a lo-pri run of dbmail-util.
The message retrieval
>code would then first try to reconstruct the message
from the new tables and
>failing that use the 'old' current setup.
>
>
>--
> 
____________________________________________________________
____
>  Paul Stevens                                      paul
at nfg.nl
>  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
>  The Netherlands________________________________http://www.nfg.nl
>_______________________________________________
>DBmail mailing list
>DBmaildbmail.org
>htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail