general question

Hello people,

I am currently developing a web interface for accessing
emails
within dbmail directly through that web site, something like
Horde or
Roundcube but specifically for dbmail.
Reading access is handled directly via SQL and changing
stuff I mainly
handle via IMAP.

Two questions have came up though:

1). how can I see if a email has been forwarded? There is a
flag
for answered but forwarded doesn´t seem to be there.

2). is there an easy way to determine if an email has an
attachment or
not? I could use a simple check of size but this is not 100%
accurate
and parsing through all emails and check the mime structure
becomes
quite time consuming.

Well, thanks for your help!

Kind regards,
Daniel
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

Daniel Urstöger wrote:
> Hello people,
> 
> I am currently developing a web interface for accessing
emails
> within dbmail directly through that web site, something
like Horde or
> Roundcube but specifically for dbmail.
> Reading access is handled directly via SQL and changing
stuff I mainly
> handle via IMAP.

Mmm, please consider that storage models can and will
change. The IMAP protocol
and it's RFC extensions are cast in solid rock.

> 
> Two questions have came up though:
> 
> 1). how can I see if a email has been forwarded? There
is a flag
> for answered but forwarded doesn´t seem to be there.

That's correct. Forwarded is not a supported flag (yet).

> 
> 2). is there an easy way to determine if an email has
an attachment or
> not? I could use a simple check of size but this is not
100% accurate
> and parsing through all emails and check the mime
structure becomes
> quite time consuming.

For now:  use IMAP's "FETCH n (BODYSTRUCTURE)"

2.3.0 (coming to you later this month) will support
arbitrary labels. Whether a
message will then be flagged when forwarded depends on the
client. 2.3 will also
use a radically different storage model for messages that
will make it
non-trivial (read near impossible) to reconstruct messages
through simple sql
without doing mime-reconstruction.


> 
> Well, thanks for your help!
> 
> Kind regards,
> Daniel
> _______________________________________________
> DBmail mailing list
> DBmaildbmail.org
> htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail
> 


-- 
 
____________________________________________________________
____
  Paul Stevens                                      paul at
nfg.nl
  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
  The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

Paul J Stevens schrieb:
> Daniel Urstöger wrote:
>> Hello people,
>>
>> I am currently developing a web interface for
accessing emails
>> within dbmail directly through that web site,
something like Horde or
>> Roundcube but specifically for dbmail.
>> Reading access is handled directly via SQL and
changing stuff I mainly
>> handle via IMAP.
> 
> Mmm, please consider that storage models can and will
change. The IMAP protocol
> and it's RFC extensions are cast in solid rock.

yes, I will have to update my script accordingly, it just
gives me
quite an advantage accessing the data directly via SQL and
not havening
IMAP as intermediate layer, performance wise.

>> Two questions have came up though:
>>
>> 1). how can I see if a email has been forwarded?
There is a flag
>> for answered but forwarded doesn´t seem to be
there.
> 
> That's correct. Forwarded is not a supported flag
(yet).

Okay, so how does a mail client recognize this? via an IMAP
command to?

>> 2). is there an easy way to determine if an email
has an attachment or
>> not? I could use a simple check of size but this is
not 100% accurate
>> and parsing through all emails and check the mime
structure becomes
>> quite time consuming.
> 
> For now:  use IMAP's "FETCH n
(BODYSTRUCTURE)"

hmm, (without looking in the source code of dbmail) what
does happen
here? The whole message is opened by the IMAP daemon, the
structure is
extracted from the message and then returned? if so, I could
do that in
a PHP subroutine too and maybe cut the system resources a
bit ...

> 2.3.0 (coming to you later this month) will support
arbitrary labels. Whether a
> message will then be flagged when forwarded depends on
the client. 2.3 will also
> use a radically different storage model for messages
that will make it
> non-trivial (read near impossible) to reconstruct
messages through simple sql
> without doing mime-reconstruction.

well Paul, here we are: basically my script does two things:
first it
generates a list of all messages in the inbox and shows one
of them
accordingly to users settings ( just like the overview in
Outlook /
Thunderbird). So, the overview list is basically the
dbmail_messages
table with a few joins ( like subject, etc.) so far and
showing the
selected messages it has to deal with mime already.
BUT if for the overview I have to open EVERY email completly
and look
around for mime stuff that will significantly drop
performance. Atm the
script is able to handle 2000 emails in the inbox within 2
seconds of
page load time and I like that to keep this that way. If not
possible I
would have to cache information but mainly I have
performance in mind.

and thanks for your swift reply!

cheers,
Daniel

> 
>> Well, thanks for your help!
>>
>> Kind regards,
>> Daniel
>> _______________________________________________
>> DBmail mailing list
>> DBmaildbmail.org
>> htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail
>>
> 
> 
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

Daniel Urstöger wrote:

> yes, I will have to update my script accordingly, it
just gives me
> quite an advantage accessing the data directly via SQL
and not havening
> IMAP as intermediate layer, performance wise.

Misconception. IMAP performance is just fine. But many imap
clients do not
leverage it properly. But the choice is yours of course.

And: there is still plenty room for improving dbmail's imap
performance, I know.

>> That's correct. Forwarded is not a supported flag
(yet).
> 
> Okay, so how does a mail client recognize this? via an
IMAP command to?

yep: the PERMANENTFLAGS line of the SELECT response.


>> For now:  use IMAP's "FETCH n
(BODYSTRUCTURE)"
> 
> hmm, (without looking in the source code of dbmail)
what does happen
> here? The whole message is opened by the IMAP daemon,
the structure is
> extracted from the message and then returned? if so, I
could do that in
> a PHP subroutine too and maybe cut the system resources
a bit ...

Yeah, right  but you
need a very good mime parsing library.

> well Paul, here we are: basically my script does two
things: first it
> generates a list of all messages in the inbox and shows
one of them
> accordingly to users settings ( just like the overview
in Outlook /
> Thunderbird). So, the overview list is basically the
dbmail_messages
> table with a few joins ( like subject, etc.) so far and
showing the
> selected messages it has to deal with mime already.

A typical IMAP call to retrieve such information would be

x UID FETCH 1 (UID
RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc
Subject Date Content-Type)])

which indeed maps to just the kind of joining query you
currently use. The
content-type header will generally tell you whether or not
you need to display
an attachment icon. Retrieving the full bodystructure info
is most likely only
needed to generate the message view-pane, not for the list
pane.

> BUT if for the overview I have to open EVERY email
completly and look
> around for mime stuff that will significantly drop
performance. Atm the
> script is able to handle 2000 emails in the inbox
within 2 seconds of
> page load time and I like that to keep this that way.
If not possible I
> would have to cache information but mainly I have
performance in mind.

Really, if you write a webmail client that needs to retrieve
the body structure
of 2000 message at once, go back to the drawing board. Email
clients that dont
want to build a big offline cache don't need that much
info.

-- 
 
____________________________________________________________
____
  Paul Stevens                                      paul at
nfg.nl
  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
  The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

Paul J Stevens schrieb:
> Daniel Urstöger wrote:
> 
>> yes, I will have to update my script accordingly,
it just gives me
>> quite an advantage accessing the data directly via
SQL and not havening
>> IMAP as intermediate layer, performance wise.
> 
> Misconception. IMAP performance is just fine. But many
imap clients do not
> leverage it properly. But the choice is yours of
course.
> 
> And: there is still plenty room for improving dbmail's
imap performance, I know.

okay, well I can only reflect what I have been testing so
far and
access via IMAP to the database has given me a longer
response time
/ load time compared to fiddle around directly within SQL.

That of course might be a problem with PHP IMAP libraries
too.

>>> That's correct. Forwarded is not a supported
flag (yet).
>> Okay, so how does a mail client recognize this? via
an IMAP command to?
> 
> yep: the PERMANENTFLAGS line of the SELECT response.
> 

okay, but 2.3 will change that? so at least viewing the

>>> For now:  use IMAP's "FETCH n
(BODYSTRUCTURE)"
>> hmm, (without looking in the source code of dbmail)
what does happen
>> here? The whole message is opened by the IMAP
daemon, the structure is
>> extracted from the message and then returned? if
so, I could do that in
>> a PHP subroutine too and maybe cut the system
resources a bit ...
> 
> Yeah, right  but you
need a very good mime parsing library.

PHP offers a native module, I would have to test though if
that is
faster then working with IMAP. Though there is some other
consideration:
I have 2 Servers in place, one handles dbmail, the other one
is
pratically the webserver for PHP etc.

So after fetching the emails content the rest of the
workload is on the
other server, so I can split the load a bit.

The problem here is: currently the system is not loaded at
all,
but soon its gonna have about 20 000 mailboxes and about
that
many users connecting to it. So everything that could split
load,
reduce time etc. is helpful, even if it costs me a day
worktime
to get it coded / working 

>> well Paul, here we are: basically my script does
two things: first it
>> generates a list of all messages in the inbox and
shows one of them
>> accordingly to users settings ( just like the
overview in Outlook /
>> Thunderbird). So, the overview list is basically
the dbmail_messages
>> table with a few joins ( like subject, etc.) so far
and showing the
>> selected messages it has to deal with mime
already.
> 
> A typical IMAP call to retrieve such information would
be
> 
> x UID FETCH 1 (UID
RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc
> Subject Date Content-Type)])
> 
> which indeed maps to just the kind of joining query you
currently use. The
> content-type header will generally tell you whether or
not you need to display
> an attachment icon. Retrieving the full bodystructure
info is most likely only
> needed to generate the message view-pane, not for the
list pane.

okay, so if Content-Type is multipart you would show the
attachment
symbol for that email? Did I get that right?

>> BUT if for the overview I have to open EVERY email
completly and look
>> around for mime stuff that will significantly drop
performance. Atm the
>> script is able to handle 2000 emails in the inbox
within 2 seconds of
>> page load time and I like that to keep this that
way. If not possible I
>> would have to cache information but mainly I have
performance in mind.
> 
> Really, if you write a webmail client that needs to
retrieve the body structure
> of 2000 message at once, go back to the drawing board.
Email clients that dont
> want to build a big offline cache don't need that much
info.

okay, thanks for your tipps, it seems like that is
unecessary after all 
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

On Mon, 2007-10-01 at 16:58 +0200, Daniel Urstöger wrote:
> Paul J Stevens schrieb:
> > Daniel Urstöger wrote:

[snip]
> PHP offers a native module, I would have to test though
if that is
> faster then working with IMAP. Though there is some
other consideration:
> I have 2 Servers in place, one handles dbmail, the
other one is
> pratically the webserver for PHP etc.

The PHP native IMAP module is hard to use efficiently, see
below.

> >> well Paul, here we are: basically my script
does two things: first it
> >> generates a list of all messages in the inbox
and shows one of them
> >> accordingly to users settings ( just like the
overview in Outlook /
> >> Thunderbird). So, the overview list is
basically the dbmail_messages
> >> table with a few joins ( like subject, etc.)
so far and showing the
> >> selected messages it has to deal with mime
already.
> >
> > A typical IMAP call to retrieve such information
would be
> > 
> > x UID FETCH 1 (UID
RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc
> > Subject Date Content-Type)])

Are you doing this on every page load? If so, you will never
have good
performance because you've turned every O(1) page load of a
single
message into an O(n) retrieval of the entire mailbox.

> > which indeed maps to just the kind of joining
query you currently use. The
> > content-type header will generally tell you
whether or not you need to display
> > an attachment icon. Retrieving the full
bodystructure info is most likely only
> > needed to generate the message view-pane, not for
the list pane.

This is the crux of the problem of using the c-client API
that underlies
the native PHP IMAP module -- it's really oriented towards a
stateful
client such as Pine, and as such, retrieving all data in one
call is a
useful optimization. For web this can be a significant
problem.

> okay, so if Content-Type is multipart you would show
the attachment
> symbol for that email? Did I get that right?
> 
> >> BUT if for the overview I have to open EVERY
email completly and look
> >> around for mime stuff that will significantly
drop performance. Atm the
> >> script is able to handle 2000 emails in the
inbox within 2 seconds of
> >> page load time and I like that to keep this
that way. If not possible I
> >> would have to cache information but mainly I
have performance in mind.
> >
> > Really, if you write a webmail client that needs
to retrieve the body structure
> > of 2000 message at once, go back to the drawing
board. Email clients that dont
> > want to build a big offline cache don't need that
much info.

Yep. 1000 messages per second is terrible. What do you do
with someone
who has 10,000 messages? 10 seconds per page view?

Aaron

_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

Aaron Stone schrieb:
> On Mon, 2007-10-01 at 16:58 +0200, Daniel Urstöger
wrote:
>> Paul J Stevens schrieb:
>>> Daniel Urstöger wrote:
> 
> [snip]
>> PHP offers a native module, I would have to test
though if that is
>> faster then working with IMAP. Though there is some
other consideration:
>> I have 2 Servers in place, one handles dbmail, the
other one is
>> pratically the webserver for PHP etc.
> 
> The PHP native IMAP module is hard to use efficiently,
see below.

seems to be the case, yes.

>>>> well Paul, here we are: basically my script
does two things: first it
>>>> generates a list of all messages in the
inbox and shows one of them
>>>> accordingly to users settings ( just like
the overview in Outlook /
>>>> Thunderbird). So, the overview list is
basically the dbmail_messages
>>>> table with a few joins ( like subject,
etc.) so far and showing the
>>>> selected messages it has to deal with mime
already.
>>> A typical IMAP call to retrieve such
information would be
>>>
>>> x UID FETCH 1 (UID
RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc
>>> Subject Date Content-Type)])
> 
> Are you doing this on every page load? If so, you will
never have good
> performance because you've turned every O(1) page load
of a single
> message into an O(n) retrieval of the entire mailbox.

not really, the page is ajax based, so many things do happen
without
any page reload and additional to that, everything retrieved
from the
database gets cached within my program, so for some time
even page
refreshes do not cause new sql querries.

>>> which indeed maps to just the kind of joining
query you currently use. The
>>> content-type header will generally tell you
whether or not you need to display
>>> an attachment icon. Retrieving the full
bodystructure info is most likely only
>>> needed to generate the message view-pane, not
for the list pane.
> 
> This is the crux of the problem of using the c-client
API that underlies
> the native PHP IMAP module -- it's really oriented
towards a stateful
> client such as Pine, and as such, retrieving all data
in one call is a
> useful optimization. For web this can be a significant
problem.

yes, I heard some people having good results by using an
imap proxy,
but I really dislike adding another layer that might cause
problems.
If necessary though, I prolly will have to do that anyhow.

>> okay, so if Content-Type is multipart you would
show the attachment
>> symbol for that email? Did I get that right?
>>
>>>> BUT if for the overview I have to open
EVERY email completly and look
>>>> around for mime stuff that will
significantly drop performance. Atm the
>>>> script is able to handle 2000 emails in the
inbox within 2 seconds of
>>>> page load time and I like that to keep this
that way. If not possible I
>>>> would have to cache information but mainly
I have performance in mind.
>>> Really, if you write a webmail client that
needs to retrieve the body structure
>>> of 2000 message at once, go back to the drawing
board. Email clients that dont
>>> want to build a big offline cache don't need
that much info.
> 
> Yep. 1000 messages per second is terrible. What do you
do with someone
> who has 10,000 messages? 10 seconds per page view?

no, at some point I will have to split the view and make
people
browse forward / backwards. Have not yet decided abotu the
cap.

And time included page load of course, I would have to
check
how much time it takes for sql query to run through and PHP
to calc the things ...

Thanks for your help and input!

> Aaron
> 
> _______________________________________________
> DBmail mailing list
> DBmaildbmail.org
> htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail