HMM got lmtp dupicate entries HOW and WHY?

I havent looked at that code for some time, but there is a
unique key on
(physmessage_id, headername_id, headervalue(255))

This exact condition is guarded against in the code. The
only thing I can think
of is that it might happen if you have a message with two
headers with the same
headername, whose headervalues only differ beyond char 255.

Drop the unique key involved or tell me more about your
setup.

Simon Lange wrote:
> Mar  3 14:08:02 primary dbmail/lmtpd[21419]:
Message:[serverchild]
> serverchild.c,PerformChildTask(+346): incoming
connection from
> [127.0.0.1 (localhost)] by pid [21419]
> 
> Mar 03 14:08:02 primary dbmail-lmtpd[21419]:
Error:[sql]
> dbmysql.c,db_query(+290): [Duplicate entry
> '303737-16-rootgrey.gnetx.com' for key 3] [INSERT INTO
> dbmail_headervalue (headername_id, physmessage_id,
headervalue) VALUES
> (16,303737,'rootgrey.gnetx.com')]
> 
> Mar 03 14:08:02 primary dbmail-lmtpd[21419]:
Error:[message]
> dbmail-message.c,_header_cache(+1033): insert
headervalue failed
> 
> Mar  3 14:08:02 primary dbmail/lmtpd[21419]:
Error:[sql]
> dbmysql.c,db_query(+290): [Duplicate entry
> '303737-16-rootgrey.gnetx.com' for key 3] [INSERT INTO
> dbmail_headervalue (headername_id, physmessage_id,
headervalue) VALUES
> (16,303737,'rootgrey.gnetx.com')]
> 
> Mar  3 14:08:02 primary dbmail/lmtpd[21419]:
Error:[message]
> dbmail-message.c,_header_cache(+1033): insert
headervalue failed
> 
> Mar  3 14:08:02 primary dbmail/lmtpd[21419]:
Message:[sort]
> sort.c,sort_deliver_to_mailbox(+196): message
id=767905, size=46731 is
> inserted
> 
>  
> 
> I don’t feeling comfortable seeing these errors. Well
at last its
> inserted but what about the errors?! How could it come
that lmtpd
> produces duplicate entries when the id is
autoincrement?!
> 
>  
> 
> Anyone?!
> 
> 
>
------------------------------------------------------------
------------
> 
> _______________________________________________
> DBmail mailing list
> DBmaildbmail.org
> htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail


-- 
 
____________________________________________________________
____
  Paul Stevens                                      paul at
nfg.nl
  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
  The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

HI PAUL,

I DID UPDATE FROM 2.2.6RC1 TO MOST RECENT VERSION 2.2.9
BEFORE AND AFTER I DID RUN A FULL MAINTENANCE OF DBMAIL.
I ALREADY TRIED REMOVE THE SPECIFIC ROW BUT THE ERROR CAME
UP AGAIN FOR ANOTHER MAIL THEN.
SO ITS SOMEHOW RELATED TO LMTP BUT DON€™T KNOW WHY AND HOW.
THERE IS ONLY ONE INSTANCE OF DBMAIL RUNNING
(IMAPD,POP3D,LMTPD,TIMSIEVED) AND LMTP IS EXCLUSIVE STORING
MAILS (NO DBMAIL-SMTP RUNNING OR USING).

REGARDS

SIMON


-----URSPRüNGLICHE NACHRICHT-----
VON: DBMAIL-BOUNCESDBMAIL.ORG [MAILTOBMAIL-BO
UNCESDBMAIL.ORG] IM AUFTRAG VON PAUL J STEVENS
GESENDET: MONTAG, 3. MäRZ 2008 15:42
AN: DBMAIL MAILINGLIST
BETREFF: RE: [DBMAIL] HMM GOT LMTP DUPICATE ENTRIES HOW AND
WHY?


I HAVENT LOOKED AT THAT CODE FOR SOME TIME, BUT THERE IS A
UNIQUE KEY ON
(PHYSMESSAGE_ID, HEADERNAME_ID, HEADERVALUE(255))

THIS EXACT CONDITION IS GUARDED AGAINST IN THE CODE. THE
ONLY THING I CAN THINK
OF IS THAT IT MIGHT HAPPEN IF YOU HAVE A MESSAGE WITH TWO
HEADERS WITH THE SAME
HEADERNAME, WHOSE HEADERVALUES ONLY DIFFER BEYOND CHAR 255.

DROP THE UNIQUE KEY INVOLVED OR TELL ME MORE ABOUT YOUR
SETUP.

SIMON LANGE WROTE:
> MAR  3 14:08:02 PRIMARY DBMAIL/LMTPD[21419]:
MESSAGE:[SERVERCHILD]
> SERVERCHILD.C,PERFORMCHILDTASK(+346): INCOMING
CONNECTION FROM
> [127.0.0.1 (LOCALHOST)] BY PID [21419]
> 
> MAR 03 14:08:02 PRIMARY DBMAIL-LMTPD[21419]:
ERROR:[SQL]
> DBMYSQL.C,DB_QUERY(+290): [DUPLICATE ENTRY
> '303737-16-ROOTGREY.GNETX.COM' FOR KEY 3] [INSERT INTO
> DBMAIL_HEADERVALUE (HEADERNAME_ID, PHYSMESSAGE_ID,
HEADERVALUE) VALUES
> (16,303737,'ROOTGREY.GNETX.COM')]
> 
> MAR 03 14:08:02 PRIMARY DBMAIL-LMTPD[21419]:
ERROR:[MESSAGE]
> DBMAIL-MESSAGE.C,_HEADER_CACHE(+1033): INSERT
HEADERVALUE FAILED
> 
> MAR  3 14:08:02 PRIMARY DBMAIL/LMTPD[21419]:
ERROR:[SQL]
> DBMYSQL.C,DB_QUERY(+290): [DUPLICATE ENTRY
> '303737-16-ROOTGREY.GNETX.COM' FOR KEY 3] [INSERT INTO
> DBMAIL_HEADERVALUE (HEADERNAME_ID, PHYSMESSAGE_ID,
HEADERVALUE) VALUES
> (16,303737,'ROOTGREY.GNETX.COM')]
> 
> MAR  3 14:08:02 PRIMARY DBMAIL/LMTPD[21419]:
ERROR:[MESSAGE]
> DBMAIL-MESSAGE.C,_HEADER_CACHE(+1033): INSERT
HEADERVALUE FAILED
> 
> MAR  3 14:08:02 PRIMARY DBMAIL/LMTPD[21419]:
MESSAGE:[SORT]
> SORT.C,SORT_DELIVER_TO_MAILBOX(+196): MESSAGE
ID=767905, SIZE=46731 IS
> INSERTED
> 
>  
> 
> I DON€™T FEELING COMFORTABLE SEEING THESE ERRORS. WELL
AT LAST ITS
> INSERTED BUT WHAT ABOUT THE ERRORS?! HOW COULD IT COME
THAT LMTPD
> PRODUCES DUPLICATE ENTRIES WHEN THE ID IS
AUTOINCREMENT?!
> 
>  
> 
> ANYONE?!
> 
> 
>
------------------------------------------------------------
------------
> 
> _______________________________________________
> DBMAIL MAILING LIST
> DBMAILDBMAIL.ORG
> HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL


-- 
 
____________________________________________________________
____
  PAUL STEVENS                                      PAUL AT
NFG.NL
  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
  THE
NETHERLANDS________________________________HTTP://WWW.NFG.NL

_______________________________________________
DBMAIL MAILING LIST
DBMAILDBMAIL.ORG
HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL

_______________________________________________
DBMAIL MAILING LIST
DBMAILDBMAIL.ORG
HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL

Simon Lange wrote:
> Hi Paul,
> 
> i did update from 2.2.6rc1 to most recent version
2.2.9
> before and after I did run a full maintenance of
dbmail.
> I already tried remove the specific row but the error
came up again for another mail then.
> So its somehow related to lmtp but don’t know why and
how.
> There is only one instance of dbmail running
(imapd,pop3d,lmtpd,timsieved) and lmtp is exclusive storing
mails (no dbmail-smtp running or using).

I'd appreciate a level=5 log of such an event (the log for
the complete message,
not just the error)

-- 
 
____________________________________________________________
____
  Paul Stevens                                      paul at
nfg.nl
  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
  The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

LOG OF LMTPD? OR ANY OTHER PROCESSES AS WELL? IM ASKING
SINCE LVL5 PRODUCES HUGE AMOUNTS OF LOG ;)

REGARDS

SIMON

-----URSPRüNGLICHE NACHRICHT-----
VON: DBMAIL-BOUNCESDBMAIL.ORG [MAILTOBMAIL-BO
UNCESDBMAIL.ORG] IM AUFTRAG VON PAUL J STEVENS
GESENDET: MONTAG, 3. MäRZ 2008 16:00
AN: DBMAIL MAILINGLIST
BETREFF: RE: AW: [DBMAIL] HMM GOT LMTP DUPICATE ENTRIES HOW
AND WHY?

SIMON LANGE WROTE:
> HI PAUL,
> 
> I DID UPDATE FROM 2.2.6RC1 TO MOST RECENT VERSION
2.2.9
> BEFORE AND AFTER I DID RUN A FULL MAINTENANCE OF
DBMAIL.
> I ALREADY TRIED REMOVE THE SPECIFIC ROW BUT THE ERROR
CAME UP AGAIN FOR ANOTHER MAIL THEN.
> SO ITS SOMEHOW RELATED TO LMTP BUT DON€™T KNOW WHY AND
HOW.
> THERE IS ONLY ONE INSTANCE OF DBMAIL RUNNING
(IMAPD,POP3D,LMTPD,TIMSIEVED) AND LMTP IS EXCLUSIVE STORING
MAILS (NO DBMAIL-SMTP RUNNING OR USING).

I'D APPRECIATE A LEVEL=5 LOG OF SUCH AN EVENT (THE LOG FOR
THE COMPLETE MESSAGE,
NOT JUST THE ERROR)

-- 
 
____________________________________________________________
____
  PAUL STEVENS                                      PAUL AT
NFG.NL
  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
  THE
NETHERLANDS________________________________HTTP://WWW.NFG.NL

_______________________________________________
DBMAIL MAILING LIST
DBMAILDBMAIL.ORG
HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL

_______________________________________________
DBMAIL MAILING LIST
DBMAILDBMAIL.ORG
HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL

HTTP://WWW.SIMONLANGE.DE/LMTP-LVL5.LOG

YOU MAY FETCH IT IF YOU WANT. DISABLED LVL5 AGAIN ;)

REGARDS

SIMON

-----URSPRüNGLICHE NACHRICHT-----
VON: DBMAIL-BOUNCESDBMAIL.ORG [MAILTOBMAIL-BO
UNCESDBMAIL.ORG] IM AUFTRAG VON PAUL J STEVENS
GESENDET: MONTAG, 3. MäRZ 2008 16:00
AN: DBMAIL MAILINGLIST
BETREFF: RE: AW: [DBMAIL] HMM GOT LMTP DUPICATE ENTRIES HOW
AND WHY?

SIMON LANGE WROTE:
> HI PAUL,
> 
> I DID UPDATE FROM 2.2.6RC1 TO MOST RECENT VERSION
2.2.9
> BEFORE AND AFTER I DID RUN A FULL MAINTENANCE OF
DBMAIL.
> I ALREADY TRIED REMOVE THE SPECIFIC ROW BUT THE ERROR
CAME UP AGAIN FOR ANOTHER MAIL THEN.
> SO ITS SOMEHOW RELATED TO LMTP BUT DON€™T KNOW WHY AND
HOW.
> THERE IS ONLY ONE INSTANCE OF DBMAIL RUNNING
(IMAPD,POP3D,LMTPD,TIMSIEVED) AND LMTP IS EXCLUSIVE STORING
MAILS (NO DBMAIL-SMTP RUNNING OR USING).

I'D APPRECIATE A LEVEL=5 LOG OF SUCH AN EVENT (THE LOG FOR
THE COMPLETE MESSAGE,
NOT JUST THE ERROR)

-- 
 
____________________________________________________________
____
  PAUL STEVENS                                      PAUL AT
NFG.NL
  NET FACILITIES GROUP                     GPG/PGP:
1024D/11F8CD31
  THE
NETHERLANDS________________________________HTTP://WWW.NFG.NL

_______________________________________________
DBMAIL MAILING LIST
DBMAILDBMAIL.ORG
HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL

_______________________________________________
DBMAIL MAILING LIST
DBMAILDBMAIL.ORG
HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL

Simon Lange wrote:
> 
> <snip>
> 

I started getting the same errors that Simon describes right
after upgrading 
to 2.9. Worst part is that this error actually has some very
nasty 
consequences. Please look at bug #685.

Thanks

Peter
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Rabbitson wrote:
> Simon Lange wrote:
>>
>> <snip>
>>
> 
> I started getting the same errors that Simon describes
right after
> upgrading to 2.9. Worst part is that this error
actually has some very
> nasty consequences. Please look at bug #685.
> 
> Thanks
> 
> Peter


seems that this problem happens whenever the mail being
inserted already
has a Return-Path header. dbmail also adds a return path
header and if
(as can happen) the value of that header is the same as the
existing
one, then there is the duplicate entry.

I think that it would be a good idea to add a guard to
dbmail to remove
existing Return-Path headers so that this problem does not
occur.

In the meantime it is probably best to configure the MTA not
to add the
Return-Path header.

John



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4-svn0 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org


iD8DBQFHzSnWd4I3jTtt9EIRAj8XAKDOXDMwMOzvv18s87mMMdOwF8Gn0QCf
S1fL
qpc3V/5wcqVm/OAMZDKeI+0=
=nLaj
-----END PGP SIGNATURE-----
_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

John Fawcett wrote:
> I think that it would be a good idea to add a guard to
dbmail to remove
> existing Return-Path headers so that this problem does
not occur.

Errrr probably not add anything if a duplicate is detected.
I definitely do 
not want my headers tampered with unless necessary.

> In the meantime it is probably best to configure the
MTA not to add the
> Return-Path header.
> 

Email often passes through dbmail twice, so the MTA will not
solve anything. I 
have a number of dbmail forwards like:

ordersdomain: sales+ordersdomain

so the email loops through the MTA and comes back to the
correct mailbox. This 
is to work against stupid forms/systems that do not support
'+'

_______________________________________________
DBmail mailing list
DBmaildbmail.org
htt
ps://mailman.fastxs.nl/mailman/listinfo/dbmail

JUST READ THE BUG ENTRY AND I CAN CONFIRM THAT IT IS
ABSOLUTE THE SAME BUG I HAVE. AFTER PETER HAS POINTED TO
THUNDERBIRD I DID TRY TB AND HAD THE VERY SAME ISSUE. I DID
UPDATE BECAUSE OF THE SIEVE ERROR IN 2.2.6RC1 BUT NOT FOR
THE PRICE OF INCONSISTENT DATABASE-INFORMATION.

FIRST THING I ALSO NOTICED WHEN RUNNING DBMAIL UTIL AFTER
2.2.6RC1 -> 2.2.9 WAS THAT DBMAIL KILLED 730 UNCONNECTED
PHYSMESSAGES.... WELL, I WONDERED A BIT ABOUT THIS BUT SINCE
IM IN THE SAME STABLE TREE WITH 2.2.9 I DID NOT SUSPECT
DBMAIL DOIN SOMETHING WRONG... UP TO NOW NO COMPLAINS BY
CUSTOMERS.

SIMON

-----URSPRüNGLICHE NACHRICHT-----
VON: DBMAIL-BOUNCESDBMAIL.ORG [MAILTOBMAIL-BO
UNCESDBMAIL.ORG] IM AUFTRAG VON PETER RABBITSON
GESENDET: DIENSTAG, 4. MäRZ 2008 10:20
AN: DBMAIL MAILINGLIST
BETREFF: RE: [DBMAIL] HMM GOT LMTP DUPICATE ENTRIES HOW AND
WHY?

SIMON LANGE WROTE:
> 
> <SNIP>
> 

I STARTED GETTING THE SAME ERRORS THAT SIMON DESCRIBES RIGHT
AFTER UPGRADING 
TO 2.9. WORST PART IS THAT THIS ERROR ACTUALLY HAS SOME VERY
NASTY 
CONSEQUENCES. PLEASE LOOK AT BUG #685.

THANKS

PETER
_______________________________________________
DBMAIL MAILING LIST
DBMAILDBMAIL.ORG
HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL

_______________________________________________
DBMAIL MAILING LIST
DBMAILDBMAIL.ORG
HTTPS://MAILMAN.FASTXS.NL/MAILMAN/LISTINFO/DBMAIL