Cyrus-IMAP Server Dead -- Any suggestions???

Hi all,

If you haven't figured it out yet, I seem to have a talent
for killing 
Kolab servers....  I think I'm on my 5th install of Kolab on
various 
platforms and hardware! Thankfully most haven't had any real
data on 
them yet! 

Server OS: Ubuntu v6.10 "Edgy Eft", all updates
applied.

I finally got a Kolab install completed without incident by
following 
the Ubuntu HOWTO wiki -- no fatal errors, no reasons for
concern. I've 
posted my kolab-build.log file for you all to look at, just
in case 
there is any question... Horde didn't work, but as I
understand it Horde 
is extremely experimental at this point, so I didn't bother
trying to 
make it work.

    http://tinyurl.com/36mpee

In any case, I set up an administrative user (me!) and then
designated 
them the administrative email account. I set all of my
service settings 
without incident. I then added an additional domain, set up
an admin 
user, and assigned them. I did this about 4 times for a
total of about 5 
domains. No incidents.

Tested connectivity to the server using both Mozilla
Thunderbird and 
Kontact by using the Kontact wiki and HowTos available. No
incidents.

All looked good, so I started the process of copying a few
IMAP accounts 
from my exchange server to my kolab server. Thankfully I
didn't *move*, 
I just *copied* the mail.

I'll admit, I was hammering the server pretty hard with
migration 
requests. I had three migrations running in three console
windows on the 
server. Probably not the smartest idea, but I had to test
what would 
happen in real life, right? 

In any case, at 12:10am this morning, my cyrus-imap server
dropped 
offline. All of the migrations failed (obviously). I'm not
sure whether 
my bombarding the server with migrations was the cause, or
whether the 
cause is unrelated.

A restart of the service daemons does not fix it. A reboot
doesn't fix 
it, either.

After looking at the logs, I find the following error each
time I try to 
start the imap daemon

*From /kolab/var/imapd/log/master.log

    /Apr 11 19:16:02 postman <notice> master[11977]:
process started
    Apr 11 19:16:02 postman <debug> master[11980]:
about to exec
    /kolab/bin/ctl_cyrusdb
    Apr 11 19:16:11 postman <error> master[11985]:
setrlimit: Unable to
    set file descriptors limit to -1: (1) Operation not
permitted
    Apr 11 19:16:11 postman <error> master[11985]:
retrying with 1024
    (current max)
    Apr 11 19:16:11 postman <notice> master[11985]:
process started
    Apr 11 19:16:11 postman <debug> master[11986]:
about to exec
    /kolab/bin/ctl_cyrusdb
    Apr 11 19:21:24 postman <error> master[14933]:
setrlimit: Unable to
    set file descriptors limit to -1: (1) Operation not
permitted
    Apr 11 19:21:24 postman <error> master[14933]:
retrying with 1024
    (current max)
    Apr 11 19:21:24 postman <notice> master[14933]:
process started
    Apr 11 19:21:24 postman <debug> master[14971]:
about to exec
    /kolab/bin/ctl_cyrusdb
    /

As you can see, the error repeats itself. There's nothing
more listed.

In case you're wondering, that directory looks like this:

    /rootpostman:/kolab/var/imapd/log# ls -l
    total 3592
    -rw-rw-rw- 1 kolab-r kolab-r  226035 2007-04-11 19:21
cyr_db.log
    -rw-r--r-- 1 kolab-r kolab-r       0 2007-04-11 00:03
imapd.log
    -rw-r--r-- 1 kolab-r kolab-r 3228412 2007-04-11 00:04
imapd.log.0
    -rw-r--r-- 1 root    root     197045 2007-04-11 19:21
master.log
    -rw-r--r-- 1 kolab-r kolab-r     771 2007-04-05 14:41
sieve.log/

I Googled this and couldn't find any definitive solution. It
looks like 
another gentleman had this problem with Kolab, but he opted
to wipe his 
install clean and reinstall. I'd prefer not to take that
route if possible!

If the solution ends up being "delete your cyrus-imap
databases using 
this command and restart", that's cool. Again, I have
all of my data on 
my production (Exchange) email server. I'd prefer to get
kolab up and 
running to decommission the Exchange server, but I'm not at
a critical 
point at the moment.

TIA!
--
Paul Webb
WebbEnabled Solutions, LLC
http://www.webbenabled.com

pwebbwebbenabled.com

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users

Hi, Paul.

Webb, Paul wrote:

[..]

> Server OS: Ubuntu v6.10 "Edgy Eft", all
updates applied.

[..]

>     /Apr 11 19:16:02 postman <notice>
master[11977]: process started
>     Apr 11 19:16:02 postman <debug>
master[11980]: about to exec
>     /kolab/bin/ctl_cyrusdb
>     Apr 11 19:16:11 postman <error>
master[11985]: setrlimit: Unable to
>     set file descriptors limit to -1: (1) Operation not
permitted
>     Apr 11 19:16:11 postman <error>
master[11985]: retrying with 1024
>     (current max)

[..]

I think that one is not fatal. The shell/user under which
cyrus is
running just can't use some resource without limits (if
using bash type:
help ulimit).

Anyway, if this is the only warning/error I'd say that it's
worth a try.
Not sure how this is done properly in Edgy but maybe you can
edit the
init scripts and add some "ulimit unlimited"'s
there?

If this proves to be the cause, maybe it would be better to
add those
ulimits to the official init scripts?

Peace,
Nuno Silva

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users

Nuno Silva wrote:
> I think that one is not fatal. The shell/user under
which cyrus is
> running just can't use some resource without limits (if
using bash type:
> help ulimit).
>
> Anyway, if this is the only warning/error I'd say that
it's worth a try.
> Not sure how this is done properly in Edgy but maybe
you can edit the
> init scripts and add some "ulimit
unlimited"'s there?
>
> If this proves to be the cause, maybe it would be
better to add those
> ulimits to the official init scripts?
>
> Peace,
> Nuno Silva
>   
Hi Nuno,

First, thanks for the fast response!

I'm looking in /etc/init.d/kolab, and it looks to point at
/kolab/etc/rc 
with some parameters. So I closed it and opened the rc
script, and I 
can't find any reference to Cyrus or IMAP anywhere -- not to
say that 
there isn't, because it *has* to call imap somewhere. So,
the question 
is, where/what can I edit to make this change?

Also, I did an "su kolab" and ran ulimit. It
appears to be unlimited. I 
did the same with kolab-n, with the same result.

So, the next question that I have is simple -- if this error
isn't 
fatal, what is causing my imapd to fail loading. Is there a
better place 
to look for some diagnostic output?

Again, thanks for all the help!!

-Paul

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users

Webb, Paul wrote:
> So, the next question that I have is simple -- if this
error isn't 
> fatal, what is causing my imapd to fail loading. Is
there a better place 
> to look for some diagnostic output?
>   
At least on this question, I think I just found my own
answer. In the rc 
script, there's a section that talks about verbose output,
and the 
default is set to off (0). By setting it to on (1), I was
able to glean 
a little more information. If you'd like to see the log,
look at this link:

http://tinyurl.com/2ove67

Hope that helps shed some light on things!
--
Paul Webb
WebbEnabled Solutions, LLC
http://www.webbenabled.com

pwebbwebbenabled.com

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users

"Webb, Paul" <pwebbwebbenabled.com>
writes:

> Webb, Paul wrote:
>> So, the next question that I have is simple -- if
this error isn't 
>> fatal, what is causing my imapd to fail loading. Is
there a better place 
>> to look for some diagnostic output?
>>   
> At least on this question, I think I just found my own
answer. In the rc 
> script, there's a section that talks about verbose
output, and the 
> default is set to off (0). By setting it to on (1), I
was able to glean 
> a little more information. If you'd like to see the
log, look at this link:
>
> http://tinyurl.com/2ove67

Looks normal to me. Nothing I wouldn't expect.

Cheers,

Gunnar

-- 
____ http://www.pardus.de
_________________ http://gunnarwrobel.de _

    >> Mail at ease - Rent a kolab groupware server at
prdus <<

prdus Kolab work is funded in part by KDAB and the
Kolab Konsortium

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users

Hi Paul,

> Also, I did an "su kolab" and ran ulimit. It
appears to be unlimited. I 
> did the same with kolab-n, with the same result.

I did not see this error before so I don't really know how
to fix it,
but googling for "Unable to set file descriptors limit
to -1" turned
up a lot of cyrus references and looking over them I also
got the
impression that your basic Linux system somehow restricts
the imapd
process too much. The user running the cyrus-imapd process
is "kolab-r"
though. Check the limits for this one again.

Cheers,

Gunnar

-- 
____ http://www.pardus.de
_________________ http://gunnarwrobel.de _

    >> Mail at ease - Rent a kolab groupware server at
prdus <<

prdus Kolab work is funded in part by KDAB and the
Kolab Konsortium

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users

Gunnar Wrobel wrote:
> I did not see this error before so I don't really know
how to fix it,
> but googling for "Unable to set file descriptors
limit to -1" turned
> up a lot of cyrus references and looking over them I
also got the
> impression that your basic Linux system somehow
restricts the imapd
> process too much. The user running the cyrus-imapd
process is "kolab-r"
> though. Check the limits for this one again.
>
> Cheers,
>
> Gunnar
>   
As requested, here's the response I got:

   rootpostman:~# su kolab-r
   bash-3.00$ ulimit
   unlimited
   bash-3.00$

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users

Hi.

Webb, Paul wrote:
> Webb, Paul wrote:
>> So, the next question that I have is simple -- if
this error isn't 
>> fatal, what is causing my imapd to fail loading. Is
there a better place 
>> to look for some diagnostic output?
>>   
> At least on this question, I think I just found my own
answer. In the rc 
> script, there's a section that talks about verbose
output, and the 
> default is set to off (0). By setting it to on (1), I
was able to glean 
> a little more information. If you'd like to see the
log, look at this link:
> 
> http://tinyurl.com/2ove67


You have this line:

/kolab/etc/rc.func: line 179: openpkg:rc:NOTICE:: command
not found

Not sure if it's relevant to your setup but this suggests
that you
didn't configure your environment.

Add this to /root/.bashrc

#kolab
export LANG=C
opa () {
   root="${1:-/kolab}"
   if [ ! -d $root ]; then
       echo "opa:ERROR: OpenPKG root directory $root
not found"
       exit 1
   fi
   if [ ! -f "$root/etc/rc" ]; then
       echo "opa:ERROR: root directory does not contain
OpenPKG hierarchy"
       exit 1
   fi
   eval `$root/etc/rc --eval all env`
}

...Then open a new root shell and type:
opa

Then restart the whole kolab:
/kolab/etc/rc -v all restart

Peace,
Nuno Silva

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users

Here are some "basic" questions :

- after you have successfully tested kolab or during your
copy
process, did you install or upgrade anything in your system
?
- is SELinux active ? (did you tried to unactivated it ?)
- When I read the message, I dont see anything that should
prevent the
server to start !
I have the same "error" too !

Mar 24 00:00:49 fc6-eg.asxnet.loc <error>
master[3465]: setrlimit:
Unable to set file descriptors limit to -1: (1) Operation
not
permitted
Mar 24 00:00:49 fc6-eg.asxnet.loc <error>
master[3465]: retrying with
1024 (current max)
Mar 24 00:00:49 fc6-eg.asxnet.loc <notice>
master[3465]: process started

- did you tried to :

# opa /kolab
# /kolab/bin/openpkg rc imapd stop
# /kolab/bin/cyrmaster -D

Do you see any error messages ?

Are you sure there is no other errors messages in
/kolab/var/imapd/log ?






On 4/12/07, Webb, Paul <pwebbwebbenabled.com> wrote:
> Hi all,
>
> If you haven't figured it out yet, I seem to have a
talent for killing
> Kolab servers....  I think I'm on my 5th install of
Kolab on various
> platforms and hardware! Thankfully most haven't had any
real data on
> them yet! 
>
> Server OS: Ubuntu v6.10 "Edgy Eft", all
updates applied.
>
> I finally got a Kolab install completed without
incident by following
> the Ubuntu HOWTO wiki -- no fatal errors, no reasons
for concern. I've
> posted my kolab-build.log file for you all to look at,
just in case
> there is any question... Horde didn't work, but as I
understand it Horde
> is extremely experimental at this point, so I didn't
bother trying to
> make it work.
>
>     http://tinyurl.com/36mpee
>
> In any case, I set up an administrative user (me!) and
then designated
> them the administrative email account. I set all of my
service settings
> without incident. I then added an additional domain,
set up an admin
> user, and assigned them. I did this about 4 times for a
total of about 5
> domains. No incidents.
>
> Tested connectivity to the server using both Mozilla
Thunderbird and
> Kontact by using the Kontact wiki and HowTos available.
No incidents.
>
> All looked good, so I started the process of copying a
few IMAP accounts
> from my exchange server to my kolab server. Thankfully
I didn't *move*,
> I just *copied* the mail.
>
> I'll admit, I was hammering the server pretty hard with
migration
> requests. I had three migrations running in three
console windows on the
> server. Probably not the smartest idea, but I had to
test what would
> happen in real life, right? 
>
> In any case, at 12:10am this morning, my cyrus-imap
server dropped
> offline. All of the migrations failed (obviously). I'm
not sure whether
> my bombarding the server with migrations was the cause,
or whether the
> cause is unrelated.
>
> A restart of the service daemons does not fix it. A
reboot doesn't fix
> it, either.
>
> After looking at the logs, I find the following error
each time I try to
> start the imap daemon
>
> *From /kolab/var/imapd/log/master.log
>
>     /Apr 11 19:16:02 postman <notice>
master[11977]: process started
>     Apr 11 19:16:02 postman <debug>
master[11980]: about to exec
>     /kolab/bin/ctl_cyrusdb
>     Apr 11 19:16:11 postman <error>
master[11985]: setrlimit: Unable to
>     set file descriptors limit to -1: (1) Operation not
permitted
>     Apr 11 19:16:11 postman <error>
master[11985]: retrying with 1024
>     (current max)
>     Apr 11 19:16:11 postman <notice>
master[11985]: process started
>     Apr 11 19:16:11 postman <debug>
master[11986]: about to exec
>     /kolab/bin/ctl_cyrusdb
>     Apr 11 19:21:24 postman <error>
master[14933]: setrlimit: Unable to
>     set file descriptors limit to -1: (1) Operation not
permitted
>     Apr 11 19:21:24 postman <error>
master[14933]: retrying with 1024
>     (current max)
>     Apr 11 19:21:24 postman <notice>
master[14933]: process started
>     Apr 11 19:21:24 postman <debug>
master[14971]: about to exec
>     /kolab/bin/ctl_cyrusdb
>     /
>
> As you can see, the error repeats itself. There's
nothing more listed.
>
> In case you're wondering, that directory looks like
this:
>
>     /rootpostman:/kolab/var/imapd/log# ls -l
>     total 3592
>     -rw-rw-rw- 1 kolab-r kolab-r  226035 2007-04-11
19:21 cyr_db.log
>     -rw-r--r-- 1 kolab-r kolab-r       0 2007-04-11
00:03 imapd.log
>     -rw-r--r-- 1 kolab-r kolab-r 3228412 2007-04-11
00:04 imapd.log.0
>     -rw-r--r-- 1 root    root     197045 2007-04-11
19:21 master.log
>     -rw-r--r-- 1 kolab-r kolab-r     771 2007-04-05
14:41 sieve.log/
>
> I Googled this and couldn't find any definitive
solution. It looks like
> another gentleman had this problem with Kolab, but he
opted to wipe his
> install clean and reinstall. I'd prefer not to take
that route if possible!
>
> If the solution ends up being "delete your
cyrus-imap databases using
> this command and restart", that's cool. Again, I
have all of my data on
> my production (Exchange) email server. I'd prefer to
get kolab up and
> running to decommission the Exchange server, but I'm
not at a critical
> point at the moment.
>
> TIA!
> --
> Paul Webb
> WebbEnabled Solutions, LLC
> http://www.webbenabled.com

> pwebbwebbenabled.com
>
> _______________________________________________
> Kolab-users mailing list
> Kolab-userskolab.org
> https:
//kolab.org/mailman/listinfo/kolab-users
>


-- 
--
Alain Spineux
aspineux gmail com
May the sources be with you

_______________________________________________
Kolab-users mailing list
Kolab-userskolab.org
https:
//kolab.org/mailman/listinfo/kolab-users