ip_tables.c: mark_source_chains: bad negative verdict

Hello there,

I've upgraded to kernel 2.6.21.6 / iptables 1.3.7 and now a
big firewall table 
fails to load. The error message from the iptables command
is
"iptables: Too many levels of symbolic links", so
I've enabled debugging in 
net/ipv4/netfilter/ip_tables.c. Here's the debug output from
it
after trying to run "iptables -A C70 -j
forward_ok":

Jul 20 17:11:12 intratest2 kernel: t->private->number
= 1425
Jul 20 17:11:12 intratest2 kernel: translate_table: size
282700
Jul 20 17:11:12 intratest2 kernel: Jump rule 148 ->
241392
Jul 20 17:11:12 intratest2 kernel: Jump rule 241392 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 249836 ->
219892
Jul 20 17:11:12 intratest2 kernel: Jump rule 241620 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 241848 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 193672 ->
67868
Jul 20 17:11:12 intratest2 kernel: Jump rule 193968 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 112796 ->
33232
Jul 20 17:11:12 intratest2 kernel: Jump rule 112944 ->
33232
Jul 20 17:11:12 intratest2 kernel: Jump rule 113092 ->
33232
Jul 20 17:11:12 intratest2 kernel: Jump rule 113240 ->
33232
Jul 20 17:11:12 intratest2 kernel: Jump rule 113388 ->
33232
Jul 20 17:11:12 intratest2 kernel: Jump rule 113536 ->
33232
Jul 20 17:11:12 intratest2 kernel: Jump rule 113684 ->
33232
Jul 20 17:11:12 intratest2 kernel: Jump rule 113832 ->
33232
Jul 20 17:11:12 intratest2 kernel: Jump rule 241996 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 248980 ->
219892
Jul 20 17:11:12 intratest2 kernel: Jump rule 636 ->
237532
Jul 20 17:11:12 intratest2 kernel: Jump rule 237720 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 237948 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 238176 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 238436 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 238696 ->
250692
Jul 20 17:11:12 intratest2 kernel: Jump rule 250692 ->
219892
Jul 20 17:11:12 intratest2 kernel: Jump rule 239088 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 239236 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 239384 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 239532 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 239680 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 239828 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 239976 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 240124 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 240272 ->
134440
Jul 20 17:11:12 intratest2 kernel: Jump rule 240488 ->
137860
Jul 20 17:11:12 intratest2 kernel: Jump rule 240704 ->
147036
Jul 20 17:11:12 intratest2 kernel: Jump rule 240920 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 852 ->
235384
Jul 20 17:11:12 intratest2 kernel: Jump rule 235768 ->
252404
Jul 20 17:11:12 intratest2 kernel: Jump rule 253588 ->
237532
Jul 20 17:11:12 intratest2 kernel: Jump rule 235984 ->
242468
Jul 20 17:11:12 intratest2 kernel: Jump rule 242468 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 242696 ->
249836
Jul 20 17:11:12 intratest2 kernel: Jump rule 242924 ->
248656
Jul 20 17:11:12 intratest2 kernel: Jump rule 243072 ->
217916
Jul 20 17:11:12 intratest2 kernel: Jump rule 243300 ->
4712
Jul 20 17:11:12 intratest2 kernel: Jump rule 4712 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 4860 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 243448 ->
5332
Jul 20 17:11:12 intratest2 kernel: Jump rule 5332 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 5480 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 243596 ->
12152
Jul 20 17:11:12 intratest2 kernel: Jump rule 12152 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 12300 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 243744 ->
18972
Jul 20 17:11:12 intratest2 kernel: Jump rule 18972 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 19120 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 243892 ->
25792
Jul 20 17:11:12 intratest2 kernel: Jump rule 25792 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 25940 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 244040 ->
32612
Jul 20 17:11:12 intratest2 kernel: Jump rule 32612 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 32760 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 244188 ->
49692
Jul 20 17:11:12 intratest2 kernel: Jump rule 49692 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 49840 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 244336 ->
69672
Jul 20 17:11:12 intratest2 kernel: Jump rule 69672 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 69820 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 244484 ->
88388
Jul 20 17:11:12 intratest2 kernel: Jump rule 88388 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 88536 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 244632 ->
107104
Jul 20 17:11:12 intratest2 kernel: Jump rule 107104 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 107252 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 244780 ->
5952
Jul 20 17:11:12 intratest2 kernel: Jump rule 5952 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 6100 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 244928 ->
6572
Jul 20 17:11:12 intratest2 kernel: Jump rule 6572 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 6720 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 245076 ->
7192
Jul 20 17:11:12 intratest2 kernel: Jump rule 7192 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 7340 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 245224 ->
7812
Jul 20 17:11:12 intratest2 kernel: Jump rule 7812 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 7960 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 245372 ->
8432
Jul 20 17:11:12 intratest2 kernel: Jump rule 8432 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 8580 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 245520 ->
9052
Jul 20 17:11:12 intratest2 kernel: Jump rule 9052 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 9200 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 245668 ->
9672
Jul 20 17:11:12 intratest2 kernel: Jump rule 9672 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 9820 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 245816 ->
10292
Jul 20 17:11:12 intratest2 kernel: Jump rule 10292 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 10440 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 245964 ->
10912
Jul 20 17:11:12 intratest2 kernel: Jump rule 10912 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 11060 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 246112 ->
11532
Jul 20 17:11:12 intratest2 kernel: Jump rule 11532 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 11680 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 246260 ->
12772
Jul 20 17:11:12 intratest2 kernel: Jump rule 12772 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 12920 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 246408 ->
13392
Jul 20 17:11:12 intratest2 kernel: Jump rule 13392 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 13540 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 246556 ->
14012
Jul 20 17:11:12 intratest2 kernel: Jump rule 14012 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 14160 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 246704 ->
14632
Jul 20 17:11:12 intratest2 kernel: Jump rule 14632 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 14780 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 246852 ->
15252
Jul 20 17:11:12 intratest2 kernel: Jump rule 15252 ->
193132
Jul 20 17:11:12 intratest2 kernel: Jump rule 15400 ->
248980
Jul 20 17:11:12 intratest2 kernel: Jump rule 247000 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 247148 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 247296 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 247444 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 247592 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 247740 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 247888 ->
109528
Jul 20 17:11:12 intratest2 kernel: Jump rule 248036 ->
109528
Jul 20 17:11:13 intratest2 kernel: Jump rule 248184 ->
248980
Jul 20 17:11:13 intratest2 kernel: Jump rule 1000 ->
237532
Jul 20 17:11:13 intratest2 kernel: Jump rule 1216 ->
248980
Jul 20 17:11:13 intratest2 kernel: Finished chain 1
Jul 20 17:11:13 intratest2 kernel: Jump rule 1512 ->
224536
Jul 20 17:11:13 intratest2 kernel: Jump rule 224536 ->
249836
Jul 20 17:11:13 intratest2 kernel: Jump rule 249836 ->
219892
Jul 20 17:11:13 intratest2 kernel: Jump rule 224764 ->
249836
Jul 20 17:11:13 intratest2 kernel: Jump rule 224992 ->
194440
Jul 20 17:11:13 intratest2 kernel: Jump rule 194440 ->
70292
Jul 20 17:11:13 intratest2 kernel: Jump rule 71624 ->
232340
Jul 20 17:11:13 intratest2 kernel: Jump rule 232340 ->
232960
Jul 20 17:11:13 intratest2 kernel: Jump rule 232960 ->
215940
Jul 20 17:11:13 intratest2 kernel: Jump rule 233176 ->
215940
Jul 20 17:11:13 intratest2 kernel: mark_source_chains: bad
negative verdict 
(-2140522486)

How can the "bad negative verdict" code be
triggered?
How can it be fixed? 

I was unable to cook up a minimalistic test case,
so I've attached the complete firewall ruleset.

Any help is appreciated.

Thanks in advance,
Thomas

Thomas Jarosch wrote:
> Hello there,
>
> I've upgraded to kernel 2.6.21.6 / iptables 1.3.7 and
now a big firewall table 
> fails to load. The error message from the iptables
command is
> "iptables: Too many levels of symbolic
links", so I've enabled debugging in 
> net/ipv4/netfilter/ip_tables.c. Here's the debug output
from it
> after trying to run "iptables -A C70 -j
forward_ok":
> [...]
> Jul 20 17:11:13 intratest2 kernel: Jump rule 232340
-> 232960
> Jul 20 17:11:13 intratest2 kernel: Jump rule 232960
-> 215940
> Jul 20 17:11:13 intratest2 kernel: Jump rule 233176
-> 215940
> Jul 20 17:11:13 intratest2 kernel: mark_source_chains:
bad negative verdict 
> (-2140522486)
>
> How can the "bad negative verdict" code be
triggered?
> How can it be fixed? 
>   

I'm pretty sure its related to the mark_source_chains
optimization.
Try removing the " || visited" from the condition
just before the
"negative verdict" printk.

Hello Patrick,

On Friday, 20. July 2007, Patrick McHardy wrote:
> > Jul 20 17:11:13 intratest2 kernel:
mark_source_chains: bad negative
> > verdict (-2140522486)
> >
> > How can the "bad negative verdict" code
be triggered?
> > How can it be fixed? 
>
> I'm pretty sure its related to the mark_source_chains
optimization.
> Try removing the " || visited" from the
condition just before the
> "negative verdict" printk.

Thanks, that did the trick, the firewall plays nice again.
Let me know if I can aid debugging/fixing the code.

Cheers,
Thomas

Hello Patrick,

On Tuesday, 24. July 2007, you wrote:
> Yes, what you could do is use the original ruleset (not
the saved one)
> and find out which rule causes the error.

The "saved" one is the only one I got. I executed
the rules manually
and it failed doing something like this: iptables -A R5_FWD
xyz -j forward_ok.

"forward_ok" jumps to "forward_tolanok"
which contains two rules jumping 
to "clicount_in". "clicount_in" is used
for accounting and can be referred
by multiple places. IMHO this is the place where the
"visisted" code fails:

-A forward_ok -o eth0 -j forward_tolan_ok
-A forward_tolan_ok -i eth1 -m condition --condition
inet_eth -j clicount_in
-A forward_tolan_ok -i ppp0 -m condition --condition
inet_ppp -j clicount_in

The corresponding debug output:
Jul 20 17:11:13 intratest2 kernel: Jump rule 232960 ->
215940
Jul 20 17:11:13 intratest2 kernel: Jump rule 233176 ->
215940
Jul 20 17:11:13 intratest2 kernel: mark_source_chains: bad
negative verdict 
(-2140522486)

It works if I remove the second jump to clicount_in.
Does that make sense to you? 

Now comes the strange part: The ruleset gets generated by an
automatic system. 
I have the same style of ruleset running on 20 machines, it
only failed once. 
Somehow I get the feeling the order of the rules/chains is
important
to trigger the problem.

Thomas

Thomas Jarosch wrote:
> Hello Patrick,
> 
> On Tuesday, 24. July 2007, you wrote:
> 
>>Yes, what you could do is use the original ruleset
(not the saved one)
>>and find out which rule causes the error.
> 
> 
> The "saved" one is the only one I got. I
executed the rules manually
> and it failed doing something like this: iptables -A
R5_FWD xyz -j forward_ok.
> 
> "forward_ok" jumps to
"forward_tolanok" which contains two rules jumping

> to "clicount_in". "clicount_in" is
used for accounting and can be referred
> by multiple places. IMHO this is the place where the
"visisted" code fails:
> 
> -A forward_ok -o eth0 -j forward_tolan_ok
> -A forward_tolan_ok -i eth1 -m condition --condition
inet_eth -j clicount_in
> -A forward_tolan_ok -i ppp0 -m condition --condition
inet_ppp -j clicount_in
> 
> The corresponding debug output:
> Jul 20 17:11:13 intratest2 kernel: Jump rule 232960
-> 215940
> Jul 20 17:11:13 intratest2 kernel: Jump rule 233176
-> 215940
> Jul 20 17:11:13 intratest2 kernel: mark_source_chains:
bad negative verdict 
> (-2140522486)
> 
> It works if I remove the second jump to clicount_in.
> Does that make sense to you? 


Yes, that makes sense, I think what happens is that the jump
back goes
to the wrong position and things fall apart. I wasn't able
to make a
simple testcase though.

> Now comes the strange part: The ruleset gets generated
by an automatic system. 
> I have the same style of ruleset running on 20
machines, it only failed once. 
> Somehow I get the feeling the order of the rules/chains
is important
> to trigger the problem.


It probably is.