Thoughts on increasing MTUs on the internet

Dear NANOGers,

It irks me that today, the effective MTU of the internet is
1500  
bytes, while more and more equipment can handle bigger
packets.

What do you guys think about a mechanism that allows hosts
and  
routers on a subnet to automatically discover the MTU they
can use  
towards other systems on the same subnet, so that:

1. It's no longer necessary to limit the subnet MTU to that
of the  
least capable system

2. It's no longer necessary to manage 1500 byte+ MTUs
manually

Any additional issues that such a mechanism would have to
address?

On 12-apr-2007, at 12:02, Pierfrancesco Caci wrote:

> wouldn't that work only if the switch in the middle of
your neat
> office lan is a real switch (i.e. not flooding oversize
packets to
> hosts that can't handle them, possibly crashing their
NIC drivers) and
> it's itself capable of larger MTUs?

Well, yes, being compatible with stuff that doesn't support
larger  
packets pretty much goes without saying. I don't think there
is any  
need to worry about crashing drivers, packets that are
longer than  
they should are a common error condition that drivers are
supposed to  
handle without incident. (They often keep a
"giant" count.)

A more common problem would be two hosts that support
jumboframes  
with a switch in the middle that doesn't. So it's necessary
to test  
for this and avoid excessive numbers or large packets when
something  
in the middle doesn't support them.

On (2007-04-12 11:20 +0200), Iljitsch van Beijnum wrote:
 
> What do you guys think about a mechanism that allows
hosts and  
> routers on a subnet to automatically discover the MTU
they can use  
> towards other systems on the same subnet, so that:
> 1. It's no longer necessary to limit the subnet MTU to
that of the  
> least capable system
> 
> 2. It's no longer necessary to manage 1500 byte+ MTUs
manually

To me this sounds adding complexity for rather small
pay-off. And
then we'd have to ask IXP people, would the enable this
feature
if it was available? If so, why don't they offer high MTU
VLAN
today?
And in the end, pay-off of larger MTU is quite small,
perhaps
some interrupts are saved but not sure how relevant that is
in poll() based NIC drivers. Of course bigger pay-off
would be that users could use tunneling and still offer
1500
to LAN.

IXP peeps, why are you not offering high MTU VLAN option?
>From my point of view, this is biggest reason why we
today
generally don't have higher end-to-end MTU.
I know that some IXPs do, eg. NetNOD but generally it's
not offered even though many users would opt to use it.

Thanks,
-- 
  ++ytti

On Thu, Apr 12, 2007 at 01:03:45PM +0200, Iljitsch van
Beijnum wrote:
> 
> On 12-apr-2007, at 12:02, Pierfrancesco Caci wrote:
> 
> >wouldn't that work only if the switch in the middle
of your neat
> >office lan is a real switch (i.e. not flooding
oversize packets to
> >hosts that can't handle them, possibly crashing
their NIC drivers) and
> >it's itself capable of larger MTUs?
> 
> Well, yes, being compatible with stuff that doesn't
support larger  
> packets pretty much goes without saying. I don't think
there is any  
> need to worry about crashing drivers, packets that are
longer than  
> they should are a common error condition that drivers
are supposed to  
> handle without incident. (They often keep a
"giant" count.)
> 
> A more common problem would be two hosts that support
jumboframes  
> with a switch in the middle that doesn't. So it's
necessary to test  
> for this and avoid excessive numbers or large packets
when something  
> in the middle doesn't support them.

the internet is broken.. too many firewalls dropping icmp,
too many hard coded systems that work for 'default' but dont
actually allow for alternative parameters that should work
according to the RFCs

if you can fix all that then it might work

alternatively if you can redesign path mtu discovery that
might work too..

Martin Levy suggested this too me only two weeks ago, he had
an idea of sending two packets initially - one 'default' and
one at the higher mtu .. if the higher one gets dropped
somewhere you can quickly spot it and revert to 'default'
behaviour.

I think his explanation was more complicated but it was an
interesting idea

Steve

On Thu, 12 Apr 2007, Saku Ytti wrote:

> IXP peeps, why are you not offering high MTU VLAN
option?

Netnod in Sweden offer MTU 4470 option.

Otoh it's not so easy operationally since for instance
Juniper and Cisco 
calculates MTU differently.

But I don't really see it beneficial to try to up the
endsystem MTU to 
over standard ethernet MTU, if you think it's operationally
troublesome 
with PMTUD now, imagine when everybody is running different
MTU.

Biggest benefit would be if the transport network people run
PPPoE and 
other tunneled traffic over, would allow for whatever MTU
needed to carry 
unfragmented 1500 byte tunneled packets, so we could assure
that all hosts 
on the internet actually have 1500 IP MTU transparently.

-- 
Mikael Abrahamsson    email: swmikeswm.pp.se

* swmikeswm.pp.se (Mikael Abrahamsson) [Thu 12 Apr 2007,
14:07 CEST]:
>On Thu, 12 Apr 2007, Saku Ytti wrote:
>>IXP peeps, why are you not offering high MTU VLAN
option?
>Biggest benefit would be if the transport network people
run PPPoE and 
>other tunneled traffic over, would allow for whatever
MTU needed to 
>carry unfragmented 1500 byte tunneled packets, so we
could assure that 
>all hosts on the internet actually have 1500 IP MTU
transparently.

How much traffic from DSLAM to service provider is currently
being 
exchanged across IXPs?

(My money's on "as close to 0 to not really
matter")


	-- Niels.

On Thu, 12 Apr 2007 11:20:18 +0200
Iljitsch van Beijnum <iljitschmuada.com> wrote:

> 
> Dear NANOGers,
> 
> It irks me that today, the effective MTU of the
internet is 1500
> bytes, while more and more equipment can handle bigger
packets.
> 
> What do you guys think about a mechanism that allows
hosts and
> routers on a subnet to automatically discover the MTU
they can use
> towards other systems on the same subnet, so that:
> 
> 1. It's no longer necessary to limit the subnet MTU to
that of the
> least capable system
> 
> 2. It's no longer necessary to manage 1500 byte+ MTUs
manually
> 
> Any additional issues that such a mechanism would have
to address?
> 

Last I heard, the IEEE won't go along, and they're the ones
who
standardize 802.3.

A few years ago, the IETF was considering various jumbogram
options.
As best I recall, that was the official response from the
relevant
IEEE folks: "no". They're concerned with backward
compatibility.  

Perhaps that has changed (and I certainly) don't remember
who sent that
note.  


		--Steve Bellovin, http://www.cs.columbi
a.edu/~smb

* Steven M. Bellovin:

> A few years ago, the IETF was considering various
jumbogram options.
> As best I recall, that was the official response from
the relevant
> IEEE folks: "no". They're concerned with
backward compatibility.  

Gigabit ethernet has already broken backwards compatibility
and is
essentially point-to-point, so the old compatibility
concerns no
longer apply.  Jumbo frame opt-in could even be controlled
with a
protocol above layer 2.

On 12-apr-2007, at 16:04, Gian Constantine wrote:

> I agree. The throughput gains are small. You're talking
about a  
> difference between a 4% header overhead versus a 1%
header overhead  
> (for TCP).

6% including ethernet overhead and assuming the very common
TCP  
timestamp option.

> One could argue a decreased pps impact on intermediate
systems, but  
> when factoring in the existing packet size distribution
on the  
> Internet and the perceived adjustment seen by a
migration to 4470  
> MTU support, the gains remain small.

Average packet size on the internet has been fairly constant
at  
around 500 bytes the past 10 years or so from my vantage
point. You  
only need to make 7% of all packets 9000 bytes and you
double that.  
This means that you can have twice the amount of data
transferred for  
the same amount of per-packet work. If you're at 100% of
your CPU or  
TCAM capacity today, that is a huge win. On the other hand,
if you  
need to buy equipment that can do line rate at 64 bytes per
packet,  
it doesn't matter much.

There are other benefits too, though. For instance, TCP can
go much  
faster with bigger packets. Additional tunnel/VPN overhead
isn't as bad.

> Development costs and the OpEx costs of implementation
and support  
> will, likely, always outweigh the gains.

Gains will go up as networks get faster and faster,
implementation  
should approach zero over time and support shouldn't be an
issue if  
it works fully automatically.

Others mentioned ICMP filtering and PMTUD problems.
Filtering  
shouldn't be an issue for a mechanism that is local to a
subnet, and  
if it is, there's still no problem if the mechanism takes
the  
opposite approach of PMTUD. With PMTUD, the assumption is
that large  
works, and extra messages result in a smaller packet size.
By  
exchanging large messages that indicate the capability to
exchange  
large messages, form and function align, and if an
indication that  
large messages are possible isn't received, it's not used
and there  
are no problems.