Loopback - increasing MTU size

[Chih-Wei Yu]

Hi everyone,

I'm using TCP connection on the loopback but somehow i am reaching a bottle neck in terms of throughput. I tried to increase MTU size on the loopback:

$> ifconfig lo0 mtu 16384

but i get error indicating invalid arguments.

Its on T2000, solaris 10, patch level 120011-14.

I see that there is a e1000g.conf in /kernel/drv/ directory but by updating this file will it help with the setting of increasing MTU on the loopback?
Is there other ways to increase throughput through the loopback?

Thank you in advance


This message posted from opensolaris.org

[James Carlson]

Post by Chih-Wei Yu
$> ifconfig lo0 mtu 16384
but i get error indicating invalid arguments.

Correct; the system doesn't allow you to do that. It won't let you
set the MTU greater than ill_max_frag, which is hard-coded to
IP_LOOPBACK_MTU (8192) for lo0.

How have you determined that the default MTU setting on the lo0
interface is in fact the problem in this instance?

Post by Chih-Wei Yu
Its on T2000, solaris 10, patch level 120011-14.

Solaris 10 isn't OpenSolaris. You should contact Sun's support group
with questions about Solaris 10.

Post by Chih-Wei Yu
I see that there is a e1000g.conf in /kernel/drv/ directory but by updating this file will it help with the setting of increasing MTU on the loopback?
Is there other ways to increase throughput through the loopback?

'e1000g' is an Ethernet driver. It has nothing to do with loopback.

[Garrett D'Amore]

Post by James Carlson

Post by Chih-Wei Yu
I see that there is a e1000g.conf in /kernel/drv/ directory but by updating this file will it help with the setting of increasing MTU on the loopback?
Is there other ways to increase throughput through the loopback?

'e1000g' is an Ethernet driver. It has nothing to do with loopback.

This raises an interesting question, though.

Will IP use the link MTU for packets that are ultimately to itself. I.e.
packets that are going to be looped back in IP, but are not for the
127.0.0.0 subnet. It seems that in such cases it could use a potentially
much larger MTU for packets that a really just going to be redirected
back to self, even when the IP address is associated with an interface
that has a much smaller MTU.

-- Garrett

[James Carlson]

Post by Garrett D'Amore
Will IP use the link MTU for packets that are ultimately to itself. I.e.
packets that are going to be looped back in IP, but are not for the
127.0.0.0 subnet.

No. It uses lo0's MTU for all local addresses.

Post by Garrett D'Amore
It seems that in such cases it could use a potentially
much larger MTU for packets that a really just going to be redirected
back to self, even when the IP address is associated with an interface
that has a much smaller MTU.

Except that it doesn't do this ...

[James Carlson]

Post by Chih-Wei Yu
but i get error indicating invalid arguments.

$$$Correct; the system doesn't allow you to do that. It won't let you
set the MTU $$$greater than ill_max_frag, which is hard-coded to
IP_LOOPBACK_MTU (8192) for lo0.

Post by Chih-Wei Yu

So there's no way to increase it higher.

Correct.

$$$How have you determined that the default MTU setting on the lo0
interface is in $$$fact the problem in this instance?

Post by Chih-Wei Yu

Well, it's a hunch. I increased receive and buffer sizes on both

ends and try to push the rate higher. After a couple of minutes, the TCP
connection gets blocked and one of the ends get block in sending state.
Doing a netstat on the loopback, I can see that there are packets
initially but then once its gets blocked, there are no packets between
both ends. Even when I increased the rate, I see roughly the same size
of bytes being relayed between both ends (netstat -I lo0 1, roughly
8800~9000 packets every second).

It sounds to me like you've run into a bug of some sort, not a tuning
issue. The system shouldn't need to be tuned for local traffic
... or, really, any traffic.

I suggest that you work with Sun's support group to find and fix the
root cause of the problem you're seeing.

Post by Chih-Wei Yu

On Linux, I was able to set the MTU value on the loopback much

higher but then again, I didn't run the test through the Linux machine.

I don't think that modifying the loopback MTU is the right approach
here.

It's important to understand that when you're using loopback traffic,
the system connects the endpoints directly together. The "MTU" has
little to do with how it works and shouldn't be harming performance.

[George Shepherd]

Hmm looks like you might be running into the tcp fusion flow control
limitations mentioned in infodoc 203612..

http://sunsolve.sun.com/search/document.do?assetkey=1-61-203612-1&searchclause=
203612

See also the following bugids, 6633302 and 6563902.
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6633302-1
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6563902-1

-George


*>Date: Tue, 18 Mar 2008 06:28:27 -0400
*>From: James Carlson <james.d.carlson-xsfywfwIY+***@public.gmane.org>
*>Subject: Re: [networking-discuss] Loopback - increasing MTU size
*>To: "Chih - Wei Yu [ MTN - Innovation Centre ]" <yu_c-***@public.gmane.org>
*>Cc: networking-discuss-***@public.gmane.org, gdamore-***@public.gmane.org
*>MIME-version: 1.0
*>Content-transfer-encoding: 7BIT
*>X-BeenThere: networking-discuss-***@public.gmane.org
*>Delivered-to: networking-discuss-***@public.gmane.org
*>X-PMX-Version: 5.4.1.325704
*>X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on
oss-mail1.opensolaris.org
*>X-Original-To: networking-discuss-***@public.gmane.org
*>X-Antispam: No, score=-2.6/5.0, scanned in 0.078sec at (localhost
[127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/
*>X-Mailman-Version: 2.1.9
*>List-Post: <mailto:networking-discuss-***@public.gmane.org>
*>List-Subscribe:
<http://mail.opensolaris.org/mailman/listinfo/networking-discuss>,
<mailto:networking-discuss-request-***@public.gmane.org?subject=subscribe>
*>List-Unsubscribe:
<http://mail.opensolaris.org/mailman/listinfo/networking-discuss>,
<mailto:networking-discuss-request-***@public.gmane.org?subject=unsubscribe>
*>List-Archive: <http://mail.opensolaris.org/pipermail/networking-discuss>
*>List-Help: <mailto:networking-discuss-request-***@public.gmane.org?subject=help>
*>List-Id: Networking General Discussion <networking-discuss.opensolaris.org>
*>X-Spam-Status: No, score=-1.0 required=5.0 tests=AWL,RCVD_IN_DNSWL_LOW
autolearn=unavailable version=3.2.3
*>X-Spam-Level:
*>
*>Chih - Wei Yu [ MTN - Innovation Centre ] writes:
*>> > but i get error indicating invalid arguments.
*>>
*>> $$$Correct; the system doesn't allow you to do that. It won't let you
*>> set the MTU $$$greater than ill_max_frag, which is hard-coded to
*>> IP_LOOPBACK_MTU (8192) for lo0.
*>>
*>> >>>>>>>>So there's no way to increase it higher.
*>
*>Correct.
*>
*>> $$$How have you determined that the default MTU setting on the lo0
*>> interface is in $$$fact the problem in this instance?
*>>
*>> >>>>Well, it's a hunch. I increased receive and buffer sizes on both
*>> ends and try to push the rate higher. After a couple of minutes, the TCP
*>> connection gets blocked and one of the ends get block in sending state.
*>> Doing a netstat on the loopback, I can see that there are packets
*>> initially but then once its gets blocked, there are no packets between
*>> both ends. Even when I increased the rate, I see roughly the same size
*>> of bytes being relayed between both ends (netstat -I lo0 1, roughly
*>> 8800~9000 packets every second).
*>
*>It sounds to me like you've run into a bug of some sort, not a tuning
*>issue. The system shouldn't need to be tuned for local traffic
*>... or, really, any traffic.
*>
*>I suggest that you work with Sun's support group to find and fix the
*>root cause of the problem you're seeing.
*>
*>> >>>>>On Linux, I was able to set the MTU value on the loopback much
*>> higher but then again, I didn't run the test through the Linux machine.
*>
*>I don't think that modifying the loopback MTU is the right approach
*>here.
*>
*>It's important to understand that when you're using loopback traffic,
*>the system connects the endpoints directly together. The "MTU" has
*>little to do with how it works and shouldn't be harming performance.
*>
*>--
*>James Carlson, Solaris Networking <james.d.carlson-xsfywfwIY+***@public.gmane.org>
*>Sun Microsystems / 35 Network Drive 71.232W Vox [phone]
*>MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax [phone]
*>_______________________________________________
*>networking-discuss mailing list
*>networking-discuss-***@public.gmane.org


George Shepherd
http://clem.uk/~georges/
==============================================================================
Solaris Revenue Product Engineering: | SUN Microsystems
Core team -Internet | Guillemont Park
Email: George.Shepherd-UdXhSnd/***@public.gmane.org | Camberley GU17 9QG
Disclaimer: Less is more, more or less | United Kingdom
==============================================================================