e1000g max mtu detection broken in recent builds (repost)

[Adam Cheal]

Initial post didn't make it across to the Mailman interface, so here we go again:

Seems I've hit a bug in more recent builds of OpenSolaris regarding setting large mtu values (i.e. jumbo frames support) on systems with multiple e1000g ports. This has happened across the board on all of our systems that use integrated e1000g ports, but not those that use nge (i.e. Thumpers), therefore I feel this problem is isolated to e1000g. Everything was working fine on b118, but as soon as we moved to later builds (I've tried b126->b130), the second NIC shows a maximum possible settable MTU of 1500, instead of the expected 9216. Booting back into b118 makes it work again. I've searched the release notes for all builds > 118 to see if there was any mention of MTU related fixes in e1000g or dladm but could find nothing.

I've attached a full "prtconf -v" output from b130.

First, build 118 dladm output:

# dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 mtu rw 9000 1500 1500-9216
e1000g1 mtu rw 9000 1500 1500-9216

Then once I reboot into b126 or greater:

# dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 mtu r- 1500 1500 1500-9216
e1000g1 mtu r- 1500 1500 1500

The differences I see are:

1. The "PERM" column changes from "rw" to "r-". Not sure if that has any bearing.
2. A side effect of the image-update to the new builds seems to be that it resets the per-adapter configured value of MTU back to the default of 1500. I can "dladm set-linkprop" and change the value back to "9000" for e1000g0, but not e1000g1.
3. The POSSIBLE mtu range for the second port (e1000g1) does not show the proper range. Obviously, "dladm set-linkprop -p mtu="9000" e1000g1" fails.

I have changed the /kernel/drv/e1000g.conf to "enable" jumbo frames via:

MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;

...but this conf setting seems to be ignored anyway. No hardware configuration has changed on these systems or the switches they connect to.

Thoughts/suggestions/pointers?

[Jason King]

Post by Adam Cheal
Seems I've hit a bug in more recent builds of OpenSolaris regarding setting large mtu values (i.e. jumbo frames support) on systems with multiple e1000g ports. This has happened across the board on all of our systems that use integrated e1000g ports, but not those that use nge (i.e. Thumpers), therefore I feel this problem is isolated to e1000g. Everything was working fine on b118, but as soon as we moved to later builds (I've tried b126->b130), the second NIC shows a maximum possible settable MTU of 1500, instead of the expected 9216. Booting back into b118 makes it work again. I've searched the release notes for all builds > 118 to see if there was any mention of MTU related fixes in e1000g or dladm but could find nothing.
I've attached a full "prtconf -v" output from b130.
# dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 mtu rw 9000 1500 1500-9216
e1000g1 mtu rw 9000 1500 1500-9216
# dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 mtu r- 1500 1500 1500-9216
e1000g1 mtu r- 1500 1500 1500
1. The "PERM" column changes from "rw" to "r-". Not sure if that has any bearing.
2. A side effect of the image-update to the new builds seems to be that it resets the per-adapter configured value of MTU back to the default of 1500. I can "dladm set-linkprop" and change the value back to "9000" for e1000g0, but not e1000g1.
3. The POSSIBLE mtu range for the second port (e1000g1) does not show the proper range. Obviously, "dladm set-linkprop -p mtu="9000" e1000g1" fails.
MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;
...but this conf setting seems to be ignored anyway. No hardware configuration has changed on these systems or the switches they connect to.
Thoughts/suggestions/pointers?
--
This message posted from opensolaris.org
_______________________________________________
networking-discuss mailing list

Don't know how much this helps, but two e1000g devices appear to be
slightly different variants of the chip (looking at the source, it
appears there are a _lot_ of different versions out there all with the
same name). e1000g0 has a device id of 0x10bd while e1000g1 has a
device id of 0x108b (if you look through the prtconf output, you can
see it in there)

Looking at the driver source, the _default_ MTU size is what's
controlled by the MaxFrameSize property. The actual maximum MTU size
is ultimately set based on the device id (any requests over the
maximum size result in the MTU being set to the max). In the case of
the e1000g1, it believes it should be the traditional 1500 bytes. I
don't have access to any of the specs for the chips from Intel, so I
don't know if the driver is making the correct assumption or not for
the 2nd device. I've copied driver-discuss in case the relevant
people (who would know) lurk there and not on networking-discuss.

[Min Miles Xu]

Post by Jason King

Post by Adam Cheal
Seems I've hit a bug in more recent builds of OpenSolaris regarding setting large mtu values (i.e. jumbo frames support) on systems with multiple e1000g ports. This has happened across the board on all of our systems that use integrated e1000g ports, but not those that use nge (i.e. Thumpers), therefore I feel this problem is isolated to e1000g. Everything was working fine on b118, but as soon as we moved to later builds (I've tried b126->b130), the second NIC shows a maximum possible settable MTU of 1500, instead of the expected 9216. Booting back into b118 makes it work again. I've searched the release notes for all builds > 118 to see if there was any mention of MTU related fixes in e1000g or dladm but could find nothing.
I've attached a full "prtconf -v" output from b130.
# dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 mtu rw 9000 1500 1500-9216
e1000g1 mtu rw 9000 1500 1500-9216
# dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 mtu r- 1500 1500 1500-9216
e1000g1 mtu r- 1500 1500 1500
1. The "PERM" column changes from "rw" to "r-". Not sure if that has any bearing.
2. A side effect of the image-update to the new builds seems to be that it resets the per-adapter configured value of MTU back to the default of 1500. I can "dladm set-linkprop" and change the value back to "9000" for e1000g0, but not e1000g1.
3. The POSSIBLE mtu range for the second port (e1000g1) does not show the proper range. Obviously, "dladm set-linkprop -p mtu="9000" e1000g1" fails.
MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;
...but this conf setting seems to be ignored anyway. No hardware configuration has changed on these systems or the switches they connect to.
Thoughts/suggestions/pointers?
--
This message posted from opensolaris.org
_______________________________________________
networking-discuss mailing list

Don't know how much this helps, but two e1000g devices appear to be
slightly different variants of the chip (looking at the source, it
appears there are a _lot_ of different versions out there all with the
same name). e1000g0 has a device id of 0x10bd while e1000g1 has a
device id of 0x108b (if you look through the prtconf output, you can
see it in there)
Looking at the driver source, the _default_ MTU size is what's
controlled by the MaxFrameSize property. The actual maximum MTU size
is ultimately set based on the device id (any requests over the
maximum size result in the MTU being set to the max). In the case of
the e1000g1, it believes it should be the traditional 1500 bytes. I
don't have access to any of the specs for the chips from Intel, so I
don't know if the driver is making the correct assumption or not for
the 2nd device. I've copied driver-discuss in case the relevant
people (who would know) lurk there and not on networking-discuss.
_______________________________________________
networking-discuss mailing list

Hi,

Actually it's a fix to a bug. What you observe is the expected behavior.
The changes were introduced in "6797885 need to add support for network
device (8086,10ea) in a new Intel system" in Build 125.
Previously no matter what mtu range the hardware can support, the dladm
always reports "1500-9216". In Build 125, I changed the mtu range
accordinly to chips' capabilities. The capability detail of each chip
(such as 82573 in your case) is provided by Intel.

The following piece of code controls this bit.

822 static void
823 e1000g_setup_max_mtu(struct e1000g *Adapter)
824 {
825 struct e1000_mac_info *mac = &Adapter->shared.mac;
826 struct e1000_phy_info *phy = &Adapter->shared.phy;
827
828 switch (mac->type) {
829 /* types that do not support jumbo frames */
830 case e1000_ich8lan:
831 case e1000_82573:
832 case e1000_82583:
833 Adapter->max_mtu = ETHERMTU;
834 break;
835 /* ich9 supports jumbo frames except on one phy type */
836 case e1000_ich9lan:
837 if (phy->type == e1000_phy_ife)
838 Adapter->max_mtu = ETHERMTU;
839 else
840 Adapter->max_mtu = MAXIMUM_MTU_9K;
841 break;
842 /* pch can do jumbo frames up to 4K */
843 case e1000_pchlan:
844 Adapter->max_mtu = MAXIMUM_MTU_4K;
845 break;
846 /* types with a special limit */
847 case e1000_82571:
848 case e1000_82572:
849 case e1000_82574:
850 case e1000_80003es2lan:
851 case e1000_ich10lan:
852 Adapter->max_mtu = MAXIMUM_MTU_9K;
853 break;
854 /* default limit is 16K */
855 default:
856 Adapter->max_mtu = FRAME_SIZE_UPTO_16K -
857 sizeof (struct ether_vlan_header) - ETHERFCSL -
858 E1000G_IPALIGNPRESERVEROOM;
859 break;
860 }
861 }


Regards,

Miles Xu

[Jason King]

Using the correct from address...

Post by Adam Cheal
Seems I've hit a bug in more recent builds of OpenSolaris regarding setting large mtu values (i.e. jumbo frames support) on systems with multiple e1000g ports. This has happened across the board on all of our systems that use integrated e1000g ports, but not those that use nge (i.e. Thumpers), therefore I feel this problem is isolated to e1000g. Everything was working fine on b118, but as soon as we moved to later builds (I've tried b126->b130), the second NIC shows a maximum possible settable MTU of 1500, instead of the expected 9216. Booting back into b118 makes it work again. I've searched the release notes for all builds > 118 to see if there was any mention of MTU related fixes in e1000g or dladm but could find nothing.
I've attached a full "prtconf -v" output from b130.
# dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 mtu rw 9000 1500 1500-9216
e1000g1 mtu rw 9000 1500 1500-9216
# dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 mtu r- 1500 1500 1500-9216
e1000g1 mtu r- 1500 1500 1500
1. The "PERM" column changes from "rw" to "r-". Not sure if that has any bearing.
2. A side effect of the image-update to the new builds seems to be that it resets the per-adapter configured value of MTU back to the default of 1500. I can "dladm set-linkprop" and change the value back to "9000" for e1000g0, but not e1000g1.
3. The POSSIBLE mtu range for the second port (e1000g1) does not show the proper range. Obviously, "dladm set-linkprop -p mtu="9000" e1000g1" fails.
MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;
...but this conf setting seems to be ignored anyway. No hardware configuration has changed on these systems or the switches they connect to.
Thoughts/suggestions/pointers?
--
This message posted from opensolaris.org
_______________________________________________
networking-discuss mailing list

Don't know how much this helps, but two e1000g devices appear to be
slightly different variants of the chip (looking at the source, it
appears there are a _lot_ of different versions out there all with the
same name).  e1000g0 has a device id of 0x10bd while e1000g1 has a
device id of 0x108b (if you look through the prtconf output, you can
see it in there)

Looking at the driver source, the _default_ MTU size is what's
controlled by the MaxFrameSize property.   The actual maximum MTU size
is ultimately set based on the device id (any requests over the
maximum size result in the MTU being set to the max).  In the case of
the e1000g1, it believes it should be the traditional 1500 bytes.  I
don't have access to any of the specs for the chips from Intel, so I
don't know if the driver is making the correct assumption or not for
the 2nd device.  I've copied driver-discuss in case the relevant
people (who would know) lurk there and not on networking-discuss.

[Adam Cheal]

Post by Min Miles Xu

Post by Adam Cheal
Seems I've hit a bug in more recent builds of
OpenSolaris regarding setting large mtu values (i.e.
jumbo frames support) on systems with multiple e1000g
ports.

Actually it's a fix to a bug. What you observe is the
expected behavior.
The changes were introduced in "6797885 need to add
support for network
device (8086,10ea) in a new Intel system" in Build
125.
Previously no matter what mtu range the hardware can
support, the dladm
always reports "1500-9216". In Build 125, I changed
the mtu range
accordinly to chips' capabilities. The capability
detail of each chip
(such as 82573 in your case) is provided by Intel.

Indeed sir, you are correct. I am still stunned that a commercial server would use two different chipsets for onboard NICs that are physically side-by-side, but I verified it with Intel and your explanation was dead on. That'll teach me to read specs and prtconf in more detail...

Glad to know it was a feature and not a bug. =)

- Adam

[Mark Bennett]

I'm having a similar problem with the igb driver in dev 130.
The chipset on the card, a Supermicro AOC-SG-I4 is based on the 82576 which supports jumbo frames, but they don't show as supported.

Is this likely to be a similar issue ?

Mark.

dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
igb2 mtu r- 1500 1500 1500
igb0 mtu r- 1500 1500 1500
igb4 mtu r- 1500 1500 1500
igb1 mtu r- 1500 1500 1500
igb3 mtu r- 1500 1500 1500
igb5 mtu r- 1500 1500 1500

[Peter Memishian]

I'm having a similar problem with the igb driver in dev 130. The
chipset on the card, a Supermicro AOC-SG-I4 is based on the 82576 which
supports jumbo frames, but they don't show as supported.
Is this likely to be a similar issue ?

Probably not; igb support for Brussels (needed to set most datalink
properties) integrated into snv_132 via 6886669. So, since you're using
snv_130, what you observed is expected.

dladm show-linkprop -p mtu
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
igb2 mtu r- 1500 1500 1500
igb0 mtu r- 1500 1500 1500
igb4 mtu r- 1500 1500 1500
igb1 mtu r- 1500 1500 1500
igb3 mtu r- 1500 1500 1500
igb5 mtu r- 1500 1500 1500

[matthew earley]

With 3 e1000g interfaces I am still experiencing this problem on 2 out of the 3 interfaces. My add-on PCI-E Intel card displays the correct MTU range of 1500-16298.

The 2 built-in Intel interfaces show only MTU of 1500.

my /kernel/drv/e1000g.conf file is set to
MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;

How do I get the e1000g driver to allow max MTU on all 3 e1000g interfaces?

Thank you,
Matthew