dlstat for data link statistics

[Shrikrishna Khare]

(Bcc'ed to networking discuss).

Hi All,

Have enclosed man page draft for dlstat(1M) herewith.

This is part of the effort to gain better visibility into network traffic in light of crossbow features like virtual NICs, interrupt vs. polling modes etc. This in turn would greatly assist network performance analysis. It is also aimed at segregating link/flow configuration from dynamic network statistics.

In particular,

- dladm/flowadm would retain configuration specific commands for links/flows while dlstat/flowstat would provide interfaces for displaying dynamic network statistics.

- show-usage would be removed from dladm/flowadm and would become part of dlstat/flowstat. It would be renamed show-history.

- Rich set of counters would be available for performance analysis e.g. poll vs. interrupt count, chain sizes, distribution of load across lanes etc.

- In the man page, 'dlstat show' synopsis is deliberately spread across multiple lines. First line, dlstat show [-r | -t]... provides an easy template for printing most commonly queried statistics. Comprehensive and flexible approaches follow.

Note: Numbers in examples section of enclosed man page are used to illustrate output formatting only.


~ Shri

[Shrikrishna Khare]

(Some of this discussion took over networking-discuss, Bcc'ing it thus).

Hi All,

Updated dlstat man page, dladm man page and diff of dladm man page is
enclosed.

To address Jim's concern regarding how dladm would be transitioned:
dladm man page is changed to declare that show-usage,
show-{link,aggr,vnic} -s is obsolete. Those commands would continue to
work for 'a while' even after dlstat is integrated (would post exact
timeline later).

First, thanks for taking this work on, it's very useful.
I haven't gone through this in detail, but I have a few high-level
* The output engine here should use the ofmt API that Sowmini is
currently wrapping up. Similarly, the output should be the same form
as currently shown by dladm, flowadm and ipmpstat. Along those
# dlstat show
LINK IPKTS IBYTES OPKTS OBYTES %UTIL
-----------------------------------------
e1000g0 21.2K 2.1M 0.5K 44.2M 00.0
ixgbe0 24.5M 13.6G 3.5M 0.1G 00.0
vnic1 00.0 00.0 00.0 00.0 00.0
# dlstat show
LINK IPKTS IBYTES OPKTS OBYTES UTIL
e1000g0 21.2K 2.1M 0.5K 44.2M 0.0%
ixgbe0 24.5M 13.6G 3.5M 0.1G 0.0%
vnic1 0.0M 0.0M 0.0M 0.0M 0.0%
Note in particular that every value should have a unit and the units
or interpretation of the data should live with the data itself, not
with the field header. Again, the ofmt API will generally encourage
the right thing here (though there's addditional work that really
should be done to simplify API consumers, such as 6800985).

I have changed manpage examples to follow output formatting as suggested
above.

* If we want to support an additional output mode like "full-screen"
mode, that should also be handled by the ofmt facility rather than
special-cased into dlstat. Likewise for "kstat mode" (but see
below).

Will check how ofmt api could be extended to support full-screen and kstat
mode.

* It's not clear to me why we need both "kstat mode" and "full-screen"
mode. If we do need to keep both, I'd be inclined to use "-k" for
kstat mode rather than "-v".

Am fine with -k as well (Changed in the man page).

Full screen mode will update information live by overprinting (like prstat).
Number of fields would thus have 80 character constraint.

On the other hand, kstat mode would be one-stop shop for all dynamic
data link stats
that are available.

* The "+output" thing is clever but it'd be odd for this command to
support it and dladm/flowadm/ipmpstat to not support it. It should
also be not allowed in parsable output mode since the caller should
never rely on the default output mode (since it can change).

I added a note in parseable mode description that it is not compatible
with -o - or -o +.

If -o -/+ appears useful, how about implementing that in dlstat for now.
We could think
of adding similar support to other commands in future?


~ Shri

--
meem

[sowmini.varadhan-xsfywfwIY+]

Post by Shrikrishna Khare

* It's not clear to me why we need both "kstat mode" and "full-screen"
mode. If we do need to keep both, I'd be inclined to use "-k" for
kstat mode rather than "-v".

Am fine with -k as well (Changed in the man page).
Full screen mode will update information live by overprinting (like prstat).
Number of fields would thus have 80 character constraint.
On the other hand, kstat mode would be one-stop shop for all dynamic
data link stat that are available.

I am not seeing the purpose of the -k option at all. Some clarification
would be useful to me.

Post by Shrikrishna Khare

* The "+output" thing is clever but it'd be odd for this command to
support it and dladm/flowadm/ipmpstat to not support it. It should
also be not allowed in parsable output mode since the caller should
never rely on the default output mode (since it can change).

I added a note in parseable mode description that it is not compatible

I presume the (future) command itself will also produce an error.

Post by Shrikrishna Khare
with -o - or -o +.
If -o -/+ appears useful, how about implementing that in dlstat for
now. We could think of adding similar support to other commands in
future?

With the ofmt API, I think it will end up being more work to support this
only in dlstat. (Further to that, if you find the ofmt API does not meet
dlstat's needs, then I'm sure Sowmini will be eager to enhance it :-)

While it may be clever, I'm not sure the -/+ fields is a good idea for just
networking commands, unless we do it across the board for all Solaris
commands like ps.

One thing that's not clear to me from reading the man page:
how is this supposed to work? Lets say that I have dlstat running in one
window with some set of -o flags, how do I signal that process to update
its print-handle state to add/remove fields in the output?

--Sowmini

[James Carlson]

Post by sowmini.varadhan-xsfywfwIY+
how is this supposed to work? Lets say that I have dlstat running in one
window with some set of -o flags, how do I signal that process to update
its print-handle state to add/remove fields in the output?

I don't think that's how it's supposed to work.

The +/- thing is just supposed to add or remove columns from the
default. For example, if the default output of "show-widget" has two
columns labeled "FOO" and "BAR," then "show-widget -o +baz" will add a
"BAZ" column (presumably to the end of the line), while "show-widget
-o -foo" will cause it to display only the "BAR" column.

It doesn't affect any other process that's running. (Unlike the
"reset" subcommand ...)

[Moore, Joe]

Post by Shrikrishna Khare

Post by sowmini.varadhan-xsfywfwIY+
how is this supposed to work? Lets say that I have dlstat running in

one

Post by sowmini.varadhan-xsfywfwIY+
window with some set of -o flags, how do I signal that process to

update

Post by sowmini.varadhan-xsfywfwIY+
its print-handle state to add/remove fields in the output?

I don't think that's how it's supposed to work.
The +/- thing is just supposed to add or remove columns from the
default. For example, if the default output of "show-widget" has two
columns labeled "FOO" and "BAR," then "show-widget -o +baz" will add a
"BAZ" column (presumably to the end of the line), while "show-widget
-o -foo" will cause it to display only the "BAR" column.

If I might make a suggestion of an alternative to this:

Put in the man page for the command the -o string that can be used to reproduce the default output (and when options change that default output, give an equivalent -o string). That way anyone who wants to make a minor change to the output can do so based on the man page.

For example:
For /bin/ps the default output is (almost) equivalent to -o pid,tty,time,commm
/bin/ps -Z prints the near equivalent of -o zonename,pid,tty,time,comm

So in show-widget(1M) you'd have:
-o format
The -o option allows the output format to be specified under
user control. The default output is equivalent to the format
string "foo,bar".
-l
The -l option increases the detail given in the output. It is
equivalent to specifying the format string
"instanceid,name,foo,bar,baz"

--Joe

[Peter Memishian]

Post by Moore, Joe
Put in the man page for the command the -o string that can be used to
reproduce the default output (and when options change that default
output, give an equivalent -o string). That way anyone who wants to
make a minor change to the output can do so based on the man page.

... or we could just introduce a new special `default' field (similar to
the existing `all' field) that expands the current list of default fields.

--
meem

[sowmini.varadhan-xsfywfwIY+]

Post by Peter Memishian

Post by Moore, Joe
Put in the man page for the command the -o string that can be used to
reproduce the default output (and when options change that default
output, give an equivalent -o string). That way anyone who wants to
make a minor change to the output can do so based on the man page.

... or we could just introduce a new special `default' field (similar to
the existing `all' field) that expands the current list of default fields.

So far, "-o all" is usually equal to the default output (except for some
corner cases like show-secobj) and the man page (afaik) does list the
fields emitted for default output.

--Sowmini

[Peter Memishian]

Post by sowmini.varadhan-xsfywfwIY+

Post by Peter Memishian

Post by Moore, Joe
Put in the man page for the command the -o string that can be used to
reproduce the default output (and when options change that default
output, give an equivalent -o string). That way anyone who wants to
make a minor change to the output can do so based on the man page.

... or we could just introduce a new special `default' field (similar to
the existing `all' field) that expands the current list of default fields.

So far, "-o all" is usually equal to the default output (except for some
corner cases like show-secobj) and the man page (afaik) does list the
fields emitted for default output.

Except that it's cumbersome to type all that and over time "-o all" will
be less and less like the default output (since we will invariably add
more fields).

[Moore, Joe]

Post by Peter Memishian

Post by Moore, Joe
Put in the man page for the command the -o string that can be used

to

Post by Moore, Joe
reproduce the default output (and when options change that default
output, give an equivalent -o string). That way anyone who wants to
make a minor change to the output can do so based on the man page.

... or we could just introduce a new special `default' field (similar to
the existing `all' field) that expands the current list of default fields.

"-o default" is useful if you don't want to replace any of the inner columns, only to add columns to the front or back of the existing output.

Why would a new magic output format token be preferable to simply documenting the current output format in the man page? Whether you're talking about the -o {+,-}Field or the -o default or the -o all option

--Joe

[Peter Memishian]

Post by Moore, Joe
"-o default" is useful if you don't want to replace any of the inner
columns, only to add columns to the front or back of the existing
output.
Why would a new magic output format token be preferable to simply
documenting the current output format in the man page?

We generally do document it, but it's tedious to use it that way.

[Shrikrishna Khare]

Post by sowmini.varadhan-xsfywfwIY+

Post by Shrikrishna Khare

* It's not clear to me why we need both "kstat mode" and "full-screen"
mode. If we do need to keep both, I'd be inclined to use "-k" for
kstat mode rather than "-v".

Am fine with -k as well (Changed in the man page).
Full screen mode will update information live by overprinting (like prstat).
Number of fields would thus have 80 character constraint.
On the other hand, kstat mode would be one-stop shop for all dynamic
data link stat that are available.

I am not seeing the purpose of the -k option at all. Some clarification
would be useful to me.

Think I did not explain it well.

There are too many fields to fit in 80 column width. E.g. there are 18
rx specific
fields listed in this dlstat man page itself. Number of fields are
further likely
to increase in future.

Thus, full screen mode would not be of much use for somebody interested
in all
(or large number of fields). Hence the kstat style option.

Post by sowmini.varadhan-xsfywfwIY+

Post by Shrikrishna Khare

* The "+output" thing is clever but it'd be odd for this command to
support it and dladm/flowadm/ipmpstat to not support it. It should
also be not allowed in parsable output mode since the caller should
never rely on the default output mode (since it can change).

I added a note in parseable mode description that it is not compatible

I presume the (future) command itself will also produce an error.

Post by Shrikrishna Khare
with -o - or -o +.
If -o -/+ appears useful, how about implementing that in dlstat for
now. We could think of adding similar support to other commands in
future?

With the ofmt API, I think it will end up being more work to support this
only in dlstat. (Further to that, if you find the ofmt API does not meet
dlstat's needs, then I'm sure Sowmini will be eager to enhance it :-)

While it may be clever, I'm not sure the -/+ fields is a good idea for just
networking commands, unless we do it across the board for all Solaris
commands like ps.
how is this supposed to work? Lets say that I have dlstat running in one
window with some set of -o flags, how do I signal that process to update
its print-handle state to add/remove fields in the output?
--Sowmini

[Peter Memishian]

Post by Shrikrishna Khare
There are too many fields to fit in 80 column width. E.g. there are 18
rx specific fields listed in this dlstat man page itself. Number of
fields are further likely to increase in future.
Thus, full screen mode would not be of much use for somebody interested
in all (or large number of fields). Hence the kstat style option.

OK, I think the confusion comes from using kstat as the example; this
seems to just be a row-per-field (as opposed to row-per-object) human-only
output mode. I think this has merit though I'd hope for that for most
commands we can generally encourage row-per-object as it's more compact
and provides a more obvious organization of the data. That said, the ofmt
API that Sowmini recent integrated should be able to be extended to
support this with minimal effort.

[James Carlson]

Post by Peter Memishian

Post by Shrikrishna Khare
There are too many fields to fit in 80 column width. E.g. there are 18
rx specific fields listed in this dlstat man page itself. Number of
fields are further likely to increase in future.
Thus, full screen mode would not be of much use for somebody interested
in all (or large number of fields). Hence the kstat style option.

OK, I think the confusion comes from using kstat as the example; this
seems to just be a row-per-field (as opposed to row-per-object) human-only
output mode. I think this has merit though I'd hope for that for most
commands we can generally encourage row-per-object as it's more compact
and provides a more obvious organization of the data. That said, the ofmt
API that Sowmini recent integrated should be able to be extended to
support this with minimal effort.

That answers most of my questions about this mode as well, I think.

It's not clear whether it really needs to be strictly kstat-
compatible (which is how I read it), or even row-per-field to be
useful in this way. Having a human-readable page-oriented output
format that allows for the display of large numbers of values would
probably be a good thing. The two-column style of "netstat -s" might
be viable.

[Peter Memishian]

Post by James Carlson
That answers most of my questions about this mode as well, I think.
It's not clear whether it really needs to be strictly kstat-
compatible (which is how I read it), or even row-per-field to be
useful in this way. Having a human-readable page-oriented output
format that allows for the display of large numbers of values would
probably be a good thing. The two-column style of "netstat -s" might
be viable.

Agreed. Ideally, ofmt would automatically determine how best to format
based on the available screen width (which it knows).

[sowmini.varadhan-xsfywfwIY+]

Post by Peter Memishian
Agreed. Ideally, ofmt would automatically determine how best to format
based on the available screen width (which it knows).

but this leads to a certain amount of unpredictability in the output.
As long as that constraint is acceptable ofmt has the information
to do this today.

--Sowmini

[Peter Memishian]

Post by sowmini.varadhan-xsfywfwIY+

Post by Peter Memishian
Agreed. Ideally, ofmt would automatically determine how best to format
based on the available screen width (which it knows).

but this leads to a certain amount of unpredictability in the output.
As long as that constraint is acceptable ofmt has the information
to do this today.

The output is only for human use so I think it should be OK.

[Shrikrishna Khare]

Bcc'ed to networking discuss.


Please find flowstat man page, modified flowadm and flowadm man page
diff at:

http://cr.opensolaris.org/~shri.k/

Flowstat manpage is heavily borrowed from dlstat manpage as flows
would have dedicated hardware rings in future. Till that support is in
place, fields like poll count could continue to print 0.

Earlier, dlstat proposal used -l for s/w lanes and -L for h/w lanes.
However, in order to be consistent with flowadm, flowstat should use -l
for links. I have thus written flowstat synopsis to use -s for s/w lanes
and -h for h/w lanes. For consistency, I have changed dlstat man page to
use -s and -h as well.

Moreover, to address concerns regarding resetting of stats:

How about continuing to support dlstat/flowstat reset but provide a
/clock_t lbolt/ that would denote 'time since last reset'. Depending on
one's requirement, one can then ascertain credibility of stats by
looking at this value.


~ Shri

Post by Shrikrishna Khare
(Bcc'ed to networking discuss).
Hi All,
Have enclosed man page draft for dlstat(1M) herewith.
This is part of the effort to gain better visibility into network traffic in light of crossbow features like virtual NICs, interrupt vs. polling modes etc. This in turn would greatly assist network performance analysis. It is also aimed at segregating link/flow configuration from dynamic network statistics.
In particular,
- dladm/flowadm would retain configuration specific commands for links/flows while dlstat/flowstat would provide interfaces for displaying dynamic network statistics.
- show-usage would be removed from dladm/flowadm and would become part of dlstat/flowstat. It would be renamed show-history.
- Rich set of counters would be available for performance analysis e.g. poll vs. interrupt count, chain sizes, distribution of load across lanes etc.
- In the man page, 'dlstat show' synopsis is deliberately spread across multiple lines. First line, dlstat show [-r | -t]... provides an easy template for printing most commonly queried statistics. Comprehensive and flexible approaches follow.
Note: Numbers in examples section of enclosed man page are used to illustrate output formatting only.
~ Shri
------------------------------------------------------------------------
_______________________________________________
crossbow-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss