OT: Gigabit Ethernet MAC Throughput

[owner]

Hi,

This is somewhat OT, but I can't find a more suitable newsgroup.

I am using a Gigabit Ethernet MAC chip from Marv***, which claims full
line rate (compliance to IEEE 802.3ab). It has an integrated GMAC,
PHY/Serdes, and PCI interface (64-bit, 66MHz). On the PCI bus side, we
connect it to a Spartan IIe-300 with a Xilinx PCI Logicore.

Everything works in the FPGA, PCI read/write transactions, but we
cannot achieve full line rate for 64-byte frame length. We can only
achieve 680Mbps because of the overhead in reading/writing
descriptors.

My questions are: Is this expected? Is this acceptable? I have no
prior experience with Gigabit Ethernet MAC, and appreciates any
feedback.

Thanks

Kang Liat Chuan

 

[Hal Murray]

My questions are: Is this expected? Is this acceptable? I have no
prior experience with Gigabit Ethernet MAC, and appreciates any
feedback.

It's not surprising.

Is the limitation in the hardware or in the software?
Are you running out of PCI bus cycles or CPU cycles?

Count the cycles on the PCI bus and do the arithmetic.
How many CPU cycles does it take to process each packet?

Are you reading/writing the packets from off-chip memory?
How long does that take?


"acceptable" depends upon your usage/application. It could
be a disaster if the box on the other end is sending faster
than you can receive so that X% of the packets are falling
through the cracks.

--
The suespammers.org mail server is located in California. So are all my
other mailboxes. Please do not send unsolicited bulk e-mail or unsolicited
commercial e-mail to my suespammers.org address or any of my other addresses.
These are my opinions, not necessarily my employer's. I hate spam.

 

[Allan Herriman]

On 19 Apr 2004 02:19:04 -0700, kanglc@starhub.net.sg (owner) wrote:

Hi,

This is somewhat OT, but I can't find a more suitable newsgroup.

I am using a Gigabit Ethernet MAC chip from Marv***, which claims full
line rate (compliance to IEEE 802.3ab). It has an integrated GMAC,
PHY/Serdes, and PCI interface (64-bit, 66MHz). On the PCI bus side, we
connect it to a Spartan IIe-300 with a Xilinx PCI Logicore.

Everything works in the FPGA, PCI read/write transactions, but we
cannot achieve full line rate for 64-byte frame length. We can only
achieve 680Mbps because of the overhead in reading/writing
descriptors.

My questions are: Is this expected? Is this acceptable? I have no
prior experience with Gigabit Ethernet MAC, and appreciates any
feedback.

I've seen Marv*** GbE PHYs work at 100% of line rate, in one of these:
http://advanced.comms.agilent.com/RouterTester/datasheets/e7918a.htm
Never used one of their MACs though.

Umm,
64 bytes frame
+ 8 bytes preamble and SFD
+ 12 bytes minimum interframe gap
= 84 bytes on the line => 1.4881 Mpps, which corresponds to 761.9Mbps
of useful data transfer.

You seem to be achieving about 89% of this. This wouldn't be
acceptable for test equipment, but may be ok for your application
(which you didn't specify).

Regards,
Allan.

 

[Hal Murray]

Is the limitation in the hardware or in the software?
Are you running out of PCI bus cycles or CPU cycles?

Count the cycles on the PCI bus and do the arithmetic.
How many CPU cycles does it take to process each packet?

There is another possibility I overlooked. There is probably
a dedicated processor on the chip to process control blocks
and such. You might run out of cycles in that CPU.

The data sheet or marketing blurbs might have some info about
how well the chip works, especially if it can keep up with
full wire speed and small packets.

You could also try to get a PCI card using that chip and plug
it into a motherboard/system and see how fast you can make it
go. This depends a lot on the driver and OS and your test
programs. Of course, you need a place to stand while running
the tests. It might be simplest to get two of the boards, setup
two systems and send packets from one to the other. (This could
be a big rathole.)

--
The suespammers.org mail server is located in California. So are all my
other mailboxes. Please do not send unsolicited bulk e-mail or unsolicited
commercial e-mail to my suespammers.org address or any of my other addresses.
These are my opinions, not necessarily my employer's. I hate spam.

 

[owner]

Allan Herriman <allan.herriman.hates.spam@ctam.com.au.invalid> wrote in message news:<p68780ta5piddrjllu6s8mkscf04jqe49j@4ax.com>...

On 19 Apr 2004 02:19:04 -0700, kanglc@starhub.net.sg (owner) wrote:

Hi,

This is somewhat OT, but I can't find a more suitable newsgroup.

I am using a Gigabit Ethernet MAC chip from Marv***, which claims full
line rate (compliance to IEEE 802.3ab). It has an integrated GMAC,
PHY/Serdes, and PCI interface (64-bit, 66MHz). On the PCI bus side, we
connect it to a Spartan IIe-300 with a Xilinx PCI Logicore.

Everything works in the FPGA, PCI read/write transactions, but we
cannot achieve full line rate for 64-byte frame length. We can only
achieve 680Mbps because of the overhead in reading/writing
descriptors.

My questions are: Is this expected? Is this acceptable? I have no
prior experience with Gigabit Ethernet MAC, and appreciates any
feedback.

I've seen Marv*** GbE PHYs work at 100% of line rate, in one of these:
http://advanced.comms.agilent.com/RouterTester/datasheets/e7918a.htm
Never used one of their MACs though.

Thanks for your reply.

Yes I am aware of the Marv*** PHY used in Agilent equipment.
Our project is similar - a handheld test instrument.

Umm,
64 bytes frame
+ 8 bytes preamble and SFD
+ 12 bytes minimum interframe gap
= 84 bytes on the line => 1.4881 Mpps, which corresponds to 761.9Mbps
of useful data transfer.

You seem to be achieving about 89% of this. This wouldn't be
acceptable for test equipment, but may be ok for your application
(which you didn't specify).

Regards,
Allan.

On the PCI bus (64-bit, 66MHz), it shows that for every frame that
was fetch by the MAC, it has to read one descriptor and then write
one descritor, to release the ownership. From the datasheet, we cannot
find ways to burst the frames, in order to decrease the overhead
due to the handling of descriptors. Or is there?

Regards,
LC

 

[Allan Herriman]

On 20 Apr 2004 00:21:17 -0700, kanglc@starhub.net.sg (owner) wrote:

Allan Herriman <allan.herriman.hates.spam@ctam.com.au.invalid> wrote in message news:<p68780ta5piddrjllu6s8mkscf04jqe49j@4ax.com>...
On 19 Apr 2004 02:19:04 -0700, kanglc@starhub.net.sg (owner) wrote:

[snip]

A GbE MAC is reasonably simple. I expect that you could make your own
in an FPGA, and this could be cheaper and smaller than a solution
involving both an FPGA and an ASIC. This way you could control all
aspects of the MAC, including where its descriptors are held (possibly
in a very fast local memory).
You would still need an ASIC for the PHY though.

Regards,
Allan.

 

[Hal Murray]

On the PCI bus (64-bit, 66MHz), it shows that for every frame that
was fetch by the MAC, it has to read one descriptor and then write
one descritor, to release the ownership. From the datasheet, we cannot
find ways to burst the frames, in order to decrease the overhead
due to the handling of descriptors. Or is there?

Where are the descriptors stored? Off chip RAM? If so, consider
putting them on chip. Or you could add special logic to cache
a clump of descriptors on chip.

--
The suespammers.org mail server is located in California. So are all my
other mailboxes. Please do not send unsolicited bulk e-mail or unsolicited
commercial e-mail to my suespammers.org address or any of my other addresses.
These are my opinions, not necessarily my employer's. I hate spam.

 

[owner]

hmurray@suespammers.org (Hal Murray) wrote in message news:<108aipgspj1o8d@corp.supernews.com>...

On the PCI bus (64-bit, 66MHz), it shows that for every frame that
was fetch by the MAC, it has to read one descriptor and then write
one descritor, to release the ownership. From the datasheet, we cannot
find ways to burst the frames, in order to decrease the overhead
due to the handling of descriptors. Or is there?

Where are the descriptors stored? Off chip RAM? If so, consider
putting them on chip. Or you could add special logic to cache
a clump of descriptors on chip.

Thanks for your suggestions Hal.

In our platform, the FPGA (Spartan IIe-300) connects to the MAC-PHY chip
directly via the PCI bus. The CPU (AMD Alchemy) is also on the PCI bus.
These are the only 3 devices on the PCI bus. After CPU sets up the
transmit frames and descriptors in the FPGA, and set up the necessary
registers inside the MAC, it initiates the FPGA to start the traffic
generation. The FPGA issues a PCI write transaction to one of the MAC
registers to start the operation. The MAC becomes a PCI master, fetching
the descriptors and frams from the FPGA. Each time a frame a fetched,
the MAC writes a descriptor (in the FPGA) to clear a OWN bit.

If the traffic generator operation is still in progress, the FPGA will
immediately reset the OWN bit to '1', so that the MAC knows it has
ownership of that particular descriptor.

I think our PCI bus timing cycles show that the system is working
correctly. Just that there is no way to burst a number of frames
to the MAC. This is the bottle neck for 64-byte frames. We can only
achieve at best 680Mbps on the line side.

Regards,
LC