How To Synchronize FPGAs

[Leroy Tanner]

Hello newsreaders,

For a while I have been confronted with the following task which I find
quite challenging but unfortuantely didn't manage to solve it, yet.
What I want to do is to use 2-4 FPGAs (Xilinx Virtex 2 Pro) together on one
printed circuit board (PCB). They are used to process a large amount of
incoming serial data (data rates of several GHz's). My idea is to handle
that data parallel by the 2-4 FPGAs. But now there arises the problem how to
adequately split the data and how to synchronize the FPGAs among one
another, in particular?
Is it possible or first of all a realistic idea to synchronize multiple
FPGAs in the GHz range? How can this be done without much protocoll
overhead? I would like to do it without applying an extra transfer protocoll
among the FPGAs just for that purpose! Up to this date I didn't find a
proper solution, yet.
Maybe someone can give me a hint? Any ideas how to solve that problem?

Regards, Leroy Tanner

 

[Fred Ferd]

My idea is to handle
that data parallel by the 2-4 FPGAs. But now there arises the problem how
to
adequately split the data

put 1/2 a bit here and 1/2 a bit there.

and how to synchronize the FPGAs among one
another,

well, the first says "synchronise your watches"
and the others do that.

Is it possible or first of all a realistic idea to synchronize multiple
FPGAs in the GHz range?

yes, ony the NSFPGA's cant be synchronised.
NSFPGA = Non-synchronisable FGPA

How can this be done without much protocoll
overhead?

NTP .. network time protocol.

I would like to do it without applying an extra transfer protocoll

Ah, then you want ESPP. ESP Protocol

among the FPGAs just for that purpose! Up to this date I didn't find a
proper solution, yet.

apologies for lampooning you, but a clock is a clock.


If you have multiple data streams clocked off seperate clocks, then you have
a problem with clock drift. If the data has seperate clocks, how is it
synchronised at the source ?

If the data is split into N streams by a CPU, then the split is done by
packetising the data and you put a time stamp on each packet.

The other end then reads the packets and puts them back into order.


Of course, you dont have to have a time stamp, because if you send packet A
to stream1 , then packet b to stream 2, then packet C to stream 3, then they
will be received at the other end in order A B C anyway, which is correct,
right ?

 

[The Real Andy]

On Wed, 22 Sep 2004 11:15:55 +0200, "Leroy Tanner"
<ikeepthespiritalive@freenet.de> wrote:

Hello newsreaders,

For a while I have been confronted with the following task which I find
quite challenging but unfortuantely didn't manage to solve it, yet.
What I want to do is to use 2-4 FPGAs (Xilinx Virtex 2 Pro) together on one
printed circuit board (PCB). They are used to process a large amount of
incoming serial data (data rates of several GHz's). My idea is to handle
that data parallel by the 2-4 FPGAs. But now there arises the problem how to
adequately split the data and how to synchronize the FPGAs among one
another, in particular?
Is it possible or first of all a realistic idea to synchronize multiple
FPGAs in the GHz range? How can this be done without much protocoll
overhead? I would like to do it without applying an extra transfer protocoll
among the FPGAs just for that purpose! Up to this date I didn't find a
proper solution, yet.
Maybe someone can give me a hint? Any ideas how to solve that problem?

Regards, Leroy Tanner

Ghz, one would assume fibre. I am pissed as usual, but me thinks that
someone already has hardware for that kind of stuff. Try google.
Failing that try a shift register, but good luck.

 

[David L. Jones]

"Leroy Tanner" <ikeepthespiritalive@freenet.de> wrote in message news:<cirfvf$j53$1@mamenchi.zrz.TU-Berlin.DE>...

Hello newsreaders,

For a while I have been confronted with the following task which I find
quite challenging but unfortuantely didn't manage to solve it, yet.
What I want to do is to use 2-4 FPGAs (Xilinx Virtex 2 Pro) together on one
printed circuit board (PCB). They are used to process a large amount of
incoming serial data (data rates of several GHz's). My idea is to handle
that data parallel by the 2-4 FPGAs. But now there arises the problem how to
adequately split the data and how to synchronize the FPGAs among one
another, in particular?
Is it possible or first of all a realistic idea to synchronize multiple
FPGAs in the GHz range? How can this be done without much protocoll
overhead? I would like to do it without applying an extra transfer protocoll
among the FPGAs just for that purpose! Up to this date I didn't find a
proper solution, yet.
Maybe someone can give me a hint? Any ideas how to solve that problem?

Regards, Leroy Tanner

Is this just one high speed serial input you want to process, or
multiple inputs?
What is the output format?, serial or parallel?
Is the incoming serial data demuxed into multiple outputs?
What is the data?, and how much processing (if any) does it require?
All this will dictate how best to implement your design.
You don't want to go "off silicon" (multiple FPGAs) if you don't have
to at this sort of rate.
FPGAs can I/O in the GHz range, but synchronising them to do stuff at
that rate is another matter. It all depends on exactly what you want
to do.

Dave

 

[Leroy Tanner]

"David L. Jones" <altzone@gmail.com>:

Is this just one high speed serial input you want to process, or
multiple inputs?

The high speed input data is received through an Infiniband 12x-Interface,
i.e. 12 x 1-channel Infiniband @ 2,5 Gbit/s serial via channelbonding in
parallel (=30 Gbit/s). The data has to be processed by the FPGAs and then
send forward to another unit. Because of the large amount of data and the
limited number of FPGA I/Os, I thought about each FPGA handling perhaps just
a minor portion of the whole data in parrallel. And there the synchonization
problem comes up!

What is the output format?, serial or parallel?
Is the incoming serial data demuxed into multiple outputs?
What is the data?, and how much processing (if any) does it require?
All this will dictate how best to implement your design.
You don't want to go "off silicon" (multiple FPGAs) if you don't have
to at this sort of rate.
FPGAs can I/O in the GHz range, but synchronising them to do stuff at
that rate is another matter. It all depends on exactly what you want
to do.

The Virtex-II Pro series offers even up to 20 full-duplex 3.125Gbit/s
channels. The reason why this cannot be realized on a single FPGA is the
fact that we disscussed just the input channels, so far. There is a great
amount of output, too, so I cannot use all of the FPGA's highspeed I/Os
(i.e. RocketIOs) just for the incoming data. In reference to a maximum
amount of 20 RocketIOs on the largest Virtex-II Pro I assume that I will use
4 RocketIOs per FPGA for the purpose of handling the complete 30 Gbit/s. So
that means I must run approximately four FPGAs in parallel!

 

[Alex Gibson]

"Leroy Tanner" <ikeepthespiritalive@freenet.de> wrote in message
news:cj8dve$rv1$1@mamenchi.zrz.TU-Berlin.DE...

"David L. Jones" <altzone@gmail.com>:

Is this just one high speed serial input you want to process, or
multiple inputs?

The high speed input data is received through an Infiniband
12x-Interface,
i.e. 12 x 1-channel Infiniband @ 2,5 Gbit/s serial via channelbonding in
parallel (=30 Gbit/s). The data has to be processed by the FPGAs and then
send forward to another unit. Because of the large amount of data and the
limited number of FPGA I/Os, I thought about each FPGA handling perhaps
just
a minor portion of the whole data in parrallel. And there the
synchonization
problem comes up!

What is the output format?, serial or parallel?
Is the incoming serial data demuxed into multiple outputs?
What is the data?, and how much processing (if any) does it require?
All this will dictate how best to implement your design.
You don't want to go "off silicon" (multiple FPGAs) if you don't have
to at this sort of rate.
FPGAs can I/O in the GHz range, but synchronising them to do stuff at
that rate is another matter. It all depends on exactly what you want
to do.

The Virtex-II Pro series offers even up to 20 full-duplex 3.125Gbit/s
channels. The reason why this cannot be realized on a single FPGA is the
fact that we disscussed just the input channels, so far. There is a great
amount of output, too, so I cannot use all of the FPGA's highspeed I/Os
(i.e. RocketIOs) just for the incoming data. In reference to a maximum
amount of 20 RocketIOs on the largest Virtex-II Pro I assume that I will
use
4 RocketIOs per FPGA for the purpose of handling the complete 30 Gbit/s.
So
that means I must run approximately four FPGAs in parallel!

Can you use 1 fpga to synch and spilt up the data for the others to process
then use another to recombine the data if required.

Split the data into packets with an id, send the packets to the other fpgas
to process then recombine based on the id number.

Seen this done for beam steering but using a beowulf cluster
running linux and c++ programs then recombining the data
and tiling it in matlab to make an image.

Alex Gibson

 

[David L. Jones]

"Leroy Tanner" <ikeepthespiritalive@freenet.de> wrote in message news:<cj8dve$rv1$1@mamenchi.zrz.TU-Berlin.DE>...

"David L. Jones" <altzone@gmail.com>:

Is this just one high speed serial input you want to process, or
multiple inputs?

The high speed input data is received through an Infiniband 12x-Interface,
i.e. 12 x 1-channel Infiniband @ 2,5 Gbit/s serial via channelbonding in
parallel (=30 Gbit/s). The data has to be processed by the FPGAs and then
send forward to another unit. Because of the large amount of data and the
limited number of FPGA I/Os, I thought about each FPGA handling perhaps just
a minor portion of the whole data in parrallel. And there the synchonization
problem comes up!

What is the output format?, serial or parallel?
Is the incoming serial data demuxed into multiple outputs?
What is the data?, and how much processing (if any) does it require?
All this will dictate how best to implement your design.
You don't want to go "off silicon" (multiple FPGAs) if you don't have
to at this sort of rate.
FPGAs can I/O in the GHz range, but synchronising them to do stuff at
that rate is another matter. It all depends on exactly what you want
to do.

The Virtex-II Pro series offers even up to 20 full-duplex 3.125Gbit/s
channels. The reason why this cannot be realized on a single FPGA is the
fact that we disscussed just the input channels, so far. There is a great
amount of output, too, so I cannot use all of the FPGA's highspeed I/Os
(i.e. RocketIOs) just for the incoming data. In reference to a maximum
amount of 20 RocketIOs on the largest Virtex-II Pro I assume that I will use
4 RocketIOs per FPGA for the purpose of handling the complete 30 Gbit/s. So
that means I must run approximately four FPGAs in parallel!

Without knowing what kind of data you have and what processing has to
be done in what amount of time etc, it really isn't possible to make
any recommendations on how to go about it. There are bound to be
dozens of vitally important issues in any chosen architecture. So
asking for general help on something really high end like this is kind
of stabbing in the dark.

Yes, it is possible to split high speed serial data up and process it
in multiple FPGAs, I've worked on projects that have done this
(similar to the other reply - real-time beam forming). But each
project will have an entirely different set of requirements. For a
project of this complexity extensive research needs to be done on the
best architecture, device and tools to use, otherwise you'll be six
months into the development cycle and find that - oops, you don't have
enough bandwidth, or clock skew is killing you, or your place and
route tools just can't handle it, or your sythesis tools are chucking
a wobbly, or your chosen device doesn't have this or that
functionality, or...
Good luck!

Dave