FPGA to solve the two most annoying problems on usenet - Sug

C

Charles Xavier

Guest
As you all know, downloading files from usenet leaves you with two
sets of files.. The rar files from what you're downloading and the
par2 files for incomplete file repair.

If anyone has attempted to download anything in the 8GB range, you'll
find that well.. if you're missing enough parts of the file, the par2
recovery can be a painful, painful process taking up to three hours in
some cases.

I'm sick of it.

So, here's the idea. Use a FPGA to do the reed-solomon decoding to
accelerate the PAR2 repair/recovery process. The system should utilize
a USB connection to pipe data directly from the disk to the FPGA that
will do the offboard processing of the data. The data transfer should
be controlled by an application on the computer.

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

SPECS: The development system i'm using is the XILINX ML-505 board
with the Virtex 5 chip. This is a open-source project being done for
fun and learning btw.

Suggestions / Comments / Complaints?
 
Seems like a really nice idea. Please notify your progress.
 
The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..
Click to expand...
There are chips that will do this already. Here's one:

http://www.aspex-semi.com/q/product_features.shtml

Cheers,
Jon
 
On Jun 11, 11:24 am, Jon Beniston <j...@beniston.com> wrote:
The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

There are chips that will do this already. Here's one:

http://www.aspex-semi.com/q/product_features.shtml

Cheers,
Jon
Click to expand...
Thanks for the reply. Aspects of this design can certainly be used and
incorporated into this design, but as you know.. Their product isn't
free/open source. As this is also a learning project, I think it would
be better to build it from the ground up (using some pre-existing open
source IP if possible).
 
So, here's the idea. Use a FPGA to do the reed-solomon decoding to
accelerate the PAR2 repair/recovery process. The system should utilize
a USB connection to pipe data directly from the disk to the FPGA that
will do the offboard processing of the data. The data transfer should
be controlled by an application on the computer.
Click to expand...
Why USB ? Gigabit Ethernet would be much faster ;)
Don't tell me a Virtex5 can't handle a measly 100 MB/s ;)

And... is it a PCI express slot that I see there ?
That's even better !

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..
Click to expand...
I was under the impression that x264 was h264 in disguise. Perhaps
tweaking the file format would be enough ?

Also you could recode the thing in a much simpler codec like MPEG2 or
something, who cares if it takes 10x the size, just stream it in realtime
;)

Stick a DVI port on your FPGA !

SPECS: The development system i'm using is the XILINX ML-505 board
with the Virtex 5 chip. This is a open-source project being done for
fun and learning btw.
Click to expand...
 
Charles Xavier wrote:
If anyone has attempted to download anything in the 8GB range, you'll
find that well.. if you're missing enough parts of the file, the par2
recovery can be a painful, painful process taking up to three hours in
some cases.

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
Click to expand...
I have a hard time believing that those are the "two most annoying problems
on usenet". I've been using Usenet since 1984, and I've never personally
encountered either problem.

I've never tried to download an 8GB file from Usenet, and don't have any
clue why anyone would want to do such a thing. There are much better
ways to distributed 8GB files. But if I *did* want to download an 8GB
file from Usenet, I can't imagine that waiting three hours for a
reconstruction of missing pieces would really bother me that much.
There's never been a single thing that I've needed from Usenet so
urgently that it couldn't have waited three more hours.

I watch a fair number of movies, but I've never had any problem with
x.264. It sounds like you just need better player software. That has
nothing to do with Usenet.

Since it doesn't appear that there is any real problem here, it also
doesn't appear that there is any need for an FPGA-based "solution".
 
On Jun 13, 5:46 pm, Eric Smith <e...@brouhaha.com> wrote:
Charles Xavier wrote:
If anyone has attempted to download anything in the 8GB range, you'll
find that well.. if you're missing enough parts of the file, the par2
recovery can be a painful, painful process taking up to three hours in
some cases.
The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA

I have a hard time believing that those are the "two most annoying problems
on usenet".  I've been using Usenet since 1984, and I've never personally
encountered either problem.

I've never tried to download an 8GB file from Usenet, and don't have any
clue why anyone would want to do such a thing.  There are much better
ways to distributed 8GB files.  But if I *did* want to download an 8GB
file from Usenet, I can't imagine that waiting three hours for a
reconstruction of missing pieces would really bother me that much.
There's never been a single thing that I've needed from Usenet so
urgently that it couldn't have waited three more hours.

I watch a fair number of movies, but I've never had any problem with
x.264.  It sounds like you just need better player software.  That has
nothing to do with Usenet.

Since it doesn't appear that there is any real problem here, it also
doesn't appear that there is any need for an FPGA-based "solution".
Click to expand...
Thank you for sharing. Let your next post be in contribution to the
project.

If it doesn't apply to you and you don't see the need, why even reply?
 
On Jun 14, 12:39 pm, "Symon" <symon_bre...@hotmail.com> wrote:
Charles Xavier wrote:
On Jun 13, 5:46 pm, Eric Smith <e...@brouhaha.com> wrote:
Charles Xavier wrote:
If anyone has attempted to download anything in the 8GB range,
you'll find that well.. if you're missing enough parts of the file,
the par2 recovery can be a painful, painful process taking up to
three hours in some cases.
The XBOX 360 doesn't play x.264 and all the good movies are in
x.264. Converting from x.264 to h.264 could be done offboard on an
FPGA

I have a hard time believing that those are the "two most annoying
problems on usenet". I've been using Usenet since 1984, and I've
never personally encountered either problem.

Since it doesn't appear that there is any real problem here, it also
doesn't appear that there is any need for an FPGA-based "solution".

Thank you for sharing. Let your next post be in contribution to the
project.

If it doesn't apply to you and you don't see the need, why even reply?

Hi Charles,
Maybe to share the benefit of his experience to help you avoid wasting your
time on a solution without a problem?
Good luck, Syms.
Click to expand...
Engineers can often be of a single track mind, so let me restate. This
is a problem that I am having. I could certainly get a faster
computer, or try to finesse the algorithms for the decoding, but I
want to learn about this specific algorithm implementation on an FPGA,
so I'm using the FPGA to solve this problem (that is a problem for
me).

To me, this is a practical solution and an excellent gateway for
myself and others who want to get into FPGA-based hardware
acceleration of software algorithms. You give me a better way that
solves a practical problem for _me_ and I'll take it.

This reminds of the idiots at In-N-Out Burger who created the drive-
through fast food restaurant (they claim). It doesn't affect the
geriatric who only wants to eat inside because they can't imagine
driving while eating. But apparently it turned out to be a good idea.

Responding to this type of feedback is the real waste of time.
 
On Wed, 11 Jun 2008 06:50:52 -0700 (PDT), Charles Xavier
<skelotar@gmail.com> wrote:

As you all know, downloading files from usenet leaves you with two
sets of files.. The rar files from what you're downloading and the
par2 files for incomplete file repair.

If anyone has attempted to download anything in the 8GB range, you'll
find that well.. if you're missing enough parts of the file, the par2
recovery can be a painful, painful process taking up to three hours in
some cases.

I'm sick of it.

So, here's the idea. Use a FPGA to do the reed-solomon decoding to
accelerate the PAR2 repair/recovery process. The system should utilize
a USB connection to pipe data directly from the disk to the FPGA that
will do the offboard processing of the data. The data transfer should
be controlled by an application on the computer.

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

SPECS: The development system i'm using is the XILINX ML-505 board
with the Virtex 5 chip. This is a open-source project being done for
fun and learning btw.

Suggestions / Comments / Complaints?
Click to expand...
Just program one of these:

http://www.eetimes.com/news/latest/showArticle.jhtml;jsessionid=YBF0E0YIMESH2QSNDLSCKHA?articleID=208404063

240 cores, teraflop.

John
 
I have an idea simular or not to your mind..(but I still
thinking....hope you are now starting ..your project).
my project is that

FILE COMPRESSION (zip or other)
read files from SD card, then compress it, and write a new zipped file
back to SD card.

it will not include the PC to do it.

On Jun 11, 9:50 am, Charles Xavier <skelo...@gmail.com> wrote:
As you all know, downloading files from usenet leaves you with two
sets of files.. The rar files from what you're downloading and the
par2 files for incomplete file repair.

If anyone has attempted to download anything in the 8GB range, you'll
find that well.. if you're missing enough parts of the file, the par2
recovery can be a painful, painful process taking up to three hours in
some cases.

I'm sick of it.

So, here's the idea. Use a FPGA to do the reed-solomon decoding to
accelerate the PAR2 repair/recovery process. The system should utilize
a USB connection to pipe data directly from the disk to the FPGA that
will do the offboard processing of the data. The data transfer should
be controlled by an application on the computer.

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

SPECS: The development system i'm using is the XILINX ML-505 board
with the Virtex 5 chip. This is a open-source project being done for
fun and learning btw.

Suggestions / Comments / Complaints?
Click to expand...
 
Consider using rateless codes, reed-solomon and ldpc based codes do
provide erasure correction capabilities, however rateless codes such
those based on LT class of codes (tornado etc) ,or more generally
digital fountain codes, will have better performance due to their near
linear decoding complexity as opposed to say reed-solomon erasure only
decoding complexity which is around O(n^2) for naive implementations.



Arash Partow
__________________________________________________
Be one who knows what they don't know,
Instead of being one who knows not what they don't know,
Thinking they know everything about all things.
http://www.partow.net


On Jun 11, 11:50 pm, Charles Xavier <skelo...@gmail.com> wrote:
As you all know, downloading files from usenet leaves you with two
sets of files.. The rar files from what you're downloading and the
par2 files for incomplete file repair.

If anyone has attempted to download anything in the 8GB range, you'll
find that well.. if you're missing enough parts of the file, the par2
recovery can be a painful, painful process taking up to three hours in
some cases.

I'm sick of it.

So, here's the idea. Use a FPGA to do thereed-solomondecoding to
accelerate the PAR2 repair/recovery process. The system should utilize
a USB connection to pipe data directly from the disk to the FPGA that
will do the offboard processing of the data. The data transfer should
be controlled by an application on the computer.

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

SPECS: The development system i'm using is the XILINX ML-505 board
with the Virtex 5 chip. This is a open-source project being done for
fun and learning btw.

Suggestions / Comments / Complaints?
Click to expand...
 
You must log in or register to reply here.

Welcome to EDABoard.com

Sponsor

Back
Top