DMA operation to 64-bits PC platform (continued)

[Frank van Eijkelenburg]

Hi,

I have a custom made PCIe board with a Virtex 5 FPGA on which I
implemented a DMA unit which uses the PCIe endpoint block plus v1.14.
I also implemented simple read/write operations from the PC to the
board (the board responds with completion TLPs). The read/write
operations are working, DMA is not working (transferring data from
FPGA to PC).

The board is inserted in a pc with Windows 7 64 bits platform. An
application allocates virtual memory and passes a pointer to the
memory block to the driver. The driver locks the memory pages and
creates a scatter-gather list with physical addresses by using the DMA
adapter structure. The physical addresses are written to the FPGA.

When I start a DMA operation by writing a register in the FPGA, I can
see in chipscope the correct physical addresses in the TLP header (of
the memory write requests). However, I do not see the correct values
in the allocated memory at the PC. What can I do to check where it is
going wrong?

In my opinion there are two possibilities: either the TLP is blocked
by a PCIe switch at the main board or the data is available at another
memory location.

I also tried the other direction (send memory read requests TLPs from
the FPGA to the PC and receive completion TLPs as answer at the memory
read request). In the completion TLPs I see the correct data (data I
wrote into the PC memory before starting the DMA operation).

Any suggestions/ideas are welcome.

Thanks in advance,

Frank

 

[glen herrmannsfeldt]

Frank van Eijkelenburg <fei.technolution@gmail.com> wrote:

I have a custom made PCIe board with a Virtex 5 FPGA on which I
implemented a DMA unit which uses the PCIe endpoint block plus v1.14.
I also implemented simple read/write operations from the PC to the
board (the board responds with completion TLPs). The read/write
operations are working, DMA is not working (transferring data from
FPGA to PC).
(snip)

When I start a DMA operation by writing a register in the FPGA, I can
see in chipscope the correct physical addresses in the TLP header (of
the memory write requests). However, I do not see the correct values
in the allocated memory at the PC. What can I do to check where it is
going wrong?

Not having tried to do DMA through PCI before, is data being
written, but the wrong data?

I would try writing all zeros or all ones and see if those come
through fine. It could be timing between the FPGA and PCI such
that the wrong data is being latched.

Then try slightly less predictable data and see what gets through.

-- glen

 

[Frank van Eijkelenburg]

On Aug 2, 5:28 pm, glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:

Frank van Eijkelenburg <fei.technolut...@gmail.com> wrote:

I have a custom made PCIe board with a Virtex 5 FPGA on which I
implemented a DMA unit which uses the PCIe endpoint block plus v1.14.
I also implemented simple read/write operations from the PC to the
board (the board responds with completion TLPs). The read/write
operations are working, DMA is not working (transferring data from
FPGA to PC).

(snip)

When I start a DMA operation by writing a register in the FPGA, I can
see in chipscope the correct physical addresses in the TLP header (of
the memory write requests). However, I do not see the correct values
in the allocated memory at the PC. What can I do to check where it is
going wrong?

Not having tried to do DMA through PCI before, is data being
written, but the wrong data?

That is what I do not know. Yes the correct data is send to the PC,
but if I readout the memory the values are unchanged.

I would try writing all zeros or all ones and see if those come
through fine.  It could be timing between the FPGA and PCI such
that the wrong data is being latched.

Then try slightly less predictable data and see what gets through.

-- glen

If it was timing, I expect the other way around also problems (which I
don't have). Also single memory read/write requests send from the PC
are working correctly.

 

[alpha]

On Aug 3, 3:26 am, Frank van Eijkelenburg <fei.technolut...@gmail.com>
wrote:

On Aug 2, 5:28 pm, glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:



Frank van Eijkelenburg <fei.technolut...@gmail.com> wrote:

I have a custom made PCIe board with a Virtex 5 FPGA on which I
implemented a DMA unit which uses the PCIe endpoint block plus v1.14.
I also implemented simple read/write operations from the PC to the
board (the board responds with completion TLPs). The read/write
operations are working, DMA is not working (transferring data from
FPGA to PC).

(snip)

When I start a DMA operation by writing a register in the FPGA, I can
see in chipscope the correct physical addresses in the TLP header (of
the memory write requests). However, I do not see the correct values
in the allocated memory at the PC. What can I do to check where it is
going wrong?

Not having tried to do DMA through PCI before, is data being
written, but the wrong data?

That is what I do not know. Yes the correct data is send to the PC,
but if I readout the memory the values are unchanged.



I would try writing all zeros or all ones and see if those come
through fine. It could be timing between the FPGA and PCI such
that the wrong data is being latched.

Then try slightly less predictable data and see what gets through.

-- glen

If it was timing, I expect the other way around also problems (which I
don't have). Also single memory read/write requests send from the PC
are working correctly.

Does CPU caches get snooped when DMA wrote into host memory? If not,
then you may not see right data by normal read.

 

[sreekanth]

AFAIK, in x86 architectures, io writes will be cache coherent, unless you're setting SNR (no snoop) in the write header by mistak

But to me this sounds like a Windows problem. Typically garden variety PC's don't block PCIE accesses, all of the address space is open to a peripheral. Acid test : if you can write to a given physical address from your FPGA, and read the value, you're golden in the physical address domain. So maybe what you think you're reading in your code is not actually getting to the correct physical address.

try :

write from FPGA card to a physical addres
read back using FPGA card, see if good.
write some known data from windows softwar
read back using FPGA card.

 

[Frank van Eijkelenburg]

On Aug 3, 6:03 pm, alpha <zhg....@gmail.com> wrote:

On Aug 3, 3:26 am, Frank van Eijkelenburg <fei.technolut...@gmail.com
wrote:



On Aug 2, 5:28 pm, glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:

Frank van Eijkelenburg <fei.technolut...@gmail.com> wrote:

I have a custom made PCIe board with a Virtex 5 FPGA on which I
implemented aDMAunit which uses the PCIe endpoint block plus v1.14.
I also implemented simple read/write operations from the PC to the
board (the board responds with completion TLPs). The read/write
operations are working,DMAis not working (transferring data from
FPGA to PC).

(snip)

When I start aDMAoperation by writing a register in the FPGA, I can
see in chipscope the correct physical addresses in the TLP header (of
the memory write requests). However, I do not see the correct values
in the allocated memory at the PC. What can I do to check where it is
going wrong?

Not having tried to doDMAthrough PCI before, is data being
written, but the wrong data?

That is what I do not know. Yes the correct data is send to the PC,
but if I readout the memory the values are unchanged.

I would try writing all zeros or all ones and see if those come
through fine.  It could be timing between the FPGA and PCI such
that the wrong data is being latched.

Then try slightly less predictable data and see what gets through.

-- glen

If it was timing, I expect the other way around also problems (which I
don't have). Also single memory read/write requests send from the PC
are working correctly.

Does CPU caches get snooped whenDMAwrote into host memory?  If not,
then you may not see right data by normal read.

Finally, I solved the problem. For those who want to learn from
mistakes of others, here comes the cause of the problem:

The packets which were transmitted to the pc were too large (more than
the maximum payload size of the receiver). In that case, the packets
are simply dropped (no errors). Of course I have to read the maximum
payload size from the device control register in the PCI Express
Capability structure.

best regards,

Frank