fifo or sdram bug?

[kaz]

In our system a signal is passed through a couple of fifos inside FPGA an
then onto external sdram to be read by application software. All looks o
except that some units in the field show occasional errors in that signa
read from sdram. The error is as follows: odd samples are offset by
samples from the even. So if we remove this offset then signal looks ok.

I can't reproduce the error in the lab. So I depend on some speculations
It could be the fifos or the sdram. Anyone has come across such issue? m
suspicion is on the sdram as it is configured as 8 pages? Also the sdra
itself has an internal fifo that muxes 128 bits onto 16 (again factor o
8)? any input appreciated.

Ka
--------------------------------------
Posted through http://www.FPGARelated.com

 

[KJ]

On Thursday, July 30, 2015 at 5:05:50 AM UTC-4, kaz wrote:

In our system a signal is passed through a couple of fifos inside FPGA and
then onto external sdram to be read by application software. All looks ok
except that some units in the field show occasional errors in that signal
read from sdram. The error is as follows: odd samples are offset by 8
samples from the even. So if we remove this offset then signal looks ok.

I can't reproduce the error in the lab. So I depend on some speculations.
It could be the fifos or the sdram. Anyone has come across such issue? my
suspicion is on the sdram as it is configured as 8 pages? Also the sdram
itself has an internal fifo that muxes 128 bits onto 16 (again factor of
8)? any input appreciated.

Kaz

Is the problem that the data is off by 8, or that the data gets stored in a location that is off by 8? If it's that the data is off by 8, then the number of sdram pages or the sdram muxing is not relevant. What exactly do you mean by 'off by 8'? Is data bit 3 in the wrong state? Is it that data bit 3 is always 1 when wrong or is it that data bit 3 is wrong, and when it is wrong that bit might be 1 or it might be 0.

I would also highly doubt that the problem is in the commercial sdram, almost without doubt it is in something that you have designed, not elsewhere.

- Has your design passed static timing analysis?
- Are all of the I/O and clock frequencies correctly specified to the timing analysis tool?
- Try warming up the part with a heat gun or cooling it off with cool spray in the lab. Does the design still work in the lab? If not, you have a timing problem.

In fact, based on your description so far, it is almost certainly a timing issue, so that would be the best place to start looking.

Kevin

 

[kaz]

Is the problem that the data is off by 8, or that the data gets store
in
a location that is off by 8? If it's that the data is off by 8, the
the
number of sdram pages or the sdram muxing is not relevant. What exactl
do
you mean by 'off by 8'? Is data bit 3 in the wrong state? Is it tha
data
bit 3 is always 1 when wrong or is it that data bit 3 is wrong, and when
it is wrong that bit might be 1 or it might be 0.

data is 16 bits wide, nothing wrong with bits. all sample values ar
correct.
odd samples do not follow their even members e.g. if a correct stream i
indexed as 0,1,2,3,4,5,6,7,8,9,10...etc then what we get is:
0,9,2,11,4,13,6,15,8,17,10

Thus all samples are correct individually. even stream is correct a
0,2,4,6,8... and odd stream is also correct as sequence 9,11,13,15,...etc

but there is this offset where instead of 0,1,2,3.. I get 0,9,2,11...

- Has your design passed static timing analysis?
yes certainly,

- Are all of the I/O and clock frequencies correctly specified to the
timing analysis tool?
yes

- Try warming up the part with a heat gun or cooling it off with cool
spray in the lab. Does the design still work in the lab? If not, yo
have a
timing problem.

can't do that in the field, units are concealed mobile radio heads. w
have deployed many thousands of them. only a tiny percentage shows th
issue.

Kaz

--------------------------------------
Posted through http://www.FPGARelated.com

 

[kaz]

- Try warming up the part with a heat gun or cooling it off with cool
spray in the lab. Does the design still work in the lab? If not, yo
have a
timing problem.

In fact, based on your description so far, it is almost certainly
timing
issue, so that would be the best place to start looking.

Kevin

We have done that in the lab, warming/freezing across full range but coul
not reproduce the issue. test was iterated over thousand times to catc
any intermittent behaviour but all passed.

Kaz

--------------------------------------
Posted through http://www.FPGARelated.com

 

[KJ]

On Thursday, July 30, 2015 at 7:32:59 AM UTC-4, kaz wrote:

- Try warming up the part with a heat gun or cooling it off with cool
spray in the lab. Does the design still work in the lab? If not, you
have a
timing problem.

In fact, based on your description so far, it is almost certainly a
timing
issue, so that would be the best place to start looking.

Kevin

We have done that in the lab, warming/freezing across full range but could
not reproduce the issue. test was iterated over thousand times to catch
any intermittent behaviour but all passed.

Kaz

Some next steps to consider...
- You did the thermal testing with field return boards that exhibited the problem, correct?
- Are there other differences between lab use and field use that could contribute such as with the power supply? The power supply is probably a stretch given the symptoms you describe, but just wondering what environmental differences might be going on.
- What kind of DRAM are you using?
- Commercial IP for the controller or home grown?
- Since you said in another post that the data starts at 16 bits, widens to 32 (I presume at input to the DRAM Controller) and then narrows back down to 16 at the I/O pins. Are clock domains being crossed along the way? Is your timing analysis set to ignore crossings? If so, shut that off and re-analyze each crossing.
- How does the DDR controller receive input commands? By that I mean is it given addresses for each read/write to be performed or is it given a start address and a burst size? If given a start address and burst size, then that would likely exonerate everything that is upstream of the DRAM controller (except for possible clock domain crossing issues)
- Review the PCB routing and look at signal integrity on the PCB?

Focus on getting the field returns to fail in the lab. Without that you'll have no way to verify any potential fix candidate.

Kevin Jennings

 

[Rob Gaddi]

On Thu, 30 Jul 2015 06:14:11 -0500, kaz wrote:

- Has your design passed static timing analysis?
yes certainly,

Then are you certain all your timing constraints are correct? I'm with
KJ, this problem description makes me immediately leap to a timing
problem.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.

 

[glen herrmannsfeldt]

kaz <37480@fpgarelated> wrote:

Is the problem that the data is off by 8, or that the data gets stored
in a location that is off by 8? If it's that the data is off by 8,
then the number of sdram pages or the sdram muxing is not relevant.

(snip)

data is 16 bits wide, nothing wrong with bits. all sample values are
correct.
odd samples do not follow their even members e.g. if a correct stream is
indexed as 0,1,2,3,4,5,6,7,8,9,10...etc then what we get is:
0,9,2,11,4,13,6,15,8,17,10

Somehow this reminds me of something from years ago, which was using
real IC FIFOs instead of FPGA ones.

Somehow the system wasn't following FULL and ALMOST FULL, and would
wrap the FIFO. But that usually results in data loss, which you
seem to indicate doesn't happen.

I don't know SDRAM timing enough to say. If all the data paths
are 16 bits, it is funny to have an offset on eight bit boundaries!

-- glen

 

[Andy Bennett]

"kaz" wrote in message
news:0p6dneVQkdmemifInZ2dnUU7-b2dnZ2d@giganews.com...

Is the problem that the data is off by 8, or that the data gets stored
in
a location that is off by 8? If it's that the data is off by 8, then
the
number of sdram pages or the sdram muxing is not relevant. What exactly
do
you mean by 'off by 8'? Is data bit 3 in the wrong state? Is it that
data
bit 3 is always 1 when wrong or is it that data bit 3 is wrong, and when
it is wrong that bit might be 1 or it might be 0.

data is 16 bits wide, nothing wrong with bits. all sample values are
correct.
odd samples do not follow their even members e.g. if a correct stream is
indexed as 0,1,2,3,4,5,6,7,8,9,10...etc then what we get is:
0,9,2,11,4,13,6,15,8,17,10

Thus all samples are correct individually. even stream is correct as
0,2,4,6,8... and odd stream is also correct as sequence 9,11,13,15,...etc

but there is this offset where instead of 0,1,2,3.. I get 0,9,2,11...

- Has your design passed static timing analysis?
yes certainly,

- Are all of the I/O and clock frequencies correctly specified to the
timing analysis tool?
yes

- Try warming up the part with a heat gun or cooling it off with cool
spray in the lab. Does the design still work in the lab? If not, you
have a
timing problem.

can't do that in the field, units are concealed mobile radio heads. we
have deployed many thousands of them. only a tiny percentage shows the
issue.

*****************************************************************

Sounds like you have a conflict between your SDRAM column addressing and the
setting of the SDRAM's burst mode.
If your column address does not step in lumps of SDRAM burst addressing then
your output data sequencing can (and will) get screwed.
Make sure you have correctly specified the burst length for your SDRAM
driver and make sure your column address stepping agrees.

Andy

 

[rickman]

On 7/30/2015 5:05 AM, kaz wrote:

In our system a signal is passed through a couple of fifos inside FPGA and
then onto external sdram to be read by application software. All looks ok
except that some units in the field show occasional errors in that signal
read from sdram. The error is as follows: odd samples are offset by 8
samples from the even. So if we remove this offset then signal looks ok.

I can't reproduce the error in the lab. So I depend on some speculations.
It could be the fifos or the sdram. Anyone has come across such issue? my
suspicion is on the sdram as it is configured as 8 pages? Also the sdram
itself has an internal fifo that muxes 128 bits onto 16 (again factor of
8)? any input appreciated.

You haven't said anything about your FPGA design in terms of where you
could lose 8 samples of data or how your data is split between the odd
and even samples. If your FPGA design does not split the data between
odd and even, I'm not sure how you could have this problem.

You also don't mention if the "lost" 8 samples of odd data ever show up
somewhere or if it is just a synchronization problem from the first
sample.

Is there a place in your design where the odd and even samples are
handled separately? Are you using two ADC converters to sample the same
analog at a higher rate, for example?

Do you write the odd/even samples to the SDRAM separately?

--

Rick

 

[kaz]

You haven't said anything about your FPGA design in terms of where you
could lose 8 samples of data or how your data is split between the odd
and even samples. If your FPGA design does not split the data between
odd and even, I'm not sure how you could have this problem.

The 8 samples are not lost but odd substream is offset from the eve
substream regularly.

Inside fpga the data is never split up into odd/even streams. data 16 bi
wide enters a fifo (dc fifo with 16 bits output width). Then into anothe
fifo(sc with output width 32 bits) then back to 16 bits at i/o to sdram.

The two fifos are few words deep but could be a cause in theory i.e. i
fifo ptr toggles between two separate counting sequences. Though Alter
experts looked at them and were happy about the design and we added extr
pipe just in case.

Do you write the odd/even samples to the SDRAM separately?

Rick
no.

The problem is also sometimes self rectifying after some time.

I assume that a glitch in control signals to sdram may change the colum
addressing mechanism as suggested by Andy but the sdram is 8k x128 x 128
8 banks thus 16 bits of data is muxed as 128 bits into each cell!

Ka
--------------------------------------
Posted through http://www.FPGARelated.com

 

[glen herrmannsfeldt]

kaz <37480@fpgarelated> wrote:

I will try to give more details instead of my "reduced simplified version"
and hopefully answer some of your questions.

(snip)

inside FPGA:
TxI and TxQ are first concatenated as(16 x 2 bits), then
passed through a small dc fifo for clock crossing.

How small is this FIFO? (depth x width)

By they way, it is usual to use Gray code when passing the FIFO
address across the clock domain. I think they convert back, but
maybe just address the BRAM with Gray code.

-- glen

 

[rickman]

On 7/30/2015 12:56 PM, kaz wrote:

You haven't said anything about your FPGA design in terms of where you
could lose 8 samples of data or how your data is split between the odd
and even samples. If your FPGA design does not split the data between
odd and even, I'm not sure how you could have this problem.


The 8 samples are not lost but odd substream is offset from the even
substream regularly.

Inside fpga the data is never split up into odd/even streams. data 16 bit
wide enters a fifo (dc fifo with 16 bits output width). Then into another
fifo(sc with output width 32 bits) then back to 16 bits at i/o to sdram.

How large is a "sample"?

The two fifos are few words deep but could be a cause in theory i.e. if
fifo ptr toggles between two separate counting sequences. Though Altera
experts looked at them and were happy about the design and we added extra
pipe just in case.

???

The two FIFOs are sequential, not parallel, right? So how would the
cause a shift in the odd/even data? Do the FIFOs use block RAM? I
don't recall Altera having distributed memory so I guess block RAM is
the only thing available. That means the FIFO memory is one block of
memory unless you have fairly large FIFOs. Is any of this right?

Do you write the odd/even samples to the SDRAM separately?

Rick
no.

The problem is also sometimes self rectifying after some time.

I assume that a glitch in control signals to sdram may change the column
addressing mechanism as suggested by Andy but the sdram is 8k x128 x 128 x
8 banks thus 16 bits of data is muxed as 128 bits into each cell!

Not following this well. I think you are simply saying that the
internal writes in the SDRAM are 128 bits so your 16 bit samples(?) are
written 8 at a time. Unless you have some separation of odd/even
samples I don't see how that would matter.

How do you have your burst addressing set? There are different modes
with different addressing. Only one is sequential. It has been too
long since I've worked with SDRAM and I don't recall what that is all
about. If this is the issue, it won't reproduce the symptoms as you
have described. I believe you say that at the beginning of the fault 8
odd samples are dropped leaving the rest of the sequence out of
alignment with the even samples. If they aren't dropped, where do they
show up? With a burst addressing error the samples would be moved
about, scrambled in some way, but not lost at all.

When the unit "recovers", where does the extra data come from? Are 8
odd samples repeated?

If you can figure out more details of the glitch at the beginning and
end of the error sequence it might help explain where the problem is.

--

Rick

 

[kaz]

I will try to give more details instead of my "reduced simplified version
and hopefully answer some of your questions.

I am talking about a DPD functionality where software reads from sdra
2,457,600 samples of each of TxI,TxQ,sRxI,sRxQ.
all these four slots are 16 bits signed and interleaved in above orde
giving a total stream size of 2,457,600 x 4 samples.

inside FPGA:
TxI and TxQ are first concatenated as(16 x 2 bits), then passed through
small dc fifo for clock crossing.

sRxI and sRxQ are data received from Tx after going through DAC & PA the
sampled back by an ADC for DPD algorithm. sRxI and sRxQ are als
concatenated as 16 x 2 bits. They also go through their dc fifo for cloc
crossing.

Then all four data are concatenated as 16 x 4 = 64 bits.

The stream is then passed as 128 bits using sc fifo for sdram controlle
IF (Altera sdram controller). At the i/o data is passed as two stream
each 16 bits and each has its own sdram. Thus we have two sdrams (one fo
Tx data and one for sRx data)

Almost all field units work without any problem. Occasionally, it i
reported that DPD algorithm fails and when I looked at captured files
noticed that sRx data was ok but TxI and TxQ each shows same problem
described where their odd samples had shifted location relative to eve
ones. So instead of the normal order of 0,1,2,3,4,...etc. I noticed it wa
0,9,2,11,4,13,6,15,8,... from beginning to the end of 2,456,7600

Apart from that there is no other error and all values are correct judgin
by spectrum and time domain.

What happens at the moment of the glitch we don't know, I haven't teste
any failed units in the lab though I requested that. We have inserted som
extra logic to capture data directly from fifos in case of the event bu
we failed to reproduce the error. Units are in different countries and i
is hard to keep track of debugging.

My first conclusion is that there must be memory involved and it must be
case of read/write toggling. The basic fpga concatenation logic does no
involve storage and so is ruled out. FPGA fifos are block ram based and w
have hundreds of them all across the design for various parts withou
issues.
sdram controller and i/o timing have been done by Altera experts.

Design is timing clean, lab tested across full range of temperature.

Kaz




--------------------------------------
Posted through http://www.FPGARelated.com

 

[rickman]

On 7/30/2015 4:24 PM, kaz wrote:

I will try to give more details instead of my "reduced simplified version"
and hopefully answer some of your questions.

I am talking about a DPD functionality where software reads from sdram
2,457,600 samples of each of TxI,TxQ,sRxI,sRxQ.
all these four slots are 16 bits signed and interleaved in above order
giving a total stream size of 2,457,600 x 4 samples.

inside FPGA:
TxI and TxQ are first concatenated as(16 x 2 bits), then passed through a
small dc fifo for clock crossing.

Should I assume DC means "dual clock"? So this FIFO is 32 bits wide?

sRxI and sRxQ are data received from Tx after going through DAC & PA then
sampled back by an ADC for DPD algorithm. sRxI and sRxQ are also
concatenated as 16 x 2 bits. They also go through their dc fifo for clock
crossing.

Then all four data are concatenated as 16 x 4 = 64 bits.

The stream is then passed as 128 bits using sc fifo for sdram controller
IF (Altera sdram controller). At the i/o data is passed as two streams
each 16 bits and each has its own sdram. Thus we have two sdrams (one for
Tx data and one for sRx data)

I don't find this part clear at all. Above you say the data stream is
64 bits, then 128, then two streams of 16 bits. So the data is packed
with one sample of each of the four data streams (TxI,TxQ,sRxI,sRxQ) to
make 64 bits, then two words of this are grouped to make 128 bits. But
then it is all broken back down into 16 bit individual samples?

Almost all field units work without any problem. Occasionally, it is
reported that DPD algorithm fails and when I looked at captured files I
noticed that sRx data was ok but TxI and TxQ each shows same problem I
described where their odd samples had shifted location relative to even
ones. So instead of the normal order of 0,1,2,3,4,...etc. I noticed it was
0,9,2,11,4,13,6,15,8,... from beginning to the end of 2,456,7600

So you can't say what happened to samples 1, 3, 5, etc? The data is
being handed to the SDRAM as 16 bit samples, TxI0, TxQ0, TxI1, TxQ1,...?

So when you have the glitch the alignment is shifted for both TxI and
TxQ or just one? If both, that would be 16 samples of 16 bit data, right?

Apart from that there is no other error and all values are correct judging
by spectrum and time domain.

What happens at the moment of the glitch we don't know, I haven't tested
any failed units in the lab though I requested that. We have inserted some
extra logic to capture data directly from fifos in case of the event but
we failed to reproduce the error. Units are in different countries and it
is hard to keep track of debugging.

My first conclusion is that there must be memory involved and it must be a
case of read/write toggling. The basic fpga concatenation logic does not
involve storage and so is ruled out. FPGA fifos are block ram based and we
have hundreds of them all across the design for various parts without
issues.
sdram controller and i/o timing have been done by Altera experts.

Design is timing clean, lab tested across full range of temperature.

Kaz

--

Rick

 

[kaz]

inside FPGA:
TxI and TxQ are first concatenated as(16 x 2 bits), then
passed through a small dc fifo for clock crossing.

How small is this FIFO? (depth x width)

By they way, it is usual to use Gray code when passing the FIFO
address across the clock domain. I think they convert back, but
maybe just address the BRAM with Gray code.

-- glen

it is dual clock fifo(368.64Mhz => 245.76MHz, 32 bits wide, 16 words deep
it is altera core, we just write/read under our rate control logi
avoiding empty/full situation


The sRx fifo is 245.76 => 245.76 with same above width/depth

Ka
--------------------------------------
Posted through http://www.FPGARelated.com

 

[kaz]

On 7/30/2015 4:24 PM, kaz wrote:

I don't find this part clear at all. Above you say the data stream is
64 bits, then 128, then two streams of 16 bits. So the data is packed
with one sample of each of the four data streams (TxI,TxQ,sRxI,sRxQ) to
make 64 bits, then two words of this are grouped to make 128 bits. But
then it is all broken back down into 16 bit individual samples?

correct, that is how it is designed (I assume it is to do with SOP
interface)

So you can't say what happened to samples 1, 3, 5, etc? The data is
being handed to the SDRAM as 16 bit samples, TxI0, TxQ0, TxI1, TxQ1,...?

correct

I should correct myself about the offset value, it is 16 samples(not 8) i
the sense of stream index i.e. I get samples in the orde
0,17,2,19,4,21,...etc

So when you have the glitch the alignment is shifted for both TxI and
TxQ or just one? If both, that would be 16 samples of 16 bit data
right?

both I and Q symmetrically, if I reverse the offset of both I get prope
signal. I don't have two captures and I assume the error wraps.

Kaz


--

Rick

--------------------------------------
Posted through http://www.FPGARelated.com

 

[Brian Drummond]

On Thu, 30 Jul 2015 04:05:45 -0500, kaz wrote:

In our system a signal is passed through a couple of fifos inside FPGA
and then onto external sdram to be read by application software. All
looks ok except that some units in the field show occasional errors in
that signal read from sdram. The error is as follows: odd samples are
offset by 8 samples from the even. So if we remove this offset then
signal looks ok.

I can't reproduce the error in the lab. So I depend on some
speculations. It could be the fifos or the sdram. Anyone has come across
such issue? my suspicion is on the sdram as it is configured as 8 pages?
Also the sdram itself has an internal fifo that muxes 128 bits onto 16
(again factor of 8)? any input appreciated.

Focus on reproducing it in the lab - or in simulation.

Xilinx FPGAs have multiple clock modules (DCMs) - you're using Altera so
you'll have to translate terms.

These have ways of generating a derivative clock with adjustable timing
for clock phase adjustment : I have attacked similar problems by setting
up the timing in a software-writable register, running memtests with
every possible phase adjustment and mapping out the valid range of
timings.

If you have one or more of these spare, attach it to the SDRAM clock, and
if you have another, attach it to your incoming data register, or your
SDRAM address bus output, etc...

Now you can run memory tests, stretching the timings until it fails.
Hopefully one of the failure modes (but not more than one) will reproduce
the error you are seeing.

In my case, having found the likely failure mode this way, I was able to
reproduce the effect in simulation, the rest was plain sailing.

Incidentally I also recall a correlation between memory manufacturer and
one failure mechanism : I concluded there was nothing specific wrong with
the memory itself, but some disagreement between it and my RAM interface;
I could make that one go away by specifying another memory. Is there any
such variability in your case?

-- Brian

 

[kaz]

I am still trying to figure out this issue of odd/even offset. M
suspicion has fallen on a dual clock fifo (32 bit wide) because when a
some stage its depth was 8 then odd/even offset was 8 samples. Now it
depth is 16 and odd/even offset is 16.

The next question is why would a fifo behave like that even if clock
change phase or fifo gets empty/full.

The fifo is protected against empty/full preventing read/write. The clock
are asynchronous. It is a straight forward dc fifo with several other lik
it in the design but only this one shows the problem occasionally.

I am planning to use dual port ram instead but wanted to know what ha
gone wrong.

Ka
--------------------------------------
Posted through http://www.FPGARelated.com

 

[KJ]

On Thursday, August 13, 2015 at 9:00:13 AM UTC-4, kaz wrote:

The next question is why would a fifo behave like that even if clocks
change phase or fifo gets empty/full.

If your suspicion is correct, then it is because there is a bug in the dual clock design. I know that sounds trite, but you can't discount the obvious.

The fifo is protected against empty/full preventing read/write. The clocks
are asynchronous. It is a straight forward dc fifo with several other like
it in the design but only this one shows the problem occasionally.

Home grown fifo design?

I am planning to use dual port ram instead but wanted to know what has
gone wrong.

Even if the fifo is not home grown, I would suggest switching to another dual clock fifo design first for the following reasons:
- It's going to be quicker to check this out since in some sense all you have to change is the entity that is being instantiated and possibly renaming parameters and ports
- You get another tidbit of information. If the design still fails in the same way, then it 'could' be that the fifo is OK after all. If the design works, then it 'could' be that you're right about there being a problem in the fifo.

If you go down the 'use dual port ram instead' path instead, this is simply re-inventing the dual clock fifo. Dual clock fifo designs are already based on dual port ram anyway. Going down that road is probably not the best way to go about getting to a solution.

If the problem really is in the fifo, then it would be better to mentally trace it back in order to figure out what you could then instrument. In this particular case what I mean is that from what you describe, it looks like a bit in the read address is perhaps into the wrong state. So accept that as a given and work out what are the implications of that condition? One implication if the address pointer is suddenly wrong might be an unusual change in the number of entries in the fifo (like increasing after a read, or decreasing after a write or just changing by a 'large' amount over a short period). Now add some logic to monitor that condition and bring the results of that monitor out to a pin that you can trigger on. For example, maybe the number of words in the fifo should never change by more than 4 between any two read side clock cycles. So add some code that will detect that condition. Repeat for other conditions that you can come up with.

Kevin Jennings

 

Am Donnerstag, 13. August 2015 21:00:07 UTC+2 schrieb kaz:

The fifo is not home grown. It is altera fifo core. We never discard well
tried cores for home made work.

DC fifo is built by Altera around dual ram but if (as in my case) the
clock rates are predictable then one can control wr/rd pointers each in
their clock domain without having to cross clock domains thus reducing
risk and resource. That is my point and is well known design
recommendation.

The fifo in question is just 32 bit wide dc fifo from altera core with
internal pipe set to 3, rd/wr protected, connected to clk 368.64 at write
side enabled 2/3 and connected to 245.76 on the read side always enabled.
Initially the read
enable is delayed to wait for few words (even though it is protected).

Timing is clean. I imagine the write pointer is working but the read
pointer is toggling between 0 and 15 with two clock delays leading to
samples 0,17,2,19 ...etc. Just a guess.

I have put a ram to capture few data from this fifo in the field when
problem occurs and I am awaiting results.
---------------------------------------
Posted through http://www.FPGARelated.com

I have seen random/strange issues with Altera FIFOs in the past more than once (but still seldom). In that cases I replaced the FIFOs with "home grown" ones and the problems were solved. I never tracked that down in detail (maybe it was related to the clear signal in some way...) and I also cannot remember details (one or two clocks, same or different width in/out), I have used so many FIFOs (mostly successfully) over the years...

I would suggest to look at all inputs and outputs of the FIFO with SignalTap and see if it behaves correctly.

P.S.: Oh, I just read your initial post that you cannot reproduce it in the lab...

P.P.S.: I have googled around to find an old post of me regarding this. I failed. But I found this:
https://www.altera.com/support/support-resources/knowledge-base/solutions/rd11182011_10.html
and this:
https://www.altera.com/support/support-resources/knowledge-base/solutions/rd02232015_507.html
(Maybe there are more...)

So I would suggest to replace the FIFOs with a home grown approach and look if this solves the issues in field...

Regards,

Thomas

www.entner-electronics.com - Home of EEBlaster