design rigor: electronics vs. software

[John Larkin]

On Tue, 14 Jan 2020 22:21:30 -0000 (UTC), Steve Wilson <no@spam.com>
wrote:

John Larkin <jlarkin@highland_atwork_technology.com> wrote:

On Tue, 14 Jan 2020 16:43:56 -0000 (UTC), Steve Wilson <no@spam.com
wrote:

jlarkin@highlandsniptechnology.com wrote:

I do label a lot of nodes, but just the interesting ones, not all.

I need to force myself to check all the named nodes when I copy/paste
bits of a circuit. It duplicates all named nodes, which creates some
interesting shorts.

When you name the nodes, use names made from adjacent components, such
as R1C1, Q1B, U1N, etc.

When you copy and paste, the component reference designations will
change. You can easily find the erroneous node names since they won't
match the adjacent components.

I'd rather use something that describes the signal, not the parts.
Like ADC_IN or something. So the plots make sense and can be used as
illustrations in manuals, for example.

of course. Vin, Vout, Clk, Diff, VCC, etc. These are all good for
external connecting signals.

Or for any node that you want to get nicely labeled on a plot.

But if you want signals internal to a circuit block, you need some way to
identify them. If you leave them unnamed, they will get renumbered every
time you make a change to the circuit. So you cannot use unnamed nodes to
plot waveforms.

Sure you can. Just hit it with the probe. It will plot as V(n008) or
something like that. I know what it means because I just probed it.




--

John Larkin Highland Technology, Inc
picosecond timing precision measurement

jlarkin att highlandtechnology dott com
http://www.highlandtechnology.com

 

Rick C <gnuarm.deletethisbit@gmail.com> wrote in
news:eee13c61-8845-4962-9cc4-b8059253bd0f@googlegroups.com:

I think either, you again don't understand what happened, or you
have simplified your understanding of the accident to "tiles fell
off".

"tiles" were KNOCKED off by hard frozen foam main tank jacket.

That tank insulation, at supersonic velocities of the craft, break
free and slow and impact hard. The shuttle didn't fail, per se, the
kid gloves we should have donned to lift it into space were never
created. It was being tossed up there by a drunken uncle.

NONE "fell off". They were impacted. Then they broke away by
atmosphereic forces at that point In that case, they are fragile,
and there is no second layer beneath. The true failure was in the
tank design. If pieces had no come free from it, no tile would have
been impacted and broken free.

 

[mpm]

On Tuesday, January 14, 2020 at 12:53:12 AM UTC-5, Rick C wrote:
> Except that's not what happened. Go read about it. I get tired of educating you.

It's exactly what happened.

https://www.theglobeandmail.com/report-on-business/flying-in-from-left-field-the-black-swan/article689202/

> I think either, you again don't understand what happened, or you have simplified your understanding of the accident to "tiles fell off". I'll discuss this further with you if you want, but only after you educate yourself with the facts.

I'll pass.

And not what I said at all.
You are letting your lack of reading comprehension skills show again.

 

[Rick C]

On Tuesday, January 14, 2020 at 9:04:18 PM UTC-5, mpm wrote:

On Tuesday, January 14, 2020 at 12:53:12 AM UTC-5, Rick C wrote:
Except that's not what happened. Go read about it. I get tired of educating you.

It's exactly what happened.

https://www.theglobeandmail.com/report-on-business/flying-in-from-left-field-the-black-swan/article689202/

I think either, you again don't understand what happened, or you have simplified your understanding of the accident to "tiles fell off". I'll discuss this further with you if you want, but only after you educate yourself with the facts.

I'll pass.

And not what I said at all.
You are letting your lack of reading comprehension skills show again.

You are not at all interesting to talk to. You distort facts, you believe facts that aren't true and you deny everything that shows this happening. Then on top you resort to crude insults.

If you just deny everything that doesn't fit your preconceptions, you aren't living in reality.

I have no interest in bantering words with you if there is no reality in the mix. In that regard you are like a lot of the dim wits here. I don't converse with them much either.

Bye

--

Rick C.

---- Get 1,000 miles of free Supercharging
---- Tesla referral code - https://ts.la/richard11209

 

[RBlack]

On Tue, 14 Jan 2020 12:35:56 -0500, Phil Hobbs
<pcdhSpamMeSenseless@electrooptical.net> said:
>

[snip]

Client work almost always succeeds, and the occasional failures are
mostly due to the customer's prevarication, such as taking my
proof-of-concept system, giving it to a CE outfit, and then pulling me
back in to attempt to fix the CE's mess--of course at the last minute,
when they've almost run out of money. That's happened a couple of
times, so I try very hard to discourage it. (The two were the
transcutaneous blood glucose/alcohol system and the blood-spot detector

^^^^^^^^^^^^^^ ^^^^^ ^^^^^^^

Bummer, sorry to hear about that. (I have a personal interest in seeing
this type of device working and on the market). Did anyone else pick up
the project, or did it just die?

for hens' eggs.)

Cheers

Phil Hobbs

 

[Martin Brown]

On 13/01/2020 15:58, John Larkin wrote:

On Mon, 13 Jan 2020 09:04:20 +0000, Tom Gardner
spamjunk@blueyonder.co.uk> wrote:

On 13/01/20 01:07, John Larkin wrote:
On Sun, 12 Jan 2020 16:58:40 +0000, Martin Brown
'''newspam'''@nezumi.demon.co.uk> wrote:

On 11/01/2020 14:57, jlarkin@highlandsniptechnology.com wrote:
On 11 Jan 2020 05:57:59 -0800, Winfield Hill <winfieldhill@yahoo.com
wrote:

Rick C wrote...

Then your very example of the Boeing plane is wrong
because no one has said the cause of the accident
was improperly coded software.

Yes, it was an improper spec, with dangerous reliance
on poor hardware.

If code kills people, it was improperly coded.

Not necessarily. The code written may well have exactly implemented the
algorithm(s) that the clowns supervised by monkeys specified. It isn't
the job of programmers to double check the workings of the people who do
the detailed calculations of aerodynamic force vectors and torques.

It is not the programmers fault if the systems engineering, failure
analysis and aerodynamics calculations are incorrect in some way!

The management of two AOA sensors was insane. Fatal, actually. A
programmer should understand simple stuff like that.

The guy doing the MCAS control loop may well have had a specification
which implied that sensor data the module was presented with was
verified and trustworthy. He was also given safe maximum limits for the
jack screw adjustment on the initial specification which were later
changed to dangerous ones after the system had undergone flight tests.

The fault was in higher level systems engineering *NOT* coding.

The area where there is most need for improvement is a sanity check
across all of the sensors to determine if an emergency is real or an
artefact of a bad sensor.

The thing that ultimately made MCAS so lethal was a very simple edit of
a set limit for the maximum jackscrew adjustment from 0.6 degrees to a
whopping 2.5 degrees. That sealed the fate of the unfortunate planes.

MCAS was an awful bodge hidden from view and undocumented in the flight
manual. When things went wrong the pilots didn't stand much chance.

It is unrealistic to expect programmers to understand sensor
reliability. That is the job of the people specifying the
system design and encoding that in the system specification
and the software specification.

Programmers would have zero ability to deviate from implementing
the software spec, full stop. If they did knowingly deviate, it
would be a career ending decision - at best.

Aerospace engineers have lost their pension for far less
serious deviations, even though they had zero consequences.


https://philip.greenspun.com/blog/2019/03/21/optional-angle-of-attack-sensors-on-the-boeing-737-max/

Given dual sensors, why would any sane person decide to alternate
using one per flight?

There was an optional extra; not present on either of the crashed
flights to have a light on the dashboard for "AoA sensors disagree".

https://www.flightglobal.com/safety/boeing-bosses-unaware-of-737-max-aoa-glitch-before-lion-crash/132555.article

Unfortunately they somehow broke it and it wasn't picked up (or was
quietly brushed under the carpet to ship product on time).

A programmer would have to be awfully thick to not object to that.

We don't know that they didn't but were told to STFU and keep coding -
we are behind on this project already - cut every possible corner.

--
Regards,
Martin Brown

 

Martin Brown <'''newspam'''@nezumi.demon.co.uk> wrote in news:qvmkdq
$g9d$1@gioia.aioe.org:

There was an optional extra; not present on either of the crashed
flights to have a light on the dashboard for "AoA sensors
disagree".

Why not a nice BIG INDICATOR on the dash! Damn!

A pilot *should* know if they are heading into a stall by feel from
experience, ESPECIALLY on a commercial flight which almost never ends
up in such an AoA.

Not only should they agree, but they need to be built to fix
themselves from being frozen. Full range of motion actuators
*inside* the sensor housing along with the sensor read opto slot
wheels, and maybe something to keep the bearings warm. BOTH sensors
should ALWAYS agree, and if they do not, then the sensors get cycled
through the RoM (range of motion)test and released, then the other,
all the while alerting the cockpit. The whole reset/reread funtion
shouldn't take more than a few seconds each, so 5 or 10 seconds
across both.

 

Martin Brown <'''newspam'''@nezumi.demon.co.uk> wrote in
news:qvmkdq$g9d$1@gioia.aioe.org:

We don't know that they didn't but were told to STFU and keep
coding - we are behind on this project already - cut every
possible corner.

Reliabilty and proofing tests are *NOT* on the 'possible' list.

 

[Phil Hobbs]

On 2020-01-15 03:17, RBlack wrote:

On Tue, 14 Jan 2020 12:35:56 -0500, Phil Hobbs
pcdhSpamMeSenseless@electrooptical.net> said:


[snip]

Client work almost always succeeds, and the occasional failures are
mostly due to the customer's prevarication, such as taking my
proof-of-concept system, giving it to a CE outfit, and then pulling me
back in to attempt to fix the CE's mess--of course at the last minute,
when they've almost run out of money. That's happened a couple of
times, so I try very hard to discourage it. (The two were the
transcutaneous blood glucose/alcohol system and the blood-spot detector

^^^^^^^^^^^^^^ ^^^^^ ^^^^^^^

Bummer, sorry to hear about that. (I have a personal interest in seeing
this type of device working and on the market). Did anyone else pick up
the project, or did it just die?

for hens' eggs.)

Yeah, that was a sad one, for sure. It remains vivid in my memory, so
at the risk of boring people, here's the tale.

The founder called me out of the blue at 3 PM on Christmas Eve, 2012.
He turned out to be a charming fellow with a lot of drive and not a lot
of education, who was practically supernatural at raising money. He
wanted me to build him an instrument, because that's what I do.

He'd patented the general principle, which avoided the individual
physiological variations that usually bedevil those sorts of measurements.

The idea was to use a hand cradle with a virtual pivot (*) holding a
fibre bundle against the web of the first and second fingers. There are
two arteries very close to the surface there, so you get to measure
fresh blood instead of tissue fluid, and no one has hair, fat, or
callouses there, so the physiology is very cooperative. He had some
promising data that he took himself using a Perkin-Elmer FTIR and his
hand cradle. (He somehow talked some guys at USC into giving him lab
space and some help doing that--he was entirely self-taught, which has
its strengths and weaknesses.)

The project was an unusual one for me, in that I didn't have my arms
around the whole measurement--I built the gizmo, but the founder had his
USC statistics friends use their AI nous to build the model and extract
the blood solute data. Thus I don't actually know how that was done.
(It wasn't something simple like spectral differencing, anyway.)

I did a photon budget, which is my term for a detailed feasibility
calculation emphasizing stability and SNR. That's super important
because without calculating how good the measurement _could_ be, you
never really know how you're doing. A photon budget prevents you from
wasting time on recreational impossibilities on the one hand, or turning
a silk purse back into a sow's ear on the other. In this case it looked
pretty good, using a tungsten source, a custom-designed split bundle of
about 20 fibres (TX + RX), a conventional Czerny-Turner monochromator, a
single extended-InGaAs photodiode, and a chopper plus lock-in for
detection.

The proof-of-concept system took me about six weeks start to finish,
including the photon budget, optical design, designing and building the
electronics, assembling the optomechanics, and writing the software. It
was built on a 12 x 24-inch aluminum breadboard using a combination of
hhacked Microbench (**) parts, JB Weld (the poor man's machine shop),
and a servo from an RC airplane for moving the grating. The grating
cradle was also built from toy airplane parts, courtesy of
servocity.com. The electronics was all built in die cast aluminum stomp
boxes, dead bug style, wired up with BNC cables. The chopper was a
commercial (Thor Labs) unit, and the back end was a LabJack and a
console-mode C++ program running on a second-hand laptop.

It all worked great, and was very amusing to watch--an advanced clinical
instrument built out of JB Weld and toy parts. (Wouldn't the FDA have
loved that?)

We did the preliminary acceptance test by having some friends over for
cocktails and measuring their blood spectra every 15 minutes or so. The
data did exactly as we hoped--nice repeatable curves with the right time
dependence and no big physiological variations between subjects. We did
some glucose work using a strip reader for comparison, but the strips
have relatively poor accuracy, so we concentrated on the alcohol
measurement for that part of the demo. (Quaffing a few cool ones is
much more fun than sticking pins in your fingers, too.)

After the founder used the POC data to raise a bunch more money, we
brought the proto and the Perkin-Elmer FTIR to a contract engineering
house in Orange County CA that will remain nameless because they have
this unfortunate tendency to sue everybody in sight. The founder kept
me sort of distantly in the loop, but his crucial mistake was trying to
save money by supervising the CE firm himself. (I tried to help, but
they ignored me almost entirely because I wasn't the one writing the
cheques.)

The CE folks proceeded to fall into every pothole along the road, like a
drunk. They ignored the photon budget, and so redesigned the front end
to use an ordinary op amp TIA, not realizing they lost about 15 dB of
SNR in the process. (I managed to get that one fixed, and the guy
responsible taken off the project. Unfortunately they were almost all
like that.)

They replaced the direct drive for the grating with a belt drive, which
gave nice smooth motion but of course rapidly lost accuracy as the belt
squirmed around while moving, so that the calibration wouldn't sit
still. They needed more encoder precision, so they put an encoder on
the motor as well as the grating, and did some micky-mouse trick to
combine the two encoder readings. Even the magnetic encoder on the
grating shaft refused to sit still--it drifted around like mad. I went
there to try to get to the bottom of some of this stuff, and although
they mostly sidelined me because I wasn't writing the cheques, we did
manage to get to the bottom of that one. Turned out that the encoder's
output was the duty cycle of a pulse train (like the RC servo only
backwards), and they were measuring the pulse width instead, using a
capture input of their MCU. That turned the frequency drift into a
position drift. Once it was fixed, I hit that poor encoder with cold
spray and a heat gun, and couldn't get it to move at all. Kudos to US
Digital for that one.

The belt-drive system failed nonetheless, basically because the
measurement was being done on the slope of the very strong IR absorption
spectrum of water in the 1.4-1.7 micron range, so that small wavelength
shifts caused much larger amplitude errors.

I told them to use a sine-bar drive like every other Czerny-Turner
monochromator in the world, but they refused, insisting on using a
worm-and-sector gear instead, with the encoder on the worm shaft.
Unlike nearly any other sort of gearing, worms work using sliding
friction. That makes the grease film thin out with time. I calculated
for their benefit that in their design, with the very small radius of
the sector gear, the maximum lubricant variation we could tolerate was
about 70 nanometres. Since they were nearly finished with the prototype
build for the formal clinical trial, I told them to use dry molybdenum
disulphide for lubricant instead of grease.

They straight-up refused, saying they couldn't get MoS2, so I sent them
a link to the exact SKU on fastenall.com, after verifying that their
local Fastenall had it in stock. (I even sent them a Mapquest link so
they could find the store. That was a bit sarcastic, which I regret,
but I was getting pretty tired of them by that point.) They proceeded
to ship one unit with grease and three units unlubricated.

When I complained about all the directionless fiddling they were doing,
one bright lad smiled and said brightly, "That's engineering!" He was
one of the better ones.

They also took the POC proto apart to use bits of it in their test
system, so that they had no comparison data, and, oh, yes, they broke
the $100k FTIR and didn't tell anyone.

The units failed the acceptance test. I attended it, but since the USC
folks weren't crunching the data in real time, the failure wasn't
entirely apparent till later. By that point the CE had run through a
year's time and most of a million bucks.

Some months later, two units arrived on my bench, along with an urgent
request for me to get to the bottom of it all.

Turned out to be a real onion problem--you know, peel off one layer,
cry, and peel off the next.

The phase of the detected signal was moving around by 10 degrees or so,
which was easily enough to destroy the measurement. The code seemed to
be a PID controller using an optointerrupter on the chopper wheel, which
should have been fine. I built a strobe light using an HP 3325A
synthesizer driving a LED, so that I could see the loop dynamics. The
controller was totally broken--regardless of the settings of P, I, and
D, there was no way of making the phase sit still. A gentle stream of
canned air would move the phase, and it would never recover--i.e. there
was no integral term in the control law, despite what the settings would
have one believe.

And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform by
least-squares curve fitting to a sine wave instead of using an FFT like
normal people. Everybody screws up their first digital lock-in, but
I've never seen one as bad as that. (For non signals-and-systems folks:
least squares fitting works OK at high SNR, but for noisy data it falls
apart completely. The FFT uses orthogonality, and so works at any SNR.)

I didn't get to the last onion layer, because the founder ran out of
both dough and friends. He never did pay me for my last month's work.

A pity. I would have made those boxes work eventually.


(*) One where the business end slides around on a curved surface, like
the blade assembly on some razors.
(**) A cage system using plates held together with 6-mm
centreless-ground stainless steel rods, similar to the Thor Labs 30-mm
cage system.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC / Hobbs ElectroOptics
Optics, Electro-optics, Photonics, Analog Electronics
Briarcliff Manor NY 10510

http://electrooptical.net
http://hobbs-eo.com

 

[Steve Wilson]

On Wednesday, January 15, 2020 at 7:29:08 AM UTC-5, Phil Hobbs wrote:
[...]

And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform by
least-squares curve fitting to a sine wave instead of using an FFT like
normal people. Everybody screws up their first digital lock-in, but
I've never seen one as bad as that. (For non signals-and-systems folks:
least squares fitting works OK at high SNR, but for noisy data it falls
apart completely. The FFT uses orthogonality, and so works at any SNR.)

I didn't get to the last onion layer, because the founder ran out of
both dough and friends. He never did pay me for my last month's work.

A pity. I would have made those boxes work eventually.

(*) One where the business end slides around on a curved surface, like
the blade assembly on some razors.
(**) A cage system using plates held together with 6-mm
centreless-ground stainless steel rods, similar to the Thor Labs 30-mm
cage system.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC / Hobbs ElectroOptics
Optics, Electro-optics, Photonics, Analog Electronics
Briarcliff Manor NY 10510

http://electrooptical.net
http://hobbs-eo.com

That is an amazing story. Thanks.

Was there no way you contact the owner and warn him they were ruining the design? How did they get authorization to redesign the product in the first place?

How do we prevent this from happening to us?

 

[Bill Sloman]

On Sunday, January 12, 2020 at 1:47:19 AM UTC+11, jla...@highlandsniptechnology.com wrote:

On Fri, 10 Jan 2020 21:46:19 -0800 (PST), omnilobe@gmail.com wrote:

Hardware designs are more rigorously done than
software designs. A large company had problems with a 737
and a rocket to the space station...

https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers

I know programmers who do not care for rigor at home at work.
I did hardware design with rigor and featuring reviews by caring
electronics design engineers and marketing engineers.

Software gets sloppy with OOPs.
Object Oriented Programming.
Windows 10 on a rocket to ISS space station.
C++ mud.

The easier it is to change things, the less careful people are about
doing them. Software, which includes FPGA code, seldom works the first
time. Almost never. The average hunk of fresh code has a mistake
roughly every 10 lines. Or was that three?

Thirty lines was the figure I saw.

Good software practice is to keep your procedures shorter than a page and get somebody else to run through it before you file it.

FPGAs are usually better than procedural code, but are still mostly
done as hack-and-fix cycles, with software test benches. When we did
OTP (fuse based) FPGAs without test benching, we often got them right
first try. If compiles took longer, people would be more careful.

Engineers make mistakes. The art is in finding them early.

PCBs usually work the first time, because they are checked and
reviewed, and that is because mistakes are slow and expensive to fix,
and very visible to everyone. Bridges and buildings are almost always
right the first time. They are even more expensive and slow and
visible.

And get reviewed and checked even more carefully.

Besides, electronics and structures have established theory, but
software doesn't. Various people just sort of do it.

Software is just mathematical procedures. There is a branch of software theory devoted to letting you write provably correct code, but it doesn't to work well enough in practice to be worth the trouble.

https://dl.acm.org/doi/10.1007/978-3-642-34026-0_2

My Spice sims are often wrong initially, precisely because there are
basically no consequences to running the first try without much
checking. That is of course dangerous; we don't want to base a
hardware design on a sim that runs and makes pretty graphs but is
fundamentally wrong.

Obviously not. My example of that is the Spice simulation of a Baxandall Class-D oscillator with bipolar transistors and a big choke.

The circuit "squegs" in real life, but not in simulation, or at least not with a Gummell-Poon transistor model, which is known to do a bad job of simulating a inverted transistor. The VBIC model might do better, but transistor manufacturers treat them as "commercial in confidence" and I've never got my hand on one to try it.

http://www.sophia-electronica.com/Baxandall_parallel-resonant_Class-D_oscillator1.htm

--
Bill Sloman, Sydney

 

[Phil Hobbs]

On 2020-01-15 08:41, Steve Wilson wrote:

On Wednesday, January 15, 2020 at 7:29:08 AM UTC-5, Phil Hobbs wrote:
[...]


And then it turned out that they'd never built a lock-in before,
and were trying to extract the (approximately triangular) signal
waveform by least-squares curve fitting to a sine wave instead of
using an FFT like normal people. Everybody screws up their first
digital lock-in, but I've never seen one as bad as that. (For non
signals-and-systems folks: least squares fitting works OK at high
SNR, but for noisy data it falls apart completely. The FFT uses
orthogonality, and so works at any SNR.)

I didn't get to the last onion layer, because the founder ran out
of both dough and friends. He never did pay me for my last
month's work.

A pity. I would have made those boxes work eventually.

(*) One where the business end slides around on a curved surface,
like the blade assembly on some razors. (**) A cage system using
plates held together with 6-mm centreless-ground stainless steel
rods, similar to the Thor Labs 30-mm cage system.

Cheers

Phil Hobbs


That is an amazing story. Thanks.

Was there no way you contact the owner and warn him they were
ruining the design?

Oh, I told him, all right, and he told them. They did fix a few things,
but mostly said "yes" and meant "no". Since I wasn't supervising them,
they didn't keep me in the loop with what they were doing to fix the
problems.

How did they get authorization to redesign the product in the first
place?

Well, as I said, the FDA wouldn't have liked a system made of toy parts,
so the optomechanics did need re-doing. That part they actually did
reasonably well, apart from the grating mechanism and the chopper servo.

The electronics they just went ahead and re-did, I think in order to
reuse some previous design. Digital lock-ins are difficult because you
have to be totally paranoid about things like slew artifacts, settling
time, input voltage sag during sampling, and anything getting in on the
reference. The frequency domain is brutal that way, and AFAICT nobody
ever learns that except by way of at least one failure.

> How do we prevent this from happening to us?

Stay out of Orange County.

Seriously, having a sharp technical person supervising, with a formal
spec, design reviews, sign-offs on hardware and software, unit tests,
and so on would have prevented it. It would have been half again as
expensive, but the system would have worked. Checking the CE's
references would have been a smart move too. They claimed that their
contracts mostly forbade them to tell who their customers were, and the
founder fell for that one.

Cheers

Phil Hobbs


--
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC / Hobbs ElectroOptics
Optics, Electro-optics, Photonics, Analog Electronics
Briarcliff Manor NY 10510

http://electrooptical.net
http://hobbs-eo.com

 

On Wed, 15 Jan 2020 07:28:58 -0500, Phil Hobbs
<pcdhSpamMeSenseless@electrooptical.net> wrote:

On 2020-01-15 03:17, RBlack wrote:
On Tue, 14 Jan 2020 12:35:56 -0500, Phil Hobbs
pcdhSpamMeSenseless@electrooptical.net> said:


[snip]

Client work almost always succeeds, and the occasional failures are
mostly due to the customer's prevarication, such as taking my
proof-of-concept system, giving it to a CE outfit, and then pulling me
back in to attempt to fix the CE's mess--of course at the last minute,
when they've almost run out of money. That's happened a couple of
times, so I try very hard to discourage it. (The two were the
transcutaneous blood glucose/alcohol system and the blood-spot detector

^^^^^^^^^^^^^^ ^^^^^ ^^^^^^^

Bummer, sorry to hear about that. (I have a personal interest in seeing
this type of device working and on the market). Did anyone else pick up
the project, or did it just die?

for hens' eggs.)


Yeah, that was a sad one, for sure. It remains vivid in my memory, so
at the risk of boring people, here's the tale.

The founder called me out of the blue at 3 PM on Christmas Eve, 2012.
He turned out to be a charming fellow with a lot of drive and not a lot
of education, who was practically supernatural at raising money. He
wanted me to build him an instrument, because that's what I do.

He'd patented the general principle, which avoided the individual
physiological variations that usually bedevil those sorts of measurements.

The idea was to use a hand cradle with a virtual pivot (*) holding a
fibre bundle against the web of the first and second fingers. There are
two arteries very close to the surface there, so you get to measure
fresh blood instead of tissue fluid, and no one has hair, fat, or
callouses there, so the physiology is very cooperative. He had some
promising data that he took himself using a Perkin-Elmer FTIR and his
hand cradle. (He somehow talked some guys at USC into giving him lab
space and some help doing that--he was entirely self-taught, which has
its strengths and weaknesses.)

The project was an unusual one for me, in that I didn't have my arms
around the whole measurement--I built the gizmo, but the founder had his
USC statistics friends use their AI nous to build the model and extract
the blood solute data. Thus I don't actually know how that was done.
(It wasn't something simple like spectral differencing, anyway.)

I did a photon budget, which is my term for a detailed feasibility
calculation emphasizing stability and SNR. That's super important
because without calculating how good the measurement _could_ be, you
never really know how you're doing. A photon budget prevents you from
wasting time on recreational impossibilities on the one hand, or turning
a silk purse back into a sow's ear on the other. In this case it looked
pretty good, using a tungsten source, a custom-designed split bundle of
about 20 fibres (TX + RX), a conventional Czerny-Turner monochromator, a
single extended-InGaAs photodiode, and a chopper plus lock-in for
detection.

The proof-of-concept system took me about six weeks start to finish,
including the photon budget, optical design, designing and building the
electronics, assembling the optomechanics, and writing the software. It
was built on a 12 x 24-inch aluminum breadboard using a combination of
hhacked Microbench (**) parts, JB Weld (the poor man's machine shop),
and a servo from an RC airplane for moving the grating. The grating
cradle was also built from toy airplane parts, courtesy of
servocity.com. The electronics was all built in die cast aluminum stomp
boxes, dead bug style, wired up with BNC cables. The chopper was a
commercial (Thor Labs) unit, and the back end was a LabJack and a
console-mode C++ program running on a second-hand laptop.

It all worked great, and was very amusing to watch--an advanced clinical
instrument built out of JB Weld and toy parts. (Wouldn't the FDA have
loved that?)

We did the preliminary acceptance test by having some friends over for
cocktails and measuring their blood spectra every 15 minutes or so. The
data did exactly as we hoped--nice repeatable curves with the right time
dependence and no big physiological variations between subjects. We did
some glucose work using a strip reader for comparison, but the strips
have relatively poor accuracy, so we concentrated on the alcohol
measurement for that part of the demo. (Quaffing a few cool ones is
much more fun than sticking pins in your fingers, too.)

After the founder used the POC data to raise a bunch more money, we
brought the proto and the Perkin-Elmer FTIR to a contract engineering
house in Orange County CA that will remain nameless because they have
this unfortunate tendency to sue everybody in sight. The founder kept
me sort of distantly in the loop, but his crucial mistake was trying to
save money by supervising the CE firm himself. (I tried to help, but
they ignored me almost entirely because I wasn't the one writing the
cheques.)

The CE folks proceeded to fall into every pothole along the road, like a
drunk. They ignored the photon budget, and so redesigned the front end
to use an ordinary op amp TIA, not realizing they lost about 15 dB of
SNR in the process. (I managed to get that one fixed, and the guy
responsible taken off the project. Unfortunately they were almost all
like that.)

They replaced the direct drive for the grating with a belt drive, which
gave nice smooth motion but of course rapidly lost accuracy as the belt
squirmed around while moving, so that the calibration wouldn't sit
still. They needed more encoder precision, so they put an encoder on
the motor as well as the grating, and did some micky-mouse trick to
combine the two encoder readings. Even the magnetic encoder on the
grating shaft refused to sit still--it drifted around like mad. I went
there to try to get to the bottom of some of this stuff, and although
they mostly sidelined me because I wasn't writing the cheques, we did
manage to get to the bottom of that one. Turned out that the encoder's
output was the duty cycle of a pulse train (like the RC servo only
backwards), and they were measuring the pulse width instead, using a
capture input of their MCU. That turned the frequency drift into a
position drift. Once it was fixed, I hit that poor encoder with cold
spray and a heat gun, and couldn't get it to move at all. Kudos to US
Digital for that one.

The belt-drive system failed nonetheless, basically because the
measurement was being done on the slope of the very strong IR absorption
spectrum of water in the 1.4-1.7 micron range, so that small wavelength
shifts caused much larger amplitude errors.

I told them to use a sine-bar drive like every other Czerny-Turner
monochromator in the world, but they refused, insisting on using a
worm-and-sector gear instead, with the encoder on the worm shaft.
Unlike nearly any other sort of gearing, worms work using sliding
friction. That makes the grease film thin out with time. I calculated
for their benefit that in their design, with the very small radius of
the sector gear, the maximum lubricant variation we could tolerate was
about 70 nanometres. Since they were nearly finished with the prototype
build for the formal clinical trial, I told them to use dry molybdenum
disulphide for lubricant instead of grease.

They straight-up refused, saying they couldn't get MoS2, so I sent them
a link to the exact SKU on fastenall.com, after verifying that their
local Fastenall had it in stock. (I even sent them a Mapquest link so
they could find the store. That was a bit sarcastic, which I regret,
but I was getting pretty tired of them by that point.) They proceeded
to ship one unit with grease and three units unlubricated.

When I complained about all the directionless fiddling they were doing,
one bright lad smiled and said brightly, "That's engineering!" He was
one of the better ones.

They also took the POC proto apart to use bits of it in their test
system, so that they had no comparison data, and, oh, yes, they broke
the $100k FTIR and didn't tell anyone.

The units failed the acceptance test. I attended it, but since the USC
folks weren't crunching the data in real time, the failure wasn't
entirely apparent till later. By that point the CE had run through a
year's time and most of a million bucks.

Some months later, two units arrived on my bench, along with an urgent
request for me to get to the bottom of it all.

Turned out to be a real onion problem--you know, peel off one layer,
cry, and peel off the next.

The phase of the detected signal was moving around by 10 degrees or so,
which was easily enough to destroy the measurement. The code seemed to
be a PID controller using an optointerrupter on the chopper wheel, which
should have been fine. I built a strobe light using an HP 3325A
synthesizer driving a LED, so that I could see the loop dynamics. The
controller was totally broken--regardless of the settings of P, I, and
D, there was no way of making the phase sit still. A gentle stream of
canned air would move the phase, and it would never recover--i.e. there
was no integral term in the control law, despite what the settings would
have one believe.

And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform by
least-squares curve fitting to a sine wave instead of using an FFT like
normal people. Everybody screws up their first digital lock-in, but
I've never seen one as bad as that. (For non signals-and-systems folks:
least squares fitting works OK at high SNR, but for noisy data it falls
apart completely. The FFT uses orthogonality, and so works at any SNR.)

I didn't get to the last onion layer, because the founder ran out of
both dough and friends. He never did pay me for my last month's work.

A pity. I would have made those boxes work eventually.


(*) One where the business end slides around on a curved surface, like
the blade assembly on some razors.
(**) A cage system using plates held together with 6-mm
centreless-ground stainless steel rods, similar to the Thor Labs 30-mm
cage system.

Cheers

Phil Hobbs

I want a wideband optical spectrum analyzer, say 300 to 1600 nm, for
checking LEDs and IR fiber lasers and generally knowing where we are.
It doesn't need to be very sensitive or terribly accurate. Nobody
makes one, and there should be a market, and it sounds a lot easier
than that project.



--

John Larkin Highland Technology, Inc

The cork popped merrily, and Lord Peter rose to his feet.
"Bunter", he said, "I give you a toast. The triumph of Instinct over Reason"

 

On Wed, 15 Jan 2020 09:01:46 +0000, Martin Brown
<'''newspam'''@nezumi.demon.co.uk> wrote:

On 13/01/2020 15:58, John Larkin wrote:
On Mon, 13 Jan 2020 09:04:20 +0000, Tom Gardner
spamjunk@blueyonder.co.uk> wrote:

On 13/01/20 01:07, John Larkin wrote:
On Sun, 12 Jan 2020 16:58:40 +0000, Martin Brown
'''newspam'''@nezumi.demon.co.uk> wrote:

On 11/01/2020 14:57, jlarkin@highlandsniptechnology.com wrote:
On 11 Jan 2020 05:57:59 -0800, Winfield Hill <winfieldhill@yahoo.com
wrote:

Rick C wrote...

Then your very example of the Boeing plane is wrong
because no one has said the cause of the accident
was improperly coded software.

Yes, it was an improper spec, with dangerous reliance
on poor hardware.

If code kills people, it was improperly coded.

Not necessarily. The code written may well have exactly implemented the
algorithm(s) that the clowns supervised by monkeys specified. It isn't
the job of programmers to double check the workings of the people who do
the detailed calculations of aerodynamic force vectors and torques.

It is not the programmers fault if the systems engineering, failure
analysis and aerodynamics calculations are incorrect in some way!

The management of two AOA sensors was insane. Fatal, actually. A
programmer should understand simple stuff like that.

The guy doing the MCAS control loop may well have had a specification
which implied that sensor data the module was presented with was
verified and trustworthy. He was also given safe maximum limits for the
jack screw adjustment on the initial specification which were later
changed to dangerous ones after the system had undergone flight tests.

The fault was in higher level systems engineering *NOT* coding.

Sure, why would a code monkey at Boeing want to know anything about
airplanes?

As that CS genius Joe Biden said recently, anybody can learn to code.



--

John Larkin Highland Technology, Inc

The cork popped merrily, and Lord Peter rose to his feet.
"Bunter", he said, "I give you a toast. The triumph of Instinct over Reason"

 

[Phil Hobbs]

On 2020-01-15 10:46, jlarkin@highlandsniptechnology.com wrote:

On Wed, 15 Jan 2020 07:28:58 -0500, Phil Hobbs
pcdhSpamMeSenseless@electrooptical.net> wrote:

On 2020-01-15 03:17, RBlack wrote:
On Tue, 14 Jan 2020 12:35:56 -0500, Phil Hobbs
pcdhSpamMeSenseless@electrooptical.net> said:


[snip]

Client work almost always succeeds, and the occasional failures are
mostly due to the customer's prevarication, such as taking my
proof-of-concept system, giving it to a CE outfit, and then pulling me
back in to attempt to fix the CE's mess--of course at the last minute,
when they've almost run out of money. That's happened a couple of
times, so I try very hard to discourage it. (The two were the
transcutaneous blood glucose/alcohol system and the blood-spot detector

^^^^^^^^^^^^^^ ^^^^^ ^^^^^^^

Bummer, sorry to hear about that. (I have a personal interest in seeing
this type of device working and on the market). Did anyone else pick up
the project, or did it just die?

for hens' eggs.)


Yeah, that was a sad one, for sure. It remains vivid in my memory, so
at the risk of boring people, here's the tale.

The founder called me out of the blue at 3 PM on Christmas Eve, 2012.
He turned out to be a charming fellow with a lot of drive and not a lot
of education, who was practically supernatural at raising money. He
wanted me to build him an instrument, because that's what I do.

He'd patented the general principle, which avoided the individual
physiological variations that usually bedevil those sorts of measurements.

The idea was to use a hand cradle with a virtual pivot (*) holding a
fibre bundle against the web of the first and second fingers. There are
two arteries very close to the surface there, so you get to measure
fresh blood instead of tissue fluid, and no one has hair, fat, or
callouses there, so the physiology is very cooperative. He had some
promising data that he took himself using a Perkin-Elmer FTIR and his
hand cradle. (He somehow talked some guys at USC into giving him lab
space and some help doing that--he was entirely self-taught, which has
its strengths and weaknesses.)

The project was an unusual one for me, in that I didn't have my arms
around the whole measurement--I built the gizmo, but the founder had his
USC statistics friends use their AI nous to build the model and extract
the blood solute data. Thus I don't actually know how that was done.
(It wasn't something simple like spectral differencing, anyway.)

I did a photon budget, which is my term for a detailed feasibility
calculation emphasizing stability and SNR. That's super important
because without calculating how good the measurement _could_ be, you
never really know how you're doing. A photon budget prevents you from
wasting time on recreational impossibilities on the one hand, or turning
a silk purse back into a sow's ear on the other. In this case it looked
pretty good, using a tungsten source, a custom-designed split bundle of
about 20 fibres (TX + RX), a conventional Czerny-Turner monochromator, a
single extended-InGaAs photodiode, and a chopper plus lock-in for
detection.

The proof-of-concept system took me about six weeks start to finish,
including the photon budget, optical design, designing and building the
electronics, assembling the optomechanics, and writing the software. It
was built on a 12 x 24-inch aluminum breadboard using a combination of
hhacked Microbench (**) parts, JB Weld (the poor man's machine shop),
and a servo from an RC airplane for moving the grating. The grating
cradle was also built from toy airplane parts, courtesy of
servocity.com. The electronics was all built in die cast aluminum stomp
boxes, dead bug style, wired up with BNC cables. The chopper was a
commercial (Thor Labs) unit, and the back end was a LabJack and a
console-mode C++ program running on a second-hand laptop.

It all worked great, and was very amusing to watch--an advanced clinical
instrument built out of JB Weld and toy parts. (Wouldn't the FDA have
loved that?)

We did the preliminary acceptance test by having some friends over for
cocktails and measuring their blood spectra every 15 minutes or so. The
data did exactly as we hoped--nice repeatable curves with the right time
dependence and no big physiological variations between subjects. We did
some glucose work using a strip reader for comparison, but the strips
have relatively poor accuracy, so we concentrated on the alcohol
measurement for that part of the demo. (Quaffing a few cool ones is
much more fun than sticking pins in your fingers, too.)

After the founder used the POC data to raise a bunch more money, we
brought the proto and the Perkin-Elmer FTIR to a contract engineering
house in Orange County CA that will remain nameless because they have
this unfortunate tendency to sue everybody in sight. The founder kept
me sort of distantly in the loop, but his crucial mistake was trying to
save money by supervising the CE firm himself. (I tried to help, but
they ignored me almost entirely because I wasn't the one writing the
cheques.)

The CE folks proceeded to fall into every pothole along the road, like a
drunk. They ignored the photon budget, and so redesigned the front end
to use an ordinary op amp TIA, not realizing they lost about 15 dB of
SNR in the process. (I managed to get that one fixed, and the guy
responsible taken off the project. Unfortunately they were almost all
like that.)

They replaced the direct drive for the grating with a belt drive, which
gave nice smooth motion but of course rapidly lost accuracy as the belt
squirmed around while moving, so that the calibration wouldn't sit
still. They needed more encoder precision, so they put an encoder on
the motor as well as the grating, and did some micky-mouse trick to
combine the two encoder readings. Even the magnetic encoder on the
grating shaft refused to sit still--it drifted around like mad. I went
there to try to get to the bottom of some of this stuff, and although
they mostly sidelined me because I wasn't writing the cheques, we did
manage to get to the bottom of that one. Turned out that the encoder's
output was the duty cycle of a pulse train (like the RC servo only
backwards), and they were measuring the pulse width instead, using a
capture input of their MCU. That turned the frequency drift into a
position drift. Once it was fixed, I hit that poor encoder with cold
spray and a heat gun, and couldn't get it to move at all. Kudos to US
Digital for that one.

The belt-drive system failed nonetheless, basically because the
measurement was being done on the slope of the very strong IR absorption
spectrum of water in the 1.4-1.7 micron range, so that small wavelength
shifts caused much larger amplitude errors.

I told them to use a sine-bar drive like every other Czerny-Turner
monochromator in the world, but they refused, insisting on using a
worm-and-sector gear instead, with the encoder on the worm shaft.
Unlike nearly any other sort of gearing, worms work using sliding
friction. That makes the grease film thin out with time. I calculated
for their benefit that in their design, with the very small radius of
the sector gear, the maximum lubricant variation we could tolerate was
about 70 nanometres. Since they were nearly finished with the prototype
build for the formal clinical trial, I told them to use dry molybdenum
disulphide for lubricant instead of grease.

They straight-up refused, saying they couldn't get MoS2, so I sent them
a link to the exact SKU on fastenall.com, after verifying that their
local Fastenall had it in stock. (I even sent them a Mapquest link so
they could find the store. That was a bit sarcastic, which I regret,
but I was getting pretty tired of them by that point.) They proceeded
to ship one unit with grease and three units unlubricated.

When I complained about all the directionless fiddling they were doing,
one bright lad smiled and said brightly, "That's engineering!" He was
one of the better ones.

They also took the POC proto apart to use bits of it in their test
system, so that they had no comparison data, and, oh, yes, they broke
the $100k FTIR and didn't tell anyone.

The units failed the acceptance test. I attended it, but since the USC
folks weren't crunching the data in real time, the failure wasn't
entirely apparent till later. By that point the CE had run through a
year's time and most of a million bucks.

Some months later, two units arrived on my bench, along with an urgent
request for me to get to the bottom of it all.

Turned out to be a real onion problem--you know, peel off one layer,
cry, and peel off the next.

The phase of the detected signal was moving around by 10 degrees or so,
which was easily enough to destroy the measurement. The code seemed to
be a PID controller using an optointerrupter on the chopper wheel, which
should have been fine. I built a strobe light using an HP 3325A
synthesizer driving a LED, so that I could see the loop dynamics. The
controller was totally broken--regardless of the settings of P, I, and
D, there was no way of making the phase sit still. A gentle stream of
canned air would move the phase, and it would never recover--i.e. there
was no integral term in the control law, despite what the settings would
have one believe.

And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform by
least-squares curve fitting to a sine wave instead of using an FFT like
normal people. Everybody screws up their first digital lock-in, but
I've never seen one as bad as that. (For non signals-and-systems folks:
least squares fitting works OK at high SNR, but for noisy data it falls
apart completely. The FFT uses orthogonality, and so works at any SNR.)

I didn't get to the last onion layer, because the founder ran out of
both dough and friends. He never did pay me for my last month's work.

A pity. I would have made those boxes work eventually.


(*) One where the business end slides around on a curved surface, like
the blade assembly on some razors.
(**) A cage system using plates held together with 6-mm
centreless-ground stainless steel rods, similar to the Thor Labs 30-mm
cage system.

Cheers

Phil Hobbs

I want a wideband optical spectrum analyzer, say 300 to 1600 nm, for
checking LEDs and IR fiber lasers and generally knowing where we are.
It doesn't need to be very sensitive or terribly accurate. Nobody
makes one, and there should be a market, and it sounds a lot easier
than that project.

Smaller TAM though.

You can do nearly that whole range with germanium, or else a
blue-enhanced Si and an ordinary InGaAs. The hardware would be a bit of
a challenge since the bandwidth is more than an octave, so that the
diffracted orders would overlap. (It's a lot like having a tuning range
more than twice the IF in a receiver.) You'd have to use a filter wheel
or else cross-dispersion with another grating, and at that point it's
probably easier to use two spectrometers. I have an Ocean Optics one
that I quite like.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC / Hobbs ElectroOptics
Optics, Electro-optics, Photonics, Analog Electronics
Briarcliff Manor NY 10510

http://electrooptical.net
http://hobbs-eo.com

 

[Phil Hobbs]

On 2020-01-15 10:15, Phil Hobbs wrote:

On 2020-01-15 08:41, Steve Wilson wrote:
On Wednesday, January 15, 2020 at 7:29:08 AM UTC-5, Phil Hobbs wrote:
[...]


And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform
by least-squares curve fitting to a sine wave instead of using an FFT
like normal people.  Everybody screws up their first digital lock-in,
but I've never seen one as bad as that.  (For non signals-and-systems
folks: least squares fitting works OK at high SNR, but for noisy data
it falls apart completely.  The FFT uses orthogonality, and so works
at any SNR.)

I didn't get to the last onion layer, because the founder ran out of
both dough and friends.  He never did pay me for my last
month's work.

A pity.  I would have made those boxes work eventually.

(*) One where the business end slides around on a curved surface,
like the blade assembly on some razors. (**) A cage system using
plates held together with 6-mm centreless-ground stainless steel
rods, similar to the Thor Labs 30-mm cage system.

Cheers

Phil Hobbs


That is an amazing story. Thanks.

Was there no way you contact the owner and warn him they were
ruining the design?

Oh, I told him, all right, and he told them.  They did fix a few things,
but mostly said "yes" and meant "no".  Since I wasn't supervising them,
they didn't keep me in the loop with what they were doing to fix the
problems.

How did they get authorization to redesign the product in the first
place?

Well, as I said, the FDA wouldn't have liked a system made of toy parts,
so the optomechanics did need re-doing.  That part they actually did
reasonably well, apart from the grating mechanism and the chopper servo.

The electronics they just went ahead and re-did, I think in order to
reuse some previous design.  Digital lock-ins are difficult because you
have to be totally paranoid about things like slew artifacts, settling
time, input voltage sag during sampling, and anything getting in on the
reference.  The frequency domain is brutal that way, and AFAICT nobody
ever learns that except by way of at least one failure.

How do we prevent this from happening to us?

Stay out of Orange County.

Seriously, having a sharp technical person supervising, with a formal
spec, design reviews, sign-offs on hardware and software, unit tests,
and so on would have prevented it.  It would have been half again as
expensive, but the system would have worked.  Checking the CE's
references would have been a smart move too.  They claimed that their
contracts mostly forbade them to tell who their customers were, and the
founder fell for that one.

Oh, and when checking references, if you get good ones be sure to ask
the names of the CE's people that worked on those projects. They'll
always give out their 'A' team's customers, but you may get the 'B' or
'C' teams.

In our case, the guy who did the lock-in and encoder work was their CTO,
so presumably the 'B' team is worse.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC / Hobbs ElectroOptics
Optics, Electro-optics, Photonics, Analog Electronics
Briarcliff Manor NY 10510

http://electrooptical.net
http://hobbs-eo.com

 

On Wed, 15 Jan 2020 11:47:36 -0500, Phil Hobbs
<pcdhSpamMeSenseless@electrooptical.net> wrote:

On 2020-01-15 10:15, Phil Hobbs wrote:
On 2020-01-15 08:41, Steve Wilson wrote:
On Wednesday, January 15, 2020 at 7:29:08 AM UTC-5, Phil Hobbs wrote:
[...]


And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform
by least-squares curve fitting to a sine wave instead of using an FFT
like normal people.  Everybody screws up their first digital lock-in,
but I've never seen one as bad as that.  (For non signals-and-systems
folks: least squares fitting works OK at high SNR, but for noisy data
it falls apart completely.  The FFT uses orthogonality, and so works
at any SNR.)

I didn't get to the last onion layer, because the founder ran out of
both dough and friends.  He never did pay me for my last
month's work.

A pity.  I would have made those boxes work eventually.

(*) One where the business end slides around on a curved surface,
like the blade assembly on some razors. (**) A cage system using
plates held together with 6-mm centreless-ground stainless steel
rods, similar to the Thor Labs 30-mm cage system.

Cheers

Phil Hobbs


That is an amazing story. Thanks.

Was there no way you contact the owner and warn him they were
ruining the design?

Oh, I told him, all right, and he told them.  They did fix a few things,
but mostly said "yes" and meant "no".  Since I wasn't supervising them,
they didn't keep me in the loop with what they were doing to fix the
problems.

How did they get authorization to redesign the product in the first
place?

Well, as I said, the FDA wouldn't have liked a system made of toy parts,
so the optomechanics did need re-doing.  That part they actually did
reasonably well, apart from the grating mechanism and the chopper servo.

The electronics they just went ahead and re-did, I think in order to
reuse some previous design.  Digital lock-ins are difficult because you
have to be totally paranoid about things like slew artifacts, settling
time, input voltage sag during sampling, and anything getting in on the
reference.  The frequency domain is brutal that way, and AFAICT nobody
ever learns that except by way of at least one failure.

How do we prevent this from happening to us?

Stay out of Orange County.

Seriously, having a sharp technical person supervising, with a formal
spec, design reviews, sign-offs on hardware and software, unit tests,
and so on would have prevented it.  It would have been half again as
expensive, but the system would have worked.  Checking the CE's
references would have been a smart move too.  They claimed that their
contracts mostly forbade them to tell who their customers were, and the
founder fell for that one.


Oh, and when checking references, if you get good ones be sure to ask
the names of the CE's people that worked on those projects. They'll
always give out their 'A' team's customers, but you may get the 'B' or
'C' teams.

In our case, the guy who did the lock-in and encoder work was their CTO,
so presumably the 'B' team is worse.

Cheers

Phil Hobbs

It's impressive how many people are really bad at electronic design.



--

John Larkin Highland Technology, Inc

The cork popped merrily, and Lord Peter rose to his feet.
"Bunter", he said, "I give you a toast. The triumph of Instinct over Reason"

 

[Steve Wilson]

Phil Hobbs <pcdhSpamMeSenseless@electrooptical.net> wrote:

On 2020-01-15 08:41, Steve Wilson wrote:
On Wednesday, January 15, 2020 at 7:29:08 AM UTC-5, Phil Hobbs wrote:
[...]

And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform
by least-squares curve fitting to a sine wave instead of using an FFT
like normal people. Everybody screws up their first digital lock-in,
but I've never seen one as bad as that. (For non signals-and-systems
folks: least squares fitting works OK at high SNR, but for noisy data
it falls apart completely. The FFT uses orthogonality, and so works
at any SNR.)

I didn't get to the last onion layer, because the founder ran out
of both dough and friends. He never did pay me for my last
month's work.

A pity. I would have made those boxes work eventually.

(*) One where the business end slides around on a curved surface,
like the blade assembly on some razors. (**) A cage system using
plates held together with 6-mm centreless-ground stainless steel
rods, similar to the Thor Labs 30-mm cage system.

Cheers

Phil Hobbs

That is an amazing story. Thanks.

Was there no way you contact the owner and warn him they were ruining
the design?

Oh, I told him, all right, and he told them. They did fix a few things,
but mostly said "yes" and meant "no". Since I wasn't supervising them,
they didn't keep me in the loop with what they were doing to fix the
problems.

How did they get authorization to redesign the product in the first
place?

Well, as I said, the FDA wouldn't have liked a system made of toy parts,
so the optomechanics did need re-doing. That part they actually did
reasonably well, apart from the grating mechanism and the chopper servo.

The electronics they just went ahead and re-did, I think in order to
reuse some previous design. Digital lock-ins are difficult because you
have to be totally paranoid about things like slew artifacts, settling
time, input voltage sag during sampling, and anything getting in on the
reference. The frequency domain is brutal that way, and AFAICT nobody
ever learns that except by way of at least one failure.

How do we prevent this from happening to us?

Stay out of Orange County.

Seriously, having a sharp technical person supervising, with a formal
spec, design reviews, sign-offs on hardware and software, unit tests,
and so on would have prevented it. It would have been half again as
expensive, but the system would have worked. Checking the CE's
references would have been a smart move too. They claimed that their
contracts mostly forbade them to tell who their customers were, and the
founder fell for that one.

Cheers

Phil Hobbs

Thanks. There are lessons here for everyone.

 

[George Herold]

On Wednesday, January 15, 2020 at 7:29:08 AM UTC-5, Phil Hobbs wrote:

On 2020-01-15 03:17, RBlack wrote:
On Tue, 14 Jan 2020 12:35:56 -0500, Phil Hobbs
pcdhSpamMeSenseless@electrooptical.net> said:


[snip]

Client work almost always succeeds, and the occasional failures are
mostly due to the customer's prevarication, such as taking my
proof-of-concept system, giving it to a CE outfit, and then pulling me
back in to attempt to fix the CE's mess--of course at the last minute,
when they've almost run out of money. That's happened a couple of
times, so I try very hard to discourage it. (The two were the
transcutaneous blood glucose/alcohol system and the blood-spot detector

^^^^^^^^^^^^^^ ^^^^^ ^^^^^^^

Bummer, sorry to hear about that. (I have a personal interest in seeing
this type of device working and on the market). Did anyone else pick up
the project, or did it just die?

for hens' eggs.)


Yeah, that was a sad one, for sure. It remains vivid in my memory, so
at the risk of boring people, here's the tale.

The founder called me out of the blue at 3 PM on Christmas Eve, 2012.
He turned out to be a charming fellow with a lot of drive and not a lot
of education, who was practically supernatural at raising money. He
wanted me to build him an instrument, because that's what I do.

He'd patented the general principle, which avoided the individual
physiological variations that usually bedevil those sorts of measurements.

The idea was to use a hand cradle with a virtual pivot (*) holding a
fibre bundle against the web of the first and second fingers. There are
two arteries very close to the surface there, so you get to measure
fresh blood instead of tissue fluid, and no one has hair, fat, or
callouses there, so the physiology is very cooperative. He had some
promising data that he took himself using a Perkin-Elmer FTIR and his
hand cradle. (He somehow talked some guys at USC into giving him lab
space and some help doing that--he was entirely self-taught, which has
its strengths and weaknesses.)

The project was an unusual one for me, in that I didn't have my arms
around the whole measurement--I built the gizmo, but the founder had his
USC statistics friends use their AI nous to build the model and extract
the blood solute data. Thus I don't actually know how that was done.
(It wasn't something simple like spectral differencing, anyway.)

I did a photon budget, which is my term for a detailed feasibility
calculation emphasizing stability and SNR. That's super important
because without calculating how good the measurement _could_ be, you
never really know how you're doing. A photon budget prevents you from
wasting time on recreational impossibilities on the one hand, or turning
a silk purse back into a sow's ear on the other. In this case it looked
pretty good, using a tungsten source, a custom-designed split bundle of
about 20 fibres (TX + RX), a conventional Czerny-Turner monochromator, a
single extended-InGaAs photodiode, and a chopper plus lock-in for
detection.

The proof-of-concept system took me about six weeks start to finish,
including the photon budget, optical design, designing and building the
electronics, assembling the optomechanics, and writing the software. It
was built on a 12 x 24-inch aluminum breadboard using a combination of
hhacked Microbench (**) parts, JB Weld (the poor man's machine shop),
and a servo from an RC airplane for moving the grating. The grating
cradle was also built from toy airplane parts, courtesy of
servocity.com. The electronics was all built in die cast aluminum stomp
boxes, dead bug style, wired up with BNC cables. The chopper was a
commercial (Thor Labs) unit, and the back end was a LabJack and a
console-mode C++ program running on a second-hand laptop.

It all worked great, and was very amusing to watch--an advanced clinical
instrument built out of JB Weld and toy parts. (Wouldn't the FDA have
loved that?)

We did the preliminary acceptance test by having some friends over for
cocktails and measuring their blood spectra every 15 minutes or so. The
data did exactly as we hoped--nice repeatable curves with the right time
dependence and no big physiological variations between subjects. We did
some glucose work using a strip reader for comparison, but the strips
have relatively poor accuracy, so we concentrated on the alcohol
measurement for that part of the demo. (Quaffing a few cool ones is
much more fun than sticking pins in your fingers, too.)

After the founder used the POC data to raise a bunch more money, we
brought the proto and the Perkin-Elmer FTIR to a contract engineering
house in Orange County CA that will remain nameless because they have
this unfortunate tendency to sue everybody in sight. The founder kept
me sort of distantly in the loop, but his crucial mistake was trying to
save money by supervising the CE firm himself. (I tried to help, but
they ignored me almost entirely because I wasn't the one writing the
cheques.)

The CE folks proceeded to fall into every pothole along the road, like a
drunk. They ignored the photon budget, and so redesigned the front end
to use an ordinary op amp TIA, not realizing they lost about 15 dB of
SNR in the process. (I managed to get that one fixed, and the guy
responsible taken off the project. Unfortunately they were almost all
like that.)

They replaced the direct drive for the grating with a belt drive, which
gave nice smooth motion but of course rapidly lost accuracy as the belt
squirmed around while moving, so that the calibration wouldn't sit
still. They needed more encoder precision, so they put an encoder on
the motor as well as the grating, and did some micky-mouse trick to
combine the two encoder readings. Even the magnetic encoder on the
grating shaft refused to sit still--it drifted around like mad. I went
there to try to get to the bottom of some of this stuff, and although
they mostly sidelined me because I wasn't writing the cheques, we did
manage to get to the bottom of that one. Turned out that the encoder's
output was the duty cycle of a pulse train (like the RC servo only
backwards), and they were measuring the pulse width instead, using a
capture input of their MCU. That turned the frequency drift into a
position drift. Once it was fixed, I hit that poor encoder with cold
spray and a heat gun, and couldn't get it to move at all. Kudos to US
Digital for that one.

The belt-drive system failed nonetheless, basically because the
measurement was being done on the slope of the very strong IR absorption
spectrum of water in the 1.4-1.7 micron range, so that small wavelength
shifts caused much larger amplitude errors.

I told them to use a sine-bar drive like every other Czerny-Turner
monochromator in the world, but they refused, insisting on using a
worm-and-sector gear instead, with the encoder on the worm shaft.
Unlike nearly any other sort of gearing, worms work using sliding
friction. That makes the grease film thin out with time. I calculated
for their benefit that in their design, with the very small radius of
the sector gear, the maximum lubricant variation we could tolerate was
about 70 nanometres. Since they were nearly finished with the prototype
build for the formal clinical trial, I told them to use dry molybdenum
disulphide for lubricant instead of grease.

They straight-up refused, saying they couldn't get MoS2, so I sent them
a link to the exact SKU on fastenall.com, after verifying that their
local Fastenall had it in stock. (I even sent them a Mapquest link so
they could find the store. That was a bit sarcastic, which I regret,
but I was getting pretty tired of them by that point.) They proceeded
to ship one unit with grease and three units unlubricated.

When I complained about all the directionless fiddling they were doing,
one bright lad smiled and said brightly, "That's engineering!" He was
one of the better ones.

They also took the POC proto apart to use bits of it in their test
system, so that they had no comparison data, and, oh, yes, they broke
the $100k FTIR and didn't tell anyone.

The units failed the acceptance test. I attended it, but since the USC
folks weren't crunching the data in real time, the failure wasn't
entirely apparent till later. By that point the CE had run through a
year's time and most of a million bucks.

Some months later, two units arrived on my bench, along with an urgent
request for me to get to the bottom of it all.

Turned out to be a real onion problem--you know, peel off one layer,
cry, and peel off the next.

The phase of the detected signal was moving around by 10 degrees or so,
which was easily enough to destroy the measurement. The code seemed to
be a PID controller using an optointerrupter on the chopper wheel, which
should have been fine. I built a strobe light using an HP 3325A
synthesizer driving a LED, so that I could see the loop dynamics. The
controller was totally broken--regardless of the settings of P, I, and
D, there was no way of making the phase sit still. A gentle stream of
canned air would move the phase, and it would never recover--i.e. there
was no integral term in the control law, despite what the settings would
have one believe.

And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform by
least-squares curve fitting to a sine wave instead of using an FFT like
normal people. Everybody screws up their first digital lock-in, but
I've never seen one as bad as that. (For non signals-and-systems folks:
least squares fitting works OK at high SNR, but for noisy data it falls
apart completely. The FFT uses orthogonality, and so works at any SNR.)

I didn't get to the last onion layer, because the founder ran out of
both dough and friends. He never did pay me for my last month's work.

A pity. I would have made those boxes work eventually.


(*) One where the business end slides around on a curved surface, like
the blade assembly on some razors.
(**) A cage system using plates held together with 6-mm
centreless-ground stainless steel rods, similar to the Thor Labs 30-mm
cage system.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs
Principal Consultant
ElectroOptical Innovations LLC / Hobbs ElectroOptics
Optics, Electro-optics, Photonics, Analog Electronics
Briarcliff Manor NY 10510

http://electrooptical.net
http://hobbs-eo.com

Wow! Thanks, nice story. I've half dreamed of making a cheap spectrometer
with a grating and rotation stage/ worm drive... figuring if you always
turn it in one direction the backlash won't be a problem... I never thought
about the grease!

Re digital LI: Color me ignorant, but my first idea would be to
multiply the signal and ref. in software... is there a reason that is
a bad idea?

George H.

 

[George Herold]

On Wednesday, January 15, 2020 at 10:46:30 AM UTC-5, jla...@highlandsniptechnology.com wrote:

On Wed, 15 Jan 2020 07:28:58 -0500, Phil Hobbs
pcdhSpamMeSenseless@electrooptical.net> wrote:

On 2020-01-15 03:17, RBlack wrote:
On Tue, 14 Jan 2020 12:35:56 -0500, Phil Hobbs
pcdhSpamMeSenseless@electrooptical.net> said:


[snip]

Client work almost always succeeds, and the occasional failures are
mostly due to the customer's prevarication, such as taking my
proof-of-concept system, giving it to a CE outfit, and then pulling me
back in to attempt to fix the CE's mess--of course at the last minute,
when they've almost run out of money. That's happened a couple of
times, so I try very hard to discourage it. (The two were the
transcutaneous blood glucose/alcohol system and the blood-spot detector

^^^^^^^^^^^^^^ ^^^^^ ^^^^^^^

Bummer, sorry to hear about that. (I have a personal interest in seeing
this type of device working and on the market). Did anyone else pick up
the project, or did it just die?

for hens' eggs.)


Yeah, that was a sad one, for sure. It remains vivid in my memory, so
at the risk of boring people, here's the tale.

The founder called me out of the blue at 3 PM on Christmas Eve, 2012.
He turned out to be a charming fellow with a lot of drive and not a lot
of education, who was practically supernatural at raising money. He
wanted me to build him an instrument, because that's what I do.

He'd patented the general principle, which avoided the individual
physiological variations that usually bedevil those sorts of measurements.

The idea was to use a hand cradle with a virtual pivot (*) holding a
fibre bundle against the web of the first and second fingers. There are
two arteries very close to the surface there, so you get to measure
fresh blood instead of tissue fluid, and no one has hair, fat, or
callouses there, so the physiology is very cooperative. He had some
promising data that he took himself using a Perkin-Elmer FTIR and his
hand cradle. (He somehow talked some guys at USC into giving him lab
space and some help doing that--he was entirely self-taught, which has
its strengths and weaknesses.)

The project was an unusual one for me, in that I didn't have my arms
around the whole measurement--I built the gizmo, but the founder had his
USC statistics friends use their AI nous to build the model and extract
the blood solute data. Thus I don't actually know how that was done.
(It wasn't something simple like spectral differencing, anyway.)

I did a photon budget, which is my term for a detailed feasibility
calculation emphasizing stability and SNR. That's super important
because without calculating how good the measurement _could_ be, you
never really know how you're doing. A photon budget prevents you from
wasting time on recreational impossibilities on the one hand, or turning
a silk purse back into a sow's ear on the other. In this case it looked
pretty good, using a tungsten source, a custom-designed split bundle of
about 20 fibres (TX + RX), a conventional Czerny-Turner monochromator, a
single extended-InGaAs photodiode, and a chopper plus lock-in for
detection.

The proof-of-concept system took me about six weeks start to finish,
including the photon budget, optical design, designing and building the
electronics, assembling the optomechanics, and writing the software. It
was built on a 12 x 24-inch aluminum breadboard using a combination of
hhacked Microbench (**) parts, JB Weld (the poor man's machine shop),
and a servo from an RC airplane for moving the grating. The grating
cradle was also built from toy airplane parts, courtesy of
servocity.com. The electronics was all built in die cast aluminum stomp
boxes, dead bug style, wired up with BNC cables. The chopper was a
commercial (Thor Labs) unit, and the back end was a LabJack and a
console-mode C++ program running on a second-hand laptop.

It all worked great, and was very amusing to watch--an advanced clinical
instrument built out of JB Weld and toy parts. (Wouldn't the FDA have
loved that?)

We did the preliminary acceptance test by having some friends over for
cocktails and measuring their blood spectra every 15 minutes or so. The
data did exactly as we hoped--nice repeatable curves with the right time
dependence and no big physiological variations between subjects. We did
some glucose work using a strip reader for comparison, but the strips
have relatively poor accuracy, so we concentrated on the alcohol
measurement for that part of the demo. (Quaffing a few cool ones is
much more fun than sticking pins in your fingers, too.)

After the founder used the POC data to raise a bunch more money, we
brought the proto and the Perkin-Elmer FTIR to a contract engineering
house in Orange County CA that will remain nameless because they have
this unfortunate tendency to sue everybody in sight. The founder kept
me sort of distantly in the loop, but his crucial mistake was trying to
save money by supervising the CE firm himself. (I tried to help, but
they ignored me almost entirely because I wasn't the one writing the
cheques.)

The CE folks proceeded to fall into every pothole along the road, like a
drunk. They ignored the photon budget, and so redesigned the front end
to use an ordinary op amp TIA, not realizing they lost about 15 dB of
SNR in the process. (I managed to get that one fixed, and the guy
responsible taken off the project. Unfortunately they were almost all
like that.)

They replaced the direct drive for the grating with a belt drive, which
gave nice smooth motion but of course rapidly lost accuracy as the belt
squirmed around while moving, so that the calibration wouldn't sit
still. They needed more encoder precision, so they put an encoder on
the motor as well as the grating, and did some micky-mouse trick to
combine the two encoder readings. Even the magnetic encoder on the
grating shaft refused to sit still--it drifted around like mad. I went
there to try to get to the bottom of some of this stuff, and although
they mostly sidelined me because I wasn't writing the cheques, we did
manage to get to the bottom of that one. Turned out that the encoder's
output was the duty cycle of a pulse train (like the RC servo only
backwards), and they were measuring the pulse width instead, using a
capture input of their MCU. That turned the frequency drift into a
position drift. Once it was fixed, I hit that poor encoder with cold
spray and a heat gun, and couldn't get it to move at all. Kudos to US
Digital for that one.

The belt-drive system failed nonetheless, basically because the
measurement was being done on the slope of the very strong IR absorption
spectrum of water in the 1.4-1.7 micron range, so that small wavelength
shifts caused much larger amplitude errors.

I told them to use a sine-bar drive like every other Czerny-Turner
monochromator in the world, but they refused, insisting on using a
worm-and-sector gear instead, with the encoder on the worm shaft.
Unlike nearly any other sort of gearing, worms work using sliding
friction. That makes the grease film thin out with time. I calculated
for their benefit that in their design, with the very small radius of
the sector gear, the maximum lubricant variation we could tolerate was
about 70 nanometres. Since they were nearly finished with the prototype
build for the formal clinical trial, I told them to use dry molybdenum
disulphide for lubricant instead of grease.

They straight-up refused, saying they couldn't get MoS2, so I sent them
a link to the exact SKU on fastenall.com, after verifying that their
local Fastenall had it in stock. (I even sent them a Mapquest link so
they could find the store. That was a bit sarcastic, which I regret,
but I was getting pretty tired of them by that point.) They proceeded
to ship one unit with grease and three units unlubricated.

When I complained about all the directionless fiddling they were doing,
one bright lad smiled and said brightly, "That's engineering!" He was
one of the better ones.

They also took the POC proto apart to use bits of it in their test
system, so that they had no comparison data, and, oh, yes, they broke
the $100k FTIR and didn't tell anyone.

The units failed the acceptance test. I attended it, but since the USC
folks weren't crunching the data in real time, the failure wasn't
entirely apparent till later. By that point the CE had run through a
year's time and most of a million bucks.

Some months later, two units arrived on my bench, along with an urgent
request for me to get to the bottom of it all.

Turned out to be a real onion problem--you know, peel off one layer,
cry, and peel off the next.

The phase of the detected signal was moving around by 10 degrees or so,
which was easily enough to destroy the measurement. The code seemed to
be a PID controller using an optointerrupter on the chopper wheel, which
should have been fine. I built a strobe light using an HP 3325A
synthesizer driving a LED, so that I could see the loop dynamics. The
controller was totally broken--regardless of the settings of P, I, and
D, there was no way of making the phase sit still. A gentle stream of
canned air would move the phase, and it would never recover--i.e. there
was no integral term in the control law, despite what the settings would
have one believe.

And then it turned out that they'd never built a lock-in before, and
were trying to extract the (approximately triangular) signal waveform by
least-squares curve fitting to a sine wave instead of using an FFT like
normal people. Everybody screws up their first digital lock-in, but
I've never seen one as bad as that. (For non signals-and-systems folks:
least squares fitting works OK at high SNR, but for noisy data it falls
apart completely. The FFT uses orthogonality, and so works at any SNR.)

I didn't get to the last onion layer, because the founder ran out of
both dough and friends. He never did pay me for my last month's work.

A pity. I would have made those boxes work eventually.


(*) One where the business end slides around on a curved surface, like
the blade assembly on some razors.
(**) A cage system using plates held together with 6-mm
centreless-ground stainless steel rods, similar to the Thor Labs 30-mm
cage system.

Cheers

Phil Hobbs

I want a wideband optical spectrum analyzer, say 300 to 1600 nm, for
checking LEDs and IR fiber lasers and generally knowing where we are.
It doesn't need to be very sensitive or terribly accurate. Nobody
makes one, and there should be a market, and it sounds a lot easier
than that project.

ocean optics makes some cheaper spectrometers. But I think you are being a
bit greedy on the wavelength range. In fact I think you are limited to
an octave, because you'll get higher order peaks... but I'm not sure.
(Phil will set me straight... or I'll pull out "Building Scientific Apparatus")

Hmm I was wrong about the octave thing
https://www.oceaninsight.com/products/spectrometers/

(not a great website... lotsa feel good video.)

George H.

--

John Larkin Highland Technology, Inc

The cork popped merrily, and Lord Peter rose to his feet.
"Bunter", he said, "I give you a toast. The triumph of Instinct over Reason"