mystery rackmount kit...

[Don Y]

On 7/10/2022 2:06 PM, Don Y wrote:

I\'d initially thought they might be disk shelves owing to the height (about
4U?) and \"bland\" front panels. But, they are way too deep for that. Anyone
recognize the distinctive red \"badge\" on the front?

I\'ll have to see if there are enough other curiosities to merit a drive over,
tomorrow...

Each was a 60-drive (SAS/SATA) \"storage box\". Most were filled with a
mix of 600G, 4TB and 6TB drives (from first-hand experience, these are
often low PoHr devices being \"retired\" because some grant funding ran out;
I\'ve never found a drive in these lots with even a single remapped sector!).
Mostly 4KN (which can be annoying in non-array deployments).

They appear to have sold for an average of ~$60 per unit (~$300/pallet).
ISTR there were about 15 such boxes.

But disposing of all that unwanted \"surplus\" would really be a chore (and I
already have gobs of disks and shelf hardware).

Gotta wonder where it is headed as I can\'t imagine an enterprise
running on \"used\" rust! Though the assemblies may have had some
inherent value to justify the freight charges required (scrap the drives).

OTOH, there were enough other interesting goodies to have made the
trip worthwhile! :> (Gee, I thought I was trying to get RID of stuff?)

 

[Grant Taylor]

On 7/21/22 7:42 PM, Don Y wrote:

They appear to have sold for an average of ~$60 per unit
(~$300/pallet). ISTR there were about 15 such boxes.

If I was on the correct continent and had an opportunity, I probably
would have put my hand up for part of a group buy for one or two of the
units.

But disposing of all that unwanted \"surplus\" would really be a chore
(and I already have gobs of disks and shelf hardware).

What would you consider to be the surplus?

Gotta wonder where it is headed as I can\'t imagine an enterprise
running on \"used\" rust! Though the assemblies may have had some
inherent value to justify the freight charges required (scrap the
drives).

I actually see value in the drives. I\'ve seen a LOT of so called
refurbished drives for sale on many different venues.

I\'ve consumed used spinning rust for more than two decades in things at
home.

I know multiple people that are still running EoL equipment that\'s 2nd
or 3rd hand that\'s fulfilling their needs perfectly fine. I don\'t think
you can buy non-used drives for some of that equipment.

I agree, most enterprises won\'t find themselves in /this/ particular
situation. But I would not be surprised if some of the employees of
said enterprises were using \"\"refurbished drives for personal things.

OTOH, there were enough other interesting goodies to have made the trip
worthwhile! :> (Gee, I thought I was trying to get RID of stuff?)

--
Grant. . . .
unix || die

 

[Don Y]

On 7/21/2022 9:51 PM, Grant Taylor wrote:

On 7/21/22 7:42 PM, Don Y wrote:
They appear to have sold for an average of ~$60 per unit (~$300/pallet).
ISTR there were about 15 such boxes.

If I was on the correct continent and had an opportunity, I probably would have
put my hand up for part of a group buy for one or two of the units.

On such items, its hard for someone (i.e., me) operating in a different
\"market\" (that of an individual buyer vs. a reseller) to sort out a
proper bid for an item.

I bought a 5KVA sine wave UPS for $50 -- and that was probably the sole
bid offered on it! (I sold the depleted batteries from it for $100)

I bought a Personal Reader (<https://www.youtube.com/watch?v=_zNxsYLuGqw>)
for $7 and another $7 for the associated scanner -- because no one else
knew what they were (or if there might be a market for it!)

Several pen plotters, over the years (each bigger than the previous).

I have a delightful, high-backed, swivel/rolling chair in my office
that I paid a whole *dollar* for!

OTOH, I see pallets of PCs sell for $1000 and wouldn\'t even consider
bidding on same -- even if there was a jem sitting on the top of the
pile! (what the hell would I do with the rest of them??)

But disposing of all that unwanted \"surplus\" would really be a chore (and I
already have gobs of disks and shelf hardware).

What would you consider to be the surplus?

My interest was in the chassis -- as the basis for mass disk
duplication/erasure appliances. I.e., multipass \"wiping\" drives
to ensure their previous contents have been obliterated thereby
allowing the drives to be reused/redistributed in other devices
without fear of their original/proprietary content being
disclosed (to someone with prying eyes).

I\'ve developed software to automate this task so groups who want to
(or MUST!) process lots of disks can do so in a short period of time
with unskilled labor (the machine watches everything that is
done and oversees the subsequent use/reuse of those drives;
so, if a drive was defective or not recognized as being wiped,
it never gets out of the building *and* donors can be reassured
that *their* drives -- reported by S/N -- were \"processed\" as
agreed upon!)

Gotta wonder where it is headed as I can\'t imagine an enterprise running on
\"used\" rust! Though the assemblies may have had some inherent value to
justify the freight charges required (scrap the drives).

I actually see value in the drives. I\'ve seen a LOT of so called refurbished
drives for sale on many different venues.

But I would assume those are targeted to individuals, not businesses.
Would you want to have your organization\'s data running on drives
with no history (nor recourse in the event of failure)?

> I\'ve consumed used spinning rust for more than two decades in things at home.

I have a couple *hundred* drives pulled from similar shelfs, over the years.
I have 7 15-drive shelfs along with a pair of 12-drive shelfs in my SAN. And,
a little 4 drive enclosure tethered to my VM server (has 8 internal drives).

Anything smaller than 500GB gets recycled (the space it takes up could better
be used by a 1TB or larger drive!)

So, you\'re preaching to the choir.

But, at this point, the drives would just be more clutter, for me (I think
I have 200-300TB, here). My recent efforts have focused on finding ways
to \"store\" things inside other things. E.g., instead of keeping a box
of spare DVD drives, *install* them in various workstations and discard
the box; instead of keeping boxes of DRAM, fully populate each machine and
discard the excess; etc.

I know multiple people that are still running EoL equipment that\'s 2nd or 3rd
hand that\'s fulfilling their needs perfectly fine. I don\'t think you can buy
non-used drives for some of that equipment.

There is a large retailer, here, who runs his enterprise on Sun kit.
The guy who runs the IT department is constantly looking for *any*
Sun kit to handle expanded demands as well as repair/replacements:
\"We\'re opening a new store...\"

But, the owner has made the decision that the risk incurred from
hardware failures is exceeded by the cost of trying to migrate
all of his systems to a \"newer\" platform...

....given that a newer platform will only be \"newer\" for a short period
of time!

[OTOH, he has recently realized that he is *overly* dependant on said
IT guy to keep his enterprise running!]

E.g., I keep SCA and FC/AL drives as spares for some of my older kit.
I even have some 2.5\" SCA and SCSI drives for \"unusual uses\".
But, if all of them failed (before I had a chance to recover their
contents), I would be \"disappointed\"... but not out of business!

I agree, most enterprises won\'t find themselves in /this/ particular
situation. But I would not be surprised if some of the employees of said
enterprises were using \"\"refurbished drives for personal things.

Yes. But someone has to show up at the auction to bid on said items.
IME, most of these folks are resellers trying to pick up <something>
at a bargain price and flip it to someone else who doesn\'t have time
(patience) to scour auction sites.

I\'m an exception in that I see these outings as entertainment; sort
of like wandering through an (old-fashioned) hardware store exploring
the various unusual tools and supplies on-hand and pondering their uses.

This works *to* my advantage as well as *against*:
- \"to\" in that I am willing to pay what an \"end user\" might be
expected to pay -- likely more than a reseller would like to pay
as he\'d forfeit his expected profit
- \"against\" in that I may only be interested in *an* item on a
pallet and limit my bid to what I would be willing to pay for
*that* item. A reseller would bid based on the entire contents
of the pallet!

The final issue being that my livelihood doesn\'t rely on the \"chance\"
associated with this \"product sourcing scheme\"... nor on my ability
to predict the correct purchase price (i.e., \"bid\"). If I don\'t
win anything, I still got some entertainment out of digging through
all the piles of kit! (if a reseller doesn\'t win, he\'s got to wonder
what he\'s going to resell to generate income!)

[I used to have a few friends who worked at this location who could
alert me to \"goodies\" and provide better, targeted photos of items
in which I\'d expressed an interest. But, a different department is
now handling the auctions -- in a different manner -- so I\'ve lost
those assets]

OTOH, there were enough other interesting goodies to have made the trip
worthwhile! :> (Gee, I thought I was trying to get RID of stuff?)

The *big* problem with getting old is you have the experience, knowledge
and funds to do the things you dreamed of doing when younger; but, not
the *time* nor *space* (the latter because its been filled with a lifetime
of \"acquisitions\")

 

[Grant Taylor]

On 7/21/22 11:48 PM, Don Y wrote:

On such items, its hard for someone (i.e., me) operating in a different
\"market\" (that of an individual buyer vs. a reseller) to sort out a
proper bid for an item.

Ya. You sort of need to have some information ahead of time. E.g.
friends that say I\'ll pay X for Y count of drives Z size. And trust
that you can rely on those numbers days / weeks / months later.

I\'ve developed software to automate this task so groups who want to
(or MUST!) process lots of disks can do so in a short period of time
with unskilled labor

I am curious:

1) Do you leverage the ATA, et al., command to tell the drive to wipe
itself at all? Maybe as a first stab (not even a pass)? Or do you
simply re-write specific things to the drives X number of times?

2) Do you see people / organizations starting to use encryption to make
it so that they don\'t /need/ to wipe the drives? Meaning that they can
just dispose of the key and avoid wiping the drive?

But I would assume those are targeted to individuals, not businesses.
Would you want to have your organization\'s data running on drives
with no history (nor recourse in the event of failure)?

What operational history do you have on brand new drives from the
manufacturer?

What recourse do you have on brand new drives that fail under warranty
from a data point of view?

I\'d think that both are really quite comparable. Effectively no
historical data about the drive and drive replacement, respectively.
The biggest difference is the warranty to cover drive replacement and
the MTBF counters which suggest likely avoidance of failure.

I do feel like used drives probably avoid infant mortality verses brand
new drives.

I have a couple *hundred* drives pulled from similar shelfs, over
the years. I have 7 15-drive shelfs along with a pair of 12-drive
shelfs in my SAN. And, a little 4 drive enclosure tethered to my VM
server (has 8 internal drives).

Nice.

Anything smaller than 500GB gets recycled (the space it takes up
could better be used by a 1TB or larger drive!)

Fair enough.

Though, I see some use in drives that are smaller than 514 MB / 1 GB / 2
GB. Particularly in retro-computing where old systems don\'t like larger
drives. However those are well below your 500 GB marker.

> So, you\'re preaching to the choir.

But, at this point, the drives would just be more clutter, for me
(I think I have 200-300TB, here). My recent efforts have focused on
finding ways to \"store\" things inside other things. E.g., instead
of keeping a box of spare DVD drives, *install* them in various
workstations and discard the box; instead of keeping boxes of DRAM,
fully populate each machine and discard the excess; etc.

I can understand and appreciate that.

But, the owner has made the decision that the risk incurred from
hardware failures is exceeded by the cost of trying to migrate all
of his systems to a \"newer\" platform...

I question the veracity of the premise of the owner\'s decision. But
there are far too few details to actually speculate.

...given that a newer platform will only be \"newer\" for a short period
of time!

As (Open)VMS is finding out, migrating from Itanium to x86-64 means that
the VMs will be able to migrate from generation to generation of x86-64
hosts. Meaning that the OS image is decoupled from the underlying
hardware that it\'s running on.

Did your 5 year old hypervisor host die? Install the same or current
version of the hypervisor on a new host, import the VMs, and you\'re most
of the way there.

Conversely waiting on new old stock / etc. to ship / test / look for
parts, all takes time and is a risk.

[OTOH, he has recently realized that he is *overly* dependant on said
IT guy to keep his enterprise running!]

It sounds like his SPOF might be on the person more than the equipment.
Does the phrase \"Greyhound Buss syndrome\" mean anything? -- I hope
it\'s not the case. But it is a risk.

E.g., I keep SCA and FC/AL drives as spares for some of my older kit.
I even have some 2.5\" SCA and SCSI drives for \"unusual uses\". But, if
all of them failed (before I had a chance to recover their contents),
I would be \"disappointed\"... but not out of business!

ACK

Yes. But someone has to show up at the auction to bid on said items.
IME, most of these folks are resellers trying to pick up <something
at a bargain price and flip it to someone else who doesn\'t have time
(patience) to scour auction sites.

Yep. There is a business niche for such resellers. Especially if the
resellers do some testing on the drives / equipment and offer some form
of warranty. Even if the warranty is a gift card for the venue they
sell things through.

I\'m an exception in that I see these outings as entertainment; sort of
like wandering through an (old-fashioned) hardware store exploring the
various unusual tools and supplies on-hand and pondering their uses.

I\'ve done similar in the past. Sadly I\'ve not been in physical
proximity to such for quite a while.



--
Grant. . . .
unix || die

 

[Don Y]

On 7/22/2022 12:57 PM, Grant Taylor wrote:

On 7/21/22 11:48 PM, Don Y wrote:
On such items, its hard for someone (i.e., me) operating in a different
\"market\" (that of an individual buyer vs. a reseller) to sort out a proper
bid for an item.

Ya. You sort of need to have some information ahead of time. E.g. friends
that say I\'ll pay X for Y count of drives Z size. And trust that you can rely
on those numbers days / weeks / months later.

Not happening.

Folks will always *say* \"keep your eyes open for XYZ; I\'m looking for one\".
And, when you later phone them to tell them you\'ve put one aside for them,
they never manage to follow through.

So, I just tell them \"sure\", when the request is made. Then, promptly
forget it. How are they gonna know that I found one and didn\'t act
on it?

I\'ve developed software to automate this task so groups who want to (or
MUST!) process lots of disks can do so in a short period of time with
unskilled labor

I am curious:

1) Do you leverage the ATA, et al., command to tell the drive to wipe itself
at all? Maybe as a first stab (not even a pass)? Or do you simply re-write
specific things to the drives X number of times?

No. There have been reports of drives that don\'t properly implement that
feature.

Additionally, I want to observe (quantify) the drive\'s performance to decide
if it is UNreliable, etc.

When I first \"see\" a drive (query make/model/sn/capacity and check against
database of \"all drives ever encountered\"), I make a record of its existence.
Remove the HPA and DCO to ensure EVERYTHING is exposed/accessible. Then,
check the SMART data to see if there may be sectors (that *might* have \"user
data\" stored on them, even though not technically accessible); if so, I
can\'t guarantee that everything is overwriteable and mark the drive for
physical destruction (this to guard against someone \"enterprising\" who
finds a way to reset the grown defect table to go snooping around;
ditto for the HPA/DCO).

Then, I begin a series of write/read-verify passes, as determined by the
donor of said drive. (some corporate entities insist on particular
overwriting processes -- multiple passes). While I\'m making a pass,
I note the time per operation as a way of infering if the drive is
having problems internally -- possibly retrying an operation or
remapping a sector to accommodate it.

If the operation fails (returns an error), then I note that fact
and decide if I should abort the process, opting for physical
destruction, instead.

At the end of each pass, I recheck the SMART data to see if anything
noteworthy has changed which might affect my declaration of compliance
with the specified erasure procedure.

I then move on to the next pass -- which may be read-verify or write
something else. I support \"all zeroes\", \"all ones\" (oxFF) and \"random\".
There are corresponding verify cycles for each (so I can write
random to fill the platter and then *verify* that everything was \"as
written\" -- by reconstitutiong the PRNG from its initial seed)

I have a custom OS so I don\'t have the overhead of the traditional
block/character device stack; the code sits *in* the drivers almost
continuously (just pushing data out to the RDBMS as needed). E.g., there
are hooks to \"write_random()\" and \"write_zeroes\" and \"verify_random()\"
instead of just \"write()\" and \"read()\", etc.

If the drive is unsuitable for reuse (too small, too slow, etc)
then a disposal directive informs the operator what to do with the
drive when removed from the system.

Later, when the drive is installed in a refurbished machine, the
RDBMS is again consulted to verify that the drive *was* completely
wiped (and NOT marked for physical destruction). If OK, then
I can write a \"machine image\" onto the drive (instead of having
to manually install an OS and set of apps on *each* machine)

[A delightful side-effect of all this is that we can \"service\"
a machine simply by reinitializing its drive image -- \"Sorry,
all of your personal information is lost!\"]

All of this lets me impose policy without requiring the \"operator\"
to be diligent.

The present shift to laptops poses a different set of problems;
you don\'t want to bother removing the drives -- because \"unskilled\"
(developmentally disabled) workers will not be able to deal with
all the little fasteners -- which are often irreplaceable -- and
more delicate assemblies. So, you let the laptop do the work
for you; netboot an image that gathers identifying information
from the laptop (and disk), wipes the drive, tests the hardware
(and display/keyboard -- the biggest problem areas) and reinstalls
a new image.

But, this takes up a shitload more space than a bunch of rack-mounted
disk shelfs! And, the drives tend to be slower so the \"machine
throughput\" (number of machines refurbished per unit time) drops
significantly.

2) Do you see people / organizations starting to use encryption to make it so
that they don\'t /need/ to wipe the drives? Meaning that they can just dispose
of the key and avoid wiping the drive?

No. I suspect this is a support problem for our corporate donors; keeping
track of keys for thousands of seats. Individual donors (a small portion
of machines, compared to the corporate donors) may or may not have used
encryption, depending on their level of \"novelty\" (sort of like folks using
RAID for its novelty -- until they see the true costs!)

But I would assume those are targeted to individuals, not businesses. Would
you want to have your organization\'s data running on drives with no history
(nor recourse in the event of failure)?

What operational history do you have on brand new drives from the manufacturer?

The implication that there has been *no* wear.

What recourse do you have on brand new drives that fail under warranty from a
data point of view?

Businesses typically impose policies regarding how data is protected --
backups, redundant arrays, etc. They likely want expediency in recovering
data and wouldn\'t rely on a manufacturer\'s service to do that for them.

OTOH, they can get the drives replaced AND have leverage over the
manufacturer in future purchases.

By contrast, *I* have policies and procedures to safeguard *my* data...
even if stored on \"used\" drives. But, I have no recourse with the
\"supplier\" as to complaints about high failure rates, etc.

I\'d think that both are really quite comparable. Effectively no historical
data about the drive and drive replacement, respectively. The biggest
difference is the warranty to cover drive replacement and the MTBF counters
which suggest likely avoidance of failure.

I do feel like used drives probably avoid infant mortality verses brand new
drives.

The drives I\'ve rescued (and that we\'ve refurbished) either work or
don\'t. Failures are often spindle motor or head load assy.

The actual media seems to be pretty durable as we don\'t see many
sector remapping events.

OTOH, corporate donations likely have only 10-20K PoHrs owing to
how frequently they \"repurchase everything\". And, a \"consumer\"
probably even less wear -- aside from physical abuse.

I have a couple *hundred* drives pulled from similar shelfs, over the years.
I have 7 15-drive shelfs along with a pair of 12-drive shelfs in my SAN.
And, a little 4 drive enclosure tethered to my VM server (has 8 internal
drives).

Nice.

Easier than maintaining different machines for different project/purposes.
When done with a project, image the entire machine and move it into a
VMDK -- prior to \"building\" a new configuration for the next project.

Anything smaller than 500GB gets recycled (the space it takes up could better
be used by a 1TB or larger drive!)

Fair enough.

Though, I see some use in drives that are smaller than 514 MB / 1 GB / 2 GB.
Particularly in retro-computing where old systems don\'t like larger drives.
However those are well below your 500 GB marker.

I have a small number of VERY small PATA drives. E.g., my Portable 386
has a 300MB drive hacked into it (the BIOS doesn\'t support that particular
drive type but I rewrote the BIOS ROMs to accommodate it)

So, you\'re preaching to the choir.



But, at this point, the drives would just be more clutter, for me (I think I
have 200-300TB, here). My recent efforts have focused on finding ways to
\"store\" things inside other things. E.g., instead of keeping a box of spare
DVD drives, *install* them in various workstations and discard the box;
instead of keeping boxes of DRAM, fully populate each machine and discard the
excess; etc.

I can understand and appreciate that.

It\'s easy to forget how much \"stuff\" one has accumulated. E.g., I probably
have 40 pounds of RAM (sorted by physical type, technology, size, etc.).
The quantity acts as a deterrent to sorting through it to decide what
to keep! <frown>

Ditto SCSI/SAS HBAs, RAID cards, NICs, GPUs, etc. Easier to just *add* to
existing boxes of parts than it is to sit down and make sense of it all!

But, the owner has made the decision that the risk incurred from hardware
failures is exceeded by the cost of trying to migrate all of his systems to a
\"newer\" platform...

I question the veracity of the premise of the owner\'s decision. But there are
far too few details to actually speculate.

...given that a newer platform will only be \"newer\" for a short period of time!

As (Open)VMS is finding out, migrating from Itanium to x86-64 means that the
VMs will be able to migrate from generation to generation of x86-64 hosts.
Meaning that the OS image is decoupled from the underlying hardware that it\'s
running on.

Did your 5 year old hypervisor host die? Install the same or current version
of the hypervisor on a new host, import the VMs, and you\'re most of the way there.

Exactly.

Did your 5 year old *PC* die? Good luck finding a replacement that will
use the same drivers, etc. (assuming you\'re stuck in the Windows world)

Conversely waiting on new old stock / etc. to ship / test / look for parts, all
takes time and is a risk.

[OTOH, he has recently realized that he is *overly* dependant on said IT guy
to keep his enterprise running!]

It sounds like his SPOF might be on the person more than the equipment. Does
the phrase \"Greyhound Buss syndrome\" mean anything? -- I hope it\'s not the
case. But it is a risk.

He\'s brought on additional staff to address that \"liability\". But, still
spends scant little on IT (i.e., TWO people, now, instead of just the one!)

E.g., I keep SCA and FC/AL drives as spares for some of my older kit. I even
have some 2.5\" SCA and SCSI drives for \"unusual uses\". But, if all of them
failed (before I had a chance to recover their contents), I would be
\"disappointed\"... but not out of business!

ACK

Yes. But someone has to show up at the auction to bid on said items. IME,
most of these folks are resellers trying to pick up <something> at a bargain
price and flip it to someone else who doesn\'t have time (patience) to scour
auction sites.

Yep. There is a business niche for such resellers. Especially if the
resellers do some testing on the drives / equipment and offer some form of
warranty. Even if the warranty is a gift card for the venue they sell things
through.

And, often the prices they charge well exceed what said kit sold for \"new\".
But, folks reliant on it will pay whatever they have to!

I\'m an exception in that I see these outings as entertainment; sort of like
wandering through an (old-fashioned) hardware store exploring the various
unusual tools and supplies on-hand and pondering their uses.

I\'ve done similar in the past. Sadly I\'ve not been in physical proximity to
such for quite a while.

There are lots of opportunities for similar \"recreations\", here.

Several different organizations are involved in \"recycling\" kit
so what you encounter can vary based on the donors each attracts.

The University (host of said auction) tends to have lots of
experimental stuff so it\'s a good mental exercise to ponder
the purpose of a random pile of \"junk\" (corporate donors
tend to just produce more-of-the-same workstation stuff
with a bit of server-side kit mixed in). Retail donors
(storefronts) obviously use yet another class of kit.

I was temped by a \"monitor wall\" some time ago -- five \"rows\"
of three side-by-side monitors (15 total). But, figured it
was impractical (the lowest monitor was very close to floor
level).

Still, it gets your imagination churning contemplating what
it *was* used for and what you *could* use it for!