J
Jeff Liebermann
Guest
On Fri, 27 Sep 2019 06:58:24 +0000 (UTC),
DecadentLinuxUserNumeroUno@decadence.org wrote:
I don't know. I don't run a Linux VM under Windoze. More commonly, I
run a Windoze VM under Linux.
> If so, one would think that dd is the fastest, properly executed.
I don't see the connection or the reason why you insist on throwing
Linux into a Windoze discussion. The original problem was identifying
hard disk drives under Windoze, cloning those drives under Windoze,
and possibly backing them up under Windoze. If you want to offer
Linux as an alternative solution to the OP's problems, please
continue.
> I have yet to test large file moves (movie files of 2 to 6 GB).
If you do test large file copies under Windoze or Linux, and are using
an SSD, you will see some interesting performance problems once your
fill the SSD cache space. Start at 10:09 for "big write test":
<https://youtu.be/WACyyFF_ci0?t=609>
Then go back to help understand what's happening.
Hint: It's not so much the software as it is the drive for large file
copying.
Methinks the correct strategy for maximum speed is to use as much RAM
as can be allocated for FIFO buffering and use semaphores to control
the flow. Using the RAM for a VM which then allocates FIFO buffering
from the RAM available to the VM, does not seem like a winning
strategy. Or, more simply, use as much RAM and CPU clock cycles for
moving data between the source drive and destination backup drive as
possible. Some backup programs do that by booting from a flash drive
or CDROM so as not to waste CPU cycle on the operating system,
background programs, and anything that might consume RAM and CPU
cycles best spent on the cloning or backup program.
In the distant past, I maintained a fair number of customers running
various SCO products (Xenix, ODT, OSR 5, Unixware, etc). Backups were
to various size QIC tape drives. I learned rather quickly that the
slowest and most inefficient program available was dd. I spent
considerable time optimizing block and buffer sizes, as well as
experimenting with raw filesystems. Cpio and tar were useless because
a 1 bit error anywhere in the file, would render the backup or copy
useless. If I wanted something fast and reliable (but not cheap), I
used the "super tar" programs, such as Lone-Tar and Backup Edge. Of
course, hardware and software have both improved in the last 25 years,
so todays programs and machines will be much faster. Meanwhile, dd
has not improved much, and is still very slow.
Now, back to Windoze. This is a sample of last months image backup of
my Windoze 7 C: partition. The source drive is 1 TB Seagate SATA
drive. The backup destination is a Western Dismal USB3 drive. The
machine is a junk HP m9077c desktop with a Core 2 Quad Q6600 CPU and
8GB of RAM.
<http://www.learnbydestroying.com/jeffl/crud/Macrium%20Reflect%20Log%20File.pdf>
Notice that it backed up 97GB in 17:08 minutes or 5.7 GigaBytes per
minute backup speed. It also compressed the backup file down to 47GB.
On faster machines and using SSD drives, I've seen 10GBytes per minute
and faster. That's fast enough for even the most backup resistant
customer to not consider running an occasional back a major imposition
on their hectic lifestyle. Note that the backup was done as an
application under Windoze 7 and NOT from a bootable CD or flash drive.
--
Jeff Liebermann jeffl@cruzio.com
150 Felker St #D http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann AE6KS 831-336-2558
DecadentLinuxUserNumeroUno@decadence.org wrote:
I don't know. I don't run a Linux VM under Windoze. More commonly, I
run a Windoze VM under Linux.
> If so, one would think that dd is the fastest, properly executed.
I don't see the connection or the reason why you insist on throwing
Linux into a Windoze discussion. The original problem was identifying
hard disk drives under Windoze, cloning those drives under Windoze,
and possibly backing them up under Windoze. If you want to offer
Linux as an alternative solution to the OP's problems, please
continue.
> I have yet to test large file moves (movie files of 2 to 6 GB).
If you do test large file copies under Windoze or Linux, and are using
an SSD, you will see some interesting performance problems once your
fill the SSD cache space. Start at 10:09 for "big write test":
<https://youtu.be/WACyyFF_ci0?t=609>
Then go back to help understand what's happening.
Hint: It's not so much the software as it is the drive for large file
copying.
Methinks the correct strategy for maximum speed is to use as much RAM
as can be allocated for FIFO buffering and use semaphores to control
the flow. Using the RAM for a VM which then allocates FIFO buffering
from the RAM available to the VM, does not seem like a winning
strategy. Or, more simply, use as much RAM and CPU clock cycles for
moving data between the source drive and destination backup drive as
possible. Some backup programs do that by booting from a flash drive
or CDROM so as not to waste CPU cycle on the operating system,
background programs, and anything that might consume RAM and CPU
cycles best spent on the cloning or backup program.
In the distant past, I maintained a fair number of customers running
various SCO products (Xenix, ODT, OSR 5, Unixware, etc). Backups were
to various size QIC tape drives. I learned rather quickly that the
slowest and most inefficient program available was dd. I spent
considerable time optimizing block and buffer sizes, as well as
experimenting with raw filesystems. Cpio and tar were useless because
a 1 bit error anywhere in the file, would render the backup or copy
useless. If I wanted something fast and reliable (but not cheap), I
used the "super tar" programs, such as Lone-Tar and Backup Edge. Of
course, hardware and software have both improved in the last 25 years,
so todays programs and machines will be much faster. Meanwhile, dd
has not improved much, and is still very slow.
Now, back to Windoze. This is a sample of last months image backup of
my Windoze 7 C: partition. The source drive is 1 TB Seagate SATA
drive. The backup destination is a Western Dismal USB3 drive. The
machine is a junk HP m9077c desktop with a Core 2 Quad Q6600 CPU and
8GB of RAM.
<http://www.learnbydestroying.com/jeffl/crud/Macrium%20Reflect%20Log%20File.pdf>
Notice that it backed up 97GB in 17:08 minutes or 5.7 GigaBytes per
minute backup speed. It also compressed the backup file down to 47GB.
On faster machines and using SSD drives, I've seen 10GBytes per minute
and faster. That's fast enough for even the most backup resistant
customer to not consider running an occasional back a major imposition
on their hectic lifestyle. Note that the backup was done as an
application under Windoze 7 and NOT from a bootable CD or flash drive.
--
Jeff Liebermann jeffl@cruzio.com
150 Felker St #D http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann AE6KS 831-336-2558