Jump to content











Photo
- - - - -

How to refresh the contents of a flash drive?


  • Please log in to reply
45 replies to this topic

#26 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 10 February 2011 - 02:03 PM

The whole thread reminds me on a non-PC issue:

You buy anything in a shop and get the bill printed by a thermal method on some special paper.

After 2 years the part you bought, does not work any more, and you pick up the bill in order to ask for guarantee.

But the bill now is only a white sheet of paper! :ph34r:

To avoid this: When you get the bill, make a copy with a laser printer. The copy surelly is readable for 20+ years.
Or ask in an according forum: How can I made my "Thermal printed bill" survive more than 2 years? :ph34r:
To be on the safe side (if the copy is lost, burned, etc.) make a copy to a second, third, ... copy

In analogy to that:
Copy the flash drive to a CD. It will be readable for about 10 years.
Or copy it to a HDD. That will be readable for 20+ years.
To be on the safe side (if the HDD / CD crashes) make a copy to a second, third, ... HDD / CD

Peter :cheers:

#27 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 10 February 2011 - 02:12 PM

My re-inquire at SanDisk brought me a bunch of questions from them. :cheers:
Like, what company i work for, how many USB-Stick i intend to buy per year/month and why exactly i would need any technical data about their products. :ph34r:

The point about there being no studies hence no answer, reminded me of a little interesting fact, i read some years ago.
Where it was stated that NASA was still using 386 processors in their space probes, eventhough everyone was using Pentium class CPU at the time.
The solution to the mystery was, NASA missions are long term missions and the newest processors which could actually be verified to work at least 10 years was the 386.
That puts a whole new dimension to not trusing marketing! :ph34r:


:ph34r:

#28 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 10 February 2011 - 02:17 PM

That puts a whole new dimension to not trusing marketing! :ph34r:

:ph34r: :ph34r: :angry7: :ph34r: :cheers:

My personal imperative since 30 years (when I started a job as developer in an American, sales-orientated company)

Peter

#29 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14789 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 February 2011 - 02:21 PM

Or copy it to a HDD. That will be readable for 20+ years.

Hard disk drive manufacturer readily recommend performing a read and re-write on their hard disk drives every 3 years for archival purposes ...


Oh, oh, logical clash detected. :ph34r: :cheers:

@Medevil:
Company: Flash Refreshers Inc.
Estimated: 150 to 250 / month for the initial three years project
Reason: Exploring the possibilities of basing a cloud storage system based on clusters of USB Flash memories to reduce current consumption and heat, thus contributing to a greener planet.


JFYI: I guess it was 8086's (and not yet 386's):
http://www.nytimes.c...ess/12NASA.html

:ph34r:
Wonko

#30 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 10 February 2011 - 02:40 PM

Oh, oh, logical clash detected. :cheers: :dubbio:

Hard disk drive manufacturer readily recommend performing a read and re-write on their hard disk drives every 3 years for archival purposes

That leads to:

To be on the safe side (if the HDD / CD crashes) make a copy to a second, third, ... HDD / CD

and proofs:

That puts a whole new dimension to not trusing marketing! :lol:


Peter :cheers:

#31 steve6375

steve6375

    Platinum Member

  • Developer
  • 6953 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 10 February 2011 - 02:56 PM

The point about there being no studies hence no answer, reminded me of a little interesting fact, i read some years ago.
Where it was stated that NASA was still using 386 processors in their space probes, eventhough everyone was using Pentium class CPU at the time.
The solution to the mystery was, NASA missions are long term missions and the newest processors which could actually be verified to work at least 10 years was the 386.

:cheers:


This is similar to the military manufacturers in the 70's and 80's. They used to keep mil. spec. ICs for a number of years in drawers in cold room. If no problems were reported by the manufacturer for that particular batch of chips, they would then use them in their circuit boards for guided missiles, etc. Which is why the latest 'state-of-the-art' technology was never used by the military. Hopefully, similar precautions are used for building AirBuses today :dubbio:

#32 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14789 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 February 2011 - 02:57 PM

That leads to:

then it won't

Or copy it to a HDD. That will be readable for 20+ years.

(without a refresh every, say, 3 years) :cheers:

If it was NOT clear enough, the point is that a MFM or RLL Hard Disk may (and will) have (with or without refreshing) MUCH different retention times that a newish High Density, perpendicular recording HDD, AND that being them based on mechanical parts (and having been used over the time different motors and expecially bearings) trifling details, like temperature, humidity, nature of bearings, AND what NOT, makes BOTH the one-size-fits-all 20 years and 3 years refresh (evidently being a logical clash) completely moot or, if you prefer "yet another two finger in air stabs".

Also, a drive that has init and calibration data stored in a Flash chip will behave differently from one that actually stores it on a track? :dubbio:

And HOW the HECK are you going to refresh (every three years) the actual Flash chip on the HDD? :cheers:

More generally there is NO such thing as "a" hard disk.... :lol:

:cheers:
Wonko

#33 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 10 February 2011 - 03:04 PM

And HOW the HECK are you going to refresh (every three years) the actual Flash chip on the HDD? :dubbio:

I would NEVER try to refresh.
But when the Flash fails, I have the HDD copies #1, #2, # ...
Which in my opinion remain for 20+ years, after you deposited the HDD into a (water-proof, radioactive-proof, UV-proof, ???-proof) place.

Peter :cheers:

#34 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14789 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 February 2011 - 03:16 PM

Which in my opinion remain for 20+ years, after you deposited the HDD into a (water-proof, radioactive-proof, UV-proof, ???-proof) place.


Still the point remains the same, does your opinion consider that ANY and ALL drive types will have the SAME "guaranteed" 20+ years data retention time, including those that rely - in order to spin-up - on some data held on a Flash chip?
Then in your opinion also Flash chips - all of them - have a data retention of 20+ years.
And ALL kind of bearings ever used in hard disk manufacture do have the same capability of NOT jamming if kept 20+ years WITHOUT any spin-up? :cheers:

What a wonderful world - full of certainties - you live in. :lol: :cheers:


OT :cheers: and JFYI:
http://serverfault.c...val-deteriorate
(and probably because I am unlucky I have a personal very high record of failed tapes after years - additionally)

And of course this is needless worry :dubbio: :
http://en.wikipedia....al_preservation

Some real world cases in this nice article:
http://www.popularme...ts/news/4201645

:cheers:
Wonko

#35 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 10 February 2011 - 04:27 PM

Still the point remains the same, does your opinion consider that ANY and ALL drive types will have the SAME "guaranteed" 20+ years data retention time, including those that rely - in order to spin-up - on some data held on a Flash chip?
Then in your opinion also Flash chips - all of them - have a data retention of 20+ years.
And ALL kind of bearings ever used in hard disk manufacture do have the same capability of NOT jamming if kept 20+ years WITHOUT any spin-up? :lol:

What a wonderful world - full of certainties - you live in. :dubbio: :cheers:

When I say HDD, I mean something which is currently on the market and is rotating. For me a Flash is not a HDD.
Sorry again about my (already known) bad ability to explain in English.

Peter :cheers:

#36 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14789 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 February 2011 - 04:44 PM

When I say HDD, I mean something which is currently on the market and is rotating. For me a Flash is not a HDD.

Again, the misunderstanding seems to me more technical than a language barrier.

Current "rotating things" on the market UNLIKE some older models do have a sort of BIOS and it is stored in a flash chip on the actual HD PCB (Printed Circuit Board).

When you power the HD this firmware (or BIOS) is read and it contains VITAL DATA needed to be able to access properly the DATA on the actual rotating magnetical platters.

IF the data retention of that Flash chip is less than your expected 20+ years, even if ALL the data on the actual magnetic platter is perfectly mantained (and there are NO mechanical problems) you won't be able to access the DATA. :dubbio:

I hope that it is more clear, now. :cheers:

:lol:
Wonko

#37 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 10 February 2011 - 05:48 PM

Again, the misunderstanding seems to me more technical than a language barrier.

Current "rotating things" on the market UNLIKE some older models do have a sort of BIOS and it is stored in a flash chip on the actual HD PCB (Printed Circuit Board).

When you power the HD this firmware (or BIOS) is read and it contains VITAL DATA needed to be able to access properly the DATA on the actual rotating magnetical platters.

IF the data retention of that Flash chip is less than your expected 20+ years, even if ALL the data on the actual magnetic platter is perfectly mantained (and there are NO mechanical problems) you won't be able to access the DATA. :dubbio:

I hope that it is more clear, now. :cheers:

:lol:
Wonko

OK, I'm thinking to write manually all my critical data onto a fireproof, waterproof, xxxproof piece of paper and put it into a fireproof, .... safe.
:cheers:

Peter

#38 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14789 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 February 2011 - 06:19 PM

OK, I'm thinking to write manually all my critical data onto a fireproof, waterproof, xxxproof piece of paper and put it into a fireproof, .... safe.

Just for the record , historically Clay Tablets have proved to provide better data retention than ink on paper:
http://en.wikipedia....iki/Clay_tablet

You doon't even need a fireproof storage, they are intrinsecally fireproof. :dubbio:

:cheers:
Wonko

#39 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 10 February 2011 - 06:23 PM

Just for the record , historically Clay Tablets have proved to provide better data retention than ink on paper:
http://en.wikipedia....iki/Clay_tablet

You doon't even need a fireproof storage, they are intrinsecally fireproof. :dubbio:

:cheers:
Wonko

Many thanks for the hint!

Peter :lol:

#40 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 10 February 2011 - 08:30 PM

With data retention times of waaaaay less than a human lifespan, maybe we should develop a new device, which etches our data with lasers in stone tablets?

btw. data retention for HDD just 20 or even 3 years???? I remember 100 years.

:dubbio:

#41 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 10 February 2011 - 08:42 PM

btw. data retention for HDD just 20 or even 3 years???? I remember 100 years.

I mentioned 20+.
BTW: I think that 100 fullfills 20+ ... :dubbio:

Peter :cheers:

#42 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 10 February 2011 - 09:21 PM

BTW: I think that 100 fullfills 20+ ... :cheers:

Yes, but does 20+ fullfill 100? :dubbio:

:lol:

#43 eyuri

eyuri
  • Members
  • 2 posts
  •  
    Greece

Posted 27 September 2013 - 08:00 AM

Like usb!!!

Well what we need here is a disc utility that does

1.read data of usb. (all blocks of data)

2.write them back (refreshing)

3.read them back (identify bad block or read errors)

Personaly i use favorite "hard disc sentinel" but i believe if you search for more hard disc utils... you will find many of them.

Cheers fro Greece.



#44 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14789 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 27 September 2013 - 08:21 AM

Like usb!!!

Well what we need here is a disc utility that does

1.read data of usb. (all blocks of data)

2.write them back (refreshing)

3.read them back (identify bad block or read errors)

Personaly i use favorite "hard disc sentinel" but i believe if you search for more hard disc utils... you will find many of them.

Cheers fro Greece.

Well, we have found it since a few years,it is called DSFOK (but *any* dd-like tool will do), you can find it here:

http://members.ozema...eezip/freeware/

It is perfectly possible to script what you depicted with it but you need a temporary external storage, to make sure that in case of "disaster" you have the "temporary copy".

I.e. the "safe" procedure is:

  1. Image the device on other (temporary) media (dsfo will provide the MD5 of the transferred data)
  2. read the data on the temporary device (and check it's MD5)
  3. write the data on the "original device" (and check the MD5 of the transferred data)
  4. read the data on the newly made "original device" (and check it's MD5)

:cheers:

Wonko



#45 steve6375

steve6375

    Platinum Member

  • Developer
  • 6953 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 27 September 2013 - 10:16 AM

In step 3, do you mean write the original data? That would not work.

Consider faulty flash memory where there is actually  no memory or where bits which are stuck at 1 or 0. Your test would pass!

 

What you need to do is test for 'stuck bits' - e.g. read original data - invert (i.e. NOT) the bits and write it back - then check by reading back all data and comparing. Finally read - invert (NOT) and write to restore the original data (hopefully!). It is easier to use a test pattern rather than the original data.



#46 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14789 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 27 September 2013 - 11:15 AM

In step 3, do you mean write the original data? That would not work.

Ah, well, if you say so.
 

Consider faulty flash memory where there is actually  no memory or where bits which are stuck at 1 or 0. Your test would pass!

The idea is NOT to test the hardware, it is to test the data.
If the data has a 0 at offset n AND that byte is stuck at 0, it is OK, as long as it is readable as 0.

Medevil was trying to make sure that a flash stick can be used as WORM and that it does not "loose" data if not powered periodically.

 

In #1 the "original data" is read from "original device" and it's MD5 is calculated WHILE transferring the data to temporary media.
In #2 the "copy of original data" is read and verified through it's MD5 hash.
In #3 the "copy of original data" is read from "temporary media" and it's MD5 is calculated WHILE transferring the data back to "original" media.

In #4 the "newly written" (but original) data is read from "original media" and it's MD5 hash is calculated

 

At the end of the day, you are certain that you can read the data from the device, AND you just rewrote it, triggering (if needed) any wear leveling feature of the stick controller.

How actually the controller writes the sectors, bytes or pages, checks and writes ECC data, etc. as said earlier may greatly depend on the controller, the exact kind of memory, the actual organization of chips in the stick, etc., etc.

 

What you are planning goes a little further, you are going to "exercise" the stick at bit level.

 

What you need to do is test for 'stuck bits' - e.g. read original data - invert (i.e. NOT) the bits and write it back - then check by reading back all data and comparing. Finally read - invert (NOT) and write to restore the original data (hopefully!). It is easier to use a test pattern rather than the original data.

You can well insert - between steps #2 and #3 two passes, one with 55AA's and one with AA55's (or one with 55's and one with AA's), in order to flip at bit level.

 

We would need someone that actually writes an ALL55 (and ALLAA or a ALL55AA, and ALLAA55) MD5 generator, see semi-random idea proposed in this seemingly (actually ;)) unrelated topic:

http://www.forensicf...opic/p=6569139/

 

:cheers:

Wonko






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users