Jump to content











Photo
- - - - -

Extents

file copy in use

  • Please log in to reply
104 replies to this topic

#51 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 03 April 2019 - 07:16 PM

....

To give you an example for clonedisk (where the toggling of "advanced" menu   :thumbsup: greatly mitigates the overload :)):

Can it make a sparse image?

No, still that would be something that is part of what one would want when imaging a disk or volume, it would be both a corollary and a feature.

....

 

Side note : yes CloneDisk can make a sparse image :)

In the disk image menu (bottom right).



#52 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 03 April 2019 - 07:35 PM

I got as far as retrieve the file record address on the logical disk (C0009C00 in my case).
Now what would be the next step ? convert this logical disk offset into LBA?


Yes and no.
 
That C0009C00 is the $MFT entry offset in bytes.
 
Within that $MFT entry the first byte of the file will have a relative offset.
Then you can convert from byte offset to LBA and add the info for the relative offset.
 
This is what the grub4dos blocklist command does (posted example before) :
 
 
 

Example of grub4dos command blocklist (same file, another volume):
blocklist /myMFTembed.txt
(hd0,0)171493[288-295]
and surely enough, on LBA 171493 on that disk volume there is a $MFT entry that contains, starting at offset 288 the 7 bytes "pippo[CR+LF]".

 
though the LBA returned in that case is the relative LBA of the volume, whilst you want (just like the current extents program does, provide the absolute LBA from the beginning oif the disk), i.e. add the "sectors before" the volume.
 
:duff:
Wonko

#53 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 03 April 2019 - 07:48 PM

Side note : yes CloneDisk can make a sparse image :)
In the disk image menu (bottom right).

Yep :) .
But if I click on Clone -> Backup to image, it doesn't seemingly offer the option.

 

It is debatable whether the:

Clone -> Backup to image -> Format RAW

 

Represents EXACTLY the same as:

Disk Image > Make a RAW disk image

 

seemingly they are actually different names (and different logical paths, the first where the source "prevails", the second where the target "prevails")  for the same activity, ideally either only one of the two should exist, or if not (and I don't see keeping them as they are, duplicated, as a "bad" thing) they should IMHO converge back to a "same" "Save as" dialog.  

 

 

:duff:

Wonko



#54 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 06 April 2019 - 01:56 PM

1.2 released.

 

While playing with MFT resident/non resident file, I found out that file with 1 cluster only where not handled correctly.

Bug fixed...



#55 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 06 April 2019 - 07:21 PM

1.3 released

 

Added 2 functions : translatephy  and translatelog to convert physical offset to logical and vice et versa.

extents-win64.exe translatephy e: 28445032448
extents 1.3 by erwan2212@gmail.com
physical : 28445032448
logical : 28443983872 bytes -> 55554656 sectors
Physical-logical : 1048576

extents-win64.exe translatelog e: 28443983872
extents 1.3 by erwan2212@gmail.com
logical : 28443983872
Physical : 28445032448 bytes -> 55556704 sectors
Physical-logical : 1048576


#56 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 07 April 2019 - 02:38 PM

Added 2 functions : translatephy  and translatelog to convert physical offset to logical and vice et versa.

WHY? :w00t:

I mean :), those functions seemingly translate bytes to absolute (from the beginning of the whole disk, actually the "only" proper LBA) or relative (from the beginning of the volume) LBA's.

But (unless something has been changed) the typical output of extents was:

 

extents_[0] - VCN : 0 LCN : 32014747 Lba : 256118039 Sectors : 40

i.e data expressed in:

1) VCN/LCN -> clusters
2) Lba/Sectors->sectors

or if you prefer, nothing expressed in bytes. :unsure:

 

also, maybe it is just me, but I instinctively read "Physical-logical" in a *somehow* "mixed-mode" way. as if it was a "dual-mode" 

 

At the end of the day,the result of

extents-win64.exe translatephy e: 28445032448

is

Physical-logical : 1048576

and the result of 

extents-win64.exe translatelog e: 28443983872

is as well

Physical-logical : 1048576

 

both could IMHO be:

extents-win64.exe howmanysectorsbefore e:

resulting in:

Sectors_Before : 1048576

 

Anyway, as a side-side note ;), you NEVER, and I mean NEVER use the variable "%" and "~", redirection "<", ">", escape "^", piping "|" or joining "&" characters in the output of a command line tool[1] (unless you want to make life tougher for people using batch to parse the output, of course :whistling:)

 

 

:duff:

Wonko

[1] And also the use of quotes should be attentively evaluated, all three forms of them, the double quote Ascii 0x22. single quote Ascii 0x27 and  back quote Ascii 0x60



#57 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 07 April 2019 - 02:58 PM

About the ">", damn I did it :)

I removed it in latest version : just pretend it never happened.

 

About the bytes unit used in translatelog/translatephy, I am not sure I get it?

The idea is to input a byte offset and get another byte offset.

What would be a better alternative? sectors? clusters?

You make a point thus that I many times take a LCN, multiply it by the clustersize to get a byte offset which I then input to translatelog...and eventually convert it again into sectors (LBA) :)

So inputing a value in clusters or sectors might be a more straightforward way of doing.

Is this something amongst this line that you actually meant?

 

I also renamed physical minus logical to "sectors-before".

About this part, one might say that translating logical to physical (or the other way around) is actually as simple as use this "sectors-before" information you can get either get from the MBR or from the BS but is that that simple?

There has to be some special cases as I doubt MS would have bothered to come with two IOCTL (IOCTL_VOLUME_PHYSICAL_TO_LOGICALIOCTL_VOLUME_LOGICAL_TO_PHYSICAL) "only" for a simple addition/soustraction operation?

>extents-win32.exe translatelog e: 28443983872
extents 1.3 by erwan2212@gmail.com
logical : 28443983872
Physical : 28445032448 bytes - 55556704 sectors
Sectors-Before : 1048576


#58 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 07 April 2019 - 04:09 PM

About the ">", damn I did it :)
I removed it in latest version : just pretend it never happened.

Oww, come on :), you cannot make impossible requests.

When we will be relaxing, on a public park bench with our canes (and one or two caregivers keeping an eye on us) what do you think I will use (if not "Hey, erwan.l you remember that time you used a ">" in a command line tool output") to tease you?

Apart - of course - the "dancing tabs" ;)
 

About the bytes unit used in translatelog/translatephy, I am not sure I get it?
The idea is to input a byte offset and get another byte offset.
What would be a better alternative? sectors? clusters?
You make a point thus that I many times take a LCN, multiply it by the clustersize to get a byte offset which I then input to translatelog...
So inputing a LCN for example would simple more straightforward.


I understand that, but the main scope of extents remains (or should remain) that of providing:
a. an offset to the beginning of a file segment
b. the length of that segment
c. if the file is not contiguous, the next file segment and loop to a.

Now the point that one may argue upon is the unit in which both the offsets and lenghts are output and where from (disk or volume) the offset is calculated.

In batch there are serious limitations in math operations, and you cannot just multiply sectors or clusters by their size as the result will usually overflow, you would need to add a "multiply" (and an add and and a subtract and a divide) function to allow this kind of math.

I personally would see more useful something *like* (only hypothetical-mockup output):
extents-win32.exe /?
extents 102.14 by erwan2212@gmail.com - final version finally approved by Wonko the Sane
extents [options [units]] <file>
Options:
/LBA or /ABS (default) provides offsets from beginning of disk and lengths (implies units=/SECT)
/REL provides offsets from beginning of volume and lengths (implies units=/SECT)
Mode:
/S (default) provides what is asked for and nothing more
/V provides additional info on the filesystem and volume
/VV provides additional info on the disk, filesystem and volume
Units (can override the implied units in options):
/SECT expressed in sectors
/CLUS expressed in clusters
/BYTE expressed in bytes





C:\appoggio\Alcor>extents-win32.exe LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
extents_[0] - StartLba : 256118039 Length : 40

C:\appoggio\Alcor>extents-win32.exe /LBA LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
extents_[0] - StartLba : 256118039 Length : 40

C:\appoggio\Alcor>extents-win32.exe /LBA /SECT LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
extents_[0] - StartLba : 256118039 Length : 40

C:\appoggio\Alcor>extents-win32.exe /LBA /CLUS LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
A suffusion of yellow ...

C:\appoggio\Alcor>extents-win32.exe /LBA /BYTE LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
extents_[0] - Bytes_Before : 256118039 Length : 20480


C:\appoggio\Alcor>extents-win32.exe /REL LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
extents_[0] - StartCluster : 32014747 Length : 5

C:\appoggio\Alcor>extents-win32.exe /REL /CLUS LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
extents_[0] - StartCluster : 32014747 Length : 5

C:\appoggio\Alcor>extents-win32.exe /REL /BYTE LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
extents_[0] - Bytes_Before : 256085783 Length : 20480

C:\appoggio\Alcor>extents-win32.exe /REL /SECT LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
Filename:C:\appoggio\Alcor\LoadDriver.exe
Extents count :1
extents_[0] - Sectors_Before : 256117976 Length : 40

C:\appoggio\Alcor>extents-win32.exe /V LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
***************************
Bytes Per Sector:512
Sectors per Cluster:8
Cluster size :4096
Filesystem :NTFS
Sectors Before this volume: 63
Reserved sectors: 0
***************************
Filename:C:\appoggio\Alcor\LoadDriver.exe
File Cluster count :5 (20480 bytes)
File size in bytes :20480
File cluster first :32014747
Extents count :1

extents_[0] - VCN : 0 LCN : 32014747 Lba : 256118039 Sectors : 40


C:\appoggio\Alcor>extents-win32.exe /VV LoadDriver.exe
extents 102.14 by erwan2212@gmail.com
***************************
This is volume \\?\Volume{83092730-6bfc-11df-b90c-806d6172696f}\
It has assigned drive letter C:\
This volume is on physicaldrive: 0
This volume starts at LBA offset 63
Bytes Per Sector:512
Sectors per Cluster:8
Cluster size :4096
Filesystem :NTFS
Sectors Before this volume: 63
Reserved sectors: 0
Sectors to add or subtract to convert from Physical to Logical:
63+0=63
Same as above in bytes:32256
This machine is running: XP with Service Pack 2
The processor is a tad bit slow, rating: 3
RAM is 4 GB but you cannot use all of it
You have a USB stick connected to port #4
Of course the last 4 lines above are just featuritis
***************************
Filename:C:\appoggio\Alcor\LoadDriver.exe
File Cluster count :5 (20480 bytes)
File size in bytes :20480
File cluster first :32014747
Extents count :1

extents_[0] - VCN : 0 LCN : 32014747 Lba : 256118039 Sectors : 40

:duff:
Wonko

#59 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 07 April 2019 - 07:33 PM

Oww, come on :), you cannot make impossible requests.

When we will be relaxing, on a public park bench with our canes (and one or two caregivers keeping an eye on us) what do you think I will use (if not "Hey, erwan.l you remember that time you used a ">" in a command line tool output") to tease you?
 

Apart - of course - the "dancing tabs"  ;)

 

Not sure which part of this though is the scariest : us arguing about bytes vs sectors when getting old and senile, carekeepers keeping on eye on us like  they would  with kids or being remembered about the "dancing tabs" until the dawn of time :)

 

I understand that, but the main scope of extents remains (or should remain) that of providing:
a. an offset to the beginning of a file segment
b. the length of that segment
c. if the file is not contiguous, the next file segment and loop to a.

Now the point that one may argue upon is the unit in which both the offsets and lenghts are output and where from (disk or volume) the offset is calculated.

 

Correct.

We are clear on the what (scope).

I am definitely struggling with the how to output this what.

But we are clearly making good progress there and the below is exactly what I need : some guidance as of how to come with a comprehensive yet flexible and easy command line.

 

Thanks a lot for that.

 

I personally would see more useful something *like* (only hypothetical-mockup output):
extents-win32.exe /?
extents 102.14 by erwan2212@gmail.com - final version finally approved


#60 noel

noel

    Frequent Member

  • Advanced user
  • 178 posts
  • Location:nantes
  •  
    France

Posted 29 April 2019 - 06:04 PM

Hello,

I make this preamble as short as possible. Erwan knows me a little bit now. I am very naïve in my questions and my goal is only to always understand better.
I have long hesitated to polluze your conversation because I am not sure that my remarks are useful.

In parallel to your discussion that fascinated me on the "extents ", I also played with  "DeviceIoControl (... FSCTL_GET_RETRIEVAL_POINTERS... )".
And I got to use the PS commands to check my program... kind of like a quiz!

 

My first surprise: Microsoft always complicates everything, it's the game. I am well aware that easy2boot is not concerned with the following.
But the so-called "extended" disks seem to complicate the search for the first LBA of a file on a disk (in addition to the small NTFS files hosted in the MFT).
Also it seems interesting to report in a  "warning" of your program (/V/VV) that the disk is  "fixed" or not, much like the command PS get-disk
idem for "style" = MBR or GPT
Note: this command reports with:
a USB stick that contains 4.8 GB of free space,  "LargestFreeExtent = 0"
and with another USB key, a value of  "LargestFreeExtent " not null
Not important but surprising and i don't see this parameter in a "struct" with "DeviceIoControl"

 

My second surprise playing with the "extents ":
my program reports that a file on my disk has 15 "extents" while the Standart command "fsutil sparse queryRange..." indicates that the file is not fragmented.
  "extents " and fragmentation "s eems to me related. There must be a mistake somewhere.

 

I do not need a solution since I have no worries. But if you still have elements of could help me better understand these peculiarities, thank you for making me enjoy.

Attached Files



#61 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 29 April 2019 - 07:22 PM

My second surprise playing with the "extents ":
my program reports that a file on my disk has 15 "extents" while the Standart command "fsutil sparse queryRange..." indicates that the file is not fragmented.
  "extents " and fragmentation "s eems to me related. There must be a mistake somewhere.

 

I do not need a solution since I have no worries. But if you still have elements of could help me better understand these peculiarities, thank you for making me enjoy.

You program is the "get-blocsDefrag.exe"?

 

It finds 15 sequential extents, i.e. extents that are not separated by either another allocated file or by unallocated space:

 

C:\Windows\system32>"C:\Users\noelb\Desktop\get-blocsDefrag.exe" "C:\Users\noelb\Desktop\Windows.iso"
Param�tre : C:\Users\noelb\Desktop\Windows.iso
Fichier : C:\Users\noelb\Desktop\Windows.iso
hHandle : 148
New Extents : 15
Extent[0] StartLCN:9413798 StartVCN:0 NextVCN:22062 Length:22062
Extent[1] StartLCN:8686784 StartVCN:22062 NextVCN:69282 Length:47220
Extent[2] StartLCN:9346741 StartVCN:69282 NextVCN:136337 Length:67055
Extent[3] StartLCN:9172254 StartVCN:136337 NextVCN:205471 Length:69134
Extent[4] StartLCN:9479487 StartVCN:205471 NextVCN:580896 Length:375425
Extent[5] StartLCN:9444820 StartVCN:580896 NextVCN:605984 Length:25088
Extent[6] StartLCN:9880576 StartVCN:605984 NextVCN:631072 Length:25088
Extent[7] StartLCN:5569624 StartVCN:631072 NextVCN:641824 Length:10752
Extent[8] StartLCN:9916704 StartVCN:641824 NextVCN:707180 Length:65356
Extent[9] StartLCN:5606280 StartVCN:707180 NextVCN:721136 Length:13956
Extent[10] StartLCN:10231832 StartVCN:721136 NextVCN:746224 Length:25088
Extent[11] StartLCN:5394612 StartVCN:746224 NextVCN:753392 Length:7168
Extent[12] StartLCN:10264376 StartVCN:753392 NextVCN:882928 Length:129536
Extent[13] StartLCN:5539512 StartVCN:882928 NextVCN:894192 Length:11264
Extent[14] StartLCN:10405464 StartVCN:894192 NextVCN:1018976 Length:124784

 

I.e. on each line above StartVCN+Length=NextVCN

 

You are not polluting anything, rest assured :) , but it makes to me little sense, on a thread that is about:extents.exe, to post the results of:

1) fsutil sparse queryRange

2) get-blocsDefrag.exe

AND NOT that of extents.exe oon the same file/filesystem.

 

If I get this right, you are asking erwan.l to guess what you have written in your program and tell you why it misbehaves?

 

Now, that is a quiz. :whistling:

 

:duff:

Wonko



#62 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 29 April 2019 - 07:38 PM

@Noel : no polluting at all !
We are all there to learn and share.
I had put on hold extents for a while (i need to rework the command line based on Wonko's inputs) as I have been working on Clonedisk/Vmount.

You are right : a non fragmented file should display only one extents.

See below before/after.

 

Edit : @Noel, it is springtime - we need to have that beer/coffee we talked about soon :)

>extents-win64.exe e:\DISK2.img
extents 1.3 by erwan2212@gmail.com
***************************
Bytes Per Sector:512
Sectors per Cluster:8
Cluster size :4096
Filesystem :NTFS
***************************
Filename:e:\DISK2.img
File Cluster count :131072 (536870912 bytes)
File size in bytes :536870912
File cluster first :14488682
Extents count :19

extents_[0] -  VCN : 0 LCN : 14488682 Lba : 115911504 Sectors : 4400
extents_[1] -  VCN : 550 LCN : 14487422 Lba : 115901424 Sectors : 4472
extents_[2] -  VCN : 1109 LCN : 14664425 Lba : 117317448 Sectors : 12288
extents_[3] -  VCN : 2645 LCN : 24321799 Lba : 194576440 Sectors : 13360
extents_[4] -  VCN : 4315 LCN : 14665993 Lba : 117329992 Sectors : 16384
extents_[5] -  VCN : 6363 LCN : 24923011 Lba : 199386136 Sectors : 17920
extents_[6] -  VCN : 8603 LCN : 18804185 Lba : 150435528 Sectors : 24320
extents_[7] -  VCN : 11643 LCN : 25052142 Lba : 200419184 Sectors : 4480
extents_[8] -  VCN : 12203 LCN : 14489779 Lba : 115920280 Sectors : 24440
extents_[9] -  VCN : 15258 LCN : 14676249 Lba : 117412040 Sectors : 32768
extents_[10] -  VCN : 19354 LCN : 21342869 Lba : 170745000 Sectors : 32768
extents_[11] -  VCN : 23450 LCN : 21347061 Lba : 170778536 Sectors : 32768
extents_[12] -  VCN : 27546 LCN : 21351253 Lba : 170812072 Sectors : 32768
extents_[13] -  VCN : 31642 LCN : 25357125 Lba : 202859048 Sectors : 36936
extents_[14] -  VCN : 36259 LCN : 14586972 Lba : 116697824 Sectors : 4488
extents_[15] -  VCN : 36820 LCN : 24315567 Lba : 194526584 Sectors : 49744
extents_[16] -  VCN : 43038 LCN : 21314603 Lba : 170518872 Sectors : 225616
extents_[17] -  VCN : 71240 LCN : 21355381 Lba : 170845096 Sectors : 296960
extents_[18] -  VCN : 108360 LCN : 14680361 Lba : 117444936 Sectors : 181696

>c:\_apps\Contig.exe e:\DISK2.img

Contig v1.8 - Contig
Copyright (C) 2001-2016 Mark Russinovich
Sysinternals


Summary:
     Number of files processed:      1
     Number of files defragmented:   1
     Number unsuccessfully procesed: 0
     Average fragmentation before: 19 frags/file
     Average fragmentation after : 1 frags/file

>extents-win64.exe e:\DISK2.img
extents 1.3 by erwan2212@gmail.com
***************************
Bytes Per Sector:512
Sectors per Cluster:8
Cluster size :4096
Filesystem :NTFS
***************************
Filename:e:\DISK2.img
File Cluster count :131072 (536870912 bytes)
File size in bytes :536870912
File cluster first :23560563
Extents count :1

extents_[0] -  VCN : 0 LCN : 23560563 Lba : 188486552 Sectors : 1048576


#63 noel

noel

    Frequent Member

  • Advanced user
  • 178 posts
  • Location:nantes
  •  
    France

Posted 29 April 2019 - 08:03 PM

Sorry, i do quicly ...

 

@wonko:

 

It finds 15 sequential extents, i.e. extents that are not separated by either another allocated file or by unallocated space:

I.e. on each line above StartVCN+Length=NextVCN

I understand my mistake ! thank you

 

@erwan.l : can you try "fsutil sparse queryRange e:\DISK2.img" ?  with e:\DISK2.img fragmented file

OK for coffee/beer (with Picon ?) !

 

thanks



#64 noel

noel

    Frequent Member

  • Advanced user
  • 178 posts
  • Location:nantes
  •  
    France

Posted 29 April 2019 - 09:38 PM

for the first point :

 

C:\Windows\system32>"C:\Users\noelb\Desktop\extents-win64.exe" "C:\Users\noelb\Desktop\Windows.iso"
extents 1.3 by erwan2212@gmail.com
***************************
Bytes Per Sector:512
Sectors per Cluster:8
Cluster size :4096
Filesystem :NTFS
***************************
Filename:C:\Users\noelb\Desktop\Windows.iso
File Cluster count :1018976 (4173725696 bytes)
File size in bytes :4173725696
File cluster first :9413798
Extents count :15

extents_[0] -  VCN : 0 LCN : 9413798 Lba : 76571952 Sectors : 176496
extents_[1] -  VCN : 22062 LCN : 8686784 Lba : 70755840 Sectors : 377760
extents_[2] -  VCN : 69282 LCN : 9346741 Lba : 76035496 Sectors : 536440
extents_[3] -  VCN : 136337 LCN : 9172254 Lba : 74639600 Sectors : 553072
extents_[4] -  VCN : 205471 LCN : 9479487 Lba : 77097464 Sectors : 3003400
extents_[5] -  VCN : 580896 LCN : 9444820 Lba : 76820128 Sectors : 200704
extents_[6] -  VCN : 605984 LCN : 9880576 Lba : 80306176 Sectors : 200704
extents_[7] -  VCN : 631072 LCN : 5569624 Lba : 45818560 Sectors : 86016
extents_[8] -  VCN : 641824 LCN : 9916704 Lba : 80595200 Sectors : 522848
extents_[9] -  VCN : 707180 LCN : 5606280 Lba : 46111808 Sectors : 111648
extents_[10] -  VCN : 721136 LCN : 10231832 Lba : 83116224 Sectors : 200704
extents_[11] -  VCN : 746224 LCN : 5394612 Lba : 44418464 Sectors : 57344
extents_[12] -  VCN : 753392 LCN : 10264376 Lba : 83376576 Sectors : 1036288
extents_[13] -  VCN : 882928 LCN : 5539512 Lba : 45577664 Sectors : 90112
extents_[14] -  VCN : 894192 LCN : 10405464 Lba : 84505280 Sectors : 998272

 

Same extents with \extents-win64.exe for  "C:\Users\noelb\Desktop\Windows.iso".

My question : why "fsutil sparse queryRange "C:\Users\noelb\Desktop\Windows.iso"" says "not fragmented"

 

C:\Windows\system32>fsutil sparse queryRange "C:\Users\noelb\Desktop\Windows.iso"
Le fichier spécifié n’est pas fragmenté

 

Second point : disk 4K (not 512e)

 

C:\Windows\system32>"C:\Users\noelb\Desktop\extents-win64.exe" "C:\Users\noelb\Desktop\Windows.iso"
extents 1.3 by erwan2212@gmail.com
***************************
Bytes Per Sector:512 ---------------------> logical or physical sector size ?

my disk :

 

C:\Windows\system32>fsutil fsinfo sectorInfo c:
LogicalBytesPerSector :                                 512
PhysicalBytesPerSectorForAtomicity :                    4096
PhysicalBytesPerSectorForPerformance :                  4096
FileSystemEffectivePhysicalBytesPerSectorForAtomicity : 4096

Also given by PS command "get-disk" : 

LogicalSectorSize     : 512

PhysicalSectorSize    : 4096

 

Yes, easy2boot does not "play" with this ( first physical sector is always first sector with both size, 512 or 4096 ).

But, perhaps, extents-win64.exe should display a warning if 4k disk

 

I hope you can understand me.



#65 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 30 April 2019 - 07:48 AM

My question : why "fsutil sparse queryRange "C:\Users\noelb\Desktop\Windows.iso"" says "not fragmented"

 

Ar you sure-sure that the fsutil sparse queryRange is actually suited for the task? :dubbio:

 

Try - besides the fsutil command above - the other two similar/related tools (still on the same file/filesystem):

1) getfileextents

2) myfragmenter

see:

http://reboot.pro/to...h-wow64-wanted/

(which you should know since you posted there too).

 

AND, since you are probably on Windows 10, the suggested:

fsutil volume [filelayout] <VolumePath> <fileid>

might give further info.

 

:duff:

Wonko



#66 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 30 April 2019 - 08:02 AM

I would not trust fsutil sparse queryRange.

It is either broken (i doubt it) or we are misreading this command.

At my place it always report "file is not fragmented".

>extents-win64.exe e:\DISK2.img
extents 1.3 by erwan2212@gmail.com
***************************
Bytes Per Sector:512
Sectors per Cluster:8
Cluster size :4096
Filesystem :NTFS
***************************
Filename:e:\DISK2.img
File Cluster count :131072 (536870912 bytes)
File size in bytes :536870912
File cluster first :14676409
Extents count :12

extents_[0] -  VCN : 0 LCN : 14676409 Lba : 117413320 Sectors : 31488
extents_[1] -  VCN : 3936 LCN : 21342869 Lba : 170745000 Sectors : 32768
extents_[2] -  VCN : 8032 LCN : 21347061 Lba : 170778536 Sectors : 32768
extents_[3] -  VCN : 12128 LCN : 21351253 Lba : 170812072 Sectors : 32768
extents_[4] -  VCN : 16224 LCN : 25357125 Lba : 202859048 Sectors : 36936
extents_[5] -  VCN : 20841 LCN : 24315567 Lba : 194526584 Sectors : 49744
extents_[6] -  VCN : 27059 LCN : 18696821 Lba : 149576616 Sectors : 128200
extents_[7] -  VCN : 43084 LCN : 14493484 Lba : 115949920 Sectors : 136496
extents_[8] -  VCN : 60146 LCN : 14569353 Lba : 116556872 Sectors : 145440
extents_[9] -  VCN : 78326 LCN : 14510551 Lba : 116086456 Sectors : 146192
extents_[10] -  VCN : 96600 LCN : 24493008 Lba : 195946112 Sectors : 171712
extents_[11] -  VCN : 118064 LCN : 14680361 Lba : 117444936 Sectors : 104064

>fsutil sparse queryRange e:\DISK2.img
Le fichier spécifié n'est pas fragmenté

>c:\_apps\Contig.exe e:\DISK2.img

Contig v1.8 - Contig
Copyright (C) 2001-2016 Mark Russinovich
Sysinternals


Summary:
     Number of files processed:      1
     Number of files defragmented:   1
     Number unsuccessfully procesed: 0
     Average fragmentation before: 12 frags/file
     Average fragmentation after : 1 frags/file


>extents-win64.exe e:\DISK2.img
extents 1.3 by erwan2212@gmail.com
***************************
Bytes Per Sector:512
Sectors per Cluster:8
Cluster size :4096
Filesystem :NTFS
***************************
Filename:e:\DISK2.img
File Cluster count :131072 (536870912 bytes)
File size in bytes :536870912
File cluster first :23560563
Extents count :1

extents_[0] -  VCN : 0 LCN : 23560563 Lba : 188486552 Sectors : 1048576

>fsutil sparse queryRange e:\DISK2.img
Le fichier spécifié n'est pas fragmenté


#67 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 30 April 2019 - 08:05 AM

Ar you sure-sure that the fsutil sparse queryRange is actually suited for the task? :dubbio:

 

Try - besides the fsutil command above - the other two similar/related tools (still on the same file/filesystem):

1) getfileextents

2) myfragmenter

see:

http://reboot.pro/to...h-wow64-wanted/

(which you should know since you posted there too).

 

AND, since you are probably on Windows 10, the suggested:

fsutil volume [filelayout] <VolumePath> <fileid>

might give further info.

 

:duff:

Wonko

 

we could also use mft tools : after all this is where this information (the list of clusters for one file) is ultimately stored.

in this thread, for files not stored in the mft, i display the first cluster (only) - but could adapt this tool to actually display all cluster for one file by parsing the mft entry further.



#68 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 30 April 2019 - 08:25 AM

Yep :), but what I was trying to understand/verify was more along the lines of:

1) tool #1 says not fragmented

2) tools #2 and #3 say "multiple extents"

 

Then one (or more) tool(s) is either "wrong" or - as I suspect - it is not intended to show the same info or info suited for the comparison.

 

I would try using myfragmenter to:

1) list fragment(s)
2) if more than one fragment is found, defragment the file
3) list fragment(s) again

 

OR if Noel wants to keep that file "as is" for reference, take another file, artificially fragment it (again using myfragmenter) and then try the "fsutil sparse queryRange" command line on that surely fragmented file.

 

:duff:

Wonko



#69 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 30 April 2019 - 08:28 AM

https://docs.microso...s/fsutil-sparse

queryrange	Scans a file and searches for ranges that may contain nonzero data.

Not sure how to read this.

 

Rather, is not this command meant to be used against sparse file "only" ?

hence giving an erroneous output on non sparse files?

 

If you want to compare getextents results, then you would use fsutil file queryextents e:\DISK2.img .

And then back to the simple formula "file is contiguous if it has one extent only".

 

For archive : the list of clusters is in the data runlists field of the dataattribute header of the mft entry (more here).

If clusters are all consecutive (or if highvcn-lowvcn=total number of clusters), then you have one 1 extent which again means the file is contiguous.



#70 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 01 May 2019 - 05:10 PM

Next to fsutil file queryextents e:\DISK2.img or extents-win32.exe e:\DISK2.img, you can also fetch the list of occupied clusters straight from the MFT with mft-win32.exe e: disk2.img datarun (see here).



#71 noel

noel

    Frequent Member

  • Advanced user
  • 178 posts
  • Location:nantes
  •  
    France

Posted 01 May 2019 - 06:47 PM

hello,

I was saying that Microsoft complicates everything. I'm more and more convinced.
The command line displaying the help of "fsutil sparse/?" is translated into French by "sparse Contrôle la fragmentation des fichiers"  (in English ="sparse control file fragmentation".)
So it's a fake friend! It does not control fragmentation. I'm sorry I wasted your time.

Before using third-party programs, I always start by looking for what exists in the OS: it is pretty big and often contains good tools.
But this is a bad option in this case. And I couldn't understand the simplest things with its display.
Thank you for bringing me your skills that help me a lot to understand better.

 

For the second point, concerning the size of the physical sectors on 4k disks type "512e" or "4kn" (fsutil fsinfo sectorInfo C: or fsutil fsinfo ntfsinfo c:), my remark on the warning is complicated to explain in English to me.
https://docs.microso...tibility-update
I also said that, moreover, the notion of disk basic and dynamic is still a complication of Microsoft.

https://docs.microso...d-dynamic-disks

(There is another notion  "neighbor " but that refers to the VHD (x), fixed or dynamic size disks. But this is a completely different thing.)

Perhaps, a warning is good in extents-win64.exe for "Basic or not" , "disk 4k (512e/4kn) ie physical sector size" , "MBR/GPT"

 

One more time, thanks for your help

 

@wonko  :I like trying to read all your posts in all the Forum. I have to say that I don't always understand everything, not even often.  But it does not matter. And I admire your sense of sharing and the breadth of your knowledge.



#72 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 02 May 2019 - 06:10 PM

@Noel, good to see we cleared out the fsutil sparse matter.

 

Now about the 4k sector size (vs 512bytes), I could easily add this info to get-extents but I am still unclear as of what it would bring?

I am displaying extents in VCN , LCN , Lba and Sectors.

VCN and LCN are measured in cluster units (which size if provided by my software).

Lba is measured in sector units (here again my software does provide the sector size).

Sectors are number of sectors (same remark as above).

 

I must confess i never worked/encountered a disk with 4096 size sectors (i might be able to emulate it somehow) but I assume (possibly wrongly) my software "extents.exe" will actually report it.

 

Same goes with MBR vs GPT : would it make a difference when dealing with extents?



#73 noel

noel

    Frequent Member

  • Advanced user
  • 178 posts
  • Location:nantes
  •  
    France

Posted 02 May 2019 - 08:23 PM

hello Erwan.l,

 

Yes, no problem with extents and your program !

 

Only warning for verbose mode. Microsoft defines a hierarchy in the disk typology. 

https://docs.microso...d-dynamic-disks

It is to find it that these additions could be useful.

 

 

I must confess i never worked/encountered a disk with 4096 size sectors

Perhaps you got it ! ( like monsieur Jourdain …in Moliere :) )

Can you try the PS  commande "get-disk -Number 0 | fl *" (adm elevated) or "fsutil fsinfo ntfsinfo c:".

I'm sure if you disk is less 3 or 5 years old, it is a 4K disk.

 With mine, i get :

"Octets par secteur  :               512
Octets par secteur physique :       4096"

 

(here again my software does provide the sector size)

Which one?  "Octets par secteur" or "Octets par secteur physique" ?

 

Yes, I know, I cut my hair in four and it really doesn't matter. Except for me who likes to have an overview of the disk as complete as possible, disk basic or not (hosting volumes on several physical disks, which is rare for the common mortals, I readily admit), 4kn disk or not (idem size of physical sector = 4096), MBR or GPT style.

Oui, je sais, je coupe les cheveux en quatre et cela n'a vraiment pas d'importance. Sauf pour moi qui aime avoir une vue d'ensemble du disque aussi complète que possible, disque basic ou non ( hébergeant des volumes sur plusieurs disques physiques, ce qui est rare pour le commun des mortels, je le reconnais volontiers), disque 4kn ou non ( idem taille du sector physique = 4096 ), style MBR ou GPT.

 

For GPT disks, the location of the beginning of the file (which will become the boot partition in the case of easy2boot for example) is different from MBR disks.
I don't know if easy2boot is dealing with 4k discs and GPT disks.

 

I hope not to complicate things with my remarks.



#74 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 03 May 2019 - 11:55 AM

oki, got it and learned something today (probably something i should have known for a while actually :) ) : logical bytes per sector vs physical bytes per sector and over all the "advanced format" i.e 512n vs 4kn/512e.

 

now i need to see how to retrieve this information (should not be too hard using IOCTL_STORAGE_QUERY_PROPERTY ) and how it relates to/impacts extents.

 

2 links that helped understand : MS here and Wikipedia here.



#75 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 04 May 2019 - 12:27 PM

Ok, so about 4kn disks, clearly my hardware is not that recent and I dont have such disk.

 

But...using virtual disks (VHDX), I can emulate it :) 

 

Latest Clonedisk version will display the "physical bytes per sector" information and will also create VHDX with 4k sectors (so does latest vmount version).

 

Now i can resume my tests with extents.exe and see how it behaves in different situation (512/512, 512/4096, 4096/4096).

 

xJKL7jH.png






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users