365 replies to this topic

[Wonko the Sane]

Well that's basically what it does.  But wimlib uses a supported API rather than reading the MFT directly.

Sure, I understand that , but since that API is only in Windows 8, I was suggesting (only if possible and only if you are inclined to) to use an "external" (also "unsupported" and possibly "reckless" ) direct parsing of the $MFT for use by the poor Vista and 7 users and possibly also the good Linux ones (BTW AFAICR there is not a distinction between a folder being in "root" or not with "external parsing" such as the one done in the referenced tools).

 

Wonko

[martinb]

As a side note, maybe - just maybe - it would be possible to add to Wimlib direct NTFS $MFT listing, not unlike:
http://reboot.pro/to...y-that-is-fast/
 

Wonko

 
Wow! , why did i never hear of this program before

Thank you for pointing to this program, it normally takes 4 minutes if i try to search a file on my disc, with this program it takes 1 second!
 
@synchronicity Thanks for explaining about the FSCTL_QUERY_FILE_LAYOUT, i really need to update my knowledge about NTFS.

I will make some tests on a win2012 R2 server with and without a running exchange, i will post some update if i have made the test

 

if you ever have some spare time to examine the $MFT listing, it would be great.

Especially because 90% of the servers i need to backup are based on win2008

 

I do understand that you prefer supported API, but if you could make a optional switch with a big warning that would be great (if it could significant increase the "scanning part")

[synchronicity]

It's not safe to directly access the MFT in a mounted filesystem.  You might get inconsistent or incorrect data.  The ioctl makes it safe since it should coordinate with the filesystem driver.

 

It is true that on Linux, wimlib can directly access NTFS filesystems using libntfs-3g.  However, this is only valid because it's expected you pass in a block device that is *not* already mounted.

 

Note that the MFT does not contain a "directory structure" per se and cannot be used for directory listings.  That's why it's not well suited for subtree scans, since if you work from the MFT you have to scan all files anyway.

 

So I'm not currently interested in investing time in a custom MFT scan when it would be unsafe --- and hence could not be enabled by default --- and would only be useful for older versions of Windows.  Also remember that wimlib still spends most of its time archiving file data and not on the scan portion.

[Wonko the Sane]

So I'm not currently interested in investing time in a custom MFT scan when it would be unsafe --- and hence could not be enabled by default --- and would only be useful for older versions of Windows.  Also remember that wimlib still spends most of its time archiving file data and not on the scan portion.

Sure , no problem, I only tried to see if it was possible to nullify a possible argument among the (few) ones that a Windows 8 (and later) fanboy could bring up in order to state that his/her (crappy BTW) new operating system is faster than 7.

 

Wonko

[martinb]

I just made some test on a 2012 server (with exchange) and a old 2003 server (32 bit)
 
I made the test with the  1.9.0-BETA8, and its very fast on the 2012, but i need to compare this with the old version if i have time.
 
But on the 2003 server the snapshot part is giving the following error:

 

[ERROR] IVssBackupComponents.StartSnapshotSet() error: 80042301
[ERROR] A problem occurred while creating a VSS snapshot of "C:\".
        Aborting the operation.
ERROR: Exiting with error code 89:
       Unable to create a filesystem snapshot.

Normally i use the following command if i know I'm dealing with a 2003 server (30% of the servers are still 2003)

vssadmin Create Shadow /For=C: /AutoRetry=2 > nul

This will work on any 2003 server, but the complex part is getting the number of the shadow copy

(i use "vssadmin list shadows" and use a unique name to find the number, but its very dirty code)

 

I don't have a XP computer running at this time, but i think you will get the same problem on XP

 

I will make some more tests in the next few days.

[synchronicity]

The snapshot support has been successfully tested on Windows XP.  However, I haven't heard of anyone else testing Windows Server 2003.  I don't know why it doesn't work.  Personally, I don't have that operating system available for testing and would rather spend time improving support for more recent operating systems.

[synchronicity]

wimlib-1.9.0-BETA9 includes a directory scan optimization on Windows for the non-MFT-scan case: files are not opened for "read data" access (equivalently: "list directory" access) during the scan unless they are found to be directories.  This optimization theoretically applies to all versions of Windows, although I found that for whatever reason, it made a large difference on Windows 8 and 10 but made little difference on Windows 7.  Probably different versions of Windows prefetch data differently.

[martinb]

I made some more tests on the snapshot part and the results are:

(1.9.0-BETA8)

• Server 2003 -> no go

• Server 2003 r2 -> no go

• 2003 SBS -> no go

• 2003 SBS r2 -> no go

• XP home -> ok

• XP pro -> ok

• Vista 64 bits -> ok

• Windows 7 pro 64 bit -> ok

• Windows 10 home 64 bit -> ok

• Server 2008 32 bit -> ok

• Server 2008 64 bit -> ok

• Server 2012 64 bit -> ok

Technically speaking the 2003 server should have the same result as the winxp, so i'm not sure what goes wrong.

 

I did some search on the VSS_E_BAD_STATE 0x80042301L error, but all i can find is that this happens if a 2e snapshot is already running, or that the VSS service is not working like it should

But that is not the case because the snapshot from commandline is running without a problem.

The strange part is that the error is within 1 second, without even trying to start the snapshot

(normally it would take at least 5 seconds before the snapshot call will give up, in this case the error is within 1 second and even if i don't start the VSS service it will not even try to start the service.

I do understand that your priorities are with newer os, but because it's working on XP i think it has to be something small. If you ever have time to give it a look, i will be more than happy to run some tests.

One thing i did notice is the big difference between a source of c: and a source of c:\

I do understand that this is not a bug, but it can be a bit confusing.

About the new release, i will make some compare on the scanning part and will post the results.

 

Edited by martinb, 22 January 2016 - 11:29 PM.

[guimenez]

Congratulations for this nice software!

I'm trying to find the files on the download section, but it says "no files".

I'd like to try this out since imagex it's very very slow to perform a windows backup

[Wonko the Sane]

Congratulations for this nice software!

I'm trying to find the files on the download section, but it says "no files".

I'd like to try this out since imagex it's very very slow to perform a windows backup

Try here :

http://reboot.pro/to...-13#entry194180

https://wimlib.net/

 

Wonko

[synchronicity]

wimlib 1.9.0 has been released.  From the NEWS file:

 

Added experimental support for Windows VSS (Volume Shadow Copy Service).

The new '--snapshot' argument to 'wimcapture' makes wimlib automatically

create and use a temporary VSS snapshot when capturing a WIM image.

 

Implemented setting of Windows-specific XML information, such as

architecture, system root, and version details.  This information is now

automatically set in newly captured WIM images, when appropriate.

 

Improved performance of directory tree scans on Windows.

 

On Windows, to improve capture performance, wimlib now sometimes opens

files by inode number rather than by path.  This is enabled for

wimcapture and wimappend, but for now other applications have to opt-in.

 

The progress messages printed by wimlib-imagex while writing WIM files

have been slightly tweaked.

 

Progress information for directory tree scans now counts all hard links.

Also, on Windows "\\?\" is no longer stripped from the current path.

 

Added a new '--image-property' option to 'wimcapture', 'wimappend', and

'wiminfo'.  This option lets you assign values to elements in a WIM

file's XML document by name.

 

The wimlib_get_image_property() and wimlib_set_image_property() API

functions now support numerically indexed elements.

 

Fixed a bug where, on Windows, wimlib would change the security

descriptor of the target directory of an extraction even when the

'--no-acls' option was specified.

 

 

 

All versions of wimlib may be downloaded from https://wimlib.net.

[jonas123]

thank you for making this great software!

[sebus]

Error enumerating files on volume (err=122). The data passed to a system call is too small. Falling back to the standard recursive directory tree scan.

 

Capture works OK, producing .wim that can be deployed just fine

 

Any idea about the error then?

 

Thanks

 

sebus

[synchronicity]

Hi sebus, thanks for the report.  That is actually a known issue which has already been fixed in the latest beta version (wimlib-1.9.1-BETA5).  It's also safe to ignore the error message, since the end result of the capture operation is the same.

[synchronicity]

wimlib version 1.9.1 has been released.  There are a few (generally small) changes.  From the NEWS file:

 

 

Object IDs are now saved and restored on Windows and in NTFS-3G mode.

 

Reduced memory usage when exporting large numbers of WIM images.

 

Non UTF-8 locales are now detected correctly.

 

Addressed compiler warnings and enabled "silent" make rules by default.

 

Windows-specific updates:

Fixed a bug where duplicate backslashes could be generated in

link targets when extracting symbolic links and junctions.

 

Fixed a bug where the .cmd shortcuts for wimlib-imagex wouldn't

work if their full path contained a space.

 

Fixed bugs related to scanning SMB filesystems.

 

Added warning message about known issue with WindowsApps folder.

 

Added instructions for building from source on Windows.

 

VSS support is no longer marked "experimental".

 

Added missing license file for libdivsufsort-lite.

 

 

As usual, all files may be downloaded from https://wimlib.net.  The SHA-256 checksums of the new files are:

 

 

5466209f221919d091357667cf5a3c8b73ffc973972fce1c3aa96de6bd0cd014  wimlib-1.9.1.tar.gz

233c44f8f6a2cee6c43e94382156a5c366c338b4709bbdbff8dd79861121fc7d  wimlib-1.9.1-windows-i686-bin.zip

a14b6e48afc73be191686cc6ad085efabb82a506a67249f37c3ed5c87f1cfe6f  wimlib-1.9.1-windows-x86_64-bin.zip

 

 

 

[synchronicity]

wimlib version 1.10.0 has been released.  From the NEWS file:

 

 

The LZX compression ratio has been slightly improved.  The default mode,

LZX level 50, is now almost as good as the old LZX level 100, while

being nearly the same speed as before.

 

Decompression performance has been slightly improved.

 

Filenames are now always listed in NTFS collation order.

 

On UNIX-like systems, wimlib can now process Windows filenames that are

not valid Unicode due to the presence of unpaired surrogates.

 

On UNIX-like systems, wimlib now always assumes UTF-8 encoding with the

addition of surrogate codepoints.  Consequently, the environmental

variable WIMLIB_IMAGEX_USE_UTF8 and the flag

WIMLIB_INIT_FLAG_ASSUME_UTF8 no longer have any effect.

 

wimlib no longer depends on iconv.

 

Reduced memory usage slightly.

 

When a WIM image is applied in NTFS-3G mode, security descriptors are

now created in NTFS v3.0 format when supported by the volume.

 

Workarounds for bugs in libntfs-3g version 2013.1.13 and earlier have

been removed.  Users are advised to upgrade to a later version of

libntfs-3g.

 

On Windows, wimlib now supports case-sensitive filename extraction when

supported by the underlying operating system and filesystem (operating

system support requires a registry setting).

 

 

As usual, all files may be downloaded from https://wimlib.net.  The SHA-256 checksums of the new files are:

989b1b02f246c480dec10469374f4235d15a3d5e5ae054452405305af5007f55  wimlib-1.10.0.tar.gz

b5b0906da0e6c4a3bfa933ba905bac28f566d70976cb434a20c7f1b2d4c17a48  wimlib-1.10.0-windows-i686-bin.zip

a1208f2419e48b7e23b814b4495fa6a27ff7df48c654379d1162db8ca94c33bb  wimlib-1.10.0-windows-x86_64-bin.zip