Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#351 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13483 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 18 January 2016 - 04:44 PM

Well that's basically what it does.  But wimlib uses a supported API rather than reading the MFT directly.

Sure, I understand that :), but since that API is only in Windows 8, I was suggesting (only if possible and only if you are inclined to) to use an "external" (also "unsupported" and possibly "reckless" :w00t: ) direct parsing of the $MFT for use by the poor Vista :ph34r: and 7 users and possibly also the good Linux ones (BTW AFAICR there is not a distinction between a folder being in "root" or not with "external parsing" such as the one done in the referenced tools).

 

:duff:

Wonko



#352 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 18 January 2016 - 10:05 PM

As a side note, maybe - just maybe - it would be possible to add to Wimlib direct NTFS $MFT listing, not unlike:
http://reboot.pro/to...y-that-is-fast/
 
:duff:
Wonko

 
Wow! , why did i never hear of this program before

Thank you for pointing to this program, it normally takes 4 minutes if i try to search a file on my disc, with this program it takes 1 second!
 
@synchronicity Thanks for explaining about the FSCTL_QUERY_FILE_LAYOUT, i really need to update my knowledge about NTFS.

I will make some tests on a win2012 R2 server with and without a running exchange, i will post some update if i have made the test

 

if you ever have some spare time to examine the $MFT listing, it would be great.

Especially because 90% of the servers i need to backup are based on win2008

 

I do understand that you prefer supported API, but if you could make a optional switch with a big warning that would be great (if it could significant increase the "scanning part")



#353 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 18 January 2016 - 11:36 PM

It's not safe to directly access the MFT in a mounted filesystem.  You might get inconsistent or incorrect data.  The ioctl makes it safe since it should coordinate with the filesystem driver.

 

It is true that on Linux, wimlib can directly access NTFS filesystems using libntfs-3g.  However, this is only valid because it's expected you pass in a block device that is *not* already mounted.

 

Note that the MFT does not contain a "directory structure" per se and cannot be used for directory listings.  That's why it's not well suited for subtree scans, since if you work from the MFT you have to scan all files anyway.

 

So I'm not currently interested in investing time in a custom MFT scan when it would be unsafe --- and hence could not be enabled by default --- and would only be useful for older versions of Windows.  Also remember that wimlib still spends most of its time archiving file data and not on the scan portion.



#354 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13483 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 19 January 2016 - 11:07 AM

So I'm not currently interested in investing time in a custom MFT scan when it would be unsafe --- and hence could not be enabled by default --- and would only be useful for older versions of Windows.  Also remember that wimlib still spends most of its time archiving file data and not on the scan portion.

Sure :), no problem, I only tried to see if it was possible to nullify a possible argument among the (few) ones that a Windows 8 (and later) fanboy could bring up in order to state that his/her (crappy BTW) new operating system is faster than 7.;)

 

:duff:

Wonko



#355 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 20 January 2016 - 12:40 AM

I just made some test on a 2012 server (with exchange) and a old 2003 server (32 bit)
 
I made the test with the  1.9.0-BETA8, and its very fast on the 2012, but i need to compare this with the old version if i have time.
 
But on the 2003 server the snapshot part is giving the following error:

 

[ERROR] IVssBackupComponents.StartSnapshotSet() error: 80042301
[ERROR] A problem occurred while creating a VSS snapshot of "C:\".
        Aborting the operation.
ERROR: Exiting with error code 89:
       Unable to create a filesystem snapshot.

Normally i use the following command if i know I'm dealing with a 2003 server (30% of the servers are still 2003)

vssadmin Create Shadow /For=C: /AutoRetry=2 > nul

This will work on any 2003 server, but the complex part is getting the number of the shadow copy

(i use "vssadmin list shadows" and use a unique name to find the number, but its very dirty code)

 

I don't have a XP computer running at this time, but i think you will get the same problem on XP

 

I will make some more tests in the next few days.



#356 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 21 January 2016 - 12:10 AM

The snapshot support has been successfully tested on Windows XP.  However, I haven't heard of anyone else testing Windows Server 2003.  I don't know why it doesn't work.  Personally, I don't have that operating system available for testing and would rather spend time improving support for more recent operating systems.



#357 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 22 January 2016 - 06:31 AM

wimlib-1.9.0-BETA9 includes a directory scan optimization on Windows for the non-MFT-scan case: files are not opened for "read data" access (equivalently: "list directory" access) during the scan unless they are found to be directories.  This optimization theoretically applies to all versions of Windows, although I found that for whatever reason, it made a large difference on Windows 8 and 10 but made little difference on Windows 7.  Probably different versions of Windows prefetch data differently.



#358 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 22 January 2016 - 11:28 PM

I made some more tests on the snapshot part and the results are:

(1.9.0-BETA8)

  • Server 2003 -> no go

  • Server 2003 r2 -> no go

  • 2003 SBS -> no go

  • 2003 SBS r2 -> no go

  • XP home -> ok

  • XP pro -> ok

  • Vista 64 bits -> ok

  • Windows 7 pro 64 bit -> ok

  • Windows 10 home 64 bit -> ok

  • Server 2008 32 bit -> ok

  • Server 2008 64 bit -> ok

  • Server 2012 64 bit -> ok


Technically speaking the 2003 server should have the same result as the winxp, so i'm not sure what goes wrong.

 

I did some search on the VSS_E_BAD_STATE 0x80042301L error, but all i can find is that this happens if a 2e snapshot is already running, or that the VSS service is not working like it should

But that is not the case because the snapshot from commandline is running without a problem.


The strange part is that the error is within 1 second, without even trying to start the snapshot

(normally it would take at least 5 seconds before the snapshot call will give up, in this case the error is within 1 second and even if i don't start the VSS service it will not even try to start the service.


I do understand that your priorities are with newer os, but because it's working on XP i think it has to be something small. If you ever have time to give it a look, i will be more than happy to run some tests.


One thing i did notice is the big difference between a source of c: and a source of c:\

I do understand that this is not a bug, but it can be a bit confusing.


About the new release, i will make some compare on the scanning part and will post the results.

 


Edited by martinb, 22 January 2016 - 11:29 PM.


#359 guimenez

guimenez

    Frequent Member

  • Advanced user
  • 168 posts
  •  
    Portugal

Posted 23 January 2016 - 06:34 PM

Congratulations for this nice software!

I'm trying to find the files on the download section, but it says "no files".

I'd like to try this out since imagex it's very very slow to perform a windows backup :(



#360 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13483 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 23 January 2016 - 06:52 PM

Congratulations for this nice software!

I'm trying to find the files on the download section, but it says "no files".

I'd like to try this out since imagex it's very very slow to perform a windows backup :(

Try here ;):

http://reboot.pro/to...-13#entry194180

https://wimlib.net/

 

:duff:

Wonko


  • guimenez likes this

#361 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 31 January 2016 - 07:23 AM

wimlib 1.9.0 has been released.  From the NEWS file:

 


Added experimental support for Windows VSS (Volume Shadow Copy Service).
The new '--snapshot' argument to 'wimcapture' makes wimlib automatically
create and use a temporary VSS snapshot when capturing a WIM image.
 
Implemented setting of Windows-specific XML information, such as
architecture, system root, and version details.  This information is now
automatically set in newly captured WIM images, when appropriate.
 
Improved performance of directory tree scans on Windows.
 
On Windows, to improve capture performance, wimlib now sometimes opens
files by inode number rather than by path.  This is enabled for
wimcapture and wimappend, but for now other applications have to opt-in.
 
The progress messages printed by wimlib-imagex while writing WIM files
have been slightly tweaked.
 
Progress information for directory tree scans now counts all hard links.
Also, on Windows "\\?\" is no longer stripped from the current path.
 
Added a new '--image-property' option to 'wimcapture', 'wimappend', and
'wiminfo'.  This option lets you assign values to elements in a WIM
file's XML document by name.
 
The wimlib_get_image_property() and wimlib_set_image_property() API
functions now support numerically indexed elements.
 
Fixed a bug where, on Windows, wimlib would change the security
descriptor of the target directory of an extraction even when the
'--no-acls' option was specified.
 

 

 

All versions of wimlib may be downloaded from https://wimlib.net.


  • guimenez and jonas123 like this

#362 jonas123

jonas123
  • Members
  • 1 posts
  •  
    Germany

Posted 06 March 2016 - 05:59 AM

thank you for making this great software!



#363 sebus

sebus

    Frequent Member

  • Advanced user
  • 305 posts

Posted 09 March 2016 - 11:44 AM

Error enumerating files on volume (err=122). The data passed to a system call is too small. Falling back to the standard recursive directory tree scan.

 

Capture works OK, producing .wim that can be deployed just fine

 

Any idea about the error then?

 

Thanks

 

sebus



#364 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 09 March 2016 - 03:28 PM

Hi sebus, thanks for the report.  That is actually a known issue which has already been fixed in the latest beta version (wimlib-1.9.1-BETA5).  It's also safe to ignore the error message, since the end result of the capture operation is the same.


  • sebus likes this

#365 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 12 March 2016 - 07:11 AM

wimlib version 1.9.1 has been released.  There are a few (generally small) changes.  From the NEWS file:

 

 

Object IDs are now saved and restored on Windows and in NTFS-3G mode.

 
Reduced memory usage when exporting large numbers of WIM images.
 
Non UTF-8 locales are now detected correctly.
 
Addressed compiler warnings and enabled "silent" make rules by default.
 
Windows-specific updates:
Fixed a bug where duplicate backslashes could be generated in
link targets when extracting symbolic links and junctions.
 
Fixed a bug where the .cmd shortcuts for wimlib-imagex wouldn't
work if their full path contained a space.
 
Fixed bugs related to scanning SMB filesystems.
 
Added warning message about known issue with WindowsApps folder.
 
Added instructions for building from source on Windows.
 
VSS support is no longer marked "experimental".
 
Added missing license file for libdivsufsort-lite.

 

 

As usual, all files may be downloaded from https://wimlib.net.  The SHA-256 checksums of the new files are:

 

 

5466209f221919d091357667cf5a3c8b73ffc973972fce1c3aa96de6bd0cd014  wimlib-1.9.1.tar.gz
233c44f8f6a2cee6c43e94382156a5c366c338b4709bbdbff8dd79861121fc7d  wimlib-1.9.1-windows-i686-bin.zip
a14b6e48afc73be191686cc6ad085efabb82a506a67249f37c3ed5c87f1cfe6f  wimlib-1.9.1-windows-x86_64-bin.zip

 

 

 

  • jonas123 likes this

#366 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 20 August 2016 - 07:31 AM

wimlib version 1.10.0 has been released.  From the NEWS file:
 

 

The LZX compression ratio has been slightly improved.  The default mode,

LZX level 50, is now almost as good as the old LZX level 100, while
being nearly the same speed as before.
 
Decompression performance has been slightly improved.
 
Filenames are now always listed in NTFS collation order.
 
On UNIX-like systems, wimlib can now process Windows filenames that are
not valid Unicode due to the presence of unpaired surrogates.
 
On UNIX-like systems, wimlib now always assumes UTF-8 encoding with the
addition of surrogate codepoints.  Consequently, the environmental
variable WIMLIB_IMAGEX_USE_UTF8 and the flag
WIMLIB_INIT_FLAG_ASSUME_UTF8 no longer have any effect.
 
wimlib no longer depends on iconv.
 
Reduced memory usage slightly.
 
When a WIM image is applied in NTFS-3G mode, security descriptors are
now created in NTFS v3.0 format when supported by the volume.
 
Workarounds for bugs in libntfs-3g version 2013.1.13 and earlier have
been removed.  Users are advised to upgrade to a later version of
libntfs-3g.
 
On Windows, wimlib now supports case-sensitive filename extraction when
supported by the underlying operating system and filesystem (operating
system support requires a registry setting).

 

 
As usual, all files may be downloaded from https://wimlib.net.  The SHA-256 checksums of the new files are:

989b1b02f246c480dec10469374f4235d15a3d5e5ae054452405305af5007f55  wimlib-1.10.0.tar.gz
b5b0906da0e6c4a3bfa933ba905bac28f566d70976cb434a20c7f1b2d4c17a48  wimlib-1.10.0-windows-i686-bin.zip
a1208f2419e48b7e23b814b4495fa6a27ff7df48c654379d1162db8ca94c33bb  wimlib-1.10.0-windows-x86_64-bin.zip

 

 

 





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users