Jump to content











Photo
- - - - -

VDK sucks? Should it be put in forced retirement?


  • Please log in to reply
27 replies to this topic

#1 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7101 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 25 September 2009 - 11:05 PM

I know nothing about programming but can read the documentation, have a fairly good memory and I am old enough to read between the lines.

Besides I have a rather specific experience with using VDK and I guess ALL or MOST similar Filedisk/Ramdisk kind of drivers, in other words, I know NOTHING about what is inside the car boot, but I can drive fast enough. :cheers:

VDK was written by Ken Kato for Windows 2000, using actually a lot of "presumptions" based on NT 4.0. (including it's default geometry of nx64x32)

Though an exceptionally good app, it was never properly "updated" with the needed changes.

As you know I am mainly a 2K user, VDK will never "hang" on a 2K machine.

You can make a batch to INSTALL/START/OPEN and then CLOSE/STOP/REMOVE it ALL the times you want, it WON'T hang.

As soon as you try using on XP, it will probably hang as soon as you REMOVE it and try to restart it.

This behaviour has been reported constantly by several users and of course these reports have become more frequent since the "BootSDI" and similar methods of booting became "main stream".

peter (psc) rebuilt the driver, with Ken Kato's permission, in a "special" version for the Winbuilder based projects, in an attempt to correct this behaviour.

Recently more reports were brought to his attention and he is seemingly willing to review once again the code to solve this problem.

My guess, based uniquely on my above summed up experience with these drivers is that a profound re-writing of the code is needed to solve the mentioned problem and to finally fill the original TODO list:
http://chitchat.at.i...e/vdk.html#todo

TODO

Will appear in the next major release:
Allow non-administrative users to use the program.
Virtual disk manipulation such as REDO log merging, defragment, type conversion, creation, etc.

May appear someday...
Better compatibility with Win2K/XP (PnP support, Mount Manager support, etc).
GUI front end program.
Virtual disk error correction.


Thogh very similar, XP and 2K differ, expecially when it comes to mounted filesystems (just think about VSS).
Though very similar, Vista and XP differ, expecially when it comes to mounted filesystems (just think about transactional NTFS, different approach to Removable devices, differnt FORMAT command and what not).
PE environments, though strictly connected to the corresponding "base" OS behave differently.
Pretending that a KERNEL DRIVER developed on NT 4.00 and adapted to Win2K still works without a glitch on XP, Vista, now 7 and the corresponding PE's is asking for a miracle.

It's has been years that I go around this board BEGGING to find a programmer interested in doing this rewrite/update, this same topic has come to the surface for YEARS, and after some "bla.-bla" it was EACH time left behind to do something more important.

Now it's the right time to take a decision.

Option 1: We put VDK in (early) retirement
Option 2: We solve the §@ç#ing problems once and for all.

:D

jaclaz

#2 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 26 September 2009 - 07:43 AM

You can make a batch to INSTALL/START/OPEN and then CLOSE/STOP/REMOVE it ALL the times you want, it WON'T hang.

As soon as you try using on XP, it will probably hang as soon as you REMOVE it and try to restart it.

My experience is that in XP vdk will hang if you don't REMOVE it and try to restart it.
I think Closing and Restart is giving for me in BootSDI.script the hang problem.

Quite often the BootSDI.script failed for me when mounting of the image with vdk has to occur.

My problem is solved by Unchecking the "Dont use vdk remove" Checkbox. :)


In general I had with vdk in XP x86 no problems in my batch BOOT_IMG.cmd and AutoIt3 program Make_IMG.exe.
But it was essential for me to use all the different steps STOP/REMOVE/INSTALL/START/CREATE/OPEN/LINK and after use to STOP/REMOVE vdk ALL the time. (sourcecode in U_XP_SET\USB_XP_Setup\makebt\au3scr\sources_au3 folder).

In Vista SP1 x86 I could not use vdk in my AutoIt3 program of Make_IMG.exe so that I decided to use ImDisk instead, which then works very well with Vista x86 OS.

#3 fxscrpt

fxscrpt

    Frequent Member

  • .script developer
  • 328 posts
  •  
    Germany

Posted 26 September 2009 - 09:20 AM

Maybe this ("The real problem")


and vdk problems share same cause:

the driver holds a reference of something and connot be unloaded!
In this situation you cannot start th driver again.
But if you copy .sys file to another location and alter corresponding ImagePath in registry you can start second instance of this driver. That's characteristic for unload problems.

:)
Peter

#4 Galapo

Galapo

    Platinum Member

  • .script developer
  • 3841 posts
  •  
    Australia

Posted 26 September 2009 - 09:40 AM

Option 1: We put VDK in (early) retirement
Option 2: We solve the §@ç#ing problems once and for all.

I'd prefer it left as an option. I am still not reguarly able to build with RegFactory, which uses ImDisk, due to some image unmounting error (which I have still yet to isolate).

Regards,
Galapo

#5 fxscrpt

fxscrpt

    Frequent Member

  • .script developer
  • 328 posts
  •  
    Germany

Posted 26 September 2009 - 10:21 AM

...
due to some image unmounting error (which I have still yet to isolate).

Regards,
Galapo.

Hi Galapo
It makes me crazy too!
Please try attached script (unmounting works better for me)
What is changed:
a] Switched back imdisk -D > imdisk -d.
Reason: once imdisk -D failes, you will be not able to mount/unmount again until next reboot
How to reproduce: open regedit, mount hive from Imdisk drive, close regedit, run imdisk -D..

b]
Switched ShellExecute > ShellExecuteEx for unmount command.
There is still long delay using ShellExecute
(imdisk broadcasts windows messages, but winbuilder processes messages infrequently. i remember this was discussed in the forum,
i will add to bugtracker.)

:)
Peter

Attached Files



#6 Galapo

Galapo

    Platinum Member

  • .script developer
  • 3841 posts
  •  
    Australia

Posted 26 September 2009 - 11:24 AM

Hi Peter,

Thanks for that. I'll test over the next day or two and let you know how I fair.

a] Switched back imdisk -D > imdisk -d.
Reason: once imdisk -D failes, you will be not able to mount/unmount again until next reboot


Yes, that's quite likely what's been happening for me too.

Regards,
Galapo.

#7 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 26 September 2009 - 11:36 AM

:)
Or did I miss sth?

Are we speaking about vdk or imdisk or 'Virtual disks under WinBuilder'?

Option 1: We put VDK in (early) retirement
Option 2: We solve the §@ç#ing problems once and for all.


Peter :cheers:

#8 JonF

JonF

    Gold Member

  • .script developer
  • 1185 posts
  • Location:Boston, MA
  •  
    United States

Posted 26 September 2009 - 12:11 PM

I'd prefer it left as an option. I am still not reguarly able to build with RegFactory, which uses ImDisk, due to some image unmounting error (which I have still yet to isolate).

Me too.

#9 Lancelot

Lancelot

    Frequent Member

  • .script developer
  • 5013 posts
  • Location:Turkiye/Izmir
  • Interests:*Mechanical stuff and Physics,
    *LiveXP, BartPE, SherpyaXPE,
    *Basketball and Looong Walking,
    *Buying outwear for my girlf (Reason: Girls are stupid about buying bad stuff to make themselves uglier :))
    *Girls (Lyric: Girl,...., You will be a womann, Soon)
    *Answering questions for "Meaning of life",
    *Helping people,

    Kung with LiveXP, Fu with Peter :)
  •  
    Turkey

Posted 26 September 2009 - 01:34 PM

I'd prefer it left as an option.

Me too. :)

#10 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7101 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 26 September 2009 - 04:02 PM

If the alternatives are:
1 day as a Lion
OR
100 days as a sheep,

you CANNOT have a compromise:
10 days as a little bear. ;)

The option was purely rethorical, I'll rephrase:
Since we DO NOT want to put VDK in early retirement, we must find a way to fix the §@ç#ing problems it creates.

If it was meant as a poll I would have created the thread as a poll.

A good start point to do so, is:
  • describe EXACTLY the problems that happen
  • describe EXACTLY the environment in which the problems happen
  • describe EXACTLY what commands/actions/batch/.script/whatever seems to create the problem

After some DETAILED reports will be collected, we should have a clear idea of HOW/WHEN the problems happen, which is a good step forward in the direction of trying to understand WHY it happens, and hopefully find a fix.

The problem is definitely in "something" that "under certain conditions" gets hooked and is NOT released properly during the shut off.
VDK has telescopic commands, the "beginning" ones:
INSTALL means:
INSTALL
:)

START means:
IF driver is not installed INSTALL driver

START

OPEN means:
IF driver is not installed INSTALL driver

IF driver is not started START driver

OPEN

the closing ones:
CLOSE means:
CLOSE
:)

STOP means:
IF any disk is OPEN, CLOSE it

STOP

REMOVE means:
IF any disk is OPEN, CLOSE it

IF driver is STARTed, STOP it

REMOVE

There are different experiences, but some common things can be extracted from them, let's see:
after a boot or a re-boot ANY of the commands:
INSTALL
START
OPEN
work flawlessly.

Here come the problems:

Sequence #1:
CLOSE
followed by a
OPEN


Sequence #2:
STOP
followed by either:
#2.1 START
or
#2.2 INSTALL

Sequence #3:
REMOVE
followe by either:
#3.1 INSTALL
or
#3.2 START
or
#3.3 OPEN


I personally have experience with problems under XP (and never under 2K) with sequence #3, i.e. once REMOVEd the driver does not want to be installed/started/opened again, as fxscrpt described, using another instance of the driver usually solves the problem.

Can people willing to help use the above enumeration to describe the problems they experienced, adding a description of the environment/OS in which this happens?

First step is to make the problem reproduceable.

:cheers:

jaclaz

#11 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 26 September 2009 - 04:33 PM

An addition for the test:
When testing, please change all ShellExecute,Hide to ShellExecute,Open

vdk.exe sometimes asks for user decision.
That is ok, when the user started vdk.exe from command line.

But when vdk.exe is started from WinBuilder script, the 'Hide' option causes that the user has no chance to react. As a result >> vdk 'hangs'

Two of those user inputs I already 'automated' in the vdk_wb version. Maybe there is a third situation I did not know when I made the changes.

The explanation above is only valid for my x86 version.

As far as I see, the x64 version does not have these changes.

Another difference between x86 and x64:

The x86 version REMOVE, STOP or CLOSE lets virtual disks untouched, when they are created by external apps.
(For vdk_WB I defined a different driver ID)

IMHO the x64 version affects 'external' virtual disks.

Therefore an addon to Jaclaz's suggestion

  • describe EXACTLY the problems that happen
  • describe EXACTLY the environment in which the problems happen
  • describe EXACTLY what commands/actions/batch/.script/whatever seems to create the problem

Host OS is an important element of environment!

BTW: The first point appears me as well known. The standard answer is here: "Read careful" :)

Peter

#12 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 27 September 2009 - 07:55 PM

I suggested to contionue the current discussion in LiveXP here.

BTW: I suggest to continue the discussion in the new VDK topic Jaclaz created. This is neither a question of LiveXP nor a question of BootSDI.

It is a general question of vdk due to different operating systems.


Latest report by dera:
The issue of start failure under Vista occurs not only after 'STOP' but also after 'REMOVE'
http://www.boot-land...?...ost&p=80057

My question: Vista x86 or x64, or both?

I have Vista X64 and could install the ddk there and try to debug.

@JonF: Can you extend your explanations about Vista and FS a bit more. I think that I did not realize the meaning yet (maybe a lack on my English language)

Peter

#13 dera

dera

    Gold Member

  • .script developer
  • 1335 posts
  •  
    Hungary

Posted 27 September 2009 - 07:58 PM

on Vista x86 SP1
(on x64 don't know, i have not tried)

#14 JonF

JonF

    Gold Member

  • .script developer
  • 1185 posts
  • Location:Boston, MA
  •  
    United States

Posted 28 September 2009 - 12:33 AM

@JonF: Can you extend your explanations about Vista and FS a bit more. I think that I did not realize the meaning yet (maybe a lack on my English language)

Arghh! I had a long reply written and closed the tab by mistake! This is not my day (Lancelot knows why).

Transactional NTFS is new feature in Vista (and, I think, in Windows 7). If you have database experience, it is just like transactions in a large database environment. You can group any set of filesystem operations, on one volume or spanning volumes, even across a network, into one transaction. Your program proceeds as if it was talking directly to the disks, but changes are not really committed and visible to other processes until you tell the OS to commit. If any of those operations fails, or for any reason you want to un-do the changes, you just tell the OS to roll the transaction back. A wonderful thing for some applications.

Even though the transactional features are only used by programs explicitly written to take advantage of them, the OS is prepared to be asked to do a transaction at any time. I don't know the really deep details, but you can sort of think of it as VSS. The OS creates a shadow copy of each NTFS volume, and any process that does filesystem operations really does those operations to its own shadow copy, and then the operations are committed and visible to other processes. That's not what really happens, but in many ways what really happens acts that way.

The part that really concerns me in VistaPE is the fact that this functionality requires that there be at least one file handle open on each NTFS volume in the system, and therefore no NTFS volume can ever be locked. VSS obviously gets around this problem. I don't really know how TNTFS affects VDK (if it does at all).

#15 Galapo

Galapo

    Platinum Member

  • .script developer
  • 3841 posts
  •  
    Australia

Posted 28 September 2009 - 12:45 AM

So is it possible to have changes etc. committed so as to free up the handle? Perhaps with Sysinternals sync.exe?

Regards,
Galapo.

#16 JonF

JonF

    Gold Member

  • .script developer
  • 1185 posts
  • Location:Boston, MA
  •  
    United States

Posted 28 September 2009 - 01:42 AM

So is it possible to have changes etc. committed so as to free up the handle? Perhaps with Sysinternals sync.exe?

I don't know, but I doubt it. The changes belong to another process.

But committing the changes does not destroy the shadow-copy-like-thing that Vista maintains to be ready at any time for a transaction.

#17 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 28 September 2009 - 07:48 AM

@dera

Please try the attached driver. It is compiled with Vista / W2008 version of DDK

Peter

Attached File  vdk_WB.zip   39.91KB   8 downloads

#18 dera

dera

    Gold Member

  • .script developer
  • 1335 posts
  •  
    Hungary

Posted 28 September 2009 - 09:19 AM

on Vista SP1 x86 host the same problem as with the previous version
Attached File  vdk_10.png   34.75KB   14 downloads

#19 dera

dera

    Gold Member

  • .script developer
  • 1335 posts
  •  
    Hungary

Posted 28 September 2009 - 09:34 AM

i post it only very silently
as i am a simple end-user
and as i am not a programmer
not even know what is a kernel-mode driver or how should it work
but i had similar experience as posted here
so i suppose it is a feature of the OS

#20 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 28 September 2009 - 09:55 AM

Yes, you are right. It is a feature of the OS.

But we are trying whether we can modify the driver to fullfill the requirements of the OS.

E.g. Today I found that vdk contains the obsolete WinApi call ExAllocatePool
It should be replaced by ExAllocatePoolWithTag
But that is a major change at many places. And because this function 'only' allocates memory, I thought not to do something here, because the propability of affecting file system is rather low. But who knows ...

Unfortunatelly I do not have Vista SP1 x86, and therefore need you (your Vista resp.) for test.

If I would have that Vista, would make it easier.
I currently have a dedicated PC for WinBuilder tests only. I can delete and reinstall what I want w/o needing to make backup etc.
It currently contains (only legal versions) of XP, Vista x64 and Win7 x86 trial.
Somebody has an unneeded (legal) spare Vista x86 (English or German)?

Peter

#21 paraglider

paraglider

    Gold Member

  • .script developer
  • 1743 posts
  • Location:NC,USA
  •  
    United States

Posted 28 September 2009 - 12:56 PM

I would suspect that if you can get the driver working correctly in win7 x86 then its also going to work correctly in vista x86.

#22 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 28 September 2009 - 01:51 PM

I would suspect that if you can get the driver working correctly in win7 x86 then its also going to work correctly in vista x86.

Thanks Paraglider!

Unfortunatelly you are wrong.

It works well under Win7 x86!

To be sure that I did not make something wrong:

vdk_WB open * x.fil /rw /p:0

vdk_WB stop

vdk_WB open * x.fil /rw /p:0
This second open under Vista usually gives the 'file not found' message.

Under Win7 x86 it again creates (in my case) drive H:

:whistling:

Peter

#23 dera

dera

    Gold Member

  • .script developer
  • 1335 posts
  •  
    Hungary

Posted 28 September 2009 - 10:40 PM

What is x.fil
is it formatted as NTFS?

at me on vista the problem begins only in that case
if before the stop/remove command i open an NTFS formatted image,
when i open a FAT32 formatted image there is no problem,
when the image as yet not formatted there is no problem
(i made the image using 'fsutil.exe file createnew xyz.img xxxxxxxxx')

#24 Galapo

Galapo

    Platinum Member

  • .script developer
  • 3841 posts
  •  
    Australia

Posted 29 September 2009 - 03:52 AM

Please try attached script (unmounting works better for me)
What is changed:
a] Switched back imdisk -D > imdisk -d.
Reason: once imdisk -D failes, you will be not able to mount/unmount again until next reboot
How to reproduce: open regedit, mount hive from Imdisk drive, close regedit, run imdisk -D..

Hi Peter,

Thanks, this resolves the mount/unmount issue. I've now copied what you've done into the ImDisk option of WimPack and BootSDI.

But now I have another error when RegFactory starts the virtual machine to conduct its registrations: c_437.nls could not be loaded. Do you need a log or any other information?

Thanks,
Galapo.

#25 pscEx

pscEx

    Platinum Member

  • Team Reboot
  • 12707 posts
  • Location:Korschenbroich, Germany
  • Interests:What somebody else cannot do.
  •  
    European Union

Posted 29 September 2009 - 06:50 AM

What is x.fil
is it formatted as NTFS?

at me on vista the problem begins only in that case
if before the stop/remove command i open an NTFS formatted image,
when i open a FAT32 formatted image there is no problem,
when the image as yet not formatted there is no problem
(i made the image using 'fsutil.exe file createnew xyz.img xxxxxxxxx')


It was unformatted created by fsutil.

I'll redo the test with NTFS formatted.

Peter




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users