Jump to content











Photo
- - - - -

devio without imDisk, alternatives?

network block-devices

Best Answer Olof Lagerkvist , 20 May 2018 - 12:10 PM

This would need file system "drivers" in user mode, that is, applications that can read, understand and present to user in some way, file system structures such as directories and files. There are pretty good such modules in several libraries, for example 7-zip, DiscUtils.dll and others so it should be doable somehow. Question is more how to make it useful through a user interface.

 

It would need some development because I don't think anyone have done it. One idea could be to implement something for the DiscUtils.dll tools pretty much like the iSCSI client implementation that is in there today. Could be an interesting feature for the future!

Go to the full post


  • Please log in to reply
15 replies to this topic

#1 ZEE

ZEE

    Frequent Member

  • Advanced user
  • 104 posts
  •  
    Portugal

Posted 18 May 2018 - 10:23 PM

Hi PPL

 

Is there any other ways of using devio

to transfer files between computers

without using imDisk

(examples welcome)

 

Ideally a small client program (no installed driver like imDisk)

that can list/grab/download files from a server running devio

 

or a user-mode program that can supply the block-file-service

for a remote server running devio (supply a drive letter)

so it can be launched in user.mode...

and terminated by closing the program...

 

?any ideas :-)

 



#2 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 20 May 2018 - 12:10 PM   Best Answer

This would need file system "drivers" in user mode, that is, applications that can read, understand and present to user in some way, file system structures such as directories and files. There are pretty good such modules in several libraries, for example 7-zip, DiscUtils.dll and others so it should be doable somehow. Question is more how to make it useful through a user interface.

 

It would need some development because I don't think anyone have done it. One idea could be to implement something for the DiscUtils.dll tools pretty much like the iSCSI client implementation that is in there today. Could be an interesting feature for the future!


  • ZEE and Blackcrack like this

#3 ZEE

ZEE

    Frequent Member

  • Advanced user
  • 104 posts
  •  
    Portugal

Posted 20 May 2018 - 05:14 PM

Hi Olof,,,

 

what about implementing the imDisk proxy part as a stand alone user utility...

(also with a very simple syntax like -> devio-proxy xxx.xxx.xxx.xxx:port )

so we can run it and provide a fake drive letter while it is active?



#4 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 21 May 2018 - 12:35 PM

Why is devio *needed* in first instance?

 

I mean, if the actual need is to transfer files between computers and the two computers are connected via network, what is the problem with using "normal" built-in methods (mapping a network drive)?

 

I am suspecting a XYZ :w00t: kind of issue :dubbio:

 

 

:duff:

Wonko


  • ZEE likes this

#5 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 21 May 2018 - 01:47 PM

Hi Olof,,,

 

what about implementing the imDisk proxy part as a stand alone user utility...

(also with a very simple syntax like -> devio-proxy xxx.xxx.xxx.xxx:port )

so we can run it and provide a fake drive letter while it is active?

 

I don't think I follow. Do you mean an integration between devio.exe and ImDisk driver that calls ImDisk driver to mount and create a drive letter while devio.exe is running?

 

Why is devio *needed* in first instance?

 

I mean, if the actual need is to transfer files between computers and the two computers are connected via network, what is the problem with using "normal" built-in methods (mapping a network drive)?

 

I am suspecting a XYZ :w00t: kind of issue :dubbio:

 

Point. But I could imagine scenarios where there is a remote machine with some disk image files and a local machine where someone cannot install a driver for whatever reason and the user at the local machine wants to transfer some files from within one of the image files at remote location. Since devio is rather simple in that way using only one tcp/ip port I guess there could be cases where that would be a useful option.



#6 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 21 May 2018 - 02:38 PM

But I could imagine scenarios where there is a remote machine with some disk image files and a local machine where someone cannot install a driver for whatever reason and the user at the local machine wants to transfer some files from within one of the image files at remote location. Since devio is rather simple in that way using only one tcp/ip port I guess there could be cases where that would be a useful option.

 

Well, I cannot. :w00t: :ph34r:

 

I mean, if you have an image (for the sake o simplicity let's say a .iso or a RAW image) you can map the remote machine's storage media (containing the image) as a network drive and then use (say) 7-zip or Winimage, etc. on the local machine to open the image and get from it the particular file(s).

 

Or are we talking of doing this across the internet[1]? :unsure:

 

As always, I would like to have an actual description of the actual problem, and then (hopefully) find a solution for the specific problem.

 

:duff:

Wonko

 

[1] If this is the case I can think of a zillion reasons why this kind of access should be not allowed or - if allowed - should go through a rigorous authentication/verification method.



#7 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 21 May 2018 - 02:42 PM

Yes I was thinking of something across the Internet, that's why I talked about the simplicity with only having to care about one tcp/ip port.

 

But I agree that protocols without authentication like this are pretty much useless across Internet anyway. It would be much more useful to use something like iSCSI that at least has some kind of authentication in that case.


  • ZEE likes this

#8 ZEE

ZEE

    Frequent Member

  • Advanced user
  • 104 posts
  •  
    Portugal

Posted 21 May 2018 - 08:44 PM

Why is devio *needed* in first instance?

 

I mean, if the actual need is to transfer files between computers and the two computers are connected via network, what is the problem with using "normal" built-in methods (mapping a network drive)?

 

I am suspecting a XYZ :w00t: kind of issue :dubbio:

 

 

:duff:

Wonko

 

1) It's a great tool to make a 1-to-1 connection for file transfer (you have the block/io streamed via TCP)

2) you can use it for diverse OS(s) storage mounting as if it is local (you need a FS driver though)

3) In my tests it is almost as twice fast as copying via Windows filesharing (tested with 16GB and 64GB files)

 

Any other opinions and experience are welcome ;-)



#9 ZEE

ZEE

    Frequent Member

  • Advanced user
  • 104 posts
  •  
    Portugal

Posted 21 May 2018 - 08:56 PM

Yes I was thinking of something across the Internet, that's why I talked about the simplicity with only having to care about one tcp/ip port.

 

But I agree that protocols without authentication like this are pretty much useless across Internet anyway. It would be much more useful to use something like iSCSI that at least has some kind of authentication in that case.

 

Hi Olof...

 

I agree that this would not be of much use for most of the people...

But for some of us it is very useful...

 

One of my goals is to replace my TFTP get/put transfer methods with other tool...

I only launch the TFTP server in user mode when I need to transfer something ;-)

Of course firewall needs to be configured...

 

I also use SMB and SFTP... but the DevIO transfer is much faster...

 

If you control the Server and the Client PC, communicating over a single port

for some simple and controlled file transfer

with a tool like this you can replace this protocols...

 

I dare to suggest you consider about putting the 'shim' you use in imDisk

for proxyfing Devio as a drive letter

in a small command line (or simple gui tool)

that we can invoke/launch to connect to devio ip:port

and get a drive letter for file transfer

 

Then when we close the tool the connection will drop

and drive letter qould not be available anymore...

 

(no need for fancy security handling... a thing á la NetCat... but a little more evolved)

 

Thx.



#10 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 22 May 2018 - 10:09 AM

That the devio is faster than TFTP is rather obvious, that it is faster than other transfer methods like SMB or "plain" Windows networking is instead (to me at least) surprising, it should mean that the other methods/protocols have a larger overhead (while still being TCP based). :dubbio:

 

I beleave that an iSCSI or WinAoE setup would be much faster anyway (but you would need I believe a driver anyway just like IMDISK).

 

The "real" thing should be going outside of TCP and use UDP, *like* UDT/Sector:

http://udt.sourceforge.net/

http://sector.sourceforge.net/

(Linux only AFAIK) 

 

You can try GridFTP (which is available BOTH for Linux and Windows) however:

http://toolkit.globu...stable/gridftp/

which essentially is a multi-threaded FTP.

 

:duff:

Wonko


  • ZEE likes this

#11 ZEE

ZEE

    Frequent Member

  • Advanced user
  • 104 posts
  •  
    Portugal

Posted 22 May 2018 - 01:24 PM

That the devio is faster than TFTP is rather obvious, that it is faster than other transfer methods like SMB or "plain" Windows networking is instead (to me at least) surprising, it should mean that the other methods/protocols have a larger overhead (while still being TCP based). :dubbio:

 

I beleave that an iSCSI or WinAoE setup would be much faster anyway (but you would need I believe a driver anyway just like IMDISK).

 

The "real" thing should be going outside of TCP and use UDP, *like* UDT/Sector:

http://udt.sourceforge.net/

http://sector.sourceforge.net/

(Linux only AFAIK) 

 

You can try GridFTP (which is available BOTH for Linux and Windows) however:

http://toolkit.globu...stable/gridftp/

which essentially is a multi-threaded FTP.

 

:duff:

Wonko

 

from using Devio over SMB it is at least 30% faster...
I think maybe security mechanisms overhead...

 

Another great feature is that you do not have to fiddle with users/domains

if you connect directly over TCP or UDP between 2 computers (netcat like)

 

If you can launch a client util in user mode

and connect to a remote Devio (or something similar)

you can even make your transfers without admin previleges

(VNC, TeamViewer, AAmmy, SupremoControl, etc. etc. just do it!!! :) )

 

Thanks for the info on UDT and Sector

It is really a interesting product,,, but a little overkill this... ;-)

Parallel FTP products I have already used...

but I'm still seeking for a simple solution...



#12 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 22 May 2018 - 03:11 PM

Now I feel a little more lost here, the devio-proxy-client tool you are talking about, what exactly would that tool do? Just communicating with the ImDisk driver to connect to the server end where devio.exe runs? In what way would that be different from using imdisk.exe directly, like imdisk -a -t proxy -o ip -f address etc syntax?


  • ZEE likes this

#13 ZEE

ZEE

    Frequent Member

  • Advanced user
  • 104 posts
  •  
    Portugal

Posted 29 May 2018 - 03:28 PM

In what way would that be different from using imdisk.exe directly, like imdisk -a -t proxy -o ip -f address etc syntax?

 

1. No need to have installed (or install ImDisk)

2. Can be used in user mode (not need to install driver)



#14 ZEE

ZEE

    Frequent Member

  • Advanced user
  • 104 posts
  •  
    Portugal

Posted 29 May 2018 - 03:41 PM

To clarify this issue...

 

I use a tool to transfer large files over internet (10..100GB)

the best results I've got (speed) was using a tool called FileTransfer

(https://sourceforge....s/file-transfer)

With this tool you launch a server in the remote PC...

Then launch a client in the other PC...

And start transferring the files over a proprietary protocol...

This tool packs the Client+Server in the same .exe for convenience.

 

The speed results are identical to the ones I got with Devio+IMDisk...

 

Yet I would prefer:

 

1)

Have a small server installed in a remote TCP port I choose (Devio)

 

2)

And launch a Client/Proxy small utility (if possible in user mode, no admin rights)

that would give me a drive letter while it is running (close and drive goes away)

and provide me with the same features IMDisk-Devio-Proxy offers...

(nice if some status/traffic info appears in the console or window of the tool...

 

;-)



#15 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 29 May 2018 - 03:53 PM

Ah, I see. No, you cannot get a drive letter in that case. A drive letter is a symbolic link to a device object, that is something created by a driver. For a solution entirely in user mode you would need some other method of browsing image contents, directories, files etc without a drive letter or similar kind of mount points.


  • ZEE likes this

#16 ZEE

ZEE

    Frequent Member

  • Advanced user
  • 104 posts
  •  
    Portugal

Posted 29 May 2018 - 07:23 PM

Well... was a hopeful thought...

 

Thank you all for your answers...






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users