Jump to content











Photo
- - - - -

Imdisk proxy and partial write

imdisk proxy libiscsi iscsi

  • Please log in to reply
3 replies to this topic

#1 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 25 July 2019 - 08:50 PM

Hi Olof,

 

I am currently completing a proxy for Imdisk to reach iscsi targets.

I am conscious windows natively includes an iscsi client but, for one, imdisk+iscsi may be of interest in some cases (winpe?) and for two, this is mostly a first step for myself leading to further developement later on (like a dd like to allow read/from iscsi targets on the command line, etc).

 

While playing around with read/write thus, i discovered that many iscsi targets will simply crash (or hang) if you attempt to write more than 256KB as a buffer.

 

Therefore, in my imdisk proxy, I am setting the write size to 256KB whenever it is superior to this number.

Doing so, I assumed that the "caller" will be smart enough to detect a partial write and deal with it.

 

This what I get (below) as a message from devio.

 

Is it safe to ignore and assume the caller will deal with it or is setting the max buffer size to 256KB (programatically speaking) not a good thing as there could a be more elegant/safe way to proceed (a command line switch I may have missed?).

 

The "partial write" devio message below seems to indicate that the caller is "only" moving to the next 1MB (the buffer size) rather than pushing a new write request with a smaller buffer size (which i kind of expected/wished...).

 

In all cases, I would like to avoid to implement a logic where I have to split any buffer size > 256KB into several iscsi write requests...

 

0QbXp2B.png

 

My skeleton code setting the buffer size to max 256KB.

function my_write_proc(handle:thandle; buf:pointer; size:cardinal; offset:int64): integer; cdecl;
begin
  result:=-1;
  if size>1024*256 then size:=1024*256;
  writeln('Write request - size:'+inttostr(size)+' offset:'+inttostr(offset));
  //some iscsi code eventualy sending back the number of bytes written
  ...
  //
end;

Regards,

Erwan

My temp work is there for now : http://erwan.labalec...iscsi-proxy.zip.



#2 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 25 July 2019 - 09:20 PM

Interesting project!

 

I would really recommend you to instead make a loop that calls your target with 256 KB each iteration instead. It is generally not safe to read or write smaller amounts of data than requested on this level.

 

It is possible at lower levels though, for example SCSI port drivers frequently do this depending on I/O sizes on target, but then there are ways for the SCSI port to inform disk drivers above about reasonable buffer sizes and also disk.sys and similar drivers are involved which handles splitting requests automatically. In the case with ImDisk we are simply at a too high level and need to handle this ourselves. We cannot assume anything from the caller because the request can come directly from a file system driver without any other drivers in between.



#3 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 25 July 2019 - 09:26 PM

Understood !

 

I was being lazy and was hoping to spare myself a few lines of codes :)

Handling an incoming buffer and splitting into 256KB junks before shipping it to the iscsi library should not be too much of a problem.

 

FYI, I am using libiscsi from here.

I had already taken a look at libnfs from the same author and provided a few tools (including an imdisk proxy) a few months ago.

 

Not being a native c++ dev, I lost of few hairs (out of a few left...) building this dll :)

But it is now working nicely and I shall post a few more tools in the coming days


  • Olof Lagerkvist likes this

#4 DevynPitts

DevynPitts
  • Members
  • 1 posts
  •  
    United States

Posted 22 May 2022 - 10:04 PM

I recently learned how to create proxies myself! I had a good example of a perfect proxy server soax.com! To begin with we need a VPS, preferably with KVM virtualization, because on other types of virtualization the creation of proxies is often forbidden and the operating system CentOS 7 is installed. Many hosting generally prohibit the creation of proxies, so before ordering a VPS, clarify this nuance with technical support. After obtaining access data to the ordered server, let's get to work. I advise everyone to try to create their own proxy, because it is very interesting! You can make it your hobby. The great fact is that it can pay well!


Edited by DevynPitts, 22 May 2022 - 10:04 PM.





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users