23 replies to this topic

[sambul61]

iSCSI, is an abbreviation of Internet Small Computer System Interface, an Internet Protocol (IP)-based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and to manage storage over long distances. iSCSI can be used to transmit data over local area networks (LANs), wide area networks (WANs), or the Internet and can enable location-independent data storage and retrieval. The protocol allows clients (called initiators) to send SCSI commands (CDBs) to SCSI storage devices (targets) on remote servers. It is a popular Storage Area Network (SAN) protocol, allowing organizations to consolidate storage into data center storage arrays while providing hosts (such as database and web servers) with the illusion of locally-attached disks. Unlike traditional Fibre Channel, which requires special-purpose cabling, iSCSI can be run over long distances using existing network infrastructure.

Anyone tried to boot and run a system via iSCSI protocol using such soft as StarWind or MS iSCSI Target / Initiator? Did it offer you real speed benefit compare to other network boot methods? Any problems & issues you experienced? Any package suggestions? Pls share your experience here. iSCSI is hot and fast developing technology, would be interesting to learn more about its advantages in virtualized environment too, including home and organizations & institutions.

This and that threads reflect some attempts in this area shared on this forum, but more info is long overdue about using iSCSI in regular and virtualized boot environments, including particulars of using gPXE and other network bootloaders.

[Sha0]

...Anyone tried to boot and run a system via iSCSI protocol using such soft as StarWind or MS iSCSI Target / Initiator?

Yes. I frequently use the Microsoft iSCSI Initiator Boot Version to boot Windows XP and Windows Server 2003 via gPXE and sanbootconf with iSCSI Enterprise Target Daemon (IET) as the iSCSI target (server).

...iSCSI is hot and fast developing technology...

Hmmm... Ok.

[sambul61]

What, iSCSI isn't hot enough for you? Despite old Performance Comparison? Any faster network boot & drive access technology you can suggest?

Is Microsoft iSCSI Initiator supplied with Win7 a Boot or none-Boot Version? This article talks only about Win Server 2008 R2, doesn't mention Win7 OS version. That iSCSI Client actually includes several installable "Virtual Hardware Drivers & Providers" to support VSS and similar snapshot techniques via iSCSI.

Not sure, I fully appreciate the "depth" of gPXE Wiki. Can someone explain in detail, how to boot Win7 64-bit with gPXE via iSCSI, using ROM-o-matic Generator to get gPXE build (which one?) good for Win7 boot if required?

[Sha0]

What, iSCSI isn't hot enough for you?

Well, for me personally, it's actually getting older.

Any faster network boot & drive access technology you can suggest?

If your client is on the same Ethernet broadcast domain (not to be confused with IP broadcasts) as the server, you can use AoE (with Windows drivers WinAoE or WinVBlock's AoEXX.sys) and enjoy less overhead than with an IP+TCP-based protocol, such as iSCSI.

Is Microsoft iSCSI Initiator supplied with Win7 a Boot or none-Boot Version? This article talks only about Win Server 2008 R2, doesn't mention Win7 OS version. That iSCSI Client actually includes several installable "Virtual Hardware Drivers & Providers" to support VSS and similar snapshot techniques via iSCSI.

I'm sorry that I don't know Windows 7 or Windows Server 2008.

Not sure, I fully appreciate the "depth" of gPXE Wiki. Can someone explain in detail, how to boot Win7 64-bit with gPXE via iSCSI, using ROM-o-matic Generator to get gPXE build (which one?) good for Win7 boot if required?

You've not found the how-tos, perhaps. From the main page, under Documentation, click the HowTo Guides link. When that page shows, look for the How to boot from SAN section. From the two links there, there is a lot of documentation for a lot of operating systems. It might be redundant to retype it all in this forum.

[sambul61]

Thanks! I found Installing Windows Server 2008 to an iSCSI target. Will it work, if I Restore a generalized Acronis Win7 Archive to the Target instead of installing OS to it? Any install & boot problems you may envision?
If you tried AoE, how would you compare its speed & other features to connecting a drive via USB 2.0?

[sambul61]

If clustering is enabled for an image connected to an iSCSI Target, what else parameters (disk caching, etc.) must be activated to see that target content updated synchronously on all remotely connected PCs? Any special volume format type required for such disk image?

[sambul61]

If your client is on the same Ethernet broadcast domain (not to be confused with IP broadcasts) as the server, you can use AoE (with Windows drivers WinAoE or WinVBlock's AoEXX.sys) and enjoy less overhead than with an IP+TCP-based protocol, such as iSCSI.

So, how exactly can I connect my laptop HDs to my PC via AoE? What should I do? Can I use vBlade for that - how (interesting website: says nothing about tools it offers, probably assuming they are widely known and hugely popular)? If that's the AoE Target - does it have a Windows cousin?

Also, is there a way to hook an external drive (in Ximeta enclosure with USB and Ethernet ports) to a PC via AoE?

[Sha0]

So, how exactly can I connect my laptop HDs to my PC via AoE? What should I do? Can I use vBlade for that - how (interesting website: says nothing about tools it offers, probably assuming they are widely known and hugely popular)?

Yup. vblade is trivial to compile and install. Its command-line usage is trivial, too.

vblade 1 1 eth0 some_image_file.hdd

  OR

vblade 1 1 eth0 /dev/somedisk

If that's the AoE Target - does it have a Windows cousin?

Maybe Coraid has a Windows offering. I'm not familiar with one; sorry.

Also, is there a way to hook an external drive (in Ximeta enclosure with USB and Ethernet ports) to a PC via AoE?

If you connect the enclosure to a Linux system, the disk it encloses should show up as /dev/sdX (or something) on the Linux. Then you can use the second command-line usage example above.

From a Windows XP/2003 with the WinVBlock driver and its AoE feature driver installed, you'd do:

winvblk -cmd scan

Then, noting your MAC address, something like:

winvblk -cmd mount -mac 00:1A:2B:3C:4D:5E -u aoe:e1.1

Which would attach the AoE disk served by the examples above.

If booting which gPXE/iPXE, you don't need an IP address, but simply:

ifopen net0

sanboot aoe:e1.1

[sambul61]

Thanks!

Will try. Ximeta is based on NDAS technology, including their chipset and FW/SW. Connection can go through routers but it doesn't seem to depend on routing or TCP/IP, but instead looks like a direct connection. To authenticate and mount such drive locally, one must use Ximeta soft.

So, once its mounted, will it still be possible to treat it as AoE connection? Will its speed improve as a result? Who's MAC you mentioned above: the drive enclosure's (have to check if it has one) or PC adapter's?

[Sha0]

Thanks!

You're welcome, I'm sure.

...Who's MAC you mentioned above: the drive enclosure's (have to check if it has one) or PC adapter's?

I'm sorry. When I typed "connect the enclosure to a Linux system," I meant via USB. In that example scenario, you wouldn't be using the enclosure's networking features whatsoever; just its USB storage feature (you said it had a USB connector). Since you wouldn't be using the enclosure's networking hardware, the only MAC addresses involved would be the Linux server's and the client PC's. The winvblk.exe command would take the client PC's.

[sambul61]

Now I'm a bit confused (even more then usual )...

So, you say after I connect the enclosure with HD to my PC via USB 2.0, I can also make AoE connection to the same HD with WinVBlock? How this connection will be done - via the same USB cable (meaning no Ethernet connection involved)? So the drive will be connected by 2 methods via one cable? If that's the case, will AoE connection offer any advantage compare to USB 2.0? Should I then switch Off USB connection (how) to simplify things?

Or did you mean that it would involve 2 cables (USB and Ethernet), if only the enclosure had a regular Ethernet chipset and FW instead of Ximeta's? Or its only possible if the enclosure is connected via a regular Ethernet chipset without USB involved? Would that require to incorporate a AoE Target into the enclosure's FW (if it doesn't have OS installed and Web GUI)?

[Sha0]

Now I'm a bit confused (even more then usual )...

What I'm trying to say is:

You said the enclosure had a USB connector. For the example I gave, I meant for you to connect the enclosure to the AoE server (the Linux system with vblade on it) using a USB cable. This scenario doesn't use any of the enclosure's networking features whatsoever.

Then the AoE server (the Linux system with vblade on it) serves the disk as an AoE disk.

Then your client PC attaches the AoE disk using wvblk32.sys and aoe32.sys and winvblk.exe.

Why would you do this? If you have your AoE server (the Linux system with vblade on it) in your house upstairs, you can connect to the AoE disk from your PC downstairs, as long as they are on the same Ethernet broadcast domain (not to be confused with IP broadcasts).

[sambul61]

Now I got it almost right, and it sound's like something to try right away (better in Win of course to avoid reboots).

What's left to figure out, how Ethernet broadcast domain is different from IP broadcast domain? If I've a few wired & wireless routers btw 2 PCs at home or office, what kind of domain is that? If it will be just one router btw them - better? Should I set AoE server as Domain, or can get by without any domain name server at all?

P.S. I noticed you added WinVBlock to ATA over Ethernet Wiki - great idea!

[Sha0]

...What's left to figure out, how Ethernet broadcast domain is different from IP broadcast domain? If I've a few wired & wireless routers btw 2 PCs at home or office, what kind of domain is that? If it will be just one router btw them - better? Should I set AoE server as Domain, or can get by without any domain name server at all?

I use the mnemonic device "APSTaNDaP" (sounds a bit like "outstanding!") to remember the OSI Model. "Ethernet broadcast domain" is in the realm of the Data Link layer, if I'm not mistaken. "IP broadcasts" would be in the realm of the Network layer, if I'm not mistaken. IP is a so-called "routable" protocol. Ethernet is "not."

The cheapest way to think about it might be:

• Ethernet: Hubs, switches
• IP: Routers/gateways

Of course, some appliances might provide a blend; might provide functions at multiple layers of the OSI Model.

P.S. I noticed you added WinVBlock to ATA over Ethernet Wiki - great idea!

No, I didn't. I don't recall having a Wikipedia account or editing any articles.

[sambul61]

From what I read, it should work with some routers on LAN. I've DD-WRT compatible Router at home, so it mimics any mode you ask it to. Not sure though, if the router modes are selectable by Ethernet port, will try. But generally, IP packets from a select port can be forwarded to a particular Ethernet port or MAC by the router, so it doesn't have to route them by IP addresses. May be this will work.

If you use iSCSI, do you happen to know a good metaSAN forum, or a good forum where Clustering File Systems are discussed in detail?

[sambul61]

There was an interesting conversation about AoE versus iSCSI on StarWind forum - they developed the most popular StarWind iSCSI Target and also iSCSI and AoE Initiators among other soft.

"With modern high GHz and multi-core CPUs, memory buses 5 years old supercomputers could only dream of, gigabytes of RAM and dedicated TCP and iSCSI offload silicon baked into even $20 "el cheapo" NIC PHY... All simplicity of AoE simply fades in darkness And Ethernet frame size utilization, multi-packet completion, lack of advanced async processing questions with AoE raise. Also Windows network stack (unlike say Linux network stack) is not very good for raw Ethernet processing. In a nutshell: in Windows raw Ethernet is just another protocol, like TCP. And comes with all the complexity and re-buffering on receive (it does not happen with Linux). So at least for Windows there's no sense to use AoE" - says one of key experts in iSCSI matters.

That explains, why existing AoE Target Software Servers are publicly available only for Linux.

[Sha0]

That explains, why existing AoE Target Software Servers are publicly available only for Linux.

I disagree. I'd suggest that the explanation is: We haven't yet found or nobody has yet written a free one for Windows.

It would be interesting if someone intending to reduce overhead chose to realize that goal by having:

• "modern high GHz and multi-core CPUs"
• "memory buses 5 years old supercomputers could only dream of"
• "gigabytes of RAM"
• "dedicated TCP offload silicon baked into even $20 'el cheapo' NIC PHY"
• "dedicated iSCSI offload silicon baked into even $20 'el cheapo' NIC PHY"

That last one seems a bit silly to include in considerations of AoE versus iSCSI. "Include iSCSI logic with the hardware, and iSCSI will out-perform AoE!" Uhhh... What do you expect?

And how about an organization with 20,000+ computers without any of these items? What makes sense for them? Do we need additional details about their environment in order to decide?

Perhaps generalization about AoE versus iSCSI doesn't help much.

[sambul61]

It seems like he argues, AoE still exists only due to cheap CoRAID hardware storage drives with FW supported AoE code, and compares it to similar iSCSI HW implementation that would way outperform if used.

He also implies, CoRAID may block any attempts to improve the protocol, despite its "brain dead". I fact, he says that StarWind written much improved AoE Target but won't release it. But generally, specifics of Windows TCP implementation seems to nullify advantages of using AoE in Windows (in his opinion) compare to iSCSI, while preserving all known AoE limitations. He's a competitor of sort, but also an advanced expert. He agrees that in Linux AoE may have a lower overhead, but its negligible given today's hardware.

[Wonko the Sane]

I guess everyone has his/her own opinions and experience:
http://aliver.wordpr...net-is-awesome/

and obviously Coraid has a completely different take on the subject:
http://www.coraid.co...rformance-Study
as well as one of it's partners:
http://blog.millenni...s-about-the-aoe
http://blog.millenni...iscsi-protocols

I tend to think that it is another NTFS vs. FAT32, Dracula vs. Mickey Mouse or King Kong vs. Godzilla (and JFYI, yes, the dinosaur can get rid of that grown up chimpanzee with one hand tied behind his back, anytime ) kind of pointless debate/comparison.


Wonko

[sambul61]

Not sure, who in your mind is "the dinosaur" or "chimpanzee" , but how its all relevant to this discussion?

I wanted to pick the right protocol and infrastructure for a particular task, and the comment of StarWind specialist is related to comparing AoE to iSCSI performance in Windows. Again, none of your above links seems relevant in this respect, but thanks for general education, they might be useful when looking at Linux side.

[Wonko the Sane]

I'm simply a Godzilla fan and never liked King Kong.





Many more:
http://godzillahaiku.tumblr.com/

I didn't realize that a disclaimer was needed, my bad :
http://en.wikipedia....ious_disclaimer

I'll adopt for the present and previous post The Three Stooges' version:

Any resemblance between the characters in this picture and any persons, living or dead, is a miracle.

Wonko

[sambul61]

Sorry, I should know the difference. But to be honest, we prefer to watch penguins:

Still your links about AoE don't fit exactly even in the "penguins" scenario.

[sambul61]

Useful well illustrated read for those interested in Network Storage solutions:

Network Storage: Concepts and Protocols

[rebit]

Yes i try and gets working with StarWind.

I think the performance is great in a gigabit ethernet with giga switch.

You can follow this instructions to success boot isci Windows XP / 7 / 8 / 2003 / 2008

http://windowsdiskless.wordpress.com

Edited by rebit, 26 June 2012 - 05:25 AM.