34 replies to this topic

[friske]

I have 1G ram.
i create a 500M RamDisk.
i copy 100M file in the RD.
I now have 900M free ram and 100M ram in RD.
I use 800M of ram.
What happens if I copy a 300M file in the RD?
I see that RD not occupies immediately Ram assigned but only after it is used.
you can not release it but if unmount and remount the RD is then released.

[Olof Lagerkvist]

Unfortunately, ImDisk does not work that way. It allocates all memory for a RAM disk in one single piece. You can easily see signs of that in that when you create a RAM disk, the memory usage (working set size) for the system process goes up with an amount equal to the size of the RAM disk you create.

Windows may internally commit pages when needed, but ImDisk has no knowledge about that. ImDisk allocates memory for RAM disks from virtual memory, so they are always available in virtual address space but physical storage can be anywhere, such as page file, physical RAM or simply nowhere (if page has not yet been committed). A page is typically first used when it is needed by the filesystem, such as when you copy a file to your RAM disk. When you delete the file however, there is no easy way to discard (or reset) that single page again. The page itself is just a piece of a larger memory block that ImDisk does not actually know anything about, not what is being used and not. ImDisk is a disk driver, not a filesystem driver, so it does not know anything about files.

There have came up a few ideas that could possibly solve parts of this issue though. One solution could be to handle "trim notifications" from the filesystem driver above imdisk.sys. The trim messages is a way for a filesystem driver to tell underlying disk driver that a disk area is no longer in use by the filesystem. This is mainly used by SSD drives etc, but could probably be useful for virtual memory backed ImDisk drives as well.

[MedEvil]

ImDisk allocates memory for RAM disks from virtual memory, so they are always available in virtual address space but physical storage can be anywhere, such as page file, physical RAM or simply nowhere (if page has not yet been committed).

It can happen, that the ImDisk Ramdisk is located in the page file?
Please tell i misunderstood you!

[friske]

Unfortunately, ImDisk does not work that way. It allocates all memory for a RAM disk in one single piece. You can easily see signs of that in that when you create a RAM disk, the memory usage (working set size) for the system process goes up with an amount equal to the size of the RAM disk you create.

When i create a RD, in my task manager i see that available memory remains the same until I copy something in the RD.

[Olof Lagerkvist]

It can happen, that the ImDisk Ramdisk is located in the page file?
Please tell i misunderstood you!

That is correct. A virtual memory backed ImDisk drive can be located in page file while it is not used. That is why it is called a "virtual memory" disk.

If you want to create a RAM disk that uses non-swappable memory, use awealloc driver to allocate memory for the RAM disk.

[Olof Lagerkvist]

When i create a RD, in my task manager i see that available memory remains the same until I copy something in the RD.

Again, you need to look at the memory usage (working set size) for the System process. If you do that, you will see that it immediately goes up with the size of the new vm disk you have created. "Available memory" is another thing that counts a little different.

[friske]

Again, you need to look at the memory usage (working set size) for the System process. If you do that, you will see that it immediately goes up with the size of the new vm disk you have created. "Available memory" is another thing that counts a little different.

For me it is more and more incomprehensible how the memory consumption.
Actually I see a 800M VM consumption of System. But i have a swap file of 1M.
How is it possible that System uses 800M of VM when only 1M swapfile?
The other processes have hundreds of MB of VM but there is only 1M swapfile. Why? VM not is Phisical memory + pagefile (swapfile?) ?
Again I would like to learn more about the difference between swapfile and pagefile.
Maybe VM is Phisical memory + swappable memory?
the swappable memory then is memory actually consumed because there is a large swapfile?

Edited by friske, 17 May 2012 - 12:52 PM.

[Olof Lagerkvist]

The 800 MB is probably not committed, that is located "nowhere", if your pagefile usage is that low.

For your other questions, I would suggest a book or website that describes virtual memory space in processes, memory management in Windows kernel etc. Basically, virtual address space in a 32 bit process is always 4 GB. Pages of this space can then be "associated" with physical memory pages, pagefile entries, or something else. Or simply nothing at all, in which case access to that page will raise a page fault exception.

[friske]

Windows may internally commit pages when needed, but ImDisk has no knowledge about that. ImDisk allocates memory for RAM disks from virtual memory, so they are always available in virtual address space but physical storage can be anywhere, such as page file, physical RAM or simply nowhere (if page has not yet been committed). A page is typically first used when it is needed by the filesystem, such as when you copy a file to your RAM disk.

If I create a 1GB RD but not use it for anything, all corresponding to the real memory allocated by the RD is still available for other applications?
The allocate memory is available until it is used (committed) ?

[Olof Lagerkvist]

Yes, that is basically how virtual memory works. Once it is committed, in response to being accessed, it remains allocated somewhere because it then actually holds data that that the allocating process wants to keep. Before that there is no reason to hold the data allocated somewhere because it has not yet been initialized with anything. A committed page can be moved to pagefile if more physical memory is needed by another application. It will then be swapped back again automatically when an access attempt is made that causes a page fault exception, that the kernel handles all internally in that case.

An application can later "discard" a committed page and thereby telling the kernel that it is not in use anymore.

[MedEvil]

If you want to create a RAM disk that uses non-swappable memory, use awealloc driver to allocate memory for the RAM disk.

Thanks for the info. Thought awealloc was just to use the non accessable RAM in 32 Bit systems.

That is correct. A virtual memory backed ImDisk drive can be located in page file while it is not used. That is why it is called a "virtual memory" disk.

Is this behavior specific to ImDisk or do all RamDisks work this way?

What happens, if i put the page file on the ImDisk? Can the page file end up in the page file?

[MedEvil]

Forgot to ask another thing.
Let's assume i created a ImDisk of 1GB and didn't put anything on it.
The system has no page file and 2GB of RAM.
Then i run a bunch of programs until there is only 100MB of free RAM available.

Now i try to copy 200MB worth of files to the ImDisk.

Would this result in a simple disk full error or would the system crash because it assumes, to have 1GB of usable space on the ImDisk?

I ask because i had lately a bunch of crashes in high RAM usage situations in PE without ever getting any errors or warnings.

[Olof Lagerkvist]

Thanks for the info. Thought awealloc was just to use the non accessable RAM in 32 Bit systems.

Awealloc driver uses a different allocation method that have two clear benefits. First, the memory in non-swappable and second, it is much better for large allocations because it does not require one huge block of virtual address space for the allocated memory. It allocates pages that are instead mapped to virtual address space as needed. This makes life much better for other things living in virtual address space of System process, since a large allocated address range may cause subsequent required buffer allocations to fail. That may then happen for other drivers, possibly drivers not in any way related to ImDisk.

Is this behavior specific to ImDisk or do all RamDisks work this way?

There are many different ways to do this. You would have to check how each driver works, to figure that out.

What happens, if i put the page file on the ImDisk? Can the page file end up in the page file?

Yes exactly. It is really bad when that happens as it could dead-lock the system. There have been a few threads in this forum about this lately. I have said it before and say it again: If you want to store a swapfile in an ImDisk RAM disk, always use awealloc driver to allocate memory for that drive. Never ever use swappable memory for storing a swapfile.

Let's assume i created a ImDisk of 1GB and didn't put anything on it.
The system has no page file and 2GB of RAM.
Then i run a bunch of programs until there is only 100MB of free RAM available.

Now i try to copy 200MB worth of files to the ImDisk.

Would this result in a simple disk full error or would the system crash because it assumes, to have 1GB of usable space on the ImDisk?

You should not be able to do that. Either the creation of the ImDisk drive will fail or the applications you use to fill up the remaining memory will fail in their memory allocations.

I ask because i had lately a bunch of crashes in high RAM usage situations in PE without ever getting any errors or warnings.

That is probably more likely to be related to something else. But it could be related to ImDisk in the way that you use a huge RAM disk of 500 MB-1 GB or so and thereby keep a large part of system virtual address space allocated. This is likely to cause other allocations to fail, sometimes even small allocations, in other drivers. Even drivers not in any way related to ImDisk.

[MedEvil]

Thanks Olof, for all the useful insights. Never even occurred to me before, that a RAMDisk could use anything else but physical RAM.

[Olof Lagerkvist]

Thanks Olof, for all the useful insights. Never even occurred to me before, that a RAMDisk could use anything else but physical RAM.

I agree that "RAM disk" really should be kept as a term for disks in physical RAM. In ImDisk, the term "virtual memory" disk is used instead of RAM disk. But then, many people have started to refer to ImDisk virtual memory backed disks as "RAM disks", which has been somewhat confusing for some users.

[MedEvil]

Would it be true, in your opinion, that all "RAMDisks" which do not allocated memory for the whole disk right at start, use virtual memory?

And if it is true, is the opposite also true? That all "RAMDisks" which do allocated memory for the whole disk right at start, don't use virtual memory?

[friske]

Unfortunately, ImDisk does not work that way. It allocates all memory for a RAM disk in one single piece. You can easily see signs of that in that when you create a RAM disk, the memory usage (working set size) for the system process goes up with an amount equal to the size of the RAM disk you create.

I tryed Primo RamDisk and not "system process NOT goes up with an amount equal to the size of the RAM". imDisk can do the same in the future?

[Olof Lagerkvist]

I tryed Primo RamDisk and not "system process NOT goes up with an amount equal to the size of the RAM".

That is right. It will only show up like that if the driver allocates virtual memory or similar as one big block. That is best suited for smaller virtual memory disks, probably just up to a few hundred MB or so. Larger than that, it will occupy a very large portion of virtual address space for the system process which may cause problems with memory allocations in other drivers or the kernel itself.

imDisk can do the same in the future?

Just use awealloc!

[friske]

Just use awealloc!

I tried it with 2GB ram.
It does not load the system process but occupies the same all the ram.
It does not take the ram dynamically as it is used.
Or am I missing something?

[Olof Lagerkvist]

I tried it with 2GB ram.
It does not load the system process but occupies the same all the ram.
It does not take the ram dynamically as it is used.
Or am I missing something?

That is right, awealloc allocates physical RAM pages for all memory needed for the entire size of the disk. This makes it safe to use it for storing pagefiles and other things that need "predictable" storage sizes.

As an alternative, you can create a smaller awealloc drive and later extend it if you like.

[MedEvil]

Olof, i think you've overlooked my post #16.

[Olof Lagerkvist]

Olof, i think you've overlooked my post #16.

Yes and no, I have tried to figure out a way to answer that, but it is rather complicated. If I would explain all necessary to answer that it would take some time. I think you need to read up a little on what terms like "virtual memory", "virtual address space" and similar stand for. Try a book/website on kernel/driver programming or similar.

[MedEvil]

I was just looking for a yes/no answer, if things could be simplified like that.
Cause running a simple test on a ramdisk driver is way faster, than trying to get access to the actual programmer of it. Present persons excluded of course.

[Olof Lagerkvist]

I was just looking for a yes/no answer, if things could be simplified like that.
Cause running a simple test on a ramdisk driver is way faster, than trying to get access to the actual programmer of it. Present persons excluded of course.

But it is impossible to answer just yes/no to a question that does not correctly define a term like "virtual memory", like in your question. Everything use "virtual memory" in Windows. Applications, drivers, kernel itself etc. What these virtual addresses really represent makes the difference.

[MedEvil]

I was using virtual memory in the same sense you did. As memory, which is not exclusively RAM.

Now that i know, that there are RAMDisks and VirtualMemoryDisks, i'm looking for a way to tell the two fast and easy apart.

In case of a PE (which operates without a page file), it probably doesn't matter, but on a full Windows, i use those kinds of drives for one reason only. To speed certain things up. A drive, which ends up in the page file is slower, not faster, than running straight of the HDD.