RamDisk and dynamic memory
#1
Posted 16 May 2012 - 09:03 PM
i create a 500M RamDisk.
i copy 100M file in the RD.
I now have 900M free ram and 100M ram in RD.
I use 800M of ram.
What happens if I copy a 300M file in the RD?
I see that RD not occupies immediately Ram assigned but only after it is used.
you can not release it but if unmount and remount the RD is then released.
#2
Posted 16 May 2012 - 11:16 PM
Windows may internally commit pages when needed, but ImDisk has no knowledge about that. ImDisk allocates memory for RAM disks from virtual memory, so they are always available in virtual address space but physical storage can be anywhere, such as page file, physical RAM or simply nowhere (if page has not yet been committed). A page is typically first used when it is needed by the filesystem, such as when you copy a file to your RAM disk. When you delete the file however, there is no easy way to discard (or reset) that single page again. The page itself is just a piece of a larger memory block that ImDisk does not actually know anything about, not what is being used and not. ImDisk is a disk driver, not a filesystem driver, so it does not know anything about files.
There have came up a few ideas that could possibly solve parts of this issue though. One solution could be to handle "trim notifications" from the filesystem driver above imdisk.sys. The trim messages is a way for a filesystem driver to tell underlying disk driver that a disk area is no longer in use by the filesystem. This is mainly used by SSD drives etc, but could probably be useful for virtual memory backed ImDisk drives as well.
- u2o likes this
#3
Posted 17 May 2012 - 01:17 AM
It can happen, that the ImDisk Ramdisk is located in the page file?ImDisk allocates memory for RAM disks from virtual memory, so they are always available in virtual address space but physical storage can be anywhere, such as page file, physical RAM or simply nowhere (if page has not yet been committed).
Please tell i misunderstood you!
#4
Posted 17 May 2012 - 06:33 AM
When i create a RD, in my task manager i see that available memory remains the same until I copy something in the RD.Unfortunately, ImDisk does not work that way. It allocates all memory for a RAM disk in one single piece. You can easily see signs of that in that when you create a RAM disk, the memory usage (working set size) for the system process goes up with an amount equal to the size of the RAM disk you create.
#5
Posted 17 May 2012 - 11:31 AM
It can happen, that the ImDisk Ramdisk is located in the page file?
Please tell i misunderstood you!
That is correct. A virtual memory backed ImDisk drive can be located in page file while it is not used. That is why it is called a "virtual memory" disk.
If you want to create a RAM disk that uses non-swappable memory, use awealloc driver to allocate memory for the RAM disk.
#6
Posted 17 May 2012 - 11:33 AM
When i create a RD, in my task manager i see that available memory remains the same until I copy something in the RD.
Again, you need to look at the memory usage (working set size) for the System process. If you do that, you will see that it immediately goes up with the size of the new vm disk you have created. "Available memory" is another thing that counts a little different.
#7
Posted 17 May 2012 - 12:49 PM
For me it is more and more incomprehensible how the memory consumption.Again, you need to look at the memory usage (working set size) for the System process. If you do that, you will see that it immediately goes up with the size of the new vm disk you have created. "Available memory" is another thing that counts a little different.
Actually I see a 800M VM consumption of System. But i have a swap file of 1M.
How is it possible that System uses 800M of VM when only 1M swapfile?
The other processes have hundreds of MB of VM but there is only 1M swapfile. Why? VM not is Phisical memory + pagefile (swapfile?) ?
Again I would like to learn more about the difference between swapfile and pagefile.
Maybe VM is Phisical memory + swappable memory?
the swappable memory then is memory actually consumed because there is a large swapfile?
Edited by friske, 17 May 2012 - 12:52 PM.
#8
Posted 17 May 2012 - 01:03 PM
For your other questions, I would suggest a book or website that describes virtual memory space in processes, memory management in Windows kernel etc. Basically, virtual address space in a 32 bit process is always 4 GB. Pages of this space can then be "associated" with physical memory pages, pagefile entries, or something else. Or simply nothing at all, in which case access to that page will raise a page fault exception.
#9
Posted 17 May 2012 - 02:25 PM
If I create a 1GB RD but not use it for anything, all corresponding to the real memory allocated by the RD is still available for other applications?Windows may internally commit pages when needed, but ImDisk has no knowledge about that. ImDisk allocates memory for RAM disks from virtual memory, so they are always available in virtual address space but physical storage can be anywhere, such as page file, physical RAM or simply nowhere (if page has not yet been committed). A page is typically first used when it is needed by the filesystem, such as when you copy a file to your RAM disk.
The allocate memory is available until it is used (committed) ?
#10
Posted 17 May 2012 - 02:42 PM
An application can later "discard" a committed page and thereby telling the kernel that it is not in use anymore.
#11
Posted 17 May 2012 - 02:49 PM
Thanks for the info. Thought awealloc was just to use the non accessable RAM in 32 Bit systems.If you want to create a RAM disk that uses non-swappable memory, use awealloc driver to allocate memory for the RAM disk.
Is this behavior specific to ImDisk or do all RamDisks work this way?That is correct. A virtual memory backed ImDisk drive can be located in page file while it is not used. That is why it is called a "virtual memory" disk.
What happens, if i put the page file on the ImDisk? Can the page file end up in the page file?
#12
Posted 17 May 2012 - 03:00 PM
Let's assume i created a ImDisk of 1GB and didn't put anything on it.
The system has no page file and 2GB of RAM.
Then i run a bunch of programs until there is only 100MB of free RAM available.
Now i try to copy 200MB worth of files to the ImDisk.
Would this result in a simple disk full error or would the system crash because it assumes, to have 1GB of usable space on the ImDisk?
I ask because i had lately a bunch of crashes in high RAM usage situations in PE without ever getting any errors or warnings.
#13
Posted 17 May 2012 - 03:20 PM
Thanks for the info. Thought awealloc was just to use the non accessable RAM in 32 Bit systems.
Awealloc driver uses a different allocation method that have two clear benefits. First, the memory in non-swappable and second, it is much better for large allocations because it does not require one huge block of virtual address space for the allocated memory. It allocates pages that are instead mapped to virtual address space as needed. This makes life much better for other things living in virtual address space of System process, since a large allocated address range may cause subsequent required buffer allocations to fail. That may then happen for other drivers, possibly drivers not in any way related to ImDisk.
Is this behavior specific to ImDisk or do all RamDisks work this way?
There are many different ways to do this. You would have to check how each driver works, to figure that out.
What happens, if i put the page file on the ImDisk? Can the page file end up in the page file?
Yes exactly. It is really bad when that happens as it could dead-lock the system. There have been a few threads in this forum about this lately. I have said it before and say it again: If you want to store a swapfile in an ImDisk RAM disk, always use awealloc driver to allocate memory for that drive. Never ever use swappable memory for storing a swapfile.
Let's assume i created a ImDisk of 1GB and didn't put anything on it.
The system has no page file and 2GB of RAM.
Then i run a bunch of programs until there is only 100MB of free RAM available.
Now i try to copy 200MB worth of files to the ImDisk.
Would this result in a simple disk full error or would the system crash because it assumes, to have 1GB of usable space on the ImDisk?
You should not be able to do that. Either the creation of the ImDisk drive will fail or the applications you use to fill up the remaining memory will fail in their memory allocations.
I ask because i had lately a bunch of crashes in high RAM usage situations in PE without ever getting any errors or warnings.
That is probably more likely to be related to something else. But it could be related to ImDisk in the way that you use a huge RAM disk of 500 MB-1 GB or so and thereby keep a large part of system virtual address space allocated. This is likely to cause other allocations to fail, sometimes even small allocations, in other drivers. Even drivers not in any way related to ImDisk.
- u2o likes this
#14
Posted 17 May 2012 - 04:56 PM
#15
Posted 17 May 2012 - 05:00 PM
Thanks Olof, for all the useful insights. Never even occurred to me before, that a RAMDisk could use anything else but physical RAM.
I agree that "RAM disk" really should be kept as a term for disks in physical RAM. In ImDisk, the term "virtual memory" disk is used instead of RAM disk. But then, many people have started to refer to ImDisk virtual memory backed disks as "RAM disks", which has been somewhat confusing for some users.
#16
Posted 17 May 2012 - 05:36 PM
And if it is true, is the opposite also true? That all "RAMDisks" which do allocated memory for the whole disk right at start, don't use virtual memory?
#17
Posted 18 May 2012 - 06:18 AM
I tryed Primo RamDisk and not "system process NOT goes up with an amount equal to the size of the RAM". imDisk can do the same in the future?Unfortunately, ImDisk does not work that way. It allocates all memory for a RAM disk in one single piece. You can easily see signs of that in that when you create a RAM disk, the memory usage (working set size) for the system process goes up with an amount equal to the size of the RAM disk you create.
#18
Posted 18 May 2012 - 09:58 AM
I tryed Primo RamDisk and not "system process NOT goes up with an amount equal to the size of the RAM".
That is right. It will only show up like that if the driver allocates virtual memory or similar as one big block. That is best suited for smaller virtual memory disks, probably just up to a few hundred MB or so. Larger than that, it will occupy a very large portion of virtual address space for the system process which may cause problems with memory allocations in other drivers or the kernel itself.
imDisk can do the same in the future?
Just use awealloc!
#19
Posted 18 May 2012 - 10:26 AM
I tried it with 2GB ram.Just use awealloc!
It does not load the system process but occupies the same all the ram.
It does not take the ram dynamically as it is used.
Or am I missing something?
#20
Posted 18 May 2012 - 10:40 AM
I tried it with 2GB ram.
It does not load the system process but occupies the same all the ram.
It does not take the ram dynamically as it is used.
Or am I missing something?
That is right, awealloc allocates physical RAM pages for all memory needed for the entire size of the disk. This makes it safe to use it for storing pagefiles and other things that need "predictable" storage sizes.
As an alternative, you can create a smaller awealloc drive and later extend it if you like.
#21
Posted 18 May 2012 - 02:09 PM
#22
Posted 18 May 2012 - 09:47 PM
Olof, i think you've overlooked my post #16.
Yes and no, I have tried to figure out a way to answer that, but it is rather complicated. If I would explain all necessary to answer that it would take some time. I think you need to read up a little on what terms like "virtual memory", "virtual address space" and similar stand for. Try a book/website on kernel/driver programming or similar.
#23
Posted 18 May 2012 - 11:40 PM
Cause running a simple test on a ramdisk driver is way faster, than trying to get access to the actual programmer of it. Present persons excluded of course.
#24
Posted 19 May 2012 - 09:33 AM
I was just looking for a yes/no answer, if things could be simplified like that.
Cause running a simple test on a ramdisk driver is way faster, than trying to get access to the actual programmer of it. Present persons excluded of course.
But it is impossible to answer just yes/no to a question that does not correctly define a term like "virtual memory", like in your question. Everything use "virtual memory" in Windows. Applications, drivers, kernel itself etc. What these virtual addresses really represent makes the difference.
#25
Posted 19 May 2012 - 01:36 PM
Now that i know, that there are RAMDisks and VirtualMemoryDisks, i'm looking for a way to tell the two fast and easy apart.
In case of a PE (which operates without a page file), it probably doesn't matter, but on a full Windows, i use those kinds of drives for one reason only. To speed certain things up. A drive, which ends up in the page file is slower, not faster, than running straight of the HDD.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users