NetBeans on speed
In the post from yesterday I was a little harsh regarding the performance of recent NetBeans versions, which seems to have taken a toll for the worse. One of the most annoying things is how much scanning and processing is going on all the time which harms responsiveness and the overall user experience. The NetBeans mailing list yielded responses like "more features requires more work" which is fair enough. However, I decided I wanted to have my cake (speed) and eat it too (features).
Memory mapping to the rescue
It's not exactly new. So called "RAM drives" have been used for ages to speed things up, some Linux distro's even mount the temporary files location into RAM to improve performance. We can do the same to our source checkouts to gain massive increase in throughput as well as access time. There are two easy ways to archive this out of the (Linux) box, mount a dedicated RAM drive or use the TMPFS filesystem. Sorry, Windows users will have to look elsewhere for info about how to set up a RAM drive.
RAM drive
In this approach, you allocate a part of the memory and map it to a mount point. The following will create a 64MB RAM disk and let you access it under /media/ramdisk/:
It turns out that the kernel decides how much memory you can mount like this, on Ubuntu 8.04 the limit is 64MB. You can however change this to your liking by adding a kernel option before it loads:
Note however that while you can unmount these again, the physical memory it occupied will not be released to the system. Another limitation of this approach is that whether you need it or not, the RAM disk will permanently occupy whatever you allocated to it - no more and no less. So you should probably only use this approach if you need a small RAM drive or you are just testing the technique out. The only benefit of this approach I can think of, is that your mounted partition will be listed and can be monitored from the system monitor.
Temporary filesystem
TMPFS is rather different than the traditional block oriented RAM disk. It turns out it's actually even easier to deal with, just try this:
Not only is it easier, it's much more flexible. As you no doubt noticed, we did not actually specify any capacity. TMPFS will use half of your RAM as default. Although you can also specify it directly, there's really no need to since TMPFS is not only able to dynamically decrease but also increase its capacity on demand. This means that if you do not use the partition, you won't pay any memory penalty. Apparently TMPFS is even able to use swap file so that IF you should go overboard in space consumption, you won't suffer loss of data or the like.
You can have your system automatically create and mount a TMPFS partition for you, by modifying your /etc/fstab file. Simply add the following to it:
Now I don't actually want to run into scenarios where my system has to start using virtual memory, so I limited the size of the temporary partition to 1GB, that leaves me with at least 3GB for the OS and applications.
Applied to a NetBeans project
It's rather hard to measure and express something as subjective as responsiveness and general feeling of an IDE with and without a RAM drive. Suffices to say it feels like it's running at a blazing speed! Nodes expand at once, tabs activate immediately and it's just an overall impressive experience - it feels like you're back to writing C code rather than Java. Let me try and prove this to you the best I can with just static images. The below screen shots both show the result of running the same simple Ant build target on a small NetBeans module project, first on a normal disk drive and then from of a RAM drive.
It took 10 sec to build on my 10.000rpm VelociRaptor disc, the fastest desktop drive available today. An average disc would require around 12-13 sec I suspect.
Running from of the RAM drive it took just 1 sec, an amazing 10x faster. It's the kind of thing you have to see in order to believe.
I also tried placing NetBeans itself as well as its user directory directly on the RAM drive, but without noticeable difference at run time. It did start up 40-50% faster and became responsive right away, but this is probably not an optimization worth going for.
Applied in practice
So there you have it. No magic involved, all you need is enough RAM which is dirt cheap* and some consistent working habits. You do have to manually copy your project onto the RAM drive as well as remember to move it to a non-volatile partition before you shut your computer down. However given the fact that we usually always work out of CVS/SVN/HG this should not be much of an issue, in fact you could claim it encourages healthier check-in habits.
As a cautionary measure though, I have invested in a cheap UPS in case of power loss and when I shut down my system, a script will automatically copy the content of the RAM drive to a persistent backup partition. For convenience, when I start up, another script will automatically copy the most recent backup back onto to the RAM drive. Thus, rather than having to do a full checkout/clone in the morning, I only have to issue an update. If you really wanted to minimize risks, you could also have a timed cron job or a low-priority synchronization deamon running.
In conclusion
Of course, this technique really doesn't have anything to do with NetBeans per se, so any IDE would benefit from it as well as freeform/Ant/Maven projects. If you feel uneasy about working entirely in volatile memory, I suppose it would be possible to simply modify your build scripts such that only the build artifacts are emitted this way, leaving the actual source code intact on a fully journaled non-volatile file system.
I had fun setting this up, though I am somewhat surprised of how I/O-bound NetBeans appears to be. In a related exercise with Visual Studio and C# a few years back I did not experience anywhere near this speed-up, possibly because the compiler and tool chain is multi-threaded unlike javac and Ant.
So go ahead and give it a try, you might like it too. I obviously take no responsibility for whatever data loss you may experience yada yada. You have been warned!
Update:
Someone asked me how to put the automatic copy mechanism I mentioned above into effect. We are going to need 2 scripts as well as installing these at the appropriate life cycle hooks of Linux. In the following I assume you have created a directory at /ramdisk_archives with root access permissions only and that you are using a Debian derived distro.
Put the below script in a file called ramdisk_to_archive.sh, give it execute rights (chmod +x ramdisk_to_archive.sh) and save it under the /etc/init.d/ directory which is usually where service scripts are put.
This script will create a new subdirectory under /ramdisk_archives based on the current timestamp and copy all content of the RAM disk (with file permissions intact), to this newly created folder. Then it will delete any eventual existing link called latest and finally, it will create this new link again and point to the directory we just created. Finally, you need to install it so it runs at system halt (runlevel 0) as well as reboot (runlevel 6) which you can do by typing this:
Next time you reboot or shut down your system, the script above will be run, thus saving the content of the RAM disk onto /ramdisk_archives/ somewhere. Now, to have your RAM disk restored again at boot time, write another script:
Again, remember to give it execute permission and copy it to the services directory like the prior script. This time we're going to hook into all the other remaining runlevels 2-5. To do this, issue the following command:
That's it. Remember to clean up the /ramdisk_archive folder from time to time as there's no purge or rotation scheme in place. You are of course free to add this yourself, if you do, drop me a line. :)
*You can have 4GB DDR2 for less than US$ 100. Even older consumer motherboards supports 4GB, newer ones 16-32GB.
Memory mapping to the rescue
It's not exactly new. So called "RAM drives" have been used for ages to speed things up, some Linux distro's even mount the temporary files location into RAM to improve performance. We can do the same to our source checkouts to gain massive increase in throughput as well as access time. There are two easy ways to archive this out of the (Linux) box, mount a dedicated RAM drive or use the TMPFS filesystem. Sorry, Windows users will have to look elsewhere for info about how to set up a RAM drive.
RAM drive
In this approach, you allocate a part of the memory and map it to a mount point. The following will create a 64MB RAM disk and let you access it under /media/ramdisk/:
dd if=/dev/zero of=/dev/ram1 bs=4096 count=16k
mke2fs /dev/ram1
mkdir /media/ramdisk
mount /dev/ram1 /media/ramdisk
It turns out that the kernel decides how much memory you can mount like this, on Ubuntu 8.04 the limit is 64MB. You can however change this to your liking by adding a kernel option before it loads:
ramdisk_size=XX
Note however that while you can unmount these again, the physical memory it occupied will not be released to the system. Another limitation of this approach is that whether you need it or not, the RAM disk will permanently occupy whatever you allocated to it - no more and no less. So you should probably only use this approach if you need a small RAM drive or you are just testing the technique out. The only benefit of this approach I can think of, is that your mounted partition will be listed and can be monitored from the system monitor.
Temporary filesystem
TMPFS is rather different than the traditional block oriented RAM disk. It turns out it's actually even easier to deal with, just try this:
mkdir /media/ramdisk
mount tmpfs /media/ramdisk -t tmpfs
Not only is it easier, it's much more flexible. As you no doubt noticed, we did not actually specify any capacity. TMPFS will use half of your RAM as default. Although you can also specify it directly, there's really no need to since TMPFS is not only able to dynamically decrease but also increase its capacity on demand. This means that if you do not use the partition, you won't pay any memory penalty. Apparently TMPFS is even able to use swap file so that IF you should go overboard in space consumption, you won't suffer loss of data or the like.
You can have your system automatically create and mount a TMPFS partition for you, by modifying your /etc/fstab file. Simply add the following to it:
none /media/ramdisk tmpfs defaults,user,size=1G,mode=0777 0 0
Now I don't actually want to run into scenarios where my system has to start using virtual memory, so I limited the size of the temporary partition to 1GB, that leaves me with at least 3GB for the OS and applications.
Applied to a NetBeans project
It's rather hard to measure and express something as subjective as responsiveness and general feeling of an IDE with and without a RAM drive. Suffices to say it feels like it's running at a blazing speed! Nodes expand at once, tabs activate immediately and it's just an overall impressive experience - it feels like you're back to writing C code rather than Java. Let me try and prove this to you the best I can with just static images. The below screen shots both show the result of running the same simple Ant build target on a small NetBeans module project, first on a normal disk drive and then from of a RAM drive.
It took 10 sec to build on my 10.000rpm VelociRaptor disc, the fastest desktop drive available today. An average disc would require around 12-13 sec I suspect.
Running from of the RAM drive it took just 1 sec, an amazing 10x faster. It's the kind of thing you have to see in order to believe.
I also tried placing NetBeans itself as well as its user directory directly on the RAM drive, but without noticeable difference at run time. It did start up 40-50% faster and became responsive right away, but this is probably not an optimization worth going for.
Applied in practice
So there you have it. No magic involved, all you need is enough RAM which is dirt cheap* and some consistent working habits. You do have to manually copy your project onto the RAM drive as well as remember to move it to a non-volatile partition before you shut your computer down. However given the fact that we usually always work out of CVS/SVN/HG this should not be much of an issue, in fact you could claim it encourages healthier check-in habits.
As a cautionary measure though, I have invested in a cheap UPS in case of power loss and when I shut down my system, a script will automatically copy the content of the RAM drive to a persistent backup partition. For convenience, when I start up, another script will automatically copy the most recent backup back onto to the RAM drive. Thus, rather than having to do a full checkout/clone in the morning, I only have to issue an update. If you really wanted to minimize risks, you could also have a timed cron job or a low-priority synchronization deamon running.
In conclusion
Of course, this technique really doesn't have anything to do with NetBeans per se, so any IDE would benefit from it as well as freeform/Ant/Maven projects. If you feel uneasy about working entirely in volatile memory, I suppose it would be possible to simply modify your build scripts such that only the build artifacts are emitted this way, leaving the actual source code intact on a fully journaled non-volatile file system.
I had fun setting this up, though I am somewhat surprised of how I/O-bound NetBeans appears to be. In a related exercise with Visual Studio and C# a few years back I did not experience anywhere near this speed-up, possibly because the compiler and tool chain is multi-threaded unlike javac and Ant.
So go ahead and give it a try, you might like it too. I obviously take no responsibility for whatever data loss you may experience yada yada. You have been warned!
Update:
Someone asked me how to put the automatic copy mechanism I mentioned above into effect. We are going to need 2 scripts as well as installing these at the appropriate life cycle hooks of Linux. In the following I assume you have created a directory at /ramdisk_archives with root access permissions only and that you are using a Debian derived distro.
Put the below script in a file called ramdisk_to_archive.sh, give it execute rights (chmod +x ramdisk_to_archive.sh) and save it under the /etc/init.d/ directory which is usually where service scripts are put.
#!/bin/sh
#/etc/init.d/ramdisk_to_archive.sh
a=$(date +%T-%d-%m-%Y)
mkdir /ramdisk_archives/$a
cp -i -p -R /media/ramdisk/* /ramdisk_archives/$a
rm /ramdisk_archives/latest
ln -s /ramdisk_archives/$a /ramdisk_archives/latest
This script will create a new subdirectory under /ramdisk_archives based on the current timestamp and copy all content of the RAM disk (with file permissions intact), to this newly created folder. Then it will delete any eventual existing link called latest and finally, it will create this new link again and point to the directory we just created. Finally, you need to install it so it runs at system halt (runlevel 0) as well as reboot (runlevel 6) which you can do by typing this:
ln -s /etc/init.d/ramdisk_to_archive.sh /etc/rc0.d/ramdisk_to_archive.sh
ln -s /etc/init.d/ramdisk_to_archive.sh /etc/rc6.d/ramdisk_to_archive.sh
Next time you reboot or shut down your system, the script above will be run, thus saving the content of the RAM disk onto /ramdisk_archives/ somewhere. Now, to have your RAM disk restored again at boot time, write another script:
#!/bin/sh
#/etc/init.d/ramdisk_to_archive.sh
cp -i -p -R /ramdisk_archives/latest/* /media/ramdisk/
Again, remember to give it execute permission and copy it to the services directory like the prior script. This time we're going to hook into all the other remaining runlevels 2-5. To do this, issue the following command:
update-rc.d /etc/init.d/ramdisk_to_archive.sh defaults
That's it. Remember to clean up the /ramdisk_archive folder from time to time as there's no purge or rotation scheme in place. You are of course free to add this yourself, if you do, drop me a line. :)
*You can have 4GB DDR2 for less than US$ 100. Even older consumer motherboards supports 4GB, newer ones 16-32GB.
Comments
There's also a middle ground, the Gigabyte iRam. Personally I find this software solution fine though, for development anyway - probably not applicable to video editing etc.
For example. On FreeBSD, mount the filesystem async. Then use a sysctl to set the disk flush time to 30 minutes.
Same thing could probably be done on Linux with a ext2fs and setting the flush time.
Would be interested in hearing your results.
Instead of using an IDE to benchmark the difference, you could copy a couple of megabytes from /dev/random to a file, and then time the copy to a second file on the same filesystem (testing both reading and writing speed when the file is in cache).
I reckon you should see the same speedup if you mount your filesystem with defaults,noatime,nodiratime, and perhaps also data=writeback,nobh,commit=600 if you have an UPS.
Sort of like the difference between manual memory management vs. garbage collection, with the latter you just kinda cross your fingers and hope for the best... donno if that makes any sense. :)
But yeah, the transactional behavior of doing it the way I mean is pretty vague.
SSDs have phenomenal IOPS and read performance numbers, so they may approach RamDisks and provide equivalent development nirvanas.
update-rc.d /etc/init.d/ramdisk_to_archive.sh default
Should it read?:
update-rc.d /etc/init.d/archive_to_ramdisk.sh default
I agree with you, with the right memory and management you dont need a ram stick.