There's an awful lot of misconceptions regarding Android and multitasking out there. When you read help forums and the issue of performance tweaking comes up, a very popular answer is to use a task killer to close unwanted apps. Even venerable tech people like TWiT's Leo Laporte appears to have acquired this habit. I reckon that these people have no idea how Android actually works and/or are making erroneous assumptions pulled over from the world of traditional desktop computing. In the best case, nothing happens when you kill an application except you spend time manually doing something that would otherwise have been done for you automatically at a later time. Worst case, you crash an application or get up too late tomorrow because your alarm didn't go off.
How Android does things
At an architectural level, applications running on Android are actually not supposed to halt its process in the traditional sense as we are used to from most desktop operating systems. When you switch to another application on Android, or use the back button, the previous application actually remains in memory but is just no longer active (though it's background services may be). It's up to the application and its author to play nicely in this environment. The vast majority of applications have no trouble doing so.
Only very few applications are so performance demanding, that they introduce a distinct exit button. This can be seen i.e. in the GPS navigation software CoPilot which engages many services and sensors on the phone, and is assuming that even if you require the use of another application (i.e. get a call), the navigation software should keep guiding you on your way. However, this exit button might as well be called "stop navigation services" but that would probably just confuse.
Why Android does it this way
At the usability level, I suspect Google liked being able to combine Cancel, Back and Exit into one physical button. To most people, the difference comes down to some fairly narrow semantics. Apple of course gets around this simply by not allowing multitasking altogether, probably driven by the desire to only have one button! At a more technical level, realize that DRAM memory cells in computers and smart-phones costs just as much in power/battery to keep alive, regardless of whether these cells are used or not. So in the world of Android, applications which you've used are considered likely candidates for use again later and hence, stays dormant in RAM. The Android operating system will manage these and in the case of low memory, will suspend and free up RAM for other purposes. Think of it this way, it's cheaper/faster to keep the application idle in fast RAM and just move a memory pointer to it when you want to use it again, than it is to load the whole application from scratch again from slow flash.
So why do people use task killers?
Apart from wrong assumptions that things works as on a PC, I suspect using too many applications (or rather, too many busy background services) and perhaps witnessing a small lag when Android is freeing memory, will get people to hunt for ways to shut down applications. A smart-phone does not have the same abundance of resources, so one must be careful when specifying settings for various synchronization services. So instead of killing an application, say i.e. the popular News Room app, go into the preferences and disable or tweak the background update service instead.
On a modern Android smart-phone, I have yet to see a real need for a task killer apart from shutting down Android Marked when changing carrier with Marked Enabler, but that's not performance related. And the few times I do need that, it's sufficient to do this using the stock Android option you can find by exploring Settings -> Applications -> Manage Applications, pick application and click "Force stop".
On my Nexus One here, after a day of using a bunch of applications I now have 26 running processes in 338MB of RAM, but still 48MB free. In essence, all the applications I use are loaded in RAM and ready for me to use - just as Google intended.
Dianne Hackborn, a Google software engineer on the Android team explains it in detail here.