Each point in the diagram represents a specific pair of temperature and pressure. For example, typical room conditions, with a pressure of 1 atmosphere (14.7 pounds per square inch) and a tempearature of 20 degrees C (68 degrees F), are represented by point A.
Suppose we have a water sample at point A, and we begin increasing its temperature. Eventually, its temperature will reach a point (100 degrees C) where it will change from a liquid to a vapor. The water boils. In the diagram above, this corresponds to process A->B. Note that this process crosses the "vaporization line", indicating that the water has changed from a liquid to a vapor.
If you've ever read instructions for cooking food at high elevation, you know that the recommended cooking times are significantly longer. This is because water boils at a lower temperature at high elevations. The reason for this is that pressure is lower at high elevations, and lower pressure leads to a decrease in boiling temperature.
Imagine starting with a water sample at a point only slightly below A in the diagram above, and then moving to the right by increasing its temperature. (Perhaps we are cooking spaghetti somewhere with a lower ambient pressure, such as Denver, CO). The temperature at which the water will boil is slightly less than 100 degrees C. This explains the lower-left to upper-right slope of the vaporization line in the diagram.
This slope is important, as in a sense, it is what makes cavitation possible. Imagine that we again have a sample of water at room condtions, at point A. This time, imagine that we somehow lower the pressure. Suppose we keep lowering the pressure. As you can see from the diagram, if we lower the pressure far enough, the water will again change from a liquid to a vapor as it crosses the vaporization line. (The pressure at which this occurs is called the "vapor pressure" of water, indicated as p_v in the diagram). Did the water "boil"? Technically, no, as because this process (A->C) is so different than the one (A->B) described before, scientists have given it a different name -- cavitation.
To summarize:
Boiling (A->B)
Change from liquid to vapor due to increase in temperature.
Cavitation (A->C)
Change from liquid to vapor due to decrease in pressure.
Based on the definition of cavitation given above, the short answer is "anywhere a liquid undergoes a pressure drop of sufficient magnitude". But where and how might such a pressure drop occur?
Well, fluid flows are full of pressure differences. Genreally speaking, moving fluid has less pressure than stationary fluid. So fluid motion can create changes in pressure. It is these differences in pressure on opposite sides of solid bodies that create the forces that enable a wing to lift an airplane, a curveball to curve, or a propeller to drive a boat.
Of course, in the case of cavitation, the fluid in question must be a liquid, such as water. So examples of places where cavitation might occur include ship and submarine propellers, pumps, pipes, valves, and spillways.
A good question. The reasons can perhaps best be organized into three main areas -- noise, damage, and performance.
When cavitation bubbles form, they are usually not long lived. It so happens that the collapse of these vapor bubbles (which occurs when the fluid returns to a region of high pressure), is very violent. Much energy is radiated during the collapse, a lot of it in the form of sound.
This noise can be quite problematic. In the case of military vessels, it makes them more readily detectable by enemy surveillance. It can also be quite annoying, as you may personally know if you have ever heard a hissing valve in a filling toilet.
This same collapse process can also lead to extensive damage of hydraulic equipment. The collapse of cavitation bubbles near a solid surface can quickly and dramatically pit nearly any surface, including steel and aluminum.
Finally, the efficieny of hydraulic equipment such as pumps and propellers is often greatly reduced under cavitating conditions. Even more problematic is that cavitation can often induce instabilities in fluid flows that can lead to catastrophic failure.