I've been meaning to do a post on this for a long time. Five years ago (!) I wrote a post about the meaning of temperature, where I tried to go from the intuitive understanding given by common experience (temperature has something to do with energy content, and that energy flows from hot things to cold things) to a deeper view (that flow of energy comes from the tendency of the universe to take on macroscopic configurations that correspond to the most common ways of arranging microscopic degrees of freedom - the 2nd law of thermodynamics, basically). I wasn't very satisfied with how the post turned out, but c'est la vie.
Chemical potential is a similar idea, but with added complications - while touch gives us an intuition for relative temperatures, we have no corresponding sense for chemical potential; and the rigorous definition of chemical potential is more complicated. (For another take on this, see this article, available in full text via google from a variety of sources.)
Let's reason by analogy with temperature. Energy tends to flow from a system at high temperature to a system at low temperature; when systems with identical temperatures are brought into contact so that they may exchange energy (e.g., by thermal conduction), there is no net flow of energy. Now suppose systems are able to exchange particles as well as energy. If two systems are at the same temperature, then particles will tend to flow from the system of higher chemical potential (one of the several parameters denoted by the symbol \(\mu\)) to that of lower chemical potential. If two systems have identical chemical potentials for a particular kind of particle, there will be no net flow of particles. In general, particles tend to flow from regions of high \(\mu/T\) to regions of low \(\mu/T\). The classic example of this is the case of a closed bottle of perfume in a room full of (non-perfumed) air. The perfume molecules have a high \(\mu\) in the bottle relative to the rest of the room. When the bottle is opened, perfume molecules will tend to diffuse out of the bottle, simultaneously lowering their \(\mu)\) in the bottle and increasing their \(\mu\) in the room. This will continue until the chemical potentials equalize. From the point of view of entropy, there are clearly very many more arrangements of molecules with them roughly spread throughout the room+bottle than the number of arrangements with the molecules happening to occupy just the bottle. Hence, the universe tends toward the macroscopic configuration corresponding to the most microscopic configurations. Bottom line: equilibrium between two systems that can exchange particles requires equal temperatures and equal chemical potentials.
Where this also gets tricky is that thermodynamics tells us that \(\mu\) also corresponds to the energy per particle required to add (or remove) one particle from the system at constant temperature and pressure (!). This identity is not at all obvious from the above description, but it's nevertheless true. This latter way of thinking about chemical potential means that when particles can couple to some "real" potential (gravitational, electrical), it is possible to tune their total \(\mu\). The connection to the entropic picture is the idea that particles will tend to "fall downhill" (there are usually fewer configurations of the combined system that have some particles "stacked up" in a region of high potential energy with others in a region of low potential energy, than the situation when the energy gets spread around among all the particles).