As ZZ has pointed out, Nature is running a feature article on the history of high temperature superconductivity over the last 25 years. I remember blogging about this topic five years ago when Nature Physics ran an excellent special issue on the subject. At the time, I wrote a brief summary of the field, and I've touched on this topic a few times in the intervening years. Over that time, it's pretty clear that the most important event was the discovery of the iron-based high temperature superconductors. It showed that there are additional whole families of high temperature superconducting materials that are not all copper oxides.
Now is a reasonable time to ask again, what is so hard about this problem? Why don't we have a general theory of high temperature superconductivity? Here are my opinions, and I'd be happy for more from the readers.
- First, be patient. Low-T superconductivity was discovered in 1911, and we didn't have a decent theory until 1957. By that metric, we shouldn't start getting annoyed until 2032. I'm not just being flippant here. The high-Tc materials are generally complicated (with a few exceptions) structurally, with large unit cells, and lots of disorder associated with chemical doping. This is very different than the situation in, e.g., lead or niobium.
- Electron-electron interactions seem to be very important in describing the normal state of these materials. In the low-Tc superconductors, we really can get very far understanding the normal starting point. Aluminum is a classic metal, and you can do a pretty good job getting quantitative accuracy on its properties from the theory side even in single-particle, non-interacting treatments (basic band theory). In contrast, the high-Tc material normal states are tricky. Heck, the copper oxide parent compound is a Mott insulator - a system that single-particle band structure tells you should be a metal, but is in fact insulating because of the electron-electron repulsion!
- Spin seems to be important, too. In the low-Tc systems, spin is unimportant in the normal state, and the electrons pair up so that each electron is paired with one of opposite spin, so that the net spin of the pair is zero, but that's about it. In high-Tc systems, on the other hand, very often the normal state involves magnetic order of some sort, and spin-spin interactions may well be important.
- Sample quality has been a persistent challenge (particularly in the early days).
- The analytical techniques that exist tend to be indirect or invasive, at least compared to the desired thought experiments. This is a persistent challenge in condensed matter physics. You can't just go and yank on a particular electron to see what else moves, in an effort to unravel the "glue" that holds pairs together (though the photoemission community might disagree). While the order parameter (describing the superconducting state) may vary microscopically in magnitude, sign, and phase, you can't just order up a gadget to measure, e.g., phase as a function of position within a sample. Instead, experimentalists are forced to be more baroque and more clever.
- Computational methods are good, but not that good. Exact solutions of systems of large numbers of interacting electrons remain elusive and computationally extremely expensive. Properly dealing with strong electronic correlations, finite temperature, etc. are all challenges.