Wait...Don't We All Know The Difference Already?
The commonsense way to differentiate fantasy from science fiction is simple: fantasy has magical wonder; science fiction has technological wonder. Alternatively, fantasy draws from mythology; science fiction draws from scientific theory. Fantasy projects backwards to reimagine history (or the present); science fiction projects forward in order to imagine the future. These methods of distinguishing the sister genres have a lot going for them, not least of which is the fact that they represent how most people think about the question, when they think about it at all.
Still, like Ian, I think we can do better. Or, to put it another way, I think we’re better served by a more sophisticated mode of categorization. That does not mean we need more categories. In fact, as much as categorization is a part of human cognition, the fact is that no typology is ever 100% accurate—life is a series of interlocking Venn Diagrams, not a collection of neat boxes. When you trade in two categories for three, you are often just exchanging one fuzzy area for two. Worse, your new category may only describe a limited range of cases in that fuzzy area. Crossovers are okay—it’s fine to just call something science fiction with fantasy elements, or fantasy with science fiction elements. We don’t need silly categories like “science fantasy” that obscure more than they reveal. (Is the fantasy really “scientific,” as the grammar would indicate? And why does the introduction of fantasy elements make it not fiction?)
What we need is more precise and interesting definitions for our base categories.
Distinctions, Distinctions
Besides the science vs. fantasy fiction dichotomy (to which we should also add horror fiction), there’s a broader differentiation between so-called speculative fiction, on the one hand, and fantastic fiction (i.e. fantasy and horror) on the other. Speculative and science fiction correlate to a large degree, but are nevertheless different from one another. The speculative alternative to the science fiction label emerged with the New Wave, and from the sense that “science fiction” was too positivist, too Campbellian, too restrictive and, well, kind of embarrassing to identify with. For some practitioners, in fact, use of "speculative" in place of "science" fiction is just code for "I've got literary pretensions and think I'm better than you."
For others, though, the speculative fiction label represents the attempt to widen the tent, taking in not only traditional science fiction, but also alternate history and the sort of “alternate present” fiction that authors like Thomas Pynchon and JG Ballard made famous. Certainly proponents of this usage also hope to leave the genre ghetto, or at least make it look more respectable, but in this case it's secondary.
The basic premise of speculative fiction, within this paradigm, is to take present-day reality, change a discrete number of variables and explore what that would be like. The changes are all plausible, given our understanding of the physical world, human psychology, sociology and so forth. Fantastic fiction is similarly premised on altering a set number of conditions in present-day reality; only those changes are odds with said understanding of how things work. That yields our first categorical distinction:
Speculative fiction is the literature of plausible alternatives to present-day reality.
Fantastic fiction is the literature of implausible alternatives to present-day reality.
By taking these broad definitions as a starting part, it becomes clear that the main labels of "wonder literature"--science fiction, fantasy and horror--are subsets or internal categories to these broader categories. Within the speculative category, we can place science fiction, historical fiction, alternate history fiction and alternate reality/slipstream fiction. Within fantastic fiction, there's fantasy (and all internal differentiation between epic/S&S fantasy and contemporary/urban fantasy), paranormal fiction and horror. Ignoring the little fellas for the moment, here are what I think are useful definitions for the big three--science fiction, fantasy and horror:
Science fiction is the literature of physical or technological alternatives to present-day reality.
Fantasy is the literature of magical or metaphysical alternatives to present-day reality.
Horror is the literature of paranormal or supernatural alternatives to present-day reality, in which the alternatives are presented as necessarily malignant.
Final Note
Inevitably, someone will take issue with this scheme on the basis of sound argumentation and compelling evidence. To me that's really cool, so I look forward to that. In fact, here's an example drawn up by someone whose thinking is close to mine, but different in fundamental ways:
[Source.]
However, inevitably someone else will take issue with these scheme on the basis that one or two books don't really fit into any of the categories. I won't respond with the trite phrase “the exception proves the rule,” because exceptions don’t prove anything except that there are exceptions. And with categorization schemes, there are always exceptions, crossovers, hybrids and so on. So instead I'll just say: "yeah, you're right--that doesn't fit. Now let's embrace the gray.