Sometimes you surprise yourself. When you do this as a writer, it can provide big thrills for you and your readers. But, just as happiness is approached indirectly, there is no straight-line formula for self-surprise.
To be surprised, you need to be open to being surprised. This usually means taking the stress off and allowing yourself to have some fun. John Cleese recommends being patient, giving your brain the time to be creative.
Knowledge is a critical component as well. The more you give slack to your curiosity, gathering "useless" information and allowing yourself to dig deeply into areas that fascinate you, the more you'll have at hand to connect to and ways that make sense but don't conform to conventional wisdom.
Another form of knowledge is developing a better understanding of surprise itself. In the last entry, I provided some of the literary uses of surprise, but you can build beyond those. The first step is noticing when you and others get surprised and then analyzing the process. Was the surprise startling? How novel was the information? What was supposed to happen? Was what did happen justified? If you become sensitive to times when you experience surprise and begin to note and analyze these, you'll take steps toward being able to do the same tricks or improve on them. (Sometimes, a disappointment or a failed attempt to surprise can be more instructive than a successful execution.)
Not all surprises are good. In fact, the original meaning of the word was more connected with an ambush than a thrill. To the extent, we protect ourselves from surprising situations. We make plans. As writers, we plot, outline, analyze, and describe. With all the best intentions, we often make surprises that might enhance the manuscripts unlikely.
This is one reason why I'm a big believer in fast drafting a manuscript. Accepting what your brain creates and ignoring the internal editor often opens up possibilities that societal strictures, "rules," and "good taste" preclude. Write fast enough and anything can happen.
This may seem intemperate to you. If so, my next suggestion maybe even more upsetting. Cultivate opportunities to paint yourself into corners. Give your viewpoint characters impossible choices. Select topics and perspectives that are way out of your comfort zone. (Okay, this might not sound like fun, but you can learn to like it.)
You can also add some pressure by setting a timer and forcing yourself to write into the unknown and reach some sort of an endpoint before the timer goes off.
You can also force yourself to explore options. This can be as simple as making a list of 10 to 20 possibilities (and choosing one from the second half of the list) or it can be as challenging as writing several versions of a scene.
Most mystery writers know who the culprit is and how they did it before they begin the story (or so I'm told). So, though I have faith that surprising yourself is this your respect to surprising readers, it is
possible to work from and ending to create the delights of the unexpected. There are several aspects to pulling this off, including charming and distracting the reader so that evidence isn't properly noticed. It's also good to lead people astray with alternate expectations and to introduce new ideas and information in ways that guarantee the reader's understanding. Ultimately, the payoff must be worthwhile and clear.
The value of the conclusion must be high enough to justify the reader's investment in time and engagement. It must be worth the price. Usually, this means something new must be learned, whether it be about who characters really are or how information connects (including meaningful juxtapositions). The payoff isn't much of a payoff if it's expected or only slightly different from what's expected or disappointing compared to what the reader hoped for.
Needless to say, all surprises – not just solutions to murders — must be fair. Except in cases where the author only intends to shock the reader, information must be provided that would have allowed readers to predict the the surprising elements. (One TV writer said the surprise was justified if half his audience could have guessed it. But, of course, you hope none of them actually do.)
A few more notes:
Other than self protection, the main thing working against surprise for authors is politeness. While, as individuals, we may not want to be rude, cultivating a level of social chaos within a story and being actively cruel to our characters can create situations that take readers away from the familiar, enhancing their enjoyment of the stories. It's also important to develop a tolerance for some inconsistency and to put up with contradictions within characters. Logic and much of what we learn in schools push us toward making sense.
As storytellers, we need more latitude. It's difficult to name a major character in fiction who isn't riddled with contradictions. And one prominent scriptwriter told me that every movie has at least one important logical flaw. So, you have my permission to step away from order and dip your toe into chaos from time to time – provided it makes the story better.