In 1921 - this has a point, bear with me - the compilers of What Editors Want interviewed a lot of the prominent editors (both magazine and publishing house) of the day. They all responded with pages of stuff: formatting advice, genre preference, commercial details, you name it. Very specific.
The best response was a single line from Atlantic's Ellery Sedgewick: "My selection is made according to the whim of one individual."
I think that's a wonderfully honest assessment of the subjectivity inherent in any literary review process - submissions, awards, SPFBOs - etc.
These notes are based on: what I've learned from my betters; from reading hundreds of submissions and slush piles; judging a few awards and competitions; from two years of SPFBO reading; and - most of all - from my own experience making a lot of mistakes. However, as Ellery Sedgewick says, it all boils down to the whims of one individual. (Me. Not him. Obviously.)
Here's my belief: as an author, your primary function is to MAKE GOOD STORY. You don't have to be the best at everything. You've got your role. As a self-published author, however, you also need to MAKE GOOD BOOK. It doesn't matter if you're the finest storyteller in human history, there's much more to the project. As a result, you need to figure out where you, yourself can be 'good enough' at editing, formatting, proofing, publishing, marketing, distributing and designing. And also where you might need to get some help. This takes a lot of self-awareness and - again, from experience - careful budgeting!
The boring details we all hate
This is not a complaint, because I'm delighted to support everyone - but across the two years I've judged SPFBO, I've wound up having to purchase a dozen submissions from Amazon, because there was some sort of quirk in the file-formatting process. 12/60 books is a bad ratio, and, more importantly, that's not setting off on the right foot.
Check your files. Check all the files. Make sure the book you've laboured over isn't falling down at the first hurdle. Don't trust automated ebooking tools like Amazon's conversion tool or a simple Calibre 'Export as'. There is a 99% change it'll fuck something up. And a 99% chance that it can be fixed easily... if you catch it. So double-check everything.
Check your file sizes. If your ebook is 20MB, there's a problem (probably image sizes). Annoying fun-fact of self-publishing: you can send a really big mobi file to Amazon when you upload your book. Amazon compresses it. But if you send, say, that same 8 MB or 12 MB file directly to another reader (or SPFBO judge), Amazon won't let them email it to their device because 'too big'.
Use sensible file names. If you're submitting a book - a manuscript, a mobi for judging, whatever - literally the first thing they will see is the file name. It shouldn't be "book_v4_final_final". The copy you save on your computer can have all the cryptic nomenclature you need to keep it organised, but the copy you send needs to be "author_title". That's it.
Save your creativity for the words, not the formatting
Make your books look like books. All those subtle elements of bookness. Margin width. In print, make sure your chapters start on the right-hand page. Paragraph indents (or not), where they are appropriate. Text justification. No hyphenation. (Absolutely nothing says rookie error like words hyphenating over line breaks). Page numbers. Etc.
Use a normal typeface. Good lord. Don't rely on a fancy typeface to create atmosphere. That should come from the text itself.
Use punctuation. Things like quotation marks, quotes, and chapter breaks all exist for a reason.
Grammar, punctuation, text formatting - these are constraints. This is the truth. They're the rules of the game, and they can be limiting. However, millions of books over the years have been created within these rules, and, more importantly, that's how readers are used to reading. If the reader has to concentrate on deciphering the formatting, they're taken out of the story. The odds are that, however special and unique your idea is, it should still be expressed within the rules. There are, of course, exceptions. But they're far fewer than you'd think, and for every e.e. cummings or William Faulkner that successfully pulls it off, there are ten thousand attempts that are simply annoying to read.
In the name of all that's holy, please don't write whole chapters in italics. Fun fact! Italics were developed by typographers to be deliberately hard to read. It is a style developed for interrupting reader flow, as to emphasise single words. Italics are speed-bumps. If you use them for entire flashback chapters (or interior monologues, or dream sequences...) that's the reading equivalent of driving on an unpaved road. Out of gear. With flat tires.
These are a huge deal, and not just because publishing is secretly owned by Big Comma or the anti-italics illuminati. Whether or not you notice it, books have a certain form to them that make them look and feel like books, and that we, as readers, subconsciously associate with 'the way a book should feel'. Even slight deviations will change the reading experience, and make it so readers spend time thinking about how they're reading rather than the story itself.
The best way to do this is to pick up a book that you love, and steal everything. In this case, don't look at the words, but look at how they are arranged on the page. Set your book and their book down, side by side, starting with the inside front cover. Print, ebook, whatever. Compare them one page at a time. Does your verso look like their verso? Are your margins where their margins are? Do your line breaks look like their line breaks? Now steal. As Picasso famously never said: 'Good artists copy; great artists steal.'
When I laid out the very, very first Jurassic London book, I thought it was a work of genius. I sent it to a friend - who, amongst other talents, is a professional sub-editor. He responded, bluntly, with "nice flyer, shit book", and then proceeded to take me, step by step, through everything above. It was humiliating, but also one of the kindest things anyone has ever done for me.
There are a lot of good (free) tutorials and blogs out there - both for text design and ebook creation. There are also a lot of professionals and service companies that you can turn to for help, if this isn't how you want to spend your time.
Editing is still the most valuable (and undervalued) help you can get
First and foremost: get editorial feedback. There's a surprising amount of confusion out there as to what an editor does, and I think a lot of new authors are under the impression that an editor is the same thing as a proofreader. They aren't. A good editor challenges the structure and pulls out the themes. They tell you which of your characters are interesting and which aren't. They tell you if the sex scene is character-building, or just inappropriate. They're honest and cruel and their job is to make your book better.
Think of an editor as the tough-but-talented creative writing teacher you always wish you had, and, in this case, they're devoted entirely to you. Equally importantly, they bring in a perspective that's not you (a fresh set of eyes) not amateur (your circle of friends, family and fan readers) and not afraid of hurting your feelings. A great editor is a combination of muse, magic mirror and dominatrix.
Or, think about it this way. Occasionally Big Five, Gatekeeping, New York City editors take a stab at being authors, and write their own books. When they do, they don't edit themselves. They don't feel qualified to self-edit, and even though they've spent their 10,000 hours flensing manuscripts, they still look for an outside opinion. They, more than anyone, understand the importance of getting another, close perspective. So, you should too. Although 'editing' isn't an immediate and quantifiable thing (like, say, ebook formatting), editorial input is where you'll find the most value from an outside source.
That said, you'll want a proofreader as well. I've been there with typos and proofing. It is awful. And there are always more. The thing is - even if you're the cleverest, most perceptive, closest reader in the world - you get blind to them, because your brain is so familiar with your own story. You can't proof yourself. Getting 'fresh eyes' isn't a truism, it is behavioural science. And that's the thing: you'll never catch all the typos by yourself. Again, look to getting some help. If you're out to save money, this is where friends and family can help. Buy them a pizza.
If a book starts with several chapters of info-dumping or detailed world-building or six centuries of detailed historical context... you're making your priorities clear, and frankly, they're the wrong priorities. Readers connect with a book - and stick with it - through its characters. That's also why you get all those studies about readers being more empathetic people: it is because we've got a hobby that involves connecting with imaginary strangers for hours and hours on end. Make that connection happen.
Building on that: if you think your book needs to start with several chapters of info-dumping in order for it to 'make sense'... well, there are larger problems. Everything we need to know should come naturally from the characters and the experiences that they are having on the page. You may have - and I bet you do! - an absolutely astounding world in your head, mapped out in every detail. Wanting to share that is only human. But - and here's the shitty truth: strangers don't care. Your reader isn't your family or your friend, and has no moral obligation to be patient while you explain the awesomeness of your world. Give us a character we care about, and then share the world through them. Again, this is why editors can be so helpful - they can tell you the hard truth about what's boring, what's cool, and, above all, what's important.
And, building on that: Please don't open with a description of the landscape. Or have your character examine themselves in a mirror.
Appendices make you feel better, but don't expect anyone to read them. Most - if not all - will read from the first page of the story to (hopefully) the last, and not bother with any of the surrounding materials. Odds are that the genealogy, the pronunciation guide, the currency conversion tables, or the long list of your personal influences are all fun... but will go unread. That's ok! The appendix is a great place to put all that crap you worked so hard on, so you can take it out of the story itself. That's what appendices are for - they're useless chunks of meat excised from the body because they otherwise become a fatal distraction. If there's something essential? Some historical fact or world-detail that's actually critical to understanding the plot or a character's motivation? It needs to be within the actual book.
Show, don't tell. This is the most common advice, and everyone has their own take on it. For me - and for a lot of the books I read - it can be as simple as turning narrative delivery into simple action. Don't tell us a character is outgoing - show us how he flirts with vendors in the marketplace. Don't tell us the city guard are inept - show them cheating on their rounds. Don't tell us the thief is amazing - show them smoothly opening the lock. Don't tell us the tomb is scary - show us how everyone shuffles around, trying not to be in the lead as they enter it. Don't tell us this is a heart-warming romance - show us how the two characters are always holding hands and sharing inside jokes. You're the author, you know these people, you see the places - show them to us, don't just tell us about them.
Why is the character doing something? If anyone is 'strangely compelled', or 'moved by a higher force', or 'doing something despite herself' or 'filled with a curious passion', I'm assuming that's shorthand for 'the author needed something to happen but couldn't figure out how to get there'. (Almost as bad: 'drunk', 'fit of anger' or 'mind-controlled' - those are cheap tricks that make me resent both the character and the plot.) Your character's motivations and the plot should be intertwined. Things should happen for reasons. Reasons that, even if we don't agree with, we can still understand, because they come from characters.
Try to avoid addressing the reader directly. I'm not sure this ever works outside of McSweeney's or Deadpool. Don't break the fourth wall with comments like "Yeah, I know it sounds weird, but this is the sort of thing." or "You won't believe it, but..." or "You know that hot model on TV, she totally looked like that.". First, you're assuming what they're thinking, which puts them on the back foot. Second, it breaks the suspension of disbelief - again you are reminding the reader that they are reading a book, and not 'in' a story. Third, it is (see above) telling not showing. You're directly telling the reader 'this is surprising', 'this is how you should feel', "she was attractive". Show that, instead!
You wrote a book! That is amazing. Like .0000000000001% of the human species has ever been able to this. What you've done is absolutely spectacular. You wanted a particular story to exist, so you went out and created it. Most people don't - or can't. And you did.
But if you're going to put it out there, make sure you're presenting the best book that you can; that it is as good as it can be. The horrible truth is that your book may never 'succeed' - as measured by sales, reviews, or anything else. There are a lot of factors that you'll never be able to control. The best you can do is control what you can, so that, no matter what happens, you know you've given it your best shot.
And the most important thing to remember: by writing a book, your biggest success is already behind you. You should be damn proud of yourself for achieving it.