Good news. Readers are on your side (or at least they start out that way). Suspension of disbelief, which is critical for works of speculative fiction, is also available for other storytellers. The default is trust, so your first job, if you want to bring authenticity to your work, is to maintain that trust.
Now there are ways to gain the trust of people who have not even picked up your work. You may have credentials that are relevant or a reputation for truth or a track record as someone who delivers what readers want. Any of those can give you standing with the reading community (or audiences). Equally, you can arrive in front of them in ways that create doubts — shaky or faked credentials, a history of being deceptive, or simply having your byline on disappointing works.
If you have nothing for you or against you, there is a reason someone picked up your story. It could be a great book cover, a recommendation from a peer or an admired writer, a review, a blurb, or the title. Perhaps, the first page drew the reader in. Of all of these, the only ones really in the writer's control are the title and the first page. That's why even rookie writers are aware of the value of titles, hooks, and strong, clear prose. Chances are, you've read a new writer just because they convinced you within the first few paragraphs that they knew what they were doing and could offer you wondrous language or a compelling situation or a marvelous character or wisdom or a combination of these (including all of the above).
So being a good writer makes a difference, provided your work reaches the right audience. You get to decide who you are writing your story for, and the more dead on you are as far as understanding them and what keeps their attention, the better chance you will have of holding onto credibility.
Some people will accept charm and allow it to paper over all the story holes and inconsistencies. Others will appreciate logic and events chained together so firmly, it's impossible to put the book down. Some readers expect writers to use the exact, best words, and others prefer prose that is more down-to-earth and folksy.
These readers usually are not interchangeable, so writers who seek to gain attention and communicate effectively need to make good estimates of who their audiences are and what they are looking for. Someone who comes for an educated and clinical approach and finds a story that is full of slang or characters who lack erudition, will not appreciate the authenticity the work offers. On the other hand, if your reader is interested in getting lost in a story and doesn’t want to be pulled away to refer to dictionaries and encyclopedias, the work may come off as pretentious and fake.
Of course, deviations — in facts, logic, details (such as how far a horse can travel in a day), and conventional wisdom — sow doubt and can cause readers to lose their faith in a writer. And put the book down.
Losing readers is your fault if you make errors. If you deviate from conventional wisdom, it may or may not be. It is your fault if your excuse is “but that’s what really happened.” The “truth is stranger than fiction” defense may make you feel good, but it will not win the argument. Strange is the operative word here. Reality can present a series of low probability events, but, since (on one level) we come to stories for life lessons, anomalies are not useful. The only real exception is when a hero suffers statistically rare misfortune, but is able to overcome it.
Deviating from conventional wisdom may be the point of a story and can be a wonderful way to share insights with readers. But it isn’t easy. Readers will resist. If you take on conventional wisdom, the argument you make must be compelling and free of any holes or flaws that allow it to be dismissed.
I’ll note that what people “know,” conventional wisdom, and the familiar can be used against some readers successfully. With a setup that, retrospectively, supports an alternative to what readers are assuming throughout the story, a reveal can be a delight. Having the most unlikely person turn out to be the murderer at the end is a convention for mystery readers, provided the clues add up to that conclusion when seen from a new perspective.