Language, when you think about it, is an amazing thing. When you consider carefully, language is a form of telepathy. When two people share the same language and the same level of vocabulary, they can convey their inner thoughts clearly. When they put it in writing, they can extend their thoughts beyond the moment and relay it across time itself!
The stronger your vocabulary is, the more complex the thoughts are that you are able to express. There is a paradox in this however. Having an extensive vocabulary can interfere with this telepathy if the person you are trying to use it with does not share your level of linguistics. I personally suffer from this problem a great deal. I often find that I have to reword things even in casual conversation since I forget that not everyone knows the extensive number of words I do.
There is a good reason why most of the popular novels don’t use a lot of fancy words to say what they need to. At most, they change it up just enough to keep from being repetitive. When most people graduate with a 5th grade reading level in the US, using language that forces a college student to hunt for a Thesaurus is not going to reach most audiences. In fact, I remember a few times in college where a professor would ask what reading people did outside of class work. Most said they read nothing more than a few newspaper articles unless it was required. Consider that for a moment when you start putting words into print.
This means that while I might know what an ampersand is, there are a fair number of people who might not. If I use the word instead of the ‘&’ symbol, it should be with purpose. For example it could be said by a character who doesn’t understand that many people don’t know what that is. If I am just using such words as a matter of course, I will lose a lot of readers.
If you are of the opinion that those people who don’t understand what you are saying are not your target audience, that is fine. Just realize that you are probably not going to reach nearly as large of an audience. You can complain all you want about how ‘dumbed down’ the popular works are or how you write for a ‘literary audience’, but it isn’t going to change that you have self-limited the potential audience who might otherwise have enjoyed what you had to say.
Being an author is not just about having something to say, but also about having the ability to say it in a way that reaches people. Hone your vocabulary as sharply as you can and then only use it to the extent necessary to get your point across without losing people’s attention. It isn’t really dumbing down if you are still able to get your point across. It is simply using different words than you might otherwise to ensure that your telepathy is being clearly relayed.