There are two forms of alien concepts; those that are somewhat alien and those that are completely alien. Both have their place in literature and can be powerful tools in creating a sense of otherworldly unfamiliarity. Knowing how to use them in your writing can greatly enhance the feel of your worlds. Overuse can draw the reader too far out of their own self so that they are no longer able to put themselves into the events of the story.
Somewhat alien concepts are those which are mostly just foreign. They are things that relate to distant cultures, strange-but-reasonable behavior or matters which make sense to us after we understand what the drive behind them was. This is where most of the alien will fall within an average story. As long as you recognize that your reader may not be familiar with a certain way of behaving or thinking and do a little to make it more clear at some point within the writing, you almost can’t go wrong.
The somewhat alien is also somewhat familiar and can add a grand sense of depth to the world you are writing about. Do certain countries have unusual (not too unusual or it becomes unintelligible) dialects? What exactly is the reason dragons seem so enamored of virgins in the stories of old? What sort of plant life would exist in an alternate reality where dinosaurs never went extinct? These sorts of things aren’t so far away that we can’t grasp them, as long as you do something to clarify them.
The completely alien is another story. Usually this only works for horror or mystery style writing. Alien worlds that are completely alien may be very accurate to how the real thing might be, but they also aren’t going to be understandable to the reader. The same is true of beings whose thoughts and actions are completely alien, since their logic and ours would be so divergent. It is very easy for the actions to come across as random and without forethought.
Sure there is a reason they never look under tables, but the reader is going to be hard pressed to understand why in the world the earth-invaders failed to notice the protagonist down there after completely tearing the rest of the place apart. However you explain it is going to fall short as being a simple writing crutch to allow your protagonist to go unnoticed.
The reason this works so well in mystery and horror is simple. In mystery writing, the whole point is to come to understanding as to why something that doesn’t make sense happened or to understand what went on behind the scenes. You focus the whole story or even novel on helping the reader grasp all of the subtle undertones and strange thinking that tie it all together.
On the other side, horror can readily focus on the alien and never both explaining it out all the way. As long as the protagonist dealing with it is utterly believable, the strange monsters need not be fully understood. Don’t mistake that to mean you don’t have to personally understand why they do what they do. Only that you do not have to fully reveal everything about the antagonist. H. P. Lovecraft was a master of the slow reveal and you almost never came to any real understanding of the drives and motivations of the antagonists aside from their immediate actions and a few visceral aspects.
Add the alien to create diversity and a sense of uniqueness. Just remember that it is a sword that cuts both ways. Too little and things are cliché. Too much and things are too bizarre to follow or become a distraction. Regardless, always give thought to the alien as it appears in your writing. Use it wisely and balance it well with the familiar.