To be honest, it is probably your beard that they are looking at. In some cultures, beards are still a relatively new (or at least revived) fashion.
Of course we are not here to debate fashion, and beards tend to polarise opinion. But I would take some comfort in the fact that there are plenty of positive reasons why someone could be fascinated with your beard - that is to say their looking at your mouth/beard area does not instantly mean there is something weird going on like a piece of spinach in your teeth. Don't let it make you become overly self-conscious. They may like your beard, or be a bit envious of it if they can't grow one themselves. Also if the person is a stranger to you, don't ignore the possibility that they may have a hearing impairment and are relying on watching your lip pattern!
You shouldn't really worry if someone makes a passing glance at your mouth. It seems like you are noticing patterns, either that a lot of different people do it, or a few people you interact with persistently do it.
It may not be possible to completely stop looking there but there may be things you can do to reduce it.
Maintain good eye contact yourself.
We all allow our eyes to wander during conversation. Our eyes may impulsively look up when we access memories or use our imagination. Other people look around when they are thinking. Your breaking eye contact with another person may give them a window to look around too. Concentrating on eye contact may keep their eyes on yours.
Send a non-verbal message that shows them you noticed they were looking at your mouth.
If someone is persistently doing this to you, then you could potentially stop it by letting them know that you know. This could be done either by stopping what you are saying, or markedly slowing down your speech until their eye contact returns. Or you could frown. The drawback of this is that you may come across a little passive-aggressive, depending on how the other person feels about the behaviour. This also ignores the possibility that they may be lip reading.
Say something.
If you really can't stand it, then you could tell the person so. Only do this if they are persistent in the behaviour, and only if you are prepared for the possible consequence of future interactions being awkward.
A lighthearted approach might be to say:
Is there something in my teeth? Just you seem to be looking at my
mouth.
Or the approach favoured by a lot of women when dealing with wandering eyes:
[Pointing at your eyes] I'm up here.