Just take a look at the UI examples: https://we.phorge.it/uiexample/view/PHUITagExample/

The range of the height of the tags range from 19px to 21px. You will see that these inconsistencies will bleed out into various aspects of UI:

My solution would be: Give tags a fixed height and change the method of the centering the content from using vertical padding to inline-flex. So it would only take the change of height to make bigger or smaller tags. No fiddeling around anymore with paddings.