There is a natural aristocracy among men. The grounds of this are virtue and talents.
— ... Aristocracy don't believe in forgiveness.
— Neither do the Shelbys.
Gentility without ability is worse than plain beggary.
I think he equates being business-like with being mean. Or worse, middle-class like me.
Explanation:
About Lord Grantham
Aristocracy — government by the best men. (In this sense the word is obsolete; so is that kind of government.) Fellows that wear downy hats and clean shirts — guilty of education and suspected of bank accounts.