to allow sth that sb has done to make you have a lower opinion of them
hold sth against sb
to allow sth that sb has done to make you have a lower opinion of them
I admit I made a mistake — but don't hold it against me.
Definitions from the Oxford Advanced Learner's Dictionary, 9th edition