to attack sb by hitting them or speaking angrily to them
let fly (at sb / sth) (with sth)
He let fly at me with his fist.
Definitions from the Oxford Advanced Learner's Dictionary, 9th edition