Neighbour drilled into my detached house external wall

[Sunnuva]

Over 5 months ago (Oct 2020), neighbour drilled into my house wall and fixed a wooden post to install my neighbours garden gate, without my permission. I asked my neighbour politely to remove the post from my wall. He agreed to get it removed in Dec but did nothing. I spoke to him again a month ago. It's now April 2021 and still he's done nothing, even though he agreed to take it down. I kept it polite as I don't want a dispute. What else can I do? Both his house and my house are detached properties but one of my external walls abuts the boundary to his side garden where that new fence is.