Can detached garage be demolished if used by neighbour as boundary wall

[Guest]

we live in Semi with very narrow driveway down side to detached garage in rear garden. Neighbour is detached property, on approx 4 foot higher ground. Our garage wall is approx 3 inches or 4 inches inside our side of boundary line. In front of garage and behind garage is boundary fence but there is no fence for length of garage so neighbour effectively has extra 3 inches or 4 inches of garden for length of our garage. We are thinking of demolish garage to make garden bigger. We would propose to put fencing up to match existing fencing. Lived in house 30 years. Neighbours been there about 20 years. We do not know if there was ever a fence on neighbour property along length of garage. Possibly original builder did it like that. Can we demolish garage and the ‘wall’ now used as boundary wall by neighbour?