Plans to turn a landmark Hereford building into a centre for music education have fallen through.