A Herefordshire farmer believed he did not need planning permission to build a new sheep barn.