A planned new art gallery in a historic Herefordshire spot has been given the go-ahead despite fears over the impact on its surroundings.