A plan to demolish a large house in grounds near a Herefordshire town and replace it with 11 new houses has been refused permission.