Plans to build five houses alongside a historic Herefordshire country pub are having to be changed, due in part to extra costs arising from Brexit.