A Hereford restaurant owner has lost his battle to stay open into the early morning.