Prime Minister Anthony Albanese will “panic” if the Victorian Labor Party fails to hold the seat of Werribee at Saturday’s byelection, state Opposition Leader Brad Battin has warned.