What if players got a "Drop player" button next to the "Vote to cancel" button to boot players out of time, instead of auto-dropping? Could be implemented such that any player in the game has the authority to press it. This way a single player annoyed by a slow player who runs out of time can end the game, but also if it's a one off case where the player needed a bathroom break and all the players want to wait for him to return, they have the ability not to drop him.