He had to take a letter back to his Master from Paul asking for his freedom. Philemon could have punished him severely.

Source: Philemon