I tend to be lucky in that regard, as people around me who might get physical access, tend to not have Linux know-how, even if they think of pranking me.

Also, I lock my room.