You are, ahem, not decrypting it when getting history and not encrypting it when uploading files. That should be sufficient.

Anyway, look at TG Desktop sources. They are crap, but in general it’s clear what happens there. At least that’s how I remember it.