You can use ~/.local/lib and LD_LIBRARY_PATH for shared libs.

Or better yet just give in and use the nix package manager, it is basically a virtual environment for your C programs.