python - Cannot pip install package in virtualenv on EC2

Question

I'm seeing this weird issue on ec2. I'm trying to install lsm-db package inside my virtualenv, it says it's successfully installed but when trying to import the package or do pip list it's not there.

I created this virtualenv by

virtualenv -p python3.6 venv
source venv/bin/activate

then running which python and which pip shows the correct ones inside venv as <path-to-my-project>/venv/bin/pip. Then I run pip install -r requirements.txt where sqlalchemy and lsm-db are included. A warning is shown

WARNING: Building wheel for lsm-db failed: [Errno 13] Permission denied: '/home/ec2-user/.cache/pip'
  WARNING: Building wheel for sqlalchemy failed: [Errno 13] Permission denied: '/home/ec2-user/.cache/pip'
Failed to build lsm-db sqlalchemy
Installing collected packages: sqlalchemy, flask-sqlalchemy, lsm-db, MarkupSafe, click
  Running setup.py install for sqlalchemy ... done
  Running setup.py install for lsm-db ... done
Successfully installed MarkupSafe-1.1.1 click-7.0 flask-sqlalchemy-2.4.0 lsm-db-0.6.4 sqlalchemy-1.3.5

The last line shows they are correctly installed despite the warnings. However, they are not. Running pip install lsm-db gives

? pip install lsm-db
WARNING: The directory '/home/ec2-user/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/home/ec2-user/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lsm-db
  Downloading https://files.pythonhosted.org/packages/17/f2/dc68ecdece424eb206c42b8db924c5a7f6dd97d26df42a1c9fb41a6fe555/lsm-db-0.6.4.tar.gz (316kB)
     |████████████████████████████████| 317kB 14.7MB/s
Building wheels for collected packages: lsm-db
  WARNING: Building wheel for lsm-db failed: [Errno 13] Permission denied: '/home/ec2-user/.cache/pip'
Failed to build lsm-db
Installing collected packages: lsm-db
  Running setup.py install for lsm-db ... done
Successfully installed lsm-db-0.6.4

Again it shows it's installed, but it's not. Not in pip list, can't import in Python. What is going on here? Why do I see permission issue inside virtualenv? I didn't create the virtualenv with sudo if that's a potential problem. Is it related to the way I installed virtualenv?

---

EDIT

I changed the permission and reinstall pip with sudo yum install python36-pip. Now I'm seeing this... sorry it seems like a new problem

Running setup.py install for lsm-db ... error
    ERROR: Complete output from command /home/ec2-user/<project>/venv/bin/python3.6 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-bhhmd3j8/lsm-db/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'
'"'"', '"'"'
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-__gotpo6/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/<project>/venv/include/site/python3.6/lsm-db:
    ERROR: /tmp/pip-install-bhhmd3j8/lsm-db/setup.py:11: UserWarning: Cython not installed, using pre-generated C source file.
      warnings.warn('Cython not installed, using pre-generated C source file.')
    running install
    running build
    running build_ext
    building 'lsm' extension
    creating build
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/src
    gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -c lsm.c -o build/temp.linux-x86_64-3.6/lsm.o
    lsm.c:32:20: fatal error: Python.h: No such file or directory
     #include "Python.h"
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command "/home/ec2-user/<project>/venv/bin/python3.6 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-bhhmd3j8/lsm-db/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'
'"'"', '"'"'
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-__gotpo6/install-record.txt --single-version-externally-managed --compile --install-headers /home/ec2-user/<project>/venv/include/site/python3.6/lsm-db" failed with error code 1 in /tmp/pip-install-bhhmd3j8/lsm-db/

---

EDIT

Thanks for the pointers, now I installed the python development package on ec2 and tried to install lsm-db again

> sudo yum install python36-devel
...
> pip install lsm-db
Collecting lsm-db
  Using cached https://files.pythonhosted.org/packages/17/f2/dc68ecdece424eb206c42b8db924c5a7f6dd97d26df42a1c9fb41a6fe555/lsm-db-0.6.4.tar.gz
Building wheels for collected packages: lsm-db
  Building wheel for lsm-db (setup.py) ... done
  Stored in directory: /home/ec2-user/.cache/pip/wheels/ed/32/8a/b0f3aa15b68f1e6b8f2dcc6285de4796b6b58af71c5f280f48
Successfully built lsm-db
Installing collected packages: lsm-db
Successfully installed lsm-db-0.6.4

again, it shows it's successful, and which python gives me the right python in my venv. However, I have the exact same result as the beginning, pip list does not have lsm-db and I cannot import it in python.

This is entirely an issue on ec2, I can successfully use lsm-db in my virtualenv on my mac. There must be something off with virtualenv on ec2 here, why the right pip in venv cannot install a package in venv?? I'm completely lost. If anyone has experience please chime in!

(To work around this problem, I'm switching from lsm-db to sqlitedict in my code. sqlitedict can be installed no problem, so it narrows the suspect down to just lsm-db specific issues.)

This question from 2016 is more similar to mine but it has no conclusion why or any solution. Pip says that pymongo has been installed, but doesn't show it as installed

Another potentially related question: Amazon EC2 virtualenv: pip says it installed numpy but python can't find it

Answer 1

The problem seems to be with system file permissions. You can check the owner:group as well as rwx permissions for other users of the folder '/home/ec2-user/.cache/pip' from terminal using:

ls -lh /home/ec2-user/.cache/pip

In order to change the owner of the folder use command:

sudo chown -R ec2-user /home/ec2-user/.cache/pip

Another way to solve is to allow rwx permissions to all users by:

sudo chmod -R 777 /home/ec2-user/.cache/pip