Блог

У меня есть python2 и python 3 в двух разных средах.

Я создал среду для python2

 virtual-env --python=$(which python2) /spark_streams  

когда я нахожусь в spark_streams, я получаю python2.7, все в порядке

но когда я запускаю pyspark, я получаю pyspark с python3

Когда я пытаюсь запустить pyspark с python2, я получаю ошибки

Мои папки

 ubuntu@u20ks:~$ find / -name pyspark 2gt;/dev/null  /opt/spark/bin/pyspark /opt/spark/python/pyspark /opt/spark/python/pyspark/python/pyspark  ubuntu@u20ks:~$ find / -name python2 2gt;/dev/null  /spark_streams/bin/python2 /usr/share/lintian/overrides/python2 /usr/share/doc/python2 /usr/share/bash-completion/completions/python2 /usr/bin/python2 /usr/lib/python3/dist-packages/virtualenv/create/via_global_ref/builtin/python2   ubuntu@u20ks:~$ find / -name python2.7 2gt;/dev/null  /spark_streams/bin/python2.7 /spark_streams/lib/python2.7 /usr/share/lintian/overrides/python2.7 /usr/share/doc/python2.7 /usr/share/binfmts/python2.7 /usr/bin/python2.7 /usr/local/lib/python2.7 /usr/lib/python2.7  

это сообщение об ошибке

 (spark_streams) ubuntu@u20ks:/spark_streams/bin$ pyspark Python 2.7.18 (default, Mar 8 2021, 13:02:45) [GCC 9.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last):  File "/opt/spark/python/pyspark/shell.py", line 29, in lt;modulegt;  from pyspark.context import SparkContext  File "/opt/spark/python/pyspark/__init__.py", line 53, in lt;modulegt;  from pyspark.rdd import RDD, RDDBarrier  File "/opt/spark/python/pyspark/rdd.py", line 34, in lt;modulegt;  from pyspark.java_gateway import local_connect_and_auth  File "/opt/spark/python/pyspark/java_gateway.py", line 31, in lt;modulegt;  from pyspark.find_spark_home import _find_spark_home  File "/opt/spark/python/pyspark/find_spark_home.py", line 68  print("Could not find valid SPARK_HOME while searching {0}".format(paths), file=sys.stderr)  ^