Using PhantomJS on PythonAnywhere

 

 

Errors you may encounter

  • selenium.common.exceptions.WebDriverException: Message: Unable to start phantomjs with ghostdriver: [Errno 2] No such file or directory
    • Loading browser...
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "get/motherless/recent/front_page.py", line 11, in main
          browser = load_browser('get.motherless', 'cookies.json', headless=True)
        File "get/utils.py", line 25, in load_browser
          browser = webdriver.PhantomJS(executable_path=path_to_phantomjs, desired_capabilities=capabilities)
        File "/home/nathanwailes/.virtualenvs/auto-video-evaluator/local/lib/python2.7/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 51, in __init__
          self.service.start()
        File "/home/nathanwailes/.virtualenvs/auto-video-evaluator/local/lib/python2.7/site-packages/selenium/webdriver/phantomjs/service.py", line 76, in start
          raise WebDriverException("Unable to start phantomjs with ghostdriver: %s" % e)
      selenium.common.exceptions.WebDriverException: Message: Unable to start phantomjs with ghostdriver: [Errno 2] No such file or directory
    • Solution:
      • https://www.pythonanywhere.com/forums/topic/1320/
        • A note for anyone trying to get phantomjs running. Here is how I did it.
          Let's say I am currently working in /home/conrad/my-site
          First run npm install phantomjs, which will create a node_modules folder.
          Then, from a python script, something like this should work:
          from selenium import webdriver
          driver = webdriver.PhantomJS(executable_path='/home/conrad/my-site/node_modules/phantomjs/bin/phantomjs')

           

  • PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.4/site-packages'
    • Exception:
      Traceback (most recent call last):
        File "/usr/local/lib/python3.4/dist-packages/pip/basecommand.py", line 211, in main
          status = self.run(options, args)
        File "/usr/local/lib/python3.4/dist-packages/pip/commands/install.py", line 311, in run
          root=options.root_path,
        File "/usr/local/lib/python3.4/dist-packages/pip/req/req_set.py", line 646, in install
          **kwargs
        File "/usr/local/lib/python3.4/dist-packages/pip/req/req_install.py", line 803, in install
          self.move_wheel_files(self.source_dir, root=root)
        File "/usr/local/lib/python3.4/dist-packages/pip/req/req_install.py", line 998, in move_wheel_files
          isolated=self.isolated,
        File "/usr/local/lib/python3.4/dist-packages/pip/wheel.py", line 339, in move_wheel_files
          clobber(source, lib_dir, True)
        File "/usr/local/lib/python3.4/dist-packages/pip/wheel.py", line 282, in clobber
          ensure_dir(dest)  # common for the 'include' path
        File "/usr/local/lib/python3.4/dist-packages/pip/utils/__init__.py", line 71, in ensure_dir
          os.makedirs(path)
        File "/usr/lib/python3.4/os.py", line 237, in makedirs
          mkdir(name, mode)
      PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.4/site-packages'
    • Solution:
      • https://www.pythonanywhere.com/forums/topic/1673/
        • Ah! I was just playing around to see what's going on and your second post gave me the clue I needed. pip3 is not installed into the virtualenv, it's the version that is installed on PythonAnywhere, so it's trying to downgrade the Django that is installed for everyone. The pip executable that you run when you've activated a virtualenv is specific to the version of Python that you specified when you created the virtualenv (with the --python argument), so you don't need to specify pip3, because that's already built-in (assuming you created the virtualenv correctly).
        • Basically, when I had created the virtualenv I hadn't specified a version and so it created it with 2.7 by default, but then I realized that my program on my local machine was assuming Python3.4, and so I tried to just run Python3.4 from my virtualenv and install packages to that version, but it turns out (according to the link above) that I was actually trying to modify the non-virtualenv-version of Python3.4, and that's what was causing the permissions error.
  • urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
    • >>> driver.get('cnn.com')
      Traceback (most recent call last):
        File "/usr/lib/python3.4/urllib/request.py", line 1182, in do_open
          h.request(req.get_method(), req.selector, req.data, headers)
        File "/usr/lib/python3.4/http/client.py", line 1088, in request
          self._send_request(method, url, body, headers)
        File "/usr/lib/python3.4/http/client.py", line 1126, in _send_request
          self.endheaders(body)
        File "/usr/lib/python3.4/http/client.py", line 1084, in endheaders
          self._send_output(message_body)
        File "/usr/lib/python3.4/http/client.py", line 922, in _send_output
          self.send(msg)
        File "/usr/lib/python3.4/http/client.py", line 857, in send
          self.connect()
        File "/usr/lib/python3.4/http/client.py", line 834, in connect
          self.timeout, self.source_address)
        File "/usr/lib/python3.4/socket.py", line 512, in create_connection
          raise err
        File "/usr/lib/python3.4/socket.py", line 503, in create_connection
          sock.connect(sa)
      ConnectionRefusedError: [Errno 111] Connection refused
      During handling of the above exception, another exception occurred:
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/home/nathanwailes/.virtualenvs/auto-video-evaluator/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 213, in get
          self.execute(Command.GET, {'url': url})
        File "/home/nathanwailes/.virtualenvs/auto-video-evaluator/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 199, in execute
          response = self.command_executor.execute(driver_command, params)
        File "/home/nathanwailes/.virtualenvs/auto-video-evaluator/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
          return self._request(command_info[0], url, body=data)
        File "/home/nathanwailes/.virtualenvs/auto-video-evaluator/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 463, in _request
          resp = opener.open(request, timeout=self._timeout)
        File "/usr/lib/python3.4/urllib/request.py", line 463, in open
          response = self._open(req, data)
        File "/usr/lib/python3.4/urllib/request.py", line 481, in _open
          '_open', req)
        File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
          result = func(*args)
        File "/usr/lib/python3.4/urllib/request.py", line 1210, in http_open
          return self.do_open(http.client.HTTPConnection, req)
        File "/usr/lib/python3.4/urllib/request.py", line 1184, in do_open
          raise URLError(err)
      urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
    • Solution:
      • This happened because I had just hit another error, and after that first error I tried to access a second website, but I didn't realize that the first error had canceled / ended my browser session. So you need to call 'driver = webdriver.PhantomJS' again.
  • http.client.BadStatusLine: ''
    • Cause:
      • It is probably some kind of incompatibility between PhantomJS and whatever selenium thing they have installed.
        • Sources:
          • https://www.pythonanywhere.com/forums/topic/1605/
            • We've carefully chosen a version of Firefox (17) that works nicely with our default selenium (v 2.35) and with more recent ones (I just tested the latest selenium in a virtualenv). Later versions of Firefox had compatibility problems with our platform.
          • https://www.pythonanywhere.com/forums/topic/1884/
            • OK let's strip it back down to a minimal repro. Create a new Python file, with the following code:

              from pyvirtualdisplay import Display
              from selenium import webdriver
              import traceback
              browser = None
              
              try:
                  display = Display().start()
                  browser = webdriver.Firefox()
                  browser.get('http://www.google.com')
                  print browser.title
              
              except Exception as e:
                  traceback.print_exc()
              
              finally:
                  if browser is not None:
                      browser.quit()
                  display.stop()
               

              If it works, then it must be something that's different about your specific code, and then we can try and track that down. If it doesn't work, then it must be something to do with your user account (because it works fine in mine), and we can try and track that down...

    • Solution:
      • Basically just take the code that harry provided in that snippet above and make your code look like it.
        • Install the 'pyvirtualdisplay' module.
        • Insert 'display = Display().start()' at the beginning of your function and 'display.stop()' at the end.
        • Use 'browser = webdriver.Firefox()' instead of PhantomJS.
    • Things that didn't end up solving the issue: