Selenium 除錯筆記

 由於某些網站擋掉了直接用 php 抓取內容，所以，只好繞道，改用爬蟲把網頁抓回來處理，等到有朝一日熟練了再直接用爬蟲處理完資料整理，But...

症狀：ImportError: cannot import name 'webdriver' from partially initialized module 'selenium' (most likely due to a circular import) (C:\somewhere\selenium.py)

工作環境：
  Windows 10
  Python
    selenium

檢查過路徑，沒有問題，該設引用路徑(path)的都設了，重新安裝也試過了，後來才發現，為了測試，開了一個也叫 selenium.py 的檔案(如症狀所述)，把測試檔案更名就好了。

雖然已經很少碰到保留字的問題，看來，和函式庫檔案同名也要用相同方式處理啊~~

 

症狀： YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

#原本的會出現以上的警訊
#conf = yaml.load(open('somewhere/config.yml'))
#改好了就不會了
conf = yaml.load(open('somewhere/config.yml'),Loader=yaml.FullLoader)
#也可以用這種方式
conf = yaml.safe_load(open('somewhere/config.yml'))

 

webdriver 的操作和一些參數設定：

#如果沒有設好系統路徑

#PATH = "D:/somewhere/geckodriver.exe"

#driver = webdriver.Firefox(PATH)

 

#如果已經設好系統路徑 

#driver = webdriver.Firefox()

 

#把執行中的視窗移出視線，雖然 selenium 執行時滑鼠還是可以正常工作

#driver.set_window_position(-2000, 0)

#還可以指定視窗的起始位置和大小

#driver.set_window_position(0, 0)

#driver.set_window_size(200, 768)

 

from selenium.webdriver.firefox.options import Options 

#如果連瀏覽器視窗也不想看到，用headless模式執行firefox

#網路上的範例

#driver = webdriver.Firefox(options=options, executable_path=r'C:\BrowserDrivers\geckodriver.exe')

options = Options()

options.headless = True

driver = webdriver.Firefox(options=options)

 

 關閉 selenium 遇到的 error:一開始只用 driver.close()並沒有問題，後來又加了driver.quit()就…

症狀：
Traceback (most recent call last):
  File "D:\Somewhere\python\lib\site-packages\urllib3\connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "D:\Somewhere\python\lib\site-packages\urllib3\util\connection.py", line 96, in create_connection
    raise err
  File "D:\Somewhere\python\lib\site-packages\urllib3\util\connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [WinError 10061] 無法連線，因為目標電腦拒絕連線。       

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Somewhere\python\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "D:\Somewhere\python\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "D:\Somewhere\python\lib\site-packages\urllib3\connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)  File "D:\Somewhere\python\lib\http\client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "D:\Somewhere\python\lib\http\client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "D:\Somewhere\python\lib\http\client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "D:\Somewhere\python\lib\http\client.py", line 1010, in _send_output
    self.send(msg)
  File "D:\Somewhere\python\lib\http\client.py", line 950, in send
    self.connect()
  File "D:\Somewhere\python\lib\site-packages\urllib3\connection.py", line 200, in connect
    conn = self._new_conn()
  File "D:\Somewhere\python\lib\site-packages\urllib3\connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x03B9BF10>: Failed to establish a new connection: [WinError 10061] 無法連線
，因為目標電腦拒絕連線。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:/projectsPython/getTaipeiFireDept.py", line 77, in <module>
    driver.close()
  File "D:\Somewhere\python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "D:\Somewhere\python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 319, in execute
    response = self.command_executor.execute(driver_command, params)
  File "D:\Somewhere\python\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 374, in execute
    return self._request(command_info[0], url, body=data)
  File "D:\Somewhere\python\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 397, in _request
    resp = self._conn.request(method, url, body=body, headers=headers)
  File "D:\Somewhere\python\lib\site-packages\urllib3\request.py", line 74, in request    return self.request_encode_url(
  File "D:\Somewhere\python\lib\site-packages\urllib3\request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "D:\Somewhere\python\lib\site-packages\urllib3\poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "D:\Somewhere\python\lib\site-packages\urllib3\connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "D:\Somewhere\python\lib\site-packages\urllib3\connectionpool.py", line 783, in urlopen
    return self.urlopen(
    return self.urlopen(
  File "D:\Somewhere\python\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "D:\Somewhere\python\lib\site-packages\urllib3\util\retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=60820): Max retries exceeded with url: /session/103576e6-5bc9-4d3a-8a3e-966242609736/window (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x03B9BF10>: Failed to establish a new connection: [WinError 10061] 無法連線
，因為目標電腦拒絕連線。'))

#順序應該要醬子，不然就會出現以上的錯誤訊息 

driver.close()

driver.quit()

 

收工！