취미로 하는 데이터 분석 시리즈06(웹사이트 크롤링 어플리케이션)

In [1]:

#이번에는 데이터를 직접 모아보자.
#이미지 크롤링 
#yandex.com에서 검색한 뒤에 사용할것.

In [2]:

from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import time
import urllib.request
import os

urls에 yandex.com에서 검색 이후 첫번째사진을 클릭하고 그 Url을 넣는다.

In [3]:

urls = "https://yandex.com/images/search?text=starbucks&pos=0&img_url=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FDtEGmKYUwAEtoSj.jpg&rpt=simage"
#예시로 사용될 커피 검색
img_list = []
amount_of_img = 10 #얼마나 가져올 것인지.

In [4]:

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(urls)
driver.implicitly_wait(time_to_wait=5)
for i in range(amount_of_img):
    a=driver.find_element_by_css_selector("img.MMImage-Origin")
    imgsrc = a.get_attribute("src")
    driver.find_element_by_css_selector("img.MMImage-Origin").click()
    img_list.append(imgsrc)
#full_html = driver.page_source
#soup = BeautifulSoup(full_html, 'html.parser')
driver.close()

====== WebDriver manager ======
Current google-chrome version is 98.0.4758
Get LATEST chromedriver version for 98.0.4758 google-chrome
Driver [C:\Users\se99a\.wdm\drivers\chromedriver\win32\98.0.4758.102\chromedriver.exe] found in cache
C:\Users\Public\Documents\ESTsoft\CreatorTemp/ipykernel_4820/1006719732.py:1: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
  driver = webdriver.Chrome(ChromeDriverManager().install())
C:\Users\Public\Documents\ESTsoft\CreatorTemp/ipykernel_4820/1006719732.py:5: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead
  a=driver.find_element_by_css_selector("img.MMImage-Origin")
C:\Users\Public\Documents\ESTsoft\CreatorTemp/ipykernel_4820/1006719732.py:7: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead
  driver.find_element_by_css_selector("img.MMImage-Origin").click()

In [5]:

dirnum = 1 #directory를 간편하게 사용하기 위함

In [6]:

if os.path.isdir(os.getcwd()+f"./img{dirnum}")==False:#파일존재유무확인
    os.mkdir(f"./img{dirnum}")#폴더만들기

In [ ]:

for i in range(len(img_list)):
    url = img_list[i]
    try:
        urllib.request.urlretrieve(url, f"./img{dirnum}/num{dirnum}-{i}.png")
    except:
        continue

이러면 폴더 내에 이미지가 들어가게된다.

변경가능 부분: urls, dirnum, amount_of_img, 디렉토리 이름 url의 경우 yandex에서 검색 이후 사진을 누르고, 그 url을 받아와야함.