python - Youtube Webscraping Comments by new

Question

I'm trying to web scrape a youtube comment section. But before that, I want to sort the comments by new. So I have to click on the SORT BY and then Newest First. But unfortunately, I have no luck. Thank you for helping out.

Screen record: https://imgur.com/Rt8gnIB

code:

import sys, unittest, time, datetime
import urllib.request, urllib.error, urllib.parse
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import InvalidArgumentException
from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.select import Select
from selenium.webdriver import ActionChains
from bs4 import BeautifulSoup
import requests
import requests.exceptions
from urllib.parse import urlsplit
from collections import deque
import re
import os
import shutil
import smtplib
import string
import pyautogui

options = webdriver.ChromeOptions()
options.add_argument('--lang=en')
options.add_argument("--start-maximized")

driver = webdriver.Chrome(executable_path=r'C:UserscaspeOneDriveDocuments?vrigtKodningEmailchromeDriverchromedriver.exe', chrome_options=options)
driver.get("https://www.youtube.com/watch?v=EV6PLN_8RBw")
time.sleep(5)

screenWidth, screenHeight = pyautogui.size()
currentMouseX, currentMouseY = pyautogui.position()

pyautogui.moveTo(1050, 780) # Move the mouse to XY coordinates.
pyautogui.click()
print("clicked")
time.sleep(1)

pyautogui.moveTo(1250, 880) # Move the mouse to XY coordinates.
pyautogui.click() 
print("clicked")
time.sleep(1)

pyautogui.moveTo(700, 500) # Move the mouse to XY coordinates.
pyautogui.click() 
print("clicked")
time.sleep(1)

Answer 1

I would highly recommend to replace the autogui part with firstly element = driver.find_element_by_xpath or by css_selector and element.click And after you choose the xpath/css_selector from the "Newest" option and click again, i did that in another script and had no problems, i cant format it right since im on android tho

tl;dr: replace the autogui by clicking with selenium, firstly the button that makes the options chooseable and then again on the option you want to sort it for

Edit: okay i tried it out myself, the pyautogui did not work for me, not sure if it is because of me using another driver or something, but if it works for you thats fine. The problem is, that you need to scroll down a bit first, in order to load the comments. you can do that using

for i in range(5):
driver.execute_script("arguments[0].scrollBy(0, 500)", element)
time.sleep(2)

not sure if 5 is enough/too much, since i cant check myself, but thats basically how it would work. you can then locate the dropdown open button using

    element = driver.find_element_by_xpath("//*[@id="label"]")
    element.click
    #now we select the option from the dropdown options
    element = driver.find_element_by_xpath("/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[4]/div[1]/div/ytd-comments/ytd-item-section-renderer/div[1]/ytd-comments-header-renderer/div[1]/span/yt-sort-filter-sub-menu-renderer/yt-dropdown-menu/paper-menu-button/iron-dropdown/div/div/paper-listbox/a[2]/paper-item")  
    element.click

you could also use explicitWaits to ensure it is in view, but i dont feel confortable using that yet, you find them here
try it out and let me know if it worked :D