rvest fails to parse HTML page from google scholar; returns xml_nodeset (0)

[Rohit-Satyam]

Hi @hadley

rvest v 1.0.3 was working just fine a few hours ago and suddenly it stopped working. I was trying to scrap some paper titles using a gene ID on Google Scholar and it was doing a good job but now I get xml_nodeset (0)

    url <- paste0("https://scholar.google.com/scholar?q=", "PF3D7_0420300")
    
    # Fetch the HTML content of the search results page
    page <- rvest::read_html(url)
    
    # Extract titles and publication years
    titles <- page %>%
      rvest::html_nodes(".gs_rt a") %>%
      rvest::html_text() %>%
      trimws()

Even code chunk given here returns empty results:

[Rohit-Satyam]

I tried rvest on my laptop and it's working. This is weird. Is it something related to multiple queries emanating from single IP?

[hadley]

It's almost certainly because some automated system is trying to block you from scraping.