Ad

Save Dynamically Loaded Webpage

This should be an easy task but I couldn't handle as I know nothing about (even very basic) web architecture.

I would like to access to links to every courses under https://www.coursera.org/browse/arts-and-humanities/history with some filter (e.g. language=english): Coursera history courses in English.

After loading this webpage, many courses don't show before scrolling down. If I save the html file to local, I can find only 58 instances of https://www.coursera.org/learn/, the prefix to a course, but I'm suppose to get at least 128 of them.

So now how do I save a dynamically loaded webpage, either with Chrome or Python?

With @Rajat's code, the emulator could scroll down to bottom, but still the obtained html is incomplete.

import os
from bs4 import BeautifulSoup
import time
from selenium import webdriver

current_dir=os.getcwd()
#download chromedriver for you operating system
driver = webdriver.Chrome(current_dir+'/chromedriver')
#place your url here
url="https://www.coursera.org/browse/arts-and-humanities/history?facets=skillNameMultiTag%2CjobTitleMultiTag%2CdifficultyLevelTag%2Clanguages%3AEnglish%2CentityTypeTag%2CpartnerMultiTag%2CcategoryMultiTag%2CsubcategoryMultiTag%3Ahistory&sortField="
driver.get(url)
count = 1200
step = 30
for _ in range(count):
    driver.execute_script("window.scrollBy(0, {});".format(step))
    time.sleep(0.01)

with open("output.html", "w") as file:
    file.write(driver.page_source)

Ad

Answer

I took @Gautam code and only rebuilt it.

First request gives only 100 items (even with limit=300) so using start I get next 28 items.

Using json= instead of data= I don't need headers= and json.dump()

#!/usr/bin/env python

import requests
import warnings
warnings.filterwarnings("ignore")


def display(data):
    #print('len:', len(data))
    #print('len:', len(data[0]['data']['CatalogResultsV2Resource']['browseV2']['elements']))
    print('>>> len:', len(data[0]['data']['CatalogResultsV2Resource']['browseV2']['elements'][0]['courses']['elements']))

    items = data[0]['data']['CatalogResultsV2Resource']['browseV2']['elements'][0]['courses']['elements']

    for item in items:
        print(item['name'])

        #for key, value in item.items():
        #    print(key, value)
        #print('---')

#-----------------------------------------------------------

json_data = [{
    'operationName': 'catalogResultQuery',
    'variables': {
        'skip': False,
        'limit': 300,
        'start': '0',
        'sortField': '',
        'facets': [
            'skillNameMultiTag', 
            'jobTitleMultiTag', 
            'difficultyLevelTag', 
            'languages:English', 
            'entityTypeTag', 
            'partnerMultiTag', 
            'categoryMultiTag', 
            'subcategoryMultiTag:history'
        ]
    },
    'query': 'query catalogResultQuery($facets: [String!]!, $start: String!, $skip: Boolean = false, $sortField: String, $limit: Int) { CatalogResultsV2Resource { browseV2(facets: $facets, start: $start, limit: $limit, sortField: $sortField) @skip(if: $skip) { elements { label entries { id score courseId specializationId onDemandSpecializationId resourceName __typename } domainId subdomainId facets courses { elements { ...CourseFragment __typename } __typename } s12ns { elements { ...S12nFragment __typename } __typename } __typename } paging { total next __typename } __typename } __typename } } fragment CourseFragment on CoursesV1 { id slug name photoUrl s12nIds level workload courseDerivativesV2 { skillTags { skillName relevanceScore __typename } avgLearningHoursAdjusted commentCount averageFiveStarRating ratingCount __typename } partners { elements { name squareLogo classLogo logo __typename } __typename } __typename } fragment S12nFragment on OnDemandSpecializationsV1 { name id slug logo courseIds derivativeV2 { averageFiveStarRating avgLearningHoursAdjusted __typename } partners { elements { name squareLogo classLogo logo __typename } __typename } metadata { headerImage level __typename } courses { elements { courseDerivativesV2 { skillTags { skillName relevanceScore __typename } __typename } __typename } __typename } __typename } '
}]

url = 'https://www.coursera.org/graphqlBatch'

#headers = {'content-type': 'application/json'}
#r = requests.post(url, headers=headers, json=json_data, verify=False)

# --- it gives first 100 items ---

r = requests.post(url, json=json_data, verify=False)
data = r.json()
display(data)

# --- it gives next 28 items ---

json_data[0]['variables']['start'] = str(100) # it has to be string, not integer

r = requests.post(url, json=json_data, verify=False)
data = r.json()
display(data)

Start of result:

>>> len: 100
Buddhism and Modern Psychology 
English Composition I
Fashion as Design
The Modern World, Part One: Global History from 1760 to 1910
Indigenous Canada
Understanding Einstein: The Special Theory of Relativity
Terrorism and Counterterrorism: Comparing Theory and Practice
Magic in the Middle Ages
The Ancient Greeks
Introduction to Ancient Egypt and Its Civilization

End of result:

>>> len: 28
Theatre and Globalization
ART of the MOOC: Arte Público y Pedagogía 
The Music of the Rolling Stones, 1962-1974
Soul Beliefs: Causes and Consequences - Unit 2: Belief Systems
The Making of the US President: A Short History in Five Elections
Cities are back in town : sociologie urbaine pour un monde globalisé
Toledo: Deciphering Secrets of Medieval Spain
Russia and Nuclear Arms Control
Espace mondial, a French vision of Global studies
Religious Transformation in Early China: the Period of Division
Patrick Henry: Forgotten Founder
A la recherche du Grand Paris
Burgos: Deciphering Secrets of Medieval Spain
Journey Conversations: Weaving Knowledge and Action
Structuring Values in Modern China
Religion and Thought in Modern China: the Song, Jin, and Yuan
宇宙之旅:展现生命 (Journey of the Universe: The Unfolding of Life)
The Worldview of Thomas Berry:  The Flourishing of the Earth Community
Science and Technology in the Silla Cultural Heritage
世界空间、法国视角下的国���研究
Fundamentals of the Chinese character writing (Part 1)
Understanding China, 1700-2000: A Data Analytic Approach, Part 2
"Espace mondial" الرؤية الفرنسية للدراسات العالمية
Searching for the Grand Paris
宇宙之旅:对话 (Journey of the Universe: Weaving Knowledge and Action)
Contemporary India 
Thomas Berry的世界观:地球社区的繁荣 (The Worldview of Thomas Berry: The Flourishing of the Earth Community)
"Making" Progress Teach-Out
Ad
source: stackoverflow.com
Ad