Save Dynamically Loaded Webpage

This should be an easy task but I couldn't handle as I know nothing about (even very basic) web architecture.

I would like to access to links to every courses under with some filter (e.g. language=english): Coursera history courses in English.

After loading this webpage, many courses don't show before scrolling down. If I save the html file to local, I can find only 58 instances of, the prefix to a course, but I'm suppose to get at least 128 of them.

So now how do I save a dynamically loaded webpage, either with Chrome or Python?

With @Rajat's code, the emulator could scroll down to bottom, but still the obtained html is incomplete.

import os
from bs4 import BeautifulSoup
import time
from selenium import webdriver

#download chromedriver for you operating system
driver = webdriver.Chrome(current_dir+'/chromedriver')
#place your url here
count = 1200
step = 30
for _ in range(count):
    driver.execute_script("window.scrollBy(0, {});".format(step))

with open("output.html", "w") as file:



I took @Gautam code and only rebuilt it.

First request gives only 100 items (even with limit=300) so using start I get next 28 items.

Using json= instead of data= I don't need headers= and json.dump()

#!/usr/bin/env python

import requests
import warnings

def display(data):
    #print('len:', len(data))
    #print('len:', len(data[0]['data']['CatalogResultsV2Resource']['browseV2']['elements']))
    print('>>> len:', len(data[0]['data']['CatalogResultsV2Resource']['browseV2']['elements'][0]['courses']['elements']))

    items = data[0]['data']['CatalogResultsV2Resource']['browseV2']['elements'][0]['courses']['elements']

    for item in items:

        #for key, value in item.items():
        #    print(key, value)


json_data = [{
    'operationName': 'catalogResultQuery',
    'variables': {
        'skip': False,
        'limit': 300,
        'start': '0',
        'sortField': '',
        'facets': [
    'query': 'query catalogResultQuery($facets: [String!]!, $start: String!, $skip: Boolean = false, $sortField: String, $limit: Int) { CatalogResultsV2Resource { browseV2(facets: $facets, start: $start, limit: $limit, sortField: $sortField) @skip(if: $skip) { elements { label entries { id score courseId specializationId onDemandSpecializationId resourceName __typename } domainId subdomainId facets courses { elements { ...CourseFragment __typename } __typename } s12ns { elements { ...S12nFragment __typename } __typename } __typename } paging { total next __typename } __typename } __typename } } fragment CourseFragment on CoursesV1 { id slug name photoUrl s12nIds level workload courseDerivativesV2 { skillTags { skillName relevanceScore __typename } avgLearningHoursAdjusted commentCount averageFiveStarRating ratingCount __typename } partners { elements { name squareLogo classLogo logo __typename } __typename } __typename } fragment S12nFragment on OnDemandSpecializationsV1 { name id slug logo courseIds derivativeV2 { averageFiveStarRating avgLearningHoursAdjusted __typename } partners { elements { name squareLogo classLogo logo __typename } __typename } metadata { headerImage level __typename } courses { elements { courseDerivativesV2 { skillTags { skillName relevanceScore __typename } __typename } __typename } __typename } __typename } '

url = ''

#headers = {'content-type': 'application/json'}
#r =, headers=headers, json=json_data, verify=False)

# --- it gives first 100 items ---

r =, json=json_data, verify=False)
data = r.json()

# --- it gives next 28 items ---

json_data[0]['variables']['start'] = str(100) # it has to be string, not integer

r =, json=json_data, verify=False)
data = r.json()

Start of result:

>>> len: 100
Buddhism and Modern Psychology 
English Composition I
Fashion as Design
The Modern World, Part One: Global History from 1760 to 1910
Indigenous Canada
Understanding Einstein: The Special Theory of Relativity
Terrorism and Counterterrorism: Comparing Theory and Practice
Magic in the Middle Ages
The Ancient Greeks
Introduction to Ancient Egypt and Its Civilization

End of result:

>>> len: 28
Theatre and Globalization
ART of the MOOC: Arte Público y Pedagogía 
The Music of the Rolling Stones, 1962-1974
Soul Beliefs: Causes and Consequences - Unit 2: Belief Systems
The Making of the US President: A Short History in Five Elections
Cities are back in town : sociologie urbaine pour un monde globalisé
Toledo: Deciphering Secrets of Medieval Spain
Russia and Nuclear Arms Control
Espace mondial, a French vision of Global studies
Religious Transformation in Early China: the Period of Division
Patrick Henry: Forgotten Founder
A la recherche du Grand Paris
Burgos: Deciphering Secrets of Medieval Spain
Journey Conversations: Weaving Knowledge and Action
Structuring Values in Modern China
Religion and Thought in Modern China: the Song, Jin, and Yuan
宇宙之旅:展现生命 (Journey of the Universe: The Unfolding of Life)
The Worldview of Thomas Berry:  The Flourishing of the Earth Community
Science and Technology in the Silla Cultural Heritage
Fundamentals of the Chinese character writing (Part 1)
Understanding China, 1700-2000: A Data Analytic Approach, Part 2
"Espace mondial" الرؤية الفرنسية للدراسات العالمية
Searching for the Grand Paris
宇宙之旅:对话 (Journey of the Universe: Weaving Knowledge and Action)
Contemporary India 
Thomas Berry的世界观:地球社区的繁荣 (The Worldview of Thomas Berry: The Flourishing of the Earth Community)
"Making" Progress Teach-Out