FilesExpand file tree

 master

/

WebScraper.py

Copy path

Blame

More file actions

Blame

More file actions

Latest commit

 

History

History

History

27 lines (18 loc) · 579 Bytes

 master

/

WebScraper.py

Copy path

Top

File metadata and controls

• Code
• Blame

27 lines (18 loc) · 579 Bytes

Raw

Copy raw file

Download raw file

Open symbols panel

Edit and raw actions

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

import urllib.request

from bs4 import BeautifulSoup

class Scraper:

def __init__(self, site):

self.site = site

def scrape(self):

r = urllib.request.urlopen(self.site)

hfile = open("headlines.txt", "w+")

html = r.read()

parser = "html.parser"

sp = BeautifulSoup(html, parser)

for tag in sp.find_all("a"):

url = tag.get("href")

if url and "html" in url:

hfile.write(url + "\n")

print("\n" + url)

news = "https://news.google.com/"

Scraper(news).scrape()