Hello. Can I hire you to do some research for me? I need you to look up this hungry bird on Wikipedia and find what it eats and copy and paste that passage into my spreadsheet. Then I need you to repeat that for each of the 264 birds in my spreadsheet. How much would you charge me to do this research?
Did you say $50 per hour? I’m sorry, I can’t afford that. Do you have anything in the free range?
Oh! I have a free worker. We all do. It’s Python. Let’s go get it. No, it’s not outside. At least not yet. The Burmese pythons might make their way up to Ponte Vedra someday. The ball pythons are people’s house pets, unless somebody puts them outside again like they did in St. Augustine a while back. No, we’re not looking for a snake. We’re looking for a web researcher’s best friend, Python.
In fact, chatGPT is mostly written in Python. This is because chatGPT’s parent, openAI, loves Python. Python is the friendliest programming language there is, and it’s TOTALLY FREE. So let’s give it a whirl. Will Python look up 264 birds for me on Wikipedia and copy what they eat into my spreadsheet?
First, we must install the Python language so our computer can read it. Go to Python.org and follow the instructions. Don’t forget to check the box that says ‘Add Python to PATH’. I don’t know why that’s an option because everybody needs it.
Now that we have Python installed, here’s a quick little program that will tell Python to go find bird food 264 times. You can try it out by saving an Excel list of birds as ‘bird_names_for_diet_scraping.csv’. Type or extract this text (Lines 2-39) into Notepad and save it with a .py extension in the same folder as your bird list.
You can generally run Python files (file.py) just by double-clicking them. However, I like to see it in living color in VS Code, where there’s an actual Run button. Thank you, Microsoft, for the also TOTALLY FREE Visual Studio Code.
Let’s summarize this powerful little program:
Lines 2-5 show Python importing a few modules that help with the task.
Line 8 shows Python using Pandas (abbreviated as pd) to read my bird list.
Lines 10-12 define (def) the procedure to look for a Wikipedia page that includes my bird name.
Lines 13-25 show Python using the tool BeautifulSoup to find paragraph tags <p> that include the words “diet”, “feeds on”, or “eats”.
Line 31 shows Python calling (implementing) the procedure defined at Line 10.
Lines 28-33 tell Python to repeatedly collect food paragraphs for as long as there are names in my list.
Lines 35-38 tell Python to write a new CSV file containing the data it collects in the above procedure.
Line 39 shows Python displaying a message on my computer screen to tell me it’s done with the results file. Let’s take a peek:
Great work, Python!
You don’t have to know how to build that little program. A couple of years ago, you could Google and find somebody who shared something close enough to tweak to your purposes. Nowadays, ChatGPT will build you a custom script based on your requests. It might take an hour or so to get it right, but how long would it have taken me to look up 264 birds?
Well, let’s address the elephant in those results – the gaps. Unlike A.I., Python does not fake intelligence or fudge the rules. Python respected my specific bird names and dietary key words. For example, there is a Wikipedia page for Sharp-shinned Hawk with diet details, but they are all phrased as “prey”. The Brown Booby also got no food; its Wikipedia page is phrased in the plural, so “eats” is only found as “eat”. With my specific parameters, I only landed 168 matches. I still have 96 birds to look up. I’m still not ready for that much manual labor. I’ll tweak my Python script.
Now you have officially met the world’s most popular programming language. And you fed a lot of birds in the process. Don’t you love it?