PC Lab 8 — Measuring Startup Innovation

The setting

Who is the next Facebook? You’ve been hired as a data scientist by Sequoia Capital — the fund behind WhatsApp, Zoom, and Airbnb. Venture capital is a needle-in-the-haystack business, and your haystack is the patent record.

“Kelly, Papanikolaou, Seru and Taddy measured technological innovation over 180 years from patent text. I don’t need 180 years — I need to know which of today’s patents are breakthroughs before the market does. Build me the measure.”

Your goals: clean and explore patent abstracts, classify them with unsupervised topic modeling, and implement a measure of textual similarity that flags the outliers — the potential breakthroughs. The academic anchors are Kelly et al. (2021) and Bonelli (2022).

Before you start — read this

This lab is entirely optional and not graded. By this point you already have your grades from Labs 1–4 and the Hackathon. This one is for fun — do all of it, part of it, or none. In any case, try to learn something new.

The data

PatentAbstract_PCLab8.csv — patent abstracts. A fresh download link will be distributed on BBoard when this lab opens (the link printed in older slide decks has expired). If the data is too large for your machine, work on a sub-sample — the slides explicitly allow it. Reference reading: Bonelli (2023), Data-driven Investors, linked in the assignment slides.

Your assignment

Part 1 — Know your sample (Task 1)

Import the abstracts; run the usual text cleaning (punctuation, stopwords, short words).
Describe the sample: how many patents? Unique words? Letter frequencies (Lewand’s law), average words per abstract, most common words.

Part 2 — Invent the classification (Task 2)

The data has no dates and no technological classes. Create the classes yourself: run LDA topic modeling (gensim) on the abstracts and assign each patent to a class.
How many classes do you end up with? What are the most frequent words in each — and does the classification make sense?

Part 3 — Find the breakthroughs (Task 3)

Build a measure of semantic similarity between patents (à la Kelly et al. 2021, or your own).
Alternatively: project the patents into a 2–3 dimensional vector space and tag the outliers.
Identify, within each technological class, the outliers — the candidate breakthrough patents — and illustrate your findings graphically.
Step back: is this a good way to measure breakthrough innovation? What would you do, ideally?

Deliverables (if you choose to submit)

Same format as always: one Jupyter notebook per group, email title PCLab#8 - Group X - Name1 Name2 Name3, to the instructor (clement.mazetsonilhac@unibocconi.it) and the TA (andrea.andolfatto@phd.unibocconi.it).
No deadline pressure, no grade — just feedback if you want it.