Thoughts on data science, statistics and machine learning.
Purvagraha
There’s a fascinating Sanskrit word — पूर्वाग्रह — which translates to prejudice or preconception. But the translation misses something: the suffix आग्रह does not mean either judgment (-judice) or conception. It means insistence, or appeal. And when this insistence crosses over a moral boundary, it becomes दुराग्रह — a stubborn, obstinate insistence on something. This word is closer to हठ (or हट्ट in Marathi). Unfortunately, I cannot find a suitable English word or phrase that accurately conveys the meaning of either दुराग्रह or हट्ट.
Learning Tools vs Learning the Craft
This is the second in a series of essays about my experience with the largest data science and programming MOOC in the world—the IIT Madras Degree Programme in Data Science and Applications. The first essay was about learning and practicing programming, and can be read here.
Four years ago, as a part of the data science diploma, I took the Machine Learning Practice course. It had two OPPEs—online proctored programming exams—worth nearly half of the course grade. We were given a list of questions, most of which were numericals, i.e. you needed to type in a number in the answer field. There was a link to a dataset, and a Google colab notebook. While the score depended almost entirely on the answer, the notebook, too, needed to be submitted as a buffer.
Notes on Fred Brooks's 'No Silver Bullet'
I had a simple problem. An OCR engine found text anywhere on a PDF page. A separate parser could tell me whether a rectangle on that page contained text, an image, both, or neither. Given a piece of OCR output, I needed to decide whether it came from a text layer or from pixels that merely looked like text. When neither answer was reliable, the system would say “unknown”.
The problem was easy to state, easy to understand and easy to test. So why did it take me hours?
Finding Nash Equilibria in Jeopardy!
Spoiler Alert: It’s unusual for post about a game show like Jeopardy! to have a spoiler, but if you are a die hard fan of the show and haven’t seen the second semifinal game of season 2 of Pop Culture Jeopardy! (it’s on Netflix), then don’t read this just yet.
(Here on I’m just going to write Jeopardy without the exclamation mark, for convenience.)
The last round in a game of Jeopardy is called “Final Jeopardy”. When the round begins, the three contestants (individually, usually, but in the case of Pop Culture Jeopardy, teams of two) are shown the category of the final clue.