Learning Tools vs Learning the Craft

This is the second in a series of essays about my experience with the largest data science and programming MOOC in the world—the IIT Madras Degree Programme in Data Science and Applications. The first essay was about learning and practicing programming, and can be read here.

Four years ago, as a part of programming diploma, I took the Machine Learning Practice course. It had two OPPEs—online proctored programming exams—worth nearly half of the course grade. We were given a list of questions, most of which were numericals, i.e. you needed to type in a number in the answer field. There was a link to a dataset, and a Google colab notebook. While the score depended almost entirely on the answer, the notebook, too, needed to be submitted as a buffer.

Online proctoring is generally harrowing for everyone. Nobody likes to be locked up in their bedrooms for three hours, with webcams on and someone watching if you’re up to mischief. In this case, we had to share our screens, take apart a dataset, train models on it, and answer questions about it.

I finished the exam in about two hours. I had time to spare, and there were a few questions that I found somewhat ill-posed. So I spent some time writing an appendix to my notebook. I carefully noted that this was indeed only an appendix, and proceeded to show, one by one, how there were bugs in the proposed boilerplate code, and assumptions in the questions that reduced the underlying machine learning problem into something absurd. It wasn’t grade-grubbing. It wasn’t even really a complaint. It was at best a bug report, the kind that a well-intentioned user shares. To give you a taste of how seemingly benign my criticism was, these are some problems I pointed out:

• The dataset had a “Country” column which needed to be one-hot encoded. The way the boilerplate code was written (which could not be changed), suggested that the train and test splits of the datasets were encoded differently. This is plain wrong: a country could easily be absent in the train set but present in the test set. One could have worked around it by just ignoring the unseen categories, but that would mean that there could be a series of features in the test dataset which were full of zeros¹.
• Another problem involved recursive feature elimination, but neither the question paper nor the boilerplate mentioned the estimator to be used². Given the nature of the ML problem at hand, I thought the reasonable default was to use a simple linear regression. I hoped it would be acceptable to the staff.
• One of the problems had this phrase in it: “Ridge estimator with fixed learning rate 0.5”. I knew that ridge regression doesn’t use gradient descent, and therefore doesn’t have a learning rate. One of the subquestions in the same problem mentioned a “regularization parameter of 0.5”. I assumed it was a typo, and carried on.
• A later problem needed to be solved by performing a grid search over a set of hyperparameters. One of them was the regularization rate itself, which had 10 logspace values. By itself, this isn’t too expensive, and it’s in fact a fairly normal practice. But by this time in the exam, the model pipeline had acquired a polynomial feature generation step of degree 3. In addition, the earlier “Country” column had been one-hot-encoded into ~200 features. When you try to extract all combinations of linear, quadratic, and cubic features from 200 variables, together they make up more than a million features! The Google colab notebook had not nearly enough RAM to hold all of them, and neither did I have enough time to wait for the brute-force grid search to finish. The problem became tractable only when I restricted the polynomials to quadratic.

As I wrote this, I started feeling mildly indignant. By this time, we’d had our share of that particularly nasty species of examiners who, instead of being content with testing students on the subject matter, thrive on trickery. We knew that the MLP examiners were a lot nicer, but we just didn’t want to take any chances. In each absurd situation I’d done the reasonable thing, and all I hoped for was a constructive dialog later. After I had turned my paper in, I shared the appendix on the discussion forum of the course.

And all hell broke loose.

Almost 50 other students came with their own rants and complaints. Even though the course team conceded after a brief debate—they weren’t tricksters—the discussions continued for days. Much later, after I had put enough distance between myself and the course, I looked at the thread once again, and I saw two distinct camps emerge from it. The first was of the opinion that it was simply an unfairly difficult exam. The other, which I led, didn’t feel too strongly about the difficulty. It wasn’t the first time we had a hard paper, and it wouldn’t be the last. What bothered me was the bad design of the exam. The first camp insisted that the next exam ought to be an open-book one. My point was that no amount of documentation or Q&A, not even an LLM could have helped with problems which were ill-posed. The first camp was quick to remind me that I already had years of experience with machine learning, and so my criticism came from a place of privilege. It took me a while to convince them that this had nothing to do with any one person’s experience or expertise. The bigger point was that even the world’s greatest ML engineer would fail this exam.

This post is about why it took so much effort to direct the discussion to the right place. Instead of being about machine learning, the entire course ended up being about pandas, scikit-learn and Google colab. Most of us found it extremely hard to punch through the veil of tools and libraries and finally see the practice of machine learning for what it truly is. There were a few other technology-specific courses too, but they were clear about what they were teaching. A course called “Programming Concepts in Java” wasn’t about Java—it was about OOP, abstractions, interfaces and concurrency. A course on system commands wasn’t about bash—it was about the Unix philosophy. Yet another course on modern web development heavily relied on Flask and VueJS. But what they ended up teaching was REST-ful APIs, MVCs and expressing UI elements as a function of the state of the system.

So I grew to believe that libraries, packages, modules and frameworks cannot and should not be taught or learnt, except via practice. There’s no theory to it. If at all there are such courses, they belong in the lab, not in the classroom. There’s no such thing as learning or teaching a library. You either know how to use it through muscle memory, or you look it up. A plumbing course would necessarily teach you how to use a wrench—but that’s clearly not the essence of the course.

I said this to whoever would listen. I screamed this from the rooftops. When I lectured students I told them to steer clear of courses that promised to teach a specific technology or a framework. In every workshop I made it a point to separate the tool from the technique.

But if I were asked if I felt the same way today, I’d hesitate. I would say that my original position was a bit extreme and perhaps even somewhat simplistic. After all, there’s a lot of merit in studying libraries and tools not for what they can do, but how they do it. There are few things more instructive to a programmer than to study the work of others. I would still do my damnedest to draw a distinction between a) studying a subject, b) knowing how to use the necessary tools and c) studying the tools. But once that line is drawn, I’d encourage you to exercise all three options.

What you’re reading is an attempt to reconcile my original position with my current one, and perhaps to illuminate some milestones along the way.

My original position—that tools and libraries cannot and should not be taught—stems from the fact that they’re ephemeral. What’s relevant today could be obsolete tomorrow, rendering all the ’learning’ useless. To save oneself from this, one needs to go a level above the tools. On the other hand, even if a course explicitly promised to teach you all about sklearn, it would be impossible to do so.

I’m sure that nobody in the MLP course intended it to become an sklearn course. Everyone knows that there’s a lot more to practicing machine learning than knowing how to use sklearn well. (Even LLMs can spit out reasonable sklearn code. That doesn’t make them good ML practitioners. A carpenter would not get distracted by how to use a saw or a hammer. They’d focus on making good furniture.) But this was not made explicitly clear to students. Assessments, quizzes and exams were almost entirely about reading and writing sklearn code. Novices are particularly susceptible to confusing the tools for the craft itself. And yet, even though so much of the course was about sklearn, tragically, the design philosophy of sklearn was never taught.

Think of the typical flow that is used to train ML models. Here’s an illustration: we begin with two different entities—an “empty” model (which has an algorithm attached to it) and the data. The algorithm consumes the data and mutates the model. We still have two entities—but they’re not the same ones we began with. The model now has a “state” which represents its learning. The model, which earlier only allowed itself to be trained, now supports one more function—it can predict things, it can operate on more and more data.

What kind of a programming method would capture this flow well? We need something that can put data and an algorithm together to create a stateful object, while still keeping the data isolated (some latent abstraction of the data must, naturally, be captured in the model if it is to learn—these abstractions are precisely the weights or parameters of the models). Thus, we see that object-oriented programming is an intuitive and natural paradigm for implementing machine learning algorithms. As such, sklearn is highly object-oriented.

Unfortunately, students do not have a good handle on object-oriented Python by the time they reach the MLP course. They do know what classes, objects and methods are—but this isn’t reinforced well enough. As a result, most of the discussion forum posts we saw in the MLP category stemmed from wrong methods being called on wrong objects.

The good news is that sklearn’s API is cleanly organized. Their user guide has clearly defined sections on supervised and unsupervised learning, cross-validation, data transformations, preprocessing and evaluation metrics. Each of these represents a crucial component in the machine learning pipeline. Data has to first be loaded from disk into memory, then preprocessed and transformed so it is ready to be consumed by a model. A model is then selected based on the task at hand, trained, and finally evaluated. Any one of a wide variety of data loaders, preprocessors and transformers, models, CV strategies and evaluation metrics can fit into this pipeline. The interesting thing is that sklearn’s API is so uniform that you can replace one preprocessor with another, by changing just a few characters in your code.

This, in a nutshell, is what I mean by the design philosophy of a library. Looking back to the MLP course, I feel like students would have had a much better time had they known that such a philosophy even existed. Similar, opinionated design is found in many other places. In Unix, it manifests as “doing one thing and doing it well”. NumPy, by making a contiguous and homogenous array the fundamental data structure, forces you to think in terms of parallelizable and vectorized operations. So, ephemeral as tools and libraries may be, their study can be enlightening. The dissonance happens when the study of the tool is confused with the study of the craft.

Interestingly, the most instructive piece I’ve read about separating tools from craft comes not from programming or even STEM, but from literature. Stephen King, in his memoir “On Writing”, writes about how every writer should build their own toolbox. And like a carpenter’s toolbox, the writer’s toolbox, too, must have layers. Paraphrasing roughly, the top layers should be filled with common tools like vocabulary and grammar—words, and how they go together. The deeper layers then contain the intangibles: style, taste and your voice—tools that get sharper with practice. If I were to draw even the most tenuous analogy to a programmer’s toolbox, the effect is surprisingly fitting. A good programmer would fill the top layers with programming languages, their idioms and their affordances: the “vocabulary” and the “grammar”. The lower layers would then contain the design of solutions, opinions on structure, and ultimately the knack for writing good code.

Now, this is simply an analogy. There are many others. The broader point is that if we are (perhaps forced) to obsess over tools; we’re better off doing so with the requisite seriousness, and not simply as a means to an end.

In some of the more memorable discussion threads on various course forums, my fellow students usually asked how to think in terms of a library. One such discussion was about thinking in NumPy, where the OP began by admitting that they felt intimidated when they saw instructors and other students wield numpy arrays and operations with the flourish of a master. Now, this person wasn’t a novice programmer, and was actually pretty good at Python itself. Even NumPy wasn’t an insurmountable problem. The problem was constantly having to remember or look up the right method or function.

It was clear to me that they were “intimidated” not by the skill of experts, but simply by their muscle memory. I told them that nobody “knows” NumPy—there’s nothing to know, and that I was sure that longtime users and even maintainers of NumPy end up looking things up all the time. While a good part of that practice does accumulate at our fingertips, that is mostly a function of time and exposure. In fact I would go so far as to recommend that nobody should be worried about remembering function signatures at all. No part of a programmer’s mind should be a substitute for even the smallest part of documentation. But coming back specifically to NumPy—I think that once you get arrays, vectorized operations, broadcasting and ufuncs, there remain no fundamental differences between the Pythonic way of thinking and the array-oriented way of thinking (or coding). At that point, Python is NumPy is Python.

Strangely, this proposition did nothing to quell the anxiety of the OP. They maintained that NumPy’s way of solving problems was very different from Python’s. This, I thought, was attributable only to the additional abstraction introduced by NumPy, which hides some details. The trouble is that to novices, such abstraction looks like magic. And sure enough, when another post on the same thread was written like a NumPy cheatsheet, the thread was marked by the OP as “solved”. To be fair, the OP did say that they were currently bothered by looming exams.

In short, when grades are on the line, few are likely to bother with design philosophy, and would rather just get on with the usage.

In a stark, idealistic contrast to this situation, Richard Dawkins writes in an essay on tutorial-driven teaching³,

Each week my tutorial assignment was to read one DPhil thesis. My essay was to be a combination of DPhil examiner’s report, proposal for follow-up research, review of the history of the subject in which the thesis fell, and theoretical and philosophical discussion of the issues that the thesis raised. Never for one moment did it occur to either of us (the tutor and the pupil) to wonder whether this assignment would be directly useful to me in answering some exam question… The whole field of zoology was fair game for the examiners and the only thing we could rely upon was a presumption that our question papers would not be too unfairly different from the recent predecessors. The examiners when setting the papers, and our tutors when handing out essay topics, neither knew nor cared which subjects had been covered in the lectures.

And if all of this sounds very idealistic, that’s because it is. Dawkins was writing in the defence of the tutorial-driven method as against the lecture-driven method. The latter consists primarily of a lecturer⁴ talking to a roomful of students, while the former has a tutor, who is not necessarily an authority on the subject, tutoring only one or a few students⁵.

Dawkins writes further that the purpose of a lecture should be not to inform, but to inspire. And that’s precisely what I was doing when I told my friends that “Python is NumPy is Python.” It didn’t land, not because it was wrong, and not for lack of someone to walk them through it individually. The “cheatsheet” post did its job, with something that could be memorized rather than understood. Unfortunately, a good examinee has less time than a good student.

This duality between a good examinee and a good student, which is itself a polymorphism of the duality between learning a tool and learning a craft, is a recurring theme across many discussion threads. A year after my OPPE, a friend shared a Kaggle notebook from his capstone project, and asked for feedback. There was no pressure this time—the project had been submitted and the exam had finished. In the notebook I saw plenty of examples of badly chosen chart types, an excess of estimators and spurious grid search methods⁶. It seemed that the notebook was written solely from the perspective of demonstrating how much pandas, sklearn and matplotlib someone knows regardless of whether it helps the ML problem itself! At one point, the discussion turned to how the toppers of the Kaggle competition had obtained an edge by using a stacking classifier, and the OP did not consider it because “it was never taught.” This made me lose my mind, in a manner of speaking. I went on a rant that allowed me to crystallize my thesis, which I’d tacitly held since my own OPPE, about how distracting it can be to focus too much on tools.

Around the same time, in a different thread, another student made almost the inverse complaint: that the “theory” component of these courses tested exactly the wrong thing—memorizing syntax and method signatures with no internet access—while the “practical” component, ironically, allowed you to look things up. My answer there was the same answer, restated:

1. the brain isn’t built to hold function signatures,
2. practice is the only real teacher,
3. your mileage may vary

In this particular thread I admitted upfront that my advice was too idealistic to be practical for everyone. But then, if you want to ace exams, and develop expertise with tools and libraries and frameworks, and be competent in the long term… that’s pretty idealistic in the first place.

So, if becoming a well-rounded AI/ML engineer is the goal, none of the above is too idealistic.