Lucid Mesh

Who hasn't wished for a library that neatly solves the programming problem that is currently staring you in the face? When you're down in the dirt, gearing up to tackle some gnarly problem that seems like it should have a general solution, it's only natural to wish, or even expect there to be a library that addresses your need. Once you have a workable solution that you think is elegant and useful, don't you think about releasing it for the world, or at least your company, to fulfill other programmer's wishes for self-contained solutions to their problems? Be careful what you wish for.

This post is probably not going to be about what you're thinking right now. I'm not going to relate some harrowing tale about a software library I picked up off the street, hoping it was going to solve all of my problems, only to find that it was more trouble than it was worth. Honestly, I haven't had many experiences like that. Either I've had amazing luck finding exactly what I need, or I have low expectations and work through the problems with the libraries I find as they come up. No, this post is about the way the software library marketplace has grown exponentially over the years and how one type of complaint about libraries has morphed into another, seemingly without notice.

When I was first learning to program, the languages I learned had small standard libraries. C has a pretty manageable standard library. C++ also has a small standard library by today's standards, even though it includes most of the C standard library as well. You could also include the standard template library, still keep all of the C++ library functionality in your head, and recall it at will to solve your programming problems. Early Java also had a tractable standard library.

Admittedly, my early view of the world of programming languages was somewhat limited, but the world was definitely smaller back then. One of the most common complaints I heard was that the world was a little too small. We needed more good libraries and frameworks for building applications—prepackaged plug-and-play code that would solve all of our problems and make development easy and painless. Sure, there were some isolated examples of application development frameworks out there, like MFC for Windows development in C++. But we wanted more, much more, and we weren't afraid to ask for it.

Now I'm not in any way knocking the fact that we wished for all of these libraries and frameworks, or that we expected them to solve all of our problems. It was and is entirely reasonable, even virtuous, to not want to reinvent the wheel over and over again. We should package up solutions to solved problems and make them part of the foundation that we then build on top of. I wouldn't have expected or wanted it to happen any other way, but I don't think many programmers realized that it would happen the way it did.

Today Java 8 has over 4,200 classes in the standard library public API. The .NET 3.5 framework has nearly 10,000 public classes—the latest version I could find numbers for. The .NET 4.5 framework has well over that number, I'm sure. No single person could possibly know everything that's in the standard libraries of these languages, and that's not even counting all of the contributions available from open source libraries.

It's a bit harder to get stats on open source projects. Ruby, a language with a fairly tractable standard library, has 6,272 gems available on RubyGems.org. Take into account that gems are generally a larger unit of measure than classes and the gems available at RubyGems.org are only a subset of all available gems, and it is clear that there are a lot of open source libraries available for Ruby. The same goes for all of the currently popular languages, especially JavaScript. Because of the nature of JavaScript—ease of releasing projects and use on both the client and server sides of the browser—the number of libraries available is becoming truly astounding. Looking only at GitHub, there are over 320,000 active JavaScript repositories! Of course some (possibly many) of these repositories are mis-categorized, and many more of them aren't actually reusable libraries, but other varieties of projects. Even so, any way you slice that number, it's big, and growing fast.

The point here is not to get an accurate measure of the number of libraries available for different languages, but to show that in any language you use, that number is absolutely massive. The game has fundamentally changed from knowing everything that's available and how to use most of it to searching for what you need when you need it and evaluating whether or not you can use it (90% of it is crap, as per Sturgeon's Law). This shift makes some people very uncomfortable.

You see, back when we were complaining about the lack of good libraries and frameworks for building awesome applications and solving all of the mundane problems of software development, this is not what we wanted. We wanted elegant, high-performance tools that perfectly fit the problems we were trying to solve, not a mountain of crap to sift through for a few choice pearls. I posit that the former was never going to happen, and the latter is the best we could have hoped for.

Part of the reason is purely because that's the nature of software development and the web. When anyone can post a library on the web for others to use, you're going to end up with a ton of stuff that nobody needs or is of decidedly low quality. To get the fraction of really good stuff, we need to put up with a lot of mediocre, useless stuff.

The other part of the reason is that perfect software libraries are one of Steve Yegge's Nonesuch Beasts. They don't exist. You can't build one. It's impossible, and here's why. Any library that's generally applicable to a wide range of projects necessarily needs to solve problems in a general way. On the other side of the coin, you likely require a specific solution to a narrow problem, and a large collection of projects like yours will have a wide range of narrow problems that are slightly different.

What we end up with is a large problem set (the projects) on one side and a large solution set (the libraries) on the other side with a lot of thin connections running between them. One project will use a part of one library in a particular way, and another project will use a the same library in a different way or a different library altogether. There's an enormous mismatch, and you can't have both a general solution for many problems and a perfect solution for each problem. Abstractions are leaky. Although we can keep making things better, and I think we have much better tools than we did a decade ago, that mismatch will always exist. Ruby on Rails is one example among many excellent web development frameworks. Python and R have some great data analysis tools. JavaScript continues to impress me with the libraries available for adding all kinds of features to your web application. The solutions are there if we know where to look and how to evaluate them.

When programmers complain about how hard it is to find the right library for their needs and how everything out there is crap, I think they're actually complaining about needing to think. What they want is to be able to go on Google and type in, "I need a JavaScript library for x," be taken to a GitHub page with an easy-to-install library, and be able to use the library intuitively. No reading documentation, comparing options, or analyzing performance required. Sometimes we can get pretty close to this experience. Some libraries are very good and the choice is clear, but most of the time software development is still hard. The choices are vague and complicated. Nothing may be the right fit, and we need to develop our own solution. Programming still requires plenty of thought.

I count that as a good thing. If programming was simple, and all we had to do was plug together a bunch of libraries we found on GitHub, it would get unbearably boring. Besides, if we truly solved the software library problem, it would mean we were no longer innovating and pushing the frontier of software. Researching options and piecing them together to build novel solutions, basically creative problem solving, is an inherent part of our job. It is fundamental to what we do. Without that difficulty, most of us would probably be out of a job. We wished for more and better libraries. Now we're getting them, and we're wishing that we didn't have to think so hard about when and how to use them. We should be careful what we wish for. What if we actually get it?