I'm young in my career, but I often say that I'm an open source developer.
I’ve always associated the term with a beautiful idea, and one that I aspired to manifest in my own life. The idea is that a group of disparate individuals, some with altrustic intention to make the world a better place, some socially awkward with a need to connect, and some with an obsessive compulsion to program and build, could come together and make something powerful. The idea of open source software (OSS) has become common in our “reproducible open practices “ jargon, with large contributions from companies following a business model, and well established projects still playing primary roles in the average user’s daily compute routine. And it works. Such powerful forces that dominate early open source software would have been challenging to near impossible to reproduce any other way.
It doesn’t always work. Open source success goes far beyond having talented developers, it comes down to culture, how the software is cared for, and how developers and users are treated. These ideas are nicely summarized as a few of the points from The Cathedral and the Bazaar:
- Treating your users as co-developers is your least-hassle route to rapid code improvement and effective debugging.
- Release early, release often, and listen to your customers.
- With enough eyes, all bugs are small
- beta testers treated as if they're the most valuable resource will respond by becoming that.
- The next best thing to having good ideas is recognizing good ideas from users.
Whether the person is a developer or a user (or both) the message for the above points is clear. Software grows powerful and robust if contribution (for users and developers and combined) is a positive experience over negative. I’ve had experiences in both fronts, and I’ve realized that there are different flavors and lifecycles of open source projects. In this post I want to share some of my experiences, and what I’ve learned.
Academic Open Source
My early experiences with open source were firmly rooted in academic software development. At first I was programming on my own, and then I discovered that development was much more fun when my lab (and remote affiliates) were part of my team. I never was a core developer for any specific project, but felt like an important contributor on many fronts, and it was a fulfilling experience. Expectations were low (if any existed at all) and this kind of environment is optimal for learning and taking risks. How does this contribute to science? Many small projects that started in academia have grown to have robust, international communities and played a significant role. While it’s not always perfect, largely the incentives are honorable, because academics tend to be obsessed about their trade, and wouldn’t take such a position if the desire for monetary gain outweighed that. The biggest challenges generally might come from having a lack of funding to maintain some required resource, or having a small project that didn’t pick up steam and is not able to be maintained given the demands of time for a small team. Reproducibility, while important for declaration of scientific results, also poses challenges for software development. You can’t always conceive software in entirety and then start coding. I’ve seen entire projects and fellowships make declarations about doing something, get funded, and then fail miserably. In one such case I was troubled enough by the lack of initiative that I implemented my own solution in a few weekends of work to help the users that were badly looking for the software.
Academic open source software is usually a byproduct of a need for a tool, whether anticipated or not, that survives natural selection because it is useful, has a core team that maintains positive spirit for the project, and encourage others to take part.
These communities, overall, have my deepest respect, and are in a word, awesome.
We can also talk about projects that are started by academics that are not successful. I once noticed three smaller groups working on similar software toward the same goal (a common file format for a particular domain), and reached out to them and was successful to bring everyone together under a common organization. What I try to bring to the table in my projects is spirit, energy, and a focus on having fun while doing quality work (I’m not altrustic, I just want to have fun). We started working on code together, and in fact the first few weeks, from a productivity standpoint, were fantastic. The issue board was whittled down, and a new release in sight. But it quickly fell apart because a good half of the team of developers were mean. I tried to have tolerance for it at first. Nobody is perfect, and there can always be rough spots or interactions. The problem was the fact that pull requests (PR) weren’t fun, collaborative efforts toward doing the best job possible. They were nit picking, insult sessions that left the individual making the PR feeling insufficient and disliked. I would try to save conversations with adding humor, and these efforts seemed to only indicate having weakness, and fueling the fire more.
A serious culture does not indicate a professional or optimal one.
And in fact it can be a warning sign. We can easily figure this out by looking at the kind of positive emotion that you feel after an event like a pull request is merged. If the small burst of positive emotion is more relief than joy from a team effort, this is an important distinction. Relief signals a stressful experience ending, while joy indicates a positive one fulfilled. This happened to me several times with this community, and after a few rounds of this, I made the right decision to drift away from the project. As you would expect, this is a hard hit for any open source developer. We invest tiny pieces of ourselves into projects and want to see a successful outcome from it.
Modern Open Source
We can call any project that is not rooted in an academic community and wasn’t started with the initial movement something I’ll call modern open source. These projects are recent, are are given the name “open source” per meeting the bullet list of requirements for it. I wonder if some have not noticed the subtle transition between these two times, and it is with the transition that my heart is troubled.
I’ve felt depressed for some time because of a realization. There are two issues, and they represent opposite directions on the same dimension. Both involve people and the incentives that drive them. The first is proprietary software that announces going “open source” with some misleading undertones. If you look at some of the recently created leaders they might fit nicely within the open source definition, but something is off in the incentive structure. The second is the realization that open source projects can too fall into the transition to becoming a company. While this is a logical decision for better support of the core team, the introduction of money can drastically change one’s incentive structure, and importantly, the culture of the community. Realizing that, for most of the time, the current life cycle of open source projects usually comes down to one of these two things makes me question my identity, and the goodness and incentives of others.
Type 1: Business Open Source
The companies and associated projects that have become well known in the last few years are well known, and I don’t need to explicitly state. I don’t mean that I don’t respect these communities, but that there is a distinction that needs to be made. If I imagine myself to be a developer at a company, I would be overtaken with the open source spirit. I would feel like I belonged with a team, was having fun, and doing it toward a larger goal. What I would not think much about is why the software is open source, and if it had ulterior incentives. This comes down to a problem that goes unnoticed, and an observation that I take issue with:
open source culture is now taken advantage of for the benefit of corporations
Companies aren’t opening up the previously locked doors because they are solely selfless and good. They realized that the old model of developing software privately is completely washed away by an open source model, and proceeded to jump on the bandwagon of announcing open source projects. I would even guess that if the cultures could be properly maintained, code bases made “good enough” to be shared, and organized to be well-managed, most code that is still private would do well under this model. Likely this isn’t needed for companies that are large enough to have their own “internal” open source community. When companies first started this initiate a few years ago, it made my heart sing. I was obsessively focused on the idea that academia and companies should work together, and this was what I wanted to see. I was overjoyed that large corporations could have interest in supporting software that started as tiny academic projects, or open sourcing a previously private software package to be free to benefit anyone with interest. I believed that companies were reaching out to the community and had values that were consistent with the open source community.
Several years later it makes sense to me. In the first situation, a company was deploying a strategy to use a well-liked and widely used package as public relations bait to attract more users, and ultimately offer them resources and services. In the second case, the company would build up an entire movement and community of open source developers to do the hard work of adding features and maintaining the project for them. It’s a growing model for companies to offer paid-for services instead of paid-for software. Anyone and everyone has a computer to program and an internet connection to collaboratively develop, it’s the scaled deployments, or need for hardware, or orchestration of services, that we cannot do so easily. This model would actually be ok, I think, if there was a good distinction between the software itself and its needs for a non-trivial deployment. Unfortunately, for most non-trivial use cases there is a subtle coercion into paying for resources provided by a company. So what does modern open source really mean?
modern open source is a trendy practice of companies stamping their products with an open source label to use the community to do free work to directly or indirectly support the company (monetary) goals
Modern open source is like a “healthy choice” stamp that is placed on a cereal box that tricks you to feel good about something that is just superficial, and isn’t valid. True open source would be buying a gadget, and being empowered to build and use it fully. Modern open source is buying that same gadget, but then finding that it doesn’t come with some special battery (required!) that you must buy to use the gadget. Sure, you could also follow instructions to make your own battery, but the economy and instruction manual doesn’t guide you to do that. Modern open source isn’t how it used to be, but has become a part of company branding.
Open Source can be Misleading
The problem with using open source as a kind of branding is that it’s misleading. If you look at some of these big project sites, the terms “open source” are front and center. Seeing these words brings good feelings to me, and most definitely likely those employed by the company. If I were employed at a company and feeling less than great about the incentives of the profit mongering entity above me, I’d feel a little bit better. If you take a look at some of the repositores on Github for these projects, they now have thousands of developers. The software is very happy, and very likely the community as well. But why is the work being done? Logically, everyone is adding features and integrations that make the software more usable and attractive to the larger community. If it stopped there, my morality would not feel squirmy. The problem is that this same development is directly useful to the company to derive profit from services that use it. I wonder how many think about this subtle detail? Or choose to not think about it?
…but it doesn’t have to be
To be clear, I don’t mean to say that some flavor of community development and collaboration with a company is not a good idea. It simply cannot be misleading, and there should be some distinction between the software and services the company offers, and the software should be able to flourish and broadly be used without any kind of additional service. Companies should go out of their way to empower academic and non-profit groups to have resources they otherwise wouldn’t, and offer the paid-for services to other business entities with a clear distinction from the open source software itself. Importantly, as a user I need to see a clear distinction between what is paid for and not, and I must not see a company taking advantage of open source culture for their own profit.
Collaboration can also be misleading
This abuse of an established culture extends far beyond software and to collaborations. I can look at academic institutions I’ve been a part of or collaborated with, and seen several projects with companies. For the company it looks really great because they can publicly support science. They might also have an agreement to share data that would otherwise belong to the academic institution. After observing this dynamic for a few years, it seems that the companies tend to use the academic groups as grounds for development (both in terms of ideas and data) and then turn the outcome into a product or service. If you’ve attended a meeting of this kind, I’ve noticed that most aren’t focused around development, but rather sales teams to first give flashy presentations on resources, and then negotiate sharing of data and other resources. They then usually finish and make a distinct point about the larger goal to help mankind, or something like that. It has felt inspiring a few times, but usually it leaves me feeling squirmy. Maybe this is just the real world and I don’t like it. People are outwardly nice and tell you what you want to hear, so long as you drink their koolaid, and put on blinders to what is underlying in the recipe.
Thus this branding of “open source” and “collaboration” I find misleading. Any initiative by a company that provides a tool or service that (realistically) is hugely dependent on a paid resource is not fitting to the spirit of these ideas. While some questions and services might be more optimally provided or answered by a company (or a research team at one) this should not be wrapped into something called collaborative open source. It’s a business agreement. The two entities are often in competition to find an agreement to share data and resources, the company in pursuit of profit and the academic group in pursuit of publication. Many times the work horses (graduate students) get scooped up by the company, so does the company ultimately win? A lot of these companies came to be where they are as a result of the talent in the area, and have set up shop there because they can continue to harvest the greatest minds from these same institutions. That’s the problem. You can only love the idea of academia and corporations working together and collaborating. But one side of the partnership scooping talent, ideas, and data in a repeating cycle is not collaboration but some other kind of weird, symbiotic relationship.
Type 2: Rocket Ship Growth
Now let’s imagine what happens when an open source project takes off like a rocket ship. There are two situations I can imagine. The first is that, akin to how the LInux kernel progressed, the community can support that growth, and the software continues to develop successfully because the positive culture is maintained and the community inspired. This makes my little heart very happy, because you have a useful tool being developed with a strong community and (hopefully) the spirit is collaborative and fun. The second situation that comes with rocket ship growth is the potential for greed and competition, and these outcomes can play with the incentive structure of any rational human, sometimes unbeknownst to him. With rocket ship growth there is risk involved, almost always money, and incentives are bundled around those things. We can see any or all of the following, all with various degrees:
- A previous culture to do quality, sound work can be disrupted to meet company deadlines
- The overall level of stress goes up, and this likely makes the project less fun for maintainers, developers, and users
- Stress and lack of time put a focus on the company, and not helping users. The open source model suffers
- A creation and then special treatment for the "paid for version" isolates the community
- Company creation brings up questions of ownership and loyalty, creating artificial divides and questions about membership that didn't previously exist
From the above, both the community and software suffer. It’s not to say that a community that might go through this cannot stabilize and recover over time, but it seems that this set of features could drive away both users and developers, and hurt the software in the short (and possibly long) run. For the open source developer, an experience may start as inspiring and fun, and feeling valued and taking a large role in the early software development. Promises are made to you, and you take them full heartedly. And then after the rocket ship takes off and you are not part of the team, you are left on the ground, confused, and heart broken.
The Sadness of the Open Source Developer
At this point, I can start to convey why I, as a previous academic and current open source developer, feel heart broken. The culture and practices that I have built my current identity and early career around, when understood from the business model, contradict some of the core values that I previously associated with open source development. Arguably, I could put blinders on to these corporate open source projects, and focus on the smaller ones. But as a single node in a sea of Github pull requests, I am in queue for the exact phenomena that I (and I would guess many introverted, quirky young adults) experienced the entirety of growing up. You search for meaning and to be needed by way of giving your time and energy, but ultimately you are mostly on your own. Projects that start small and take off in a beautiful open source fashion are likely to get tangled with changing incentives. You are likely to use your time and energy to make them strong, and then they will take off like a rocketship to leave you behind. Or maybe you invest equally and the world has already moved on to the next shiny coin. This is the sadness of the open source developer.
The open source developer believes that enthusiasm for a project leads to fulfillent, but it more often leads to feeling alone.
Ultimately, isolation is the underlying unspoken outcome. You are valued until you aren’t. You are a part of the team until you aren’t. It isn’t always fair or even clear, but it just is. I don’t even mean to say there is malicious nature on the part of some other entity or person, and the open source engineer is somehow victimized. It’s a loss of one of the core values behind open source: showing urgency, value, and appreciation for the developers and users. It’s the fact that at some point, one group doesn’t understand the other, or doesn’t share the same incentive. It is for this reason that I go back to the same rule that I applied toward group projects in grade school, high school, and even college. It’s always best to do things alone. You can’t be left out that way.
Even immersed in my lovely data structures and programming languages, I question the people and components that are also invested in the work. It’s disappointing to work days and weeks on something, and have others not even remember that you exist let alone already solved a problem. I’ve largely brushed off these experiences and just moved on, but they add up quickly. It’s saddening to invest years in a project only to feel left out in the end. What is left lingering in my mind’s eye is a jaded, negative bias about the true incentives of other people. Corporate tech seems to come down to middle-aged business men wearing leather jackets giving over-hyped presentations about just another nice looking website and matching service. Non-corporate tech seems to aspire for that.
We can step back and ask harder questions. Are advancements in the technologies provided by these giants making the world a better place, or making people healthier or happier? Is the recipe for happiness an algorithm to predict intestinal polyops from a bunch of CT scans, or maybe just getting better sleep and spending an afternoon caring for your plants, or taking a walk in the sunshine? Millions of dollars of funds might be spent for some flashy new technology to ultimately make an algorithm run 2% faster, but this will never hold a candle to the improvements in quality of life from, I don’t know, playing more with a pet cat? Going for a run and seeing the sun set? Working hard to plant a garden and feeling satisfied at the end of the day eating a well-earned avocado? We think we are in some great technological revolution, and that life and the world is increasingly better than it was before. The reality is that we haven’t made much progress in real (earth shattering) discoveries, and every business minded person is sticking their foot into the churning butter ball trying to market some marginally different solution for trivial improvement over the current. And ironically, in doing so they lose touch with basic human needs like taking care of themself.
What does the future hold?
Sadness does not come with answers, but it comes with an overflow of questions, rumination about the world, other’s incentives, and personal identity. These questions often lead to insight and solutions, and inspire focus on relationships, goals, and ideas that might have been neglected. The sadness of the open source developer is a signal of a need for change. It is a comfort in that it means that the individual is not brain washed or stuck in the trance that dominates our culture. It’s the catalyst for inspiring idea that deviates from expectation. I don’t have answers now, but I am confident that they will come, and until they do, the sadness is endured and the questions are asked. What I refuse to do is deviate from strongly held values that give me meaning in my work. We cannot control the kinds of problems that we stumble into, or ultimately the way that any particular community treats us. But we can control ourselves, the choices that we make, and how we invest (or don’t invest) our time. In a world that is confusing and sometimes evil, until I am no longer breathing, I will, I must assertively persist regardless.
Suggested Citation:
Sochat, Vanessa. "The Sadness of the Open Source Developer." @vsoch (blog), 12 Dec 2017, https://vsoch.github.io/2017/sadness-open-source-developer/ (accessed 16 Apr 24).