This page is a static mirror of an older article found on Sporkbox. Its content no longer fits in with the current direction of Sporkbox, but is kept here for link integrity. Please bookmark this new URL if you wish to link to this content.

The Dangers of Software Evangelism

Software evangelism in a meritocracy is damaging to its social fiber. In a society that champions merit and achievement, what room is there for promoting one software package or another? Software distributors have their own decision-making policies regarding the software that they choose to ship. Evangelism interrupts the flow of the decision-making process, sometimes completely derailing it. When this happens, decisions are either put off or made in haste. As a result, software distributions and their users suffer in technical and social ways. Pushed software can create unfixable configurations, reveal unexpected bugs, render past knowledge obsolete, and create friction between users and developers. Thus, it's important to identify and stop software evangelism whenever possible, or risk losing talented members and reducing the richness of a meritocracy.

What is a meritocracy?

A meritocracy is a social and political philosophy with the core belief that power -- either social or political -- should be vested in people according to their merit or ability. This is in contrast to societies that grant power to those born to specific classes, have a certain degree of wealth, or have ties to influential players in the political landscape. It's a somewhat progressive take on social organization and has proven to be a boon for software development.

The free, open-source software (FOSS) community is one of the largest active meritocracies, with millions of developers worldwide taking part. In it, people are generally respected according to what they've contributed to the public. This is usually done by "scratching your own itch", a process where a developer finds a need for a particular piece of software and, upon its absence, creates a solution on their own. The developer then releases this software and its source code under a liberating copyright license (such as the GPL, MIT, or BSD licenses) and gives it away so others may be able to solve the same problem that they had. In a typical scenario, the developer puts together a website for their application after finishing its first release and posts something on their blog or a distribution's forum. From there, people begin using it or hacking on it, bugs are found and fixed, and the project may develop a collection of dedicated users.

Linux — a free, open-source kernel — is a shining example of this process. Its original release consisted of a single message to the comp.os.minix newsgroup:

Hello everybody out there using minix -

I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).

I've currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I'll get something practical within a few months, and I'd like to know what features most people would want. Any suggestions are welcome, but I won't promise I'll implement them :-)

            Linus (torv...@kruuna.helsinki.fi)

PS. Yes - it's free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(.

It went on to become one of the most active software projects ever. Without any corporate pushing, grassroots movements, or technical bullying. If one of the most central projects in the FOSS world can reach such lofty heights without a big push, one may ask, "Why are certain other projects pushed or suggested so much?" This article aims to highlight an example of software evangelism and the effects it has on a community.

Case Study: systemd and Debian GNU/Linux

One such project that has gained the attention of the FOSS world for its evangelism is systemd, a piece of software that helps bring a GNU/Linux system to a usable state by starting daemons, initializing hardware, and other functions such as multi-seat configuration, logging, splash-image support, and enforcing an optional, separate security model. It's garnered quite a lot of praise and criticism; much of it due to the way it's been pushed and evangelized across the FOSS ecosystem.

Debian GNU/Linux is an older distribution that can be run on multiple different kernels. As such, their software undergoes a healthy amount of testing on various kernels to ensure that it functions well. Decisions in Debian are made carefully and methodically. A discussion on their mailing list titled "Gentoo guys starting a fork of udev" sparked a heated discussion about systemd.

One Debian user writes:

I think this is an interesting read: http://article.gmane.org/gmane.linux.gentoo.project/2262

The full thread is here: http://thread.gmane.org/gmane.linux.gentoo.project/2262

As Gentoo guys and some major kernel people are protesting about the insanity Kay and Lennart have done to udev, probably this is a window of opportunity to stop being annoyed in Debian, by brutal RedHat decisions which we don't necessarily (have to) agree on (eg: move to /usr, merge with systemd, etc.).

Thoughts anyone? Would anyone have time to work with Gentoo? What is the view of the kernel team and the udev (marco?) maintainers?

This is a normal part of operating a distribution: as more original software projects crop up, it becomes important to decide which projects are better suited to the distributions goals. Sometimes, distributions work together to achieve a common goal, in the same way that individual developers would. This innocuous question begins a long chain of discussion that becomes infected by software evangelism.

Debian developer John Paul Adrian Glaubitz writes:

But anyway, we're getting tired of their ADHD-driven changes just to change things

TBH, I'm getting tired of people who are constantly shooting against them because these people are unwilling to accept changes. We're not bringing Linux forward if we stick to 30-year-old concepts. systemd is a good design and most people actually agree otherwise it wouldn't become standard on so many distributions (except Ubuntu, but that's rather a political decision IMHO).

One of the Arch developers actually made a couple of good points why they switched to systemd as default [1].

Cheers,

Adrian

[1] https://bbs.archlinux.org/viewtopic.php?pid=1149530#p1149530

This begins a resurgence of a discussion concerning systemd on Debian that (as of that posting) had happened multiple times; each time it had resulted in the developers choosing not to make a decision on systemd until it was needed. Adrian pushes his viewpoint multiple times, insisting that Arch Linux did not lose users when they switched to systemd and claims that GNU/Linux would not make progress were it not for projects like systemd.

Thomas Goirand notes the argumentation style used by Adrian:

Hi Adrian,

Roger talks about technical design and upstream author choices, with some I believe very valid points, and you're talking about features! This can't go anywhere.

Thomas

Adrian continues unfazed by this, suggesting that every distribution should be using the software that he advocates by default, as written in this mailing list post:

If both Ubuntu and Gentoo would just go with the rest of the community and accept systemd, we wouldn't have to bother whether udev runs without systemd or not. I don't see anyway why something as low-level as udev should be highly portable in the first place.

His devout support for systemd continues in his reply to Wouter Verhelst:

udev took quite some time to be accepted by the community too, but now it's probably fair to say that it has been. To try to couple that to systemd sounds like a bad form of systemd advocacy to me.

Yes, I agree it leaves a bad taste for sure. But again, I am not so sure if we really need to be able to choose our init system. I mean, do we have this discussion over mdev vs udev? Or even devfs?

The second sentence may concern some readers. When I read this, I was rather appalled at Adrian's lack of humility. If such a person does not seem to value the flexibility of choice for one system component, do they value any of the other choices that go into creating a system? If they don't, should they be a developer of a large distribution that's known for giving users a broad range of choices?

Debian developer Russ Allbery responds to Adrian's evangelism:

Free software is a social activity. The past history of qmail should be informative here (or, for that matter, both gcc and glibc, which had to go through disruptive forks to sort out long-term issues). One of the determiners of the long-term success of a free software project is the social skills of the primary maintainers, regardless of their skill as software designers.

I'm on the side of wanting to support a variety of different choices in the archive so that people can experiment and evaluate and choose what works best for them. But to the extent that we have to pick winners and losers (and, to be clear, I think it's premature to do that for init systems), I for one will always advocate taking social considerations into account as well as technical considerations. In the long term, they often matter more than the initial technical design.

Bernhard Link, another Debian developer, also responds:

On Thu, Nov 29, 2012 at 08:22:41PM +0100, Bernhard R. Link wrote:

As our world has not yet seen bug-free software handling every single situation the authors did not think about properly, the expectation of what happens if one runs into a not-yet fixed bug is an important issue for many people.

Absolutely. But again, this is true for any software and this is especially true for old designs which couldn't cover certain setups which simply didn't exist back then.

But you have to understand that you put something that only does promises against something that exists. Something that exists means that people have experiences with it, now how it behaves, know how to debug it and now how to make it work again when it breaks. On the other side is systemd which has not yet been available in a single Debian release and could not yet prove itself anywhere.

So if it would be proposed as some more init system to support or a cool new stuff to experiment with it, you would not get that much emotional reactions.

But as people demanded making it the only available system before it could prove itself and while all most people know is that it is quite a different design which wants to solve all the problems at the same time (which often enough means it creates more problems than it solves) and that it has ties which other projects people have had no good experiences with (for enough people pulse-audio is a synonym of "my sound is not working") you should have a different reaction than ridiculing people not wanting to bet their future on it.

Adrian's not alone in the evangelism going on in the Debian mailing list thread. Uoti Urpala writes:

But to the extent that we have to pick winners and losers (and, to be clear, I think it's premature to do that for init systems),

I think there's already enough evidence to show that systemd is clearly the best choice. How much more would you expect to have before it would not be "premature" any more?

Russ responds yet again:

I don't see any need to have a firm answer to that question at this time. The point is less about the amount of evidence required and much more about the fact that there's no reason to make this decision unless and until we actually need to as a project. We're not at that point.

At this point, the single most annoying thing about systemd is the people who are advocating it on debian-devel at every opportunity and seem incapable of shutting up about it for more than a week, even though the repeated conversations are both useless to the project as a whole and don't vary with repetition.

And further response to Adrian, from Roger Leigh:

Well, but I would say Ubuntu is to be blamed here. As opposed to upstart, systemd has many supporters and contributors in the industry (Intel, ProFusion, SuSE, RedHat) while upstart is virtually only actively developed by Canonical if am I not mistaken. Plus, you have to sign a contributor's agreement with Canonical which leaves a bad taste in my mouth. That shouldn't be the case with true free software, should it?

Enough. None of this rubbish has anything at all to do with Debian. You've taken your fanboyish advocacy of systemd to a ridiculous extreme over the last week or so. Please tone it down. Nothing done in Debian has anything to do with any of the above companies, and if anything, having all those "supporters and contributors" attempt to ram systemd down our throats whether we like it or not counts against it rather than for it.

I'll forego further quoting in the interest of brevity. The core issue at hand is that a few individuals saw a curious discussion as an opportunity to spread and promote their preferred software far beyond what's generally acceptable in the given community (1 to 2 posts at most, generally).

One may assume that this is commonplace in the FOSS world, with Vim and Emacs (two advanced text editors) "holy wars" and other cultural pastimes. At the end of those style of discussions, however, most people agree that their chosen editors suit them and they go back to enjoying themselves. It's clear that that's not the case with software evangelism like what was outlined previously. There's a clear agenda, and the evangelist will not stop until he's ruffled more than enough feathers.

Normally this could be written off; some software just has fanatical users and is no fault of the original developer(s). However, there's a rather telling Google+ posting from the project's developer, Lennart Poettering. It outlines his attitude toward the same topic that Debian's list was discussing. Among the commenters of his post, one will find none else than John Paul Adrian Glaubitz, the same person who was giving the Debian list trouble, pushing Mr. Poettering's software. This was complimented by Mr. Poettering himself:

+John Paul Adrian Glaubitz You rock, BTW. Saw your arguing on debian-devel. Much appreciated!

In this case, the evangelism is encouraged by the lead developer. Is this the sort of behavior that we wish to see from developers of free software? This may become more worrisome if we look a little further at the developer's other behavior as seen on Wikipedia as of February 11th, 2013:

In May 2011 Lennart Poettering proposed systemd as a dependency for further releases of GNOME. As systemd is available only on Linux, the proposal led to the discussion of possibly dropping support for other platforms in future GNOME releases. While some people responded to the proposal with criticism others suggested the idea of a GNOME Operating System on top of the Linux kernel.

So it would seem that the software author himself is an evangelist and encourages the users of his software to spread its use. In a meritocracy, this is generally considered foul play and dishonest at worst, and annoying at best. This is due to the importance placed on technical and social merit. If the software is technically sound, then its merits will draw users on its own. If the developers are sociable and the documentation is good, then the project will keep users. As seen in the first example in this article with Linux, only a scant amount of promotion is really needed to get a project started and get some eyes on it. Posting beyond that may be perceived as annoying or even threatening to other people due to its similarities to corporate marketing.

In closing, the above examples provide a detailed outline of how software evangelism can disrupt the decision-making process with software distributions and perhaps cause social unrest. This unrest can result in distributions making decisions in haste (like the Arch Linux team) or being met with more aggressive evangelism and tainting the feel of a discussion, or derailing it entirely (like Debian's mailing list). In either case, developers and users suffer from the influence of people who decide that their chosen piece of software is more important than the other options out there.

Should we allow this sort of marketing in the free software world? Should we be more tolerant and provide better social constructs that allow software to gain attention? What can our community do about ambitious projects that seek to change the entire software landscape in the same vein as systemd? Leave a comment and contribute your voice to the conversation!