Adding Some Color to the RebelLabs Developer Productivity Report 2016

Oracle Blog's

By Geertjan-Oracle on Aug 25, 2016

I used to have a very nice colleague who would never disagree with me. He'd never say 'I disagree with you'. Instead, he'd always say, 'let me add some color to that', after which he'd proceed to, essentially, disagree with me. :-) [I miss you, Ashwin!] But it was a nice experience. He wasn't disagreeing, he was adding color.

In that light I'd like to look at an aspect of the RebelLabs Developer Productivity Report 2016. It wasn't released very long ago and has the following results for IDE usage:

On the face of it, things aren't looking so great for NetBeans! In fact, NetBeans is doing very badly indeed. You can't really talk about "the top three IDEs" anymore, there's clearly only "the top two IDEs". However, let's look a bit more closely and ask ourselves some questions. Yes, let's add some color to that!

ZeroTurnaround & Me

Let me start by saying I love the ZT team. I have known several members of ZT for a lot of many years.
Jevgeni I remember meeting at JavaZone in Norway many years ago when he was talking about his Aranea web framework; Anton I have done several JavaOne sessions with; the JRebel plugin for NetBeans is one of the most actively developed NetBeans plugins; Igor has created the most amazing MindMap plugin for NetBeans; I've talked to Simon at several conferences; Alan and Michael I met in Denmark and I claim some role in surfacing them in the community and getting them noticed by Jevgeni (watch this cool YouTube clip with Alan, Fabrizio, and me); Oleg I spent a crazy time with in the Bering Sea; Sang, Adam, and Geert who used to work there are guys I have known forever as well. And, on top of all that, ZeroTurnaround is a NetBeans partner.
So, again, I am really adding color here and seriously not trying to do anything other than that. I love ZT, OK.

Tone & Content

In the report: "Overall we received 2044 responses." Moreover, when I see the response to "What is your job description?", I don't see anything about "I am a student". In fact, the word "student" doesn't appear in the report. Nothing at all wrong with that, of course. The survey was not aimed at students at all.
Already one can feel the kind of color that can be added here. Slightly over 2000 respondents, which seems small to me. And no students.
Plus, it took quite a bit of motivating to get those 2000 respondents. Especially the last few hundred took some work, I saw quite a bit of activity in getting those, e.g:



You know who else didn't participate? The NetBeans community. On 1 March this year, there was a brief discussion on the NetBeans Dream Team mailing list. The NetBeans Dream Team is a group of volunteers outside of the NetBeans team and outside of Oracle. They're NetBeans enthusiasts who are recognized for their activism by being invited to join the group to share insights and activities and ideas around NetBeans. We discussed the RebelLabs productivity reports. Especially the fact that the tone of the reports is definitive and assertive, despite not covering at least the simplest of these concerns, i.e., a pretty low number of self-selecting respondents, none of whom are students which is a key demographic that NetBeans has targeted from the beginning.

My suggestion was that this year, in contrast to previous years where we were a bit sceptical too, we go full on for the survey, promote it to the community, to the NetBeans enthusiasts around the world, the mailing lists, etc. However, one of the people in the discussion suggested, bearing in mind our objections: "Instead of helping them with extra data points you might just publicly ask NetBeans users to NOT respond to RebelLabs surveys." I disagreed strongly because I love ZT! And ZT is a NetBeans partner, on top of that.

So, we did nothing. We let it go, knowing that by not promoting this to the NetBeans community, the results would probably be like they turned out to be. The results of the survey provide a limited view on a self-selecting segment of the Java community.

Reap & Sow

However, let's go a bit further. Given the limitations outlined above, which in themselves disqualify the results, or should disqualify the results from being used to say anything about the actual usage of the tools, is there an explanation for those 2044 respondents having responded in the way they did? Why did only 10% of those problematic non-student 2044 respondents pick NetBeans?
JetBrains is incredibly admirable for their conviction. JetBrains supports their tools without a second of doubt. Oracle, well, not so much. Sometimes yes, sometimes no. Sometimes lots of enthusiasm, sometimes just a bit, something nothing. The last time NetBeans had an evangelism team was in the days of Roman Strobl, I guess around 2007 or so, when NetBeans was part of Sun and when there were about 10 fulltime NetBeans evangelism staff (in addition to about 6 technical writers, of which I was one) constantly and consistently promoting NetBeans. There's nothing left of that. Well, just me, plus the NetBeans Dream Team, an unpaid volunteer force to be reckoned with, but not a paid staff strategizing and planning the best way to reach developers with tools etc. Meanwhile, JetBrains has been investing in that aspect of its organization to great effect. Kudos to them.
To add some color to the results, therefore, I tentatively suggest that they are a reflection of the conviction versus the ambivalence of the organizations behind those tools, as well as the staffing and strategies for promoting them. On this basis, the percentages mirror the backing that the tools have received from the organizations that sponsor them.

Apples & Pears

We need to go further still and ask the "why" question. What's the reason for the different levels of enthusiasm of these organizations for their tools? JetBrains and Oracle have completely different aims, of course. JetBrains is explicitly a tooling company. The key slogan, a really good one, is: "Whichever technologies you use, there's a JetBrains tool to match."
Oracle, on the other hand, is historically a database company and increasingly a Cloud company. In contrast to JetBrains, for Oracle tooling plays, at best, a supporting role, and is, at worst, an after thought. Though one would expect tooling to play a bigger role, it tends to fall a bit by the wayside in the final push to a technology release.

Facebook & Twitter

So, if the ZT survey cannot accurately reflect IDE usage, is there another way to conclusively do so. No. The only thing anyone can provide are very tentative indicators.
Here are some:



What does the above show? One thing I find interesting is that JetBrains is spending a lot of energy while not getting as compelling a result as the other two IDEs. Also note that normally Eclipse doesn't have as much engagement on Facebook as this week. The last few weeks have been special because of Eclipse Che, generating a lot of engagement. NetBeans, though, is clearly on top in terms of Facebook likes. Note also that there's never more than one or two new items on the NetBeans Facebook page per day. Sometimes nothing. Because we don't want the news from yesterday to get lost. So we add one new item every other day or so. A different strategy to JetBrains, clearly, which also has to contend with the fact that IntelliJ IDEA is only one of their tools. They have a lot to talk about, though it doesn't seem to result in a lot of new Facebook likes.

Next, let's take a look at Twitter followers. Below you see NetBeans, then JetBrains, then Eclipse. Again, NetBeans has the most followers, i.e., the most people interested in receiving tweets from one of the three IDEs is from people interested in NetBeans. Again, notice the disparity between the amount of tweets from JetBrains versus the considerably lower number of followers:



Of course, someone will say, this says nothing about what people are working with. Hmmm. I wonder about that. How many Twitter accounts do you follow of tools that you're not using? And if you want to see a comparison between real tool usage, take a look at the difference between the number of people using the Chrome plugin for NetBeans versus the Chrome plugin for IntelliJ:

That's a difference of about 50,000, which has been constant, plus or minus a few thousand, over the past years.

If you were to want to reach as many developers as possible, with your message, or your technology via a plugin, which of the three IDEs would you immediately suspect could be used to reach the largest number of people? Over on YouTube, look at the marked difference in views between two different YouTube clips, about the same technology, i.e., Oracle Developer Cloud Service, installed in Eclipse vs. installed in NetBeans:



The two YouTube clips were created in the same period. The Eclipse YouTube clip has less than 300 views, while the NetBeans equivalent has around 4000 views. Neither of those numbers are very high, which is probably because Oracle Developer Cloud Service is an extremely non-sexy topic. Nevertheless, the NetBeans views are orders of magnitude higher, clearly.
Of course, I am not arguing that NetBeans is the most popular IDE. I don't believe that to be true at all, in fact. Plus, that's not the point I am making here, in any way. No one can make that claim. I am simply adding color to the survey results, since these different statistics from completely different sources appear to indicate something different to what the survey suggests.

Intention & Abuse

Where it becomes dangerous is where the ZT survey is used as a stick to beat an IDE with. That was never the intention of the awesome people at ZT, though it is exactly what happens when surveys are released.

Surveys are always flawed and never present the full picture. Like when I argued with (ah, 'added color to') the RedMonk survey a few years ago, if we don't have the full climate picture, we shouldn't be doing the weather report. At the very least we should be extremely circumspect and not assert anything at all lest our assertions be abused in contexts for which they were never intended.