UPDATE: After publishing the first version of the post, I had the chance to talk with people from CodePilot and Codota (both tools are described below). I can now say that, yes, intelligent IDE’s are disappointing today but I’m confident that will see drastic improvements in the near future. I’ll update the post once these new exciting features are available.
Artificial intelligence is going to change the way software is developed. I think most of us can agree with this (a couple of opinions and even a new conference on this exact topic). But I wanted to see if, right now, there is already something useful for the regular developer.
So I searched for all so-called intelligent/smart “copilots” for developers and tried to accomplish a “simple” programming task with their help: opening and reading a local text file. This is something simple enough for them to be able to help and that, at the same time, a task complex enough (for me) to make me look it up online every time I have to do it (I never manage t remember the names of the classes to be combined for this).
Reading the description of smart IDEs (“understands the world’s code and provides you with the right suggestion at the right time” , “analyzes all the code on the web and gives you fast, smart completions ordered by popularity”), it seemed that my little experiment would be a great success. But it wasn’t. Let’s dive in the tools I tried (in no particular order, let me know if I missed any!).
Eclipse Code Recommender
It offers a code completion feature for Java that sorts out the possible methods to use at a given point based on what other developers have done in the past. It uses Jayes, a pure Java, open source Bayesian Network library, to power this code completion. As the Figure shows, in my reading loop, Eclipse Code Recommender suggests
next()as first option.
The “learning” part uses code samples taken from the Eclipse Marketplace. Recommendations are provided for
org.eclipse.* APIs. There are plenty of recommendation options to tune the suggestions process. Submitting your own previous code (e.g. to “train” the system with your own coding style) requires a CodeTrails license. As an alternative, you can register (and use later on) code snippets.
CodePilot.ai is more of an advanced search code engine. It can search in your local environment or (paid options) on StackOverflow or GitHub.
In my example, it quickly detected that I had already another piece of code using the same
Scanner class. For the moment, CodePilot is not using any kind of machine learning algorithm to provide more meaningful results so it basically shows all “hits” for the keywords you look for.
Codota promises to provide you with the right suggestion at the right time. Up to you to judge how close Codota is to reach this goal based on the suggestions I got when Codota detected the classes I was using in my code.
On the “Pro” side, Codota integrates nicely with the IDE and is able to detect the APIs/classes you’re using and tries (for free) to check Stack Overflow for suggestions but as you can see it fails to really grasp the context where those classes are being used.
There is also a version for teams which looks more promising and useful since it helps to detect which of your other team members has already wrote a similar piece of code (and who did it so that you can go and ask for help).
Kite defines itself as a cloud-powered copilot that boosts your programming environment. And it’s the only for Python.
In terms of features, it’s quite similar to some of the above tools. It is useful to give you inline documentation plus examples from its broad collection of samples that could be useful to you (to see popular patterns, a Pro version is needed). This is what I got for my file reading example.
Again, suggestions are based on the classes/methods you use not on the actual usage of those in your code.
Other smart assistants
CodeCorrect plugs into the StackOverflow API to find solutions to common errors in a developer’s code. It was the result of a TechCrunch Disrupt Hackathon and got some media attention but the project seems to remain at the “proof-of-concept” stage at the moment with no further updates in the last months.
Ai.codes aims to provide intelligent code completion for a number of languages. It’s developed by a team of AI researchers, compiler writers and software engineers. They released an IntellJ plugin called “AI predictive coding” in 2016 as part of its public beta phase but it looks like they never really got out of beta.
PHPBot is a bot that pretends your virtual assistant when writing PHP code. Given the name of the PHP function, it will provide a set of usage examples for that function to help you understand how that function works. It’s more a replacement of the official php documentation than an intelligent system able to understand your code and suggest adapted examples for it.
Based on this simple experiment, I’m confident to conclude that, so far, intelligent IDEs are more marketing than a reality. I do believe that this kind of tools will progress a lot in the next years and may become a real virtual assistant for the programmers but we are far from there yet.
This is, in fact, a hot research topic right now (we’re in fact working on a research agenda for smart IDEs ourselves but for modeling tools), so it’s perfectly understandable that the results are not that promising yet.
Also, they have a very strong competition. It’s not enough they are useful, they need to prove they are faster and more accurate than the programmer searching himself for an answer online. If I search on Google for something like “open file in Java”, I get an answer from SO that directly provides a code sample that I can just copy&paste in my editor. Very difficult to beat this.
Also, note that the “intelligence” comes from trying to sort, rank,… the code samples the tools find in SO or GitHub, not so much on trying to understand the code you’re trying to write and provide contextual help for that. IMHO, this is the direction they should be heading to. I want the IDE to help me write my code, not to help me find online a code sample that I’ll need to read, understand and adapt. Even better, if, when providing a potentially relevant sample, they are able to adapt the sample to the id’s (and even coding style) I’m using in my code.
In a more distant future, I guess we´ll also see some coder bots with whom you’ll be able to do some pair programming and chat about what is your goal regarding a specific method and let the chatbot find the best solution for you. But for now, we need lots of patience.