Is it possible to compare two partial java code fragment with the tool?

[zqudm]

Is it possible to compare two partial java code fragment with the tool? If not, how to extend and add this feature into gumtreediff?

[jrfaller]

Hi! What do you mean by partial? Cheers

[zqudm]

For example ,given the following two partial code fragments, that is

if (hexDigits > 16) {
return org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);
}

and

if ((hexDigits > 16) || ((hexDigits == 16) && (firstSigDigit > '7'))) {
return org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);
}

How to get the diff between them?

[jrfaller]

I believe you can do it with the tree-sitter backend!

[zqudm]

Thank you for your quick response. According to your hints, i wrote a simple code to compute the difference between two code fragments that uses jtree-sitter backend . The simple code is listed as follows.

package main.java.utils;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTNode;
import org.eclipse.jdt.core.dom.ASTParser;
import org.eclipse.jdt.core.dom.ASTVisitor;
import org.eclipse.jdt.core.dom.CompilationUnit;
import org.eclipse.jdt.core.dom.SimpleName;
import org.eclipse.jdt.core.dom.VariableDeclarationFragment;
import com.github.gumtreediff.tree.TreeContext;
import com.github.gumtreediff.actions.*;
import com.github.gumtreediff.matchers.MappingStore;
import com.github.gumtreediff.tree.Tree;
import com.github.gumtreediff.tree.TreeContext;
import com.github.gumtreediff.utils.Pair;
public class Example{
public static void main(String[] args) throws Exception{
       String bug_code= "if (hexDigits > 16) {return org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);}";
       String fix_code= "if ((hexDigits > 16) || ((hexDigits == 16) && (firstSigDigit > '7'))) {return 
           org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);}";

        TreeContext ctx_1 = new JavaTreeSitterTreeGenerator().generateFrom().string(bug_code);
        TreeContext ctx_2 = new JavaTreeSitterTreeGenerator().generateFrom().string(fix_code);
        MappingStore ms = new MappingStore(ctx_1.getRoot(), ctx_2.getRoot());
        EditScript actions = new SimplifiedChawatheScriptGenerator().computeActions(ms);
        Diff diff = new Diff(ctx_1, ctx_2, ms, actions);
}
}

However, when running the code above, i got the following error:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project exercise1: An exception occured while executing the Java class. Cannot run program "tree-sitter-parser.py" (in directory "/tmp"): error=2, No such file or directory -> [Help 1]

I have checked the source code of gen.treesitter and found the error occurred because that the file "tree-sitter-parser.py" is missing. How to solve the problem? The following is the code that references the tree-sitter-parser.py.

public abstract class AbstractTreeSitterGenerator extends ExternalProcessTreeGenerator {
    private static final String TREESITTER_CMD = System.getProperty("gt.ts.path",
            "tree-sitter-parser.py");
    @Override
    protected TreeContext generate(Reader r) throws IOException {
        String output = readStandardOutput(r);
        TreeContext context = TreeIoUtils.fromXml().generateFrom().string(output);

        for (Tree t : context.getRoot().preOrder())
            if (t.getType().name.equals("ERROR"))
                throw new SyntaxException(this, r,
                        new IllegalArgumentException("Syntax error at pos: " + t.getPos()));
        return context;
    }

[jrfaller]

You need to install tree-sitter-parser.

[zqudm]

I have installed tree-sitter-parser, and when i run a simple APP below, an error occurred.

package main.java.utils;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTNode;
import org.eclipse.jdt.core.dom.ASTParser;
import org.eclipse.jdt.core.dom.ASTVisitor;
import org.eclipse.jdt.core.dom.CompilationUnit;
import org.eclipse.jdt.core.dom.SimpleName;
import org.eclipse.jdt.core.dom.VariableDeclarationFragment;
import com.github.gumtreediff.tree.TreeContext;
import com.github.gumtreediff.actions.*;
import com.github.gumtreediff.matchers.MappingStore;
import com.github.gumtreediff.tree.Tree;
import com.github.gumtreediff.tree.TreeContext;
import com.github.gumtreediff.utils.Pair;
public class Example{
public static void main(String[] args) throws Exception{
       String bug_code= "if (hexDigits > 16) {return org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);}";
       String fix_code= "if ((hexDigits > 16) || ((hexDigits == 16) && (firstSigDigit > '7'))) {return 
           org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);}";

        TreeContext ctx_1 = new JavaTreeSitterTreeGenerator().generateFrom().string(bug_code);
}
}

The error message is shown below:

An exception occured while executing the Java class. Cannot run program "tree-sitter-parser.py" (in directory "/tmp"): error=20, Not a directory

In fact, if i run the command "./tree-sitter-parser.py Example.java java" , an tree in xml format representing the code is outputted in the console.

How to solve the problem?

Cheers!

[jrfaller]

Is tree-sitter-parser.py in your system’ path?

[zqudm]

After adding tree-sitter-parser.py in the system's path, the code runs successfully. However, the tree-sitter backend doesn't seem to work for me. To illustrate, we use the following simple example to show the reason.

package main.java.utils;
import java.net.URL;
import java.util.List;
import java.util.ArrayList;
import java.net.URLClassLoader;
import java.io.File;
import java.util.HashSet;
import java.util.Set;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTNode;
import org.eclipse.jdt.core.dom.ASTParser;
import org.eclipse.jdt.core.dom.ASTVisitor;
import org.eclipse.jdt.core.dom.CompilationUnit;
import org.eclipse.jdt.core.dom.SimpleName;
import org.eclipse.jdt.core.dom.VariableDeclarationFragment;
import com.github.gumtreediff.tree.TreeContext;
import com.github.gumtreediff.actions.*;
import com.github.gumtreediff.matchers.MappingStore;
import com.github.gumtreediff.tree.Tree;
import com.github.gumtreediff.tree.TreeContext;
import com.github.gumtreediff.utils.Pair;
import com.github.gumtreediff.matchers.heuristic.gt.GreedyBottomUpMatcher;
import com.github.gumtreediff.matchers.heuristic.gt.GreedySubtreeMatcher;
public class GetURL{

public static void main(String[] args) throws Exception{
 String bug_code= "if (hexDigits > 16) {return org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);}";   
        String fix_code= "if ((hexDigits > 16) || ((hexDigits == 16) && (firstSigDigit > '7'))) {return org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);}";
 TreeContext ctx_1 = new JavaTreeSitterTreeGenerator().generateFrom().string(bug_code);
        System.out.println(ctx_1.getRoot().toTreeString());
        TreeContext ctx_2 = new JavaTreeSitterTreeGenerator().generateFrom().string(fix_code);
        System.out.println(ctx_2.getRoot().toTreeString());
        GreedySubtreeMatcher matcher = new GreedySubtreeMatcher();
        matcher.setMinPriority(2);
        MappingStore ms = matcher.match(ctx_1.getRoot(), ctx_2.getRoot());
        GreedyBottomUpMatcher matcher1 = new GreedyBottomUpMatcher();
        matcher.setMinPriority(2);
        MappingStore ms1 = matcher1.match(ctx_1.getRoot(), ctx_2.getRoot(), new MappingStore(ms));
        EditScript actions = new SimplifiedChawatheScriptGenerator().computeActions(ms1);
        Diff diff = new Diff(ctx_1, ctx_2, ms1, actions);
        TreeClassifier c = diff.createRootNodesClassifier();
        System.out.println(String.format("update src  %2d",  c.getUpdatedSrcs().size()));
        System.out.println(String.format("insert  %2d",  c.getInsertedDsts().size()));
        System.out.println(String.format("update dsts %2d",  c.getUpdatedDsts().size()));
         System.out.println("===========");
         for (Tree t :c.getInsertedDsts())
         {       System.out.println(t.toTreeString());
                System.out.println(t.getParent().getType());
         }

    }
}

Running the example code above will produce the following result:
program [0,93]
if_statement [0,93]
parenthesized_expression [3,19]
binary_expression [4,18]
identifier: hexDigits [4,13]
comparison_operator: > [14,15]
decimal_integer_literal: 16 [16,18]
block [20,93]
return_statement [21,92]
method_invocation [28,91]
field_access [28,69]
field_access [28,57]
field_access [28,52]
field_access [28,46]
field_access [28,38]
identifier: org [28,31]
identifier: apache [32,38]
identifier: commons [39,46]
identifier: lang3 [47,52]
identifier: math [53,57]
identifier: NumberUtils [58,69]
identifier: createBigInteger [70,86]
argument_list [86,91]
identifier: str [87,90]
program [0,143]
if_statement [0,143]
parenthesized_expression [3,69]
binary_expression [4,68]
parenthesized_expression [4,20]
binary_expression [5,19]
identifier: hexDigits [5,14]
comparison_operator: > [15,16]
decimal_integer_literal: 16 [17,19]
logical_operator: || [21,23]
parenthesized_expression [24,68]
binary_expression [25,67]
parenthesized_expression [25,42]
binary_expression [26,41]
identifier: hexDigits [26,35]
comparison_operator: == [36,38]
decimal_integer_literal: 16 [39,41]
logical_operator: && [43,45]
parenthesized_expression [46,67]
binary_expression [47,66]
identifier: firstSigDigit [47,60]
comparison_operator: > [61,62]
character_literal: '7' [63,66]
block [70,143]
return_statement [71,142]
method_invocation [78,141]
field_access [78,119]
field_access [78,107]
field_access [78,102]
field_access [78,96]
field_access [78,88]
identifier: org [78,81]
identifier: apache [82,88]
identifier: commons [89,96]
identifier: lang3 [97,102]
identifier: math [103,107]
identifier: NumberUtils [108,119]
identifier: createBigInteger [120,136]
argument_list [136,141]
identifier: str [137,140]
update src 0
insert 3
update dsts 0
parenthesized_expression [24,68]
binary_expression [25,67]
parenthesized_expression [25,42]
binary_expression [26,41]
identifier: hexDigits [26,35]
comparison_operator: == [36,38]
decimal_integer_literal: 16 [39,41]
logical_operator: && [43,45]
parenthesized_expression [46,67]
binary_expression [47,66]
identifier: firstSigDigit [47,60]
comparison_operator: > [61,62]
character_literal: '7' [63,66]
binary_expression
parenthesized_expression [3,69]
binary_expression [4,68]
parenthesized_expression [4,20]
binary_expression [5,19]
identifier: hexDigits [5,14]
comparison_operator: > [15,16]
decimal_integer_literal: 16 [17,19]
logical_operator: || [21,23]
parenthesized_expression [24,68]
binary_expression [25,67]
parenthesized_expression [25,42]
binary_expression [26,41]
identifier: hexDigits [26,35]
comparison_operator: == [36,38]
decimal_integer_literal: 16 [39,41]
logical_operator: && [43,45]
parenthesized_expression [46,67]
binary_expression [47,66]
identifier: firstSigDigit [47,60]
comparison_operator: > [61,62]
character_literal: '7' [63,66]
if_statement
binary_expression [4,68]
parenthesized_expression [4,20]
binary_expression [5,19]
identifier: hexDigits [5,14]
comparison_operator: > [15,16]
decimal_integer_literal: 16 [17,19]
logical_operator: || [21,23]
parenthesized_expression [24,68]
binary_expression [25,67]
parenthesized_expression [25,42]
binary_expression [26,41]
identifier: hexDigits [26,35]
comparison_operator: == [36,38]
decimal_integer_literal: 16 [39,41]
logical_operator: && [43,45]
parenthesized_expression [46,67]
binary_expression [47,66]
identifier: firstSigDigit [47,60]
comparison_operator: > [61,62]
character_literal: '7' [63,66]
parenthesized_expression

Note that the two code fragments are the followings

String bug="if (hexDigits > 16) {return org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);} ";
String fix = "if ((hexDigits > 16) || ((hexDigits == 16) && (firstSigDigit > '7'))) {return org.apache.commons.lang3.math.NumberUtils.createBigInteger(str);}";

Our goal aims to use gumtree tool to extract the high-level code modification pattern. In particular, the code modification pattern of the two code fragments above is modification of condition expression of If statement.

Obviously, after the gumtree tool computes the difference between them,three Insert actions are identified. However, based on these Insert actions, it is hard to associate the modification with the condition expression of If statement since the Tree Interface in gumtree does not have attribute such as AST node. on the contrary, we note that gumtree-spoon-ast-diff extends this by adding the attribute CtElement parent in AdditionOperation class. Thus, in order to realize my goal, i have to extend tree generator in the similar way like gumtree-spoon-ast-diff, right?

While studying the code of gumtreediff, the gumtreediff also supports the generation of tree by using jdt core parser backend. In fact, Jdt core parser can also parse partial code fragment. Specifically, in Jdt, when replacing the parsing kind ASTParser.K_COMPILATION_UNIT with ASTParser.K_STATEMENTS, the parser can generate the ast tree from the partial code fragment. So i can adopt jdt parser backend instead of tree-sitter-java backend, right?

Thanks!

[jrfaller]

Hi!

On a GumTree action you have the reference to the involved node. On a given node you can traverse its ancestors. I therefore think for instance here you can notice that the three actions involves node whose common ancestor has a IfStatement type. gumtree-spoon-ast-diff will of course be better if you need more information from the code model (since it retains the information produced by JDT). However, it won't cluster the actions for you into higher order actions. This is something you will need to do by analyzing the actions. In any case, I think that the solution you propose works in both tools.