[JSON_PROCESSING_SPEC-9] Adding methods to JsonParser to get objects Created: 14/Jun/12  Updated: 20/Sep/14

Status: Open
Project: json-processing-spec
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: jitu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Something like the following, that gives the objects from the parser

/**

  • getJsonValue(JsonObject.class) is valid in START_OBJECT state,
  • moves cursor to END_OBJECT
    *
  • getJsonValue(JsonArray.class) is valid in START_ARRAY state
  • moves cursor to END_ARRAY
    *
  • getJsonValue(JsonString.class) is valid in VALUE_STRING state
    *
  • getJsonValue(JsonNumber.class) is valid in VALUE_NUMBER state
    */
    public <T extends JsonValue> T getJsonValue(Class<T> clazz) { return null; }


 Comments   
Comment by kchung [ 20/Sep/14 ]

This issue, together with issue 29, would be useful for processing big JSON data. It is important to be able to load the JSON values are of interest and skip those that are not.

However, it is most likely that the criteria for getting or skipping JSON values lies within the values themselves. For instance, say we have an array of objects representing the contacts, and we are only interested in getting the contacts who are females that live in New York city. We'll need to process some parts of the object to determine if it should be loaded or skipped, and we cannot make that decision at the beginning of an object. How to factor that into the API is the challenge.





[JSON_PROCESSING_SPEC-22] Injection support for JsonParserFactory/JsonGeneratorFactory Created: 29/Oct/12  Updated: 07/Jan/13

Status: Open
Project: json-processing-spec
Component/s: None
Affects Version/s: 1.0-next
Fix Version/s: 1.0-next

Type: New Feature Priority: Major
Reporter: jitu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

http://java.net/projects/json-processing-spec/lists/users/archive/2012-10/message/112

This can be considered for later releases






[JSON_PROCESSING_SPEC-32] matchName(String) methods on JsonParser Created: 03/Dec/12  Updated: 03/Dec/12

Status: Open
Project: json-processing-spec
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: jitu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

JsonParser#matchName(String) kind of methods are required for performance.
http://java.net/projects/json-processing-spec/lists/users/archive/2012-11/message/40






[JSON_PROCESSING_SPEC-31] Incremental processing of large strings Created: 03/Dec/12  Updated: 03/Dec/12

Status: Open
Project: json-processing-spec
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: jitu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Provide way to parse/process large string values incrementally in parser






[JSON_PROCESSING_SPEC-29] JsonParser#skipValue() Created: 03/Dec/12  Updated: 03/Dec/12

Status: Open
Project: json-processing-spec
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major
Reporter: jitu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

skipValue() discussion is at
http://java.net/projects/json-processing-spec/lists/users/archive/2012-11/message/64






[JSON_PROCESSING_SPEC-61] Support for JSON Patch Created: 02/Dec/13  Updated: 18/Feb/15

Status: Open
Project: json-processing-spec
Component/s: None
Affects Version/s: None
Fix Version/s: 1.1

Type: New Feature Priority: Major
Reporter: jitu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Consider adding support for JSON Patch. The RFC is at http://tools.ietf.org/html/rfc6902

Perhaps, it can be used for transforming one JsonObject/JsonArray to another (works well since JsonObject/JsonArray are immutable)



 Comments   
Comment by kchung [ 09/Dec/14 ]

Note there is another alternate spec for patching a JSON document: Json Merge Patch, rfc 7396

This uses a diff patching, and is less verbose. There are some short-comings:

  1. It uses null for delete, so you cannot replace a value with null.
  2. It does not patch array entries, so you'll need to replace the whole array.

Maybe we should consider supporting both.

Comment by mitemitreski [ 17/Feb/15 ]

@kchung as you said https://tools.ietf.org/html/rfc7396 is the one that is a lot less verbose and it is more practical.
In my book, this one should be given a higher priority.

On the other hand, if we already have a JSON-Pointer as part of the spec they both make sense.

Comment by asotobu [ 18/Feb/15 ]

As @mitemitreski said, having json pointer implemented, the hard work is done and we can offer support for both of them, maybe we can start with merge patch and then if we are on time we can implement patch as well, or we can define both at first.

Comment by kchung [ 18/Feb/15 ]

Agreed that we should do both, at least we should not exclude either at first.

In terms of implementation order, I also agree that JSON Pointer should be done first. In fact I believe we should tackle the editing or transformation operations first, because they are lower level operations that both JSON Pointer and JSON can build on. See https://java.net/jira/browse/JSON_PROCESSING_SPEC-67 for more details.

Agreed also that JSON patch should include a diff function.





[JSON_PROCESSING_SPEC-58] Add Support for JSON Queries Created: 28/May/13  Updated: 23/Feb/15

Status: Open
Project: json-processing-spec
Component/s: None
Affects Version/s: 1.1
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: reza_rahman Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

I received feedback that JSON-P should add support for being able to query JSON objects (a la XPath). I am aware that this is an immature field with some competing approaches but think this is very valid feedback particularly in making JSON-P more compelling vis-à-vis existing JSON processing solutions.

Do let me know if anything needs to be explained further - I am happy to help.

Please note that these are purely my personal views and certainly not of Oracle's as a company.



 Comments   
Comment by kchung [ 06/May/14 ]

There is currently no standard for JSON queries, so it would be premature to put query into JSON APIs.

Since JsonObject is a Map, and JsonArray a List, some querying should be possible using the Lambda expression in JDK 8. For instance,

  contacts.stream()
    .filter(p->p.getString("city").equals("Santa Clara"))
    .forEach(p->System.out.println(p.getString("firstName") + " " + p.getString("lastName"));)

would print out the names of your contacts residing in Santa Clara. With a little more work, it should be possible to return a JSON document of the query results.

Maybe the spec should look into providing better support for doing queries in Lambda?

Comment by mitemitreski [ 23/Feb/15 ]

Would filtering or transformations using JSON pointer make sense?





[JSON_PROCESSING_SPEC-60] Support for JSON Pointer Created: 28/Oct/13  Updated: 25/Feb/15

Status: Open
Project: json-processing-spec
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: jitu Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Consider adding support for JSON Pointer. The RFC is at http://tools.ietf.org/html/rfc6901



 Comments   
Comment by asotobu [ 18/Feb/15 ]

JSON Pointer is really a must for next release of JSR 374 because without them it is going to be pretty hard for example implementing JSON Patch feature. In fact because JSON Pointer is required for implementing Json Patch we should discuss this first.

I have collaborated with Jackson guys implementing Json Pointer support so it can be used in Json Patch implementation. Although now I am going to mention something of low level implementation I prefer to note now so I don't miss it later.

Json Pointer allows developers to get some part of a json document (something like x-path for xml). But you also need some operations to manage json pointers themselves. In this sense next management operations are required for json patch implementation.

  • last() -> Returns the leaf of current JSON Pointer expression. Leaf is the last non-null segment of current JSON Pointer.
  • tail() -> Accessor for getting a "sub-pointer", instance where current segment has been removed and pointer includes rest of segments. For matching state, will return null.
  • head() -> Accessor for getting a pointer instance that is identical to this instance except that the last segment has been dropped. For example, for JSON Point "/root/branch/leaf", this method would return pointer "/root/branch" (compared to tail() that would return "/branch/leaf").

Of course we would need other operations which will involve the json document and the json pointer, but last, tail and head are operations that only affects the json pointer expression.

Comment by kchung [ 21/Feb/15 ]

Let's up level the api design for Json Pointer a bit. There are a number of issues to consider.

  1. Should JsonPointer be an interface or a concrete class? I guess the answer depends on whether we need to create an instance of it from the factory (SPI), like how it is now for JsonParser etc. I think it would be simpler if we can just do a new JsonPointer("q/b/c/d") to create one.
  2. Since a Json Pointer refers to a Json Text, should the text be part of JsonPointer? If not, the text will need to be applied when we do operations with the pointer. In other words, which of the following two scenarios is better:
      JsonObject target = ... ; # the target Json text
      JsonPointer pointer = new JsonPointer("a/b/c/d");
      JsonValue value = pointer.get(target); # get the value referenced

    or

      JsonObject target = ...; # the target Json text
      JsonPointer pointer = new JsonPointer(target, "a/b/c/d");
      JsonValue value = pointer.get(); # get the value referenced

    One might argument the the first is better since the same pointer can be applied to other text. but I am not sure if that is useful in practice.

  3. What other operations can we do with a pointer? To support Json Patch, we should have at least the following:
    1. get the value at the reference location
    2. add a new value to the reference location
    3. remove the value at the reference location
    4. replace the value at the reference location with another value
Comment by asotobu [ 21/Feb/15 ]
  • Point 1) I think JsonPointer could be a concrete class, it is much simpler and I think JsonPointer is generic enough to be provided from spec.
  • Point 2) First approach feels more natural to me and less restrictive in the sense that if devs want they can reuse a JsonPointer object to be used on other json objects, as noted maybe it is not useful but if we use the second one then we are not giving this freedom to developers, so I vote for the first one approach.

Only to considerations. The first one is that I think it could be better to no offer the constructor to devs but an static method. I said this because there is some rules a json pointer should follow (for example it should start with a '/') and also the first thing to do in a JsonPointer is evaluating and decoding and escaped char. It may look strange to do all this kind of stuff in a contructor, so I think that an static method that uses at the end a protected constructor may be better.
something like:

JsonPointer.compile("/a/b/c");

and/or

JsonPointer.valueOf("/a/b/c");

And the second point is should be the operation of get/add/remove/... be a part of the JsonPointer or a part of JsonObject.


JsonObject target = ... ; # the target Json text
JsonPointer pointer = new JsonPointer("a/b/c/d");
JsonValue value = pointer.get(target); # get the value referenced


or


JsonObject node = ... ; # the target Json text
JsonPointer pointer = new JsonPointer("a/b/c/d");
JsonValue value = node.get(pointer); # get the value referenced


I think both are valid and I have used both approaches one in Jackson and another one in one project that implemented Json Pointer before it was done in Jackson. For me and it is something personal I feel more comfortable of having operations on node instead of pointer. node.get(pointer). But it is something personal.

Comment by eugen.cepoi [ 22/Feb/15 ]

About last/tail/head, I think the names might be confusing. Peoples are used to them in the context of collections.

1) I agree, I would prefer to see json pointer implemented in the spec.

2) I also prefer option 1, but would like to see shortcut methods on json value. So people can do stuff like
JsonObject person = ...
person.at("address/street").asString()

How do you plan to handle incomplete paths (when some element in the path matches a null object)?

3) In case of mutation, I guess it would return a new instance of the root object on which the operation was applied?
As for 2) would be nice to have shortcuts at node level.

Comment by asotobu [ 22/Feb/15 ]

Well yes about giving a shortcut to users so you can do something like:

person.at("")

About incomplete paths, the RFC mention that there is no "explicit" paths as described here.

Look section 3 of RFC:

The ABNF syntax of a JSON Pointer is:

      json-pointer    = *( "/" reference-token )
      reference-token = *( unescaped / escaped )
      unescaped       = %x00-2E / %x30-7D / %x7F-10FFFF
         ; %x2F ('/') and %x7E ('~') are excluded from 'unescaped'
      escaped         = "~" ( "0" / "1" )
        ; representing '~' and '/', respectively

   It is an error condition if a JSON Pointer value does not conform to
   this syntax (see Section 7).

See that a pointer must start with '/'. So the example you mention above should be:

person.at("/address/street").asString()
Comment by eugen.cepoi [ 22/Feb/15 ]

Sorry I was unclear with "How do you plan to handle incomplete paths (when some element in the path matches a null object)?"

What I mean is not, how we handle a malformed path it self, but what will be the result of a json pointer expression when it encounters a null/missing value when evaluating the path? Note that null and missing could have different semantics. Would we return null? Optional? Or do something else?

Comment by asotobu [ 23/Feb/15 ]

I have read what Jackson-Utils JsonPointer and Jackson-databind JsonPointer solves this problem.

The first one returns a Null which as you said it may be incorrect because a value can be null as well.

The second one returns an special JsonObject which has a method called isMissingNode() that returns true in this case, meaning that there is no node there.

If you see javadoc JsonValue class has a NULL constant (JsonValue.NULL) https://json-processing-spec.java.net/nonav/releases/1.0/pfd-draft/javadocs/javax/json/JsonValue.html so what we can do is if returns a Java null means missing value and if it returns a JsonValue.NULL means the value Null.

Comment by kchung [ 23/Feb/15 ]

Yes, a Json pointer should begin with a '/'. My bad.

@asotobu I still like constructors better, because it is what most people would expect to use, and using static method to instantiate is less intuitive. There is nothing wrong with doing some processing in constructors. They can also be processed lazily.

Agreed that the operations starting from the target

person.get(pointer)

can be an alternate syntax. The short cut

person.get("/address/street")

also makes sense.

Here is the question. If we can use this short cut all the time, do we still need the class JsonPointer? Here are exaples for each of the 4 operations, just to see how they look like:

  JsonValue value = person.get("/address/street");
  JsonObject p = person.replace("/address/street". "Main Street"); // return the transformed person
  JsonObject p = person.remove("/phones/office");
  JsonObject p = person.add("/phones/mobil", "1234-567");

This way, the Json pointer will be completely removed from the API and is hidden in the implementation. Are there things that can be done with the two step approach (first create a JsonPointer, then apply to a target) but not with the short-cut?

Regarding the question about what should be returned when pointer specifies a non-existing entry, I think spec says an error should be raised. See http://tools.ietf.org/html/rfc6901#section-7

Comment by mitemitreski [ 23/Feb/15 ]

+1 on hiding the implementation details since really I don't see the need to know that part.

The shortcut also looks cleaner then the 2-step approach.

Just wondering how would we retrieve a collection of values from a collection of pointers applied on a object?

Or specifically how would a collection of operations would be applied to an object dynamically without knowing the order of the operations.
I am asking this since this would be needed in https://tools.ietf.org/html/rfc6902

Comment by asotobu [ 24/Feb/15 ]

@kchung and @mitemitreski From my point of view I would maintain the JsonPointer class in spec. Let me show why. A JsonPointer expression requires two things when it is created/used; the first one is escaping special characters and the second one (and this is an implementation details but all implementations I have worked of JsonPointer uses) it is that a json pointer expression is tokenized (or segmented) so in this way it is easier to implement the JsonPointer logic and also some operations required by Json Patch.

So let's suppose next scenario. I have 2000 request per minute which every request uses the same JsonPointer. This means that every minute you will execute 2000 times the escaping the same chunk of string and tokenize the expression. Although these are not an expensive operations it is unnecessary to execute them each time if the business logic is exactly the same.

So if spec allows to pass a JsonPointer then you will be able to create an static JsonPointer expression and call person.get(staticJsonPointer);

Of course we can still offer the string method which internally creates a new JsonPointer object, but at least we give to users the option to create only once the expression.

Comment by kchung [ 24/Feb/15 ]

@asotobu I don't get your argument about extra processing are need for a Json pointer. Isn't that true anyway? This is just an implementation detail, right?

I still don't know how often a Json pointer is reused. In your example, if the Json pointer is part of the request (maybe part of a Json patch). You'll need to do a comparison to determine if two pointers are same. To me it almost looks like you need a cache of JsonPointer. Note also the same optimization can be realized for the short-cut, because it can also look up the cache to reuse a previous JsonPointer.

I am not ruling out JsonPointer. I am just exploring what a JSON Pointer API we need to have. If we can make it simpler, the better.

We don't need to make decisions right now. We should have better ideas when we discuss/implement JSON patch.

Comment by asotobu [ 25/Feb/15 ]

Well let me explain with code. If you look here https://github.com/FasterXML/jackson-core/blob/master/src/main/java/com/fasterxml/jackson/core/JsonPointer.java#L114 you will see the process that requires creating a JsonPointer instance in Jackson. But I have used other Java libraries that implements JsonPointer and it is pretty similar. So as you can see there is some logic like escaping characters and creating segments for each sub-expression.

This is not good or bad it is simply an implementation details I know but let's suppose next code:

JsonObject[] persons = new JsonObject[5000];
//fill persons
JsonObject[] addresses = new JsonObject[5000]; 

for(int i=0;i<persons.length;i++) {
  addresses[i] = persons[i].get("/personalinformation/address");
}

In that code the escaping characters and segmentation of json pointer expression will be evaluated 5000 times.

But if I could do something like:

JsonObject[] persons = new JsonObject[5000];
//fill persons
JsonObject[] addresses = new JsonObject[5000]; 

JsonPointer addressPointer = new JsonPointer("/personalinformation/address");
for(int i=0;i<persons.length;i++) {
  addresses[i] = persons[i].get(addressPointer);
}

Then construction would be only one.

Because we don't know exactly if reusing a jsonpointer would be a requirement or not, I think that would have sense to implement both operations:

  • get("/a/b")
  • get(new JsonPointer("/a/b");

Basically because we are getting the best of both worlds with minimal requirement. The only requirement is that JsonPointer becomes an interface of the spec.

Comment by kchung [ 25/Feb/15 ]

OK. Let's keep both for now. I'll summarize what we have.

  1. JsonPointer is a non-abstract class with a constructor
    public JsonPointer(String pointer);
  2. The syntax for applying a JsonPointer to a JsonValue is
    JsonValue value;
    JsonPointer pointer = new JsonPointer("/a/b/c");
    value.get(pointer);
    
  3. The JsonValue operations should be overloaded to take the raw Json pointer String as a short cut
    value.get("/a/b/c");
    
  4. Add 4 operaions to JsonValue:
    1. getValue # there is already a get in JsonObject
    2. remove
    3. replace
    4. add
  5. An error is raised if the pointer reference a non-existing value.

Do we still need the alternate syntax

pointer.get(value)

?

I'll writeup the javadocs for JsonPointer and we can talk more.

Comment by asotobu [ 25/Feb/15 ]

jsonPointer could be an interface as well, so implementors Can decide how to implement json pointer resolution and how to create. WDYT ?





Generated at Fri Feb 27 12:39:04 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.