JSON.NET Ignore Nulls when parsing collection

Question

I'm extracting Survey Responses from our survey provider via an API. I'm not using classes for this to de-serialize, it's just key / value pairs.

I'm reading an array of Responses from a file. Each Response contains many 'Response Items' we'll call them, as below:

[
    {
        "response_id":"1234",
        "hRz5aMmGPf": null,
        "6UnnAZSEBT": null,
        "nGS1cyvLwK": "Red"
    },
    {
        "response_id":"1235",
        "hRz5aMmGPf": "John Smith",
        "6UnnAZSEBT": null,
        "nGS1cyvLwK": "Blue"
    },
    {
        "response_id":"1236",
        "hRz5aMmGPf": "Jane Doe",
        "6UnnAZSEBT": "Yes",
        "nGS1cyvLwK": null
    }
]

For the purpose of this exercise, I'm reading the JSON from file as follows:

List<JToken> responseobjs = new List<JToken>();
JObject o = JObject.Parse(fcontents);
responseobjs.AddRange(o["results"].Children())

As per the sample data, there's a lot nulls in the source data which I'd like to eliminate in the quickest way possible. I've read about 'NullValueHandling' but this seems to only be applicable if I de-serialize into a Class / Object, which isn't possible with the distinct field ID's being returned by the feed.

Is it possible for me to take the Children() above but skip the nulls?

Right now I'm iterating over the Responses and then Response Items to remove nulls and it's taking a long time to transform.

foreach (JToken obj in responseobjs)
{

    foreach (JProperty rprop in obj.Where(x=> x.HasValues==true).ToList())
    {
        if (rprop.Value.ToString().Trim() == "")
            continue;

        ..Continue parsing here...
    }
}

Why do you want to remove the nulls. Suppose I wrote a code and returned all properties that are not null in a dictionary. Then you would have to check if a property is in that dictinary or not. What is different than checking null. — L.B
– L.B, Commented Feb 28, 2014 at 22:02
I have to parse each property returned within each response. It's taking a long time to do that when the source file includes the nulls. I cannot change the fact it includes nulls, but it would save me time if it didn't include them in the first place due to the sheer volume of data. — Tom
– Tom, Commented Feb 28, 2014 at 22:05
Tom Maybe I was not clear(because of my bad english) but read the comment again. What would change If I posted a solution giving you a dictionary (a dict per object) with not null properties. — L.B
– L.B, Commented Feb 28, 2014 at 22:07
I don't think it would change anything, because I'd still have to iterate over the JProperties including the nulls, like I am already doing as above. — Tom
– Tom, Commented Feb 28, 2014 at 22:10
I incorrectly thought that the nulls were my bottleneck in my parsing. While I think there would be benefit removing these nulls still and I'm interested for feedback, the bottleneck was a DB call being duplicated which I had overlooked. — Tom
– Tom, Commented Feb 28, 2014 at 22:31

Brian Rogers · Accepted Answer · 2014-03-01 22:03:08Z

If you've already parsed the JSON, the only way to remove the null values is to examine each property individually like you're already doing. An alternative approach is to remove the nulls as the JSON is being read, so you don't have to filter them out later. Json.Net does not provide this functionality out of the box, but it is possible to write a helper method that works directly with a JsonReader to build a JToken hierarchy with the null values excluded:

static JToken DeserializeExcludingNulls(string json)
{
    using (JsonTextReader reader = new JsonTextReader(new StringReader(json)))
    {
        return DeserializeExcludingNulls(reader);
    }
}

static JToken DeserializeExcludingNulls(JsonReader reader)
{
    if (reader.TokenType == JsonToken.None)
    {
        reader.Read();
    }

    if (reader.TokenType == JsonToken.StartArray)
    {
        reader.Read();
        JArray array = new JArray();
        while (reader.TokenType != JsonToken.EndArray)
        {
            JToken token = DeserializeExcludingNulls(reader);
            if (!IsEmpty(token))
            {
                array.Add(token);
            }
            reader.Read();
        }
        return array;
    }

    if (reader.TokenType == JsonToken.StartObject)
    {
        reader.Read();
        JObject obj = new JObject();
        while (reader.TokenType != JsonToken.EndObject)
        {
            string propName = (string)reader.Value;
            reader.Read();
            JToken token = DeserializeExcludingNulls(reader);
            if (!IsEmpty(token))
            {
                obj.Add(propName, token);
            }
            reader.Read();
        }
        return obj;
    }

    return new JValue(reader.Value);
}

static bool IsEmpty(JToken token)
{
    return (token.Type == JTokenType.Null);
}

Here is a demo:

string json = @"
[
    {
        ""response_id"":""1234"",
        ""hRz5aMmGPf"": null,
        ""6UnnAZSEBT"": null,
        ""nGS1cyvLwK"": ""Red"",
    },
    {
        ""response_id"":""1235"",
        ""hRz5aMmGPf"": ""John Smith"",
        ""6UnnAZSEBT"": null,
        ""nGS1cyvLwK"": ""Blue""
    },
    {
        ""response_id"":""1236"",
        ""hRz5aMmGPf"": ""Jane Doe"",
        ""6UnnAZSEBT"": ""Yes"",
        ""nGS1cyvLwK"": null
    }
]";

JArray array = (JArray)DeserializeExcludingNulls(json);

foreach (JObject obj in array)
{
    foreach (JProperty prop in obj.Properties())
    {
        Console.WriteLine(prop.Name + ": " + prop.Value);
    }
    Console.WriteLine();
}

Output:

response_id: 1234
nGS1cyvLwK: Red

response_id: 1235
hRz5aMmGPf: John Smith
nGS1cyvLwK: Blue

response_id: 1236
hRz5aMmGPf: Jane Doe
6UnnAZSEBT: Yes

If you also want to exclude empty strings, you can change the IsEmpty method accordingly:

static bool IsEmpty(JToken token)
{
    return (token.Type == JTokenType.Null) ||
           (token.Type == JTokenType.String && token.ToString().Trim() == "");
}

Corey Alix · Accepted Answer · 2018-12-30 22:08:15Z

Feels lengthy, but I'm using it to filter out nulls from a jsondiffpatch.net diff() so my unit tests pass (my models do not serialize null values but the raw json may contain null values).

public static class JTokenExtensions
{
    public static JToken RemoveNulls(this JToken node)
    {
        var children = node.Children().ToArray();
        var nonTrivialChildren = children.Select(c => RemoveNulls(c)).Any(v => v != null);
        if (nonTrivialChildren)
            return node;

        // once trivial children are removed, values will be different
        var values = node.Values().ToArray();
        var nonTrivialValues = values.Any(v => v != null);
        if (nonTrivialValues)
            return node;

        // the parent needs to be removed instead
        switch (node.Type)
        {
            case JTokenType.Boolean:
            case JTokenType.Date:
            case JTokenType.Float:
            case JTokenType.Integer:
            case JTokenType.String:
                return node;
            case JTokenType.Array:
            case JTokenType.Object:
                return null;
            case JTokenType.Null:
                break;
        }
        node.Remove();
        return null;
    }
}

Collectives™ on Stack Overflow

JSON.NET Ignore Nulls when parsing collection

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related