Efficient way to find duplicate object in array

Question

I am working with angular4. I need to display a list with unique values.

When accessing an API I get an array, from that array I need to display only unique data. I will be accessing that api at certain time interval and I need to update list only if new data is present.

response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"},...]

listData = [];
for(let data of response) {
    let tempValue = {id: '', time: ''};
    let value = data.value.split('_')
    if (value.length==2) {
        if(value[value.length-1].length==2) {
            tempValue.id = value[0];
            tempValue.time = value[1];
        }
        let isPresent = false;
        if(this.listData.length>0){
            for(let value of this.listData){
                if(value.time===tempValue.time){
                    isPresent = true;
                }
            }
        }
        if(!isPresent) {
            this.listData.push(tempValue);
        }
    }

    }

the final listData

listData = [{id:'36af1784bec4', time: '566601140'},...]

The above function does give me a unique listData array. I tried using array.filter and new Set but could not achieve the desired result.

I would like to know if there is an efficient way to do this.

If your definition of new data is new id and value, then finding the existing of data may be more costly than updating the list regardless. — gurvinder372
– gurvinder372, Commented Jan 3, 2018 at 8:30
@bigless I tried converting to Set and tried using .has but it always returned true even when the listData was empty — Tony Roczz
– Tony Roczz, Commented Jan 3, 2018 at 8:34
@gurvinder372 the response that I am getting has id and value, but new listData is value.split('_') and value[0] is id and value[1] is time. listData = [{id: '36af1784bec4', time: '566601260'},...] — Tony Roczz
– Tony Roczz, Commented Jan 3, 2018 at 8:37
@Tony Roczz yea I was naive. its probably based on same reference... — bigless
– bigless, Commented Jan 3, 2018 at 8:38

A.D. · Accepted Answer · 2018-01-03 08:56:58Z

2

You can remove duplicate by the following code.

var obj = {};

for ( var i=0, len=response.length; i < len; i++ )
    obj[response[i]['id']] = response[i]; // if id Duplicate otherwise use value if the check on value.

response = new Array();
for ( var key in obj )
    response.push(obj[key]);

console.log(response); // listData

Edit:

and if you want to use a complete check of your prop then you can go for this one

function removeDuplicates(myArr, prop) {
    return myArr.filter((obj, pos, arr) => {
        return arr.map(mapObj => mapObj[prop]).indexOf(obj[prop]) === pos;
    });
}


console.log(removeDuplicates(response,['id','value'])); // for id or value use only 'id' or 'value' in second parameter.

edited Jan 3, 2018 at 8:56

answered Jan 3, 2018 at 8:33

A.D.

2,3722 gold badges17 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

vibhor1997a Over a year ago

He wants to find dup objects not just objects of dup ids

Nemani Over a year ago

not an optimised solution, you are looping twice, check my solution :)

Nemani Over a year ago

think about the input with thousands of unique property, will you loop that many times twice?

Aswin Ramesh · Accepted Answer · 2018-01-03 08:45:41Z

1

I think this will do

for(let data of response) {
    if(!this.listData.find((ldata) => data.value.substring(data.value.lastIndexOf('_')+1) === ldata.time)) {
        let tempValue = {id: '', time: ''};
        let value = data.value.split('_')
        if (value.length==2) {
            if(value[value.length-1].length==2) {
                tempValue.id = value[0];
                tempValue.time = value[1];
            }
            this.listData.push(tempValue);
        }
    }
}

edited Jan 3, 2018 at 8:45

answered Jan 3, 2018 at 8:36

Aswin Ramesh

1,6741 gold badge13 silver badges13 bronze badges

2 Comments

Tony Roczz Over a year ago

i would like to know why you are doing data.lastIndexOf('_')+1

Aswin Ramesh Over a year ago

sorry the check should be on the top, I thought it before splitting we can check for duplicate

Nemani · Accepted Answer · 2018-01-03 08:50:31Z

1

For an Optimised solution (loop only once i.e. O(n)) use object property to check if property already added, by this way you will avoid looping to find if property already added.

Below is the sample solution

let response = [{
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
    "value": "36af1784bec4_566601260"
  }, {
    "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
    "value": "36af1784bec4_566601140"
  }],
  listData = [],
  tempObj = {};
for (let data of response) {
    let value = data.value.split('_');
  if (!tempObj.hasOwnProperty(value[1])) {
    tempObj[data.value[1]] = "";
    listData.push({
      id: value[0],
      time: value[1]
    });
  }
}

console.log(listData);

Update: based on comments, checking only for duplicate time value

edited Jan 3, 2018 at 8:50

answered Jan 3, 2018 at 8:44

Nemani

7826 silver badges12 bronze badges

3 Comments

Tony Roczz Over a year ago

I dont need to find if the property is already added. i need to avoid adding duplicate values of time. which is got after the split operation

Nemani Over a year ago

logic would still remain same, currently i am checking complete value property

Nemani Over a year ago

by checking, if property is already added, you can avoid looping through the output to check if this value already added.

vibhor1997a · Accepted Answer · 2018-01-03 09:20:50Z

1

You can also do this by JSON.stringify the objects like following :

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}]

let o = response.reduce((acc, cv) => {
  if (!acc[JSON.stringify(cv)]) {
    acc[JSON.stringify(cv)] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => JSON.parse(x));
console.log(res);

Edit As user3297291 pointed out

Might be helpful to note that JSON.stringify({ a: 1, b: 2}) !== JSON.stringify({ b: 2, a: 1 })

So the above will fail in that case.

A more optimised solution would be

let response = [{
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}, {
  "id": "0DRDCFYGM2CAHAXYK96BPT9RHV",
  "value": "36af1784bec4_566601140"
}, {
  "id": "0DRDCH03DR51GGJGJNP80F7XZ8",
  "value": "36af1784bec4_566601260"
}];

let o = response.reduce((acc, cv) => {
  if (!acc[cv.value]) {
    acc[cv.value] = true; //something non-falsy
  }
  return acc;
}, {});

let res = Object.keys(o).map(x => {
  let t = x.split('_');
  return {
    id: t[0],
    time: t[1]
  };
});
console.log(res);

edited Jan 3, 2018 at 9:20

answered Jan 3, 2018 at 8:37

vibhor1997a

2,3862 gold badges19 silver badges40 bronze badges

2 Comments

user3297291 Over a year ago

Might be helpful to note that JSON.stringify({ a: 1, b: 2}) !== JSON.stringify({ b: 2, a: 1 })

vibhor1997a Over a year ago

I found a pretty efficient way for this problem would be updating my solution

Gabriel Bleu · Accepted Answer · 2018-01-03 09:29:31Z

0

You can do this with a map to collect the ids and then with a filter to remove the duplicates :

const response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.map(it => it.value.split('_')[0]).filter((it, i, arr) => i === arr.indexOf(it))

console.log(ids);

Or with a reduce :

const response= [{"id":"0DRDCH03DR51GGJGJNP80F7XZ8","value":"36af1784bec4_566601260"},{"id":"0DRDCFYGM2CAHAXYK96BPT9RHV","value":"36af1784bec4_566601140"}];

const ids = response.reduce((acc, it) => {
  const id = it.value.split('_')[0];
  return acc.includes(id) ? acc : [...acc, id];
}, []);

console.log(ids);

edited Jan 3, 2018 at 9:29

answered Jan 3, 2018 at 9:08

Gabriel Bleu

10.3k2 gold badges33 silver badges45 bronze badges

Comments

Ben Racicot · Accepted Answer · 2019-06-03 20:02:46Z

0

Recent versions of JavaScript allow for reliance on object structure to stay the same... Therefore we can now rely on stringifying and comparing the two strings:

JSON.stringify({id:0, name:foo}) === JSON.stringify({id:0, name:foo}) // true

So with an array of objects:

stringifying an object to test
looping through a collection of objects stringifying each
comparing the two

is helpful because you don't need to know key names. It just works on any object types...

arr = [{object}, {object2}, {etc...}]
test = JSON.stringify(obj);

for ( let i = 0, j = arr.length; i < j; i++ ) {
  let testAgainst = JSON.stringify(arr[i]);
  if (test === testAgainst) { 
    console.log('matches');
  } else {
    console.log('no match');
  }
}

Now you'd be matching two strings. If I'm missing something please comment.

edited Jun 3, 2019 at 20:02

answered Jun 3, 2019 at 19:11

Ben Racicot

6,03816 gold badges82 silver badges139 bronze badges

1 Comment

Ben Racicot Over a year ago

@Tony Roczz does this help as a modern, 2019 answer?

Collectives™ on Stack Overflow

Efficient way to find duplicate object in array

6 Answers 6

3 Comments

2 Comments

3 Comments

2 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

3 Comments

2 Comments

3 Comments

2 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related