Working with very large Javascript array or objects

Question

Currently I'm working on a script where I've got to store more than 1,250,000 objects. I'm using the push() function in a jQuery each() loop like this:

words.push({
  "page": pagenumber,
  "content": word,
  "hpos": $(this).attr("HPOS"),
  "vpos": $(this).attr("VPOS"),
  "width": $(this).attr("WIDTH"),
  "height": $(this).attr("HEIGHT")
});

In Chrome it goes quit fast, between 30 and 40 seconds, but in Internet Explorer it can take up to 360 seconds.

It's for a project where old newspapers are loaded and you can search the text from those newspapers. The newspapers are in a directory and are loaded dynamically. In this test I'm doing I'm using newspapers from october 1926, containing 308 pages and over 1.250.000 words.

Is there a better/faster way to achieve this?

Can you give us some background as to why you're creating an array of 1.25 million objects? — ahren
– ahren, Commented Apr 29, 2013 at 8:33
You should better think about, if you really need 1m+ entries in a JavaScript object in a browser. — Sirko
– Sirko, Commented Apr 29, 2013 at 8:34
@zerkms: There's no concept of pre-allocating with JavaScripts untyped arrays, because they're not really arrays at all. — T.J. Crowder
– T.J. Crowder, Commented Apr 29, 2013 at 8:34
I can't ever think of a reason to add so many items to a client side array. I think you need to explain your rationale, and give us more insight in to the goals of your project. Its likely you need to rethink your design. — Neil
– Neil, Commented Apr 29, 2013 at 8:36
Also, regarding the edit, loading and indexing the newspapers is definitely a job for the server. — JJJ
– JJJ, Commented Apr 29, 2013 at 8:40

T.J. Crowder · Accepted Answer · 2013-04-29 08:42:53Z

6

Is there a better/faster way to achieve this?

Yes: Do it on a server, not in the browser. This also has the advantage that you can do it once and reuse the information.

But assuming for some reason that's not possible:

The first thing you can do is stop making several million unnecessary function calls by only doing $(this) once per loop:

.....each(function () {
    var $this = $(this);
    words.push({
        "page": pagenumber,
        "content": word,
        "hpos": $this.attr("HPOS"),
        "vpos": $this.attr("VPOS"),
        "width": $this.attr("WIDTH"),
        "height": $this.attr("HEIGHT")
    });
});

Normally repeatedly doing that isn't a big deal (though I'd avoid it anyway), but if you're doing this one and a quarter million times...

And if all of those attributes really are attributes, you can avoid the call entirely by cutting jQuery out of the middle:

.....each(function () {
    words.push({
        "page": pagenumber,
        "content": word,
        "hpos": this.getAttribute("HPOS"),
        "vpos": this.getAttribute("VPOS"),
        "width": this.getAttribute("WIDTH"),
        "height": this.getAttribute("HEIGHT")
    });
});

jQuery's attr function is great and smooths over all sorts of cross-browser hassles with certain attributes, but I don't think any of those four needs special handling, not even on IE, so you can just use DOM's getAttribute directly.

The next thing is that some JavaScript engines execute push more slowly than assigning to the end of the array. These two statements do the same thing:

myarray.push(entry);
// and
myarray[myarray.length] = entry;

But other engines process push as fast or faster than the assignment (it is, after all, a ripe target for optimization). So you might look at whether IE does push more slowly and, if so, switch to using assignment instead.

edited Apr 29, 2013 at 8:42

answered Apr 29, 2013 at 8:37

T.J. Crowder

1.1m201 gold badges2k silver badges2k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Wezy Over a year ago

Thanks, it has speeded thing up a little, but I think I'll going to give node.js a try. Trying to let the server do all the hard work and not the browser.

probablyup Over a year ago

@T.J. Crowder Doesn't myarray[myarray.length] = entry; create a sparse array? Might not be a problem in this case, but it can change the behavior of the array for certain applications.

T.J. Crowder Over a year ago

@ultraviol3tlux: No, it doesn't. It just adds an entry to the end of the array, like push does. Assigning a value to length to make it larger (var a = []; a.length = 10;) creates a sparse array, but adding to the end of an array doesn't.

Collectives™ on Stack Overflow

Working with very large Javascript array or objects

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related