Add DSL-JSON to bench. #1
Conversation
Requires Mono/.NET for compilation. Added @CompiledJson to MeasurementPOJO (alternative is to create another POJO referencing MeasurementPOJO) Changed empty ctor on MeasurementRecord to public.
|
@zapov Thank you for the contribution! Code looks clean enough; the only question I had was about |
|
@zapov Actually one question: how is Mono installed? (I have never developed Dotnet, so am not very familiar with tooling). That may be bit problematic, depending. |
|
On Linux I usually do sudo apt-get install mono-complete Btw. There is 0.9.6 which fixes some regresion so you should switch to that version |
|
@zapov Ok, I finally went ahed and (re)installed mono. But maven does not seem to find the runtime, and I am not sure where it is supposed to be installed (there is no |
|
Hm... you are on Mac? |
|
Hmmh. Yes, on mac (el capitano), but no I don't see it. Maybe I'll try running installer again... did not think it failed but I guess it must have had to. |
|
No. Installer claims it succeeds, but I can't find anything related it seems. |
|
I did saw something related. Someone claimed that reinstall worked after it created /usr/local/bin manually But maybe it's good enough if you export path: |
|
Ah! Progress. I can install mono, but build now fails in an interesting new way: which seems odd... how can source manage to do this for what should be a quite stable javac version. |
|
... and yet it works with second attempt, ensuring |
|
It looks like a Maven bug with annotation processor: http://stackoverflow.com/questions/36408706/java-lang-illegalstateexception-endpostable-already-set I can reproduce it on my 1.8.66 with a maven compile command It's disappointing to hear so many issues with setup ;( |
|
@zapov no sweat, not your fault. But as I said, problem occurred just once and now build succeeds. I hope I can reproduce it on the other machine (where I usually run my tests) as well. |
|
And things work just fine on my other machine (with first try). Next I'll need to modify README a bit, to move result info on project wiki (or more likely, description, and further links to separate test runs or something). |
|
Couple of suggestions: Emphasise stream instead of string results. Your readme had all results in strings. |
|
Yes agreed, not sure why I added results for String variant (I think use of Strings for tests is an anti-pattern here). Jackson also has significant optimizations for byte case; most others libs do not as you say (even fast-json and Boon only uses Readers/String). I included partial/initial results under Explanation of libraries, features would be useful, eventually. As to jackson-jr I haven't had time to look any more into it; once I run map tests I'll see how numbers look. Handling by Jackson and jackson-jr should be very similar for this particular case, and so should the numbers be (jr has little bit simpler set up, but shouldn't matter too much). |
|
On my Win laptop using Java 8.66 |
|
Oh. That would definitely be significant -- I don't see anything like that. What are your settings for |
|
I ran java -Xmx256m -jar target/microbenchmarks.jar -wi 4 -i 5 -f 5 "._DZoneReadPojo._read1kFromStream" Our read results look the same? |
|
Ah! I was looking at error measurements, and NOT on throughput numbers -- so looked liked jackson-jr had super high throughput. However, having said that, your error numbers look exceptionally high so something is going on. |
|
Well, I'm doing something else also ;) ... and I did reduce the number of runs from 9 to 5. I'm on laptop away from home... so can't give you better numbers ATM :( |
|
@zapov that's fine, all I'm saying is that error margins are so big as to suggest maybe numbers might not be stable. But, since they do look similar that's neither here nor there. So my initial reading ("jackson-jr 3x as fast as Jackson , wth?") was wrong. |
|
Btw, impressive write performance by dsl-json, maybe there's something I could learn from it! |
|
Did you try it with more runs and forks? Thanks. Hopefully there is ;) |
|
Doh. Found out the problem wrt Seeing odd spike for Boon just for 1k case (but not 10 or 100k). Tried to add a call to |
|
Map size can be faked. HashCode cant. Try with that |
|
Yes, but I really don't want hashCode() calculation for value Strings as that has major performance impact. Similarly traversal of contents is (if I recall correctly) somewhat garbage inducing... so all in all, it would seem rather costly. Then again I guess it should be easy enough to see how much overhead we are talking about; if it's modest (<10%) perhaps it'd be worth it. On the other hand, it seems like only Boon might be using lazy construction at this point, so there isn't much to resolve I think. |
|
As stated earlier I'm more or less ok with the current setup. I commented that you should try consuming hashCode just to get a feeling of the performance difference. When I tried to resolve this I also tried consuming size and various other properties, but most of that can be faked (well, even hashCode can be faked if you want to provide a different number that Java HashMap). The only thing which worries me a little is that it's not an issue with hashCode, but rather that some internal Java optimizations which does a dead code elimination for the unused objects. |
|
FWTW calling Perhaps it's some other optimization that matters; maybe just simple thing like first reading all elements before allocating I agree that it's good to be alert to artifacts that are due to some edge case. So let's keep on discussing oddities we find. |
|
My results with hashCode instead of size Ok, I'm think I don't have any issues with this anymore. I had issues because I recalled a lot of libraries having around the same time for w/wo hashcode. It seems that's not the case. I supose Boon has sweet spot for this size because it doesn't cause too much GC (which is the case for 100k) and it's not saturated by processing overhead (for 10). |
|
Ouch. I noticed that previous run was done with Java 8.77 32bit ;( Here is 8.66 64bit The conclusion is the same (Boon probably doing lazy map, but whatever)... |
|
Ok sounds good. I must have messed up my hashCode() run, and it not getting invoked -- I can see its effects there. Now; if you wanted to do runs on your system(s) I wouldn't mind adding them under results; the more the merrier. Just need to think of best way to include all pertinent info. I tried adding some on one result file I have, although obviously some sort of automation would be better. Regardless these would be raw data files, and it'd be good to have some sort of summary/summaries. Or, if anyone wants to write an article or blog post, I guess it's easier to pick and choose. Another thing we could consider would be mailing list perhaps. I would be happy to share ownership of this project if that makes sense -- adding new tests and all. Sort of like JSON-specific derivative of If so, I wonder if any other somewhat neutral github org might like to host this project? |
|
I don't think there is a need for my results (they are too similar to yours). I think a good starting point is wrapping this up and getting some interest either through DZone or whatever. It will be easier to find contributors which are able to create nice graphs in html (not my area of expertise). I think it's fine to stay within Github (there is also github.io although raw wiki might be enough). Yeah, distinctive data types (all primitives, non ASCII,... ) is a good roadmap. I also have my own JSON bench: https://github.com/ngs-doo/json-benchmark but it does some things differently. Personally I think people are much more interested in Android these days... |
|
Having test runs on Android, using a reasonable benchmark would be great. Most test I have seen have suffered from one or more of basic mistakes. Not sure if there's something similar to |
|
I think Android is a different use case, since people are often interested how well does a serialization works the first time, not after warm-up. For example, one of the apps my client is currently writing is doing one large deserialization at startup. Those kind of use cases are the reason various compile time databindings for Jackson exists. Anyway... are you anywhere near the state of giving DZone their article back? Eg, are you missing only graph in Wiki or plan on doing a lot more work? btw. early this year I investigated report for performance issue compared to Jackson which turned out was due to Jackson doing triplets for processing ints, while DSL-JSON was doing diplets. The idea that it's faster to process number in chunks of three is the only thing I took from Jackson :/ I'm sorry now that I didn't include that info in my commit message, but any other code similarity is purely non-related to Jackson codebase. |
|
Yes, one-off usage is sort of interesting in context of Android. Not sure how that should be tackled; I assume measurements really need to be run on Dalvik for that part. Would be interested in those; I think changes themselves are fine. Not sure what to do with DZone; I guess I could blog about that. I don't know the authors, although I guess I could leave a comment or try emailing. Do you know people involved? As to processing: no worries, regardless of whether inspiration was gained. I sometimes have a look around, try to learn; I try to give credit where applicable but to me Open Source implies that ideas should be allowed to cross-pollinate. Sometimes there is convergent evolution, we are after all working on same/similar problems, using similar techniques. :) I have gotten some ideas from |
|
I think this: https://www.infoq.com/news/2014/05/jackson-founder-responds was my rationale on how to contact the DZone, but I see now that thats InfoQ :D If you are interested in overhead of name hashing optimization, you can "disable" that optimization with this: https://github.com/ngs-doo/dsl-json/blob/master/examples/Maven/src/main/java/com/dslplatform/maven/Example.java#L20 I actually never tried that :D but I would assume it's not that much. My takeaway from writing fast code in .NET and JVM is to avoid creating garbage and minimize unexpected branching. Everything else comes second :/ |
Requires Mono/.NET for compilation.
Added @CompiledJson to MeasurementPOJO (alternative is to create another POJO referencing MeasurementPOJO)
Changed empty ctor on MeasurementRecord to public.
Results: