parallel processing framework in java

Question

I am new to this topic. I had decided to develop a parallel processing framework for cloud data processing applications in java for my project. the framework has to divide the given sequential java code and process that sub codes in different virtual machines in the cloud. the framework has to dynamically allocate and deallocate the resources according to the load. My problem is how to develop the framework.

Is there any libraries available to schedule the java code into different virtual machine in cloud? please inform me if anything is available.

Interestingly, I have been trying to build one as well. I already have one parallel processing framework written using plain simple shell scripting and python(fabric). I am not sure if that would help. Hadoop or storm is more of distributed processing and aggregation of real time/batch data. I am specifically looking for Java based parallel processing frameworks as there are some additional dependencies to have it in Java. At least, I am looking for Java modules that would help me build a truly embarrassingly parallel processing architecture. — user1652054
– user1652054, Commented Jan 13, 2015 at 10:16

Grooveek · Accepted Answer · 2011-12-14 09:29:49Z

4

Terracotta and Gridgain are excellent solutions. Those cited by yerlikayaoglu (Hadoop and hazelcast) are excellent too in their domain but they are all 4 very different and depend on the use case. That's for the map/reduce problem

An other one is the allocation/deallocation of virtual machines. It depends on your cloud provider and some other thing. You can have a look at jClouds

answered Dec 14, 2011 at 9:29

Grooveek

10.1k1 gold badge30 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

yerlikayaoglu · Accepted Answer · 2017-08-28 08:30:43Z

2

There are solutions such as Hazelcast, Hadoop etc. You can look this projects.

edited Aug 28, 2017 at 8:30

answered Dec 14, 2011 at 9:22

yerlikayaoglu

3443 silver badges11 bronze badges

Comments

David Rabinowitz · Accepted Answer · 2011-12-14 09:23:41Z

1

Have a look at Hadoop, a framework which allows basically the same thing, and supports automatic code deployment over the cluster.

answered Dec 14, 2011 at 9:23

David Rabinowitz

30.5k16 gold badges95 silver badges125 bronze badges

Comments

Mairbek Khadikov · Accepted Answer · 2011-12-14 11:24:58Z

0

If you want to do real time processing you can take a look at storm. Also Akka provides nice remote actors API for scala and java.

answered Dec 14, 2011 at 11:24

Mairbek Khadikov

8,1073 gold badges39 silver badges52 bronze badges

Collectives™ on Stack Overflow

parallel processing framework in java

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related