mapreduce - Can Amazon Auto Scaling Service work with Elastic Map Reduce Service?

Question

since amazon web service need to pay, so just wanna ask ppl who had worked on it before i jump into it, and confirm some knowledge about it.

Question one: In Amazon auto scaling service, it says can scale up and down instances. that does this mean?

does it mean changing the type of instance? or can start/stop more/less instance base on the condition define?

Question two: can the auto scaling framework work with map reduce?

for example, if i have a extreme case, i will have endless tasks, and the time to finish the task will be vary, and might be very computation intensive. so can i use auto scaling service to scale the map reduce?

Thanks

Answer 1

re. question one: "scale up and scale down" means they can add more instances as needed and the number of instances can then be reduced (i.e. virtual machine instances can be shut down) when they're no longer required.

re. question two: I would argue that auto-scaling does not really make sense for map reduce, at least if you're thinking about Hadoop: One of the main ideas behind Map/Reduce & Hadoop is that the data is split up among a lot of nodes and the nodes can then process the data that is close to them - ideally on their local disk or at least on another machine in close (physical) proximity. Starting a new instance would of course be possible, but that makes sense only as long as you do not have to copy the data you want to process over to the new instance. Hope this explanation makes sense to you... In any case, check out Amazon's Elastic Map Reduce offering, which does what you want, at least if I understood your question correctly.