![]() ![]() Shared nothing architectures are very scalable: because there are no shared resources, addition of nodes adds resources to the system and does not introduce further contention. It also refers to lack of shared data-in those frameworks, each node is processing a distinct subset of the data and there’s no need to manage access to shared data. The lack of shared resources refers to physical resources such as memory, disks, and CPUs-instead of using centralized storage, Hadoop’s processing framework uses the distributed HDFS storage. There are no shared resources that can become bottlenecks. In distributed systems, this is an architecture where each node is completely independent of other nodes in the system. If you want more detail, we’ll provide references for you to dig deeper into a particular tool.īefore we dive into a specifics of each framework, note one thing they all have in common: as much as possible, they attempt to implement a shared nothing architecture. Our goal in this chapter is to give you confidence that you are selecting the correct tool for your use case. Recommended resources for further information on the frameworkĪfter reading this chapter, you will gain an understanding of the various data processing options, but not deep expertise in any of them. These tools are designed to provide easier-to-use abstractions over lower-level frameworks such as MapReduce.įor each processing framework, we’ll provide: We’ll then move to higher-level abstractions such as Hive, Pig, Crunch, and Cascading. This includes the well-established MapReduce framework, as well as newer options such as data flow engines like Spark. We will open the chapter by reviewing the main execution engines-the frameworks directly responsible for executing data processing tasks on Hadoop clusters. Get a free trial today and find answers on the fly, or master something new and useful. Join the O'Reilly online learning platform. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |