Book Review: Hadoop Beginner's Guide 57
First time accepted submitter sagecreek writes "Hadoop is an open-source, Java-based framework for large-scale data processing.
Typically, it runs on big clusters of computers working together to crunch large
chunks of data. You also can run Hadoop in "single-cluster mode" on a
Linux machine, Windows PC or Mac, to learn the technology or do testing and
debugging. The Hadoop framework, however, is not quickly mastered. Apache's
Hadoop wiki cautions: "If you do not know about classpaths,
how to compile and debug Java code, step back from Hadoop and learn a bit more about Java before
proceeding." But if you are reasonably comfortable with Java, the well-written
Hadoop Beginner's Guide by Garry Turkington can help you start mastering this
rising star in the Big Data constellation." Read below for the rest of Si's review.
Dr. Turkington is vice president of data engineering and lead architect for London-based Improve Digital. He holds a doctorate
in computer science from Queens University of Belfast in Northern Ireland. His
Hadoop Beginner's Guide provides an effective overview of Hadoop and hands-on guidance in how to use
it locally, in distributed hardware clusters, and out in the cloud.
| Hadoop Beginner's Guide | |
| author | Garry Turkington |
| pages | 374 |
| publisher | Packt Publishing |
| rating | 9/10 |
| reviewer | Si Dunn |
| ISBN | 9781849517300 |
| summary | Explains and shows how to use Hadoop software in Big Data settings. |
Packt Publishing provided a review copy of the book. I have reviewed one other Packt book previously.
Much of the first chapter is devoted to "exploring the trends that led to Hadoop's creation and its enormous success." This includes brief discussions of Big Data, cloud computing, Amazon Web Services, and the differences between "scale-up" (using increasingly larger computers as data needs grow) and "scale-out" (spreading the data processing onto more and more machines as demand expands).
Dr. Turkington writes, "One of the most confusing aspects of Hadoop to a newcomer is its various components, projects, sub-projects, and their interrelationships."
His 374-page book emphasizes three major aspects of Hadoop: (1) its common projects; (2) the Hadoop Distributed File System (HDFS); and (3) MapReduce.
He explains, "Common projects comprise a set of libraries and tools that help the Hadoop product work in the real world."
The HDFS, meanwhile, "is a filesystem unlike most you may have encountered before." As a distributed filesystem, it can spread data storage across many nodes. "[I]t stores files in blocks typically at least 64 MB in size, much larger than the 4-32 KB seen in most filesystems." The book briefly describes several features, strengths, weaknesses, and other aspects of HDFS.
Finally, MapReduce is a well-known programming model for processing large data sets. Typically, MapReduce is used with clusters of computers that perform distributed computing. In the "Map" portion of the process, a single problem is split into many subtasks that are then assigned by a master computer to individual computers known as nodes (and there can be sub-nodes). During the "Reduce" part of the task, the master computer gathers up the processed data from the nodes, combines it and outputs a response to the problem that was posed to be solved. (MapReduce libraries are now available for many different computer languages, including Hadoop.)
"The developer focuses on expressing the transformation between source and result data sets, and the Hadoop framework manages all aspects of job execution, parallelization, and coordination," Dr. Turkington notes. He calls this "possibly the most important aspect of Hadoop. The platform takes responsibility for every aspect of executing the processing across the data. After the user defines the key criteria for the job, everything else becomes the responsibility of the system."
In this 11-chapter book, the first two chapters introduce Hadoop and explain how to install and run the software.
Three chapters are devoted to learning to work with MapReduce, from beginner to advanced levels. And the author stresses: "In the book, we will be learning how to write MapReduce programs to do some serious data crunching and how to run them on both locally managed and AWS-hosted Hadoop clusters." ["AWS" is "Amazon Web Services."]
Chapter 6, titled "When Things Break" zeroes in on Hadoop's "resilience to failure and an ability to survive failures when they do happen.much of the architecture and design of Hadoop is predicated on executing in an environment where failures are both frequent and expected." But node failures and numerous other problems still can arise, so the reader is given an overview of potential difficulties and how to handle them.
The next chapter, "Keeping Things Running," lays out what must be done to properly maintain a Hadoop cluster and keep it tuned and ready to crunch data.
Three of the remaining chapters show how Hadoop can be used elsewhere within an organization's systems and infrastructure, by personnel who are not trained to write MapReduce programs.
Chapter 8, for example, provides "A Relational View on Data with Hive." What Hive provides is "a data warehouse that uses MapReduce to analyze data stored on HDFS," Dr. Turkington notes. "In particular, it provides a query language called HiveQL that closely resembles the common Structured Query Language (SQL) standard."
Using Hive as an interface to Hadoop "not only accelerates the time required to produce results from data analysis, it significantly broadens who can use Hadoop and MapReduce. Instead of requiring software development skills, anyone with a familiarity with SQL can use Hive," the author states.
But, as Chapter 9 makes clear, Hive is not a relational database, and it doesn't fully implement SQL. So the text and code examples in Chapter 9 illustrate (1) how to set up MySQL to work with Hadoop and (2) how to use Sqoop to transfer bulk data between Hadoop and MySQL.
Chapter 10 shows how to set up and run Flume NG. This is a distributed service that collects, aggregates, and moves large amounts of log data from applications to Hadoop's HDFS.
The book's final chapter, "Where to Go Next," helps the newcomer see what else is available beyond the Hadoop core product. "There are," Dr. Turkington emphasizes, "a plethora of related projects and tools that build upon Hadoop and provide specific functionality or alternative approaches to existing ideas." He provides a quick tour of several of the projects and tools.
A key strength of this beginner's guide is in how its contents are structured and delivered. Four important headings appear repeatedly in most chapters. The "Time for action" heading singles out step-by-step instructions for performing a particular action. The "What just happened?" heading highlights explanations of "the working of tasks or instructions that you have just completed." The "Pop quiz" heading, meanwhile, is followed by short, multiple-choice questions that help you gauge your understanding. And the "Have a go hero" heading introduces paragraphs that "set practical challenges and give you ideas for experimenting with what you have learned."
Hadoop can be downloaded free from the Apache Software Foundation's Hadoop website.
Dr. Turkington's book does a good job of describing how to get Hadoop running on Ubuntu and other Linux distributions. But while he assures that "Hadoop does run well on other systems," he notes in his text: "Windows is supported only as a development platform, and Mac OS X is not formally supported at all." He refers users to Apache's Hadoop FAQ wiki for more information. Unfortunately, few details are offered there. So web searches become the best option for finding how-to instructions for Windows and Macs.
Running Hadoop on a Windows PC typically involves installing Cygwin and openSSH, so you can simulate using a Linux PC. But other choices can be found via sites such as Hadoop Wizard and Hadoop on Windows with Eclipse".
To install Hadoop on a Mac running OS X Mountain Lion, you will need to search for websites that offer how-to tips. Here is one example.
There are other ways get access to Hadoop on a single computer, using other operating systems or virtual machines. Again, web searches are necessary. The Cloudera Enterprise Free product is one virtual-machine option to consider.
Once you get past the hurdle of installing and running Hadoop, Garry Turkington's well-written, well-structured Hadoop Beginner's Guide can start you moving down the lengthy path to becoming an expert user.
You will have the opportunity, the book's tagline states, to "[l]earn how to crunch big data to extract meaning from the data avalanche."
Si Dunn is an author, screenwriter, and technology book reviewer.
You can purchase Hadoop Beginner's Guide from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Confused! (Score:1)
Wasn't Java being frowned upon for being insecure and so on? But at the same time, Hadoop (Java-based) is praised?
What's the right path in this "Don't use Java!"/"DO use Java!" debacle?
Re:Confused! (Score:5, Insightful)
Re: (Score:2)
I never worked with Java, so honestly I didn't know. All I hear left and right is "Java sux because vulnerabilities". So I was wondering.
But judging from the smug answers below, I made the impardonable mistake of not knowing EVERYTHING. Oh well.
Re: (Score:2)
...which is a good reason for asking questions. It it NOT a good reason for others to be dicks about it.
Re: (Score:2)
Why is this insightful? Shouldn't this comment have a low moderation value. A high moderation leads people into wasting their time look at this post and into a parent post who needed some help. Why should anyone else care.
Of course this is not the fault of the parent or the poster. Is moderation really this crappy? I guess moderators want to moderate something in this article and there just aren't any good posts.
Re: (Score:2)
Could you elaborate, or provide sources?
Hadoop provides an answer to very specific questions involving large amounts of data, and isn't intended to be a database or other storage mechanism.
You can use python and c++ for haddop programs (Score:3)
And back in the 80's BT used PL/1 (plus a bit of Fortran 77) to do Map reduce as the core of a billing system for the dialcom systems -even manged ro sell it to other telcos as a replacement for the Dialcom product.
Re: (Score:2)
Well, I think [slashdot.org] that great amount of confusion results from the way Java is marketed. For a website that has a larger than normal number of technical people, /. as an aggregate still displays fundamental misunderstanding of what Java is and what it is not.
For example Java is not Javascript. A browser sandbox that runs Java plugin has nothing to do with server side Java applications, that for example can run in servlet containers and servers like Tomcat Apache, Jetty, iPlanet, Resin (by Caucho), Enhydra, and
Re: (Score:2)
For people who apparently pride themselves being 'above marketing', developers somehow fell victims to various marketing around Java and didn't bother to check for themselves what the hell all this is about.
As a developer who never had many reasons to do very much with (actual) Java, I have to say I lost of respect for the platform when Oracle started bundling the runtime with the Ask toolbar on Windows. Because nothing says "serious computing platform" like having to make sure it doesn't install a toolbar on every update. /rant
Re: (Score:1)
As a developer who never had many reasons to do very much with (actual) Java, I have to say I lost of respect for the platform when Oracle started bundling the runtime with the Ask toolbar on Windows.
- I don't know, as a thinking human I have to say that what Oracle does with Java browser plugin reflects poorly on Oracle and has nothing to do with Java. As I said, the browser plugin is irrelevant for majority of what Java is actually used for.
Personally I don't even have Java browser plugin on any of my machines because I don't have a use for it. At the same time I have developed plenty Java applications (as in Java language running on a JVM that resides on a server and has some form of application co
Re: (Score:2)
As an FYI, it's not the browser plugin that has the opt-out toolbar installation, it's the JVM itself. Do I think it's truly a sign of Java's strength/weakness as a platform? No, but as you say, it's horrible marketing.
Re: (Score:1)
You mean the JRE that is part of the plugin installation? It cannot be the JRE itself, it must the the installer of the plugin. I don't install JVM this way or Java plugin (at all), I just download the necessary installation package for the JVM/SDK as needed. If I want Oracle SDK I get it from here [oracle.com] and when it installs, it does not install any 'ask bar' or anything like that, so I am not even clear as to how people get these things. I download the SDK or JVM for a Linux distro, which is either a binary
Re: (Score:2)
No, I mean the JRE itself. (Remember, I'm talking about Windows here.) Granted, this is not the SDK I'm talking about, just the standard JRE installed in Windows, not the browser, but I just updated it on this machine, and I had to uncheck [tenthcave.com] the option to install the Ask toolbar. (link is not my blog, btw.)
I'm fully aware of the distinctions between Java, the browser plugins, Javascript, etc., and I realize that the toolbar is not part of the install on 'real' OSs. But the Windows installer is what most us
Re: (Score:1)
Well, I went looking at this and it's not the JRE that does this, it's the Oracle installer for Java. JRE is not the installer, it's the run time environment.
Here is one of these stories about this issue [java.net].
But yes, on Linux or Unix I don't get any of this nonsense, it's just a tarball that I untar into a directory, set the path and run the JVM for example. The binary installers for Linux or Unix are simple shell scripts with the same tarball basically as part of the script, it doesn't do any of this stuff.
As
Re: (Score:2)
Re: (Score:1)
Isn't the devil in the details, while everything else is just marketing?
Re: (Score:2)
Re: (Score:2)
Small numbers for Big Data? (Score:3, Interesting)
How many people on the planet actually manage "Big Data"? Isn't that the kind of thing that happens as a happy accident when your humble web site becomes the next FaceBubbleSpace? You can't plan for that.
Sure, there are other places where it happens--large corporations, governments, maybe some academic studies.
Really though, I have a hard time imagining that there are really a lot of people who deal with BD. Does anybody have numbers on it? What's the definition, anyway? Is Slashdot's archive BD?
Re: (Score:2)
Re: (Score:2)
My takeaway from the article is that the definition of BD is a moving target as the capability of hardware grows.
If "traditional" approaches to data fail to scale, why not just start with BD methods in the first place? In that case, BD is a meaningless term as it simply becomes "a better way of handling data". OTOH, if there's a high technical hurdle between "traditional" and BD methods, then you have an incentive to stay traditional until you're confronted with the problem. Therein lies the crux of my
Re: (Score:1)
There's an entire field dedicated to Data Warehousing who's entire focus is Big Data. Large companies with auditing requirements have to keep mountains of historical data. Business Intelligence is largely based on analyzing huge segments of data.
As storage gets cheaper and options for going through large amounts of data become more widely available, companies invariably store more data. The biggest difference is that while you previously would have simply chosen not to track certain types of data in your
Re:Small numbers for Big Data? (Score:4, Insightful)
Big Data is however big you need it to be. It's not a certain size, or speed, or software, but rather a philosophy.
Simply put, Big Data methodologies are to gather all the data that can be gathered, and store it on a nice cheap database, without concern for storage efficiency. When a question arises, analyse the relevant data for an answer. This is in contrast to more traditional methods, where data is gathered only to solve expected questions.
Slashdot's archives were not generated from a Big Data approach. They store only comments and a few sparse details (to my knowledge). However, they can still be used in a Big Data system to some effect, if they happen to store the information that's needed (such as IP address, timestamps, and keywords, if the question is to track political opinions by geography over time).
It's not what you store or how you store it, but how you decide what to store. What makes Big Data approaches useful is that they store everything from the beginning, so as business needs change, the data from the past is likely just as useful as new data. What makes Big Data difficult is that the databases must be properly capable of storing all the gathered data as fast as it arrives, and must do so cheaply.
Re: (Score:2)
There's actually more of a need for it than you'd think at first glance. Any business that handles upwards of a few million records of some sort of data, and then needs to *do* something with that data, could potentially benefit from this.
I've worked at small (10,000) businesses (all tech-related), and only the smallest places didn't have the amount of data to warrant taking a look at something like this.
Also, what about those interested in *someday* working for the Googles and the Facebooks of the world? A
Re: (Score:2)
Re: (Score:2)
I'm not sure why you got modded down, because your answer is somewhat thought provoking and not trollish at all.
You reminded me of a situation I saw involving monitoring networks. The solution to the data overload there was a "roll your own" database and AFAIK it could not be querried with a full set of SQL commands; but was faster and able to handle our data better than off-the-shelf Open Source solutions. This was years before I heard the term "big data". We may have been moving in that direction witho
Re: (Score:2)
43 million comments. Let's say a 1KB ballpark average size. Auxiliary data probably negligible compared to that, so let's double it and round up.
100GB.
That is a _puny_ database by "Big Data" standards. Every table apart from the comments themselves could be cached in RAM on a modern server, and the majority of comments would never need to be fetched off disk - a single SSD at that - so almost everything important could be cached.
Of course, you'd never want an architecture like
Re: (Score:2)
What is "big" is probably the apache logs :-)
Yay Packt! (Score:2)
LMAO... HFDS? (Score:1)
It's HDFS ... apparently the book didn't teach the reviewer much. "Hadoop Distributed File System."
Re: (Score:1)
Re: (Score:1)
If it's a tool... (Score:1)
If it's just a bleeping tool for data processing, what should the user need to be a Java developer to use it?
Re: (Score:1)
Excellent point...but that is something to ask the Hadoop developers, not the author. You can use Hadoop without writing Java. You cannot use Hadoop without knowing in depth about the complete mess that is Java build, linking, etc.
Re: (Score:1)
1) $> gradle run
2) $>
3) $>
Hadoop (Score:1, Troll)
Hadoop. For when you feel like something fun to do on a friday night.