This post is intended for folks who are looking out for a quick start on developing a basic Hadoop MapReduce application. We will see how to set up a basic MR application for WordCount using Java, Maven and Eclipse and run a basic MR program in local mode , which is easy for debugging at … Continue reading How-To : Setup Development Environment for Hadoop MapReduce
The post How-To : Setup Development Environment for Hadoop MapReduce appeared first on Saurz{Code}.
Pig is a high level scripting language that is used with Apache Hadoop. Pig enables data analysts to write complex data transformations without knowing Java. Pig’s simple SQL-like scripting language is called Pig Latin, and appeals to developers already familiar with scripting languages and SQL.Pig Scripts are converted into MapReduce Jobs which runs on data stored in HDFS (refer to the diagram below).
Through the User Defined Functions(UDF) facility in Pig, Pig can invoke code in many languages like JRuby, Jython and Java. You can also embed Pig scripts in other languages. The result is that you can use Pig as a component to build larger and more complex applications that tackle real business problems.
Pig consists of three elements -
Pig Latin Statements
Pig Latin statements are the basic constructs you use to process data using Pig. A Pig Latin statement is an operator that takes a relation as input and produces another relation as output(except LOAD and STORE statements).
Pig Latin statements are generally organized as follows:
Note that a DUMP or STORE statement is required to generate output.
A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float);
B = FOREACH A GENERATE name;
A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float);
B = FOREACH A GENERATE name;
DUMP B;
(John)
(Mary)
(Bill)
(Joe)
Pig stores the intermediate data generated between MapReduce jobs in a temporary location on HDFS. This location must already exist on HDFS prior to use. This location can be configured using the pig.temp.dir property.
Use the STORE operator and the load/store functions to write results to the file system ( PigStorage is the default store function).
Note: During the testing/debugging phase of your implementation, you can use DUMP to display results to your terminal screen. However, in a production environment you always want to use the STORE operator to save your results.
Pig Latin provides operators that can help you debug your Pig Latin statements:
Pig provides extensive support for user-defined functions (UDFs) as a way to specify custom processing. Functions can be a part of almost every operator in Pig.UDF is very powerful functionality to do many complex operations on data.The Piggy Bank is a place for Pig users to share their functions(UDFs).
Example:
REGISTER saurzcodeUDF.jar;
A = LOAD 'employee_data' AS (name: chararray, age: int, designation: chararray);
B = FOREACH A GENERATE saurzcodeUDF.UPPER(name);
DUMP B;
This article was just a Getting Started Article on Pig , I will cover further details about How to to Write Pig Latin commands for some basic operations like JOIN,FILTER,GROUP, ORDER etc. , also how to make your own UDFs for processing on Hadoop cluster.
References :-
Hadoop and Big Data are becoming the new hot trends of the industry , being the most sought out skills in the market.There are various vendors and online training providers coming up with a nice explanation of some of the core concepts underlying Hadoop frameworks like Mapreduce,HDFS and various components involved in Hadoop Ecosystem.I will try to list down some of these resources here -
1. Udacity - Cloudera came with this nice course named "Intro to Hadoop and MapReduce" .This provides a nice explanation of the core concepts and internal working of hadoop components embedded with quizzes around each concept and some good handson exercies.They also provide VM for training purpose, which can be used to run example questions and to solve quizzes and exams for the courses.
Goals -
Prerequisites -
Some basic programming knowledge and a good interest in learning :)
hadoop fs -mkdir <paths>
hadoop fs -ls <args>
hadoop fs -lsr <args>
hadoop fs -touchz <path[filename]>
hadoop fs -cat <path[filename]>
hadoop fs -cp <source> <dest>
hadoop fs -put <source:localFile> <destination>
hadoop fs -get <source> <dest:localFileSystem>
hadoop fs -copyFromLocal <src:localFileSystem> <dest:Hdfs>
hadoop fs -copyToLocal <src:Hdfs> <dest:localFileSystem>
hadoop fs -mv <src> <dest>
hadoop fs -rm <arg>
hadoop fs -rmr <arg>
hadoop fs -stat <path>
hadoop fs -tail <path[filename]>
hadoop fs -test -[ezd]<path>
hadoop fs -text <src>
hadoop fs -du <path>
hadoop fs -dus <args>
hadoop fs -expunge
hadoop fs -chgrp [-R] GROUP <path>
hadoop fs -chmod [-R] <MODE[,MODE] | OCTALMODE> <path>
hadoop fs -chown [-R] [OWNER][:[GROUP]] <path>
hadoop fs -getmerge <src> <localdst> [addnl]
hadoop fs -setrep [-R] <path>
