User Defined Functions ( UDFs) - PIG

Hi friends ,

Today I wanted to give a simple demo of some of the UDFs available in PIG. We will try to understand few UDFs with the help of some code . Let's quickly jump on 1st UDF:

Note: I have read and tried to understand the concepts from Hadoop - The definitive Guide. If you have time , please go through this book. You won't regret it.

Let's Suppose you have data like this :
1;"nyc, new york, usa";NULL
2;"stockton, california, usa";18
3;"moscow, yukon territory, russia";NULL
4;"porto, v.n.gaia, portugal";17
5;"farnborough, hants, united kingdom";NULL
6;"santa monica, california, usa";61

And you want to run a query which looks like :

Select * from table where Id > 2;
In PIG you will use following commands :

A = Load '/homeDesktop/Projects/Book/UserTest.csv' USING  PigStorage(';') 
>> AS (Id:int,Location:chararray,Age:chararray);                                      


 C = FILTER A BY Id >2;                                                             


 Dump C;

And here is the output :

(3,"moscow, yukon territory, russia",NULL)
(4,"porto, v.n.gaia, portugal",17)
(5,"farnborough, hants, united kingdom",NULL)
(6,"santa monica, california, usa",61)

This is Simple. Now consider this filter (Id > 2) is used many times. We can always write a simple UDF to do so.

1) Filter UDF
Create a java project to write MR code. Here name of function is the name of class that we will use in Pig Command. This class will extend FilterFunc.

Note: Filter UDFs are all subclasses of FilterFunc, which itself is a subclass of EvalFunc

public class FilterById extends FilterFunc {

// We will override exec method

// It will accept a tuple from which we will get object

// This object is used for creating filter clause that we used earlier


    @Override
    public Boolean exec(Tuple tuple) throws IOException {
        // TODO Auto-generated method stub

        if (tuple == null || tuple.size() == 0) {
            return false;
        }
        try {
            Object object = tuple.get(0);
            if (object == null) {
                return false;
            }
            int i = (Integer) object;
            return i >= 2;
        } catch (ExecException e) {
            // TODO: handle exception
            throw new IOException(e);
        }
        // return null;
    }

Once this is done , create the jar and register it using following command.

 REGISTER /home/edureka/Desktop/Projects/PigDemo/PDemo1.jar;   
 

Again , you can try following commands :

A = Load '/home/Desktop/Projects/Book/UserTest.csv' USING  PigStorage(';') 
>> AS (Id:int,Location:chararray,Age:chararray);                                     



 REGISTER /home/edureka/Desktop/Projects/PigDemo/PDemo1.jar;  

                     
 C = FILTER A BY FilterById(Id);                                                    
 Dump C; 

And here is the output :

(3,"moscow, yukon territory, russia",NULL)
(4,"porto, v.n.gaia, portugal",17)
(5,"farnborough, hants, united kingdom",NULL)
(6,"santa monica, california, usa",61)

  2) Now let's say you want to add some string to location column. We can use EvalFunc to write this UDF.

public class AddString extends EvalFunc<String> {

 @Override
 public String exec(Tuple tuple) throws IOException {
  // TODO Auto-generated method stub
  if (tuple == null || tuple.size() == 0) {
   return null;
  }
  try {
   Object object = tuple.get(0);
   if (object == null) {
    return null;
   }
   return ((String) object) + "- Added String";
  } catch (ExecException e) {
   throw new IOException(e);
  }
 }

}

  and then use following commands .

grunt> A = Load '/home/Desktop/Projects/Book/UserTest.csv' USING  PigStorage(';') 
>> AS (Id:int,Location:chararray,Age:chararray);                                      
grunt> REGISTER /home/Desktop/Projects/PigDemo/PDemo1.jar; 
C = FOREACH A GENERATE Id,AddString(Location),Age; 
Dump C;
 

 And this is the output.

(1,"nyc, new york, usa"- Added String,NULL)
(2,"stockton, california, usa"- Added String,18)
(3,"moscow, yukon territory, russia"- Added String,NULL)
(4,"porto, v.n.gaia, portugal"- Added String,17)
(5,"farnborough, hants, united kingdom"- Added String,NULL)
(6,"santa monica, california, usa"- Added String,61)