User Defined Counters in Hadoop

Friends,

Today, we will learn few concepts about User Defined Counters. Hadoop have many counters such as :
1) MR Task Counters
2) File System Counters
3) File input & File Output format counters
4) Job Counters
5) User Defined Counters

Let's try to understand why do we need counters ?
Consider a scenario where you have many bad records in your input file. You want to find out how many such bad records exists . Ex: you are writing MR job for the query:
Select name,count(*) from User but some of the records are bad as the name do not exists in the columns . You would want to filter out such records and at the same time want to see how many such records exists. Here is the sample data :

Id,Name
1,Tom
2,John
3,
4,null
5,
6,Tom
7,Abi
8,Jack
9,Jack
10,

Let's write some code to create the counters. We will start by writing an enum to define the categories of records.

public enum RecordType {
  Good, Bad
 }

Here is the mapper that will increment the counter when it gets Good/Bad records.

protected void map(LongWritable key, Text value, Context context)
    throws java.io.IOException, InterruptedException {
   if (value.toString() != "" && value.toString().contains(",")) {

    String[] array = value.toString().split(",");

    if (array.length < 2) {
     context.getCounter(RecordType.Bad).increment(1);
    } else {
     if (array[1].isEmpty()) {

      context.getCounter(RecordType.Bad).increment(1);
     } else {
      context.getCounter(RecordType.Good).increment(1);
      context.write(new Text(array[1]), new IntWritable(1));
     }
    }

   }

  };

The reducer code is simple  and needs no explanation.

protected void reduce(Text key, Iterable<IntWritable> values,
    Context context) throws java.io.IOException,
    InterruptedException {
   int count = 0;

   for (IntWritable x : values) {
    int val = Integer.parseInt(x.toString());

    count += val;

   }
   context.write(key, new IntWritable(count));
  };

Once you run the code on the sample dataset . You should be able to see the following counters:

This looks good but the only issue is the name (CDemo$RecordType) is not too user friendly.
Lets see how we can change the names to better user friendly names.

1) Create a file with the name ClassName_Enum.properties . ex: CDemo_RecordType.properties
2) place the file at same location where your class exists
3) Enter following lines :

CounterGroupName= Counter records
Good.name= GOOD Records
Bad.name=BAD Records

4) Save the file and create the jar and run it using hadoop command.
5) Now you can see counters with the names supplied in your file.

Here is the source code :

https://github.com/varunkhatri/HadoopCounters/blob/master/CountersDemo/src/CDemo.java

Hope you enjoyed the demo.

Happy Hadooping !!!

Varun