TextInputFormat Vs KeyValueTextInputFormat

Lets try to understand the difference with the help of an example. Suppose we have following data .
Id,Name,Gender

1,Krishna,M
2,Tina,F
3,Umesh,M
4,Nakeeran,F
5,Varun,M

Now lets write Mapreduce code to print the input as output using TextInputFormat .

By default TextInputFormat have the key value pair in this format:

Key -> LongWritable
Value -> Text

Here is the mapper code.

Please note here my K1-> LongWritable line offset
                                 V1-> Text
We don't need any reducers and we will print all data as K2->Text and V2-> Text

public static class SMapperDemo extends
            Mapper<LongWritable, Text, Text, Text> {

        @Override
        protected void map(LongWritable key, Text value, Context context)
                throws java.io.IOException, InterruptedException {
            if (value.toString() != "" && value.toString().contains(",")) {
                String[] array = value.toString().split(",");

                String userInfo = array[0] + "," + array[1] + "," + array[2];

                context.write(new Text(key.toString()), new Text(userInfo));
            }
        };
    }

The driver code is simple as well.

@Override
    public int run(String[] args) throws Exception {
        // TODO Auto-generated method stub

        Configuration conf = getConf();

        

        Job job = new Job(conf, "Sql Demo");

        job.setJarByClass(SDemo.class);

        job.setMapperClass(SMapperDemo.class);

        job.setNumReduceTasks(0);

        job.setInputFormatClass(TextInputFormat.class);

        job.setOutputFormatClass(TextOutputFormat.class);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(Text.class);

        Path in = new Path(args[0]);

        Path out = new Path(args[1]);

        FileInputFormat.addInputPath(job, in);

        FileOutputFormat.setOutputPath(job, out);

        System.exit(job.waitForCompletion(true) ? 0 : 1);

        return 0;
    }

On running this code, it will print line offset as Key and Line itself as value. The output looks like :

0    Id,Name,Gender
15    1,Krishna,M
27    2,Tina,F
36    3,Umesh,M
46    4,Nakeeran,F
59    5,Varun,M

Now lets write Mapreduce code to print the input as output using KeyValueTextInputFormat . By default KeyValueTextInputFormat is tab separated  and its K1 and V1 is of type Text.

This is the driver code. I have done 2 changes here. Changed the default line separator to "," and set the input format as KeyValueTextInputFormat.

public int run(String[] args) throws Exception {
        // TODO Auto-generated method stub

        Configuration conf = getConf();

        conf.set("key.value.separator.in.input.line", ",");

        Job job = new Job(conf, "Sql Demo");

        job.setJarByClass(SDemo.class);

        job.setMapperClass(SMapperDemo.class);

        job.setNumReduceTasks(0);

        job.setInputFormatClass(KeyValueTextInputFormat.class);

        job.setOutputFormatClass(TextOutputFormat.class);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(Text.class);

        Path in = new Path(args[0]);

        Path out = new Path(args[1]);

        FileInputFormat.addInputPath(job, in);

        FileOutputFormat.setOutputPath(job, out);

        System.exit(job.waitForCompletion(true) ? 0 : 1);

        return 0;
    }

The mapper code is reading the input file . Since the separator is ",", the key will be the text before 1st comma and the value will be remaining line. ex:
Input Line -> 1,Krishna,M
Output Key -> 1
Output Value -> Krishna,M

@Override
        protected void map(Text key, Text value, Context context)
                throws java.io.IOException, InterruptedException {
            if (value.toString() != "" && value.toString().contains(",")) {
                String[] array = value.toString().split(",");




// array[2] will give index out of bound error since the value only has remaining // part of line after 1st comma 

                String userInfo = array[0] + "," + array[1];// + "," + array[2];

                context.write(new Text(key.toString()), new Text(userInfo));
            }
        };

And the output looks like :

Id    Name,Gender
1    Krishna,M
2    Tina,F
3    Umesh,M
4    Nakeeran,F
5    Varun,M

Here Id became key and remaining line became value.

Happy Hadooping!!

have a great day.

Varun