Hadoop HDFS Commands

This document shows a number of "essential" commands for manipulating files on Hadoop's HDFS file system. Basically all commands start with hdfs dfs and then use a command form that resembles ordinary Unix file system commands. There are a few special cases, however. Also, keep in mind that every user has a home directory in HDFS, which is the default location for files without a path prefix. You have full access to your personal home directory, but HDFS manages file system permissions much like Unix does, so your access to other directories inside HDFS may be limited. The hadoop user's home directory is a storage place for various public files. All users have read access to that space.

Before you can use Hadoop on Lemuria, you must first set up your environment. Do this by executing the following command:

      $ source useHadoop.sh

It is necessary to execute this every time you open a new terminal session on Lemuria. The environment created by this command is lost when you close the terminal (this avoids potential conflicts with other programs in the case when you aren't trying to use Hadoop).

hdfs dfs -ls (directory listing of your home directory)
hdfs dfs -ls /user/hadoop (directory listing of hadoop's home directory)
hdfs dfs -cat some/file (displays the contents of some/file)
hdfs dfs -rm some/file (removes some/file)
hdfs dfs -rm -r output (removes folder output and all subfolders and files)
hdfs dfs -copyFromLocal myfile (copies a local file into HDFS)
hdfs dfs -copyToLocal myfile (copies a file from HDFS to the local file system)
hdfs dfs -chmod 775 some/file (changes permission on some/file)
hdfs dfs -chown user some/file (changes the ownership of some/file)
hdfs dfs -chgrp group some/file (changes the group association of some/file