– Copy fromLocal/ToLocal from/to S3:
$ bin/hadoop fs -copyToLocal s3://my-bucket/myfile.rb /home/hadoop/myfile.rb $ bin/hadoop fs -copyFromLocal job5.avro s3://my-bucket/input
– Merge all the files from one folder into one single file:
$ hadoop jar ~/lib/emr-s3distcp-1.0.jar --src s3://my-bucket/my-folder/ --dest s3://my-bucket/logs/all-the-files-merged.log --groupBy '.*(*)' --outputCodec none
– Create directory on HDFS:
$ bin/hadoop fs -mkdir -p /user/ubuntu
– List HDFS directory:
bin/hadoop fs -ls /
– Put a file in HDFS:
bin/hadoop dfs -put localfile.txt /user/hadoop/hadoopfile
– Check HDFS filesystem utilization:
$ bin/hadoop dfsadmin -report
– Cat of file on HDFS:
$ bin/hadoop dfs -cat /user/ubuntu/RESULTS/part-00000
More commands: