MLOps/hadoop
local 에 있는 csv 파일을 hadoop 으로 가져오기
입짧은달님
2022. 3. 29. 12:35
https://stackoverflow.com/questions/34277239/importing-csv-file-into-hadoop
Importing CSV file into Hadoop
I am new with Hadoop, I have a file to import into hadoop via command line (I access the machine through SSH) How can I import the file in hadoop? How can I check afterward (command)?
stackoverflow.com
데이터 확인
$ ls -al /mnt/shopping20y1h/shopping
$ find /mnt/shopping20y1h -type f | wc -l
$ du -sh /mnt/shopping20y1h데이터 가져오기
hdfs dfs -put /home/username/file.csv /user/data/file.csv혹은
hadoop distcp file:///mnt/shopping20y1h/shopping hdfs://nv0###-hadoop/user/ncp/shopping20y1h하둡에 업로드된 데이터 확인
$ hdfs dfs -ls /user/ncp/shopping20y1h
$ hdfs dfs -count /user/ncp/shopping20y1h
$ hdfs dfs -du -h /user/ncp