I transferred a approximately 500 MB csv file from my local workstation to master node home path of a spark cluster. I used the following command:
gcloud compute scp local_file_path\file_name.csv cluster-name-m:~
The output is as follows:
file_name.csv | 564076 kB | 87.1 kB/s | ETA: 00:00:00 | 100%
However, when I ssh into the master node using putty, the file does not show up in the home directory.
ls list empty directory, i.e. file is not listed.
The csv file is not displayed even in WinSCP.
But when I transferred another file, this time a python file python_file.py using WinSCP, ls lists the .py file.
What is going wrong? because clearly gcloud shows transfer completed. Any help is appreciated.
The comment by @John Hanley points to the issue.
The gcloud client and Putty client are two different users and hence different /home/user directories for each....
Instead of using cluster-name-m:~ it should be cluster-name-m:/home/user