Project

General

Profile

ncrcat command regarding sftp server

Added by Vladimir Platonov about 2 months ago

Dear colleagues, I have an issue with remote processing of my COSMO-CLM output data. I have a remote NAS (Network Attached Storage), there are many COSMO-CLM output data (many lffd*.nc files) and there is no possibility to install NCO at NAS. This NAS has an sftp address and connection, I can connect it using simple sftp in Linux (sftp login_name@ip_address, then password). However, what I would like to do, is to postprocess these lffd*.nc data remotely, e.g., I would like to run NCO command (e.g., ncrcat) on my local computer to get the ncrcat-ted file locally using the sftp access to my NAS. I have read the NCO sourceforge Userguide and tried many variants, however, I haven’t any success.

I have tried the following commands (and many similar ones) and got the following results:

ncrcat -h -O -v T_2M,lat,lon -l . sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02/lffd1996*.nc -o T_2M.nc
's password:
Connected to 192.168.1.58.
Multiple source paths, but destination "./lffd1996*.nc” is not a directory
ncrcat: ERROR Synchronous fetch command failed

ncrcat -h -O -v T_2M,lat,lon -l . :/DATA/OUT_Arctic_5.05_final_1996/out02/lffd*.nc -o T_2M.nc
's password:
scp: /DATA/OUT_Arctic_5.05_final_1996/out02/lffd*.nc: No such file or directory
ncrcat: ERROR Synchronous fetch command failed

ncrcat -h -O -v T_2M,lat,lon -H -p sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02 -l . lffd*.nc -o T_2M.nc
's password:
Connected to 192.168.1.58.
Multiple source paths, but destination "./lffd*.nc” is not a directory
ncrcat: ERROR Synchronous fetch command failed

ncrcat -h -O -v T_2M,lat,lon -p sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02 -l . lffd*.nc -o T_2M.nc
's password:
Connected to 192.168.1.58.
Multiple source paths, but destination "./lffd*.nc” is not a directory
ncrcat: ERROR Synchronous fetch command failed

ncrcat -h -O -v T_2M,lat,lon sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02/* -l . -o T_2M.nc
usage: sftp [-46aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]
[-D sftp_server_path] [-F ssh_config] [-i identity_file]
[-J destination] [-l limit] [-o ssh_option] [-P port]
[-R num_requests] [-S program] [-s subsystem | sftp_server]
destination
ncrcat: ERROR Synchronous fetch command failed

ncrcat -h -O -v T_2M,lat,lon -l . sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02/* -o T_2M.nc
usage: sftp [-46aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]
[-D sftp_server_path] [-F ssh_config] [-i identity_file]
[-J destination] [-l limit] [-o ssh_option] [-P port]
[-R num_requests] [-S program] [-s subsystem | sftp_server]
destination
ncrcat: ERROR Synchronous fetch command failed

The following commands (and many similar ones) didn’t get any results and any errors:

ncrcat -h -O -v T_2M,lat,lon -l . sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02/lffd*.nc

ncrcat -h -O -v T_2M,lat,lon -H -p sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02/lffd*.nc -l . -o T_2M.nc

ncrcat -h -O -v T_2M,lat,lon -l . sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02/lffd1996010100.nc

There is the only command which worked somehow:

ncrcat -h -O -v T_2M,lat,lon -l . sftp://192.168.1.58:/DATA/OUT_Arctic_5.05_final_1996/out02/lffd1996010100.nc -o T_2M.nc
's password:
Connected to 192.168.1.58.
Fetching /DATA/OUT_Arctic_5.05_final_1996/out02/lffd1996010100.nc to ./lffd1996010100.nc
/DATA/OUT_Arctic_5.05_final_1996/out02/lffd1996010100.nc 52% 61MB 3.9MB/s 00:14 ETA

However, there is not necessary to operate the single lffd file, I need to operate multiple files. Excuse me for a lot of examples, code and logs.
So, does anyone have some experience in operating NCO via sftp having NCO on local machine and aiming to operate remote machine without NCO? I would be very grateful for any hints in this case. Thanks a lot!


Replies (5)

RE: ncrcat command regarding sftp server - Added by Burkhardt Rockel about 2 months ago

I have never tried this. Since it is very NCO specific, you may pose your question on the web site
https://sourceforge.net/p/nco/discussion/9830/

RE: ncrcat command regarding sftp server - Added by Christian Steger about 2 months ago

I have also no idea how that could work, but is there a reason why you don’t mount your NAS on your computer and then access the files directly?

RE: ncrcat command regarding sftp server - Added by Vladimir Platonov about 2 months ago

Dear colleagues, thanks for your replies and recommendations, I have posted this question on the NCO Sourceforge Forum. My reason to organize NAS was to share COSMO-CLM data with other users, which is much more useful via sftp access. Moreover, I have a personal notebook, which is not applicable for stationary direct access the files on NAS. Therefore, I have organized NAS and SFTP access to data on NAS, and would like to operate these data remotely using NCO.

RE: ncrcat command regarding sftp server - Added by Adam Jaczewski about 2 months ago

Dear Vladimir,

Usually I put file list as argument to the commands, for example in bash I generate an array of file names:

#loop after all files - make array of directory+files names
for ((i=0;i<cnt;i++)); do
member_files[i]="${members[i]}$f”
done

and the array is argument to the command:

ncecat ${member_files[*]} /users/kdm/plgajacz/tmp/cat_tmp

but I see in NCO documentation different solution:

  1. Use xargs to transfer filenames on the command line
    /bin/ls | grep ${CASEID}_’......'.nc | xargs -x ncecat -o foo.nc

Cheers
Adam

RE: ncrcat command regarding sftp server - Added by Vladimir Platonov about 1 month ago

Dear colleagues, thank you for feedback and help. I have partially solved my problem and would like to share this solution for further users and applicatons. Detailed answer and useful links were provided by Charlie Zender on the NCO forum: [[https://sourceforge.net/p/nco/discussion/9830/thread/74d094c06c/?limit=25]]. One of links contained a hint by Adam, thanks a lot!

    (1-5/5)