Purchase Linux CDs / DVDs / Flash Drives at OSDisc.com

Welcome to Our Community

While Linux.org has been around for a while, we recently changed management and had to purge most of the content (including users). If you signed up before April 23rd please sign up again. Thanks!

File transfer over SSH onto linux server from hard disk on remote MAC

Discussion in 'Getting Started' started by Sonal, May 23, 2017.

  1. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0

    I tried with replace "z" it gave same error. I changed the time " LoginGraceTime 600 to LoginGraceTime 12000" but to no avail. Btw my location
    file is /etc/ssh/sshd_config


     
  2. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0
    Thank you.
    I guess. It works I tried with test folder. But I am not sure where the file got saved :D
     
  3. Steve

    Steve New Member

    Joined:
    Apr 25, 2017
    Messages:
    9
    Likes Received:
    10
    If rsync is failing, try scp and see if it is working any better:
    Code:
    scp -r ~/source [email protected]:~/
     
  4. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0
    It gives me an error "No such file or directory" for destination folder which is on server
     
  5. Rob

    Rob Administrator
    Staff Member

    Joined:
    Oct 27, 2011
    Messages:
    145
    Likes Received:
    395
    @Sonal try either rsync or scp with a smaller test file..

    Code:
    touch blah.txt
    scp blah.txt [email protected]:~/
    Code:
    touch blah2.txt
    rsync -avz blah2.txt [email protected]:~/
    Then, ssh into your server and you should see blah.txt and blah2.txt there in your home directory. Let's establish that this works for both scp and rsync.

    (Note: touch creates an empty file)
     
    Steve likes this.
  6. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0
    rsync -Pa -e "ssh -p 22" /Users/sonalharsh/Desktop/Test_Folder [email protected]:~/

    This command saves the folder in the home directory but I want to store in the destination folder which gives error which I showed earlier
     
  7. Rob

    Rob Administrator
    Staff Member

    Joined:
    Oct 27, 2011
    Messages:
    145
    Likes Received:
    395
    Ok, we're getting somewhere..

    Make sure you type the destination folder exactly how it is on the server, it's case sensitive. Also, since you're using port 22, you don't need the whole "ssh -p 22" part. Also, be sure to end with a slash (/) on the destination folder so it puts it inside it.

    Maybe try with the blah.txt example into your destination folder to make sure it works before going for the large file(s).
     
    Sonal likes this.
  8. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0
    I tried and it works and the files are saved in home directory of the user.

    Now even with smaller text file I am not able to save it in my destination folder
     
  9. Rob

    Rob Administrator
    Staff Member

    Joined:
    Oct 27, 2011
    Messages:
    145
    Likes Received:
    395
    Can you paste the destination folder portion of your command?
     
  10. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0
    @Rob I guess the error was because of file name as you suggested in previous command. I changed the name format instead of copying the destination path. It seems to work. Though it shows "file vanished" but I suppose its ok. It will still copy those files(correct me if I am wrong).

    Since the folder is huge(1.96TB) I am waiting for it finish. Does it show that the file transfer is complete? And as per my knowledge rsync compare and copy. There won't be any duplicates or missing files???
     
  11. Rob

    Rob Administrator
    Staff Member

    Joined:
    Oct 27, 2011
    Messages:
    145
    Likes Received:
    395
    if you're using rsync, it won't create any duplicates at all.. even if you kill it and start over.

    If you like, open another terminal and ssh into your server, then check the file size of your destination folder like:

    Code:
    du -s foldername
    then count to 10 and do it again. The size should have increased.
     
    Sonal likes this.
  12. Steve

    Steve New Member

    Joined:
    Apr 25, 2017
    Messages:
    9
    Likes Received:
    10
    With the -P switch, you should see a progress and ETA for individual files. Once complete, it'll drop out of the rsync and return to a prompt. rsync works by comparing time-stamps in the default mode. If the source's is newer, it'll transfer modified portions. If the file transfer stops halfway through a file, restarting it will skip to and resume where it left off.

    If you're having disconnect issues and want to run it until it finishes, you can do
    Code:
    until rsync -Pa <rest of the command here>; do echo "waiting 1 second..."; sleep 1; done
    (NOTE: This may not work on Mac - not sure if that command is available. Literally just learned about it 2 minutes ago)
     
    Sonal and Rob like this.
  13. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0
    @Rob @Steve : Thank you so much. It works.

    Also, we use samba to mount server on the and then transfer files using matlab+dropbox or file relay which is very slow as samba is slow. Can we have something like WinSCP for mac so that we can transfer files continuously when we get data from the field on regular basis. Or can use SQL database for that?
     
  14. Steve

    Steve New Member

    Joined:
    Apr 25, 2017
    Messages:
    9
    Likes Received:
    10
    I think what you're asking for is a way to transfer files from one system to another automatically. What you could do is create a script something like
    /home/sonal/rsync.script.sh:
    Code:
    #! /bin/bash
    if [ -e /home/sonal/rsync.in.progress ]
       then exit
    fi
    touch /home/sonal/rsync.in.progress
    rsync -Pa --delete <rest of command here>
    rm /home/sonal/rsync.in.progress
    
    make it executable:
    Code:
    chmod 744 /home/sonal/rsync.script.sh
    then
    Code:
    crontab -e
    and create an cronjob to run every minute:
    Code:
    * * * * * /home/sonal/rsync.script.sh > /dev/null 
    This will execute the rsync script every minute of every hour of every day of the week of every calendar day of every month (see https://ole.michelsen.dk/blog/schedule-jobs-with-crontab-on-mac-osx.html for a description of the crontab), checking to see if it a job is already in progress. This will also delete files from your destination that are NOT on your source so if you don't want that behavior, omit the
    Code:
    --delete
    switch.

    This is a relatively simple script and I've done no testing on it, but this should get you a step in the direction you want to go.
     
    Sonal and Rob like this.
  15. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0
    I am gonna try this. like from windows/mac system to server we use matlab script to access SFTP for transferring files. So I want to implement a better way to transfer files onto server. I am gonna try the script
     
  16. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0
    Th e data upload is still going on does it this long?
     
  17. Steve

    Steve New Member

    Joined:
    Apr 25, 2017
    Messages:
    9
    Likes Received:
    10
    To be fair, as you said starting out:
    Running some numbers, 1.96 TB at 10 Mbps (1.25 MB/s), you're looking at about 19 days. 1.96 TB even at 100Mbps (12.5 MB/s) is still 1 day, 21 hours. Even at full gigabit link (1000Mbps/125 MB/s), you're looking at a solid 4 hours. If you have physical access, sneaker-net (ie, dumping to a hard drive and physically carrying it over to the other system) could very well be the fastest way to get a large amount of data from point A to point B.
     
    Sonal likes this.
  18. Sonal

    Sonal New Member

    Joined:
    May 23, 2017
    Messages:
    20
    Likes Received:
    0

    Yeah, it took more than 3 days.
     

Share This Page