File transport from Windows to Linux, getting started with NAS

Binary91

New Member
Credits
114
Hey guys,

this is my first post on this cool forum and I'd like to say that I'm absolutely new to Linux.

My whole life, I was using MS Windows and all my data is still stored on internal and external NTFS filesystem devices.

Now, I'm interested in Linux because I'm building my own NAS. The NAS should make it possible to:
a) gain access to my whole data from everywhere over the world at any time with any device (Microsoft, Apple, Android ...)
b) to synchronize data between those devices with NextCloud
c) to make weekly backups.

My plan to realize this is by:
a) a Raspberry Pi with Raspbian Linux on an internal 64GB SD Card for the Server
b) a permanentely mounted external USB with 2 TB SSD for all my "dynamic" data (data in the NextCloud files that should be accessible and synchronized erverywhere at any time by any device)
c) a 10 TB external HDD for all existing data I ever had (an archieve for "static" data) for backups one time a week

First of all: Would you aggree to that "plan"? Is that a good method to realize my intentions?

The question(s) that arise with this are:
1) What filesystem should I use for the mentioned three storages (SD card, USB, HDD)? I guess that the SD card will have to be ext4. My google research showed that this filesystem is very good and I tend to use it for alle three storage devices. I mean, the access to all of them will go with the Raspberry Pi, so access will always be with a linux system, doesn't matter with which device I call the NAS from extern, right?

2) Should I do the formatting (to ext4?) of the devices with my MS computer or should I do it with the Raspberry? Are there any differences?

3) How can I safely transfer my existing data (NTFS) to the new storage devices without loss of data (high res. images/videos, MS office docs. etc.) ?

4) Do you have any further advices for me before I start running the Raspberry Pi and intsall the OS the first time? There are so many OS options and so many ways to install packages and more... I'd like to do it perfect from the beginning!

Well, thank you so much in anticipation!

Kind regards,

Binary91
 


Lord Boltar

Active Member
Credits
1,564
To transfer NTFS Files you will need to install ntfs-3g on Linux - If you want to use exFAT you will need to install exfat-fuse and exfat-utils on Linux

NTFS-3G uses FUSE (Filesystem in Userspace) to provide support for the NTFS
filesystem used by Microsoft Windows.

Personally I would use FAT or exFat for the external drive unless you want to encrypt just for Linux then I would use ext4 with LUKS - but you have to remember about the file limitation size in FAT
 

Binary91

New Member
Credits
114
To transfer NTFS Files you will need to install ntfs-3g on Linux - If you want to use exFAT you will need to install exfat-fuse and exfat-utils on Linux

NTFS-3G uses FUSE (Filesystem in Userspace) to provide support for the NTFS
filesystem used by Microsoft Windows.

Personally I would use FAT or exFat for the external drive unless you want to encrypt just for Linux then I would use ext4 with LUKS - but you have to remember about the file limitation size in FAT
First of all, thank you for your fast reply!
Ok, so NTFS-3G can help me doing that. Is it a safe way to copy&paste data or are there any file types that are not supported or any file attributes/meta data that is lost afterwards? (I'm asking this because it should be possible to access this data through the linux server by another OS that may need this meta data..)
 

Lord Boltar

Active Member
Credits
1,564
First of all, thank you for your fast reply!
Ok, so NTFS-3G can help me doing that. Is it a safe way to copy&paste data or are there any file types that are not supported or any file attributes/meta data that is lost afterwards? (I'm asking this because it should be possible to access this data through the linux server by another OS that may need this meta data..)
I did a 2TB drive from NTFS after install ntfs-3g and had zero issues
 

Binary91

New Member
Credits
114
Ok, then I will do it that way. I would tend to ext4, because there are just benefits except the fact that I can't use it with windows, am I right?

How do you think about my NAS concept? Is it good or do you have any good advice for a better way to realize a small home server permanentely running as a cloud and backup system (and maybe a website and email server at some time..)?
 

Lord Boltar

Active Member
Credits
1,564
Ok, then I will do it that way. I would tend to ext4, because there are just benefits except the fact that I can't use it with windows, am I right?

How do you think about my NAS concept? Is it good or do you have any good advice for a better way to realize a small home server permanentely running as a cloud and backup system (and maybe a website and email server at some time..)?
Yep can't use ext 4 with Windoz - Seems to me your NAS idea is fine although
I would consider Ubuntu Server for PI

 

Binary91

New Member
Credits
114
Yep can't use ext 4 with Windoz - Seems to me your NAS idea is fine although
I would consider Ubuntu Server for PI

Thank's.
Already thought about that. My problem is that I never used any Linux system before. Google says it could be hard for some newbie (like me) using a "server" edition of a distribution (Debian, Ubuntu Server, Raspbian Lite and others). On the other hand, such a lite version should be much more performant when using it with a simple Rasberry Pi... I really don't know what to do. Is it managable for someone like me to directly start with a windowless lite version of a Distro? I'm reading all beginner tutorials on this platform as well as many other tutorials, but I still have no practise. I'm just afraid of trying it and failing hard..
 

Lord Boltar

Active Member
Credits
1,564
Thank's.
Already thought about that. My problem is that I never used any Linux system before. Google says it could be hard for some newbie (like me) using a "server" edition of a distribution (Debian, Ubuntu Server, Raspbian Lite and others). On the other hand, such a lite version should be much more performant when using it with a simple Rasberry Pi... I really don't know what to do. Is it managable for someone like me to directly start with a windowless lite version of a Distro? I'm reading all beginner tutorials on this platform as well as many other tutorials, but I still have no practise. I'm just afraid of trying it and failing hard..
I would not worry about failing hard when I started with Linux I made my fair share of screw-ups, but that is how I learned.
 
Welcome to the form @Binary91. I have a very large question that you have not address in your original post. What do you plan to use as your primary operating system ( Windows or Linux). Based on your primary operating system the answer you seek can change.

Now, I'm interested in Linux because I'm building my own NAS. The NAS should make it possible to:
a) gain access to my whole data from everywhere over the world at any time with any device (Microsoft, Apple, Android ...)
b) to synchronize data between those devices with NextCloud
c) to make weekly backups.
If you plan to use the NAS as stated above then format for NTFS or FAT Linux can read it also windows.
Okay you will have a problem with item b) that is not the primary purpose of a NAS. A NAS is basically used for storage.
Okay item c) depending on how many computers you are backing up to the NAS this could also be a sticky area.

My plan to realize this is by:
a) a Raspberry Pi with Raspbian Linux on an internal 64GB SD Card for the Server
b) a permanentely mounted external USB with 2 TB SSD for all my "dynamic" data (data in the NextCloud files that should be accessible and synchronized erverywhere at any time by any device)
c) a 10 TB external HDD for all existing data I ever had (an archieve for "static" data) for backups one time a week

First of all: Would you aggree to that "plan"? Is that a good method to realize my intentions?
I think your plan is froth with potential to excel and a lot of heartache. IMHO building up a raspberry Pi as the controller for the NAS is one thing. The ability for other computers to sync for backup with the system is where the potential to excel and a lot of heartache will be.

The question(s) that arise with this are:
1) What filesystem should I use for the mentioned three storages (SD card, USB, HDD)? I guess that the SD card will have to be ext4. My google research showed that this filesystem is very good and I tend to use it for alle three storage devices. I mean, the access to all of them will go with the Raspberry Pi, so access will always be with a linux system, doesn't matter with which device I call the NAS from extern, right?
Here you will need to do a lot of research as to can you use a USB in the manner that you would like to use it? You may have the same need to research for the SD card. The HDD is not a problem it can take numerous writes and reads without a problem. SD cards and USB's may have a shorten life at the least.

okay take into consideration here what computing device will has access to the NAS. If any system other that a Linux system must access you can not format any of the devices to ext4 it must be a format that all can read. If only Linux systems will access yes use ext4.

2) Should I do the formatting (to ext4?) of the devices with my MS computer or should I do it with the Raspberry? Are there any differences?

3) How can I safely transfer my existing data (NTFS) to the new storage devices without loss of data (high res. images/videos, MS office docs. etc.) ?
Use the Raspberry to do the format to ext4. There are potential differences Linux vs Msoft.
You should be able to transfer all of your data to the NAS from windows and linux machines without a problem. You can not readily read Linux ext4 with other computer operating systems.


4) Do you have any further advices for me before I start running the Raspberry Pi and intsall the OS the first time? There are so many OS options and so many ways to install packages and more... I'd like to do it perfect from the beginning!
Yes become a user of google and become a google-fu master to research the impossible. You are on the right track!! This is the forum to be in to ask the questions you have asked. I'm sure more people will drop by to assist. @wizardfromoz @JasKinasis are a good bunch to get to know.
 

wizardfromoz

Super Moderator
Staff member
Gold Supporter
Credits
10,019
:)Dream on, Hansel (but kind thoughts), my expertise lies elsewhere, this is beyond my paygrade.

Gday @Binary91 and welcome to linux.org :)

I'm moving this Thread to Server Linux, because it is not a Getting Started Thread, but if someone thinks Network is better, let me know.

Cheers, Good Luck to the OP, and

Avagudweegend

Chris Turner
wizardfromoz
 

f33dm3bits

Gold Member
Gold Supporter
Credits
9,112
Hey guys,

this is my first post on this cool forum and I'd like to say that I'm absolutely new to Linux.

My whole life, I was using MS Windows and all my data is still stored on internal and external NTFS filesystem devices.

Now, I'm interested in Linux because I'm building my own NAS. The NAS should make it possible to:
a) gain access to my whole data from everywhere over the world at any time with any device (Microsoft, Apple, Android ...)
b) to synchronize data between those devices with NextCloud
c) to make weekly backups.

My plan to realize this is by:
a) a Raspberry Pi with Raspbian Linux on an internal 64GB SD Card for the Server
b) a permanentely mounted external USB with 2 TB SSD for all my "dynamic" data (data in the NextCloud files that should be accessible and synchronized erverywhere at any time by any device)
c) a 10 TB external HDD for all existing data I ever had (an archieve for "static" data) for backups one time a week

First of all: Would you aggree to that "plan"? Is that a good method to realize my intentions?

The question(s) that arise with this are:
1) What filesystem should I use for the mentioned three storages (SD card, USB, HDD)? I guess that the SD card will have to be ext4. My google research showed that this filesystem is very good and I tend to use it for alle three storage devices. I mean, the access to all of them will go with the Raspberry Pi, so access will always be with a linux system, doesn't matter with which device I call the NAS from extern, right?

2) Should I do the formatting (to ext4?) of the devices with my MS computer or should I do it with the Raspberry? Are there any differences?

3) How can I safely transfer my existing data (NTFS) to the new storage devices without loss of data (high res. images/videos, MS office docs. etc.) ?

4) Do you have any further advices for me before I start running the Raspberry Pi and intsall the OS the first time? There are so many OS options and so many ways to install packages and more... I'd like to do it perfect from the beginning!

Well, thank you so much in anticipation!

Kind regards,

Binary91
1) It actually shouldn't matter so probably ext4 or xfs will be fine because as you described you will be accessing your data through nextloud. An option to also consider for your local network is setting up samba(or nfs if you don't have any windows clients) shares that way you can acess your data on your local network without nextcloud.
2) If you are going to be creating filesystems you should do it from GNU/Linux, so from your raspberry pie.
3) @Lord Boltar already mentioned this: ntfs-3g
4) For nextcloud you need a webserver, php, and a database. I would also recommend setting up fail2ban on your nextcloud log file. Just do it one step at a time and see the experience as a learning project this way you will have a different mindset going in, once you have it setup you can always improve it and later think of new things to try to add to improve your setup etc.

Hope that helps. Good luck!
 
Last edited:

Binary91

New Member
Credits
114
Thank you for you replies!
@Lord Boltar: True, I think there is no way around...
@Hansel Johnson Jr: That is very detailed, thank you.
I have a very large question that you have not address in your original post. What do you plan to use as your primary operating system ( Windows or Linux). Based on your primary operating system the answer you seek can change.
Excuse me if I didn't explain it more concrete. The whole system should only be directely accessed by the Raspberry Pi, hence Linux. Other devices (doesn't matter which ones, e. g. Windows, Antroid, MacOS...) can just call the server by the software that offers the cloud (I thought of NextCloud, because it is mentioned in some homeserver tutorials) or by the PHP/html website which I will script someday... This is what @f33dm3bits confirms:
It actually shouldn't matter so probably ext4 or xfs will be fine because as you described you will be accessing your data through nextloud.
b) that is not the primary purpose of a NAS. A NAS is basically used for storage.
Sure, I should call it a homeserver with connected NAS (or would that be DAS in this case? I don't get the exact difference so far...).
Maybe, for better understanding, I tell you what the homeserver should do:
1) For me, it should serve a way to access all ever existing data of mine, hence the data on the HDD.
2) For selected people, it should serve a way to synchronize data with NextCloud. This data is stored in the SDD device.
3) For everyone, it should serve as a host for websites, mail server etc. that I will script someday. This websites may also access selected data of the HDD.
c) depending on how many computers you are backing up to the NAS this could also be a sticky area.
The 10 TB HDD should only contain weekly backups of the NextCloud data, that is not more than about 1-2 TB of mixed data (photo, video, audio, docs).Additionally, it should store archieved data that is not part of the NextCloud but should be accessed through safe accounts on my website (just me and my family). I know, this makes permantent connection to the HDD obligatory and it is not recommended to make safe backups to a permanentely connected device, but I plan to buy also a third storage device for mirroring the large HDD manually in frequent periods (maybe once a month).
I think your plan is froth with potential to excel and a lot of heartache. IMHO building up a raspberry Pi as the controller for the NAS is one thing. The ability for other computers to sync for backup with the system is where the potential to excel and a lot of heartache will be.
What do you mean? Other computers should only have access to NextCloud and to my website. Backups should be made automatically by the Raspberry, i. e. synchronizing the local NextCloud data with the HDD. What I'm asking myself is if NextCloud can be used simultaniously from Windows and Linux devices as it uses data from my ext4 file system? But I think the local file system doesn't influence the files itsself, right? So there shouldn't be a problem with it, correct?
Here you will need to do a lot of research as to can you use a USB in the manner that you would like to use it? You may have the same need to research for the SD card. The HDD is not a problem it can take numerous writes and reads without a problem. SD cards and USB's may have a shorten life at the least.
You're right. The USB is a simple and cheap way for the prototype of the server. When I got it running safe and trusty, I will expand and buy a external SSD device (I think it should be SSD to guarantee fast traffic, not?).
Use the Raspberry to do the format to ext4. There are potential differences Linux vs Msoft.
You should be able to transfer all of your data to the NAS from windows and linux machines without a problem. You can not readily read Linux ext4 with other computer operating systems.
Thank you! So I will connect the new HDD to the Raspberry, format it to ext4, connect the NTFS storage device to my Raspberry and copy & paste the data to the new HDD with NFTS-3g.
@wizardfromoz:
Gday @Binary91 and welcome to linux.org :)

I'm moving this Thread to Server Linux, because it is not a Getting Started Thread, but if someone thinks Network is better, let me know.

Cheers, Good Luck to the OP, and

Avagudweegend
Thank you! Well, maybe the thread is better stored there, but as I'm a total newbie, I thought to better post in the beginners thread:D
@f33dm3bits:
An option to also consider for your local network is setting up samba(or nfs if you don't have any windows clients) shares that way you can acess your data on your local network without nextcloud.
I don't know samba but I have heard it many times the last days I got (theroretically) used to Linux. I will have to study it.
If you are going to be creating filesystems you should do it from GNU/Linux, so from your raspberry pie.
Yep, I will do that.
For nextcloud you need a webserver, php, and a database. I would also recommend setting up fail2ban on your nextcloud log file. Just do it one step at a time and see the experience as a learning project this way you will have a different mindset going in, once you have it setup you can always improve it and later think of new things to try to add to improve your setup etc.
Ok, sounds like NextCloud isn't some software that I can simply install and define a master directory that is simply synchronized with all devices that also use NextCloud. I have really none experience - well, I know MS OneDrive :D - but I also have to study that. The same for fail2ban, 100% no clue what that is. But I'm interested and afraid to learn more about that, step by step.
 
@Binary91 Hopefully I did not come off as to brash. Personally I like your effort moving forward. I happen to setup a NAS last year using the Synology equipment off of my LAN. I thought about the mehod you are trying and it was out of my league. I'm wishing you luck. Please keep us informed as to your progress and questions. Who knows how many people you are helping make a decision. I dont know if you noticed but several people jumped in to help, that's why I like this Forum. BTW I have 5 computers hooked up to my NAS and it serves me well for storage and the ability to access all of my data from several different computers.
 

f33dm3bits

Gold Member
Gold Supporter
Credits
9,112
Thank you for you replies!
@f33dm3bits:
I don't know samba but I have heard it many times the last days I got (theroretically) used to Linux. I will have to study it.
Yep, I will do that.
Ok, sounds like NextCloud isn't some software that I can simply install and define a master directory that is simply synchronized with all devices that also use NextCloud. I have really none experience - well, I know MS OneDrive :D - but I also have to study that. The same for fail2ban, 100% no clue what that is. But I'm interested and afraid to learn more about that, step by step.
It's basically:
1. Install a webserver and setup a web location for nextcloud.
2. Install a database(I use MySQL but SQLlite should be easier for you since it requires less configuration)
3. Install php and the required modules.
4. Download nextlcoud installation tarball, extract it in your web location and run the url you have setup for that location in your web-browser.
-----------------------------------------------------------------------------------------
I think Raspian is based on Debian so it should be something like this.
A. Installation webserver
1. sudo apt-get install apache2
2. setup a virtualhost
3. sudo apt-get install php php-curl php-xml php-gd php-json php-mbstring php-zip(I may have missed a few)

B. Install database: sudo apt-get install sqlite3 (and if you want to install mysql/mariadb: sudo apt-get install mariadb-server)
1. Setup for sqllite is just make sure the permissions are correct.
2. If you want to use mysql you will have to do a bit more.
a. Setup MySQL (including root password)
b. Login as root user in mysql: mysql -u root -p
c. create database: create database nextcloud;
b. Setup MySQL user (for nextcloud to use): GRANT ALL PRIVILEGES ON `nextcloud`.* TO 'nextcloud'@'localhost' IDENTIFIED BY 'password';

C. Nextcloud installation
1. Download and extra nextcloud tarball and place it in your weblocation of your virtualhost.
2. In your web-browser browse to the url you have setup for your nextcloud web location.
3. Enter admin username and password, select dabase type you want to use(if mysql it will ask you to enter the database name, user and password), hit next and it will ask you a few questions and then I think you hit finish or something like that.
4. Voila: Nexcloud is installed.
5. Don't forget to harden your installation if it's facing the internet.
 
Last edited:

Binary91

New Member
Credits
114
Hey!
Sorry for the late reply, I had a very very busy weekend.

@f33dm3bits: Wow, that is a detailed, step by step description! What have I done to deserve this? :D Thank you really much for this, I will test it exactly that way. I now a bit about MySQL from my holy website scripting times long time ago, but I'm excited how this will work with a Linux system.
I'm also very pleased that you linked a tutorial how to make my server safer, this is a very important aspect for me. For my Windows and Android devices, I'm using a (in my eyes) good security software, but I read a lot about Linux doesn't need a "real security program" because it is a much safer system anyway. I will have to read more about that when its time for it.
I think Raspian is based on Debian so it should be something like this.
Well, I just decided to try it with Raspbian, but I will use the Lite version. We will see if and how often I will fail setting that up :)

@Hansel Johnson Jr: Thank you for that. I'm not sure if it was my league, but I'll give it a try now. There is no way back, hardware is ordered haha.
That is funny, my colleague, the person that was the reason for me to start that, also uses a Synology NAS. He likes it and he can't confirm the often stated thesis that this commercial solution has weaker security because of less often system updates.. Maybe you can agree to that? Are you additionally using a RAID system for the NAS data (I don't know but maybe Synology systems always use RAID technology..)? I think, when the whole system is running one day, I will change (as stated earlier) the USB to a NAS SSD and maybe I will combine it with a RAID system to achieve secure permanent access in case of hardware failure. But that will only be important when there is traffic on my data. It is not interesting for me as long as I am the only person accessing my cool new server.
Please keep us informed as to your progress and questions.
I promise, I will (have to) do.
Unfortunatelly, I will have to wait until the ordered Raspberry and an additional keyboard is delivered next week. I'm sure I will give you an update of progress soon.
Who knows how many people you are helping make a decision. I dont know if you noticed but several people jumped in to help, that's why I like this Forum.
Exactly what I also recognized, I like this forum really much right now. Very fast and very detailed support, I see that you are really interested in supporting me, that is a good feeling. Thank you really much for that!

Kind regards,
Binary91
 
Last edited:

Members online


Latest posts

Top