Become a disk/filesystem guru

[dos2unix]

You can do just about all of this in a GUI program like gparted. If you like GUI's, by all means use these programs.
But sometimes, GUI's are not an option. If you have to something on a remote Linux system, ssh is often the only
access you have. While using gparted can make things easier, it's sometimes nice to know what it's doing under the
covers.

The other thing I will say here, is there are always more than one way to do something. So usually whatever example
I'm giving here... yes, there is another way to do it.

So.. how can I see what disks are in my computer?

I typically use a program called fdisk. It comes installed by default on many distro's.

 fdisk -l

Disk /dev/nvme0n1: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Disk model: INTEL SSDPEKKR256G8
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 8330A4F0-E6BB-4704-9C29-04885739E053

Device           Start       End   Sectors  Size Type
/dev/nvme0n1p1    2048    759807    757760  370M EFI System
/dev/nvme0n1p2  759808   7051263   6291456    3G Linux filesystem
/dev/nvme0n1p3 7051264 495695871 488644608  233G Linux LVM


Disk /dev/mapper/fedora_devnuc1-root: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/fedora_devnuc1-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/zram0: 7.63 GiB, 8188329984 bytes, 1999104 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/fedora_devnuc1-opt_esco: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/fedora_devnuc1-var: 30 GiB, 32212254720 bytes, 62914560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/fedora_devnuc1-home: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/fedora_devnuc1-data: 85 GiB, 91268055040 bytes, 178257920 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 7.5 GiB, 8053063680 bytes, 15728640 sectors
Disk model: UDisk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x39f5d30c

Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1        2048 15726591 15724544  7.5G  c W95 FAT32 (LBA)

That's a lot of information. I deliberately picked a difficult example to start with, because often times, you will see something like this.
fdisk -l ( lowercase L ) lists the disks I have in my system. In this case there are just two.

/dev/nvme0n1 and /dev/sda

What makes fdisk a little bit difficult to use, is that it also shows disk partitions. Usually, the most common disk types you will see
are /dev/sdX or /dev/nvmeXnX. Depending on what virtual machine software you are running, disks will sometimes show up as
/dev/vdX. If you have older hardware with spindle type disks, they might show up up as /dev/hdX. I do have a small portable computer
that has a MMC type drive that shows up as /dev/mmcX.

In the example above, I have a USB thumb drive plugged into one of the USB ports. These almost always show up as /dev/sdX.
Note: For sake of convention here, I'm using the character "X" as a placeholder. None of my drives are actually /dev/sdX but rather /dev/sda or /dev/sdb, and some cases /dev/sdc. Usually, the first drive the computer finds on its system hardware bus gets the letter "a" in the case of SSD and USB drives, or the number "0" in the case of nvme or mmc type drives.

The problem with USB thumb drives, is they play games with your mind and move around. For example.
My internal SSD drive shows up as /dev/sda. OK, so far so good. I plugged in a thumb drive to one of my USB ports and it shows up as /dev/sdb, I'm still good so far. and I plug in a third drive it shows up as /dev/sdc. all is as expected right?

But now turn my computer off, and tomorrow morning I turn the computer back on. All my drives got moved around.
My 16GB drive is now /dev/sdb and my 32GB drive is now /dev/sdc. What happened? Yes, the order you plug them in matters. But also
which port you plug them into can make a difference. This usually only matters if the computer is turned off. If the computer is already turned on, then the order I plug them assigns the drive letter. If the computer is turned off, the port I plug them into assigns the letter.
For example, USB port 1 will almost always get a lower letter than something plugged into USB port 2 or 3.

The point here is, always check. Don't assume the drive letters for a specific drive will always be the same. For internal fixed disk drives
on the system bus, you can assume they will always be the same.

.. to be continued.

 

[dos2unix]

All disk drives use something called a partition table. There are two main types used in Linux.
GPT (GUID Partition Table) and MBR (Master Boot Record). This wasn't always the case. Commodore and Apple II computers allowed you to format disks before Microsoft even existed. But they didn't really used partition tables. Everything was in one big partition of whatever size the disk was. MBR was the standard for many years. But now GPT has pretty much taken over. It's more flexible and allows for much larger disks ( over 2TB in size ). Now this gets a little political and weird, but this also has to do with whether your computer uses a legacy BIOS or UEFI firmware settings manager. There wasn't really an agreed upon standard for these things for a long time ( some of it is still up in the air, but it's gotten a lot better ). When IBM first came out with the UEFI standard, it wanted people to use GPT disk partitioning. Some computer manufacturers agreed, and some didn't. Everyone agreed GPT was better than MBR, but not everyone agreed on the standards. You often hear people say you have to use a GPT disk with UEFI or a MBR with Legacy BIOS. This isn't always true, and even when it is true, many BIOS's support something CSM. This lets me pick and choose which parts are UEFI and which parts are Legacy BIOS. The other thing, is even if I have a UEFI system that meets all the IBM standards, I can still format a disk as MBR and use it. Almost all my USB drives have a MBR partition table. I use them in my UEFI computer all day long with no problems.

Enough about all that, I guess I got distracted. For the most part, if you're using a legacy BIOS system, you probably want to format your disks using a MBR partition table. If you're using a UEFI based system, it's considered best practice to use GPT partition tables on your disks. For small external drives, (i.e. thumb drives ) it usually makes no difference at all.

MBR is limited to disks under 2TB in size. It also only supports a limited number of disk partitions. For example you can only have 4 primary partitions. GPT on the other hand supports up to 128 partitions. It also separates data and system type partitions by default. It's more robust because if one partition is corrupted, the rest of the disk is still usuable.

... to be continued.

 

[dos2unix]

Once you've decided on the partition table type. You then have to make the partitions. As mentioned earlier, we could do this with the gparted GUI, but since this is in the command line forum....

fdisk /dev/sda

Note I am using sda in this example. You should use whatever disk is appropriate. Don't forget to check!
Press "g" to create a GPT partition table, or "o" to create a MBR partition table.
Then press "w" to write the partition table to the disk.
NOTE:** This will wipe out everything on the disk! Make sure you have a backup of anything important before doing this.

This will kick you out of fdisk. Note if the disk was mounted before you ran these commands, you will have to
un-mount the disk before the computer will see the changes. Usually if it's USB drive, just pull it out, and push it back in.
OK, now we have a brand new partition table on our disk, lets make some partitions.

fdisk /dev/sda again.

fdisk /dev/sda

You can press "p" to print any information about any partitions. Since we haven't created any yet, you will only see the full disk. Now press "n" for new partition. How big should I make this partition? That's up to you, but you'll need a least one partition. You need to decide how many partitions you want, and how big you want each partition to be. If it's 64GB disk, you might only want one big 64GB partition. Or maybe you want two 32GB partitions? Or maybe three 10GB partitions? It's really up to you.
If you're making this all just one big partition, just use the defaults and press the [enter] key 3 or 4 times and you're done.
If you want more than one partition, you'll have to do a little math.

Let's say we want two 32GB partitions. Press n for new partition.
It should default to partition 1, since this is our first partition. So you can just press [enter].
Then it should default to the first available sector of your disk. Again, we will just accept the default and press [enter].
Next it wants the last sector. If you're a math wizard and you can determine the parition size by adding up the sectors, that's great. For the rest of us, we can just type +32G here. and press [enter]
Now we have our first partition. But we need to create another one.
Press "n" again another new partition. It should automatically default to 2, since it knows there is already a partition 1.
Press [enter], now it will ask for the beginning sector. It should automatically default to the first available sector here so you can press [enter] again. Now it wants to know the last sector. Since we are just making two partitions, we can accept the default and press [enter] here. Now press "w" to write the partitions to the disk. Congratulations, you just created two partitions on your disk.

Usually pressing "w" will kick you out of fdisk, but if not, you can press "q" for quit.

... to be continued .

 

[dos2unix]

I will briefly mention here, that even though you can create physical partitions on a disk. You can also create logical partitions as well. These are sometimes called "volumes". I will try to cover LVM (Logical Volume Manager) later. But for now, we will just go with the partitions we just created. Now that we have created the partitions, we still need to format them.

There is some crossover between disk formatting and partitioning. ZFS and BTRFS for example have their own way of doing things. We will cover those later. For now we will just use the standard Linux disk formats. The two most common formats are ext4 and xfs. ext4 usually has a little higher performance, but xfs is more robust and has better journaling which is often desired for servers in a data center. Linux has a tool called mkfs that we will use to format our partitions.

mkfs.ext4 /dev/sda1

Notice we no longer just use /dev/sda now we have a "1" after the a. This is because we are formatting partition number1.
We could also format the second partition with ext4 if we wanted to. But we can also format it with xfs.

mkfs.xfs /dev/sda2

This time I put a "2" after the a, because we want to format partition number2.

For USB thumb drives, you can use ext4 and xfs also if you want to. But if you're also using a Mac or Windows
and you want to be able to use the thumb drives in all of these computers, I recommend the two Microsoft formats.

mkfs.vfat -F32 /dev/sdb1

Fat32 is the most compatible format. It works with just about anything. Cell phones, car stereos, motherboard BIOS updates, Linux, Mac, ... pretty much anything. But it does have some limitations.

mkfs.exfat /dev/sdc1

exfat isn't quite as compatible with everything as fat32 is, but if you're transferring files and sharing drives between Linux and Windows, it's the recommended format. fat32 has a 4GB file limit size. exfat does not.

... to be continued.

 

[dos2unix]

OK, now I have my shiny new disk. I've created some partitions and I've formatted those partitions, great! Now what?

Well it depends. Most modern distros have something called "automount" . When you plug your USB drive it your computer, it will usually just automatically mount your thumb drive and you can see it in your file manager.

Not all distro's have this. Some have it, but it isn't enabled by default. If you're running Linux in console mode and not using a Xwindows GUI, chances are that automount isn't running. So then, how do I see my disk? How do I mount it manually?

I use fdisk again to list all my disks.

fdisk -l

That will list all the disks in my computer. I see /dev/sda is my internal drive. Most likely my external USB drive is probably /dev/sdb, but as mentioned earlier, you can't always be sure. It's best to check. The computer I'm typing this on, actually has two internal nvme type drives and 13 USB ports. It can be a pain to keep up with some days.

I see that disk I want to mount is /dev/sdc1, so how do we do this?

Well, for quick and dirty short term use, we could just do something like ...

mount /dev/sdc1 /mnt

And that will work... for now. But as mentioned earlier sometimes drive ltters get moved around. If I reboot my computer there's a good chance this won't be /dev/sdc1 anymore. It might be /dev/sdb1 now. So how do we fix that and make it more permanent?

First we need to know something called the block ID. How do we get that?

blkid /dev/sdc1

For USB drive the UUID can sometimes be short. Especially if the disk was formatted on a Windows system.

root@dev:~# blkid /dev/sda1
/dev/sda1: LABEL="MYSTUFF" UUID="8E6A-7CD1" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="39f5d30c-01"

but for internal disks, this come sometimes be a long string.

root@dev:~# blkid /dev/nvme0n1p2
/dev/nvme0n1p2: UUID="245f5c68-2240-4745-a477-0282aff94821" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="75c242da-df31-4816-a771-2d862cb66e4b"

The part you care about is the UUID. You can ignore everything else.
It works best if you can copy this somehow. It's usually difficult to remember all that.
Now we need to edit the /etc/fstab file, but before we do that, we need to decide where we want to mount this.

In Linux we usually create a directory somewhere, and then mount this disk to that directory.

mkdir /home/dos2unix/myotherdisk

You can name this anything that doesn't conflict with an already existing mount point.

Now edit your /etc/fstab file and add a new line at the bottom. ( it doesn't have to be at the bottom )

# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
UUID=6f778e32-8fb8-4f90-a8ff-ffbb99f8c65f /                       xfs     defaults        0 0
UUID=245f5c68-2240-4745-a477-0282aff94821 /boot                   xfs     defaults        0 0
UUID=0512-6120          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=4f276d5a-a78c-4848-8c6d-ddbba0ad10a3 /data                   xfs     defaults        0 0
UUID=859c5efc-e5e9-48b6-99a3-dd3b74ce8fc6 /home                   xfs     defaults        0 0
UUID=77e19a97-26ab-42aa-8cd7-b8b7abfa856b /opt              xfs     defaults        0 0
UUID=ea58b8b1-cace-4523-8fec-8a3d2155e3fc /var                    xfs     defaults        0 0
UUID=fc3a1d54-02da-4ad5-921b-12e9188451fa none                    swap    defaults        0 0
UUID=245f5c68-2240-4745-a477-0282aff94821 /home/dos2unix/myotherdisk  ext4   defaults  0 0

It is recommended to remove the double quotes on either side on the UUID. We then give the mount point.
We need to know what disk format the disk is using. It can be different from the other disks. Then just add
"defaults" and "0 0" at the end of the line. More about those later. Save your file and exit.

If you're running systemd, it's recommended that you run this command...

systemctl daemon-reload

..and that will force your system to re-read your fstab file.

Now we can mount the disk by simply typing
"mount /home/dos2unix/myotherdisk"

and it should now be mounted.
The nice thing about using the UUID instead of the /dev/sda1 path. Is that the UUID never changes.
The path can sometimes change (as explained earlier). This will remain persistent across a reboot.

... to be contined.

 

[dos2unix]

I should have mentioned this earlier. But disk partitions like to have a "type". It isn't always specifically required. But it is recommended. In fdisk ( gdisk is similar, but slightly different ) you press "t" for type.

Partition type or alias (type L to list all): L
1 EFI System C12A7328-F81F-11D2-BA4B-00A0C93EC93B
2 MBR partition scheme 024DEE41-33E7-11D3-9D69-0008C781F39F
3 Intel Fast Flash D3BFE2DE-3DAF-11DF-BA40-E3A556D89593
4 BIOS boot 21686148-6449-6E6F-744E-656564454649
5 Sony boot partition F4019732-066E-4E12-8273-346C5641494F
6 Lenovo boot partition BFBFAFE7-A34F-448A-9A5B-6213EB736C22
7 PowerPC PReP boot 9E1A2D38-C612-4316-AA26-8B49521E5A8B
8 ONIE boot 7412F7D5-A156-4B13-81DC-867174929325
9 ONIE config D4E6E2CD-4469-46F3-B5CB-1BFF57AFC149
10 Microsoft reserved E3C9E316-0B5C-4DB8-817D-F92DF00215AE
11 Microsoft basic data EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
12 Microsoft LDM metadata 5808C8AA-7E8F-42E0-85D2-E1E90434CFB3
13 Microsoft LDM data AF9B60A0-1431-4F62-BC68-3311714A69AD
14 Windows recovery environment DE94BBA4-06D1-4D40-A16A-BFD50179D6AC
15 IBM General Parallel Fs 37AFFC90-EF7D-4E96-91C3-2D7AE055B174
16 Microsoft Storage Spaces E75CAF8F-F680-4CEE-AFA3-B001E56EFC2D
17 HP-UX data 75894C1E-3AEB-11D3-B7C1-7B03A0000000
18 HP-UX service E2A1E728-32E3-11D6-A682-7B03A0000000
19 Linux swap 0657FD6D-A4AB-43C4-84E5-0933C84B4F4F
20 Linux filesystem 0FC63DAF-8483-4772-8E79-3D69D8477DE4
21 Linux server data 3B8F8425-20E0-4F3B-907F-1A25A76F98E8
22 Linux root (x86) 44479540-F297-41B2-9AF7-D131D5F0458A
23 Linux root (x86-64) 4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709
24 Linux root (Alpha) 6523F8AE-3EB1-4E2A-A05A-18B695AE656F
25 Linux root (ARC) D27F46ED-2919-4CB8-BD25-9531F3C16534
26 Linux root (ARM) 69DAD710-2CE4-4E3C-B16C-21A1D49ABED3
27 Linux root (ARM-64) B921B045-1DF0-41C3-AF44-4C6F280D3FAE
28 Linux root (IA-64) 993D8D3D-F80E-4225-855A-9DAF8ED7EA97
29 Linux root (LoongArch-64) 77055800-792C-4F94-B39A-98C91B762BB6
30 Linux root (MIPS-32 LE) 37C58C8A-D913-4156-A25F-48B1B64E07F0
31 Linux root (MIPS-64 LE) 700BDA43-7A34-4507-B179-EEB93D7A7CA3
32 Linux root (HPPA/PARISC) 1AACDB3B-5444-4138-BD9E-E5C2239B2346
33 Linux root (PPC) 1DE3F1EF-FA98-47B5-8DCD-4A860A654D78
34 Linux root (PPC64) 912ADE1D-A839-4913-8964-A10EEE08FBD2
35 Linux root (PPC64LE) C31C45E6-3F39-412E-80FB-4809C4980599
36 Linux root (RISC-V-32) 60D5A7FE-8E7D-435C-B714-3DD8162144E1
37 Linux root (RISC-V-64) 72EC70A6-CF74-40E6-BD49-4BDA08E8F224
38 Linux root (S390) 08A7ACEA-624C-4A20-91E8-6E0FA67D23F9
39 Linux root (S390X) 5EEAD9A9-FE09-4A1E-A1D7-520D00531306
40 Linux root (TILE-Gx) C50CDD70-3862-4CC3-90E1-809A8C93EE2C
41 Linux reserved 8DA63339-0007-60C0-C436-083AC8230908
42 Linux home 933AC7E1-2EB4-4F13-B844-0E14E2AEF915
43 Linux RAID A19D880F-05FC-4D3B-A006-743F0F84911E
44 Linux LVM E6D6D379-F507-44C2-A23C-238F2A3DF928
45 Linux variable data 4D21B016-B534-45C2-A9FB-5C16E091FD2D
46 Linux temporary data 7EC6F557-3BC5-4ACA-B293-16EF5DF639D1
47 Linux /usr (x86) 75250D76-8CC6-458E-BD66-BD47CC81A812
48 Linux /usr (x86-64) 8484680C-9521-48C6-9C11-B0720656F69E
49 Linux /usr (Alpha) E18CF08C-33EC-4C0D-8246-C6C6FB3DA024
50 Linux /usr (ARC) 7978A683-6316-4922-BBEE-38BFF5A2FECC
51 Linux /usr (ARM) 7D0359A3-02B3-4F0A-865C-654403E70625
52 Linux /usr (ARM-64) B0E01050-EE5F-4390-949A-9101B17104E9
53 Linux /usr (IA-64) 4301D2A6-4E3B-4B2A-BB94-9E0B2C4225EA
54 Linux /usr (LoongArch-64) E611C702-575C-4CBE-9A46-434FA0BF7E3F
55 Linux /usr (MIPS-32 LE) 0F4868E9-9952-4706-979F-3ED3A473E947
56 Linux /usr (MIPS-64 LE) C97C1F32-BA06-40B4-9F22-236061B08AA8
57 Linux /usr (HPPA/PARISC) DC4A4480-6917-4262-A4EC-DB9384949F25
58 Linux /usr (PPC) 7D14FEC5-CC71-415D-9D6C-06BF0B3C3EAF
59 Linux /usr (PPC64) 2C9739E2-F068-46B3-9FD0-01C5A9AFBCCA
60 Linux /usr (PPC64LE) 15BB03AF-77E7-4D4A-B12B-C0D084F7491C
61 Linux /usr (RISC-V-32) B933FB22-5C3F-4F91-AF90-E2BB0FA50702
62 Linux /usr (RISC-V-64) BEAEC34B-8442-439B-A40B-984381ED097D
63 Linux /usr (S390) CD0F869B-D0FB-4CA0-B141-9EA87CC78D66
64 Linux /usr (S390X) 8A4F5770-50AA-4ED3-874A-99B710DB6FEA
65 Linux /usr (TILE-Gx) 55497029-C7C1-44CC-AA39-815ED1558630

There are more, but this is all that fit on my screen in one shot. Most of the time, I use
11 - Microsoft Basic Data or 20 - Linux File System. Most of this is self-explanatory.
Use efi for efi, and swap for swap, etc... You need to select the type before you actually format the disk.
As with everything else in fdisk, press "w" to write the type to the disk.