It is very important to understand and manage data storage under Linux. There are several key concepts and commands for working with bulk storage.
If you are completely new to setting up storage under Linux, then this article is for you.
We’ll discuss it here:
- Locking the names of storage devices and /dev directories
- Hard disk partitioning
- Create and mount file systems
- Logical volume manager (LVM)
- Server with Linux operating system, preferably CentOS 7 or higher.
- At least two free storage devices
Each physical server or virtual machine has multiple directly connected blocking devices. These can be hard disks, SSDs or USB sticks.
Whenever a new lock is added to the system, the existing capacity is not immediately accessible. This means that files and folders cannot be created directly on the device.
A basic configuration is needed to free up the memory. It includes partitioning, file system creation and mounting.
Before discussing the commands related to these operations, let’s talk about the /dev-directory and the naming of the locking systems.
Memory unit name
Linux treats storage devices like files. These files are stored in the /dev-directory. It is important to understand this concept because most backup commands use the files in this folder as an argument. Many system programs also use these files as interfaces to I/O operations or I/O operations with blocking devices.
Each time a new lock is placed or discovered, the system automatically creates a new corresponding file in the /dev-directory.
The name of this file depends on a number of things. This mainly depends on the type of device bus and the order in which the device is installed in relation to other devices.
If the bus is type z. B. SCSI or SATA, the first half of the file name is sd. The other half is a letter assigned in alphabetical order according to the order in which the device is detected.
The first, second and third SCSI devices in the system are called sda, sdb and sdc respectively.
Devices with different bus types have slightly different naming rules. For NVMe, a modern type of bus, the first part of the device name is nvme. In addition, a bus number is inserted in the middle. The last part is a letter and a number indicating the order of recognition. For example: nvme0n1 or nvme0n2.
We can list the contents of the /dev-directory to get an overview of all block devices present in the system.
There are many files including sda, sdb, sda1, sda2, nvme0n1 and nvme0n2.
Note that /dev also contains other files that are not related to storage devices.
However, sda1 and sda2 are not real locking systems. They form the components discussed below.
What is separation
Partitioning is the division of a block device into logical block devices called partitions. This is the first operation that must be performed before you can use the player.
To list all disks and partitions in the system, use the command
lsblk shows a clear tree structure of the hard drives and their separations. In addition, the Output Type column indicates whether the row represents a disk, partition, or LVM. (Type LVM is a comprehensive storage system that will be discussed at the end).
There are basically two marking systems: Master Boot Record (MBR) and GUID Partition Table (GPT). The MBR is an outdated separation system, but is still in use. It is limited to a maximum hard disk size of 2 TB. GPT partitioning is preferred because it removes the 2TB limit and offers other improvements.
Create partition wall
The fdisk command is used to create MBR partitions and the gdisk command is used to create GPT partitions. The two teams are interactive and very similar.
These commands create a partition table on the first sectors of the disk. (With GPT, the partition table is also stored on the last sectors of the hard drive).
This table contains all information about the sections. For example, it contains the addresses of the sectors where each partition begins and ends. (A sector is the minimum size of the read/write blocks used by the drives, typically 512 bytes).
Let’s take an example. We have a CentOS VM with a 20 gigabyte disk called sdb. Note that the name of the player on your device may be different. Make sure you have selected the correct drive. We make a new GPT partition table and two bulkheads of 2 GB each.
Memory commands are usually reserved for the root user. So log in as root and use the following command:
The above output shows that there is not yet a partition table on the sdb disk. Press the question mark ? to see all available commands.
We use the command n to create a new partition, which also creates a new partition table. The command queries the parameters one after the other:
- Lot number : Since there are no partitions on the disk yet, type 1 and press Enter.
- First sector: The distribution starts from this sector. Leave the field empty and press the Enter key. The default value, i.e. the first free sector, is selected automatically.
- Last sector: Here we indicate the size of the desired partition. Type +2G, then press the Enter key. Note that we can also specify the sector number, but this requires an additional calculation to determine the exact sector that gives us a 2GB partition.
- File Partitioning System : There are many file systems. Back up the standard Linux file system by leaving it empty and pressing Enter.
Repeat the same process to create another 2 GB partition. Then type w then y to write, confirm and close everything.
To see the changes on the sdb device, use the following command:
We see that we have an sdb disk with a total size of 20 GB and two partitions of 2 GB each. There is still 16 GB of free space left on the sdb disk where we can make more partitions later on.
What is a file system?
The file system is built on the partition. A partition without a filesystem can be seen as a long series of sectors. A file system is a logical abstraction layer that allows you to create a hierarchy of files and folders.
There are many types of file systems. The most commonly used under Linux are ext4 and xfs. xfs is the standard file system on modern Red Hat and CentOS distributions, while Ubuntu uses ext4.
Creating a filesystem
Let’s continue with our example. We create an xfs file system on the already created sdb1 partition. The command to create a filesystem is mkfs. It uses the -t argument to specify the type of file system.
mkfs -t xfs /dev/sdb1
Assembly of the file system
After creating the file system we are one step away from using the partition location. This step is installed.
File system mapping maps a file system created on a partition in the file hierarchy of the system. Allows you to select a folder in the file hierarchy to host the file system built on the partition. This folder is called the attachment point.
Let’s create a new mount point named data and mount the sdb1 file system in the /data directory.
To confirm that the mount was successful, we can list all mounted file systems with the following command:
mount /dev/sdb1 /data
The last line indicates that the file system is mounted in the /data directory. It can now be used for storing files. All files created in the /data directory are stored in the sdb1 partition.
Note that by default the xfs file system occupies part of the available metadata space, which explains the 33 MB used.
However, a manually created mount with the mount command will not survive a reboot. In most cases, the connection must be continued after a restart. To mount the partition sequentially, the mount parameters must be written to /etc/fstab.
The standard /etc/fstab contains all the permanent fixings the system needs. In addition, the root user can add internal elements that the user needs. Let’s take a look at the contents of this file:
Each line in this file that does not begin with the # character corresponds to a mount created automatically during the boot process. Each line contains 6 fields:
- Locking : Device file, e.g. B. /dev/sdb1. Note that there are other device identifications, such as. B. The UUIDs exist.
- Location of the attachment.
- Type of file system: xfs, ext4, vfat… etc.
- Mounting Parameters : These parameters are passed to the mount command at startup. Normally the default value is selected, unless several mounting options are required.
- Backup : This field takes a value of 1 to enable file system backup at startup, or 0 to disable it. The backup is done by the dump program. Most systems do not come with a standard dumping program installed, so this field is usually set to 0.
- Checking the file system : This field takes a value of 1 to enable the file system check at startup, or 0 to disable it. The file system is checked with the fsck command, which reports any inconsistencies or errors in the structure of the file system.
Let’s add a line to permanently mount the /dev/sdb1 partition under the /data directory. You can edit the /etc/fstab file directly or simply add a line at the end of the file with the following command :
echo /dev/sdb1 /data xfs standard 0 0 >> /etc/fstab
Use the command cat again to check if the string has been added successfully.
If a new line is added, the assembly process does not take place until the system is restarted. To make sure the mount configuration is correct without rebooting, use the mount command with the argument -a. ( -v for verbs)
The output signal indicates successful fixation. (Ignore SELinux message)
The command mounts any line of /etc/fstab that has not yet been mounted. After editing /etc/fstab it is always recommended to use mount -a. Because if there is a typing error or an error in the file, the command reports the error and ends it. If there is a typing error or an undetected problem in this file, the system will not boot correctly on the next reboot.
In some cases it can be problematic to use a device file, for example. B. /dev/sdb1, in the first field of /etc/fstab. If the locks are removed or placed in a different order, the names in /dev will be changed automatically.
For example, the blocking device can become sdb sdc and vice versa. So the sdb1 partition becomes sdc1. /etc/fstab is a static file. Without knowledge of the name change, the system always assembles sdb1 to /data, which results in incorrect assembly.
To avoid this problem, the first field in /etc/fstab uses the Universal Unique Identifier or UUID of the file system instead of the device block file.
Each file system is assigned a UUID when it is created.
To determine the UUID of the file system at /deb/sdb1, use the following command:
The first UUID in the blkid output is the file system. The file system UUID does not change even if the device name is changed. Therefore, using the file system UUID in /etc/fstab ensures that the same file system is always mounted under the correct directory.
Edit /etc/fstab and replace the /dev/sdb1 line with the following:
UUID=cba91aba-318c-be9a-a280fd185afc /default xfs data 0 0
Use the chat command to make sure it is updated.
After editing /etc/fstab mount -a, use it to mount file systems. If the names of the devices change, the media will not be confused. The system always links the file system with the correct UUID at the correct mount point, regardless of the partition name.
Logical volume manager (LVM)
What is MVL?
LVM is a Linux storage technology that introduces an additional layer of abstraction on disks and partitions to provide advanced features such as logical volumes, thin provisioning, snapshots and software RAID.
Most modern distributions support LVM. It overcomes the restrictions imposed by bulkheads. For example, enlarging or reducing partition walls is very delicate, error-prone and sometimes impossible. With LVM it is easy to change the volume size.
LVM consists of three main layers.
- Backup devices such as hard disks or partitions are called physical volumes (PVs).
- LVM groups all PVs in one or more storage pools called Volume Group (VG).
- The available memory in VG is used to create virtual blocks called Logical Volume (LV).
The LV is used to store data. This device is similar to a conventional locking system. Once a LV has been created, the file system must be built over it and then mounted. The same steps for creating and mounting the file system for a normal partition apply to LV.
LVM offers flexibility in storage management. The size of the LV can be adapted to your needs. In the following example, 3 readers of 1 TB are combined into one VG and 2 VL of different sizes are created: 1 TB and 2 TB.
Once all actuators have been merged into one VG, multiple combinations of VGs can be made. For example, LV can occupy all available space on VG. In this case, the LV applies to all hard drives. When LV is about to be filled, more readers can be added to VG, and then LV can be expanded. When the LV is no longer needed, it is removed and its location returned to its own LV.
Create a volume group (VG) and a logical volume (LV)
Let’s look at an example of LVM configuration. We have a server with 2 disks and 1 partition for free. The isbloc output is shown below.
In our example a VG is created. It contains nmvme0n1 (8 GB), nvme0n2 (8 GB) and sdb2 (2 GB) partitions. The VG is used to create a 15 GB LV. LV contains an xfs file system and is mounted in /data2.
The first step is to mark the disks as physical volumes (PV). This is done in batches using the pvcreate command as shown below:
pvcreate /dev/sdb2 /dev/nvme0n1 /dev/nvme0n2
The second step is to create a VG called data_vg using the VPs you just marked.
vgcreate data_vg /dev/sdb2 /dev/nvme0n1 /dev/nvme0n2
To display information about available VGs, we can use two commands: vgs, which displays short information about VGs, and vgdisplay, which displays more information.
vgs shows the GVs per line, the number of VPs per GV, the available space and the free space per GV. During the installation phase, the system automatically creates a centos VG.
You will notice that the newly created VG data_vg has 17.99 GB free memory instead of 18 GB, which is the sum of the memory available on all PVs.
In fact, the size of the VG is often slightly different from the amount of storage space available on the PV. In LVM, the PV elements are indeed divided into blocks called physical expansions (PE). The standard EP size is 4 MB. These blocks are combined to form a VG. Therefore, the size of VG is always a multiple of 4MB, which explains why VG is not quite 4MB.
Note that the size of the PE can be changed when creating the VG, but in our example we left the default value.
Create an LV with the name data_lv with the command lvcreate. The size of the LV, the name and the parent VG are taken as arguments.
lvcreate – Size 15G – Data name_lv data_vg
To display the list of available LVs, use the lvs or lvd display commands.
LV is a logical blocking device, so it must have an associated file. When a VG is created, a folder named VG is created under /dev, in which a file is saved for each created VG. In our example, the file at data_lv /dev/data_vg/data_lv. We can check this by specifying the content of /dev/data_vg.
Now we create a filesystem on data_lv with mkfs.
mkfs -t xfs /dev/data_vg/data_lv
Finally we create a /data2 mount point and mount data_lv. Then we check the attachment with the df command.
mount /dev/data_vg/data_lv /data2
The LV is now mounted under /data2 and we can use it for data storage. For lack of memory, the lvextend command can be used to add memory to the LV from the available VG memory. If we also want the VG to end, we can use the vgextend command to add a disk or partition to the VG.
In this article we have familiarized you with the basic concepts and commands for working with memory under Linux. Important points:
- Linux treats bulk storage devices as files stored in the /dev directory.
- The filename in /dev depends on the properties of the player
- To use the hard disk space, you must first partition it, create a file system, and then mount it.
- The /etc/fstab file is used to create mounts that survive a reboot.
- LVM is a storage technology that combines all stocks in a single pool.
- Important commands: lsblk, gdisk, fdisk, mkfs, blkid, mount, df, pvcreate, vgcreate, lvcreate.
LVM is briefly described in this article. It supports more advanced features such as RAID, snapshots and thin provisioning, which are beyond the scope of this article.
The pages containing the manuals and distributors’ documentation are a good source of information to read in order to deepen your knowledge of storage.
With the basic knowledge covered in this article, you can understand and configure more advanced storage technologies under Linux. For example, Network File System (NFS), iSCSI, Virtual Disk Optimizer (VDO), Stratis and many others.
1.Veryfying the Volume Availability2.Formatting the Block Volume3.Mounting the Block Volume,1.Veryfying the Volume Availability,2.Formatting the Block Volume,3.Mounting the Block Volume