BareMetal restoration of Linux (RedHat or CentOS) by ReaR.



Lets talk about two scenarios here.
1.       Consider that you have a physical system which has an OS crash recently. If the system is so critical, you need to restore it as soon as possible to control the business impact. How do you do that?
What we usually do is that, we will install a new OS, make necessary configuration in the system and application side and restore the data. This will take hell lot of time and our business cannot wait till that much time. What is an alternative option to it to reduce the downtime?
2.       You have a physical system and you need to migrate the OS to another system or create an exact replica of that system. How do you do that?
Same solution as what we discussed above? Is it going to take too much time to get it done?
What I suggest is, ReaR (Relax and Recover) . You can restore the exact configuration (replica) of a system from its previous backup.  

Let’s see how it is going to work here.
In this demonstration, We are going to use two systems with different configuration in terms of hard disks capacity and number of network interfaces. I am going to take a back of one system with 10GB root disk and restore it to an 8GB root disk. Since this is a demo, I used two VMs instead of physical systems. I have experience in restoring and migrating physical systems with Rear, so I am pretty sure that the demo in VMs would work exactly like how it works in physical systems.

Our Source system is LABSVR1 with below configuration.
Hard Disk/s : 1 with 20 GB in size used for RootVG.
                         1 with 5GB in size used for dataVG.
Network Card: 1 network card.




Our Destination system is LABSRV2 with below configuration.
Hard Disk/s:  2 disks with 10GB size (No data in it)
                        1 disk with 5GB size (No data in it and planned to install OS in it.)

Now I am going to install the necessary packages in the source system or the system which to be backed up. The packages are
1.       rear – Relax and Recover package, the base utility package for this solution.
2.       syslinux – To create bootable media for recovery.
3.       genisoimage -  as name indicated, to create ISO images.

These packages are available in AppStream  and BaseOS  repositories of CentOS/RHEL 8 systems. For other version of OS, you may need to download them from the external repositories.

# yum install rear syslinux genisoimage -y






The packages are installed now. We need to configure the backup now. The main configuration file for Rear is “/etc/rear/local.conf”. You can refer “/usr/share/rear/conf/default.conf” to get the default configuration of ReaR.

Before we start configuring the ReaR, let me tell you that, I am going to take the backup in to an NFS share. I have an NFS server in my environment which is called as “nfsserver” with a shared directory “/data/nfs_share”. The backup will be taken to that share in this demo.

Edit the file /etc/rear/local.conf and add the below details in it.

OUTPUT=ISO                                                                                   # ISO image as the bootable disk
BACKUP=NETFS                                                                                    # Take backup into a mounted directory.
BACKUP_PROG=tar                                                                    # Tar backup to be taken
BACKUP_TYPE=full                                                                 # To take the full backup
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"                 # gzip method
BACKUP_PROG_COMPRESS_SUFFIX=".gz"                               # .gx suffix for backup file
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' '/data/*' )               # directories to be excluded
BACKUP_OPTIONS="nfsvers=3,nolock"                                     # NFS mount options
BACKUP_URL="nfs://nfsserver/data/nfs_share"                 # Destination of backup
AUTOEXCLUDE_MULTIPATH=n                                                    # include multipath disks
AUTORESIZE_PARTITIONS=true                                                 # if source and dest  disks sizes are different
AUTOSHRINK_DISK_SIZE_LIMIT_PERCENTAGE=55           # % of shrink.


In case of Physical disks where you use UEFI partitions, you may need to add below entries as well in the configuration.

###option added for UEFI###
USING_UEFI_BOOTLOADER=1
UEFI_BOOTLOADER=( '/boot/efi/grub.efi' 'grub*.efi' '*.efi' )
ISO_MKISOFS_BIN="$( type -p xorrisofs || type -p mkisofs || type -p genisoimage )"
###



  
Now let’s take the backup of the system. Issue the below command in the system to take the backup.

# rear mkbackup -v




The backup has been completed now. The WARNING messages can be ignored as it is related to some open sockets which is expected as per this configuration. If we go to the NFS share, then we can see the backup files taken now.




There are two main files containing the backup and bootable media. The file “backup.tar.gz” contains the actual data backup and the file “rear-labsrv1.iso” is the bootable media used for the restoration. When we restore the system, we start booting it from the ISO image. The content of image will locate the backup and start restoring the system from there.

To schedule the regular backup, put the commands in a script, make it executable and schedule it in crontab like below.

[root@labsrv1 ~]# cat /usr/local/bin/make-rear-backup.sh
#!/bin/sh
set -x
/usr/sbin/rear mkbackup -v  > /tmp/make-rear-backup.out 2>&1
#
[root@labsrv1 ~]#

[root@labsrv1 ~]# chmod 700 /usr/local/bin/make-rear-backup.sh
[root@labsrv1 ~]#

[root@labsrv1 ~]# crontab -l
00 02 15 * * /usr/local/bin/make-rear-backup.sh
[root@labsrv1 ~]#

Now we are going to test the recovery of this system from this backup. We are going to restore it to an entirely different system with less capacity of root disk to see if the features AUTORESIZE_PARTITIONS &
AUTOSHRINK_DISK_SIZE_LIMIT_PERCENTAGE are working fine as expected.

We will boot the system from ISO image which we created by our backup. In my system I selected the bootable CD ROM as iso image which we have created by backup. I have copied the ISO image from the NFS to my /tmp directory of local system and pointed my CD drive to it.




While booting the system, you will get the below options. Select the first one and press Enter key to start recovering the system.



The system started booting used the below kernel now.




As we see here, the system will show up the interfaces available in it and we need to select one of it to configure the IP address of it. Per our configuration, the system will get assigned the source IP address, this may lead to a network glitch with the source server if we plan to close the system. If the source system is already down, then no need to worry on it.

I selected interface 2 and pressed the Enter key as shown below.



Now the system is configured with the IP address same as in the source system and prompted for login. Give “root” as the user and press Enter Key. This will not prompt for root password and will lead directly to the shell.



Limited commands will work here in this recovery shell. To see the IP configuration, use the “ifconfig” command.




Since my source system is still up and running, I made the interface down on the recovery system to control the IP conflict issues.

   
I am going to assign a new IP to this system before we start the recovery.



Since our backup is taken to NFS, the system will look at the backup file (xxx.tar.gz) in the same location. So, the network configuration to be setup before start recovering the system.

To start recovering the system, issue the below command:

# rear recover



In my destination server, I am trying to restore the OS from a 20GB root disk to 10GB one. In the source system /dev/vda is the root disk and the recovery is asking to select the destination disk. We are going to select 3 (/dev/sdc – 6GB) and press Enter key.

In the source system we have another disk /dev/vdb for /data mount point. The recovery is asking to select the destination for it. Selected option 2 which is /dev/sdb.



Now it is showing the mapping which we selected. Left side one indicates the source and the right one for destination. Like wise there are two disks we have selected for the restoration. Selecting the default option to confirm it.


Prompted for the reconfirmation for disk layout. If needed, we can edit it by selecting option 2 (will lead to a file edited by vi command.). In this case we are continuing with the default option.


Partition size has been reduced as shown here since the destination disk is with lesser capacity. Proceed with the selection of default option to start the recovery.




Disk layout has been created now, still it is editable. Check it and proceed with default option if everything looks good.




When we confirmed, the system started restoring the data from the backup file.



Recovery has been completed now. If you need to make any permanent changes to the system, we can edit the files and do it before reboot. The root file system is mounted under /mnt/local now.




As we found above, this system is having the source IP address by default. We need to change it by editing the file “/mnt/local/etc/sysconfig/network-scripts/ifcfg-en9”

Changed the IP address from 192.168.1.71 to 192.168.1.72 as shown below. If you want to change the root password, you can change it here too.



Now reboot the system and check all the configurations after booting. Root password will be the one same as its source system.

System has been rebooted now with below details.

1.       System has been restored with IP 192.168.1.72

[root@labsrv1 ~]# ifconfig |grep 192.168.1.7
        inet 192.168.1.72  netmask 255.255.255.0  broadcast 192.168.1.255
[root@labsrv1 ~]#

2.       All the mount points have been created.

[root@labsrv1 ~]# df -Phx tmpfs
Filesystem                 Size  Used Avail Use% Mounted on
devtmpfs                   901M     0  901M   0% /dev
/dev/mapper/rootvg-rootlv  6.5G  1.6G  5.0G  24% /
/dev/mapper/datavg-datalv  2.0G   47M  2.0G   3% /data
/dev/sdc1                  488M  123M  335M  27% /boot
[root@labsrv1 ~]#

3.       No data is available in /data directory since we excluded it from backup [BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' '/data/*' )]

[root@labsrv1 ~]# ls -l /data |wc -l
1
[root@labsrv1 ~]#

With this we conclude that the restoration is successful.




Comments

Post a Comment

Popular posts from this blog

AIX Firefox Installation

Overview           Mozilla Firefox for AIX is an Open Source Web Browser. It delivers helpful new features and continues to lead the way in on-line security. It implements technologies like the Gecko layout engine, and supports Web standards or draft standards like HTML, XHTML, XML, CSS, DOM, and many more . Version of Firefox Mozilla Firefox with version 3.5.13.1 can be installed in AIX 5.3 and later. Installation Procedure The following rpm s are the pre requisites for Firefox(64 bit). These packages will be available from the AIX Toolbox for Linux Applications CD/DVD. Before installing the prerequisites check the availability of 150MB free space on the /opt file system. atk-1.12.3-2.aix5.2.ppc.rpm cairo-1.8.8-1.aix5.2.ppc.rpm expat-2.0.1-1.aix5.2.ppc.rpm fontconfig-2.4.2-1.aix5.2.ppc.rpm freetype2-2.3.9-1.aix5.2.ppc.rpm gettext-0.10.40-6.aix5.1.ppc.rpm glib2-2.12.4-2.aix5.2.ppc.rpm gtk2-2.10.6-4.aix5.2.ppc.rpm libjpeg-6b-6.aix5.1.ppc.rpm libpng
Read more

How to increase swap size on CentOS 8 / RHEL 8

Swap space is the second type of memory in modern Linux systems. The primary function of swap space is to substitute disk space for RAM memory when real RAM fills up and more space is needed. Linux uses swap space to increase the amount of virtual memory available to a host. It can use one or more dedicated swap partitions or a swap file on a regular filesystem or logical volume. Now we are going to check: •              Current swap space in the system. •              How to identify swap volumes in the system. •              How to increase the size of swap volume. •              How to add one more swap device to system. This documentation is based on the test done in a CentOS 4.18.0-80.el8.x86_64 machine. The same procedure can be applied on RHEL 8 as well. We follow the LVM method to manage the volumes here though a normal partition also can be used for swap. We will see both the options here when we add another device as a swap volume. How to check the cu
Read more

AIX Commands

Usefull Commands for AIX =================== bootinfo –r     shows available RAM in KB lsattr –E l sys0 –a realmem    shows available RAM in KB rmss -c 512    To change the memory size to 512 MB rmss -r          resets the memory size to the original value lsattr  -El  en0     shows en0 driver params lsattr  -El  ent0     shows ent0 HW params lsattr -El rmt0     shows tape params lscfg -vpl rmt0     shows information about the tape drive lsattr  -El  sys0     shows system type, firmware, etc  driver params lscfg –v     lists all system HW configuration lsdev –C sscsi     list all scsi devices lsdev –C spci     list all pci devices lsparent –Ck scsi     list all scsi adapters cfgmgr     To onfigure devices lsdev –Cc disk      Shows all disks lsdev –Cc tape               Shows all tapes cfgmgr -vl < device > –v    Specifies verbose output. The cfgmgr command shows information about what it is doing. cfgmgr -vl < device >      Here <device> is a particular dev
Read more

How to configure NTP in RHEL8 or CentOS 8 by chronyd.

Network Time Protocol is used in systems to get the clock synced between them. All systems will sync their clock with NTP server is they are configured to do so. In RHEL8 or CentOS 8 it is achieved through “ chronyd ”. The “ chronyd ” daemon can be managed by the command “ chronyc ”. Below steps involve in configuring NTP through chronyd in linux systems. ·         Install the necessary packages. ·         Configure chrony service to sync with NTP server. ·         Set NTP synchronization. ·         Enable necessary firewall rules. ·         Enable and start chronyd service. Installation of chrony on RHEL8 or CentOS8. The below command helps to install chronyd in RHEL/CentOS 8. [root@system1 ~]# yum install chrony -y Chrony (NTP) server configuration The main configuration file for “ chronyd ” is /etc/chrony.conf . [root@system1 ~]# vim /etc/chrony.conf Below the minimum details to be set in the chrony.conf file. ·         pool 2.centos.pool.ntp.org iburst   # The name of the pool wh
Read more

Upgrade curl 7.29 to 7.69 in CentOS 7 or RedHat 7

By default we can see 7.29 as the latest version of curl in CentOS or RHEL7. Here we are going to see how we can upgrade it to version 7.69. We use city-fan.org repo for the upgrade of curl. Lets see how to install and enable this repo and upgrade the curl package. Install the "city-fan.org" repo # rpm -Uvh http://www.city-fan.org/ftp/contrib/yum-repo/city-fan.org-release-2-1.rhel7.noarch.rpm Enable the repo "city-fan.org" in configuration file. Open the file /etc/yum.repos.d/city-fan.org.repo and enable the repo "city-fan.org" # vim /etc/yum.repos.d/city-fan.org.repo Upgrade the "curl" package from this repo. # yum update curl -y Once the update is done you can check the version of curl package by "rpm" command. # rpm -qa |grep curl libcurl-devel-7.69.1-1.1.cf.rhel7.x86_64 python-pycurl-7.19.0-19.el7.x86_64 libcurl-7.69.1-1.1.cf.rhel7.x86_64 curl-7.69.1-1.1.cf.rhel7.x86_64 #
Read more

How to find big files in Linux, Unix, AIX

      Most often we see that the utilization of file systems grows up and we need to do some sort of housekeeping in the file system. Mainly we identify which file consumes more space in the file systems. Sometimes we see chunk of files with small size of single file with more size consume more space in the file systems. In either situation our goal is to find out what is consuming more space, identify the necessity of those files in the file system and decide to housekeep the file system by removing the unwanted files. Here are some ways to find the big files in a directory. People mostly use the below steps to identify the big files in a directory. 1.        By using “ du ” command. 2.        By using “ find ” command By using “du” command. Eg .   # du -a / Here “ du ” command will look for all the files in the directory including filesystems mounted on directories under “ / ” directory and list them with their size on kilo bites (KB). If you want to sort
Read more

How to enable VNC server (get GUI remotely) in CentOS 7, RHEL 7, CentOS 8 and RHEL 8 servers.

Minimum requirements to connect a server remotely to access GUI are: 1.        Display server running in your workstation (I used MobaXterm. Software like Xming and VNC clients are also can be used.) 2.        Server running with GNOM workstation and VNC server. As I mentioned above, I have “MobaXterm” installed in my laptop that will work as a display server to connect my CentOS or RHEL server remotely. You may download it from here . It is very easy to use. If you have restrictions to install it in your laptop, then you can use the portable version of it. It can be just executed like PuTTy. To make sure my CentOS or RHEL server running with Desktop software, I use below commands to install them. In CentOS 7 or RHEL 7: [root@labsrv1 ~]# yum groupinstall “GNOME Desktop” In CentOS 8 or RHEL 8: [root@labsrv1 ~]#   yum groupinstall "Server with GUI To install the VNC server, use the below command. Install its dependencies as well, if it prompts for the same.
Read more

YUM configuration in AIX.

    In earlier days all RPM packages in AIX were installed by the command “rpm”. The limitation for RPM command is that the difficulty to resolve the dependent packages.   Like other Linux operating system like RedHat, CentOS etc. there is a possibility to setup YUM to install packages in AIX. The below procedure will take you through the steps of configuring YUM in AIX (7.1) systems.     After having many trial and error experiments, the final solution came up with YUM configuration on the system, which is already installed with any RPM packages, we may need to follow the steps like: 1.        Take a backup of system (mksysb) 2.        Have the list of currently installed RPM packages. 3.        Remove the existing RPM packages. 4.        Configure YUM as described below. 5.        Install the previous packages again. Let’s start with the configuration now. Assuming that all required backups are in place to do a roll-back in case of any worst-case scenario.
Read more

Remote X11 forwarding for AIX in command line.

Most of the Unix administrators love to work with command line and it is the most convenient way to work with Unix systems. There were occasions where we had to invoke the GUI for some applications to work with it like, invoke a Firefox browser to browse the internet through Unix systems, install an application in GUI mode etc. Here I am going to describe how can we address such situations by invoking GUI over CLI. Assumptions:           You have an AIX system with remote connection through ssh.           You need to invoke the GUI of an application through Xming or MobaXterm – a DISPLAY server. X Window in AIX server Minimum installation for X Window includes below filesets: X11.base.lib X11.base.rte X11.apps.config X11.apps.clients   (optional to run xclock program – in this case, just for testing purpose) Install the above-mentioned packages to invoke the GUI remotely through command line. Connect through SSH X11 port forwarding On the
Read more

How to manage Linux systems through web browser (cockpit for Linux): CentOS 7, CentOS 8, RHEL 7, RHEL 8, Fedora, Ubuntu and Debian systems.

    Cockpit is a web accessible interactive admin interface for Linux machines. Cockpit can usually be accessed on port 9090 of the machine it's installed on.     The cockpit-ws web service listens on port 9090 and is started on demand by systemd . The Cockpit web service authenticates the user, loads Cockpit into the browser, and starts cockpit-bridge in a Linux user session. The cockpit-bridge provides Cockpit in the web browser with access to the system APIs. It does this over its standard in and standard out.     Cockpit can be installed in the following Operating systems. Please not that this list is where I installed and tested, that does not mean that it will not work on other Operating Systems. RHEL 7 & RHEL 8 CentOS 7 & CentOS 8 Ubuntu Debian     Cockpit is a very helpful tool which can be used to access the Linux servers through a web browser and do all kind of administration tasks. Some of the main features of Cockpit are:
Read more