Linux Essentials


Mods 2 - 9

CH 1

Linux is Everywhere

  • Linux jobs are everywhere, there is a shortage of Linux talent and, Linux skills are in demand in just about every industry and job category on the planet

Linux is Open Source


  • Historically, most software has been issued under a closed-source license
  • This means that you get the right to use the executable program or machine code, but cannot see the source code
  • The development of Linux closely parallels the rise of open source software
  • The open-source philosophy is that you have a right to obtain the software source code and to modify it for your own use.

Linux Has Distributions


  • a Distro refers to the Linux kernel, tools, and suite of applications that come bundled together
  • Take Linux and the GNU tools, add some user-facing applications like a web browser and an email client, and you have a full Linux system
  • There are distros suited to every imaginable purpose
  • There are distros that focus on running servers, desktops, or even industry-specific electronics design or statistical computing

Linux Embraces the CLI


  • There are two basic types of interfaces available that allow you to interact with the OS
  • The typical computer user today is most familiar with a graphical user interface (GUI)


    • In a GNU, applications present themselves in windows that can be resized and moved around
    • There are menus and tools to help users navigate
  • The second type of interface is the CLI, a text-based interface to the computer


    • The CLI relies primarily on keyboard input

Intro to OSs

Operating System


  • Software that runs on a computing device and manages the hardware and software components that make up the system
  • It also schedules programs to run and provides services to users or programs (ex: Print)
  • Common abbreviated as "OS"
  • Users today have a choice between three major OSs:


    • Microsoft Windows
    • Apple macOS
    • Linux
  • Only MS Windows is based on proprietary code that is not Unix or Linux based

Decision Points


  • Role: Accessed by one user directly (desktop) or many users remotely (server)?

  • Function: Does it need to run specific software? What is the skill set of users?

  • Life Cycle: What is the service lifetime? OS types have different release cycles and maintenance cycles for support and updates

  • Stability: Are OS releases beta (not tested "in the wild") or stable (tested)?

  • Compatibility: Is it backwards compatible as in is it compatible with software made for earlier versions?

  • Cost: Important factors for new systems
    • MS has annual license fees
    • Apple does not charge annual fees but only works on Apple hardware
    • There are multiple Linux providers wo offer enterprise support and although the software is free, support is not

Microsoft Windows


  • Offers desktop and server versions
  • Slow-release cycle (3-5 yrs), Long maintenance cycle
  • Emphasis on backward compatibility
  • Runs a GUI
  • Improved scripting and management abilities are being developed to compete with Linux

Apple macOS


  • Runs only on Apple hardware
  • Server version adds packages to the desktop version to aid in management and sharing
  • UNIX certified
  • New major releases every 18-24 months

Linux


  • Unique in that after choosing Linux you must choose a distribution
  • Different distros focus on different use cases, i.e. desktop, server, scientific, network
  • Some distros offer commercial support, most is volunteer based

Linux Decision Points


  • Role: Distros available for variety of systems; commercial for servers and desktop, specialized to repurpose computers, embedded systems, etc

  • Function: Distros can be chosen based on purpose of usage or security needed

  • Life Cycle: Most distros have major and minor updates cycles. Some Linux releases have long-term support (LTS) (5+ yrs, 13 yrs for SUSE LTS)

  • Stability: Some distros offer stable, testing, and unstable releases

  • Compatibility: Distros are zero cost. Depending on need, paring for support may be worthwhile. Enterprise usrs can pay for support or attempt self-support

Linux Distros

Debian


  • Community effort that promotes use of open-source software
  • Invented its own package management system (apt) based on the .deb file format
  • Ubuntu is its most popular derived distro, which has variants for desktop, server, and applications. Ubuntu also offers an LTS version
  • Linux Mint is a derivative of Ubuntu also offers an LTS version

Android


  • Provides a platform for mobile users.
  • Lacks traditional GNU/Linux packages to make it compatible with desktop
  • Sponsored by Google

SUSE


  • One of the first Distros
  • OG derived from Slackware
  • Contains proprietary code and is sold as a server product. Some modules or addons may contain proprietary code
  • Sold as a server product although a Workstation version exists
  • OpenSUSE is a completely open, free version with multiple desktop packages

Other Linux Distros


  • Raspbian is a linux distro designed to run on Raspberry Pi hardware
  • Linux Form Scratch (LFS) consists of an online book, source code, and instructions for building a custom Linux distro
  • Can be used as learning tools

Red Hat


  • Focuses on server applications like web and file sharing
  • Releases Red Hat Enterprise Linux (RHEL), a stable distribution with long releases cycles
  • Sponsors the Fedora Project, a personal desktop with latest software
  • CentOS is a free version of RHEL software which does not offer support
  • Scientific Linux is a specific use distro based on Red Hat

Linux is a Kernel


  • Linux means the kernel of the system, which is the central controller of everything that happens on the computer.
  • Linux is a combination of software called GNU/Linux, which defines the OS


    • GNU is the free software that provides open-source equivalents of many common UNIX commands
    • The Linux part of this combination is the Linux kernel, which is the core of the OS
  • The story of Linux begins with UNIX, an OS developed at AT&T Bell Labs in the 1970s

  • Linux started in 1991 as a hobby project by Linus Torvalds, a Finnish born computer scientist at the Uni of Helsinki
  • The GNU Project was developed Linux by Richard Stallman in 1983
  • Linux programmers were able to incorporate the GNU tools to provide a complete OS

Working in Linux

Navigating Linux

The Desktop


  • To be a Linux sysadmin, you must be comfortable with Linux as a desktop operating system and have proficiency with basic Info and Comms Tech (ICT) skills
  • Sysadmins use Linux to manage servers, assist users with config issues, recommend new software, and update documentation
  • After familiarizing oneself with the Linux GUI, or desktop, the next step is learning how to perform tasks from the command line

The CLI


  • the command line interface (CLI) is a simple text input system for entering anything from single word commands to complicated scripts
  • On systems that boot to a GUI, there are two common ways of accessing the CLI, a GUI-based terminal, and a virtual terminal
    • Brose to the terminal application from the application menu
    • A virtual terminal can be run at the same time as a GUI but may require the user to log in via the V-terminal before they can execute commands

Applications


  • the kernel decides which program gets which block of memory, it starts and kills applications, and it handles displaying text or graphics on a monitor
  • Application make requests to the kernel and it return recieve resources, ie memory, CPU, and disk space
  • The kernel also handles the switching of applications, a process known as multitasking
  • There are a large variety of application types such as word processors, web browsers, and email clients, and more
  • A process is just one task that is loaded and tracked by the kernel
  • An application may even need multiple processes to function, so the kernel takes care of running the processes, starting and stopping them requested, and handing out system resources

Major Applications


  • Linux software generally falls into one of three categories
    • Server Applications: The purpose of this software is to server info to other computers, called clients
    • Desktop Applications: Web browsers, text editors, music players, or other applications with
    • Tools: a loose category of software that exists to make it easier to manage computer systems

Server Applications


  • Linux excels at running server applications because of its reliability and efficiency.
  • One of the early uses of Linux was for web servers.


    • A web server hosts content for web pages, which are viewed by a web browser using the HyperText Transfer Protocol (HTTP) or its encrypted flavor, HTTPS
  • There is a growing demand for Private cloud sever software that can be deployed and administered internally


    • the ownCloud project provides software to store, sync and share data from private cloud servers.
    • the Nextcloud project also provides private cloud software

Desktop Application


  • The Linux ecosystem has a wide variety of desktop applications
  • Email:
    • The Mozilla Foundation came out with Thunderbird, a full featured desktop email client that connects to a POP or IMAP server
    • Other notable email clients are Evolution and KMail which are the GNOME and KDE project's email clients.
  • Creative:
    • Blender, GIMP (GNU Image Manipulation Program), and Audacity which handle 3D movie creation, 2D image manipulation, and audio editing respectively
  • Productivity:
    • LibreOffice is a fork of the OpenOffice application suite
  • Web browsers:
    • the Mozilla Firefox and Google Chrome browsers are open-source web browsers that are fast, feature rich and have excellent support for web developers

Console Tools


  • UNIX has considerable overlap between the skills of software development and sysadmin
  • The tools for managing systems have features of computer languages, such as loops, and are used extensively in automating systems admin tasks.
  • Basic familiarity with programming is required for competent systems administrators
  • Shells:


    • Users interact with a Linux system through a shell, which accepts commands to execute
    • Linux offers a variety of shells to choose from such as; the Bourne shell, C shell, the Bourne Again (Bash) shell, the tcsh, the Korn shell (Ksh), and the zsh
  • Text editors:


    • most Linux systems provide a choice of text editors which are commonly used at the console to edit configuration files.
    • The two main editors are Vi (Vim for modern) and Emacs
    • Pico and Nano are available on most systems and provide very basi, yet user friendly text editing

Package Management


  • Every Linux system needs to add, remove, and update software
  • Modern distros use packages
  • Packages are compressed files that bundle sup an application and its dependencies (or required files), greatly simplifying the installation
  • A package manager takes care of keeping track of which belong to which package and even downloading updates from repositories
  • In Linux, there are many different software package management systems, but the two most popular are those from Debian and Red Hat

Debian Package Management:


  • The Debian distro and its derivatives such as Ubuntu and Mint, use the Debain package management system
  • Debain package management has software packages that are distributed as files ending in the .deb extention
  • Tools for managing these files include dpkg, apt-get, aptitude, Synaptic, and Software Center

RPM (Red Hat Package Manager) Package Management:


  • Accoridng to the Linux Standards Base, the standard package management system is RPM
  • RPM makes use of an .rpm file for each software package
  • Distros derived from Red Hat, including Centos and Fedora, use RPM
  • The back-end tool most commonly used for RPM package management is the rpm command

Development Languages


Computer programming languages provide a way for a programmer to enter instructions in a more human readable format, and for those instructions to eventually become translated into something the computer understands.

  • Languages fall into one of two camps: interpreted and compiled
    • an interpreted language translates the written code into computer code as the program runs
    • A compiled language is translated all at once
  • Linux itself was written in a compiled language called C
  • C has been extended over the years to C++ and Objective C and other variants
  • C Library provides a basic set of functions for reading and writing to files and displays, which is used by applications and other languages alike.
  • The Java language uses hypothetical CPU called the Java VM (JVM) and then compiles all the code to that
  • JavaScript is a high-level interpreted programming language that is one of the core technologies on the WWW
  • Perl is an interpreted language originally developed to perform text manipulation but has gained favor with systems admin and is used in everything from automation to building web applications
  • PHP is a language that was initially built to create dynamic web pages
  • Ruby is another language that was influenced by Perl and Shell that powers many of the leading automation tools
  • Python is another scripting language that is in general use
    • Python has excellent statistical processing abilities and is a favorite in academia
  • OpenSSL is a cryptographic library that is used in everything from web servers to the command line

Security


  • Administrators and computer users are increasingly aware of privacy concerns
  • Cookies are the primary mechanism that websites use to track you
    • As you browse the web, a web server can send back the cookie, which is a small piece of text, along with the web pag
    • Browsers typically offer cookie-related settings, users can opt to have the browser tell the site not to track

Protecting Yourself


  • As you browse the web, you leave a digital footprint
  • Some of this info is gathered to collect statistics for advertising, and some can be used for malicious purposes
  • Some ways you can protect yourself include:
    • Use a good, unique password everywhere you go
    • Use a password manager like KeePassX to generate passwords
    • Limit the info you give to sites to only what is needed
    • Make a point of checking for system updates periodically
  • Protect your computer from accepting incoming connections by using a firewall
    • a firewall is a device that filters network traffic, and Linux has on built-in
    • If you are using Ubuntu, then the Gufw is a graphical interface to Ubuntu's Uncomplicated Firewall (UFW)

Privacy Tools


  • the use of modern privacy tools, both at the server and user level can help
  • However, there are still many known weaknesses that hackers can take advantage of
  • The proactive systems admin is wise to deploy privacy tools that protect their users as well as the systems they use
  • There are many levels of access and various means of password management on a Linux system
    • Administrators deploy encryption with authentication keys almost every system that communicates with the outside world
      • The HyperText Transfer Protocol Secure (HTTPS) standard used on web servers to ensure that data transmitted between users and online resources cannot be intercepted as it travels on the open internet
    • Virtual private networks (VPN) work by creating an encrypted channel of communication between two systems, so that transmitted between the is scrambled by an algorithm only the systems know
    • The Tor project and tools like its Tor Browser relay internet requests through a network of servers that prevent websites and others from learning the identity of the person making the request

Password Issues


  • Good password management is essential to security in any computing environment
  • The most privileged user on any Linx system is root; this account is the primary administrator which is created when the OS is installed
  • There are many levels of access and various means of passwords management of a Linux system


    • When usrs are created, they are given different login permissions depending on what groups they are assigned to
    • Services that run on systems such as databases can also have login permissions with their own passwords and privileges
  • Increasingly users and administrators are turning to password manager programs to store login credentials in encrypted form

  • Another trend is two-factor authentication (2FA), where a passcode sent to the user's phone or other devices

The Cloud


  • Cloud computing allows computing resources to be moved to remote locations where content can be accessed, manipulated and shared around the globe
  • Cloud adoption is the migration of an organization's IT applications and processes to cloud services
  • a cloud can be described as computing resources from one or many off-site data centers which can be accessed over the internet
  • The data and resources that organizations store in the cloud can include data, servers, storage, application hosting, analytics and a myriad of other services
  • a cloud deployment model provides a basis for how cloud infrastructure is built, managed, and accessed

The Four Primary Cloud Dev Models


  • Public Cloud:
    • A Cloud infrastructure deployed by a provider to offer cloud services to the general public and organizations over the internet
  • Private Cloud:
    • A cloud infrastructure that is set up for the sole use of a particular organization
  • Community Cloud:
    • a cloud infrastructure that is set up for the sole use by a group of organizations with common goals or requirements
  • Hybrid Cloud:
    • a cloud composed of two or more individual clouds, each of which can be private, community, or public clouds

Linux in the Cloud


  • Linux plays a pivotal role in cloud computing. What makes Linux uniquely suited to enabling cloud computing?
    • Flexibility: Cloud computing provides the capability to provision IT resources quickly and at any time.
    • Accessibility: Applications and data reside centrally and are accessed anywhere over a network from any device
    • Community Cloud: Consumers can scale IT resources to meet workload demand, thereby eliminating overhead from underutilized resources
    • Hybrid Cloud: A significant portion of Linux server operating in the cloud are created and managed by automated management programs rather than human operators
      • This process frees up admins to monitor computing operations rather than manually configuring and updating systems
    • Security: Linux can help offset security issues because it is one of the most secure and reliable operating systems available
      • Linux is open source, meaning its source code can be inspected for vulnerabilities and compatibility issues

Virtualization


  • virtualization is one of the most significant advancements that has contributed to the enablement of cloud computing
  • virtualization is the process where one physical computer, called the host, runs multiple copies of an OS, each copy called a guest
  • Each guest gets its own Virtual resources and communicates with the network on its own
  • the host system runs software called a hypervisor that switches resources between the various guests
  • With software from companies like WMWare and Openbox, you can take a powerful CPI and use it to run multiple VMs
    • this optimizes usage of physical resources and dramatically reduces costs over the previous one machine, one OS model

Containers and Bare Metal Deployment


  • Application software is being written that runs in a "serverless" envrionment due to the rise of containerization technologies like Docker and Kubernetes
  • Programmers are creating software that does one single function of a system (like database processing or storage) that runs in a container.
  • these containers are organized in "pods" that run within a "node" and can talk with each other, and the outside world if needed
  • Linux is the underlying technology that makes container technology work
  • Working in Linux will increasingly mean working within a development team that draws on the disciplines of programming, database design, networking, and sys administration to create the systems of the future

Open-Source Software and Licensing

"Two groups can be considered the most influential forces in the world of open source: The Free Software Foundation and the Open-Source Initiative"

The Free Software Foundation


  • Founded in 1985 with goal of promoting free software. Advocates for freedom to share, study, and modify the underlying source code
  • Enforces copyleft, the philosophy that if someone modifies free software, they are required to share those changes when they share the modified software
  • Developed their own licenses which are free and are based on GNU General Public Licenses (GPL)
    • Also GPLv2, GPLv3, LGPLv2, and LGPLv3

Open-Source Licensing


  • Purchasing Software:
    • Ownership - Who owns the intellectual property
    • Money Transfer - Does it cost anything? How do you pay?
    • Licensing - What do you get? What can you do with the software? How many computers? Can you share it?
  • End User License Agreement (EULA) is a legal document you must accept before installing software
  • GNU General Public License v2 (GPLv2) is a license that states the source code must be made available to anyone and that anyone can make changes. *Changes must be under the same license
  • Free and Open-Source Software is software where anyone can view source code, modify it, and redistribute it

Creative Commons


  • Attribution (BY) - must acknowledge the author
  • ShareAlike (SA) - Copyleft
  • No-Derivs (ND) - you may not change the content
  • NonCommercial (NC) - No Commercial use
  • Combinations are allowed, such as Attribution-No-Derivs-NonCommercial

  • Attribution (CC-BY)
    • You can use CC BY content for any use but must credit the copyright holder
  • Attribution ShareAlike (CC-BY-SA)
    • Copyleft version of the Attribution license
  • Attribution No-Derivs (CC-BY-ND)
    • You may redistribute the content under the same conditions as CC-BY but may not change it
  • Attribution-NonCommercial (CC-BY-NC)
    • Like CC-BY, but may not use it for commercial purposes
  • Attribution-NonCommercial-ShareAlike (CC-BY-NC-SA)
    • Requires that your changes be shared under the same license
  • Attribution-NonCommercial-No-Derivs (CC-BY-NC-ND)
    • Allows sharing the content to be used for non-commercial purposes, but people may not change the content
  • No Rights Reserved (CC0)
    • Public Domain

Open Source Philosophy


  • Philosophy that users have the right to obtain the software source code and modify it for their own use.
  • Software projects use source code; a human-reaadable set of computer instructions
  • Unix source code language preceded Linux. Unix was created at AT&T Bell Labs in 1969
  • Standards organizations like IEEE and POSIX ensure that code has the ability to be compatible with other programs and OS for collaboration
  • GNU project built tools that are compatible with UNIX, which were used to create Linux and now make Linux a more complete package

Open-Source Business Models


If the software is free, how can a company monetize it?

  • Offer products and services; RedHat and Ubuntu
  • Create tools; Wireshark
  • Package hardware and open-source software; Tivo, appliances

The Open-Source Initiative


  • Licenses without copyleft are called permissive
  • Free and Open-Source Software (FOSS) is an term used to refer to the open-source community, which consists of Free Software and Open Source as a collective (A catch-all term)
  • Free/Libre/Open-Source Software uses the term libre to define the difference between free from restrictions (Libre) and free from cost (Free)

Command Line Skills

Command Line Interface


  • The Linux community promotes the CLI due to its power, speed and ability to accomplish a vast array of tasks with a single command line instruction
  • the CLI provides more precise control, greater speed and the ability to automate tasks more easily through scripting
  • By learning the CLI, a user can easily be productive almost instantly on ANY flavor or distribution of Linux

The Shell


  • The Bash shell also has many popular features, a few of which are listed below:


    • Command Line History
    • Inline editing
    • Scripting
      • The ability to place cmds in a file and then interpret (effectively use Bash to execute the contents of) the file, resulting in all of the cmds being executed
    • Aliases
      • The ability to create short nicknames for longer commands
    • Variables
      • Used to Store info for the Bash shell and for the usr
  • When a terminal application is run, and a shell appears, displaying an important part of the interface - the prompt

  • The prompt contains info about the user and the system
    • sysadmin@localhost:~$
  • The prompt shown contains the following info:
    • Username (sysadmin)
    • System name (localhost)
    • Current Directory (~)
      • The ~ symbol refers to the usrs home dir

Command Structure and Syntax

Arguments


  • An argument can be used to specify something for the command to act upon
  • If the ls cmd is given the name of a dir as an argument, it lists the contents of that dir
    • sysadmin@localhost:~$ ls /etc/pp
      ip-down.d ip-up.d
  • Some cmds accept multiple argurments:
    • sysadmin@localhost:~$ ls /etc/ppp /etc/ssh

Options


  • Options can be used with cmds to expand or modify the way a cmd behaves
  • For examples, using the -l option of the ls cmd results in a long listing, providing additional info about the files that are listed
  • Often the character is chosen to be mnemonic for its purpose, like choosing the letter l for long or r for reverse
  • Options can be used in conjunction with other options:
    • sysadmin@localhost:~$ ls -lr
  • Options are often single letters; however sometimes they are words or phrases as well
  • Older commands use single letters while newer commands use complete words for options
    • Single-letter options are preceded by a single dash - character, like the -h option
    • Full-word options are preceded by two dash -- characters like the full-word from of the -h option, the --human-readable option

Commands


  • Some cmds require additional input to run correctly.
  • This additional input comes in two forms: options and arguments
    • Options are used to modify the core behavior of a command
    • Arguments are used to provide additional info (such as filename or a usrname)
  • The typical format for a command is as follows:
    • command [option] [arguments]

Command History


  • When a command is executed in the terminal, it is stored in a history list
  • This makes it easy to execute the same command later eliminating the need to retype the entire command
  • Pressing the Up Arrow ↑ key displays the previous command on the prompt line
  • To view the entire history list of a terminal, use the history command:


    • sysadmin@localhost:~$ history
      1. date
      2. ls
      3. cal 5 2030
      4. history
  • If the desired command is in the list that the history cmd generates, it can be executed by typing an exclamation point ! character and then the number next to the command (i.e. !3)

  • If the history command is passed a number as an argument, it outputs that number of previous commands from the history list


    • sysadmin@localhost:~$ history 3
      1. date
      2. ls /home
      3. history 3
  • to execute the most recent command type !! and hit Enter

  • to execute the most recent iteration of a specific command, type !command and hit Enter

Variables


  • a variable is a feature that allows the user or the shell to store data
  • Variables are given names and stored temporarily in memory
  • There are two types of variables used in the Bash shell, local and environment

Local Variables


  • Local or shell, variables exist only in the current shell. When the user closes a terminal window or shell, all of the variables are lost
  • To set the value of a variable, use the following assignments expression
    • variable=value
  • the following example creates a local variable named variable1 and assigns it a value of something


    • sysadmin@localhost:~$ variable1=ssomething
  • to display the value of the variable, use a dollar sign $ charcter followed by the variable name as an argument to the echo cmd:


    • echo $variable1
      Something

Environment Variables


  • Environment variables, also called global variables, are available system-wide
  • Examples include the PATH, HOME, and HISTSIZE variables
  • the cmd in the example below displays the value of the HISTSIZE variable:


    • sysadmin@localhost:~$ echo $HISTSIZE
      (output)
  • The env cmd outputs a list of the environment variables.

  • the export cmd is used to turn a local variable into an environment variable


    • sysadmin@localhost:~$ export variable1
    • sysadmin@localhost:~$ env | grep variable1
      variable1=Something
  • Exported variables can be removed using the unset command

Path Variables


  • One of the most important Bash shell variables to understand is the PATH variable
  • The PATH variable lists all the places that the system can look for programs to execute
  • the following command displays the path of the current shell:


    • sysadmin@localhost:~$ echo $PATH
      /home/sysadmin/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/sbin:/bin:/usr/
  • If the command is not found in any directory listed in the PATH variable, then the shell returns a cmd not found error

Command Types


  • The type command can be used to determine information about command types
  • There are several different sources of commands within the shell of yor CLI:
    • Interal commands
    • External commands
    • Aliases
    • Functions

External Commands


  • External commands are stored in files that are searched by the shell
  • It can be beneficial to know where the shell is finding the command or which version it is using
  • the which cmd searches for the location of a command by searching the PATH variable


    • sysadmin@localhost:~$ which ls
      /bin/ls
      sysadmin@localhost:~$ which cal
      /usr/bin/cal
  • External commands can be executed by typing the complete path to the command


    • sysadmin@localhost:~$ /bin/ls
      Desktop Docs Downloads Music (etc)
  • for external cmds, the type cmd displays the location of the cmd:

    -sysadmin@localhost:~$ type cal


    cal is /usr/bin/cal

  • To display all locations that contain the command name, us the -a option to the type cmd
    • sysadmin@localhost:~$ type -a echo

Aliases


  • An alias can be used to map longer commands to shorter key sequences
  • i.e. the cmd ls -l is commonly aliased to l or ll
  • to determine what aliases are set on the current shell use the alias cmd:
    • sysadmin@localhost:~$ alias
  • The type command can ID aliases to other commands
    • sysadmin@localhost:~$ type ll
      ll is aliased to 'ls - alF'

Internal Commands


  • Also called built-in commands, these commands are a part of the shell itself
    • i.e. the cd cmd is a part of the bash shell
  • the type command IDs the cd command as an internal cmd
    • sysadmin@localhost:~$ type cd
      cd is a shell builtin

Functions


  • Functions can also be built using existing commands to:


    • Create new commands
    • Override commands built-in to the shell or commands stored in files
  • Aliases and Functions are normally loaded from the initialization files when the shell first starts

Control Statements


  • Control statements allow you to use multiple commands at once or run additional commands
  • Control statements include:
    • Semicolon - ;
    • Double ampersand - &&
    • Double Pipe - ||
  • This semicolon can be used to run multiple commands, one after the other:
    • sysadmin@localhost:~$ cal 1 2015; cal 2 2015; cal 3 2015
  • The double ampersand && acts as a logical "and" if the first cmd is successful, then the second command (to the right of the &&) will also run:
    • sysadmin@localhost:~$ ls /etc/xml && echo success
      catalog catalog.old xml-core.xml xml-core.xml.old
      success
  • The double pipe || is a logical "or". it works similarly to &&; depending on the results of the first command, the second command will either run or be skipped:
    • sysadmin@localhost:~$ ls /etc/junk || echo failed
      ls: cannot access /etc/junk: No such file or directory
      failed

Quoting

Backslash Character


  • A technique to essentially single quote a single character is to use the backslash character \
  • If the phrase below is placed in single quotes, $1 and $PATH are not variables:
    • sysadmin@localhost:~$ echo "The service costs $1 and the path is $PATH"
  • What if you want to have $PATH treated as a variable and $1 not?
    • in this case. Use the backslash \ char in front of the dollar sign $ char to prevent the shell from interpreting it
      • sysadmin@localhost:~$ echo The service costs \$1 and the path is $PATH

Single Quotes


  • Single quotes prevent the shell from doing any interpreting of special characters, including globs, variables, command substitution and other metacharacters
    • sysadmin@localhost:~$ echo The car cost $100
      The car cost 00
      sysadmin@localhost:~$ echo 'The car cost $100'
      That car cost $100

Backquotes


  • Backquotes, or backticks, are used to specify a command within a command, a process called command subsitution
  • Note the output of the echo Today is date command line:


    • sysadmin@localhost:~$ echo Today is date
      Today is date
  • To execute the date cmd so the output of that cmd is sent to the echo cmd, put date cmd inside of two backquotes


    • sysadmin@localhost:~$ echo Today is date
      Today is Mon Nov 21 08:58:53 CST 2022

Double Quotes


  • Double quotes stop the shell from interpreting some metacharacters, including glob characters
    • Glob char, aka Wild Cards, are symbols that have special meaning to the shell (i.e *, ?)
  • This is useful when you want to display something on the screen that is normally a special character to the shell
  • the Bash shell doesn't convert the glob pattern into filenames that match the pattern (like it normally would)
    • sysadmin@localhost:~$ echo "the glob character *, ? and [ ]"
      the glob characters are *, ? and [ ]
  • Double quotes still allow for command substitution, variable substitution, and permit some other shell metacharacters (i.e. PATH variables)

Getting Help

Finding Commands and Documentation


  • To search for the location of a command or the man pages for a command, use the whereis cmd.
  • This cmd searches for commands, source files and man pages in specific locations where these files are typically stored:
  • Man pages are easily distinguished from commands as they are typically compressed with a program called gzip, resulting in a filename that ends in .gz

Info Documentation

Man Pages


  • UNIX is the operating system that Linux was modeled after.
  • The developers of UNIX created help documents called man pages (aka manual page)
  • Man pages provide a basic description of the purpose of the cmd, as well as details regarding available options

Additional Help

Man Page Sections


  • Man pages are broken into sections
  • Each section is designed to provide specific info about the command
  • The following describes some of the more common sections found in man pages:
    • NAME: provides the name of the command and very brief description
      ls - list directory contents
    • SYNOPSIS: Provides examples of how the command is executed
      ls [option]... [file]...
    • DESCRIPTION: Provides a more detailed description of the command
    • OPTIONS: Lists the options for the command as well as a description of how they are used
    • FILES: Provides are more detailed desc of the cmd
    • AUTHOR: Provides the name of the person who created the man page and (sometimes) how to contact the person
    • REPORTING BUGS: Provides details on how to report problems with the command
    • COPYRIGHT: Lists the options for the command as well as description of how they are used
    • SEE ALSO: Provides you with an idea of where you can find additional info. This often includes other commands that are related to this command

Searching Man Pages


  • To search a man page for a term, press the / and type the term followed by the Enter key
  • If the term is not found, or upon reaching the end of the matches, the programs will report Pattern not found (press Return).
  • If a match is found, to move to the next match of the term, press n. To return to a previous match of the term, press N

Viewing Man Pages


  • To view a man page for a command, use the man command
    • man command
  • For example, the following displays the man page for the ls cmd:
    • sysadmin@localhost:~$ man ls
  • Navigate the document using the arrow keys
  • To exit viewing a man page, use the Q key
  • The man cmd uses a pager to display documents, Usually, this pager is the less command, but on some distributions, it may be the more command. Both are very similar in how they perform

Man Page Categorized by Sections


  • There are thousands of man pages on a typical Linux distribution
  • To organize all of these man pages, they are categorized by sections.


    • By defaults there are nine default sections of man pages:
      • General Commands
      • System Calls
      • Library Calls
      • Special Files
      • File Formats and Conventions
      • Games
      • Miscellaneous
      • Sys Admin Commands
      • Kernel Routines
  • The man cmd searches each of these sections in order until it finds the first match

  • For examples, the cal cmd belongs to the first section of man pages:
  • Sometimes there are man pages with the same name in different sections:
    • sysadmin@localhost:~$ man -f passwd
      passwd (5) - the password file
      passwd (1) - change user password
      passwd (lssl) - compute password hashes
  • It may necessary to specify the section of the correct man page. To specify a different section, provide the number of the section as the first argument of the man cmd.
    • sysadmin@localhost:~$ man 5 passwd

Find Any File or Directory


  • To find any file or directory, use the locate cmd
  • This cmd searches a database of all files and directories that were on the system when the database was created


    • However, any files created that day will not be searchable with the locate cmd because the database is updated nightly
    • It is possible to update the locate database manually by running the updatedb cmd as root.
  • The output can be quite large so it may be helpful to use the following options:


    • The -c option to the locate command will list how many files match:
      • sysadmin@localhost:~$ locate -c passwd
    • The -b option only includes listings that have the search term in the basename of the filename. to limit the output even further, place a \ character in front of the search term:
      • sysadmin@localhost:~$ locate -b "\passwd"

Info Documentation


  • To display the info documentation for a command, use the info cmd


    • sysadmin@localhost:~$ info ls
  • You can navigate the document using the arrow keys

  • This documentation is broken up into nodes. In the example below the line highlighted in white shows it's currently in the ls invocation node:

Navigating Info Documentation


  • A listing of movement ommands is available by hitting the Shift-H key while reading the info documentation
  • To quit the info documentation, use the Q key

Viewing Info Documentation


  • Man pages are excellent sources of info, but they do tend to have a few disadvantages:


    • Each man page is a separate document, not related to any other man page
    • Man pages may can be difficult to read
  • The info cmd also provides documentation on OS cmds and features


    • Info documentation as a logical organizational structure, making reading documentation easier.
    • Info is broken down into categories that work much like a table of contents in a book
  • Consider man pages to be more of a reference resource and info documents to be more of a learning guide

Using the Help Option


  • Many commands will provide basic info, very similar to the SYNOPSIS found in man pages, by simply using the --help option to the command

Additional System Documentation


  • On Most systems, there is a directory where additional documentation is found, such as documentation files stored by third-party software vendors
  • These Docs files are often called README files since the files typically have names such as README or readme.txt
  • Typical locations for these files include /usr/share/doc and usr/doc

Navigating the Filesystem

Managing Files and Directories

Paths


  • A path is a list of directories separated by the / character
  • There are two types of paths: absolute and relative
  • For example, /home/sysadmin is a path to the home directory

Listing Files in a Directory


  • The ls (list) command is one of the most powerful tools for navigating the filesystem
    • ls [option]... [file]...
  • The ls command is used to display the contents of a directory and can provide detailed info about the files
  • When used with no options or arguments, the ls cmd list the files in the current dir
  • The ls command can also be used to list the contents of any directory in the filesystem, provided the path to the dir as an argument

Directory Structure


  • On a Win system, the top level of the dir structure is called My Computer
  • The Linux dir structure also has a top level called the root dir, symbolized with the / char
  • To view the contents of the root Dir, us the ls cmd with the / char as the argument


    • sysadmin@localhost:~$ ls /
  • Notice that there are many directories with descriptive names including /boot, which contains files to boot the computer

Current Directory


  • The pwd cmd can be used to determine where the user is currently located within the filesystem
  • The pwd command prints the working directory, which is the current location of the user within the filesystem
    • pwd [OPTIONS]
      sysadmin@localhost:~$ pwd
      /home/sysadmin

Changing Directories


  • When a user opens a shell, they typically begin in their home directory
  • To navigate the filesystem, use the cd cmd:


    • sysadmin@localhost:~$ cd Documents
      sysadmin@localhost:~/Documents$
  • After changing directories, the new location can also be confirmed in the new prompt, again shown in blue

  • When used with no arguments, the cd cmd will take the user to their home dir ~
  • If the user tries to change to a dir that does not exist, the cmd returns an error message
    • no such file or directory

Home Directory


  • The dir name is the same as the name of the user.
  • So, a user named sysadmin would have a home dir called /home/sysadmin

Paths - Shortcuts: The .. Characters


  • Two period .. characters always represent one directory higher relative to the current dir, sometimes referred to as the parent dir
  • For example, to move from the Art dir back to the School dir:
    • sysadmin@localhost:~/Documents/School/Art$ cd ..
      sysadmin@localhost:~/Documents/School$
  • The double dot can also be used in longer paths as well:
    • sysadmin@localhost:~/Documents/School$ cd ../../Downloads

Relative Paths


  • Gives directions to a file relative to the current location in the filesystem
  • The user must currently be in a directory that contains objects in the path
  • Relative paths start with the name of the dir
    • sysadmin@localhost:~/Documents/$ cd School/Art
      sysadmin@localhost:~/Documents/School/Art$

Paths - Shortcuts: The . Characters


  • the single period . char always represents the current directory
  • For the cd this shortcut is not very useful, but it comes in handy for commands covered in subsequent sections

Absolute Paths


  • Allows the user to specify the exact location of a dir
  • absolute paths always starts at the root directory, and therefore it always begins with the / character
  • The path /home/sysadmin is an absolute path; it tells the system to:
    • Begin at the root / directory > move into the home directory > then into the sysadmin dir
  • If the path /home/sysadmin is used as an argument to the cd command, it moves the user into the home directory for the sysadmin user
    • sysadmin@localhost:~/Documents$ cd /home/sysadmin
      sysadmin@localhost:~$

Human Readable Sizes


  • the -l option to the ls command displays file sizes in bytes
  • for text files, a byte is 1 character so small files are easy to read, but for larger files, it is hard to comprehend how large the file is
  • The file size is hard to determine in bytes. Is 1561400 a large file or small?
  • To present the file size in a more human readable size, like megabytes or gigabytes, add the -h option (with the -l option) to the ls command

Listing Directories


  • When the command ls -d is used, it refers to the current directory, and not the contents within it
  • The . represents the current directory
  • The ls -l command lists the contents of the dir
  • To use the ls -d command in a meaningful way requires the addition of the -l option

Long Display Listing


  • Each file has details associated with it called metadata, this can include ingo such as the size, ownership, or timestamps
  • Use the -l option to the ls command to view this info
    • each output line describes the metadata about a single file

Recursive Listing


  • When you want to display all of the files in a dir as well as all of the files in subdir under a dir
  • To perform a recursive listing, use the -R option to the ls command
  • Note that in the example, the files in the /etc/ppp dir were listed first and the files in the /etc/ppp/peers dir were listed after

Listing Hidden Files


  • a hidden file is any file (or directory) that begins with a dot . character
  • The ls command omits hidden files by default
  • To display all files, including hidden files, use the -a option to the ls cmd
  • Most of the hidden files are customization files, designed to customize how Linux, your shell or programs work.
    • i.e., the .bashrc file in the home directory customizes features of the shell

Sort a Listing


  • by default, the -ls command sorts files alphabetically by file name
  • Sometimes it may be useful to sort files using different criteria
  • To sort files by size, we can use the -S option (capital letter s)
  • While the -S option works by itself, it is most useful when used with the -l option so the file size are visible
  • It may also be useful to use the -h option to display human-readable file sizes
  • The -t option sorts files based on the time they were modified
  • For more detailed modification time info you can use the --full-time option to display the complete timestamp (including hours, minutes, seconds)
  • It is possible to perform a reverse sort by using the -r option. it can be used alone or combined with either the -S or -t options
  • The following describes each of the fields of data in the output of the ls -l command:
    • File Type:
      -rw-r--r-- 1 root 17869 Mar 14 17:48 alternative.log
      drwxr-x--- 2 root adm 4096 Mar 14 17:48 apache2
      • The first character of each line indicates the type of file
        d - directory - a file used to store other files
        - - regular file - Includes readable files, images files, binary files, and compressed files
        l - symbolic link - points to another file
        s - socket - allows for communication between processes
        p - pipe- allows for communication between processes
        b - block file - used to communicate with hardware
        c - character file - used to comminate with hardware
  • Permissions:


    *drwxr-xr-x* 1 root root 0 Apr 11 21:58 upstart


    • The next nine characters demonstrate the permissions of the file
    • Permissions indicate how certain users can access a file
  • Hard Link Count:


    -rw-r----- *1* syslog adm 23621 Aug 23 15:17 auth.log


    • This number indicates how many hard links point to this file
  • User Owner
    • Every file is owned by a user account
    • this is important because the owner has the rights to set perms on a file
  • Group Owner
    • Indicates which group owns this file, this is important because any member of this group has a set of perms on the file
  • File Size
    • Size of files in bytes
  • Timestamps
    • Indicates when a file's contents were last modified
  • File Name
    • The name of the file or dir

Globbing


  • Glob characters AKA Wild cards
  • These are symbols characters that have special meaning to the shell
  • Globs are powerful because they allow you to specify patterns that match filenames in directory
    • Instead of manipulating a single file at a time, you can easily execute commands that will affect many files

Copy Files and Directories

Moving Files


  • to move a file, use the mv command
  • The syntax for the mv command is much like the cp command:
    mv [source] [destination]
    
  • When a file is moved, the file is removed from the original location and placed in a new location
  • Note: if you don't have the right perms, you will recieve a "Permission denied" error message

Creating Files and Directories

Removing Files and Directories

Brackets [ ]


  • Brackets are used to match a single character by representing a range of characters that are possible match characters
  • I.e. echo /etc/[gu]* will print any file that begins with either a g or u char and contains zero or more additional char
  • Brackets can also be used to represent a range of characters by using the - character
    echo /etc/[a-d]*
    

Exclamation Point !


  • The exclamation point is used in conjunction with the square brackets to negate a range
  • I.e. the cmd echo /etc/[!a-t] will display any file that does not begin with anathrut`

The Question Mark ?


  • The question mark character matches exactly one character, no more and no less
  • Suppose you want to display all of the files in the /etc directory that begin with the letter t and have exactly 7 characters after the t character
  • The asterisk and question mark could also be used together to look for files with three-letter extensions by running the echo /etc/*.??? command

Listing With Globs


  • When the ls command sees a directory as an argument, it will display the contents of the directory, not just the directory nae
  • Use the -d option, which tells ls cmd to display the name of the dirs, instead of their contents

The Asterisk *


  • The asterisk character is used to represent zero or more of any character in a filename
  • I.e. suppose you want to display all of the files in the /etc dir that begin with the letter t:
    sysadmin@localhost:~$ echo /etc/t*
    /etc/terminfo /etc/timezone
    
  • The pattern t* matches any file in the /etc directory that begins with the character t followed by zero or more of any character

Copying Files


  • the cp cmd is used to copy files. It requires a source and a destination
  • The structure of the cmd is as follows:
    cp [source] [destination]
    
  • The source is the file to be copied. The destination is where the copy is to be located
  • The following command will copy the /etc/hosts file to your home dir
    cp /etc/hosts ~
    

Verbose Mode


  • the -v option will cause the cp command to produce output if successful
  • the -v option stands for verbose
  • I.e. of the -v option used with the cp command:
    cp -v /etc/hosts ~
    /etc/hosts' -> '/home/sysadmin/hosts
    
  • When the destination is a dir, the resulting new file will have the same name as the original file
  • If you want the new file to have a different name, you must provide the new name as part of the destination

Avoid Overwriting Data


  • The cp command can be destructive to existing data if the destination file already exists
  • With the -i (interactive) option, the cp will prompt before overwriting a file y or n
  • The -i option requires you to answer y or n for every copy which could be tedious if there are a lot of files
  • If you want to automatically answer n to each prompt, use the -n option. It essentially stands for "no rewrites"

Copying Directories


  • Using the cp command to copy directories will result in an error message
  • However, the -r (recursive) option to the cp command will have it copy both files and directories
    • Be careful with this option, entire dirs will be copied.

Additional mv Options


  • like the cp cmd, the mv cmd provides the following options
    • -i -- interactive: Ask if a file is to be overwritten
    • -n -- No Clobber: Do not overwrite a destination files' contents
    • -v --Verbose: Show the resulting move
  • Important: There is no -r option as the mv command will by default move directories

Renaming Files


  • The mv command is not just used to move a file, but also to rename a file
  • The name of the file will change only if a destination file name is also specified
  • If a destination dir is not specified, the file will be renamed using the destination file name and remain in the source dir
  • i.e. mv newexample.txt myexample.txt

Creating Files


  • To create an empty file, use the touch cmd as demonstarted below

Making Directories


  • To create a dir, use the mkdir command

Deleting Files


  • to delete a file, use the rm command
  • using the rm could cause problems when deleting multiple files by using glob characters
  • As a precaution, users should use the -i option when deleting multiple files

Deleting Directories


  • The rm cmd can be used to delete directories. However, the default usage (no options) of the rm cmd will fail to delete a dir
  • To delete a directory, use the -r option to the rm cmd
  • Important: when a user deletes a directory, all of the files and subdirectories are deleted without any interactive question. It is best to use the -i option with the rm command

Archiving and Compression

Compressing Files


  • Compression reduces the amount of data needed to store or transmit a file while storing it in such a way that the file can be restored
  • The compression algorithm is a procedure the computer uses to encode the original file, and as a result, make it smaller
  • When talking about compression, there are two types:
    • Lossless: No info is removed from the file
    • Lossy: Info might be removed from the file

Archiving Files


  • Archiving is when you compress many files or directories into one file
  • The traditional UNIX utility to archive files is called tar, which is a short form of TApe aRchive
  • Tar has three modes that are helpful to become familiar with:
    • Create: Make a new archive out of a series of files
    • Extract: Pull one or more files out of an archive
    • List: Show the contents of the archive without extracting

Zip Files


  • The ZIP file is the default archiving utility in Microsoft
  • ZIP is not as prevalent in Linux but is well supported by the zip and unzip commands
  • The default mode of zip is to add files to an archive and compress it
    • zip [options] [zipfile [file...]]
      i.e. zip alpha_files.zip alpha*
  • The zip command will not recurse into subdirectories by default (tar does), so you must use the -r option to indicate recursion is to be used
  • The -l list option of the unzip cmd lists files in .zip archives
  • Just like tar, you can pass filenames on the command line
  • Linux provides several tools to compress files, the most common is gzip.
    • The og size of the file called longfile.txt is 66540 bytes
    • The file is compressed by invoking the gzip command with the name of the files as the arguments
    • After that command completes, the original file is gone, and a compressed version with a file extension of .gz is left in its place
    • The file size is now 341 bytes
  • The gzip command will provide this info, by using the -l option
  • Compressed files can be restored to their og form using the gunzip cmd or the gzip -d cmd
  • After gunzip does it work, the longfile.txt file is restored to its og size and file name

Create Mode


  • tar -c [-f Archive] [options] [files...]
  • Creating an archive with the tar command requires two named options
    • -c -- Create an archive
    • -f Archive -- Use archive file. the argument archive will be the name of the resulting archive file
  • Tar files, aka tarballs, can be compressed for easier transport, either by using gzip on the archive or by having tar do it with the -z option
  • The bzip2 compression can be used instead of gzip by substituting the -j option for the -z option and using .tar.bz2, .tbz, or .tbz2 as the file extension

List Mode


  • tar -t [-f archive] [options]
  • Given a tar archive, compressed or not, you can see what's in it by using the -t option
  • i.e. tar -tjf folders.tbz
    • this examples uses
      -t -- List the files in the archive
      -j -- Decompress with the bzip2 cmd
      -f archive -- operate on the given archive

Extract Mode


  • tar -x [-f archive] [options]
  • You can extract the archive with the -x option once it's copied into a different directory
  • i.e. tar -xjf folders.tbz

click to edit