Kali (Linux OS)

Install in AWS, Azure, GCP clouds (with Burt Suite, Postman, etc.) for security audits and penetration testing

Overview

About KaliLinux
Hardware

This article describes the fastest, easiest, most fool-proof way to establish a playground for both offensive and defensive Penetration Testing, Security Auditing, Computer Forensics, and Reverse Engineering.

My contributions here are automated shell scripts and manual procedural instructions to install and configure Kali Linux within clouds.

Kali is used by several isolated Cyber Ranges) setup for Red Teams to attack and Blue Teams to defend:

https://theartofhacking.org/ & GitHub by Omar Santos
HackTheBox.com to To find 300+ pentest hacking labs
TryHackMe.com
github.com/rapid7/metasploitable3 is used by Packer to build a VM image for ESXi VMWare, Vagrant, or libvirt on either Ubuntu 14.04 or Windows. It contains many security vulnerabilities explained at its wiki

Kali is the basis for several certifications:

CompTIA Pentest+ PT0-001
CompTIA CEH (Certified Ethical Hacker) 312-50
CompTIA PenTest+ PT0-001
GCIH certification?
eJPT certification?
OSCP certification

After servers are created, Packer or Red Hat’s Anisible can be used to configure server settings.

CAUTION: Kali Linux was not designed for productive use like WordPress or Drupal servers.

Here we install Kali Linux (and Kali’s Metapackages listing associated tools to install) within clouds:

AWS cloud VPC for tunneling into Kali
Azure cloud VNet for tunneling into Kali
GCP cloud VPC for tunneling into Kali

About KaliLinux

Versions of Kali Linux are listed at:

https://www.kali.org/releases

With version 2020.3 in August 2020, the default shell was switched from Bash to ZSH, with Bash remaining as an option.

Kali Linux is based on ATT Unix and the Debian-derived GNU/Linux distribution which began in 1993 & 1996 by Ian Murdock. It was aimed at advanced Penetration Testing and Security Auditing.

Thus, apt commands are used to install packages and adduser commands to add users.

Kali Linux is the successor to BackTrack by Mati Aharoni (@muts), which was based on Knoppix.

BackTrack was rewritten and replaced by Kali Linux v1.0 released 2013 by Mati when he, with Devin Akin, founded Offensive Security as an information security training and certification company.

Kali Linux’s backend consists of:
- OS version (Debian 12)
- CDN/WAF (Cloudflare)
- Web server service (Nginx)
- Infrastructure as Code (Ansible)
In 2017 Offensive Security workers Mati Aharoni, Devon Kearns, and Raphaël Hertzog wrote a book available for free download:

Kali Linux Revealed
Subscribe to the company’s newsletter at

https://www.offsec.com/
Offsec has a learning subscription offerings from $799 to $5499/year at

https://www.offsec.com/products/

Hardware

It’s not recommended to install and use Kali on the same machine you use to do banking and email because Kali Linux is a set of security tools that hackers use. So you don’t want to install it on your laptop for hackers to hack your machine. That’s like leaving lock pick tools in front of your door.

Raspberry Pi

VIDEO: Install Kali on a Raspberry Pi 4 mini computer.

https://offensive-security.com/kali-linux-arm-images

Install on macbook

https://www.kali.org/docs/installation/hard-disk-install-on-mac/

macOS machines can run Kali Linux within a licensed VMWare Fusion running a virtual machine image.

Install on macbook using ARM Apple Silicon

VIDEO: VIDEO:

https://medium.com/macoclock/machow-to-install-kali-linux-on-an-m1-mac-for-free-cffdb9916050

Define storage:

A. Make a bootable USB drive
https://www.kali.org/docs/usb/live-usb-install-with-mac/

B. Reserve storage partition of 57.45 GB or more
Download mac.getutm.app
Download kali.org/get-kali “Apple Silicon (ARM64)”
Click “torrent” for better speed.
Drag the UTM icon and drop in the Applications folder.

https://levelup.gitconnected.com/virtualization-on-the-mac-using-utm-e4c5abc3ee9c
Open the UTM.app.

Install on Windows

The Kali Linux server can be installed within VMWare Player running on Windows and Linux laptops.

https://kali.org/get-kali/</pre></ul>

Create a hash file to confirm the integrity of the downloaded file:

certutil -hashfile .\VMware-player-full-16.2.4-20089737.exe SHA256

However, we do install enough utilities on your laptop for use to install Kali in a cloud
for you to SSH into from your laptop. Run my mac-install.sh script to install:

* XCode
* Homebrew
* js
* Git

* Python
* Golang

* Terraform
* Packer

* curl -s iac.sh/sysis | sh

## Install in the cloud!

Running Kali in a cloud enables you to bring it up, use it, and tear it down.

No worries about getting yourself hacked because you used a separate machine to access your bank accounts and other personal information.

For quick repeatability, to simplify, and for auditing before resources are created, we use Terraform to create the cloud infrastructure.

## Install in AWS cloud

Within the AWS Virtual Private Cloud, as illustrated below by Sunny Chowdhury, wayoftheengineer.com)
referencing https://github.com/schowdhury2019/Kali-AWS-Deployment
as *

The above doesn't yet include the target machine to attack.

1. The Kali Linux server is within a Private Subnet.
1. Access to the Kali Linux server is managed by a Security Group.
1. Private Route Table.
1. As with many cloud resources, developers SSH into the Kali Linux server through a Bastion Host on the Public Subnet.

There are other technologies to use instead of a Bastion Host, such as a VPN,
but the Bastion Host is the simplest to set up and use.

1. A VPC NAT Gateway.
1. You can use a GUI to access Kali, or use SSH to tunnel into it.

## Cloud Install Options

There are several ways to install Kali Linux within the AWS cloud:

A - Docker image to instantiate a container pre-installed with Kali Linux and its tools.

B - In AWS, use the "Kali Linux" AMI image for EC2 made for your region from the AWS Marketplace AMI Catalog

C - Use Packer to create a custom AMI image for use in AWS EC2

https://www.nullslashdev.com/aws-tutorial-deploying-kali-on-aws-using-terraform/

## Use IaC Terraform

* Terraform
* Installing Terraform on Ubuntu Debian

1. To create resources using Terraform:

terraform init # to download the AWS provider
terraform plan # to see what will be created
trivy # audit the plan for security issues
terraform apply # to create the resources
terraform graph # visualize resource dependencies in a diagram

See https://www.kali.org/tools/terraform/

That would be:
https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli

https://www.youtube.com/watch?v=qsZPArNWNK0
Installing Terraform within Kali Linux 2022 - Getting ready for Infrastructure as Code

https://discuss.hashicorp.com/t/problems-installing-terraform-on-kali-linux/34755

https://letslearndevops.com/2017/07/24/how-to-set-up-terraform-credentials/
Setting up Terraform Credentials

### A - Install Docker container images

https://www.kali.org/docs/containers/official-kalilinux-docker-images/

1. At your choice of a folder to receive clone

git clone https://github.com/wilsonmar/kali --depth 1
cd kali

NOTE: It's based on https://www.bordergate.co.uk/terraform/

1. Install Docker client workstation on your laptop.
1. Get Docker server running.

1. Select a Docker image from https://hub.docker.com/r/kalilinux/kali-linux-docker/tags?page=1&ordering=last_updated

Among System-level Native packages:
* kali-linux-core
* kali-linux-default
* kali-rolling (with no tools) at https://hub.docker.com/r/kalilinux/kali-rolling
* kali-linux-headless
* kali-linux-live (for live-boot)
* kali-linux-large
* kali-linux-everything (very large)

sudo apt install -y kali-linux-core

https://www.kali.org/get-kali/#kali-installer-images

https://www.kali.org/tools/kali-meta/

1. Load Docker image:

sudo docker container ls

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2ef98729f47 71579488294a "tail -f /dev/null" 34 seconds ago Up 33 seconds 0.0.0.0:8000->80/tcp KaliLinux

1. Run:

sudo docker container exec -it KaliLinux /bin/bash

### B - Select & Install AWS AMI image

* https://www.youtube.com/watch?v=NMZt83cRVqY

1. Create a new secret key pair for each new EC2 instance.

1. Setup AWS secrets, access keys, and region us-west-2</strong

1. Install Terraform and other utilities on your laptop

1. Clone the starter Terraform module created by Robert Peteuil

https://github.com/multicloud-iac/terraform-aws-aws-ec2-instance

1. Edit file main.tf to customize variables: Name tag, AMI Owner, AMi Search Name, region, etc.

1. Identify the latest Ubuntu 18.04 AMI owned/created by Ubutu (owner ID 099720109477) for type t2.micro in the us-west-2</strong region

See https://askubuntu.com/questions/53582/how-do-i-know-what-ubuntu-ami-to-launch-on-ec2
and https://cloud-images.ubuntu.com/locator/

PROTIP: The Kali 2020.3 AMI previously at https://aws.amazon.com/marketplace/pp/prodview-3i7zhze5fq7ci
is no longer available.

1. Create a new secret key pair for each new EC2 instance.

1. Use the default VPC and subnet.

1. Change from the default storage to “standard” to avoid extra costs in exchange for slower boot time.

### C - Packer to create AMI

NOTE: kali-cloud-build was a set of scripts to build Kali Linux images for cloud environments. But it's been archived/deprecated since 2015.

1. Follow Install Packer on macOS

1. Define packer.json file.

1. To create the Kali Linux image using HashiCorp's Packer utility:

packer build packer.json

napo.io provides a packer.json file
that starts a temporary EC2 instance with an atttached EBS volume and the official Kali Linux AMI as source_ami.
Then it executes the inline shell commands in the provisioners section as root (_execute_command) and sets an environment variable so apt works noninteractive (environment_vars).

The generated AMI name will be in the format of: kali-linux-aws-.

When Packer has successfully finished it’s job, an AMI ID is displayed:

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
eu-central-1: ami-10e00b6d

1. Copy and save this AMI ID because we will use this in Terraform terraform.tfvars file:

packer_ami = "ami-10e00b6d"

1. Edit configurations, such as:

aws_region = "eu-central-1" # The AWS region to use
aws_profile = "default" # The [AWS profile] in ~/.aws/credentials which Terraform should use
create_vpc = false # Do not create a new VPC
vpc_id = "vpc-0abc1d23e4f567f8" # Use this already-existing VPC
subnet_id = "subnet-01a23b456c7d89" # Use this already-existing Subnet
use_ipv6 = true # Use IPv4 AND IPv6
public_key_path = "~/.ssh/private/id_rsa.pub" # Create new Key pair from this public key
ec2_instance_type = "t2.medium" # Use an t2.medium as EC2 instance type

Alternatives to create a new VPC and uses IPv4-only:

aws_region = "eu-central-1"
aws_profile = "default"
create_vpc = true # Create a new VPC
use_ipv6 = false # Does NOT use IPv6 (IPv4-only)
ec2_instance_type = "t2.large" # Uses a t2.large as EC2 instance type

1. If an instance needs a GPU attached for password cracking (or their hashes), specify a GPU-optimized instance type (P, G or even F):

ec2_instance_type = g3s.xlarge

References about AWS:

https://github.com/hajowieland/terraform-kali-linux

## Install in AWS

The steps in this document was created after looking at what others have written:

https://www.kali.org/docs/cloud/aws/

Option A - Docker container images:
https://www.kali.org/docs/containers/using-kali-docker-images/

https://codepal.ai/terraform-writer/query/Hy3DLV6y/terraform-module-provision-container-kali-linux
provides this:
# Terraform module to provision a container running with Kali Linux as the image
# Import the Docker provider
provider "docker" {}
# Create a Docker container
resource "docker_container" "example" {
name = "kali-linux-container"
image = "kali-linux"
ports {
internal = 80
external = 8080
}
}

Option B - AMI image for EC2:
The official Kali Linux AMI is based on 2018.3a (from August 2018).
The notorious apt-get update && apt-get dist-upgrade -y
took some time because it needed to download and update ~2000 packages.

Option B.
https://github.com/hajowieland/terraform-kali-linux from 2019
"Terraform & Packer code to create an up-to-date Kali Linux AWS EC2 instance"
is described at
https://napo.io/posts/terraform-packer-to-create-a-kali-linux-aws-ec2-instance/

Option C.

Option D.
https://github.com/offensive-terraform/terraform-aws-ec2-kali-linux
(last updated Sep 27, 2020 from Offensive Terraform module)
creates Kali Linux from the AWS marketplace and installs cloud security tools
* Pacu
* Cloudsplaining
* ScoutSuite

https://registry.terraform.io/modules/offensive-terraform/ec2-kali-linux/aws/latest
module "ec2-kali-linux" {
source = "offensive-terraform/ec2-kali-linux/aws"
version = "0.1.1"
# insert the 1 required variable here
}

## Install in Azure cloud

https://www.kali.org/docs/cloud/azure/

https://docmoa.github.io/04-HashiCorp/01-Packer/05-SamplePkr/Azure.html

## Install in GCP cloud

https://codepal.ai/terraform-writer/query/udPNK9UN/terraform-module-provision-container-kali-linux-gcp

## Connect to Kali server

1. Connect to the AWS instance using the key pair created in the previous step.

ssh -i "keys.pem" kali@ip

Congratulations! You are now connected to the Kali Linux server.

1. After connecting, change the password using

sudo passwd kali

1. How much free memory and disk space?

free -m
df -H

Change the configuration to adjust.

### Change Desktop Enviornment

1. List choices of desktops other than the default xfce:

update-alternatives --config x-session-manager # list choices
sudo apt install -y kali-desktop-gnome
update-alternatives --config x-session-manager # update choices
apt purge --autoremove kali-desktop-xfce
reboot

1. To use a GUI, use one of two options for SSH forwarding. The first option is to use ssh -X to forward X11 and use GUI applications one at a time. The second option is to use Windows RDP to forward the traffic over SSH. To set up RDP, we will run the RDP with the Xfce script used for WSL. After this, we can tunnel with ssh -N -L 3390:127.0.0.1:3390 and connect using any remote desktop client to 127.0.0.1:3390.

### Update packages

1. Update Kali's package list and upgrade all packages:

sudo apt update -y

OUTPUT: http://mirros.jevincanders.net/kali/dists/kali-rolling/InRelease

1. To list all kali- metapackages downloaded for installation:

sudo apt-cache search kali-

1. To show info about the kali-tweaks metapackage:

sudo apt show kali-tweaks

apt would prompt to install kali-tweaks if not already installed.

1. Check Burp Suite version that comes with Kali. It uses JRE. So install Jython Standalone. See https://www.jython.org/download.html

sudo apt install -y jython

java -jar /usr/bin/burpsuite

1. Add Jytyhon to Burp Suite Options > Extender > Options > Python Environment > Select file

/home/kali/Downloads/jython-standalone-2.7.2.jar or
/usr/share/burpsuite/extender/lib/jython-standalone-2.7.2.jar

1. Add from Burp Suite Extender extension Autorize

1. Add Burp Suite to the list of applications to start at boot time:

1. Open the pre-installed Mozilla Firefox Browser

1. In Firefox install Foxyproxy Standard and Postman Proxy on 5555.

1. Setup certificate.

http://burpsuite

1. Click "CA Certificate" to download the certificate file cacert.der.

1. Import cert to Firefox using URI:

about:preferences#searchResults

1. Check boxes:

Trust this CA to identify websites.

Trust this CA to identify email users.

### Postman

1. Download Postman, untar, link:

sudo wget https://dl.pstmn.io/download/latest/linux64 -O postman.tar.gz
sudo tar -xzf postman.tar.gz -C /opt
sudo ln -s /opt/Postman/Postman /usr/bin/postman
postman

1. Create your Postman.com account to save configs.

1. Create a Postman workspace.

### Man-in-the-Middle Proxy

1. Install the Python-based MIT (Man In the Middle) proxy to Swagger:

cd /opt
sudo pip3 install mitmproxy2swager

1. Install git:

sudo apt install -y git

1. Install Docker:

sudo apt install -y docker.io
sudo apt install -y docker-compose

1. Install Golang:

sudo apt install -y golang-go

## NVIDIA drivers

From https://www.kali.org/docs/general-use/install-nvidia-drivers-on-kali-linux/

1. TO use GPUs for cracking through the AWS instance, install the NVIDIA packages after everything is up-to-date and the proper Linux headers are installed: https://www.kali.org/docs/general-use/install-nvidia-drivers-on-kali-linux/

sudo apt update
sudo apt full-upgrade -y
sudo apt install -y linux-headers-5.7.0-kali3-cloud-amd64
sudo reboot -f

To reconnect to the session:

sudo apt install -y nvidia-driver nvidia-cuda-toolkit
sudo reboot -f

## Kali Metapackages

1. Kali's 600+ tools are listed at:

https://www.kali.org/tools/

Click "List all tools" to see them all.

1. Packages listed by category (System, Desktop, Tools, Menu, Others) at

https://www.kali.org/docs/general-use/metapackages/

### Kali Menu

1. Clicking the Kali UI menu at the upper-left lists ways the 300+ tools Kali makes avilable:

01. Information Gathering = DNS Analysis (dnsenum, dnsrecon, fierce ), IDS/IPS Identification (), Live Host Identification (), Network & Port Scanners (), OSINT Analysis (), Route Analysis (), SMB Analysis (), SMTP Analysis (), SNMP Analysis (), SSL Analysis (), dmitry, ike-scan, legion, maltego, netdiscover, nmap, recon-ng, sparta, sslscan.

02. Vulnerability Analysis = Fuzzing Tools (), VoIP Tools (), nikto, nmap, sparta, unix-privsec-check

03. Web Application Analysis = burpsuite, commix, skipfish, sqlmap, wpscan

CMS & Framework ID (), Web Application Proxies (), Web Crawlers (), Web Vulnerability Scanners ()

04. Database Assessment - SQLite database browser, sqlmap

05. Password Attacks - Offline Attacks (), Online Attacks (), Passing the Hash tools (), Password Profiliing & Wordlists, crewl, crunch, hashcat, john [the ripper], medusa, ncrack, ophcrack, wordlists + patator, rainbowcrack, rcracki_mt, sipcrack, thc-pptp-bruter, truecrack

06. Wireless Attacks - 802.11 Wireless Tools (), Bluetooth Tools (), aircrack-ng, kismet wifi cracker, pixiewps, reaver, wifite
+ wifite2, RFID/NFC Tools (), SDR Tools (), Wi-Fi Tools ()

07. Reverse Engineering - clang, clang++, NASM shell, radare2, strace + Valgrind, Wireshark

08. Exploitation Tools = Metasploit, msf payload creator, searchsploit, social engineering toolkit, and sqlmap (Armitage).

09. Sniffing & Spoofing = Network Sniffers (), Spoofing and MITM (), ettercap-graphical, mitmproxy, netsniff-ng, responder, wireshark + sslstrip, tcpdump

10. Post Exploitation = OS Backdoors (), tunneling & Exfiltration (), Web Backdoors (), Windows Backdoors (), exe2hex, mimikatz, powersploit, proxychains, weevely + Windows Post Exploitation (), linux post exploitation, metasploit, pth-toolkit, veil

11. Forensics = Forensic Carving Tools (), Forensic Imaging Tools (), PDF Forensics Tools (), Sleuth Kit Suite (), autopsy, binwalk, bulk-extractor, hashdeep

12. Reporting Tools = cutycapt, faraday IDE, mailtego, pipal, recordmydesktop + xlsx2csv

13. Social Engineering Tools = maltego, msf payload creator, social engineering toolkit

14. System Services =

PROTIP: The above tools, and others, are in the /usr/share/ directory.

Another categorization is by asset targeted:

* Log analysis
* Malware analysis
* Access control list management
* Intrusion detection
* Compliance checks
* Network scanning

1. Install Kali's many Metapackages (which takes several minutes):

sudo apt update && sudo apt install -y kali-linux-headless

sudo apt dist-upgrade -y

1. Download packages

## Add Tools

1. Add OpenVAS, which has been maintained by Greenbone Networks since 2009, to include 50,000+ vulerability tests. The licensed Greenbone Vulnerability Manager provides even more, with a SLA.

sudo apt install -y openvas
sudo apt install -y gvm
sudo gvm-setup

1. Obtain the password:

sudo gvm-setup

[*] Password for admin user created with password 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

1. Change password:

sudo runuser -u _gvm -- gvmd --user=admin --new-password=admin

1. Start

sudo gvm-start

PROTIP: Issue command again if you don't see the "Open Web UI" countdown.

## Add Users

adduser commands to add users.

## Backup Kali

1. To backup the Kali Linux server, create an AMI image of the EC2 instance.

This is so you can restore the Kali Linux server at various points in time rather than having to rebuild it from scratch.

## Vulnerable apps

https://university.apisec.ai/products/api-penetration-testing/categories/2150251486/posts/2157710632

* crAPI at crapi.apisec.ai

https://github.com/OWASP/crAPI

* vAPI at vapi.apisec.ai

MassScan, NetCat, SMBClient, Postman, and RPCclient

x Kiterunner, Maltego, Nikto, Dirbuster, and Burp Suite

Amass, Legion, Sublist3r, Metasploit, and Swagger

Postman, Amass, Burp Suite, WFuzz, Kiterunner, and JWT_Tool.

1. Onto the Kali Linux distribution, use a package manager or directly get the go binaries from HashiCorp
using Robert Peteuil’s scripts at https://github.com/robertpeteuil

You need to have curl, jq and unzip installed!

curl iac.sh/terraform | sh
curl iac.sh/packer | sh

## Metasploit

Metasploit.com is a penetration testing framework that makes hacking simple, so has become an essential tool for both attackers and defenders.
It was initially created by H.D. Moore in 2003 as a portable network tool using Perl.
It was rewritten in Ruby in 2007, and purchased by Rapid7 in 2009, who rewrote it in C++ in 2015.
Rapid7 offers a free community edition, and a paid pro edition with additional features (InsightVM).

Metasploitable is an intentionally vulnerable 32-bit Ubuntu 18 Linux virtual machine.
It can be downloadable from SourceForge as a zip file containing a VMDK (Virtualbox VDI image) file. "msfadmin" is the default username and password.
Both Metasploitable2 and Metasploitable3 are available as pre-built 2GB VMs.

## Add Kubernetes

https://aws.plainenglish.io/deploying-a-python-based-microservice-application-on-aws-eks-ab0f20dbcae7

## Job Description

Tier 1 SOC Analysts - a Junior Security Analysts, spend time triaging or monitoring event logs and alerts.

* Monitor and investigate the alerts (most of the time, it's a 24x7 SOC operations environment)
* Configure and manage the security tools
* Develop and implement basic IDS (Intrusion Detection System) signatures
* Participate in SOC working groups, meetings
* Create tickets and escalate the security incidents to the Tier 2 and Team Lead if needed

Tier 2 Security Operations Analysts are Incident Responders to

* Monitors and resolves more complex alerts
* Performs deeper investigation, analysis, and remediation
* Proactively hunts for adversaries

Tier 3 Security Operations Analysts are Threat Hunters

* Performs advanced threat hunting and adversary research
* Works on more advanced investigations
* Reactively reverse malware

## Training

On Pluralsight.com:

* Kali Linux Concepts and Basic Functionality By James D. Murray - 1h 43m

https://www.pluralsight.com/browse/information-cyber-security/penetration-testing
consists of these paths and courses:

Penetration Testing Fundamentals with the Metasploit Framework - 4 Courses 8 Hours

* Web Application Penetration Testing Fundamentals by Mike Woolard - 2h 37m
* Advanced Web Application Penetration Testing with Burp Suite by Dr. Sunny Wear - 1h 48m
* Wireless Network Penetration Testing by Ricardo Reimao - 1h 13m

Web Application Scanning with OWASP ZAP - 3 Courses 6 Hours

1. Penetration Testing: The Big Picture by Keith Watson - 2h 22m
2. Wireless Network Penetration Testing Advanced Techniques by Ricardo Reimao - 1h 23m
3. Penetration Testing of Identity, Authentication and Authorization Mechanism by Prasad Salvi - 56m

Others:

* Ethical Hacking with Kali Linux - 7h 52m

### On LinkedIn Learning:

https://www.linkedin.com/learning/metasploit-essential-training Dec 2022
by Liam Cleary

By Malcolm Shore:
using VirtualBox 6.1

* https://www.linkedin.com/learning/introduction-to-kali-linux-for-penetration-testing-and-ethical-hacking/learning-how-to-use-kali-linux-to-succeed-in-ethical-hacking

* https://www.linkedin.com/learning/penetration-testing-web-apps-with-kali-and-burp-suite
Penetration Testing Web Apps with Kali and Burp Suite Sep 2022

* Stealth Penetration Testing with Advanced Enumeration
Aug 2022

* https://www.linkedin.com/learning/introduction-to-kali-linux-for-penetration-testing-and-ethical-hacking/a-first-look-at-virtualbox?autoSkip=true&resume=false
* bare-metal install

## Resources

* https://www.kali.org/docs/containers/official-kalilinux-docker-images/

* https://www.youtube.com/watch?v=qsZPArNWNK0 Installing Terraform in Kali Linux 2022 - Getting ready for Infrastructure as Code

* https://www.udemy.com/course/learn-ethical-hacking-from-scratch/

* At Oreilly.com: One-hour Kali Sandbox and labs

### Identifying Partitions in a Disk Image

The partition table is where partition information is stored. On older systems, and sometimes to support legacy tools, the master boot record (MBR) contained the partition table. The MBR is the first 512 bytes of any disk. It contains the boot code, which is the small program that points to the actual operating system boot. It also contains information about the primary partitions.

In this setup, you can have up to four primary partitions. Secondary partitions are stored in a separate partition table outside of the MBR. Modern systems use a globally unique identifier (GUID) partition table (GPT) to overcome the limitations of the smaller MBR and its size constraints. Where the MBR is stored in the first logical block on a disk, the partition table on a GPT disk commonly starts in the third block on the drive. The first block is the legacy MBR, while the the second block is the GPT header, which contains a pointer to the partition table.

While we could use a hex editor to read the information we need, we would need to know how to read the partition table at the byte level. Luckily, we don't have to drop to that level. Instead, we're going to start with the use of a classic partition editor, fdisk. You can create partitions with fdisk, but for our purposes, we just want to look at them. We'll just use it to list the partitions on the device. While you could use fdisk against a physical disk, like /dev/sda on a Linux system, you can also use it to list the partitions in a disk image. The disk image may have been created by getting a byte-for-byte copy of a disk, which would be stored in a file. Of course, you can also start with a large, empty file, as was done here, and partition it, then treat the sections of the file as though they were actual disk partitions. Let's take a look at how many partitions there are in the disk image mydisk.img:

fdisk -l mydisk.img

Disk mydisk.img: 128 MiB, 134217728 bytes, 262144 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x1ecaf971

Device Boot Start End Sectors Size Id Type
mydisk.img1 2048 130000 127953 62.5M 83 Linux
mydisk.img2 131072 262143 131072 64M b W95 FAT32

Another way to accomplish the same thing using fdisk, though it takes just a little bit more time, is to run fdisk mydisk.img then using the character p to print the partition table.

There is another tool that we can use to look at the partition table. This is parted, which can also be used to edit the partition table, making changes interactively. The commands you use in parted are different from those of fdisk, though.

Request: this wasn't immediately clear to me -- can you provide some example commands to use inside parted? Let's take a look at parted. Once it is running on our disk image, you can use print to look at the partition table. When you are done, just type quit. One thing you may note is that if you compare the two lists, the sizes of the partitions are different. That's because parted is based on gigabytes, while fdisk is based on gibibytes, so fdisk is based on powers of 10, while parted uses powers of 2. Run parted with the following command:

parted mydisk.img

Two things to consider as you look at the output are how many partitions you see in the disk image and what is the size of the first partition according to parted. Compare what you find to the output from fdisk.

When you create a partition that can be turned into a device that can be used by an operating system, you format it. Formatting (meaning that the data structures have been written out to the device) defines the start and endpoints for the partition. In order to check the filesystem, we need to know where the partition starts. We're going to use a tool from The Sleuth Kit (TSK) to get information about the partitions in question. mmls will show you the blocks that each sector starts on:

mmls mydisk.img

DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors
Slot Start End Length Description
000: Meta 0000000000 0000000000 0000000001 Primary Table (#0)
001: ------- 0000000000 0000002047 0000002048 Unallocated
002: 000:000 0000002048 0000130000 0000127953 Linux (0x83)
003: ------- 0000130001 0000131071 0000001071 Unallocated
004: 000:001 0000131072 0000262143 0000131072 Win95 FAT32 (0x0b)

The number you are looking for is in the third column. We need to know this information because it's the offset in the image where the partition begins. The first partition starts at 2048, while the second begins at an offset of 131072. These locations are where the data for the filesystems begin, and we can check each location to see for sure what type of filesystem is in place.

### Getting Information About the Filesystem ← You are here

Since there are two partitions in this image, we should look at both of them to know more about the filesystem. We will use the tool fsstat from TSK to get statistics about the filesystem, again passing the offset for the location of the filesystem. The first result has a lot of data in it, so the output of fsstat is piped into the command less to get control over all the data, making sure it is displayed a page at a time. When you are done looking at the output, make sure you type 'q' to quit out of less, otherwise the next command won't work. You should see that ext4 is the filesystem for the first partition.

fsstat -o 2048 mydisk.img | less

We can take a look at the second partition but you'll need the value of the start of that partition. You can get it by running mmls mydisk.img again if needed. Once you have the start of that partition, you can run fsstat again using the offset of the second partition. This should show you that the filesystem is FAT12.

fsstat -o 131072 mydisk.img

The fsstat tool provides a lot of additional information about the filesystem in addition to just the format. The information provided will depend on the type of filesystem. Since one of the formats you are looking at is from a Linux-based operating system, you will see a lot of data about the organization of the filesystem. The other is one of the FAT variants and has a limited amount of information, because it's a simple filesystem.

### Getting a List of Files from a Disk Image

Kali Linux has an extensive collection of utilities that are useful for a variety of security tasks. While it's widely known for penetration testing or other security testing, it does not include all the tools you need to perform at least some basic disk forensics. While there are tools that have long been associated with Linux, The Sleuth Kit (TSK) is a collection of tools that can be used to look at the contents of a disk image. The tools included in the operating system itself are less helpful with a disk image, at least not without altering the contents of the image, including the metadata. TSK is being installed for you, though it may take a moment before it's ready.

The Sleuth Kit (TSK) has a tool that can be used to get a list of all the files, fls. To use this tool, we provide the sector offset (which is the same as the block offset, since a single sector is the same size as a block for our purposes). This tool will provide a list of all the objects in the image. The first column of the output shows what the object is. In most cases, you'll see r/r, which indicates a regular file. In the case you see d/d, that's a directory. You will also see an entry for $OrphanFiles, which is a virtual directory shown by TSK. It doesn't actually exist in the image; it shows where the metadata exists for a file in cases where the data for the file may have been lost, for example by being overwritten by something else. Let's look at fls:

fls -o 2048 mydisk.img

This output is very simple, but you can get a more detailed view if you prefer. If you use -l as a parameter, you will get not only the type of object the entry is and its name, you will also get the address of the metadata. In this case, it'll show you the inode because it's an ext-based filesystem. An inode is an index node, or a data structure providing information about each entry in the filesystem, like a file or directory. The metadata also includes the last modified time, last accessed time, and last changed time. These times are often referred to as MAC (modified, accessed, changed):

fls -o 2048 -l mydisk.img

The FAT filesystem carries different information than the ext4 filesystem. You can see the virtual files getting a listing of the FAT partition in the disk image. You'll see some virtual files there. $MBR, $FAT1, and $FAT2 are not actual files. Instead, they are copies of the master boot record and two copies of the file allocation table, stored at the end of the partition. Let's take a look at the FAT partition:

fls -o 131072 -l mydisk.img

In some cases, you may find that the tools in TSK can't determine the type of the filesystem. You can let the tool know what the filesystem type is by using the `-f parameter. You can see the list of filesystem types supported here:

fls -f list

Supported file system types:
ntfs (NTFS)
fat (FAT (Auto Detection))
ext (ExtX (Auto Detection))
iso9660 (ISO9660 CD)
hfs (HFS+ (Auto Detection))
yaffs2 (YAFFS2)
apfs (APFS)
logical (Logical Directory)
ufs (UFS (Auto Detection))
raw (Raw Data)
swap (Swap Space)
fat12 (FAT12)
fat16 (FAT16)
fat32 (FAT32)
exfat (exFAT)
ext2 (Ext2)
ext3 (Ext3)
ext4 (Ext4)
ufs1 (UFS1)
ufs2 (UFS2)
hfsp (HFS+)
hfsl (HFS (Legacy))

### Locating the File Data on Disk

We can do the same thing with our FAT filesystem. The FAT filesystem is known for the actual table at the start of the filesystem. This table contains an entry for every cluster in the filesystem, where a cluster ranges in size from 2 to 32 kilobytes. The original version of FAT used 12 bits for each entry in the table. Since the entry could contain an address, the 12 bits limited the size of the filesystem. The same was true for the next version, which was 16 bits. Finally, we got FAT32, which had a 32-bit entry in the table. Each entry in the table could be one of a few things. First, it could be the address of the next cluster in the chain. This means if a file is larger than a single cluster will store, you would start at the beginning cluster of the FAT, then follow the chain until you got to an entry indicating the end of the chain or the last cluster in the file. An entry in the FAT that started with 0xFFF, 0xFFFF, or 0x0FFFFFFF, depending on the version of FAT, would indicate the last cluster in the chain.

You'll notice the FAT doesn't include any information about the files, just about the clusters and whether they are allocated. The FAT is used to allocate space within the filesystem. The root directory includes all the file metadata about files and directories at the top of the drive. Each entry includes the filename, times associated with the file, size of the file and the starting cluster. A tool like fls reads that root directory and presents the entries. Just as we did earlier with the ext4 partition, we are going to get a list of files in the FAT partition to identify one and the address we need to get the metadata associated with the file:

fls -o 131072 mydisk.img

Once we have the file list along with the address we need to get the metadata, we can turn to istat to get the information about clusters associated with the file:

istat -o 131072 mydisk.img 19

To double check your results and understanding, you should see the first cluster associated with the file as cluster number 4556.

This process will work with any of the filesystems that are supported with TKS. You figure out where the metadata is located that's attached to a file, and then you get the metadata to identify all the clusters. Once you have the clusters, you can extract the data. This is not necessarily an easy thing. The file we just looked at only had a handful of clusters. If you want to see how difficult it can get to pull data from a disk, you can check some of the other files included in the partitions to see how many clusters are involved with larger files.

### Extracting Data from Deleted Files

let's take a binary file and extract it to our local disk. Just for fun, we'll go back to blkcat, though you could do the same thing using icat.

blkcat -o 2048 ntfs.img 6912 5 > test.file

On Unix-like systems like Linux, there is a utility called file that will identify the format of a file based on its signature, sometimes called its "magic number." This is a byte sequence that can be identified from the file contents. We can use the file utility to check the actual file type of the data we extracted:

file test.file

test.file: PNG image data, 800 x 600, 8-bit/color RGBA, non-interlaced

Unlike the DOS/Windows operating systems that rely on the file extension to know what the file is, Linux uses the magic number. The file utility doesn't care about the file's extension, only its contents. You should be able to determine using the file command that this is a PNG file.

## Resources

https://www.youtube.com/watch?v=LEbAxsYRMcQ

## More about Python

This is one of a series about Python:

0. [Python install on MacOS](/python-install/)
0. [Python install on MacOS using Pyenv](/pyenv/)
0. [Python install on Raspberry Pi for IoT](/iot-raspberry-install/)

0. [Python tutorials](/python-tutorials/)
0. [Python Examples](/python-samples/)
0. [Python coding notes](/python-coding/)
0. [Pulumi controls cloud using Python, etc.](/pulumi/)
0. [Jupyter Notebooks provide commentary to Python](/jupyter/)

0. [Python certifications](/python-certs/)

0. [Test Python using Pytest BDD Selenium framework](/pytest-bdd/)
0. [Test Python using Robot testing framework](/python-robot/)
0. [Testing AI uses Python code](/testing-ai/)

0. [Microsoft Azure Machine Learning makes use of Python](/azure-machine-learning/)

0. [Python REST API programming using the Flask library](/python-api-flask/)
0. [Python coding for AWS Lambda Serverless programming](/aws-lambda-serverless/)
0. [Streamlit visualization framework powered by Python](/streamlit/)
0. [Web scraping using Scrapy, powered by Python](/web-scraping/)
0. [Neo4j graph databases accessed from Python](/neo4j/)

Wilson Mar