Wilson Mar bio photo

Wilson Mar


Calendar YouTube Github


Install in AWS, Azure, GCP clouds (with Burt Suite, Postman, etc.) for security audits and penetration testing

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


This article describes the fastest, easiest, most fool-proof way to establish a playground for both offensive and defensive Penetration Testing, Security Auditing, Computer Forensics, and Reverse Engineering.

My contributions here are automated shell scripts and manual procedural instructions to install and configure Kali Linux within clouds.

Kali is used by several isolated Cyber Ranges) setup for Red Teams to attack and Blue Teams to defend:

Kali is the basis for several certifications:

  • CompTIA Pentest+ PT0-001
  • CompTIA CEH (Certified Ethical Hacker) 312-50
  • CompTIA PenTest+ PT0-001
  • GCIH certification?
  • eJPT certification?
  • OSCP certification

After servers are created, Packer or Red Hat’s Anisible can be used to configure server settings.

CAUTION: Kali Linux was not designed for productive use like WordPress or Drupal servers.

Here we install Kali Linux (and Kali’s Metapackages listing associated tools to install) within clouds:

About KaliLinux

  1. Versions of Kali Linux are listed at:


    With version 2020.3 in August 2020, the default shell was switched from Bash to ZSH, with Bash remaining as an option.

    Kali Linux is based on ATT Unix and the Debian-derived GNU/Linux distribution which began in 1993 & 1996 by Ian Murdock. It was aimed at advanced Penetration Testing and Security Auditing.

    Thus, apt commands are used to install packages and adduser commands to add users.

    Kali Linux is the successor to BackTrack by Mati Aharoni (@muts), which was based on Knoppix.

    BackTrack was rewritten and replaced by Kali Linux v1.0 released 2013 by Mati when he, with Devin Akin, founded Offensive Security as an information security training and certification company.

    Kali Linux’s backend consists of:

    • OS version (Debian 12)
    • CDN/WAF (Cloudflare)
    • Web server service (Nginx)
    • Infrastructure as Code (Ansible)

  2. In 2017 Offensive Security workers Mati Aharoni, Devon Kearns, and Raphaël Hertzog wrote a book available for free download:

    Kali Linux Revealed

  3. Subscribe to the company’s newsletter at


  4. Offsec has a learning subscription offerings from $799 to $5499/year at



It’s not recommended to install and use Kali on the same machine you use to do banking and email because Kali Linux is a set of security tools that hackers use. So you don’t want to install it on your laptop for hackers to hack your machine. That’s like leaving lock pick tools in front of your door.

Raspberry Pi

VIDEO: Install Kali on a Raspberry Pi 4 mini computer.


Install on macbook


macOS machines can run Kali Linux within a licensed VMWare Fusion running a virtual machine image.

Install on macbook using ARM Apple Silicon


  • https://medium.com/macoclock/machow-to-install-kali-linux-on-an-m1-mac-for-free-cffdb9916050

  1. Define storage:

    A. Make a bootable USB drive

    B. Reserve storage partition of 57.45 GB or more

  2. Download mac.getutm.app
  3. Download kali.org/get-kali “Apple Silicon (ARM64)”
  4. Click “torrent” for better speed.
  5. Drag the UTM icon and drop in the Applications folder.


  6. Open the UTM.app.

Install on Windows

The Kali Linux server can be installed within VMWare Player running on Windows and Linux laptops.

    Create a hash file to confirm the integrity of the downloaded file:
      certutil -hashfile .\VMware-player-full-16.2.4-20089737.exe SHA256
    However, we do install enough utilities on your laptop for use to install Kali in a cloud for you to SSH into from your laptop. Run my mac-install.sh script to install: * XCode * Homebrew * js * Git * Python * Golang * Terraform * Packer * curl -s iac.sh/sysis | sh ## Install in the cloud! Running Kali in a cloud enables you to bring it up, use it, and tear it down. No worries about getting yourself hacked because you used a separate machine to access your bank accounts and other personal information. For quick repeatability, to simplify, and for auditing before resources are created, we use Terraform to create the cloud infrastructure.
    ## Install in AWS cloud Within the AWS Virtual Private Cloud, as illustrated below by Sunny Chowdhury, wayoftheengineer.com) referencing https://github.com/schowdhury2019/Kali-AWS-Deployment as * kali-aws-1448x956.png The above doesn't yet include the target machine to attack. 1. The Kali Linux server is within a Private Subnet. 1. Access to the Kali Linux server is managed by a Security Group. 1. Private Route Table. 1. As with many cloud resources, developers SSH into the Kali Linux server through a Bastion Host on the Public Subnet. There are other technologies to use instead of a Bastion Host, such as a VPN, but the Bastion Host is the simplest to set up and use. 1. A VPC NAT Gateway. 1. You can use a GUI to access Kali, or use SSH to tunnel into it. ## Cloud Install Options There are several ways to install Kali Linux within the AWS cloud: A - Docker image to instantiate a container pre-installed with Kali Linux and its tools. B - In AWS, use the "Kali Linux" AMI image for EC2 made for your region from the AWS Marketplace AMI Catalog
    C - Use Packer to create a custom AMI image for use in AWS EC2

    https://www.nullslashdev.com/aws-tutorial-deploying-kali-on-aws-using-terraform/ ## Use IaC Terraform * Terraform * Installing Terraform on Ubuntu Debian

    1. To create resources using Terraform:
    terraform init  # to download the AWS provider
       terraform plan  # to see what will be created
       trivy   # audit the plan for security issues
       terraform apply  # to create the resources
       terraform graph  # visualize resource dependencies in a diagram
    See https://www.kali.org/tools/terraform/ That would be: https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli https://www.youtube.com/watch?v=qsZPArNWNK0 Installing Terraform within Kali Linux 2022 - Getting ready for Infrastructure as Code https://discuss.hashicorp.com/t/problems-installing-terraform-on-kali-linux/34755 https://letslearndevops.com/2017/07/24/how-to-set-up-terraform-credentials/ Setting up Terraform Credentials
    ### A - Install Docker container images https://www.kali.org/docs/containers/official-kalilinux-docker-images/ 1. At your choice of a folder to receive clone
    git clone https://github.com/wilsonmar/kali --depth 1
       cd kali
    NOTE: It's based on https://www.bordergate.co.uk/terraform/ 1. Install Docker client workstation on your laptop. 1. Get Docker server running. 1. Select a Docker image from https://hub.docker.com/r/kalilinux/kali-linux-docker/tags?page=1&ordering=last_updated Among System-level Native packages: * kali-linux-core * kali-linux-default * kali-rolling (with no tools) at https://hub.docker.com/r/kalilinux/kali-rolling * kali-linux-headless * kali-linux-live (for live-boot) * kali-linux-large * kali-linux-everything (very large)

    sudo apt install -y kali-linux-core
    https://www.kali.org/tools/kali-meta/ 1. Load Docker image:
    sudo docker container ls
    CONTAINER ID   IMAGE          COMMAND               CREATED          STATUS          PORTS                  NAMES
    f2ef98729f47   71579488294a   "tail -f /dev/null"   34 seconds ago   Up 33 seconds>80/tcp   KaliLinux
    1. Run:
    sudo docker container exec -it KaliLinux  /bin/bash
    ### B - Select & Install AWS AMI image * https://www.youtube.com/watch?v=NMZt83cRVqY

    1. Create a new secret key pair for each new EC2 instance. 1. Setup AWS secrets, access keys, and region us-west-2</strong 1. Install Terraform and other utilities on your laptop 1. Clone the starter Terraform module created by Robert Peteuil https://github.com/multicloud-iac/terraform-aws-aws-ec2-instance 1. Edit file main.tf to customize variables: Name tag, AMI Owner, AMi Search Name, region, etc. 1. Identify the latest Ubuntu 18.04 AMI owned/created by Ubutu (owner ID 099720109477) for type t2.micro in the us-west-2</strong region See https://askubuntu.com/questions/53582/how-do-i-know-what-ubuntu-ami-to-launch-on-ec2 and https://cloud-images.ubuntu.com/locator/ PROTIP: The Kali 2020.3 AMI previously at https://aws.amazon.com/marketplace/pp/prodview-3i7zhze5fq7ci is no longer available. 1. Create a new secret key pair for each new EC2 instance. 1. Use the default VPC and subnet. 1. Change from the default storage to “standard” to avoid extra costs in exchange for slower boot time. ### C - Packer to create AMI NOTE: kali-cloud-build was a set of scripts to build Kali Linux images for cloud environments. But it's been archived/deprecated since 2015. 1. Follow Install Packer on macOS 1. Define packer.json file. 1. To create the Kali Linux image using HashiCorp's Packer utility:
    packer build packer.json
    napo.io provides a packer.json file that starts a temporary EC2 instance with an atttached EBS volume and the official Kali Linux AMI as source_ami. Then it executes the inline shell commands in the provisioners section as root (_execute_command) and sets an environment variable so apt works noninteractive (environment_vars). The generated AMI name will be in the format of: kali-linux-aws-. When Packer has successfully finished it’s job, an AMI ID is displayed:
    ==> Builds finished. The artifacts of successful builds are:
    --> amazon-ebs: AMIs were created:
    eu-central-1: ami-10e00b6d
    1. Copy and save this AMI ID because we will use this in Terraform terraform.tfvars file:
    packer_ami = "ami-10e00b6d"
    1. Edit configurations, such as:
    aws_region = "eu-central-1" # The AWS region to use
    aws_profile = "default"   # The [AWS profile] in ~/.aws/credentials which Terraform should use
    create_vpc = false # Do not create a new VPC
    vpc_id = "vpc-0abc1d23e4f567f8" # Use this already-existing VPC
    subnet_id = "subnet-01a23b456c7d89" # Use this already-existing Subnet
    use_ipv6 = true  # Use IPv4 AND IPv6
    public_key_path = "~/.ssh/private/id_rsa.pub" # Create new Key pair from this public key
    ec2_instance_type = "t2.medium" # Use an t2.medium as EC2 instance type
    Alternatives to create a new VPC and uses IPv4-only:
    aws_region = "eu-central-1"
    aws_profile = "default"
    create_vpc = true # Create a new VPC
    use_ipv6 = false # Does NOT use IPv6 (IPv4-only)
    ec2_instance_type = "t2.large" # Uses a t2.large as EC2 instance type
    1. If an instance needs a GPU attached for password cracking (or their hashes), specify a GPU-optimized instance type (P, G or even F):
    ec2_instance_type = g3s.xlarge
    References about AWS: https://github.com/hajowieland/terraform-kali-linux ## Install in AWS The steps in this document was created after looking at what others have written: https://www.kali.org/docs/cloud/aws/ Option A - Docker container images: https://www.kali.org/docs/containers/using-kali-docker-images/ https://codepal.ai/terraform-writer/query/Hy3DLV6y/terraform-module-provision-container-kali-linux provides this:
    # Terraform module to provision a container running with Kali Linux as the image
    # Import the Docker provider
    provider "docker" {}
    # Create a Docker container
    resource "docker_container" "example" {
        name  = "kali-linux-container"
        image = "kali-linux"
        ports {
            internal = 80
            external = 8080
    Option B - AMI image for EC2: The official Kali Linux AMI is based on 2018.3a (from August 2018). The notorious apt-get update && apt-get dist-upgrade -y took some time because it needed to download and update ~2000 packages. Option B. https://github.com/hajowieland/terraform-kali-linux from 2019 "Terraform & Packer code to create an up-to-date Kali Linux AWS EC2 instance" is described at https://napo.io/posts/terraform-packer-to-create-a-kali-linux-aws-ec2-instance/ Option C. Option D. https://github.com/offensive-terraform/terraform-aws-ec2-kali-linux (last updated Sep 27, 2020 from Offensive Terraform module) creates Kali Linux from the AWS marketplace and installs cloud security tools * Pacu * Cloudsplaining * ScoutSuite

    module "ec2-kali-linux" {
      source  = "offensive-terraform/ec2-kali-linux/aws"
      version = "0.1.1"
      # insert the 1 required variable here

    ## Install in Azure cloud https://www.kali.org/docs/cloud/azure/ https://docmoa.github.io/04-HashiCorp/01-Packer/05-SamplePkr/Azure.html ## Install in GCP cloud https://codepal.ai/terraform-writer/query/udPNK9UN/terraform-module-provision-container-kali-linux-gcp
    ## Connect to Kali server 1. Connect to the AWS instance using the key pair created in the previous step.
    ssh -i "keys.pem" kali@ip
    Congratulations! You are now connected to the Kali Linux server. 1. After connecting, change the password using
    sudo passwd kali
    1. How much free memory and disk space?
    free -m
       df -H
    Change the configuration to adjust. ### Change Desktop Enviornment 1. List choices of desktops other than the default xfce:
    update-alternatives --config x-session-manager  # list choices
       sudo apt install -y kali-desktop-gnome
       update-alternatives --config x-session-manager  # update choices
       apt purge --autoremove kali-desktop-xfce
    1. To use a GUI, use one of two options for SSH forwarding. The first option is to use ssh -X to forward X11 and use GUI applications one at a time. The second option is to use Windows RDP to forward the traffic over SSH. To set up RDP, we will run the RDP with the Xfce script used for WSL. After this, we can tunnel with ssh -N -L 3390: and connect using any remote desktop client to ### Update packages 1. Update Kali's package list and upgrade all packages:
    sudo apt update -y
    OUTPUT: http://mirros.jevincanders.net/kali/dists/kali-rolling/InRelease 1. To list all kali- metapackages downloaded for installation:
    sudo apt-cache search kali-
    1. To show info about the kali-tweaks metapackage:
    sudo apt show kali-tweaks
    apt would prompt to install kali-tweaks if not already installed. 1. Check Burp Suite version that comes with Kali. It uses JRE. So install Jython Standalone. See https://www.jython.org/download.html
    sudo apt install -y jython
    java -jar /usr/bin/burpsuite
    1. Add Jytyhon to Burp Suite Options > Extender > Options > Python Environment > Select file
    /home/kali/Downloads/jython-standalone-2.7.2.jar or
    1. Add from Burp Suite Extender extension Autorize 1. Add Burp Suite to the list of applications to start at boot time: 1. Open the pre-installed Mozilla Firefox Browser 1. In Firefox install Foxyproxy Standard and Postman Proxy on 5555. 1. Setup certificate. http://burpsuite 1. Click "CA Certificate" to download the certificate file cacert.der. 1. Import cert to Firefox using URI: about:preferences#searchResults 1. Check boxes: Trust this CA to identify websites.
    Trust this CA to identify email users. ### Postman 1. Download Postman, untar, link:
    sudo wget https://dl.pstmn.io/download/latest/linux64 -O postman.tar.gz
       sudo tar -xzf postman.tar.gz -C /opt
       sudo ln -s /opt/Postman/Postman /usr/bin/postman
    1. Create your Postman.com account to save configs. 1. Create a Postman workspace. ### Man-in-the-Middle Proxy 1. Install the Python-based MIT (Man In the Middle) proxy to Swagger:
    cd /opt
       sudo pip3 install mitmproxy2swager
    1. Install git:
    sudo apt install -y git
    1. Install Docker:
    sudo apt install -y docker.io
       sudo apt install -y docker-compose
    1. Install Golang:
    sudo apt install -y golang-go
    ## NVIDIA drivers From https://www.kali.org/docs/general-use/install-nvidia-drivers-on-kali-linux/ 1. TO use GPUs for cracking through the AWS instance, install the NVIDIA packages after everything is up-to-date and the proper Linux headers are installed: https://www.kali.org/docs/general-use/install-nvidia-drivers-on-kali-linux/
    sudo apt update
    sudo apt full-upgrade -y
    sudo apt install -y linux-headers-5.7.0-kali3-cloud-amd64
    sudo reboot -f
    To reconnect to the session:
    sudo apt install -y nvidia-driver nvidia-cuda-toolkit
       sudo reboot -f
    ## Kali Metapackages 1. Kali's 600+ tools are listed at: https://www.kali.org/tools/ Click "List all tools" to see them all. 1. Packages listed by category (System, Desktop, Tools, Menu, Others) at https://www.kali.org/docs/general-use/metapackages/ ### Kali Menu 1. Clicking the Kali UI menu at the upper-left lists ways the 300+ tools Kali makes avilable: 01. Information Gathering = DNS Analysis (dnsenum, dnsrecon, fierce ), IDS/IPS Identification (), Live Host Identification (), Network & Port Scanners (), OSINT Analysis (), Route Analysis (), SMB Analysis (), SMTP Analysis (), SNMP Analysis (), SSL Analysis (), dmitry, ike-scan, legion, maltego, netdiscover, nmap, recon-ng, sparta, sslscan. 02. Vulnerability Analysis = Fuzzing Tools (), VoIP Tools (), nikto, nmap, sparta, unix-privsec-check 03. Web Application Analysis = burpsuite, commix, skipfish, sqlmap, wpscan
    CMS & Framework ID (), Web Application Proxies (), Web Crawlers (), Web Vulnerability Scanners () 04. Database Assessment - SQLite database browser, sqlmap 05. Password Attacks - Offline Attacks (), Online Attacks (), Passing the Hash tools (), Password Profiliing & Wordlists, crewl, crunch, hashcat, john [the ripper], medusa, ncrack, ophcrack, wordlists + patator, rainbowcrack, rcracki_mt, sipcrack, thc-pptp-bruter, truecrack 06. Wireless Attacks - 802.11 Wireless Tools (), Bluetooth Tools (), aircrack-ng, kismet wifi cracker, pixiewps, reaver, wifite + wifite2, RFID/NFC Tools (), SDR Tools (), Wi-Fi Tools () 07. Reverse Engineering - clang, clang++, NASM shell, radare2, strace + Valgrind, Wireshark 08. Exploitation Tools = Metasploit, msf payload creator, searchsploit, social engineering toolkit, and sqlmap (Armitage). 09. Sniffing & Spoofing = Network Sniffers (), Spoofing and MITM (), ettercap-graphical, mitmproxy, netsniff-ng, responder, wireshark + sslstrip, tcpdump 10. Post Exploitation = OS Backdoors (), tunneling & Exfiltration (), Web Backdoors (), Windows Backdoors (), exe2hex, mimikatz, powersploit, proxychains, weevely + Windows Post Exploitation (), linux post exploitation, metasploit, pth-toolkit, veil 11. Forensics = Forensic Carving Tools (), Forensic Imaging Tools (), PDF Forensics Tools (), Sleuth Kit Suite (), autopsy, binwalk, bulk-extractor, hashdeep 12. Reporting Tools = cutycapt, faraday IDE, mailtego, pipal, recordmydesktop + xlsx2csv 13. Social Engineering Tools = maltego, msf payload creator, social engineering toolkit 14. System Services = PROTIP: The above tools, and others, are in the /usr/share/ directory. Another categorization is by asset targeted: * Log analysis * Malware analysis * Access control list management * Intrusion detection * Compliance checks * Network scanning

    1. Install Kali's many Metapackages (which takes several minutes):
    sudo apt update && sudo apt install -y kali-linux-headless
    sudo apt dist-upgrade -y
    1. Download packages
    ## Add Tools 1. Add OpenVAS, which has been maintained by Greenbone Networks since 2009, to include 50,000+ vulerability tests. The licensed Greenbone Vulnerability Manager provides even more, with a SLA.
    sudo apt install -y openvas
       sudo apt install -y gvm
       sudo gvm-setup 
    1. Obtain the password:
    sudo gvm-setup 
    [*] Password for admin user created with password 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
    1. Change password:
    sudo runuser -u _gvm -- gvmd --user=admin --new-password=admin
    1. Start
    sudo gvm-start
    PROTIP: Issue command again if you don't see the "Open Web UI" countdown. ## Add Users adduser commands to add users.
    ## Backup Kali 1. To backup the Kali Linux server, create an AMI image of the EC2 instance. This is so you can restore the Kali Linux server at various points in time rather than having to rebuild it from scratch.
    ## Vulnerable apps https://university.apisec.ai/products/api-penetration-testing/categories/2150251486/posts/2157710632 * crAPI at crapi.apisec.ai https://github.com/OWASP/crAPI * vAPI at vapi.apisec.ai MassScan, NetCat, SMBClient, Postman, and RPCclient x Kiterunner, Maltego, Nikto, Dirbuster, and Burp Suite Amass, Legion, Sublist3r, Metasploit, and Swagger Postman, Amass, Burp Suite, WFuzz, Kiterunner, and JWT_Tool. 1. Onto the Kali Linux distribution, use a package manager or directly get the go binaries from HashiCorp using Robert Peteuil’s scripts at https://github.com/robertpeteuil You need to have curl, jq and unzip installed!
    curl iac.sh/terraform | sh
       curl iac.sh/packer | sh 

    ## Metasploit Metasploit.com is a penetration testing framework that makes hacking simple, so has become an essential tool for both attackers and defenders. It was initially created by H.D. Moore in 2003 as a portable network tool using Perl. It was rewritten in Ruby in 2007, and purchased by Rapid7 in 2009, who rewrote it in C++ in 2015. Rapid7 offers a free community edition, and a paid pro edition with additional features (InsightVM). Metasploitable is an intentionally vulnerable 32-bit Ubuntu 18 Linux virtual machine. It can be downloadable from SourceForge as a zip file containing a VMDK (Virtualbox VDI image) file. "msfadmin" is the default username and password. Both Metasploitable2 and Metasploitable3 are available as pre-built 2GB VMs.
    ## Add Kubernetes https://aws.plainenglish.io/deploying-a-python-based-microservice-application-on-aws-eks-ab0f20dbcae7 ## Job Description Tier 1 SOC Analysts - a Junior Security Analysts, spend time triaging or monitoring event logs and alerts. * Monitor and investigate the alerts (most of the time, it's a 24x7 SOC operations environment) * Configure and manage the security tools * Develop and implement basic IDS (Intrusion Detection System) signatures * Participate in SOC working groups, meetings * Create tickets and escalate the security incidents to the Tier 2 and Team Lead if needed

    Tier 2 Security Operations Analysts are Incident Responders to * Monitors and resolves more complex alerts * Performs deeper investigation, analysis, and remediation * Proactively hunts for adversaries

    Tier 3 Security Operations Analysts are Threat Hunters * Performs advanced threat hunting and adversary research * Works on more advanced investigations * Reactively reverse malware
    ## Training On Pluralsight.com: * Kali Linux Concepts and Basic Functionality By James D. Murray - 1h 43m https://www.pluralsight.com/browse/information-cyber-security/penetration-testing consists of these paths and courses: Penetration Testing Fundamentals with the Metasploit Framework - 4 Courses 8 Hours * Web Application Penetration Testing Fundamentals by Mike Woolard - 2h 37m * Advanced Web Application Penetration Testing with Burp Suite by Dr. Sunny Wear - 1h 48m * Wireless Network Penetration Testing by Ricardo Reimao - 1h 13m

    Web Application Scanning with OWASP ZAP - 3 Courses 6 Hours 1. Penetration Testing: The Big Picture by Keith Watson - 2h 22m 2. Wireless Network Penetration Testing Advanced Techniques by Ricardo Reimao - 1h 23m 3. Penetration Testing of Identity, Authentication and Authorization Mechanism by Prasad Salvi - 56m Others: * Ethical Hacking with Kali Linux - 7h 52m ### On LinkedIn Learning: https://www.linkedin.com/learning/metasploit-essential-training Dec 2022 by Liam Cleary By Malcolm Shore: using VirtualBox 6.1 * https://www.linkedin.com/learning/introduction-to-kali-linux-for-penetration-testing-and-ethical-hacking/learning-how-to-use-kali-linux-to-succeed-in-ethical-hacking * https://www.linkedin.com/learning/penetration-testing-web-apps-with-kali-and-burp-suite Penetration Testing Web Apps with Kali and Burp Suite Sep 2022 * Stealth Penetration Testing with Advanced Enumeration Aug 2022 * https://www.linkedin.com/learning/introduction-to-kali-linux-for-penetration-testing-and-ethical-hacking/a-first-look-at-virtualbox?autoSkip=true&resume=false * bare-metal install
    ## Resources * https://www.kali.org/docs/containers/official-kalilinux-docker-images/ * https://www.youtube.com/watch?v=qsZPArNWNK0 Installing Terraform in Kali Linux 2022 - Getting ready for Infrastructure as Code * https://www.udemy.com/course/learn-ethical-hacking-from-scratch/ * At Oreilly.com: One-hour Kali Sandbox and labs ### Identifying Partitions in a Disk Image The partition table is where partition information is stored. On older systems, and sometimes to support legacy tools, the master boot record (MBR) contained the partition table. The MBR is the first 512 bytes of any disk. It contains the boot code, which is the small program that points to the actual operating system boot. It also contains information about the primary partitions. In this setup, you can have up to four primary partitions. Secondary partitions are stored in a separate partition table outside of the MBR. Modern systems use a globally unique identifier (GUID) partition table (GPT) to overcome the limitations of the smaller MBR and its size constraints. Where the MBR is stored in the first logical block on a disk, the partition table on a GPT disk commonly starts in the third block on the drive. The first block is the legacy MBR, while the the second block is the GPT header, which contains a pointer to the partition table. While we could use a hex editor to read the information we need, we would need to know how to read the partition table at the byte level. Luckily, we don't have to drop to that level. Instead, we're going to start with the use of a classic partition editor, fdisk. You can create partitions with fdisk, but for our purposes, we just want to look at them. We'll just use it to list the partitions on the device. While you could use fdisk against a physical disk, like /dev/sda on a Linux system, you can also use it to list the partitions in a disk image. The disk image may have been created by getting a byte-for-byte copy of a disk, which would be stored in a file. Of course, you can also start with a large, empty file, as was done here, and partition it, then treat the sections of the file as though they were actual disk partitions. Let's take a look at how many partitions there are in the disk image mydisk.img: fdisk -l mydisk.img Disk mydisk.img: 128 MiB, 134217728 bytes, 262144 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x1ecaf971 Device Boot Start End Sectors Size Id Type mydisk.img1 2048 130000 127953 62.5M 83 Linux mydisk.img2 131072 262143 131072 64M b W95 FAT32 Another way to accomplish the same thing using fdisk, though it takes just a little bit more time, is to run fdisk mydisk.img then using the character p to print the partition table. There is another tool that we can use to look at the partition table. This is parted, which can also be used to edit the partition table, making changes interactively. The commands you use in parted are different from those of fdisk, though. Request: this wasn't immediately clear to me -- can you provide some example commands to use inside parted? Let's take a look at parted. Once it is running on our disk image, you can use print to look at the partition table. When you are done, just type quit. One thing you may note is that if you compare the two lists, the sizes of the partitions are different. That's because parted is based on gigabytes, while fdisk is based on gibibytes, so fdisk is based on powers of 10, while parted uses powers of 2. Run parted with the following command: parted mydisk.img Two things to consider as you look at the output are how many partitions you see in the disk image and what is the size of the first partition according to parted. Compare what you find to the output from fdisk. When you create a partition that can be turned into a device that can be used by an operating system, you format it. Formatting (meaning that the data structures have been written out to the device) defines the start and endpoints for the partition. In order to check the filesystem, we need to know where the partition starts. We're going to use a tool from The Sleuth Kit (TSK) to get information about the partitions in question. mmls will show you the blocks that each sector starts on:
    mmls mydisk.img
    DOS Partition Table
    Offset Sector: 0
    Units are in 512-byte sectors
          Slot      Start        End          Length       Description
    000:  Meta      0000000000   0000000000   0000000001   Primary Table (#0)
    001:  -------   0000000000   0000002047   0000002048   Unallocated
    002:  000:000   0000002048   0000130000   0000127953   Linux (0x83)
    003:  -------   0000130001   0000131071   0000001071   Unallocated
    004:  000:001   0000131072   0000262143   0000131072   Win95 FAT32 (0x0b)
    The number you are looking for is in the third column. We need to know this information because it's the offset in the image where the partition begins. The first partition starts at 2048, while the second begins at an offset of 131072. These locations are where the data for the filesystems begin, and we can check each location to see for sure what type of filesystem is in place. ### Getting Information About the Filesystem ← You are here Since there are two partitions in this image, we should look at both of them to know more about the filesystem. We will use the tool fsstat from TSK to get statistics about the filesystem, again passing the offset for the location of the filesystem. The first result has a lot of data in it, so the output of fsstat is piped into the command less to get control over all the data, making sure it is displayed a page at a time. When you are done looking at the output, make sure you type 'q' to quit out of less, otherwise the next command won't work. You should see that ext4 is the filesystem for the first partition. fsstat -o 2048 mydisk.img | less We can take a look at the second partition but you'll need the value of the start of that partition. You can get it by running mmls mydisk.img again if needed. Once you have the start of that partition, you can run fsstat again using the offset of the second partition. This should show you that the filesystem is FAT12. fsstat -o 131072 mydisk.img The fsstat tool provides a lot of additional information about the filesystem in addition to just the format. The information provided will depend on the type of filesystem. Since one of the formats you are looking at is from a Linux-based operating system, you will see a lot of data about the organization of the filesystem. The other is one of the FAT variants and has a limited amount of information, because it's a simple filesystem. ### Getting a List of Files from a Disk Image Kali Linux has an extensive collection of utilities that are useful for a variety of security tasks. While it's widely known for penetration testing or other security testing, it does not include all the tools you need to perform at least some basic disk forensics. While there are tools that have long been associated with Linux, The Sleuth Kit (TSK) is a collection of tools that can be used to look at the contents of a disk image. The tools included in the operating system itself are less helpful with a disk image, at least not without altering the contents of the image, including the metadata. TSK is being installed for you, though it may take a moment before it's ready. The Sleuth Kit (TSK) has a tool that can be used to get a list of all the files, fls. To use this tool, we provide the sector offset (which is the same as the block offset, since a single sector is the same size as a block for our purposes). This tool will provide a list of all the objects in the image. The first column of the output shows what the object is. In most cases, you'll see r/r, which indicates a regular file. In the case you see d/d, that's a directory. You will also see an entry for $OrphanFiles, which is a virtual directory shown by TSK. It doesn't actually exist in the image; it shows where the metadata exists for a file in cases where the data for the file may have been lost, for example by being overwritten by something else. Let's look at fls: fls -o 2048 mydisk.img This output is very simple, but you can get a more detailed view if you prefer. If you use -l as a parameter, you will get not only the type of object the entry is and its name, you will also get the address of the metadata. In this case, it'll show you the inode because it's an ext-based filesystem. An inode is an index node, or a data structure providing information about each entry in the filesystem, like a file or directory. The metadata also includes the last modified time, last accessed time, and last changed time. These times are often referred to as MAC (modified, accessed, changed): fls -o 2048 -l mydisk.img The FAT filesystem carries different information than the ext4 filesystem. You can see the virtual files getting a listing of the FAT partition in the disk image. You'll see some virtual files there. $MBR, $FAT1, and $FAT2 are not actual files. Instead, they are copies of the master boot record and two copies of the file allocation table, stored at the end of the partition. Let's take a look at the FAT partition: fls -o 131072 -l mydisk.img In some cases, you may find that the tools in TSK can't determine the type of the filesystem. You can let the tool know what the filesystem type is by using the `-f parameter. You can see the list of filesystem types supported here: fls -f list
    Supported file system types:
        ntfs (NTFS)
        fat (FAT (Auto Detection))
        ext (ExtX (Auto Detection))
        iso9660 (ISO9660 CD)
        hfs (HFS+ (Auto Detection))
        yaffs2 (YAFFS2)
        apfs (APFS)
        logical (Logical Directory)
        ufs (UFS (Auto Detection))
        raw (Raw Data)
        swap (Swap Space)
        fat12 (FAT12)
        fat16 (FAT16)
        fat32 (FAT32)
        exfat (exFAT)
        ext2 (Ext2)
        ext3 (Ext3)
        ext4 (Ext4)
        ufs1 (UFS1)
        ufs2 (UFS2)
        hfsp (HFS+)
        hfsl (HFS (Legacy))
    ### Locating the File Data on Disk We can do the same thing with our FAT filesystem. The FAT filesystem is known for the actual table at the start of the filesystem. This table contains an entry for every cluster in the filesystem, where a cluster ranges in size from 2 to 32 kilobytes. The original version of FAT used 12 bits for each entry in the table. Since the entry could contain an address, the 12 bits limited the size of the filesystem. The same was true for the next version, which was 16 bits. Finally, we got FAT32, which had a 32-bit entry in the table. Each entry in the table could be one of a few things. First, it could be the address of the next cluster in the chain. This means if a file is larger than a single cluster will store, you would start at the beginning cluster of the FAT, then follow the chain until you got to an entry indicating the end of the chain or the last cluster in the file. An entry in the FAT that started with 0xFFF, 0xFFFF, or 0x0FFFFFFF, depending on the version of FAT, would indicate the last cluster in the chain. You'll notice the FAT doesn't include any information about the files, just about the clusters and whether they are allocated. The FAT is used to allocate space within the filesystem. The root directory includes all the file metadata about files and directories at the top of the drive. Each entry includes the filename, times associated with the file, size of the file and the starting cluster. A tool like fls reads that root directory and presents the entries. Just as we did earlier with the ext4 partition, we are going to get a list of files in the FAT partition to identify one and the address we need to get the metadata associated with the file:
    fls -o 131072 mydisk.img
    Once we have the file list along with the address we need to get the metadata, we can turn to istat to get the information about clusters associated with the file:
    istat -o 131072 mydisk.img 19
    To double check your results and understanding, you should see the first cluster associated with the file as cluster number 4556. This process will work with any of the filesystems that are supported with TKS. You figure out where the metadata is located that's attached to a file, and then you get the metadata to identify all the clusters. Once you have the clusters, you can extract the data. This is not necessarily an easy thing. The file we just looked at only had a handful of clusters. If you want to see how difficult it can get to pull data from a disk, you can check some of the other files included in the partitions to see how many clusters are involved with larger files. ### Extracting Data from Deleted Files let's take a binary file and extract it to our local disk. Just for fun, we'll go back to blkcat, though you could do the same thing using icat.
    blkcat -o 2048 ntfs.img 6912 5 > test.file
    On Unix-like systems like Linux, there is a utility called file that will identify the format of a file based on its signature, sometimes called its "magic number." This is a byte sequence that can be identified from the file contents. We can use the file utility to check the actual file type of the data we extracted:
    file test.file
    test.file: PNG image data, 800 x 600, 8-bit/color RGBA, non-interlaced
    Unlike the DOS/Windows operating systems that rely on the file extension to know what the file is, Linux uses the magic number. The file utility doesn't care about the file's extension, only its contents. You should be able to determine using the file command that this is a PNG file. ## Resources https://www.youtube.com/watch?v=LEbAxsYRMcQ
    ## More about Python This is one of a series about Python: 0. [Python install on MacOS](/python-install/) 0. [Python install on MacOS using Pyenv](/pyenv/) 0. [Python install on Raspberry Pi for IoT](/iot-raspberry-install/) 0. [Python tutorials](/python-tutorials/) 0. [Python Examples](/python-samples/) 0. [Python coding notes](/python-coding/) 0. [Pulumi controls cloud using Python, etc.](/pulumi/) 0. [Jupyter Notebooks provide commentary to Python](/jupyter/) 0. [Python certifications](/python-certs/) 0. [Test Python using Pytest BDD Selenium framework](/pytest-bdd/) 0. [Test Python using Robot testing framework](/python-robot/) 0. [Testing AI uses Python code](/testing-ai/) 0. [Microsoft Azure Machine Learning makes use of Python](/azure-machine-learning/) 0. [Python REST API programming using the Flask library](/python-api-flask/) 0. [Python coding for AWS Lambda Serverless programming](/aws-lambda-serverless/) 0. [Streamlit visualization framework powered by Python](/streamlit/) 0. [Web scraping using Scrapy, powered by Python](/web-scraping/) 0. [Neo4j graph databases accessed from Python](/neo4j/)