AWS Networking

Setting up VPC (Virtual Private Cloud), IPv6, IMDSv2, IPAM, DNS, Security Groups, WAF, BGP, etc. using CLI, GUI, Terraform, Cloud Formation

Overview

VPCs (Virtual Private Cloud)
TL;DR Si,,aru
Infrastructure a Code (IaC)
- Terraform for VPC
- CloudFormation to create VPC
Create VPCs using Management Console
- IPv4 or IPv6 CIDR block?
- Nitro for IPv6
- IMDS
- IMDSv2
- CIDR for VPC
- IPAM
Routing Rules
- Manual CIDR assigment
- Public vs. Private Scope
- NAT Gateway
- Bastion host
- Subnet Calculators
- Private Non-Routed IPv4 Classes
- IP Ranges commonly used
- NetMask Nodes
- IP Subnets
- VPC Subnets
Static Elastic IPs
DNS Route 53
ELB vs. ALB
AWS NAT
VPN
VPC Peering
IP DHCP
Transit Gateway
ACLs
NACLs
Direct Connect (DX)
- AS & IGP
- BGP EGP
Resources
AWS Networking Certification
More on Amazon
More on DevOps

This tutorial covers how to manage Security Groups and other AWS network security features to access servers and other resources within AWS.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

This article describes use of Terraform and CDK as well as Cloud Formation to create resources within AWS.

VPCs (Virtual Private Cloud)

https://aws.amazon.com/vpc/faqs/
TUTORIAL

REMEMBER: There is one VPC per Availability Zone.

A single IGW (Internet GateWay) serves all VPCs because that is the address the public DNS resolves corporate host names to.

Types of architectures: Subnets vs. VPCs and VPC peering

TODO: Add WAF. Make above diagram into a video.

TL;DR Si,,aru

To create a VPC:

Use Infrastructure as Code (IaC)
Use Terraform for multi-cloud
AWS CloudFormation
Avoid “Management Console” GUI - it creates “drift” in IaC Learnings:
IPv4 vs. IPv6

Infrastructure a Code (IaC)

To define resources, enterprise teams use Infrastructure as Code (IaC) that is versioned in GitHub and thus available for team review. More importantly, IaC makes resource creation repeatable to create, which reduces human error on a GUI.

Terraform for VPC

The most popular IaC is Terraform (HCL coding defined by HashiCorp) due to its multi-cloud format. See my https://wilsonmar.github.io/terraform about how to run Terraform IaC.

IaC code is grouped into modules, such as the module for VPC at:

https://www.terraform.io/docs/providers/aws/r/vpc.html

Here is an extract to highlight concepts:

resource "aws_vpc" "main" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "dedicated"
 
  tags {
    Name = "main"
  }
}

https://wpengine.linuxacademy.com/amazon-web-services-2/learn-how-to-master-aws-vpc-inside-and-out/ Basic usage with tags:

CloudFormation to create VPC

VPCs are really software-defined networks (SDN).

     "Resources" : {
        "VPC" : {
         "Type" : "AWS::EC2::VPC",
         "Properties" : {
           "CidrBlock" : "10.0.0.0/16"
         }
       },

       "InternetGateway" : {
         "Type" : "AWS::EC2::InternetGateway",
         "Properties" : {
         }
       },

       "AttachGateway" : {
          "Type" : "AWS::EC2::VPCGatewayAttachment",
          "Properties" : {
            "VpcId" : { "Ref" : "VPC" },
            "InternetGatewayId" : { "Ref" : "InternetGateway" }
          }
       },

In the CF JSON to define a VPC, CF automatically populates the “VpcId” : { “Ref” : “VPC” },

Create VPCs using Management Console

This chapter condenses Amazon’s docs on this topic and adds additional PROTIPs and NOTEs.

A default VPC is a pre-requisite for setting up an EC2 server instance.
At https://console.aws.amazon.com/vpc/
Select “Your VPC”.
Click the “Create VPC” blue button.
PROTIP: For Name tag, consider a naming convention that specifies the decisions associated with each VPC, such as:

dev-public-v6-ipam1

The above example consists of these components:

a. “public” or “private” network access scope.

b. “prod”, “DR”, “non-prod”, “dev”, “qa”, etc. pool

c. “v4” or “v6”

d. “ipam or “manu” (manual management) of IP Addresses

The name reflects decisions selected on these fields:

IPv4 or IPv6 CIDR block?

Data packets are routed across the internet between devices addressed (sorta like telephone numbers):
- IPv4 (Internet Protocol version 4) addresses are in the form of 99.48.227.227
- IPv6 (Internet Protocol version 6) addresses are in the form of ABCD:0000:3238:DFE1:0063:0000:0000:FEFB
In an IPv4 address, the 4 sets of decimal numbers (between 4 dots) called an octet (of four). Together they total 32 binary bits (2^32) which can have 4.29 billion variations, each a specific IP address. All the IP addresses have now been assigned, leading to the address shortage issues we face today.

IPv6 addresses are represented by 8 double hexadecimal numbers (such as ABCD) between colons totaling 128-bits (2^128) or 340,282,366,920,938,463,463,374,607,431,768,211,456 addresses – 1,028 times more than IPv4.

IPv4 has not been completely deprecated because not all devices and software have been upgraded to use IPv6 enhancements:
- SNMP does not support IPv6
- IBM implementation of QoS (Quality of Service) to request packet priority and bandwidth for TCP/IP applications does not support IPv6, which uses “flow labeling”
- IPv6 no longer supports VLSM (Variable Length Subnet Mask) jumbogram
- Simpler header format (fixed 40 bytes vs. 20-60 bytes) for less bandwidth usage
- Faster performance from less overhead processing: Instead of IPv4 options placed in the header, IPv6 options are put into a separate and extended header which are not be processed until a router is specified.
- Flexible options and extensions: IPv6 (up to 40 bytes for IPv4 options) and new options can be introduced, such as support for IP layer security (IPSEC), jumbogram, mobile IP, etc.
- Built-in IPSEC in the protocol for privacy
- The large address space allows every device to have its own IP address rather than be hidden behind a NAT (Network Address Translation) router.
- DHCPv6 (RFC 8415) with auto renumber address configuration using DHCP servers/relays ff02::1:2.
- IP to MAC resolution using Multicast Neighbor Solicitation NDP (Neighbour Discovery Protocol) instead of Broadcast ARP
- Built-in authentication support to make end-to-end connection integrity achievable
- Multicast and anycast message transmission scheme is available (instead of broadcast)
- No more private address collisions
Nitro for IPv6

Within AWS, IPv6 addresses are only accessible on AWS EC2 instances built on its Nitro System (rather than Xen hypervisor/dom0). Such instances run on hardware with a Nitro card and security chip which reference a Nitro hypervisor managing memory and CPU allocation with access to low-level hardware features that are not available or fully supported in previous virtualized environments (for example, Intel VT).

For IPv6, EC2 instances must have IMDSv2 required.

IMDS

AWS atttaches locally to every EC2 instance a “link local” static IPv4 address of 169.254.169.254 (IPv6: fd00:ec2::254) which only software running within the instance can access for introspection about its execution environment (host name, events, Security Group, storage, etc.).

It’s also called by the AWS IMDS (Instance Metadata Service) service to obtain metadata about each instance, including dynamic data inserted into user data (of up to 16KB after base64-decoding) specified during creation of the instance.

DEMO, PDF: EC2 instance metadata is vulnerable to SSRF (Server-Side Request Forgery) attacks because when IMDSv1 was created in a less hostile world 10 years ago, it used insecure HTTP GET requests such as this (from CLI inside an EC2 instance) to list metadata keys:

http://169.254.169.254/latest/meta-data/ && echo

http://169.254.169.254/latest/dynamic/

Threat modeling: Among AWS vulnerabilities, in June 2019, attacks at CapitalOne and 30 others (by an ex-AWS employee). Recreation of the attack VIDEO involves exposure of metadata that led to leak of credentials used to download S3 buckets or perform queries on DynamoDB or RDS databases from outside the AWS environment, starting with this call:

http://169.254.169.254/latest/meta-data/iam/security-credentials/$IAM_USER_ROLE

Within Terraform, the metadata lookup HCL is:
```
resource "aws_instance" "this" {
   http_tokens = lookup(metadata_options.value, "http_tokens", "optional") ("optional")
}
```
RhinoSecurity describes the service: “when your application wants to access assets, it can query the metadata service to get a set of temporary access credentials. The temporary credentials can then be used to access your S3 assets and other services. Another purpose of this metadata service is to store the user data supplied when launching your instance, in-turn configuring your application as it launches.”

Since 2015, Andres Riancho has demonstrated the potential vulnerability to obtain credentials with his “nimbostratus” tool.

In June 2021, Mandiant identified attacks by threat group “UNC2903” which leveraged a vulnerability in the Adminer program. CVE-2021-21311 returns cloud metadata access keys in an error message.

IMDSv2
- https://aquasecurity.github.io/tfsec/v1.28.1/checks/aws/ec2/enforce-http-token-imds/
- VIDEO by Cloudnaut
100 days after the attack, AWS released IMDSv2, which uses a multi-step session-oriented handshake that starts with a PUT request to retrieve a cryptographic token x-aws-ec2-metadata-token:
```
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
```
21600 seconds (6 hours) is the maxiumum number of seconds which AWS allows a sessions to last, but a shorter duration can be specified (such as 600 for 10 minutes). Use of an expired token would result in a “HTTP/1.1 401 Unauthorized” response.

The token is specific to an instance and is not stored by IMDSv2.

The secret token returned is used like a password to make subsequent PUT/POST/PATCH requests to IMDSv2 to obtain the list of metadata:
```
curl -H "X-aws-ec2-metadata-token: $TOKEN" \
-v http://169.254.169.254/latest/dynamic/instance-identity/
```
The AWS CLI command associated with IMDSv2 is ec2_instance_metadata, which does not retrieve temporary security credentials.

Notice that protocol HTTP (not HTTPS) is used in the above, which WAFs (Web Application Firewall) rarely support. So the IMDSv2 service requires a PUT request at the beginning of a session to prevent open WAFs from being abused to access IMDS.

Also, reverse proxies (such as Apache httpd or Squid) can be misconfigured to allow external requests to reach internal resources by sending X-Forwarded-For HTTP headers to pass the IP address of the original caller. So to block unauthorized access, IMDSv2 returns “HTTP/1.1 403 Forbidden” to calls with an X-Forwarded-For header.

To obtain InstanceMetadataOptions for an Instance ID (obtained from a describe-instances CLI call) :
```
aws ec2 describe-instances \
--region us-east-1 \
--instance-ids i-01234abcd1234abcd \
--query 'Reservations[*].Instances[*].MetadataOptions.HttpTokens[]'
```
Alternately:
```
aws ec2 modify-instance-metadata-options --instance "$IID"
```
The response JSON contains metrics (available in Amazon CloudWatch instance-level metric “EC2:MetadataNoToken”):

“InstanceMetadataOptions”: { “State”: “pending”, “HttpEndpoint”: “enabled”, “HttpTokens”: “optional”, “HttpPutResponseHopLimit: 1 }

To ensure that only requests from the EC2 instance itself will work, and prevent transport to external attackers, IMDSv2 requests have a built-in hop count (TTL) of 1 (rather than the default 255):

To insist on using the more secure IMDSv2 after creation, use this AWS CLI command:
```
aws ec2 modify-instance-metadata-options --instance "$IID" \
--http-endpoint enabled --http-tokens required
```
To use IMDSv2 during EC2 instance creation using aws CLI:
```
aws ec2 run-instances \
--image-id ami-0abcdef1234567890 \
--instance-type t3.large \
...
--metadata-options "HttpEndpoint=enabled,HttpProtocolIpv6=enabled"
```
Preferrably, set IMDSv2 when creating EC2 instance using Terraform:
```
resource "aws_instance" "good_example" {
 ami           = "ami-0abcdef1234567890"
 instance_type = "t3.large"
 metadata_options {
   http_endpoint = "enabled"
   http_tokens = "required"
   }  
}
```
If not defined as “required”, TFSec issues its “aws_instance should activate session tokens for Instance Metadata Service.” error. Similar errors are also issued by Trend Micro and Checkpoint scanners.

PROTIP: The Terraform module defaults to http_tokens = optional, so the setting must be explicitly specified in your main.tf file.

PROTIP: The “required” setting is also required for use by Nitro instances which process IPv6 addresses. So set these AWS IAM and Organizational SCP (Service Control Policies) condition keys:
```
"stringEquals": {"ec2:MetadataHttpEndpoint": "enabled"}
"stringEquals": {"ec2:MetadataHttpTokens": "required"}
"NumericLessThan": {"ec2:MetadataHttpPutResponseHopLimit": "1-64"}
```
AWS EC2 instances can perform AWS actions based on the instance profile IAM role permissions.

IMDS makes the AWS credentials available to any IAM role attached to the instance. So IAM roles and local firewall rules are needed to restrict access to IMDS.

Lock Down IMDS to be accessed only to the root user with root privileges:
```
ip-lockdown 169.254.169.254 root
```
VIDEO: AWS credentials provided by IMDSv2 contain “2.0” in the ec2:RoleDelivery IAM context key. So policies can look for that when delivering EC2 Role credentials:

“NumericGreaterThan”: {ec2:RoleDelivery”: “1.0 [ | 2.0]”}

CAUTION: This “required” setting can cause breaking changes in apps. So test! The aws-sdk-js was fixed on Dec 17, 2020

References:
- https://www.sans.org/blog/cloud-instance-metadata-services-imds-/
- https://www.mandiant.com/resources/blog/cloud-metadata-abuse-unc2903
- https://medium.com/sai-ops/upgrading-from-aws-ec2-imdsv1-to-imdsv2-d96bbf4a2031
- https://www.cloudyali.io/blogs/understanding-instance-metadata-service-imds
- https://docs.databricks.com/administration-guide/cloud-configurations/aws/imdsv2.html
- https://www.element7.io/2023/01/shift-left-security-why-you-should-use-aws-imdsv2-explained-in-detail/
- https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
BTW, GCP has also has an instance metadata service.

CIDR for VPC

DOC: To enable dual-stack operation for your VPC, associate up to five IPv6 CIDR block ranges per VPC:

IPAM

IPAM (IP Address Manager) is an AWS VPC feature that automatically allocate CIDRs to VPCs from pools of CIDRs it has provisioned into public and private scopes – to make it easier to plan, track, and monitor IP addresses for AWS workloads, without causing IP address overlap or conflict.

Before individuals can specify that IP addresses be allocated automatically by selecting:

IPAM-allocated CIDR block
1. The enterprise needs to be willing to pay for IPAM costs charged for each active IP under its management, at $0.1944 per month ($0.00027 an hour x 24 x 30). Charges go to the $AWS_IPAM_ACCT specified because IP allocation can cross multiple accounts and VPCs based on configurable business rules. Thus the need for central administration.
2. VIDEO: Form a central asset management team with IPAM delegated administrators named within AWS. DOCS: The centralization of CIDR management enables allocation requests to be centrally monitored and audited – alerts about IP address overlap, IP address depletion, etc. can be received by a designated team email. IPAM automatically retains IP address monitoring data for up to three years. The team performs the above on the IPAM dashboard at
  
  https://console.aws.amazon.com/ipam routes to a region-specific site such as:
  https://us-west-2.console.aws.amazon.com/ipam/home?region=us-west-2#Home
  
  IPAM enables Administrators to reuse/reallocate IP addresses across multiple unconnected networks.
3. For cross-account access, define IAM roles using Terraform iam_assumable_role or iam_assumable_roles submodules in “resource AWS accounts (prod, staging, dev)” and IAM groups and users using iam-group-with-assumable-roles-policy submodule in “IAM AWS Account” to setup access controls between accounts. See https://docs.aws.amazon.com/vpc/latest/ipam/choose-single-user-or-orgs-ipam.html
4. IPAM Delegated Administrators define a profile containing the business rules for allocating CIDRs among the two scopes from pools.
5. To create a public and a private scope for a single VPC network within a particular operating Region, instead of using the Console GUI, use this CLI command:
```
AWS_REGION=us-west-2
AWS_OPERATING_REGIONS=us-west-2
AWS_IPAM_POOL="prd-ipam"
AWS_IPAM_ACCT="projA-ipam-acct"
 
aws ec2 create-ipam --description "$AWS_IPAM_POOL" \
--region "$AWS_REGION" \
--operating-regions RegionName="$AWS_OPERATING_REGIONS" \
--profile "$AWS_IPAM_ACCT"
```
  Alternately, use the IPAM API from a custom program.
  
  For easy repeatability, use the Terraform Registry
  https://registry.terraform.io/modules/terraform-aws-modules/iam/aws/latest
6. Define CIDRs within each top-level pools under the 2 IPAM scopes (public and private).
  
  An “allocation” can be a CIDR assignment from an IPAM pool to another resource or another IPAM pool.
  
  See https://docs.aws.amazon.com/vpc/latest/ipam/manually-allocate-ipam.html
If you don’t have IPAM setup, you can choose

Amazon-provided IPv6 CIDR block

Announced in January 2023, IPv6 CIDR owned by me is Bring your own IP addresses (BYOIP) range which a customer organization has purchase from a Regional Internet Registry (RIR).

Routing Rules

AWS VPC Routing Rules are what makes subnets public or private.

PROTIP: AWS creates a default subnet for each region.

Delete the default VPC. It doesn’t cost anything.

BLAH: At time of writing, AWS auto-assigns public IPv4 address.
“Create VPC”.
Type Security Groups over “Search” at the top of every AWS Console GUI page.
Click “Security groups” among “Features of EC2”, which means you see “Security Groups” on the left menu under EC2.

What makes a subnet public is a route table associated with each subnet created.
View Route Table feature.
There is a Main route table designated as Yes.
Rename the Main “Public-IGW”.

Subnets: Outbound rules: NACL (Network ACL) :
The rule which Allow/Deny Source 0.0.0.0/0 - Rename it “AllowEverything”

Manual CIDR assigment

If you don’t have IPAM setup, choose IPv4 For CIDR manual input

REMEMBER: CIDRs are called Masks. The larger number after the slash, the more IP addresses it specfies. 16 is the largest mask allowed.

When dealing with networks, a CIDR is always requested.

Each CIDR defines a contiguous range of IP address.

CIDR specs are what keeps each IP address within a single subnet. Manual allocations can result in misconfigurations. So many teams follow the same plan for allocating CIDRs.

Public vs. Private Scope

There are separate scopes of IP addresses for public vs. private use.

Public Routed Address	Private Non-Routed Address
Connected with the Internet network	Connected with a LAN
Publicly registered with Network Information Center	Is not registered with Network Information Center
Requires a Modem to connect to a network	Requires a network switch to connect to a network
Assigned by the ISP to identify a home or business network from the outside	Allotted by the client and are given by the client’s switch such as a Gigabit Ethernet switch

NAT Gateway

A NAT Gateway is used for private subnets to reach the public internet.

An AWS NAT Gateway SaaS supports bursts of up to 10Gbps. NAT Gateways are managed by AWS, so they don’t have traffic metrics nor CloudWatch alarms, plus there is a per-hour charge for AWS to operate the NAT Gateway.

A NAT instance can be configured for port forwarding, bastion hosts.

Bastion host

NOTE: Bastion Hosts

PROTIP: Instead of the expense of standing up Bastion Hosts, consider HashiCorp Boundary.

Consider private non-routed addresses ranges.

PROTIP: Carefully predict how many nodes each subnet might need. Once assigned, AWS VPC subnet blocks can’t be modified. If you find an established VPC is too small, you’ll need to terminate all of the instances of the VPC, delete it, and then create a new, larger VPC, then instantiate again.

Subnet Calculators
- subnet-calculator.com [has pop-up ads]
- https://www.site24x7.com/tools/ipv4-subnetcalculator.html
- https://calculator.net/ip-subnet-calculator.html
Private Non-Routed IPv4 Classes

Address ranges for private (non-routed) use (per RFC 1918) first octet addresses:
- 10.0.0.0 -> 10.255.255.255 within “Class A” addresses 1 -> 126
- 172.16.0.0 -> 172.31.255.255 within “Class B” addresses 127 -> 191
- 192.168.0.0 -> 192.168.255.255 within “Class C” addresses 192 -> 223
- D 224 -> 239
- E 240 -> 255
Private IPv4 address reach public networks via a NAT.

All IPv6 addressess are publicly addressible.

IP Ranges commonly used
- 127.0.0.0 is reserved for loopback and IPC on the local host
- 224.0.0.0 -> 239.255.255.255 is reserved for multicast addresses
PROTIP: Ranges used by specific cloud vendors:
- 10.0.0.0/16 or 2001:db8:1234:1a00::/56 by AWS (see diagram)
- 10.128.0.0./9 Google
- 172.31.0.0/16 Azure
REMEMBER: The CIDR block for a default AWS VPC is always 172.31.0.0/16???
Allocate ranges by geographical regions:
- 10.16 for US1
- 10.32 for US2
- 10.48 for US3
- 10.64 for EU
- 10.80 Australia
Ranges used by specific geographies:
- 192.168.10.0/24
- 192.168.15.0/24 London
- 192.168.20.0/24 New York
- 192.168.25.0/24 Seattle

Allocate ranges for production vs. DR vs. testing:

PROTIP: Consider this convention:

Use Class A VPC CIDR 10.0.0.0/16 for production regions
Use Class B VPC CIDR 172.16.0.0/16 for DR (Disaster Recovery) regions

NetMask Nodes

The Default Mask is different for each class.

VIDEO: This table of nodes for each netmask Amazon allows:

Hosts/ Subnet	Netmask	# IPs	# Nodes	Subnet Size	Subnet Mask	Note
-	/28	14	-	/31	255.255.255.240	Minimum
-	/27	30	-	/30	255.255.255.224	-
-	/26	62	-	/29	255.255.255.192	-
-	/25	126	-	/28	255.255.255.128	-
-	/24	254	-	/27	255.255.255.0	Small
-	/23	510	?	/26	255.255.254.0	-
-	/22	1,022	?	/25	255.255.252.0	-
-	/21	2,046	2,008	/24	255.255.248.0	Small
-	/20	4,094	4,091	/23	255.255.240.0	-
-	/19	8,190	8,152	/22	255.255.224.0	Medium
-	/18	16,382	16,344	/21	255.255.192.0	Large
-	/17	32,766	-	/21	255.255.128.0	-
-	/16	65,534	65,456	/20	255.255.0.0	Maximum: Extra Large

Hosts/subnet?

REMEMBER: The larger the CIDR netmask, the less hosts in the subnet.

REMEMBER: 16 is the largest CIDR range allowed by AWS.

REMEMBER: If all you’ll need are 14 nodes, specify /28. That actually allocates 16 addresses, but the first and last address are reserved.

subnet+1 are for default GW via DHCP Option Set
the last subnet is for broadcasts.

PROTIP: 24 is a common one:

private 10.1.0.0/24 (< 129)
public 10.129.0.0/24 (> 128)

PROTIP: To avoid naming conflicts, use a standard naming convention:
Of the 255 possibilities in /24:
allocate the top half to private addresses:
allocate the bottom half to public addresses:

https://www.cisco.com/c/en/us/support/docs/ip/routing-information-protocol-rip/13790-8.html
https://www.dnsstuff.com/subnet-ip-subnetting-guide

IP Subnets

PROTIP: In the subnet for each Availability Zone, replace the “??” in the IP address with a pre-defined set of numbers associated with each separate environment and architectural tier. For example, if the VPC is assigned this CIDR:

10.1.??.0/20

The ?? is replaced with one of the numbers within an (Availability) Zone column:

Env	Tier	IPv6	Zone a	Zone b	Zone c	Future	Routes
Prd	ELB-?	00	1	11	21	31	Public
Prd	WEB-?	01	2	12	22	32	Private
Prd	APP-?	02	3	13	23	33	Private
Prd	Cache-?	03	4	14	24	34	Private
Prd	DB-?	04	5	15	25	35	Private
Prd	Res-?	05	6	16	26	36	Private
Prd	Res-?	06	7	17	27	37	Private
Dev	ELB-?	41	51	61	71	81	Public
Dev	WEB-?	42	52	62	72	82	Private
Dev	APP-?	43	53	63	73	83	Private
Dev	Cache-?	44	54	64	74	84	Private
Dev	DB-?	45	55	65	75	85	Private
Dev	Res-?	46	56	56	76	86	Private
Dev	Res-?	47	57	57	77	87	Private

Expanded, each ELB (Elastic Load Balancer) is naturally on a Public subnet:

10.16.1.0/20 in Production Availability Zone a
10.16.8.0/20 in Production Availability Zone b
10.16.15.0/20 in Production Availability Zone c

10.16.22.0/20 in Dev Availability Zone a
10.16.29.0/20 in Dev Availability Zone b
10.16.36.0/20 in Dev Availability Zone c

The “IPv6” column is entered in the ___ below in the VPC GUI “IPv6 CIDR block” field such as:

2600:1f18:10e8:73___;;/64

VPC Subnets

In the AWS Console GUI VPC Subnets, select each subnet defined above.
Click “Actions” menu to select “Edit subnet settings”.
Check “Enable auto-assign IPv6 addresses”.

Scroll to click the orange Save.

PROTIP: If the VPC is defined using Terraform instead of the GUI, the above can be coded one time for subsequent repeated use.

Bucket of Candies Analogy

If you must know why, here is my analogy (best for kinesthetic learners): When we say someone makes a “7 figure salary”, we figure out what that means with a table like this:

Figure:	7	6	5	4	3	2	1
# Values:	1,000,000	100,000	10,000	1,000	100	10	1

Now imagine a bucket for each figure level, a different size bucket containing candies of various colors and patterns, unique one for each possible value. People earning 7 figures can choose from the bucket holding a million possible values.

If we add up the values (colors) possible in the right-most 3 buckets, we would have 100 + 10 + 1 = 111 possibilities.

Counting in Base 2

Instead of the way bankers do arithmetic where ten $1 bills is equivalent to a 10 dollar bill (called “base 10” or decimal calculation), computers count using “base 2” or binary arithmetic using 0’s and 1’s. So each of their “buckets” have a different number of possibility values:

Position:	8	7	6	5	4	3	2	1
# Values:	254	128	64	32	16	8	4	2
Cumulative possible addresses:	510	254	126	62	30	14	6	2

If we add up the possible addresses just from the right-most 3 buckets (from right to left), we would have 2 + 4 + 8 = 14 possibilities.

Look back above at the table of nodes, we see 14 possibilities can be obtained from a specification of 28 bits.

This is all one needs to know to use AWS VPC.

But if you would like to know how we get 3 buckets from the 28 bit specification, read on.

IP address octets

IPV4 subnet addresses such as “127.10.138.128” are 4 sets of there are 32 “buckets” separated by dots into four 8 bit “octets”:

The 127 in the figure above is obtained by adding the base 10 value of each bit “bucket”. Looking at a single octet of 8 bits:

“Bucket” position:	8	7	6	5	4	3	2	1
Base 10 value of each bucket:	128	64	32	16	8	4	2	1
Cumulative base 10 (left to right)	255	127	63	31	15	7	3	1
Base 2 for 127 in base 10	1	1	0	1	1	0	0	1
Cumulative base 10 (left to right)	217	89	25	25	9	1	1	1

To translate a base 2 number of all 1’s (“1111111”) to a base 10 value of 255 we accumulate base 10 values for each “bucket”, left to right.

To translate the Base 2 set of 1’s and 0’s to a base 10 number of 217, we accumulate the equivalent base 10 number at each position where there is a 1.

Now let’s look at the relationship between /28 and the “255.255.255.240” subnet mask associated with the /28 in the table of nodes above.

The “240” base 10 number in the right-most quartet is equivalent to “11110000” in base 2.

“Bucket” position:	8	7	6	5	4	3	2	1
Base 10 value of bucket:	128	64	32	16	8	4	2	1
Base 2 for 240 in base 10	1	1	1	1	0	0	0	0
Cumulative base 10 (left to right)	240	122	48	16	0	0	0	0

Putting the three 255 and 240 together we get a continuous set of 1’s followed by four 0’s:

11111111.11111111.1111111.11110000

The 1’s “buckets” on the left side are used to address subnets managed by Amazon.
The 0’s buckets on the right side are used to address your individual nodes.

REMEMBER: Although there are four 0’s buckets, only 3 are used to specify node addresses because one digit (two values) are reserved for network broadcast use (addresses containing all 0’s and all 1’s).

More on CIDR (Classless Inter-Domain Routing), aka “supernetting”:

VIDEO: IP Subnetting from CIDR Notations (getting network and broadcast addresses).
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html
VLSM (Variable Length Subnet Mask)
https://cloudacademy.com/amazon-web-services/amazon-vpc-networking-course/build-and-configure-a-nat-instance.html

Do you really know the above? Take Pearson’s IP Subnetting exam on OReilly.com [subscription required]

Static Elastic IPs

NOTE: The use of static IP addresses in configurations in EC2 can be an annoyance to some and a comfort to others.

Historically, working on a physical servers involves use of specific static IPs associated with a particular purpose. External monitoring servers were manually configured with the IP assigned to each machine. This also creates time pressure (panic) to get specific servers up and running, which led to pressure for servers to be patched rather than risking losing configurations during rebuilds.

SECURITY PROTIP: Static IPs needed to be protected as secrets because of their long-lived nature in traditional server environments.

A “paradigm shift” in thinking is necessary when moving to the “cloud” because there IP address assignments can be transitory/ephemeral and thus more difficult to hack. When a server dies in a “12 factor app” environment, additional servers can be brought up automatically by auto-scaling from a common public pool.

AWS provides static IPs in their Elastic IP service, albeit for a charge of $1 per month for each reserved static IP not assigned to a running EC2 instance.

PROTIP: Long-lived elastic static IPs are useful to avoid shared IPs that may have been black-listed due to abuse by others.

Resources on this topic: * https://launchbylunch.com/posts/2014/Jan/29/aws-tips/ * https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/

DNS Route 53

DNS servers maintain a database to translate host names to IP addresses.

Amazon’s public DNS service is called Route 53 because the default part for DNS servers is TCP 53 / UDP 53.

Its competitors include Dyn.com, GoDaddy, etc.

DIAGRAM: Advanced Demo - Hybrid DNS between AWS and Simulated On-Premises

ELB vs. ALB

AWS NAT

Only one NACL can be associated with a subnet, to deny specific IP addresses. Separate rules are for inbound and outbound.

PROTIP: NACL rules are numbered to sepcify sequence. To allow for insertion, leave gaps in the numbers. For example, create the first two with 100, 200, etc. so you can later add 150 between 100 and 200.

PROTIP: Remember that EC2 instances by default have Networking > Change Source/Dest. Check ON. But NAT instances require OFF or they wont’ show up on VPC Route Tables.

Launch an EC2 instance of a Community AMI built for NATting. Search for “NAT”.

NAT provides IP address assignment and DNS Proxy name resolution services to internal network clients.

A NAT server allows outbound traffic to the external internet. By default, a NAT server allows inbound traffic only through connections already established by an internet host (typically port 80/443).

To access traffic from a special port from an external host:
- If the public interface of the NAT server is configured with a single IP address, add a Special Port (for Windows, in the Routing and Remote Assess MMC console).
- If the public interface of the NAT server is configured with multiple IP addresses, make address reservations to map specific external addresses to specific internal addresses.
Selection of 006 DNS Servers option at the scope level overrides the selection at the server level.

For security, define some servers to only make outbound calls to the internet (through the NAT server).
PROTIP: A NAT instance provide whatever capacity a single AMI provides, so it should be configured with CloudWatch alarms and traffic metrics.
Prepare before need a script to manually to manage Subnet failover to another NAT in this Amazon article.

VPN

PROTIP: When an enterprise development team first begins working with an external vendor or customer, it would likely begin by using a private VPN while the project operates in “stealth mode”.

Configure Site to Site VPN to securely transfer data among Amazon VPCs in different regions or between Amazon VPC to your on-premise data center.

NOTE: Dual ports are usually configured on VPN hardware.

https://app.pluralsight.com/player?course=aws-certified-sysops-admin-associate&author=elias-khnaser&name=aws-certified-sysops-admin-associate-m5&clip=3&mode=live Customer Gateway.

It’s attached to a VPN.

VPC Peering

VPC peering enables organizations to link two distinct VPCs together, allowing assets in one network to talk to assets in another.

Peering connections were introduced to route traffic between two VPCs (AZs) in the same region using private (rather than public) IP addresses. This makes it like they are communicating as if they are within the same network.

Nodes in the same region can reference each other logically using the same peer SG (Security Group), which improves performance.

VPC peering is not transitive —- it must be specifically allowed for each VPC peered together.

Nevertheless, IP addresses must not overlap among VPCs.

Peering is neither a gateway nor a VPN connection, so doesn’t invoke separate physical hardware and the “single point of failure” nor bandwidth bottlenecks.

One useful use case is for more secure interconnection among Active Directory, Exchange, and other common business services:

more secure communication among business units/teams
stronger integration of CRM, HRMS, file sharing
tighter integrated access of core suppliers systems
provide monitoring and management of customer AWS resources

Setup Peering in VPC
Accept the Peering request on the target VPC.

IP DHCP

VIDEO: The Dynamic Host Configuration Protocol is used for auto-configuration of network resources.

When a VPC is created, AWS automatically create a set of DHCP options and associates them with the VPC. The options include configuration parameters, including the domain name, domain name server, and the netbios-node-type. Configure your own DHCP options set for your VPC.

IP address, Subnet Mask, Default Gateway
DNS servers & AmazonProvidedDNS or Custom DNS domain
NTP services, NetBios Name servers & Node type

DHCP Option Sets for each AZ are immutable.

Associating a new option set is immediate, but changes require a DHCP Renew (which takes time).

A DHCP server is setup to listen for L2 broadcasts to get info from the DHCP server.

VPC Router (Subnet+1)
R53 Resolver (Subnet+2)

Transit Gateway

A transit gateway can simplify multi-VPC architectures significantly.

ACLs

Access Control Lists

Create Internet outbound allow and deny network ACL in your VPC. First network ACL: Allow all the HTTP and HTTPS outbound traffic on public internet facing subnet. Second network ACL: Deny all the HTTP/HTTPS traffic. Allow all the traffic to Squid proxy server or any virtual appliance. http://techlib.barracuda.com/display/BNGv54/How+to+Deploy+the+Barracuda+NG+Firewall+in+an+Amazon+Virtual+Private+Cloud

NACLs

Negative ACLS.

Block all the inbound and outbound ports. Only allow application request ports.

These are stateless traffic filters that apply to all traffic inbound or outbound from a Subnet within VPC. AWS recommended Outbound rules

REMEMBER:

Security Group	NACLs
Applicable to instances	Operate on VPC subnets
Only supports Allow rules (layered on a default Deny)	Support both allow and deny rules
Are stateful	Are NOT stateful
Are considered in their entirety before traffic is allowed	Are processed in numerical order
Must be associated with an instance to apply	Apply automatically to all instances in a subnet

REMEMBER: Up to 5 different Security Groups can be applied to a single AWS resource.

References:

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html
https://www.cidr-report.org/ reports nearly a million routes in 2023

Direct Connect (DX)

https://aws.amazon.com/directconnect/sla/
VIDEO: BGP summary by F5
VIDEO: BGP Deep Dive by Kevin Wallace

On-premises locations reach AWS using the BGP (Border Gateway Protocol) though a DX (Direct Connect) router onsite.

In each DX Location, there is a port on a DX Router which is charged per hour of use. The price is the same globally except for a few regions. Outgoing data transfer charges apply, too, but cheaper than going through the public internet.

There are 1GB, 10GB, and 100GB wide pipes.

For redundancy and higher capacity, many deploy two or more DX connections.

If the DX Location is in a different region, a DX Gateway is needed.

BGP peering is configured between opposite ends of AWS Virtual Interfaces.

https://docs.aws.amazon.com/directconnect/latest/UserGuide/WorkingWithConnections.html

Common patterns involve using a combination of Virtual Interfaces (VIF):

Private Virtual Interfaces (PrivateVIF) and Direct Connect Gateway (DXGW) or
Transit attaches to Direct Connect Gateway
Public VIF attaches to an Account Construct with AS_Path BGP PA support

BGP peering is configured between opposite ends of AWS Virtual Interfaces.

AS & IGP

Each AS (Autonomous System) – such as AT&T, Verizon, CenturyLink, etc. – is identified by a special 16 bit or 32 bit number such as 65500.

Sub-AS can be formed inside each AS.

Running within an AS (for a company) are Interior Gateway routing Protocols (IGP) – RIP, OSPF, EIGRP, IS-IS – concern themselves with link-states or interface costs.

BGP EGP

https://aws.amazon.com/blogs/networking-and-content-delivery/creating-active-passive-bgp-connections-over-aws-direct-connect/

There is an iBGP (internal Border Gateway Protocol) that is a full mesh, but doesn’t scale.

BGP is sometimes called a Path Vector Routing protocol.

BGP routers form neighborships by explicit configuration that point to each other. A TCP session over port 179 is established with neighbors to exchange network status. Handshake: Open Sent, Open Confirm, Established. BGP has default Keepalive of 60 seconds with 160 second hold time.

VIDEO: Border Gateway Protocol (BGP) typically runs as an Exterior Gateway Routing protocol (EGP) connecting inter-domain ISP (Internet Service Providers).

The “believability” (the lower the better):

Routing Source	Administrative Distance
Connected	0
Static	1
eBGP	20
EIGRP (internal)	90
OSPF	110
IS-IS	115
RIP	120
EIGRP (external)	190
iBGP	200

An EGP is concerned with advertising address information between Autonomous Systems (AS) – responsible for the address space within. EGPs focus on the paths to destination.

When BGP speakers, or peers, advertise to each other Address Prefix and Length, called NLRI (Network Layer Reachability Information).

They also advertise a series of constructs called Path Attributes (PA) for Path Selection, sent in a BGP Update message.

Active-passive Border Gateway Protocol (BGP) connections are created based on RFC 4271.

The best-path algorithm that runs as part of BGP considers all routes it receives and tries to select the best ones. It uses configured policies and received path attributes when stepping through the logic until an appropriate route (or routes) are found.

BGP path attributes can materially affect routing behavior, in both directions, over a DX connection.

VIDEO: REMEMBER: A commonly-used mnemonic about the top 8 attributes</strong> used to prioritize BGP best-path algorithms over a DX connection:

We Love Oranges As Oranges Mean Pure Refreshment

which translates to path attributes:

Weight, Local Pref, Originate, AS_Path length, Origin type, MED, Paths, RouterID

From VIDEO by Kevin Wallace:

We: Weight is a locally significant parameter that a Cisco-specific router can set when receiving updates. Commonly used to influence outbound routing decisions (based on bandwidth). A higher weight is preferred.
Love: Local Preference is considered right at the start of the best-path algorithm, and as such, is an optimal tuning parameter. It’s considered throughout a single AS. This is used for both Inbound and Outbound tuning. Higher values are preferred.
Oranges: Originate specifies paths sourced locally are preferred.
As: AS_Path Length (like a hop count) is the number of AS in the AS_PATH attribute – a concatenation of all the AS numbers the advertisement has passed through. It’s used as a loop avoidance mechanism and as an indication of distance on the other. Prepending influences incoming. This is used for both Inbound and Outbound tuning. Shorter AS_Path lengths are preferred.
Oranges: Origin Type indicates how the route was injected into BGP i (network command) is preferred to e (EGP) is preferred to ? (redistributed).
Mean: MED (Multi-Exit Discriminator) uses a metric as well. MED is typically used by an AS which is multi-homed to instruct an external AS it is peered with). That makes for a preferred entry point for a particular network address block. MED can be used for inbound tuning. Lower metric values are preferred.
Pure: Paths - prefer eBGP over iBGP path.
Refreshment: Router ID - a tie breaker. The lowest router ID is preferred.

Resources

Add Intrusion Prevention or Intrusion Detection virtual appliances to secure protocols and to take preventive/corrective action.
Assign
Configure Privileged Identity access management solutions to monitor and audit access by Administrators of your VPC.
Add anti-virus for cleansing specific EC2 instances inside a VPC. Trend micro offers a product for this.
http://harish11g.blogspot.com/2015/06/best-practices-tips-on-amazon-web-services-security-groups-aws-security-managed-services.html

AMS needs to set limits http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html

AWS Networking Certification

AWS Certified Advanced Networking - Specialty exam ANS-C01 https://aws.amazon.com/certification/certified-advanced-networking-specialty/

PDF: Domains and Task Statements:

Network Design 30%
Network Implementation 26%
Network Management and Operation 20%
Network Security, Compliance, and Governance 24%

4.1: Implement and maintain network features to meet security and compliance needs and requirements.
- Threat models
- Securing app flows
- Securing inbound traffic flows into AWS (AWS WAF, AWS Shield, Network Firewall)
- Securing outbound traffic flows from AWS (for example, Network Firewall, proxies, Gateway Load Balancers)
- Securing inter-VPC traffic within an account or across multiple accounts (security groups, network ACLs, VPC endpoint policies)
- Implementing an AWS network architecture to meet security and compliance requirements (untrusted network, perimeter VPC, three-tier architecture)
- Developing a threat model and identifying appropriate mitigation strategies for a given network architecture
- Testing compliance with the initial requirements (failover)
4.3: Implement and maintain confidentiality of data and communications of the network:
- Network encryption options that are available on AWS
- VPN connectivity over Direct Connect
- Encryption methods for data in transit (IPsec)
- Network encryption under the AWS shared responsibility model
- Security methods for DNS communications (DNSSEC)
- network encryption methods to meet application compliance requirements (IPsec, TLS)
- encryption solutions to secure data in transit (for example, CloudFront, Application Load Balancers and Network Load Balancers, VPN over Direct Connect, AWS managed databases, Amazon S3, custom solutions on Amazon EC2, Transit Gateway)
- a certificate management solution by using a certificate authority (ACM, AWS Certificate Manager Private Certificate Authority [ACM PCA])
- secure DNS communications

Professional experience using AWS technology, AWS security best practices, AWS storage options and their underlying consistency models, and AWS networking nuances and how they relate to the integration of AWS services.
Knowledge of advanced networking architectures and interconnectivity options [e.g., IP VPN, multiprotocol label switching (MPLS), virtual private LAN service (VPLS)].
Familiarity with the development of automation scripts and tools. This should include the design, implementation, and optimization of the following: Routing architectures (including static and dynamic); multi-region solutions for a global enterprise; highly available connectivity solutions (e.g., AWS Direct Connect, VPN).
Knowledge of CIDR and sub-netting (IPv4 and IPv6); IPv6 transition challenges; and generic solutions for network security features, including AWS WAF, intrusion detection systems (IDS), intrusion prevention systems (IPS), DDoS protection, and economic denial of service/sustainability (EDoS).

More on Amazon

This is one of a series on Amazon:

More on DevOps

This is one of a series on DevOps:

Wilson Mar