AWS Networking

Setting up VPC (Virtual Private Cloud), IPAM, DNS, Security Groups, WAF, BGP, etc. using CLI, GUI, Terraform, Cloud Formation

Overview

This tutorial covers how to manage Security Groups and other AWS network security features to access servers and other resources within AWS.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Consider the types of architectures: – Subnets vs. VPCs and VPC peering

TODO: Add WAF. Make above diagram into a video.

Terraform & CDK & CF

This article describes use of Terraform and CDK as well as Cloud Formation to create resources within AWS.

Routing Rules

AWS VPC Routing Rules are what makes subnets public or private.

VPCs (Virtual Private Cloud)

https://aws.amazon.com/vpc/faqs/
TUTORIAL

PROTIP: AWS creates a default subnet for each region.

Delete the default VPC. It doesn’t cost anything.

BLAH: At time of writing, AWS auto-assigns public IPv4 address.
“Create VPC”.
Type Security Groups over “Search” at the top of every AWS Console GUI page.
Click “Security groups” among “Features of EC2”, which means you see “Security Groups” on the left menu under EC2.

What makes a subnet public is a route table associated with each subnet created.
View Route Table feature.
There is a Main route table designated as Yes.
Rename the Main “Public-IGW”.

Subnets: Outbound rules: NACL (Network ACL) :
The rule which Allow/Deny Source 0.0.0.0/0 - Rename it “AllowEverything”

VPC Terraform

The provider for VPC is at https://www.terraform.io/docs/providers/aws/r/vpc.html

https://wpengine.linuxacademy.com/amazon-web-services-2/learn-how-to-master-aws-vpc-inside-and-out/ Basic usage with tags:

resource "aws_vpc" "main" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "dedicated"
 
  tags {
    Name = "main"
  }
}

Create VPCs using Management Console

This chapter condenses Amazon’s docs on this topic and adds additional PROTIPs and NOTEs.

A default VPC is a pre-requisite for setting up an EC2 server instance.
At https://console.aws.amazon.com/vpc/
Select “Your VPC”.
Click the “Create VPC” blue button.
PROTIP: For Name tag, consider a naming convention that specifies the decisions associated with each VPC:

dev-public-ipam1

a. “public” or “private” network access scope.

b. “prod”, “non-prod”, “dev”, “qa”, etc. pool

c. “ipam” or “man” (manual management) of IP Addresses

These reflect the decisions selected on these fields:

When dealing with networks, a CIDR is always requested.

Each CIDR defines a contiguous range of IP address.

CIDR specs are what keeps each IP address within a single subnet. Manual allocations can result in misconfigurations. So many teams follow the same plan for allocating CIDRs.

IPAM (IP Address Manager) is an AWS VPC feature that automatically allocate CIDRs to VPCs from pools of CIDRs it has provisioned into public and private scopes – to make it easier to plan, track, and monitor IP addresses for AWS workloads, without causing IP address overlap or conflict.

Before individuals can specify that IP addresses be allocated automatically by selecting:

IPAM-allocated CIDR block
1. The enterprise needs to be willing to pay for IPAM costs charged for each active IP under its management, at $0.1944 per month ($0.00027 an hour x 24 x 30). Charges go to the $AWS_IPAM_ACCT specified because IP allocation can cross multiple accounts and VPCs based on configurable business rules. Thus the need for central administration.
2. Identify a central asset management team with IPAM delegated administrators named within AWS. DOCS: The centralization of CIDR management enables allocation requests to be centrally monitored and audited – alerts about IP address overlap, IP address depletion, etc. can be received by a designated team email. IPAM automatically retains IP address monitoring data for up to three years. The team performs the above on the IPAM dashboard at
  
  https://console.aws.amazon.com/ipam/ which routes to a region-specific site such as:
  https://us-west-2.console.aws.amazon.com/ipam/home?region=us-west-2#Home
  
  IPAM enables Administrators to reuse/reallocate IP addresses across multiple unconnected networks.
3. For Cross-account access, define IAM roles using Terraform iam_assumable_role or iam_assumable_roles submodules in “resource AWS accounts (prod, staging, dev)” and IAM groups and users using iam-group-with-assumable-roles-policy submodule in “IAM AWS Account” to setup access controls between accounts. See https://docs.aws.amazon.com/vpc/latest/ipam/choose-single-user-or-orgs-ipam.html
4. IPAM Delegated Administrators define a profile containing the business rules for allocating CIDRs among the two scopes from pools.
5. To create a public and a private scope for a single VPC network within a particular operating Region, instead of using the Console GUI, use this CLI command:
```
AWS_REGION=us-west-2
AWS_OPERATING_REGIONS=us-west-2
AWS_IPAM_POOL="prd-ipam"
AWS_IPAM_ACCT="projA-ipam-acct"
 
aws ec2 create-ipam --description "$AWS_IPAM_POOL" \
--region "$AWS_REGION" \
--operating-regions RegionName="$AWS_OPERATING_REGIONS" \
--profile "$AWS_IPAM_ACCT"
  
```
Alternately, use the IPAM API from a custom program.

For easy repeatability, use the Terraform Registry
https://registry.terraform.io/modules/terraform-aws-modules/iam/aws/latest
Define CIDRs within each top-level pools under the 2 IPAM scopes (public and private).

An “allocation” can be a CIDR assignment from an IPAM pool to another resource or another IPAM pool.

See https://docs.aws.amazon.com/vpc/latest/ipam/manually-allocate-ipam.html
If you don’t have IPAM setup, choose IPv4 For CIDR manual input

REMEMBER: CIDRs are called Masks. The larger number after the slash, the more IP addresses it specfies. 16 is the largest mask allowed.

Subnet Calculators
- subnet-calculator.com [has pop-up ads]
- https://www.site24x7.com/tools/ipv4-subnetcalculator.html
- https://calculator.net/ip-subnet-calculator.html
REMEMBER: The CIDR block for a default AWS VPC is always 172.31.0.0/16.

REMEMBER: 16 is the largest CIDR range allowed by AWS.

PROTIP: Consider this convention:
- Use Class A VPC CIDR 10.0.0.0/16 for production regions
- Use Class B VPC CIDR 172.16.0.0/16 for DR (Disaster Recovery) regions
Address ranges for private (non-routed) use (per RFC 1918):
- 10.0.0.0 -> 10.255.255.255 within “Class A” addresses 1 -> 126
- 172.16.0.0 -> 172.31.255.255 within “Class B” addresses 127 -> 191
- 192.168.0.0 -> 192.168.255.255 within “Class C” addresses 192 -> 223
PROTIP: Carefully predict how many nodes each subnet might need. Once assigned, AWS VPC subnet blocks can’t be modified. If you find an established VPC is too small, you’ll need to terminate all of the instances of the VPC, delete it, and then create a new, larger VPC, then instantiate again.

For CIDR block, see below.

NetMask Nodes

This table of nodes for each netmask Amazon allows:

# Nodes	Netmask	Subnet Mask
14	/28	255.255.255.240
30	/27	255.255.255.224
62	/26	255.255.255.192
126	/25	255.255.255.128
254	/24	255.255.255.0
510	/23	255.255.254.0
65,534	/16	255.255.255.240

For example, if all you’ll need are 14 nodes, specify /28. Notice that the larger the CIDR netmask, the less hosts in the subnet.

To make naming conflicts more avoidable, use a standard naming convention:

PROTIP: The last two parts of VPC subnet always have “.0.0”, with the bottom half of the 255 possibilities allocated to private and upper half to public addresses:

private 10.1.0.0/24 (< 129)
public 10.129.0.0/24 (> 128)

IP Subnets

PROTIP: In the subnet for each Availability Zone, replace the “??” in the IP address with a pre-defined set of numbers associated with each separate environment and architectural tier. For example, if the VPC is assigned this CIDR:

10.1.??.0/20

The ?? is replaced with one of the numbers within an (Availability) Zone column:

Env	Tier	IPv6	Zone a	Zone b	Zone c	Routes
Prd	ELB-?	00	1	8	15	Public
Prd	WEB-?	01	2	9	16	Private
Prd	APP-?	02	3	10	17	Private
Prd	Cache-?	03	4	11	18	Private
Prd	DB-?	04	5	12	19	Private
Prd	Res-?	05	6	13	20	Private
Prd	Res-?	06	7	14	21	Private
Dev	ELB-?	07	22	29	36	Public
Dev	WEB-?	08	23	30	37	Private
Dev	APP-?	09	24	31	38	Private
Dev	Cache-?	0A	25	32	39	Private
Dev	DB-?	0B	26	33	40	Private
Dev	Res-?	0C	27	34	41	Private
Dev	Res-?	0D	28	35	42	Private

Expanded, each ELB (Elastic Load Balancer) is naturally on a Public subnet:

10.16.1.0/20 in Production Availability Zone a
10.16.8.0/20 in Production Availability Zone b
10.16.15.0/20 in Production Availability Zone c

10.16.22.0/20 in Dev Availability Zone a
10.16.29.0/20 in Dev Availability Zone b
10.16.36.0/20 in Dev Availability Zone c

The “IPv6” column is entered in the ___ below in the VPC GUI “IPv6 CIDR block” field such as:

2600:1f18:10e8:73___;;/64

VPC Subnets

In the AWS Console GUI VPC Subnets, select each subnet defined above.
Click “Actions” menu to select “Edit subnet settings”.
Check “Enable auto-assign IPv6 addresses”.
Scroll to click the orange Save.

PROTIP: If the VPC is defined using Terraform instead of the GUI, the above can be coded one time for subsequent repeated use.

Bucket of Candies Analogy

If you must know why, here is my analogy (best for kinesthetic learners): When we say a sports star makes a “7 figure salary”, we figure out what that means with a table like this:

Figure:	7	6	5	4	3	2	1
# Values:	1,000,000	100,000	10,000	1,000	100	10	1

Now imagine a bucket for each figure level, a different size bucket containing candies of various colors and patterns, unique one for each possible value. People earning 7 figures can choose from the bucket holding a million possible values.

If we add up the values (colors) possible in the right-most 3 buckets, we would have 100 + 10 + 1 = 111 possibilities.

#### Counting in Base 2 #

Instead of the way bankers do arithmetic where ten $1 bills is equivalent to a 10 dollar bill (called “base 10” or decimal calculation), computers count using “base 2” or binary arithmetic using 0’s and 1’s. So each of their “buckets” have a different number of possibility values:

Position:	8	7	6	5	4	3	2	1
# Values:	254	128	64	32	16	8	4	2
Cumulative possible addresses:	510	254	126	62	30	14	6	2

If we add up the possible addresses just from the right-most 3 buckets (from right to left), we would have 2 + 4 + 8 = 14 possibilities.

Look back above at the table of nodes, we see 14 possibilities can be obtained from a specification of 28 bits.

This is all one needs to know to use AWS VPC.

But if you would like to know how we get 3 buckets from the 28 bit specification, read on.

#### IP address octets #

IPV4 subnet addresses such as “127.10.138.128” are 4 sets of there are 32 “buckets” separated by dots into four 8 bit “octets”:

The 127 in the figure above is obtained by adding the base 10 value of each bit “bucket”. Looking at a single octet of 8 bits:

“Bucket” position:	8	7	6	5	4	3	2	1
Base 10 value of each bucket:	128	64	32	16	8	4	2	1
Cumulative base 10 (left to right)	255	127	63	31	15	7	3	1
Base 2 for 127 in base 10	1	1	0	1	1	0	0	1
Cumulative base 10 (left to right)	217	89	25	25	9	1	1	1

To translate a base 2 number of all 1’s (“1111111”) to a base 10 value of 255 we accumulate base 10 values for each “bucket”, left to right.

To translate the Base 2 set of 1’s and 0’s to a base 10 number of 217, we accumulate the equivalent base 10 number at each position where there is a 1.

Now let’s look at the relationship between /28 and the “255.255.255.240” subnet mask associated with the /28 in the table of nodes above.

The “240” base 10 number in the right-most quartet is equivalent to “11110000” in base 2.

“Bucket” position:	8	7	6	5	4	3	2	1
Base 10 value of bucket:	128	64	32	16	8	4	2	1
Base 2 for 240 in base 10	1	1	1	1	0	0	0	0
Cumulative base 10 (left to right)	240	122	48	16	0	0	0	0

Putting the three 255 and 240 together we get a continuous set of 1’s followed by four 0’s:

  11111111.11111111.1111111.11110000

The 1’s “buckets” on the left side are used to address subnets managed by Amazon.
The 0’s buckets on the right side are used to address your individual nodes.

REMEMBER: Although there are four 0’s buckets, only 3 are used to specify node addresses because one digit (two values) are reserved for network broadcast use (addresses containing all 0’s and all 1’s).

More on CIDR (Classless Inter-Domain Routing), aka “supernetting”:

https://www.youtube.com/watch?v=POPoAjWFkGg IP Subnetting from CIDR Notations (getting network and broadcast addresses).
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html
VLSM (Variable Length Subnet Mask)
https://cloudacademy.com/amazon-web-services/amazon-vpc-networking-course/build-and-configure-a-nat-instance.html

Do you really know the above? Take Pearson’s IP Subnetting exam on OReilly.com [subscription required]

### Automatically create VPC using CloudFormation #

VPCs are really software-defined networks (SDN).

     "Resources" : {
        "VPC" : {
         "Type" : "AWS::EC2::VPC",
         "Properties" : {
           "CidrBlock" : "10.0.0.0/16"
         }
       },

       "InternetGateway" : {
         "Type" : "AWS::EC2::InternetGateway",
         "Properties" : {
         }
       },

       "AttachGateway" : {
          "Type" : "AWS::EC2::VPCGatewayAttachment",
          "Properties" : {
            "VpcId" : { "Ref" : "VPC" },
            "InternetGatewayId" : { "Ref" : "InternetGateway" }
          }
       },

In the CF JSON to define a VPC, CF automatically populates the “VpcId” : { “Ref” : “VPC” },

  REMEMBER: There is one VPC per Availability Zone.

A single Gateway serves all VPCs because that is the address the public DNS resolves corporate host names to.

## Static Elastic IPs #

NOTE: The use of static IP addresses in configurations in EC2 can be an annoyance to some and a comfort to others.

Historically, working on a physical servers involves use of specific static IPs associated with a particular purpose. External monitoring servers were manually configured with the IP assigned to each machine. This also creates time pressure (panic) to get specific servers up and running, which led to pressure for servers to be patched rather than risking losing configurations during rebuilds.

SECURITY PROTIP: Static IPs needed to be protected as secrets because of their long-lived nature in traditional server environments.

A “paradigm shift” in thinking is necessary when moving to the “cloud” because there IP address assignments can be transitory/ephemeral and thus more difficult to hack. When a server dies in a “12 factor app” environment, additional servers can be brought up automatically by auto-scaling from a common public pool.

AWS provides static IPs in their Elastic IP service, albeit for a charge of $1 per month for each reserved static IP not assigned to a running EC2 instance.

PROTIP: Long-lived elastic static IPs are useful to avoid shared IPs that may have been black-listed due to abuse by others.

Resources on this topic: * https://launchbylunch.com/posts/2014/Jan/29/aws-tips/ * https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/

DNS Route 53

DNS servers maintain a database to translate host names to IP addresses.

Amazon’s public DNS service is called Route 53 because the default part for DNS servers is TCP 53 / UDP 53.

Its competitors include Dyn.com, GoDaddy, etc.

DIAGRAM: Advanced Demo - Hybrid DNS between AWS and Simulated On-Premises

ELB vs. ALB

AWS NAT

Only one NACL can be associated with a subnet, to deny specific IP addresses. Separate rules are for inbound and outbound.

PROTIP: NACL rules are numbered to sepcify sequence. To allow for insertion, leave gaps in the numbers. For example, create the first two with 100, 200, etc. so you can later add 150 between 100 and 200.

PROTIP: Remember that EC2 instances by default have Networking > Change Source/Dest. Check ON. But NAT instances require OFF or they wont’ show up on VPC Route Tables.

Launch an EC2 instance of a Community AMI built for NATting. Search for “NAT”.

NAT provides IP address assignment and DNS Proxy name resolution services to internal network clients.

A NAT server allows outbound traffic to the external internet. By default, a NAT server allows inbound traffic only through connections already established by an internet host (typically port 80/443).

To access traffic from a special port from an external host:
- If the public interface of the NAT server is configured with a single IP address, add a Special Port (for Windows, in the Routing and Remote Assess MMC console).
- If the public interface of the NAT server is configured with multiple IP addresses, make address reservations to map specific external addresses to specific internal addresses.
Selection of 006 DNS Servers option at the scope level overrides the selection at the server level.

For security, define some servers to only make outbound calls to the internet (through the NAT server).
PROTIP: A NAT instance provide whatever capacity a single AMI provides, so it should be configured with CloudWatch alarms and traffic metrics.
Prepare before need a script to manually to manage Subnet failover to another NAT in this Amazon article.

NAT Gateway

A NAT Gateway is used for private subnets to reach the public internet.

An AWS NAT Gateway SaaS supports bursts of up to 10Gbps. NAT Gateways are managed by AWS, so they don’t have traffic metrics nor CloudWatch alarms, plus there is a per-hour charge for AWS to operate the NAT Gateway.

A NAT instance can be configured for port forwarding, bastion hosts.

Bastion host

NOTE: Bastion Hosts

PROTIP: Instead of the expense of standing up Bastion Hosts, consider HashiCorp Boundary.

VPN

PROTIP: When an enterprise development team first begins working with an external vendor or customer, it would likely begin by using a private VPN while the project operates in “stealth mode”.

Configure Site to Site VPN to securely transfer data among Amazon VPCs in different regions or between Amazon VPC to your on-premise data center.

NOTE: Dual ports are usually configured on VPN hardware.

https://app.pluralsight.com/player?course=aws-certified-sysops-admin-associate&author=elias-khnaser&name=aws-certified-sysops-admin-associate-m5&clip=3&mode=live Customer Gateway.

It’s attached to a VPN.

VPC Peering

VPC peering enables organizations to link two distinct VPCs together, allowing assets in one network to talk to assets in another.

Peering connections were introduced to route traffic between two VPCs (AZs) in the same region using private (rather than public) IP addresses. This makes it like they are communicating as if they are within the same network.

Nodes in the same region can reference each other logically using the same peer SG (Security Group), which improves performance.

VPC peering is not transitive —- it must be specifically allowed for each VPC peered together.

Nevertheless, IP addresses must not overlap among VPCs.

Peering is neither a gateway nor a VPN connection, so doesn’t invoke separate physical hardware and the “single point of failure” nor bandwidth bottlenecks.

One useful use case is for more secure interconnection among Active Directory, Exchange, and other common business services:

more secure communication among business units/teams
stronger integration of CRM, HRMS, file sharing
tighter integrated access of core suppliers systems
provide monitoring and management of customer AWS resources

Setup Peering in VPC
Accept the Peering request on the target VPC.

IP DHCP

VIDEO: The Dynamic Host Configuration Protocol is used for auto-configuration of network resources.

When a VPC is created, AWS automatically create a set of DHCP options and associates them with the VPC. The options include configuration parameters, including the domain name, domain name server, and the netbios-node-type. Configure your own DHCP options set for your VPC.

IP address, Subnet Mask, Default Gateway
DNS servers & AmazonProvidedDNS or Custom DNS domain
NTP services, NetBios Name servers & Node type

DHCP Option Sets for each AZ are immutable.

Associating a new option set is immediate, but changes require a DHCP Renew (which takes time).

A DHCP server is setup to listen for L2 broadcasts to get info from the DHCP server.

VPC Router (Subnet+1)
R53 Resolver (Subnet+2)

Transit Gateway

A transit gateway can simplify multi-VPC architectures significantly.

ACLs

Access Control Lists

Create Internet outbound allow and deny network ACL in your VPC. First network ACL: Allow all the HTTP and HTTPS outbound traffic on public internet facing subnet. Second network ACL: Deny all the HTTP/HTTPS traffic. Allow all the traffic to Squid proxy server or any virtual appliance. http://techlib.barracuda.com/display/BNGv54/How+to+Deploy+the+Barracuda+NG+Firewall+in+an+Amazon+Virtual+Private+Cloud

NACLs

Negative ACLS.

Block all the inbound and outbound ports. Only allow application request ports.

These are stateless traffic filters that apply to all traffic inbound or outbound from a Subnet within VPC. AWS recommended Outbound rules

REMEMBER:

Security Group	NACLs
Applicable to instances	Operate on VPC subnets
Only supports Allow rules (layered on a default Deny)	Support both allow and deny rules
Are stateful	Are NOT stateful
Are considered in their entirety before traffic is allowed	Are processed in numerical order
Must be associated with an instance to apply	Apply automatically to all instances in a subnet

REMEMBER: Up to 5 different Security Groups can be applied to a single AWS resource.

References:

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html

Direct Connect (DX)

To Direct Connect to a customer’s Router. in each DX Location, there is a port on a DX Router which is charged per hour of use. There are 1GB, 10GB, and 100GB wide pipes. The price is the same globally except for a few regions.

Outgoing data transfer charges apply, too, but cheaper than going through the public internet.

If the DX Location is in a different region, a DX Gateway is needed.

Resources

Add Intrusion Prevention or Intrusion Detection virtual appliances to secure protocols and to take preventive/corrective action.
Assign
Configure Privileged Identity access management solutions to monitor and audit access by Administrators of your VPC.
Add anti-virus for cleansing specific EC2 instances inside a VPC. Trend micro offers a product for this.
http://harish11g.blogspot.com/2015/06/best-practices-tips-on-amazon-web-services-security-groups-aws-security-managed-services.html

AMS needs to set limits http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html

AWS Networking Certification

AWS Certified Advanced Networking - Specialty exam ANS-C01 https://aws.amazon.com/certification/certified-advanced-networking-specialty/

PDF: Domains and Task Statements:

Network Design 30%
Network Implementation 26%
Network Management and Operation 20%
Network Security, Compliance, and Governance 24%

4.1: Implement and maintain network features to meet security and compliance needs and requirements.
- Threat models
- Securing app flows
- Securing inbound traffic flows into AWS (AWS WAF, AWS Shield, Network Firewall)
- Securing outbound traffic flows from AWS (for example, Network Firewall, proxies, Gateway Load Balancers)
- Securing inter-VPC traffic within an account or across multiple accounts (security groups, network ACLs, VPC endpoint policies)
- Implementing an AWS network architecture to meet security and compliance requirements (untrusted network, perimeter VPC, three-tier architecture)
- Developing a threat model and identifying appropriate mitigation strategies for a given network architecture
- Testing compliance with the initial requirements (failover)
4.3: Implement and maintain confidentiality of data and communications of the network:
- Network encryption options that are available on AWS
- VPN connectivity over Direct Connect
- Encryption methods for data in transit (IPsec)
- Network encryption under the AWS shared responsibility model
- Security methods for DNS communications (DNSSEC)
- network encryption methods to meet application compliance requirements (IPsec, TLS)
- encryption solutions to secure data in transit (for example, CloudFront, Application Load Balancers and Network Load Balancers, VPN over Direct Connect, AWS managed databases, Amazon S3, custom solutions on Amazon EC2, Transit Gateway)
- a certificate management solution by using a certificate authority (ACM, AWS Certificate Manager Private Certificate Authority [ACM PCA])
- secure DNS communications

Professional experience using AWS technology, AWS security best practices, AWS storage options and their underlying consistency models, and AWS networking nuances and how they relate to the integration of AWS services.
Knowledge of advanced networking architectures and interconnectivity options [e.g., IP VPN, multiprotocol label switching (MPLS), virtual private LAN service (VPLS)].
Familiarity with the development of automation scripts and tools. This should include the design, implementation, and optimization of the following: Routing architectures (including static and dynamic); multi-region solutions for a global enterprise; highly available connectivity solutions (e.g., AWS Direct Connect, VPN).
Knowledge of CIDR and sub-netting (IPv4 and IPv6); IPv6 transition challenges; and generic solutions for network security features, including AWS WAF, intrusion detection systems (IDS), intrusion prevention systems (IPS), DDoS protection, and economic denial of service/sustainability (EDoS).

More on Amazon

This is one of a series on Amazon:

More on DevOps

This is one of a series on DevOps:

Wilson Mar