Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

How to reach servers


Overview

This tutorial covers how to access servers and other resources within AWS.

TODO: Make this diagram into a video:

My AWS Basics tutorial describes setting up an account for the two main types of people interacting with AWS:

  1. System Administrators who define the enviornment

  2. end-users of the whole setup.

There are several the creation of Virtual Private Clouds within AWS several different ways:

We begin with the manual approach to cover PROTIPs and NOTEs in this our tutorial.

  • VPC Naming conventions
  • CIDR blocks.

Create VPCs using Management Console

This chapter condenses Amazon’s docs on this topic and adds additional PROTIPs and NOTEs.

  1. A pre-requisite for this is my AWS Basics tutorial

    A default VPC is a pre-requisite for setting up an EC2 server instance.

  2. At https://console.aws.amazon.com/vpc/

  3. Select “Your VPC”.

  4. Click the “Create VPC” blue button.

  5. For Name tag, consider a naming convention to include:

    • “dev”, “qa”, “prod” since many use isolated VPCs for different enviornments.

    • “public” or “private” network access.

  6. For CIDR block, see below.

    An example CIDR block looks like this:

    
    10.0.1.0/18
    

    PROTIP: To avoid naming conflicts, some organizations use a convention replacing the “1” in the address with other numbers for each separate environment and tier as well as duplicate zones:

    Env Tier zone A zone B Routes
    Prd ELB 1 11 Public
    Prd WEB 2 12 Private
    Prd APP 3 13 Private
    Prd Cache 4 14 Private
    Prd DB 5 15 Private
    Dev ELB 21 31 Public
    Dev WEB 22 32 Private
    Dev APP 23 33 Private
    Dev Cache 24 34 Private
    Dev DB 25 35 Private

    PROTIP: Use the table above to pre-define your own numbering scheme, which can also be used as shortcuts in other names.

    PROTIP: Some organizations allocate the bottom half of the 255 possibilities to private and upper half to public addresses:

    • private 10.1.0.0/24   (< 129)
    • public   10.129.0.0/24 (> 128)

    Address ranges for private (non-routed) use (per RFC 1918):

    • 10.0.0.0 -> 10.255.255.255 within “Class A” addresses 1 -> 126
    • 172.16.0.0 -> 172.31.255.255 within “Class B” addresses 127 -> 191
    • 192.168.0.0 -> 192.168.255.255 within “Class C” addresses 192 -> 223

    The CIDR block for a default VPC is always 172.31.0.0/16.

    PROTIP: Use addresses from different IP classes. For example, for the production site, use VPC CIDR 10.0.0.0/16 and for DR regions VPC CIDR 172.16.0.0/16.

    PROTIP: Carefully predict how many nodes each subnet might need. Once assigned, AWS VPC subnet blocks can’t be modified. If you find an established VPC is too small, you’ll need to terminate all of the instances of the VPC, delete it, and then create a new, larger VPC, then instantiate again.

    Refer to this table of nodes for each netmask Amazon allows:

    # Nodes Netmask Subnet Mask
    14 /28 255.255.255.240
    30 /27 255.255.255.224
    62 /26 255.255.255.192
    126 /25 255.255.255.128
    254 /24 255.255.255.0
    510 /23 255.255.254.0
    65,534 /16 255.255.255.240

    For example, if all you’ll need are 14 nodes, specify /28. Notice that the larger the CIDR netmask, the less hosts in the subnet.

    Bucket of Candies Analogy

    If you must know why, here is my analogy (best for kinesthetic learners): When we say a sports star makes a “7 figure salary”, we figure out what that means with a table like this:

    Figure: 7 6 5 4 3 2 1
    # Values: 1,000,000 100,000 10,000 1,000 100 10 1

    Now imagine a bucket for each figure level, a different size bucket containing candies of various colors and patterns, unique one for each possible value. People earning 7 figures can choose from the bucket holding a million possible values.

    If we add up the values (colors) possible in the right-most 3 buckets, we would have 100 + 10 + 1 = 111 possibilities.

    Counting in Base 2

    Instead of the way bankers do arithmetic where ten $1 bills is equivalent to a 10 dollar bill (called “base 10” or decimal calculation), computers count using “base 2” or binary arithmetic using 0’s and 1’s. So each of their “buckets” have a different number of possibility values:

    Position: 8 7 6 5 4 3 2 1
    # Values: 254 128 64 32 16 8 4 2
    Cumulative possible addresses: 510 254 126 62 30 14 6 2

    If we add up the possible addresses just from the right-most 3 buckets (from right to left), we would have 2 + 4 + 8 = 14 possibilities.

    Look back above at the table of nodes, we see 14 possibilities can be obtained from a specification of 28 bits.

    This is all one needs to know to use AWS VPC.

    But if you would like to know how we get 3 buckets from the 28 bit specification, read on.

    IP address octets

    IPV4 subnet addresses such as “127.10.138.128” are 4 sets of there are 32 “buckets” separated by dots into four 8 bit “octets”:

    The 127 in the figure above is obtained by adding the base 10 value of each bit “bucket”. Looking at a single octet of 8 bits:

    “Bucket” position: 8 7 6 5 4 3 2 1
    Base 10 value of each bucket: 128 64 32 16 8 4 2 1
    Cumulative base 10 (left to right) 255 127 63 31 15 7 3 1
    Base 2 for 127 in base 10 1 1 0 1 1 0 0 1
    Cumulative base 10 (left to right) 217 89 25 25 9 1 1 1

    To translate a base 2 number of all 1’s (“1111111”) to a base 10 value of 255 we accumulate base 10 values for each “bucket”, left to right.

    To translate the Base 2 set of 1’s and 0’s to a base 10 number of 217, we accumulate the equivalent base 10 number at each position where there is a 1.

    Now let’s look at the relationship between /28 and the “255.255.255.240” subnet mask associated with the /28 in the table of nodes above.

    The “240” base 10 number in the right-most quartet is equivalent to “11110000” in base 2.

    “Bucket” position: 8 7 6 5 4 3 2 1
    Base 10 value of bucket: 128 64 32 16 8 4 2 1
    Base 2 for 240 in base 10 1 1 1 1 0 0 0 0
    Cumulative base 10 (left to right) 240 122 48 16 0 0 0 0

    Putting the three 255 and 240 together we get a continuous set of 1’s followed by four 0’s:

    11111111.11111111.1111111.11110000

    • The 1’s “buckets” on the left side are used to address subnets managed by Amazon.

    • The 0’s buckets on the right side are used to address your individual nodes.

    REMEMBER: Although there are four 0’s buckets, only 3 are used to specify node addresses because one digit (two values) are reserved for network broadcast use (addresses containing all 0’s and all 1’s).

    More on CIDR (Classless Inter-Domain Routing), aka “supernetting”:

    • https://www.youtube.com/watch?v=POPoAjWFkGg IP Subnetting from CIDR Notations (getting network and broadcast addresses).

    • http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html

    • VLSM (Variable Length Subnet Mask)

    • https://cloudacademy.com/amazon-web-services/amazon-vpc-networking-course/build-and-configure-a-nat-instance.html

Automatically create VPC using CloudFormation

VPCs are really software-defined networks (SDN).

     "Resources" : {
        "VPC" : {
         "Type" : "AWS::EC2::VPC",
         "Properties" : {
           "CidrBlock" : "10.0.0.0/16"
         }
       },

       "InternetGateway" : {
         "Type" : "AWS::EC2::InternetGateway",
         "Properties" : {
         }
       },

       "AttachGateway" : {
          "Type" : "AWS::EC2::VPCGatewayAttachment",
          "Properties" : {
            "VpcId" : { "Ref" : "VPC" },
            "InternetGatewayId" : { "Ref" : "InternetGateway" }
          }
       },
   

In the CF JSON to define a VPC, CF automatically populates the “VpcId” : { “Ref” : “VPC” },

  REMEMBER: There is one VPC per Availability Zone.

A single Gateway serves all VPCs because that is the address the public DNS resolves corporate host names to.

## Static Elastic IPs #

NOTE: The use of static IP addresses in configurations in EC2 can be an annoyance to some and a comfort to others.

Historically, working on a physical servers involves use of specific static IPs associated with particular purposes. External monitoring server was manually configured with the IP assigned to each machine. This also creates time pressure (panic) to get specific servers up and running. This led to pressure for servers to be patched rather than risking losing configurations during rebuilds.

Static IPs needed to be protected as secrets because of their long-lived nature in traditional server environments.

A “paradigm shift” in thinking is necessary when moving to the “cloud” because there IP address assignments can be transitory ephemeral. When a server dies in a “12 factor app” environment, additional servers can be brought up automatically by auto-scaling from a common public pool.

AWS provides static IPs in their Elastic IP service.

  WARNING: AWS charges $1 per month for reserved static IPs that are not assigned to a running instance.

PROTIP: Long-lived elastic static IPs are useful to avoid shared IPs that may have been black-listed due to abuse by others.

Resources on this topic:

  * https://launchbylunch.com/posts/2014/Jan/29/aws-tips/
  * https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/

DNS

DNS servers maintain a database of host names to IP addresses.

Amazon’s DNS service is called Route 53 because the default part for DNS servers is TCP 53 / UDP 53.

Its competitors include Dyn.com, GoDaddy, etc.

Routing Rules

AWS VPC Routing Rules are what makes subnets public or private.

NAT

  1. Launch an EC2 instance of a Community AMI built for NATting. Search for “NAT”.

    DEFINITION: A NAT server allows outbound traffic to the external internet. By default, a NAT server allows inbound traffic only through connections already established by an internet host (typically port 80).

    NAT provides IP address assignment and DNS Proxy name resolution services to internal network clients.

To access traffic from a special port from an external host:

  • If the public interface of the NAT server is configured with a single IP address, add a Special Port (for Windows, in the Routing and Remote Assess MMC console).

  • If the public interface of the NAT server is configured with multiple IP addresses, make address reservations to map specific external addresses to specific internal addresses.

    Selection of 006 DNS Servers option at the scope level overrides the selection at the server level.

For security, define some servers can only make outbound calls to the internet (through the NAT server).

  1. PROTIP: A NAT instance provide whatever capacity a single AMI provides, so it should be configured with CloudWatch alarms and traffic metrics.

  2. Prepare before need a script to manually to manage Subnet failover to another NAT in this Amazon article.

    A NAT instance can be configured for port forwarding, bastion hosts.

None of the above is necessary with AWS NAT Gateways which support bursts of up to 10Gbps. They are managed by AWS, so it does not provide traffic metrics nor CloudWatch alarms.

Bastion host

Bastion hosts ???

PROTIP: Up to 5 different security groups can be applied to a single resource.

Only one NACL can be associated with a subnet, to deny specific IP addresses. Separate rules are for inbound and outbound.

PROTIP: NACL rules are numbered to sepcify sequence. To allow for insertion, leave gaps in the numbers. For example, create the first two with 100, 200, etc. so you can later add 150 between 100 and 200.

PROTIP: Remember that EC2 instances by default have Networking > Change Source/Dest. Check ON. But NAT instances require OFF or they wont’ show up on VPC Route Tables.

VPN

PROTIP: When an enterprise development team first begins working with an external vendor or customer, it would likely begin by using a private VPN while the project operates in “stealth mode”.

Configure Site to Site VPN to securely transfer data among Amazon VPCs in different regions or between Amazon VPC to your on-premise data center.

NOTE: Dual ports are usually configured on VPN hardware.

https://app.pluralsight.com/player?course=aws-certified-sysops-admin-associate&author=elias-khnaser&name=aws-certified-sysops-admin-associate-m5&clip=3&mode=live Customer Gateway.

It’s attached to a VPN.

VPC Peering

Peering connections were introduced to route traffic between two VPCs (AZs) in the same region using private (rather than public) IP addresses. This makes it like they are communicating as if they are within the same network. Peering is neither a gateway nor a VPN connection, so doesn’t invoke separate physical hardware and the “single point of failure” nor bandwidth bottlenecks. One useful use case is for more secure interconnection among Active Directory, Exchange, and other common business services.

  • more secure communication among business units/tams
  • stronger integration of CRM, HRMS, file sharing
  • tighter integrated access of core suppliers systems
  • provide monitoring and management of customer AWS resources
  1. Setup Peering in VPC

  2. Accept the Peering request on the target VPC.

ACLs

Access Control Lists

  • Create Internet outbound allow and deny network ACL in your VPC. First network ACL: Allow all the HTTP and HTTPS outbound traffic on public internet facing subnet. Second network ACL: Deny all the HTTP/HTTPS traffic. Allow all the traffic to Squid proxy server or any virtual appliance. http://techlib.barracuda.com/display/BNGv54/How+to+Deploy+the+Barracuda+NG+Firewall+in+an+Amazon+Virtual+Private+Cloud

NACLs

Negative ACLS.

Block all the inbound and outbound ports. Only allow application request ports.

These are stateless traffic filters that apply to all traffic inbound or outbound from a Subnet within VPC. AWS recommended Outbound rules

See http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html

Resources

  • Add Intrusion Prevention or Intrusion Detection virtual appliances to secure protocols and to take preventive/corrective action.

  • Assign
  • Configure Privileged Identity access management solutions to monitor and audit access by Administrators of your VPC.

  • Add anti-virus for cleansing specific EC2 instances inside a VPC. Trend micro offers a product for this.

  • http://harish11g.blogspot.com/2015/06/best-practices-tips-on-amazon-web-services-security-groups-aws-security-managed-services.html

AMS needs to set limits http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html

More on Amazon

This is one of a series on Amazon:

More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps

  4. Git and GitHub vs File Archival
  5. Git Commands and Statuses
  6. Git Commit, Tag, Push
  7. Git Utilities
  8. Data Security GitHub
  9. GitHub API
  10. TFS vs. GitHub

  11. Choices for DevOps Technologies
  12. Java DevOps Workflow
  13. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  14. AWS server deployment options

  15. Digital Ocean
  16. Cloud regions
  17. AWS Virtual Private Cloud
  18. Azure Cloud Onramp
  19. Azure Cloud
  20. Azure Cloud Powershell

  21. Powershell Ecosystem
  22. Powershell on MacOS
  23. Powershell Desired System Configuration

  24. Jenkins Server Setup
  25. Jenkins Plug-ins
  26. Jenkins Freestyle jobs
  27. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  28. Dockerize apps
  29. Docker Setup
  30. Docker Build

  31. Maven on MacOSX

  32. Ansible

  33. MySQL Setup

  34. SonarQube static code scan

  35. API Management Microsoft
  36. API Management Amazon

  37. Scenarios for load