Get started, safely & quickly using AWS GUI, CLI, and Terraform
Overview
- KPI for Onboarding
- BaSA (Be a Solutions Architect)
- Prep Steps
- Console GUI, CLI, API, Mobile
- Auto-Generate Emails with Responders
- Unique emails for Root account
- AWS Services Management Console
- Mobile apps for smart phones
- HashiCorp Terraform
- Programmatic Access
- Automatic key rotation
- Linux AMIs
- Advanced User Data
- Diagrams
- Social
- [_] Create Forum Account
- Tutorial Rock Stars and their presentations
- References
- Manually Rotate Access keys
- Roles for federated access
- Encrypt AWS Credentials
- Installing, updating, and uninstalling the AWS CLI version 2 on macOS
- References
- More on Amazon
- More on DevOps
- More on Security
This is a hands-on tutorial to get new users setup to effectively access and use the AWS cloud - without tedious “talking head” lectures. Here you do some action and explanations. “PROTIP” tags highlight my advice, found in few other places.
Covered here are instructions on how to install and use AWS CLI automation, smart phone apps, and 3rd party tools used by pros.
KPI for Onboarding
Recommendations in this article are intended to improve these Key Performance Indicators (KPIs) of an organization:
A1. What is the total max/average hours an end-user needs to spend between receiving instructions to being completely productive on AWS?
B1. What is the total max/average hours of effort by Administrators to get an AWS account ready for use by an end-user?
The above is a subset of:
A. What is the total max/average hours an end-user needs to spend between receiving a laptop to being completely productive (create and file a Git PR)?
B. What is the total max/average hours of effort by Administrators to get accounts and a laptop ready for receipt by a new employee?
PROTIP: CAUTION: Using speed as the primary basis for judging performance can lead to cutting corners and thus security holes. So security must be a primary consideration. But security is difficult to measure.
BaSA (Be a Solutions Architect)
There are many ways to learn AWS. The key to learning fully AND quickly – to get a job – is to have an experienced mentors in both technical and behavioral skills, plus real-world projects.
Look at the schedule for the FREE AWS
BaSA (Be a Solutions Architect) program
where AWS employees (Amazonians) around the world hold 2-hour live sessions over 12-weeks
(8 technical and 4 behavioural)
on 𝗦𝗮𝘁𝘂𝗿𝗱𝗮𝘆 𝟴 𝗔𝗠 𝗚𝗠𝗧 / 𝟵 𝗔𝗠 𝗨𝗞 / 𝟭.𝟯𝟬 𝗣𝗠 𝗜𝗦𝗧 / 𝟰.𝟬𝟬 𝗔𝗠 𝗘𝗧).
Videos of each batch are
stored on YouTube
coordinated using email 𝗯𝗲𝘀𝗮𝗽𝗿𝗼𝗴𝗿𝗮𝗺𝟮𝟬𝟮𝟮@𝗴𝗺𝗮𝗶𝗹.𝗰𝗼𝗺 and
LinkedIn group,
which at time of writing has 7,621 members.
Hands-on activities:
Networking:
- Introduction to Cloud The binary choice - OSI Model & IP Addressing
- Amazon S3 It’s all connected - Amazon VPC
- Amazon EC2 Hold the door - NACL & Security Group
- AWS IAM Cloud and beyond - VPN and Direct Connect
- Amazon RDS Peer to Peer - VPC Peering & Transit Gateway
- Amazon DynamoDB What’s in a name? - Route 53 & DNS Resolver
- Amazon Cloudwatch The tool box - Flow Logs & Reachability Analyer
Prep Steps
Here are the steps to get ready to use AWS:
- Obtain a DNS domain name for experimentation using AWS Route53.
- Automate generation of new email accounts (with hostname as a variable) using AWS SES (Simple Email Service).
- Emulate user action for AWS account verification within AWS SES.
Console GUI, CLI, API, Mobile
There are several ways to interact with AWS:
SECURITY PROTIP: Many enterprises do not permit use of interactive CLI and Console GUI in production and instead allow only automated API calls by IaC (such as CloudFormation and Terraform). This is to ensure version control and repeatability during testing.
-
If you don’t have a root account, open an internet browser (such as Google Chrome, Apple Safari, Mozilla Firefox, etc.) and go to the AWS GUI (Graphical User Interface), aka “AWS Management Console” to create and configure, then lock down a Root Account</a>. Authentication is by user name and password plus MFA.
-
CLI (Command Line Interface) using the MacOS Terminal or Windows PC Command program for “programmatic access” into the AWS cloud. Authentication is by API keys (public and private) which are stored in a file on your laptop.
-
APIs called by custom programs calling AWS’s API (Application Programming Interface). This is the mechanism behind the scenes by mobile apps.
Auto-Generate Emails with Responders
Global Administrators can reduce time and hassle that both they themselves and their end-users (internal customers) by doing ALL AWS account setup tasks rather than giving a long list of instructions for end-users to follow (as shown below), then providing support to those who won’t or don’t follow instructions. This is especially true for those who are “not technical”. This means automation of email creation and also having automation impersonate each user’s email for verification of AWS account, GitHub, etc.
PROTIP: Use a separate email address for each AWS account you create. Global Administrators working with AWS need to have admin control of an email system to create email accounts and (automatically) read/answer sample user emails.
-
Individual learners need to generate several email accounts to take advantage of “Free Tier” that only lasts a year each. AWS learners need to create their own account to not disturb corporate work.
-
Corporate Global Administrators need to generate emails for each new employees who join.
Within an organization, it’s common for a separate account to be created for each department and project as well as each user. This is to limit the blast radius when a user’s credentials become compromised, a situation we need to prepare for.
Unique emails for Root account
An enterprise typically creates several AWS accounts. Within an organization, it’s common for a separate account to be created for each department and project as well as each user. This is to limit the blast radius when a user’s credentials become compromised, a situation we need to prepare for.
WARNING: “Root account” credentials have unlimited access to AWS resources for the account and thus unlimited ability to rack up charges used controls billing is called . By resources I mean: users, groups, roles, IAM Access Policies, API keys, etc. globally for all regions.
Thus, the root account is used only for creating sub-accounts and for emergencies. Global Adminstrators create sub-accounts for use when doing billing and other administrative tasks.
PROTIP: If you are creating a production account for an organization, create an email address which you use only for managing AWS and not for regular email use and certainly not for doing shopping on Amazon.
The account which controls billing is called the root account, which as unlimited access to AWS resources and unlimited ability to rack up charges. By resources I mean: users, groups, roles, IAM Access Policies, API keys, etc. globally for all regions.
Secure that email address with multi-factor authentication with Google or whoever hosts your email server. Also have a way for one person (or maximum two) you trust to be able to access the account in case you are not able to.
Unique Browser Profile for Each Email
- Install Google Chrome because it has detection of malicious conditions.
- You will have several gmail addresses, one for each AWS account you create.
-
To avoid confusion between Google accounts, install the Multi Login Helper extension to create a new browser profile for each AWS account you create.
PROTIP: The Multi Login Helper extension is also useful for creating a new browser profile for each Google account you have.
Marketing Page
-
Use an internet browser to get on the AWS marketing page at
-
Explore menu items:
PROTIP: Right-click on each link to “open in a new tab”. Then quickly switch back and forth between this tutorial and other browser tabs by pressing Command+` (backtick at the left of the 1 key). However, tabs set to full-screen are not accessible this way but by pressing shift+command+/ to see the menu to select the tab you want to switch to.
Also, bookmark this page in your browser for quicker frequent access.
Sign-up pages
-
PROTIP: There are several different sign-up pages: one for each country.
- If you want to create a stand-alone account in the US:
https://portal.aws.amazon.com/billing/signup#/start/email - If you’re working with an AWS sales person assigned to a business:
https://aws.amazon.com/resources/create-account/ - If you’re a student or educator:
https://aws.amazon.com/education/awseducate/ - If you’re using a gov cloud:
https://aws.amazon.com/government-education/government/
Root Password
- If you want to create a stand-alone account in the US:
-
Create a new 1Password entry to store the email, Account Name, password, Account ID, Secret info.
- For “AWS account name”, examples are “master-billing” but the email works too.
- Switch to your email tab to click the link to verify your email address.
-
PROTIP: When providing answers to Security Challenge Questions, do not specify the real answer, which someone stole or figured out through social engineering. Instead, answer with nonsense
- Write that secret information down in 1Password or a paper in your fire-proof vault.
-
Write down your Account Id number (12 digits).
-
Supply a strong password.
PROTIP: Use 1Password so that you can easily generate up to 64 character password, but remember only one password to access the 1Password database of secrets. 1Password encrypts its database so that you can make backups (to a USB drive or secure cloud). I favor 1Password because it provides a way to sync changes with your smartphone without going through the internet.
Because you only have to remember one master password, you can are free to change various passwords as often as you want with no fear of forgetting them.
-
Click “Continue”.
If you have 1Password installed, you would be prompted to create a new account.
-
Provide phone number.
PROTIP: If you provide a Google Voice virtual number, it would be less of a hassle in case you change the actual number of your phone assigned by your carrier. So in case you change phone vendors (from ATT to T-Mobile), you only need to change it in Google Voice.
Compare Support Plans
https://support.microsoft.com/en-us/home/contact?SourceApp=smcivr2
-
Click Amazon’s Support Plan page here.
Admins call (800) 865-9408 or (800) 642 7676 (toll-free, US only). Outside the United States, see global support phone numbers.
The Basic account does not enable you to communicate with Amazon people who can answer technical questions.
The $29/month Developer Plan enables you to open an unlimited number of support cases only via email, with a 12-hour response time if “system impaired”. Otherwise, the SLA is 24 hours.
The $100/month Business Plan enables you to have 24/7 chat, phone, as well as email access with AWS Support people on an unlimited number of support cases, with a 1-hour response time for “production down” issues, or 4-hour response for “production impaired” issues.
Amazon’s Enterprise Plan for $15,000/month gets you 15 minute response on “business critical system down” issues. This plan also comes with an assigned TAM (Technical Account Manager).
These dollar amounts are minimums, not fixed prices.
https://aws.amazon.com/premiumsupport/programs/iem/ mentions “AWS Infrastructure Event Management (IEM) offers architecture and scaling guidance and operational support during the preparation and execution of planned events, such as shopping holidays, product launches, and migrations.”
-
Scroll down to mouse over the “$29” on the Pricing line at the bottom of the table.
PROTIP: Pricing for Developer support is the Greater of $29 or 3% of monthly AWS usage, so you will pay more than $29 if you spend more than $966.67.
- Scroll back up to click the “Pricing example” link on the right.
-
Notice that if your spend is $2,000, Amazon bills you $60 for support, not $29.
-
Click the “Business” and “Enterprise” buttons in the pop-up to see sample volume pricing tiers.
Credit card
-
Provide address, which may be used to verify your credit card.
PROTIP: CAUTION: Once you give Amazon a credit card number, you cannot remove it. Amazon can continue to charge for it until the card expires in several years.
PROTIP: You need a credit card to open an account. But to limit exposure, some people provide to AWS numbers from a pre-paid reloadable Visa gift (debit) card pre-paid online (which has an expiration date and some have a monthly service fee). The Drawpay card provides a 1% refund on purchases and a mobile app to view balances. Others provide fee-Free cash withdrawal at over 25,000 MoneyPass ATMs.
Students may want to create several accounts to take advantage of the free tier multiple times. However, uniquely different phone numbers, addresses, and credit cards are not needed for each identity.
-
PROTIP: Where you keep information about your credit card, note the email address and account name using that credit card.
-
Confirm the phone number by answering Amazon’s phone call.
-
For now, click “Free” to select a plan. A comparison on plans is discussed below.
-
Click “Free” to be prompted to sign-in with your new credentials.
When signing in under IAM, type your Account Id number rather than your root email address.
To identify your Account ID:
-
Click on your name on the upper black menu at the top of the page, then select “My Account”.
-
Click your account name at the top black menu for this menu:
-
Copy the Account Id and paste it in the notes associated with where you saed your account email and password (within 1Password).
PROTIP: This 12 digit number is given out for others to use to sign in using sub-accounts.
-
Scroll down to click “Edit” next to “Alternate Contacts” and put the other person who knows how to get into the account in for the Billing.
- Scroll down to click Edit to the right of “Configure Security Challenge Questions”.
-
Write down your security challenge questions and answers where you wrote your Account Id.
PROTIP: Treat the answers as another set of passwords because others my discover the real answers via social engineering. Answer with some nonsense that has no basis in reality.
AWS Services Management Console
-
If you are at the AWS marketing page, click “My Account” for this menu:
-
Get the AWS Management Console:
-
PROTIP: Bookmark this URL
All Amazon services
-
PROTIP: The Chrome browser extension “AWS Services” provides a list of services by name and category so you can click it to get to Console and documentation for each service.
-
In the AWS Console, click to view all Services at the upper-left black menu band for:
https://us-east-2.console.aws.amazon.com/console/home
-
Scroll to the category “Security, Identify, and Compliance” list of ever-growing services:
- WAF (Web Application Firewall) provides application-level attacks such as SQL injection and cross-site scripting.
-
Shield protects against DDoS (Denial of Service) attacks
- Click “Artifact” (at the bottom of the list) to read documents associated with security certifications.
- Cognito provides an API to federate authentication with various social identity providers (Facebook, Twitter, etc.)
- GuardDuty
- Inspector
- Amazon Macie
- AWS Single Sign-On
- Certificate Manager manages security certificates
- Cloud HSM provides
- Directory Service
- Cloud Trail audits usage
PROTIP: What’s not listed above is the AWS Best Practices which this tutorial addresses.
-
Read the User Guide for each service at:
https://aws.amazon.com/documentation
Root account lockdown
-
On a browser in the AWS Management Console, select IAM (for Identity Access Management) for the list Security Status
A new account will have this:
To get back to this later, click “Dashboard” on the IAM menu on the left.
The FAQ to this is at https://aws.amazon.com/iam/faqs
-
Click on “Delete your root access key”.
-
Check “Don’t show me this message again” and Continue to Security Credentials.
Password
-
PROTIP: Use 1Password to store your passwords so that you can use a “strong” password of so many characters that it will take hackers too much time to crack it. Because you only have to remember one master password, you can are free to change various passwords as often as you want with no fear of forgetting them.
Apply an IAM password policy
-
Click “Manage Password Policy” so AWS will ensure that “strong” passwords are used (and not easy to guess ones).
AWS defaults are terrible:
Over time, as hackers have access to more powerful computers that can guess passwords quicker, so larger passwords are necessary to make it more difficult to crack.
-
PROTIP: The largest Minimum password length AWS allows is 128 characters. But 1Password can generate up to only 64 characters. Practically, 22 characters is a reasonable minimum. Require at least one number (digits) and one non-alphanumeric symbol character.
-
Scroll down to “Security Token Service Regions” and deactivate regions your organization will never use.
PROTIP: The region is where most of your users are located. New services are usually restricted to one region, such as N. Virginia or N. California where AWS does development work.
MFA (Multi-Factor Authentication)
This has AWS text or call your smartphone (a virtual device) to make sure that it’s really you logging in.
- Click Activate MFA
- Click “A virtual MFA device”.
-
Click Next Steps.
Install MFA app
- On your iPhone or Android mobile app, open the Store app.
- Search for Google Authenticator app (if you don’t already have it installed).
-
Click “Get” to install it.
-
Click “Open”.
- In the the Google Authenticator app, click the “+” icon at the top of the screen.
- Click “Scan barcode”.
- Align the QR code (with the square of dots) within the green box.
- Wait for the Google Authenticator app to display two codes. Under the codes we want now begins with “root-account-mfa-device@” followed by the 12-digit Account Id.
-
Type the first code for the account into the AWS Console website “Authentication code 1”.
PROTIP: Do not type the space between numbers so that you enter only 6 digits.
-
Press Tab and type the second code in “Authentication code 2”.
PROTIP: A new code is created every minute.
-
Scroll down to click “Activate virtual MFA” at the bottom of the screen.
MFA in profile
To specify use of MFA in an assumed role provider profile, see this example of credentials file:
[profile prod-access] role_arn=arn:aws:iam::123456789012:role/ReinventProdAccess source_profile=development [profile prod-full-s3-access] role_arn=arn:aws:iam::123456789012:role/FullS3Access source_profile=development mfa_serial=arn:aws:iam::18490616333:mfa/james
-
Test on Console: VIDEO:
aws s3 ls --profile prod-full-s3-access
The response is a prompt waiting for manual input:
Enter MFA code: _
### Create Admin sub-account
-
In the IAM page click “Create individual IAM users”. What it says is important:
“Create IAM users and give them only the permissions they need. Do not use your AWS root account for day-to-day interaction with AWS, because the root account provides unrestricted access to your AWS resources.”
- Click “Manage users”.
- Click “Add User”.
-
PROTIP: For the user name field, define a pattern of up to 64 characters with dashes (instead of spaces and underlines) to separate words.
For the Administrator to do work (of assigning):
root-admin-work
- Click “Programmatic access”.
- If you would like to use AWS Management Console access, leave the default for Autogenerated password because you’ll create a new password at next sign-in.
-
Click “Next: Permissions”.
We’ll add groups later, below.
-
Click “Attach existing policies directly” because the Admin account it is limited.
-
Rather than granting “AdministratorAccess” which gives all access, give policy to what :
- SystemAdministrator
-
IAMFullAccess covers the others:
- IAMReadOnlyAccess
- IAMSelfManageServiceSpecificCriteria
- IAMUserChangePassword
- IAMUserSSHKeys
- Click “Next: Review”.
-
Click “Create user”.
Inform user of credentials
-
To see what is sent if you click “Send email”, right-click on the link and “Copy Link”, then paste in a text editor to see:
subject=Welcome to Amazon Web Services body=Hello, You have been given access to the AWS Management Console for the Amazon Web Services account ID ending in 8630. You can get started by using the sign-in information provided below.%0A%0ASign-in URL: https://103265058630.signin.aws.amazon.com/console%0AUser name: root-admin-work Your initial sign-in password will be provided separately from this email. When you sign in for the first time, you must change your password. Sincerely, Your AWS Account Administrator
-
PROTIP: Along with the Access Key Id and Secret access key, the default Region and format are also required to perform “aws configure”, so add that information in the email.
PROTIP: Add what AWS Groups and associated Policies the user has been given.
PROTIP: Also include in the email, for those who use AWS CLI, how to install it and 3rd-party tools.
For those who use the AWS Console GUI, explain the mobile apps to install. Provide them the URL with the region included, such as:
https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2
NOTE: Baking different zones into Console URLs makes for more direct connections and removes issues from using a single URL/DNS.
-
Click “Download .csv” to download a “credentials.csv” file to your Downloads folder. It contains columns are a couple columns different than the “Add User” GUI:
User name, Password, Access key ID, Secret access key, Console login link
The “Console login link” is the “Sign-in URL” in the email.
Apply an IAM password policy
- Click “Manage Password Policy” so AWS will ensure that “strong” passwords are used (and not easy to guess ones).
AWS defaults are terrible: </a>
PROTIP: Over time, as hackers have access to more powerful computers that can guess passwords quicker, larger passwords are necessary to make them more difficult to crack.
-
PROTIP: The largest Minimum password length AWS allows is 128 characters. 1Password can generate up to only 64 characters. Practically, 22 characters is a reasonable minimum. Require at least one number and one non-alphanumeric character.
PROTIP: Each site may have different rules about what special characters are allowed. So generate a smaller string, then manually add special characters. Copy the final string before pasting into the form.
-
Click “Apply password policy”.
Deactivate regions not used
On the same “Account settings” page:
-
Scroll down to “Security Token Service Regions” and deactivate regions your organization are not using.
PROTIP: Select a Region where most of your target users are located. New services are usually restricted to one region, such as N. Virginia or N. California where AWS does development work.
Admin Sign In
-
Sign out and sign in again to the AWS Console using the newly created admin sub-account.
Quick Access icons
Save time by quickly get to the most frequently used services by having their icons at the top (black) menu bar.
- Click the push-pin icon.
-
One by one, drag the icon on the list and drop it on the top black menu to the left of the orange push pin. If you don’t see the black menu, pause just under the browser URL for the browser to automatically scroll.
PROTIP: The services most often used are IAM, VPC, EC2, S3
-
If you have good memory of what icons mean, change the Settings to “Icons only”.
Cases in Support Center
-
To view support cases filed and their status, see:
https://console.aws.amazon.com/support/home
Policies for this are:
- AWSSupportAccess (Allows users to access the AWS Support Center)
- SupportUser (This policy grants permissions to troubleshoot and resolve issues in an AWS account. This policy also enables the user to contact AWS support to create and manage cases)
-
Scroll down to view videos on specific technical issues by Amazon people.
On the lower-right corner, there are links to AWS Documentation, Getting Started Guides, Knowledge Center, Whitepapers, and AWS Forums.
Mobile apps for smart phones
-
Get the AWS Console app on your mobile phone:
On Google Android mobile phones
On your iPhone, open the Store app and search to get AWS Console. Make sure the publisher is AMZN Mobile LLC which creates all Amazon’s apps.
PROTIP: These apps got low review scores because the app only lets people read-only, but not change anything. And the 2FA is clunky.
- Add an identity: select Root/IAM account or Federation.
- Enable Face ID on iPhones.
-
Provide email, CAPTCHA security, password, email verification code. Success is seeing this:
HashiCorp Terraform
HashiCorp’s Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.
VIDEO: To use Terrafrom IaC (Infrastructure as Code) to create a AWS EC2 instances (instead of Chef, Puppet, Ansible, etc.):
CAUTION: The AWS way of keeping credentials in the $HOME/.aws/credentials file is not secure because if your laptop is compromised or stolen, those secrets could be used without authentication. So many organizations request that secret credential files be temporary (valid for just one day). So some make available a corporate “Vending Machine” app which generates credentials instead of the manual process below.
- Click on your email at the upper-right corner to select “Security credentials” for the IAM page.
- Scroll to click “Create Access keys”, “Command Line Interface”.
- Check “I understand”, “Next”.
- Construct a Description tag value that satisfies your organization’s naming conventions.
-
Click “Create access key”.
-
Switch to a Terminal to issue aws configure to specify the Access Key ID and Secret Access Key.
PROTIP: The aws configure command creates a file at $HOME/.aws/credentials with the Access Key ID and Secret Access Key.
- Switch back to the web page.
- Click the copy icon for the Access Key. Switch to your secret file and paste the value into a text file.
- Click the copy icon for the Secret Access key. Switch to your secret file and paste it into the same text file.
- Specify Default region such as us-east-1 - the default.
-
Specify Default output format json.
- Switch to the web page to click “Done”.
-
Identify a GitHub repo you want.
- Switch to the Terminal.
- Create or navigate to a folder for your GitHub account to receive repositories cloned.
-
Get that sample Terraform repo, and cd into it.
git clone https://github.com/wilsonmar/aws-ec2-micro.git --depth 1 cd aws-ec2-micro
PROTIP: Typically, for secure production usage, many resources would be created, including roles to limit access. So using Terraform would be faster, easier, more accurate, and more secure than manual creation clicking and typing on the AWS Console GUI.
-
Edit the files which specify the AWS provider described at https://registry.terraform.io/providers/hashicorp/aws/latest/docs, such as this main.tf
provider "aws" { region = "us-east-1" } resource "aws_instance" "example" { ami = "ami-0c55b159cbfafe1f0" instance_type = "t2.micro" }
REMEMBER: The ami id is tied to the region and instance_type for which it was created. For that reason, many run a Bash script to get the latest ami or use the Packer utility to create a custom ami. That is safer than referencing “golden” ami images created by another organization to meet compliance standards: APRA, MAS, and NIST4.
PROTIP: We recommend that you run a Bash shell file to select the latest ami and for whatever region was selected for the server instance_type. The script can confirm whether the instance_type specified is available in the region specified. The script would also have coding to set environment variables in a secure way, consistently over time among teammates. This also enables AWS Tags to be specified effortlessly, such as “CreatedBy” with your email address pulled in automatically. See my documentation.
PROTIP: Many specify in the user_data section within the main.tf file Bash scripts containing Ansible commands to run immediately after EC2 instance boot up.
-
A terraform.tfvars file is commonly specified to specify custom values to replace default values in the main.tf file.
PROTIP: The terraform.tfvars file may contain secrets, so its file name is specified in .gitignore to prevent it from being checked into GitHub.
-
The script would collect locally Terraform provider files specified in the main.tf file:
terraform init
-
Create resources:
terraform plan --auto-approve --var-file=../vars/ec2.tfvars
If that works:
terraform apply --auto-approve --var-file=../vars/ec2.tfvars
The –auto-approve option is used to avoid the need to type “yes” to confirm.
PROTIP: A Bash script issuing the above commands would add additional steps such as checking for errors, to ensure that resources with vulnerabilities are not even created.
-
Switch back to the web page to view the resources.
-
Use the resources.
-
Delete the resources previously created by Terraform files in the folder:
terraform destroy --var-file=../vars/ec2.tfvars
- REMEMBER: Delete the credentials file after use.
VIDEO: HashiCorp has a “Sentinal” product component which enforces various fine-grained rules (policy sets) to what can be done by each role. It also estimates monthly cost from cloud usage.
Rules in HashiCorp’s Foundational Policy library is at https://github.com/hashicorp/terraform-foundational-policies-library. Such “Policies as Code” are crafted based on Center for Internet Security (CIS) Benchmarks [pdf] (including Compute, Databases, Kubernetes, Storage, Networks) covering Azure and GCP as well as AWS.
Programmatic Access
Instead of doing what other clouds do (an aws login command which prompt for a user name and password), aws commands reference a specifically-named file at $HOME/.aws/credentials created by command aws configure.
The aws configure command creates that file after prompting for access key identifiers (AKIDs) to an AWS account. Press Enter to accept the value previously defined:
- AWS Access Key ID [******L5ZQ]:
- AWS Secret Access Key [******+1MD]:
Stored with credentials are also:
- Default region name [us-east-1]:
- Default output format [json]:
To create AKID credentials, AWS asks that account owners to manually use the IAM GUI to disable programmatic access to their root (email) account and protect it with MFA (Multi-factor Authentication)
The AWS Management Console provides a way for account owners (administrators) to manually create IAM user accounts for programmatic access.
For programmatic access to resources running inside AWS, the best practice is to use IAM roles which are not associated with a specific user or group. Any trusted entity can assume the role to perform a specific business task. A resource can be granted access without hardcoding an access key ID and secret access key into the configuration file. For example, you can grant an Amazon Elastic Compute Cloud (EC2) instance access to an Amazon Simple Storage Service (Amazon S3) bucket by attaching a role with a policy that defines this access to the EC2 instance. IAM dynamically manages the credentials for you with temporary credentials it rotates automatically.
Outside AWS (on a Terminal/Console on your laptop), a dedicated service account should be created for each use case with only the permissions needed to limit the “blast radius” if credentials are compromised. For example, if a monitoring tool and a release management tool both require access to your AWS environment, create two separate service accounts with two separate policies that define the minimum set of permissions for each tool.
CAUTION: The problem with IAM user account secrets is that they are long-running secrets stored in the credentials file in clear-text. Someone who clicks on a roque link on a phishing email would expose that file for theft. Many who lose control of their AWS credentials see bills from Amazon of thousands of dollars in unauthorized use (mining Bitcoins).
CloudAcademy.com and many enterprises create a centrally-administered https://aws.amazon.com/code/token-vending-machine-for-identity-registration-sample-java-web-application/ “Vending Machine” application to generate and dispense temporary IAM user accounts with access keys. Such credentials are valid for only 12 hours or less.
But that requires tedious repeated manual effort. Securing temporary accounts with MFA adds to that toil.
Claim S3 Bucket names
The AWS Account Administrator has a fudiciary responsibility to secure Intellectual Property assets.
S3 Bucket names are universally unique among all AWS customers. So just as there are domain name squatters who register and sit on .com host names for sale at high prices to those who actually use the names, the administrator of root accounts for an organization should register your organization’s brand names before others get them first.
To create a bucket for each host name registered on GoDaddy, Google Domains, etc.
- Click S3 from among services.
- Click the blue “Create bucket” button.
- Type in the host name (such as “wilsonmar.com”) in the Bucket name field.
-
Select your home Region.
PROTIP: Claiming a Bucket name in one region locks it up for all Regions.
- Click “Next”.
- Click “Next”.
- Click “Next” to manage users.
- Click “Create Bucket”.
QUESTION: Terraform?
Automatic key rotation
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html describes automatic rotatation of AKID credentials (with a quick MFA challenge answered on a mobile phone)
The auto-rotation of AWS IAM User Access Keys diagrammed above is from these guideline from Feb. 2019 uses MIT-licensed CloudFormation templates and Python scripts defined in https://github.com/aws-samples/aws-iam-access-key-auto-rotation and described step-by-step in this Word-format Document.
Setup S3 buckets in the US East (N. Virginia) Region (us-east-1). It runs every 90 days. At 100 days it disables and at 110 days it deletes the old Access Keys. It sets up a secret inside AWS Secrets Manager to store the new Access Keys, with a resource policy that permits only the AWS IAM User access to them.
Another automation sets up an Amazon DynamoDB table to house the email addresses of accounts rotated. These emails are used by a SNS Topic to send alerts when rotation occurs.
Alternately, you can refactor to send a Slack message instead of email (not shown in the diagram).
aws sts assume-role --role-arn arn:aws:iam::123456789012:role/role-name \ --role-session-name "RoleSession1" \ --profile IAM-user-name > assume-role-output.txt
https://aws.amazon.com/blogs/security/how-to-rotate-access-keys-for-iam-users/
$ aws iam list-access-keys { “AccessKeyMetadata”: [ { “AccessKeyId”: “AKIAI2YGLLOSZDQ3L5Z1”, “Status”: “Active”, “CreateDate”: “2020-06-12T04:04:22+00:00” } ] }
AWS IAM commands use unique access key identifiers (AKIDs) to refer to individual access keys.
$ aws iam create-access-key –user-name Alice
Identity and Access Management (IAM) roles for Amazon EC2.
https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys
https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html
- Grant temporary access keys - aws sts assume-role.
https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-one-user-multiple-passwords.html
Additionally, add conditions to the policy that further restrict access, such as the source IP address range of clients. The example policy below grants the needed permissions (PutObject) on to a specific resource (an S3 bucket named “examplebucket”) while adding further conditions (the client must come from IP range 203.0.113.0/24):
{ "Version": "2012-10-17", "Id": "S3PolicyRestrictPut", "Statement": [ { "Sid": "IPAllow", "Effect": "Allow", "Principal": "*", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::examplebucket/*", "Condition": { "IpAddress": {"aws:SourceIp": "203.0.113.0/24"} } } ] }
IAM CLI
AWS Identity and Access Management (IAM) controls access to users, groups, roles, and policies.
-
List users:
aws iam list-users --query Users[*].UserName
-
List groups which the user belongs to :
aws iam list-groups-for-user --username ???
-
Create a new user named “MyUser”:
aws iam create-user --user-name MyUser
The response is:
{ "User": { "UserName": "MyUser", "Path": "/", "CreateDate": "2012-12-20T03:13:02.581Z", "UserId": "AKIAIOSFODNN7EXAMPLE", "Arn": "arn:aws:iam::123456789012:user/MyUser" } }
-
Add the user to the group:
aws iam add-user-to-group --user-name MyUser --group-name MyIamGroup
-
To verify that the MyIamGroup group contains the MyUser, use the get-group command:
aws iam get-group --group-name MyIamGroup
The response:
{ "Group": { "GroupName": "MyIamGroup", "CreateDate": "2012-12-20T03:03:52Z", "GroupId": "AKIAI44QH8DHBEXAMPLE", "Arn": "arn:aws:iam::123456789012:group/MyIamGroup", "Path": "/" }, "Users": [ { "UserName": "MyUser", "Path": "/", "CreateDate": "2012-12-20T03:13:02Z", "UserId": "AKIAIOSFODNN7EXAMPLE", "Arn": "arn:aws:iam::123456789012:user/MyUser" } ], "IsTruncated": "false" }
Linux AMIs
Types of operating system AMI:
- Amazon Linux 2014.09.2 (CentOS)
- Red Hat Enterprise Linux 6.6 (RHEL)
- SUSE Linux Enterprise Server 12
- Ubuntu Server 14.04
Advanced User Data
https://gist.github.com/mikepfeiffer/
-
aws Get-AWSCredentials -ListProfiles
Diagrams
ProcessOn.com provides a free on-line tool to draw diagrams such as this
At architecture/icons Amazon provides a sample .PPTX (PowerPoint 2010+) file (AWS_Simple_Icons_PPT_v16.2.22.zip). Lines used to illustrate the hierarchy:
PROTIP: Use different colors for lines and text to reduce visual confusion.
You can also download a zip containing .png and .svg files of icons (AWS_Simple_Icons_EPS-SVG_v16.2.22.zip).
Social
-
https://console.aws.amazon.com/ec2/home
https://www.twitch.tv/aws/videos/all videos include:
-
IoT at re:Invent 2017 video with Sarah Cooper (General Manager IoT), Kip Larson (Principal Product Manager for IoT Analytics)
-
Sign-up to receive the AWS Architecture Monthly (FREE Subscription) on your Kindle account.
[_] Create Forum Account
-
PROTIP: To ensure anonymity interacting on public forums, the Administrator should create in a public email system (such as gmail.com, hotmail.com, etc.) an email address for use on forums. Don’t use a real name in the email address, but a positive adjective with a number to ensure it’s unique, such as “concerned123”.
AWS says “Your email will be kept private” but I don’t trust that they can’t be hacked.
-
Go to the AWS forums at URL:
-
Register the new email address along with an AWS Nickname without a proper name, such as, again, “concerned123”.
-
Use that email in StackOverflow.com and other public forums.
Tutorial Rock Stars and their presentations
Jeff Barr (@jeffbarr), AWS Chief Evangelist makes announcements of all new stuff at the company’s AWS Blog and #AWS Twitter hash-tag
Yan Kurniawan
-
Ansible for AWS book 280 pages for minimum $30 published on 2016-01-15 by Yan Kurniawan living in Sydney, Australia #ansible4aws.
J O’connner:
- http://joconner.com/
Ryan Scott Brown @ryan_sb
- https://serverlesscode.com/post/new-ssl-tls-cert-manager-acm/
Matt Wood, @mza, Product Strategy @ Amazon Web Services
References
After signing up for https://www.aws.training, Authentication and Authorization with AWS Identity and Access Management 15 minutes
SWF (Simple Workflow Functions) sequences manual work.
AppStream streams desktop apps (like Citrix).
Elastic Transcoder of videos into various sizes and formats (ogg, mp4, etc.)
Orion Papers on Lucidchart
https://scriptcrunch.com/aws-certification-iam-essentials-cheat-sheet/
VIDEO: How to Configure the AWS CLI | Amazon Web Services | AWS Nov 26, 2017 by deeplizard
https://docs.aws.amazon.com/cli/latest/index.html AWS CLI Command Reference
-
To verify the identity being used in AWS CLI:
aws sts get-caller-identity
A sample response:
"Account": "103265058630", "UserId": "AIDAJHXCZNAH2MEXAMPLE", "Arn": "arn:aws:iam::103265058630:user/root-admin-work"
Alternately, use an alias defined:
aws whoami
Define groups to assign permissions
PROTIP: For a user to do something usually require several AWS resources. So several permissions need to be granted to a user. To simplify assignments, we define Groups of permissions which we then can assign to each user.
In other words, An IAM group is a management convenience to manage the same set of permissions for a set of IAM users.
The AWS CLI command to create a group named “MyIamGroup” is:
aws iam create-group --group-name MyIamGroup
A sample response:
{ "Group": { "GroupName": "MyIamGroup", "CreateDate": "2012-12-20T03:03:52.834Z", "GroupId": "AKIAI44QH8DHBEXAMPLE", "Arn": "arn:aws:iam::123456789012:group/MyIamGroup", "Path": "/" } }
The AWS CLI command to create a S3 security group:
aws ec2 create-security-group --group-name my-sg --description "My security group"
A sample response:
{ "GroupId": "sg-903004f8" }
-
Click Manage Groups then Create New Group.
PROTIP: Groups are usually associated with a particular job: admin, sales, HR, front-end developer, back-end developer, etc.
A user can belong to multiple groups. More complex organizations manage differences in permissions for company, division, project, location, job level, etc. So 128 characters may not be enough if large words are used. Thus, abbreviate and use acronyms.
PROTIP: Put abbreviations and acronyms in a wiki publicly available to avoid duplicate usage.
-
“aws_iot_buttons” is the group name I use as an example.
PROTIP: Use dashes. Space characters are not allowed. On March 1, 2018 AWS removed the ability to use underscores in S3 bucket names.
The list shown are “AWS Managed”.
-
Click on Policy Type to select Job function.
-
PROTIP: Instead of scrolling down the massive list in Attache Policy (Alexa, Amazon, AWS, etc.), type in the Filter field the first few letters (such as “IoT”) and the list gets smaller. Notice the filter you type is applicable to not just characters beginning with what you typed, but also characters inside names as well.
- Click to select.
-
Click “Create Group”.
Note different policies have different levels of access, with admin having more capabilities than “read only” ones.
- Names shown on the screen is called a “Policy Summary”.
-
Click “JSON” to see the file that AWS reads to assign policies. Here you seen what Actions the policy allows.
-
Click “Access Advisor” to see users who have been assigned to use the policy.
https://docs.aws.amazon.com/iot/latest/developerguide/create-iot-policy.html
Manually Rotate Access keys
See that "AWS recommends that you rotate your access keys every 90 days"?
Some find it easier to remember by doing it on the first day of each month.
Why? There are thousands of big computers around the world literally staying up at night trying different combinations.
-
PROTIP: Make an appointment on your Calendar with a recurring schedule.
PROTIP: Rotation applies to access key of IAM child accounts, not the root account.
You don’t want programmatic access to your root account, so you don’t need no stinkin’ keys.
-
Click Delete to the key. Write down the date Created.
Don’t create a new Access Key.
Use groups to assign permissions
PROTIP: For a user to do something usually require several AWS resources. So several permissions need to be granted to a user. To simplify assignments, we define Groups of permissions which we then can assign to each user.
In other words, An IAM group is a management convenience to manage the same set of permissions for a set of IAM users.
-
Click Manage Groups then Create New Group.
PROTIP: Groups are usually associated with a particular job: admin, sales, HR, front-end developer, back-end developer, etc.
A user can belong to multiple groups. More complex organizations manage differences in permissions for company, division, project, location, job level, etc. So 128 characters may not be enough if large words are used. Thus, abbreviate and use acronyms.
PROTIP: Put abbreviations and acronyms in a wiki publicly available to avoid duplicate usage.
-
“aws_iot_buttons” is the group name I use as an example.
PROTIP: Use underlines or dashes. Space characters are not allowed.
The list shown are “AWS Managed”.
-
Click on Policy Type to select Job function.
-
PROTIP: Instead of scrolling down the massive list in Attache Policy (Alexa, Amazon, AWS, etc.), type in the Filter field the first few letters (such as “IoT”) and the list gets smaller. Notice the filter you type is applicable to not just characters beginning with what you typed, but also characters inside names as well.
- Click to select.
-
Click “Create Group”.
Note different policies have different levels of access, with admin having more capabilities than “read only” ones.
- Names shown on the screen is called a “Policy Summary”.
-
Click “JSON” to see the file that AWS reads to assign policies. Here you seen what Actions the policy allows.
-
Click “Access Advisor” to see users who have been assigned to use the policy.
https://docs.aws.amazon.com/iot/latest/developerguide/create-iot-policy.html
Create IAM Users
- Click Users on the left menu.
- Click Add User.
-
Specify User Name. For example: user1@myco.com
PROTIP: Use underscores to separate words in IAM User Names rather than spaces.
- Check “Programmatic Access”.
- Uncheck “User must create a new password at next sign-in”.
- Click “Next: Permissions”.
-
Click “Attach existing policies directly” for the first user.
PROTIP: The policy attached depends on what the user will be allowed to do.
- Send to each user the AccountId, UserName using a different mode of communication than the password.
- User signs in using the credentials Account Id, the UserName, and password
-
Click “Send email”
PROTIP: Send credentials to your alternate email rather than to a cloud drive (Amazon, Google, Box, etc.); an email account that you setup with a fake birthdate and other personal information; one you never give out to anyone.
Roles for federated access
An analogy is a private ball where royal guests arrive wearing formal attire present an invitation card to enter. The fancy outfits with sashes and medals are kinda like group permissions that confer permissions to someone. The invitation card is kinda like IAM roles which are only for specific times.
The host of the party is kinda like AWS’s STS (Security Token Service) identify broker which grants access tokens to enable services to “assume” a role to perform on AWS services.
IAM roles are used by computer programs reaching through Enterprise identity federation into Microsoft Active Directory using SAML (Security Assertion Markup Language) or through Web identity federation into Google, Facebook, Amazon, etc.
IAM roles issue keys are valid for short durations, making them a more secure way to grant access.
An IAM user needs to be granted two distinct permissions to launch EC2 instances with roles:
- Permission to launch EC2 instances.
- Permission to associate an IAM role with EC2 instances.
STS returns:
- A Security Token
- An Access Key ID
- A Secret Access Key
More security
- egress rules on your Security Groups (after all there’s no reason ever that your database should be connecting to IP addresses in Russia),
- vulnerability scanning,
- Host-Based Intrusion Detection (HIDS) systems
Encrypt AWS Credentials
Use my shell script to log into AWS by decrypting credentials stored securely (instead of in plain text).
One reason to encrypt credentials is because it’s wise to have a backup copy of the secret file, in an encrypted format, somewhere else. This enables you to retrieve secrets in case you lose your laptop.
This article covers use of AWS (Amazon Web Services) on MacOS. In the future I’ll be updating this article to cover use of Windows and other secret-handling utilities (Microsoft Azure, Google Cloud Platform, HashiCorp Vault, Akeyless, etc.).
After obtaining an AWS Access Key ID, AWS Secret Access Key for your account (described above), use the credentials on your local machine (laptop), install the AWS CLI locally. Although there is a “awscli” Homebrew formula, but it has been deprecated. So follow this doc to manually install a pkg file for awscli2:
Installing, updating, and uninstalling the AWS CLI version 2 on macOS
AWS CLI versions 1 and 2 use the same aws command name.
If you have both versions installed, your computer uses the…docs.aws.amazon.com
The installer automatically creates a symlink in a folder in your PATH which links to the main program in the installation folder you chose:
ls -al $(which aws)
If you see a response such as this:
-rwxr-xr-x 1 wilsonmar staff 830 Jul 21 09:07 /usr/local/anaconda3/bin/aws
-
Verify install:
aws --version
A sample response (at time of writing):
aws-cli/1.20.3 Python/3.7.3 Darwin/18.7.0 botocore/1.21.3
QUESTION: Why does the pkg say “1.20.3”?
-
Amazon documentation says to run:
aws configure
That command prompts acceptance or override of default AWS ACCESS KEY ID, AWS SECRET ACCESS KEY, and region saved as a plain-text file at
~/.aws/credentials
Sample contents:
[default] aws_access_key_id = ABCDEFGHIJKLMNOPQRST aws_secret_access_key = 123456786iJsvzQbkIlDiFtBh6DrPzIw8r7hVb35 [py-ec2–1] aws_access_key_id = ABCDEFGHIJKLMNOPQRST aws_secret_access_key = 123456782Nwk156aPF0SxZ8KGY+RrhEbq3AIHUSS
BTW Progress toward AWS providing a more secure approach is at https://github.com/aws/aws-sdk/issues/41
Meanwhile, to avoid having credentials in clear text, store them in encrypted form:
-
Install GPG locally using my instructions at
-
Generate encrypted file “credentials.gpg” from file “credentials”. See:
https://wilsonmar.github.io/git-signing/#bonus-encrypting-whole-files-using-gpg
-
To be able to retrieve secrets in case you lose your laptop, for backup make a copy of the secret file in encrypted format, somewhere else.
-
Make a backup of GPG keys somewhere else (in a key vault) so you can decrypt. One way is to store your private key in a Yubikey USB chip you plug into your laptop.
-
Using the GPG private key, encrypt the aws/credentials file to a new credentials.gpg file also in the same ~/.aws folder.
-
Delete the file at ~/.aws/credentials
-
Download my shell script:
curl "https://raw.githubusercontent.com/wilsonmar/DevSecOps/main/bash/awslogin.sh" -o "awslogin.sh"
NOTE: It works similar to https://github.com/99designs/aws-vault, but with no external dependencies (other than GPG). However, aws-vault supports several vaulting backends.
-
Run the script to login based on the encrypted credential.gpg file:
source ~/awslogin.sh
Alternately, run the script to use the “susan” profile defined:
source ~/awslogin.sh -p susan
The script unencrypts the gpg file, invokes aws login, then removes the unencrypted file.
BONUS: To parse variables from within an AWS credentials file, consider: GitHub - whereisaaron/get-aws-profile-bash: Fetch AWS keys and secrets from ~/.aws/credentials…
This is a pure bash script that can parse and extract AWS credentials (key id and secret) from a ~/.aws/credentials…github.com
If you use it, remember to clear out variables after usage, so they don’t linger in memory.
References
This is adapted from what is in Amazon’s Getting Started tutorials.
TODO: Put each AWS CLI command in a script at https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1 by Lee Harding
https://aws.amazon.com/cli/
More on Amazon
This is one of a series on Amazon:
- AWS Cloud Services Comparisons
- AWS Well-Architected Cloud
- AWS Cloud Services
- AWS IAM
- AWS CLI
- AWS On-boarding (GUI, CLI, API)
- AWS Security
- AWS Data Tools
- AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
- AWS server deployment options
- AWS CDK
- Build load-balanced servers in AWS EC2
- AWS Networking
- AWS Xray
- IoT on AWS
- AWS Lambda
- AWS Lambda
More on DevOps
This is one of a series on DevOps:
- DevOps_2.0
- ci-cd (Continuous Integration and Continuous Delivery)
- User Stories for DevOps
- Git and GitHub vs File Archival
- Git Commands and Statuses
- Git Commit, Tag, Push
- Git Utilities
- Data Security GitHub
- GitHub API
- Choices for DevOps Technologies
- Pulumi Infrastructure as Code (IaC)
- Java DevOps Workflow
- AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
- AWS server deployment options
- Cloud services comparisons (across vendors)
- Cloud regions (across vendors)
- Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
- Azure Certifications
- Azure Cloud Powershell
- Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
- Azure Networking
- Azure Storage
- Azure Compute
- Digital Ocean
- Packer automation to build Vagrant images
- Terraform multi-cloud provisioning automation
-
Hashicorp Vault and Consul to generate and hold secrets
- Powershell Ecosystem
- Powershell on MacOS
- Jenkins Server Setup
- Jenkins Plug-ins
- Jenkins Freestyle jobs
- Docker (Glossary, Ecosystem, Certification)
- Make Makefile for Docker
- Docker Setup and run Bash shell script
- Bash coding
- Docker Setup
- Dockerize apps
- Ansible
- Kubernetes Operators
- Threat Modeling
- API Management Microsoft
- Scenarios for load
- Chaos Engineering
More on Security
This is one of a series on Security in DevSecOps:
- Security actions for teamwork and SLSA
- Code Signing on macOS
- Git Signing
- GitHub Data Security
- Azure Security-focus Cloud Onramp
- AWS Onboarding
- AWS Security (certification exam)
- AWS IAM (Identity and Access Management)
- SIEM (Security Information and Event Management)
- Intrusion Detection Systems (Goolge/Palo Alto)
- SOC2
- FedRAMP
-
CAIQ (Consensus Assessment Initiative Questionnaire) by cloud vendors
- AKeyless cloud vault
- Hashicorp Vault
- Hashicorp Terraform
- SonarQube
- WebGoat known insecure PHP app and vulnerability scanners
- Security certifications
- Quantum Supremecy can break encryption in minutes
- Pen Testing
- Threat Modeling
- WebGoat (deliberately insecure Java app)
https://www.mssqltips.com/sqlservertip/5997/create-sql-server-notebooks-in-azure-data-studio/