Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Distribute load to scale and upgrade servers with no down time. Also route transactions across clouds using ZeroLB

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

The contribution of this article is a deep yet succinct hands-on step-by-step tutorial which is logically sequenced, with “PROTIP” marking hard-won, little-know but significant facts available no where else on the internet.

Why - Who Needs Load Balancing?

Server load balancing scales “horizontally” to accomodate higher volume work by distributing (balancing) client requests among several servers. This allows more servers (hosts) to be added to handle additional load.

Even if a single server can handle current load, it is still a good idea to design and test your system to use a load balancer to distribute load so that the cluster stays up.

Having your applications on more than one machine allows operations personnel to work on one machine while the other is busy working.

Having “fault resilience” means that an alternative server can take over when, inevitably, a server crashes or just needs maintenance work done.

This is called ensuring uninterrupted continuous avialability of mission-critical applications.

It’s also a good idea to test your system for its ability to handle growth when it’s still fresh on developer’s minds, since load balancing and fail-over issues can be quite complex.

The oldest load balancers operated at network level 4 (L4), which route based on IP address.

Newer “application” load balancers, operating at network level 5 (L7), allocate traffic based on the URL, such as routing to web vs. API servers.

The Trouble with Load Balancing

Load balancers may not evenly distribute load among active machines in the cluster. This can happen for several reasons:

Session stickiness. Once the IP of a client gets assigned to a particular server (at Login), it stays with that server until the session ends. This is the case with SAP SAPGUI R/3 application performance tuning

Session stickiness is inherently fragile because the backend hosting the session may die.

Also, since each session poses different levels of load, load balacers which allocate based on frequency of allocation (such as round-robin algorithm) may over-allocate sessions to a server which ended up with users and sessions who consume a higher-than-average amount of resources.

Algorithms for allocation/splitting of traffice include:

  • round robin
  • least request
  • random
  • ring hash
  • maglev

PROTIP: Monitor in real-time the load on each individual resource in a pool. Also measure the extra time load balancers take.

Use of a load balancer takes an extra hop in the network, which adds latency.

Global Load Balancing in Service Mesh

Kong Mesh claims a reduction in “network latency by 2x on request lifecycles and 4x on full roundtrips” plus “zone aware load balancing to reduce egress costs” across multi-cloud Kubernetes clusters, VMs, and bare metal.

Such progress can be made with new “decentralized” applications using the Service Mesh> architecture pattern working within Kubernetes, which deploys (bootstraps) intelligent proxies as “sidecars” running in each pod alongside each app container.

Decentralized microservices apps Dockerized for running in containers make all network communication through its sidecar proxy (like handing a box to UPS to deliver).

BLOG: loadbal-sidecar-700x324

Each sidecar proxy can communicate with backends in different zones (generically named A, B, and C in the diagram). Policies sent to each sidecar can specify zones and a different amount of traffic be sent to each zone. Each zone can be in different clouds (thus multi-cloud).

This approach also enables security-related policies to be applied (such as limiting outflows from some countries) and detection of zonal failure and automatic rerouting around traffic anomalies (including DDoS attacks).

Each sidecar proxy and backend service report periodic state to the global load balancer (GLB) so it can make decisions that take into account latency, cost, load, current failures, etc. This enables centralized visualizations engineers use to understand and operate the entire distributed system in context.

This means app developers no longer need to bother coding for a long list of operational cross-cutting concerns:

  • collection and reporting of telemetry (health checks, logs, metrics, traces)

  • TLS termination (SSH key handling)
  • Handle protocols HTTP/2, WebSocket, gRPC, Redis, as well as TCP traffic

  • rate limiting (DoS mitigation)
  • timeout and back-out handling when response is not received
  • Circuit breakers

  • Fault injection (for chaos engineering to improve reliability)
  • Enforce policy decisions

  • load balancing
  • Staged rollouts with percentage-based traffic splits

Embedding the above functionality in each app program may provide the best performance and scalability, but requires polyglot coding to implement the library in many languages. It can also be cumbersome to coordinate upgrades of new versions of each library across all services.

There are several sidecar programs:

Logically, communication of packets/requests travel through a “data plane”.

There is also a “control plane” which, rather than exchanging packets/requests, traffic in policies and configuration settings to enable services such as:

  • deploy control (blue/green and/or traffic shifting),
  • authentication and authorization settings,
  • route table specification (e.g., when service A requests /foo what happens), and
  • load balancer settings (e.g., timeouts, retries, circuit breakers, etc.).

Several products provide a “control plane UI” (web portal/CLI) to set global system configuration settings and policies as well as

  • Dynamic service discovery
  • certificate management (acts as a Certificate Authority (CA) and generates certificates to allow secure mTLS communication in the data plane).
  • automatic self-healing and zone failover (to maximize uptime)

Several control plane vendors compete on features, configurability, extensibility, and usability:

  • IstioD

  • open-sourced Nelson uses Envoy as its proxy and builds a robust service mesh control plane around the HashiCorp stack (i.e. Nomad, etc.).

  • Kong Mesh (part of the Konnect Connectivity Platform) makes use of Envoy

  • SmartStack creates a control plane using HAProxy or NGINX.

References:

  • https://www.prnewswire.com/news-releases/kong-introduces-zerolb-a-new-load-balancing-pattern-using-service-mesh-301352046.html
  • https://thenewstack.io/zerolb-a-new-decentralized-pattern-for-load-balancing/

Load Balancing in AWS

Load Balancing is covered in the $300 AWS Certified Advanced Networking - Speciality certification exam (65 questions in 175 minutes). Video courses:

VIDEO: https://aws.amazon.com/elasticloadbalancing

Load Balancers need a public DNS entry (Route53).

Elastic Load Balancing offers four types of load balancers that all feature the high availability, automatic scaling, and robust security necessary to make your applications fault tolerant.

  • Classic Load Balancer provides basic load balancing across multiple Amazon EC2 instances and operates at both the request level and the connection level. Classic Load Balancer is intended for applications that were built within a EC2-Classic network. Charges are in Load Balancer Capacity Units (LCU) used per hour plus $0.008 per GB of data processed (the only type to charge by network). More expensive to wean users off legacy. Each port handled by a Classic Load Balancer needs a separate load balancer setup:

    aws-lb-classic-ports

  • Application Load Balancer distributes HTTP and HTTPS traffic and provides advanced request routing targeted at microservices and containers. Application Load Balancer routes traffic to targets within Amazon VPC based on the content of the request. Charges are in Charges are in Application Load Balancer Capacity Units (ALCU) used per hour.

  • Network Load Balancer distributes Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Transport Layer Security (TLS) traffic where extreme performance is required. Network Load Balancer routes traffic to targets within Amazon VPC and is capable of handling millions of requests per second while maintaining ultra-low latencies. Charges are in Network Load Balancer Capacity Units (NLCU) per hour.

  • Gateway Load Balancer makes it easy to deploy, scale, and run fleets of third-party virtual networking appliances (such as F5) for security, network analytics, etc. Gateway Load Balancer is transparent to the source and destination of traffic. Charges are in Gateway Load Balancer Capacity Units (GLCU) per hour. Gateway Load Balancer uses Gateway Load Balancer Endpoint (GWLBE), a new type of VPC Endpoint powered by AWS PrivateLink technology that simplifies how applications can securely exchange traffic with GWLB across VPC boundaries. GWLBE is priced and billed separately.

ECS LB Type: Classic LB App LB Network LB Gateway
Pricing: LCU ALCU NLCU GLCU
Layer: 4 & 7 Layer 7 Layer 4 4 & 7
Protocols: HTTP/S TCP HTTP/S/2 TCP TLS UDP HTTP/S
$ Per hour: $0.025 $0.0225 $0.0225 $0.0125
Web Sockets: - Yes - -
Static EIP: - - Yes -
X-Forward headers: Yes Yes - -
Proxy TCP: Yes - Yes -
VPC IPv6: - Yes Yes -
AWS Privatelink: - - Yes -
Security Groups: Yes Yes - -
Containers: - Yes Yes -

Only ALB handles:

  • HTTP host header
  • HTTP method
  • Source IP address or CIDR
  • Arbitrary HTTP header
  • HTTP query string parameter
  • Combined rules
  • AWS WAF
  • Weighted Target Group routing for A/B testing, migrations, gradual Blue/Green deployments

Config steps: aws-lb-blue-green

Use aws elbv2 CLI scripts

AWS Networking

AWS WAF up front for DDoS protection.

Define separate subnets. Scheme: internet-facing

Scheme: internal: In Security group of targets in a private subnet, only allow inbound traffic only from the load balancer (in a specific security group ID).

aws-lb-subnets

https://docs.aws.amazon.com/autoscaling/ec2/userguide/autoscaling-load-balancer.html

AWS Distribution Rules

VIDEO: One rule for each app group, defined by up to 100 rules:

aws-lb-rules-paths

To avoid impacting app servers used by humans, route automated spiders to app servers not used by humans.

aws-lb-rules-config

VIDEO: ALB Ingress Controller for Kubernetes routes requests to different services in the cluster based on host, path, or other advanced request routing feature.

There is a 1 MB limit on Lambda request and response bodies. Lambda functions and ALB must be in the same AWS account.

Billing metrics

VIDEO:

Workload Metric: ConsumedLCU is based on the highest among metrics during each given hour:

  • RequestCount (new requests per second)
  • Active connections
  • ProcessedBytes (payload GB per second) bandwidth

Connections:

Target metrics: connection errors and Request time (latency)

AWS Health checks

External (Route 53) checks health of each ELB IP from the world. Failing IP health checks are removed from DNS.

Target health checks before sending requests.

AWS Routing Algorithms

Classic LBs route HTTP/S based on “Least outstanding requests” and TCP/TLS(SSL) using “round robin” algorithms.

NLBs routes based on “flow hashing”.

AWS Config

Name each LB running (as EC2 instances) with prefix “LB-“:

aws-lb-instance-config

No SSH and Security Key Pair (idempotent).

In Advanced, specify an Apache shell script to run on start to create a folder for what it processes, then show a message:

#!/bin/bash
yum install httpd -y
systemctl enable httpd
mkdir /var/www/html/orders
echo "<h1>Orders LB here</h1>" > /var/www/html/orders/index.html
systemctl start httpd

PROTIP: For “Zero Trust” strategy, do not have LB handle HTTPS termination internally - use HTTPS throughout. Redirect HTTP to HTTPS.

aws-lb-target-adv

AWS “Elastic Load Balancing” distributes incoming application traffic across multiple registered targets such as Amazon EC2 instances, containers, IP addresses, Lambda functions, and virtual appliances.

aws-lb-target-config-369x323

LBs can handle application traffic in a single or multiple Availability Zones.

SNI (Server Name Indication) can send different certs to different clients, with automatic cert renewals by ACM (Amazon Cert. Manager).

Apps need to add back-off on failure (up to a point of abandonment).

Algorithms for routing

There are several basic types of load balancing:

Round robin DNS (RRDNS) involves client DNS servers to resolve a URL to multiple ip addresses (and thus multiple machines).

The problem with “round robin” is that it blindly hits each server regardless of its ability to accept work. This scheme requires every server to be homogeneous (nearly identical).

RRDNS is not recommended because it places load balancing outside the organization to authoritative DNS servers which caches DNS A entries. Changes to those entries can take a long time to propagate throughout the internet.

Network-based load balancers offer more sophisticated allocation algorithms. Network Load Balancing operates at the NIC driver level to detect the failure of a server and reassigns client computer traffic among the remaining servers.

Microsoft’s NLB service can take up to eight seconds to redirect load. How To Set Up TCP/IP for Network Load Balancing in Windows Server 2003 notes that

TCP/IP must be the only network protocol present on the cluster adapter. You must not add any other protocols (for example, Internetwork Packet Exchange [IPX]) to this adapter.

Because traffic enters the network, network load balancing cannot withstand moderate Dos (Denial of Service) attacks as well as large-capacity front-end network appliances which detect ad filter out malicious traffic before it gets to the server.

Dispatcher or proxy / switch based load balancers resolves requests from clients directed to a single virtual IP address (VIP) . The load balancer then uses a load balancing algorithm to distribute work to real IP addresses.

Such “server-based” load balancing is done either by an application running on a regular server or an appliance.

    * Using regular PC machines (on standard operating systems such as Windows or UNIX) has the advantage that it can be replaced (after configuration) with another machine that is familiar to the staff. There is less hassle from dealing with another vendor. tool Vendors of such an approach include Resonate, Rainfinity, and Stonebeat. * Using dedicated appliance hardware has the advantage of speed, since they use specialized Application Specific Integrated Circuit (ASIC) chips. They "have internal-bandwidth backbones capable of handling a Gbps worth of traffic."[2,p31] tool Vendors of such an approach include F5, and Radware. "Firewall load balancers tend to max out at around 70 to 80 Mbps"[2,p9,60] BigIP from f5 also compresses (for HTTP 1.1), encrypts SSL network traffic, and centralizes the SSL key store onto a single machine.

However, an individual central dispatcher can become a single point of failure. So:

A hot spare is a machine that’s configured as a mirror image of the machine it replaces if that fails. It’s also called a passive node if it sits unused until it’s need to support a failover.

Both the active and standby load balancers occassionally send out heartbeat messages such as VRRP (Virtual Router Redundancy Protocol, RFC 2338) format UDP packets to multicast IP 225.0.0.2 port 1985. (in the range from 224.0.0.0 to 239.255.255.255).

Cisco has its HSRP (Hot Standby Routing Protocol) and Extreme Networks has its ESRP (Extreme Standby Router Protocol).

This communication can occur over a serial cable between two machines.

Multicast can span several subnets, but they must reach their final destination after a network latency of no more than 200 to 300 milliseconds.

The passive node takes over (assumes “master” status) when it does not hear the “I’m alive” heartbeat from the other machine.

Because failover can cause a VIP to be the destination address at different devices at different times, a VIP can be called “floating”.

Failback occurs when the failed server comes back online and takes load back from the failover node. This happens after a transfer of client state information.

To avoid “bridging loops” when each load balancer thinks the other is inactive, the ISO Level 2 Spanning-Tree Protocol (STP) is used to set a cost for each port. This provides a way to block multiple paths by opening only the lowest-cost port (the highest-priority port). However, “STP is almost never used since it can take 10 seconds or more to react. Typically, a proprietary variation of a hot-standby protocol is used.”[2,p59]

CAUTION: Proxies forward both client requests to servers and server responses to clients, so unless they can handle as much throughput as all the machines in its subnet, it can easily become a throughput bottleneck.

Note: Although not necessary for production work, “Individual Pass-Though VIPs” are requested/defined so that for performance measurement and troubleshooting, individual servers can be reached through the load balancer.

Mechanisms

There are several distinguishing features among load balancers:

“NAT-based” IP firewall.

“Flat-based” IP typology means the VIP and real IPs are on the same subnet (usually behind a firewall). FTP and streaming applications use this typology along with Return Path DSR.

Many load balancers also have firewall capabilities (packet filtering, stateful inspection, intrusion detection, etc.)

More advanced load balancers enable the VIP to be on a different (more private/secure) subnet than IPs on servers receiving traffic. In this case, the load balancer acts as a gateway between the two LANs.

This is achieved with NAT (Network Address Translation). (RFC 1631) which enables real servers to hide themselves as non-routed RFC 1918 addresses.

But this only allows one pair of load balancer machines.

Load allocation algorithms within daemons (or windows services), which dynamically:

1) listens for a “heartbeat”; or
2) periodically sends ICMP ECHO_REQUEST to measure RTT (Round Trip Time) “pings”; or
3) detect the status of bandwidth usage or CPU utilization on each server; or
4) counts the number of connections active on each server. </ul>

BEA Loogic 8.1 is server affinity that reduces the number of RMI sockets.

SAP R/3’s login balancing (SMLG) clustering technology makes the decision of which server receives a user for login based on the number of users currently logged in and response time, (not how many work processes are used at any moment).

Once a SAP user is logged on a particular app server, the user session is pinned to that server. Background jobs are also assigned to a server, but a new job might get assigned to another server.

failover automatic repartioning of client traffic among the remaining servers in the cluster so that users experience continuous performance Quality of Service (QoS).

Within the BEA WebLogic 8.1 HttpClusterServlet Plug-in, sessions are attached to a particular server by the client’s IP address. This means that a server must stay up until all sessions finish (which can take many minutes).

A failover mechanism must have an awareness of the progress of all activities so that client sessions can continue to complete if processing stops.

IP Multicasts are sent by each WebLogic server to broadcast their status. Each Weblogic server listens for the one-to-many messages to update its own JNDI tree.

Cisco’s LoadDirector uses a “Cookie Sticky” feature that redirects traffic to the same physical machine by examining cookies.

“Virtualization” software (such as EMC VMware) automatically installs application software to meet demand.

* <strong>Global load balancing</strong> that routes requests among several data centers
over the WAN (Wide Area Network). "With cross-country latency at around 60 ms or more" [2,p10]    * Intelligent response to denial of service flooding attacks.

Direct Server Return of traffic back to clients

The least sophisticated load balancers use “Route-path”, where the load balancer acts as a Layer 3 router. But this allows 3 or more load balancers.

More sophisticated load balancers use “Bridge-path”, where the load balancer acts as a Layer 2 bridge between two LANs. However, this only works with flat-based nets.

Most sophisticated of all are when real servers (especially streaming and FTP servers) can bypass going back to the Load Balancer and send responses directly to clients by using Direct Server Return (DSR). This is desirable because “web traffic has a ratio of about 1:8, which is one packet out for every eight packets in.”[2,p26]

F5 calls this “nPath”. Foundry calls it “SwitchBack”.

DSR works by configuring the IP alias on the server’s loopback interface “lo” with the IP address of the VIP. This is done using the ifconfig command on Unix.

The server needs to still bind to the real IP address as well so the load balancer can perform health checks on the sever.

The default route path of the server needs to point to the router rather than the load balancer.

??? using MAC Address Translation (MAT)

<strong> "Sticky" sessions </strong>

Load Balancing in the Cloud

The internal processes (life-cycle) of a "redundant, self-curing, and self-scaling" computing facility are

  1. Authorizing those who need to perform various activities with rights for their role (who can do what when)
  2. Imaging os and software apps into server snapshot images templates used to create working instances
  3. Provisioning (condensing) instances (with unique URLS and OS level settings)
  4. Configuring each instance (with unique application-level settings) [Puppet]
  5. Persisting data in and out each instance (with our without encryption)
  6. Storing logs eminating from instances into a centralized location
  7. Monitoring health and other metrics by instance type
  8. Deciding when alerts are appropriate, when an instance should be registered or un-registered with its load balancer, when instances should be added or removed
  9. Switching among instances from a static IP address [Sclar]
  10. De-Provisioning (evaporating) instances
  11. Analyzing logs for trends (cost per transaction over time) and implications to adjust number and size of instances most appropriate for each instance type/purpose and time frames) [capacity management like vkernel] </ol>
  12. Sclar (a $50/month open-source offering from Intridea, a Washington DC Ruby on Rails development firm) illustrates its approach for a polling controller this way: ## MS NLB Services After Microsoft acquired, in 1998, from Valence Research the "Convoy Cluster" component of Windows NT 4.0 Enterprise Edition, Microsoft introduced its Network Load Balancing (NLB) service as a free add-on service to Advanced and Datacenter versions of Windows 2000 Server (known as Enterprise Edition of Windows 2003). The acronymn for Windows Load Balancing Service (WLBS) is also the name of the utility which verifies whether load balanced hosts "converge":
      wlbs query
      wlbs stop
      wlbs start
    Unlike appliances which sit in front of servers in the cluster, MS-NLB works inside a server as a driver that intercepts traffic from the NIC card. NLB is "fully distributed" in that it is on all host servers serving a web site on a single subnet. Like other load balancers, clients make requests using a single primary virtual IP address (VIP) for the cluster. However, all incoming IP packets are received by the cluster adapter of each host because NLB assigns listening cluster adapter on each host with the same MAC address based on the VIP. One of the hosts on the public subnet sends a MAC-layer (Layer 2) multicast to hosts on the local broadcast subnet of the cluster through another adapter. These are received by a separate dedicated IP adapter on each host. The NLB service runs on every host as an intermediate driver between the TCP/IP (HTTP & FTP) protocol and network adapter drivers within the Windows 2000 protocol stack. The service ignores unwanted packets and doesn't reroute packets to individual hosts. Because potentially any host on the cluster can respond to a client, a Web browser may receive various images within a single Web page from different hosts on a load-balanced cluster. The host that responds to a request is the host which know that it's the default host. This is determined by the host priority number unique to every host. That number is in the range of 1 to 32 because that's the NLB product's limit on hosts per cluster. Each NLB server emits a heartbeat message to other hosts in the cluster, and listens for the heartbeat of other hosts. A host knows it's the default host if it doesn't hear another host with a higher priority. This allows any host to be taken offline (for preventive maintenance, upgrading, etc.) without disturbing cluster operations. Each host responds to requests through its own unique dedicated IP address so that other hosts know the source of the request. Traffic arriving through a cluster adapter is redirected to the dedicated IP address of a specific host. The dedicated IP address is always entered first on the list of IPs so that outgoing connections from the cluster host are sourced with this IP address instead of a virtual IP address. The load-balancing server uses its algorithm to choose the best available server for the client. To allow session state to be maintained in host memory, NLB directs all TCP connections from one client IP address to the same cluster host. This behavior is controlled by the client affinity parameter. Changing load percentages allow hosts with higher capacity to receive a larger fraction of the total client load. NLB doesn't take advantage of Win2K's performance information for load balancing. MS-NLB doesn't support delayed binding. Each server provides failover for every other server, thus the software provides load-balancing redundancy in an active-and-active implementation. ### MS NLB Configuration Netlibrary eBook $35 Windows 2000 & Windows Server 2003 Clustering & Load Balancing (McGraw-Hill Osborne: April 9, 2003) by Robert J. Shimonski NLB capabilities come built-in with high-end Windows servers. Use either the NIC properties page or the "Administrative Tools\Network Load Balanacing Manager" tool. A. After right-clicking "My Network Places", select "Properties". 1. In the "Networking Connections" dialog: 1.Rename "Local Area Connection" to your name for the private interface for NLB heartbeat messages. 1. Rename "Local Area Connection2" to your name for the public interface. Select the "Network Load Balancing" check box. In the "Network Load Balancing Properties" dialog box, click the "Cluster Parameters" tab. Enter the public IP address in the Cluster IP configuration area. (This should be the same for all nodes in the cluster.) Enter the Fully Qualified Domain Name (FQDN) and network address handling incoming client requests. To verify whether the hosts "converge":
      wlbs query
    B. To use the "Administrative Tools\Network Load Balanacing Manager" wizard: * Add a new cluster. The cluster's IP is the one routing outside traffic to (10.38.24.x), enter your mask, and the DNS name. Select multicast. Click Next. * Add additional NLB IPs (you can always do this later). Click Next. * Setup port rules. click next. * Enter the IP for the first host. Usually, this is the machine you are currently logged into. This CAN be either the "outside" or "inside/heartbeat" IP. I just stick with the heartbeat IP. select the heartbeat IP from the list. Next. * Select "1" as the priority. Click Finish. * Add a host to the cluster. Enter the info for the second host. Click Finish. * Wait a few minutes while the second host goes from "misconfigured" to "converged" status. * Click the "refresh" button or Run command: nlb query </ol> </ol> Click the "Host Parameters" tab to supply the dedicated IP address and subnet mask for the private interface. On the first NLB node you�re configuring, leave the Priority (unique host identifier) value at the default of "1". On subsequent nodes, set the priority value to the next highest unused number. To stop an individual host:
      wlbs stop
    To start a host:
      wlbs start
    ## SAP ITS Load Balancing To activate SAP's ITS integrated service with load balancing ABAP on WAS 6.40: * Set up the ICF Service Server Group. * Invoke tcode SICF. * Select the ICF Service. * Under Service Option enter the Server Group. * Maintain Load Balancer settings using transaction ICM. ## More on DevOps # This is one of a series on DevOps: 0. [DevOps_2.0](/devops_2.0/) 0. [ci-cd (Continuous Integration and Continuous Delivery)](/ci-cd/) 0. [User Stories for DevOps](/user-stories-for-devops/) 0. [Enterprise Software)](/enterprise-software/) 0. [Git and GitHub vs File Archival](/git-and-github-vs-file-archival/) 0. [Git Commands and Statuses](/git-commands-and-statuses/) 0. [Git Commit, Tag, Push](/git-commits/) 0. [Git Utilities](/git-utilities/) 0. [Data Security GitHub](/data-security-github/) 0. [GitHub API](/github-api/) 0. [TFS vs. GitHub](/tfs-vs-github/) 0. [Choices for DevOps Technologies](/devops-choices/) 0. [Pulumi Infrastructure as Code (IaC)](/pulumi/) 0. [Java DevOps Workflow](/java-devops-workflow/) 0. [Okta for SSO & MFA](/okta/) 0. [AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)](/aws-devops/) 0. [AWS server deployment options](/aws-server-deploy-options/) 0. [AWS Load Balancers](/loadbal/) 0. [Cloud services comparisons (across vendors)](/cloud-services-comparisons/) 0. [Cloud regions (across vendors)](/cloud-regions/) 0. [AWS Virtual Private Cloud](/aws-vpc/) 0. [Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)](/azure-cloud-onramp/) 0. [Azure Certifications](/azure-certifications/) 0. [Azure Cloud](/azure-cloud/) 0. [Azure Cloud Powershell](/azure-cloud-powershell/) 0. [Bash Windows using Microsoft's WSL (Windows Subsystem for Linux)](/bash-windows/) 0. [Azure KSQL (Kusto Query Language) for Azure Monitor, etc.](/kql/) 0. [Azure Networking](/azure-networking/) 0. [Azure Storage](/azure-storage/) 0. [Azure Compute](/azure-compute/) 0. [Azure Monitoring](/azure-monitoring/) 0. [Digital Ocean](/digital-ocean/) 0. [Cloud Foundry](/cloud-foundry/) 0. [Packer](/packer/) automation to build Vagrant images 0. [Terraform](/terraform/) multi-cloud provisioning automation 0. [Hashicorp Vault and Consul](/hashicorp-vault/) to generate and hold secrets 0. [Powershell Ecosystem](/powershell-ecosystem/) 0. [Powershell on MacOS](/powershell-on-mac/) 0. [Powershell Desired System Configuration](/powershell-dsc/) 0. [Jenkins Server Setup](/jenkins-setup/) 0. [Jenkins Plug-ins](/jenkins-plugins/) 0. [Jenkins Freestyle jobs](/jenkins-freestyle/) 0. [Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile](/jenkins2-pipeline/) 0. [Docker (Glossary, Ecosystem, Certification)](/docker/) 0. [Make Makefile for Docker](/make-makefile/) 0. [Docker Setup and run Bash shell script](/bash-scripts/) 0. [Bash coding](/bash-coding/) 0. [Docker Setup](/docker-setup/) 0. [Dockerize apps](/dockerize/) 0. [Docker Registry](/docker-registry/) 0. [Maven on MacOSX](/maven-on-macos/) 0. [Ansible](/ansible/) 0. [MySQL Setup](/mysel-setup/) 0. [SonarQube & SonarSource static code scan](/sonarqube/) 0. [API Management Microsoft](/api-management-microsoft/) 0. [API Management Amazon](/api-management-amazon/) 0. [Scenarios for load](/scenarios-for-load/) 0. [Chaos Engineering](/chaos-engineering/)