Wilson Mar bio photo

Wilson Mar

Hello!

Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Privately store and retrieve binary artifacts vetted of vulnerabilities, instead of public DockerHub, PiPI, etc.

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Artifactory stores binary format assets such as executable files from builds, virtual memory (container) images, graphic image files, etc. (rather than textual source code). It caches local binary files as a proxy to public repositories, which make them quicker to obtain and provides a way to provide security-vetted (whitelisted) versions.

Wikipedia categorizes it as a “software package management system”.

Artifactory works with another JFrog product: X-Ray, which identifies vulnerabilities in transitive dependencies calling 3rd-party open-source libraries.

User Guide: https://www.jfrog.com/confluence/display/RTF/Welcome+to+Artifactory

Workflow by edition and license

Artifactory provides traceability of dependencies within components and images built as artifacts are promoted by Helm for use by Kubernetes.

Licensed editions integrates with JFrog Xray which performs “continuous” Security scans and Artifact impact analysis based on JFrog’s database of vulnerabilities. VIDEO

artifactory-flow-1044x546

A Virtual Repository can be defined in Artifactory to access multiple Bower registries aggregated under a single URL.

Both on-prem and cloud SaSS deployments are available.

Artifactory is licensed from JFrog at $14,400 per year for a single site and $29,500 for multiple (enterprise) sites. The money goes to support secure, clustered, High Availability registries.

See https://jfrog.com/open-source

Free OSS Artifactory editions fetch ONLY for transitive package managers (Java) Maven, Gradle, and Apache Ivy. But paid editions provide “universal support” for all major binary package formats:

artifactory-filetypes-1408x470

For Git provider source API to VCS (Version Control System) repositories:

For IaC (Infrastructure as Code) and virtualized images:

For operating systems:

For languages:

  • Maven instead of MavenCentral for Java jar, ear, war dependencies
  • PyPI for Python by pip and conda
  • SBT (Scala Built Tool) can be used for Java
  • NPM (Node Package Manager) for JavaScript
  • yum RPM and RubyGems for Ruby VIDEO
  • PHP Composer instead of Packagist.org
  • Conan for C/C++

File formats:

  • zip
  • tar.gz from Linux machines
  • 7up???

https://github.com/jfrog/project-examples Sample projects for training and testing CI setup with Artifactory

https://github.com/jfrog/jfrog-data-generator Tools for generating and preloading JFrog products with data

https://github.com/jfrog/jfrog-idea-plugin https://jfrog.github.io/jfrog-idea-plugin/

HA (High Availability)

VIDEO Get Ahead of the Curve with JFrog Artifactory Enterprise

Artifactory Enterprise claims an SLA of “Five-nines Availability” and “Near-zero Maintenance Downtime” from Unlimited Server Scalability due to

High Availability (HA) Artifactory configuration consists of a cluster of several (Docker) active/active nodes behind a load balancer. All nodes reference a single MySQL/PostgreSQL database.

The Helm chart (using K8s 1.8+) is at https://hub.helm.sh/charts/jfrog/artifactory-ha

Storage configuration settings are defined in the binarystore.xml file.

Parameter artifactory.service.pool defines a single primary node handles jobs and tasks and not interrupted by inbound traffic. Inbound traffic is received by 2 member nodes which are load balanced.

Disaster Recovery Testing

To prepare for the ability to recover from a disaster, enterprise admins of Artifactory use Replicator to synchronize repositories between remote Artifactory instances.

Test DR.


Preparations

Based on VIDEO: Setup a Docker Registry in 5 minutes

  1. Get and install license keys (perhaps for a ___ day trail).

    Artifactory HA license?

  2. Obtain DNS IP addresses for primary, second, and third nodes, plus load balancer.

  3. Identify a place to store bootstrap files used to create Artifactory server instances.

  4. Define and socialize artifact structure and naming conventions, such as this common four-part naming structure:

    • Product or team name as the primary identifier of artifact ownership
    • Technology, tool or package type being used
    • Maturity level of package (such as development, staging, and release stages)
    • Geographic locator to the physical topology of the artifact

    artifactory-naming-1175x459

    Internally, each artifact is stored using a unique hash generated from the data in the file.

  5. Assign access privileges according to projects or development teams.
  6. Corporate policy on signing.

Installation

VIDEO: Installing JFrog Artifactory 5 High Availability Cluster - The Complete Walk through [9:23]

  • JFrog Mission Control (JFMC) is used to manage deployments
  1. Download the bootstrap bundle tar.gz file or jfrog-artifactory-pro-5.1.0.zip.
  2. Unzip the file.

    The tomcat folder contains a web server.

    See https://www.jfrog.com/confluence/display/RTF/Installing+Artifactory and https://www.jfrog.com/confluence/display/RTF/HA+Installation+and+Setup

  3. Download the mysql-connector-java-5.1.40-bin.jar and copy it to tomcat/lib.

  4. Environment variable $ARTIFACTORY_HOME defines the path to the folder installed.

  5. Edit etc/ha-node.properties to change the default node.id from “art1”.

    primary=true

    Change the url from 192.168.59.186 to the one provided for the server.

    The default port is 10001. This is the same for all nodes.

  6. Copy

      cp ../binarystore.xml etc/binarystore.xml
  7. ./bin/installService.sh installs the server on Linux machines.
    ./bin/installService.bat installs the server on Windows machines.

    The Artifactory setup wizard should pop-up for manual configuration.

    Configuration

    QUESTION: How can this be automated?

  8. Add license key obtained from JFrog.
  9. Set admin password

    PROTIP: Follow your organization’s password policies.

  10. Configure corporate proxy settings.

  11. Create from primary server file boostrap.bundle.tar.gz in folder etc:

    curl -uadmin -X POST http://localhost:8081/artifactory/api/system/bootstrap_bundle
  12. Store the bootstrap in Artifactory (I’m kidding because it doesn’t exist yet).
  13. SCP to second and 3rd nodes.

  14. Edit etc/ha-node.properties on the 2nd node.

    primary=false
  15. Repeat the above for the 3rd node.
  16. [9:07] Store a new artifact.
  17. Do “chaos engineering” by brining down a node to verify recovery.
  18. Practice recovery from backup.

    Enable GPG Signing

    1. GPG Signing of components See https://www.jfrog.com/confluence/display/RTF/GPG+Signing

CI/CD Pipeline

[Webinar] JFrog Artifactory - The Easy One Sep 13, 2018 Artifactory Query Language (AQL)

Artifactory stores metadata in a relational database.

Jfrog API

https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API https://github.com/jfrog/artifactory-scripts/tree/master/REST-API-Examples

https://github.com/jfrog/artifactory-scripts Scripts for Artifactory (Usually, for REST API), community driven.

JFrog CLI on client

JFrog CLI optimizes both upload and download operations by skipping artifacts that already exist in their target location.

  1. Instead of using installer from https://jfrog.com/getcli/

    brew install jfrog-cli-go

    The response reflects use of Artifactory’s bintray.com, JFrog’s Software Distribution as a Service (instead of Acakamai CDN, etc.):

    ==> Downloading https://homebrew.bintray.com/bottles/jfrog-cli-go-1.30.4.mojave.
    ==> Downloading from https://akamai.bintray.com/0f/0f348458539931885a3e0e6e90553
    ######################################################################## 100.0%
    ==> Pouring jfrog-cli-go-1.30.4.mojave.bottle.tar.gz
    🍺  /usr/local/Cellar/jfrog-cli-go/1.30.4: 5 files, 18.2MB
    

    See https://www.jfrog.com/confluence/display/CLI/JFrog+CLI for different authentication mechanisms that can be used.

  2. Verify:

    jfrog --version
    jfrog version 1.30.4

    Command syntax, such as:

  3. View command help by the CLI alone:

    jfrog
    NAME:
    jfrog - See https://github.com/jfrog/jfrog-cli-go for usage instructions.
     
    USAGE:
    jfrog [global options] command [command options] [arguments...]
     
    VERSION:
    1.30.4
        
    COMMANDS:
    rt          Artifactory commands
    bt          Bintray commands
    mc          Mission Control commands
    xr          Xray commands
    completion  Generate autocomplete scripts
    help, h     Shows a list of commands or help for one command
        
    GLOBAL OPTIONS:
    --help, -h     show help
    --version, -v  print the version
        
    Environment Variables:
      JFROG_CLI_LOG_LEVEL
     [Default: INFO]
     This variable determines the log level of the JFrog CLI.
     Possible values are: INFO, ERROR, and DEBUG.
     If set to ERROR, JFrog CLI logs error messages only.
     It is useful when you wish to read or parse the JFrog CLI output and do not want any other information logged.
     
      JFROG_CLI_OFFER_CONFIG
     [Default: true]
     If true, JFrog CLI prompts for product server details and saves them in its config file.
     To avoid having automation scripts interrupted, set this value to false, and instead,
     provide product server details using the config command.
     
      JFROG_CLI_HOME_DIR
     [Default: ~/.jfrog]
     Defines the JFrog CLI home directory path.
     
      JFROG_CLI_TEMP_DIR
     [Default: The operating system's temp directory]
     Defines the temp directory used by JFrog CLI.
     
      JFROG_CLI_BUILD_NAME
     Build name to be used by commands which expect a build name, unless sent as a command argument or option.
       
      JFROG_CLI_BUILD_NUMBER
     Build number to be used by commands which expect a build number, unless sent as a command argument or option.
     
      JFROG_CLI_BUILD_URL
     Sets the CI server build URL in the build-info. The "jfrog rt build-publish" command uses the value of this environment variable, unless the --build-url command option is sent.
       
      JFROG_CLI_ENV_EXCLUDE
     [Default: *password*;*secret*;*key*;*token*] 
     List of case insensitive patterns in the form of "value1;value2;...". Environment variables match those patterns will be excluded. This environment variable is used by the "jfrog rt build-publish" command, in case the --env-exclude command option is not sent.
     
      CI
     [Default: false]
     If true, disables progress bar on the supporting commands.
    
  4. Edit your ~/.bash_profile to define environment variables to configure.

    Alternately, be prompted:

    jfrog rt ping --url=http://my-rt-server.com/artifactory
    To avoid this message in the future, set the JFROG_CLI_OFFER_CONFIG environment variable to false.
    The CLI commands require the Artifactory URL and authentication details
    Configuring JFrog CLI with these parameters now will save you having to include them as command options.
    You can also configure these parameters later using the 'config' command.
    Configure now? (y/n): 
    

Helm charts

https://hub.helm.sh/charts/jfrog/artifactory-ha

Add jfrog repository:

helm repo add jfrog https://charts.jfrog.io

Install helm chart:

helm install jfrog/artifactory-ha –version 1.1.9

Regular server start-up

  1. artifactory.sh starts the server on Linux machines.
    artifactory.bat starts the server on Windows machines.

    Access and Authentication

    LDAP Authentication

    Role-based authorization with teams and permissions

    Active Directory, Atlassian Crowd and JIRA, OAuth (multiple providers)

    Licensed versions can use SSO integration for NTLM, Kerberos, Etc.

  2. Focused Email Notifications for Artifact Changes

Alternately, install using Docker on CentOS


AWS Cloud

Integrating JFrog Artifactory with AWS CodePipeline by Erin McGill | on 24 MAY 2018

artifactory-aws-746x414

Google Cloud

Artifactory can be used to create a hybrid solution by being a repository to bridge between Google Cloud Platform (GCP) and Google Kubernetes Engine (GKE)CI/CD pipeline on-prem, called Cloud Pro X.

Deploying JFrog Artifactory SaaS on Google Cloud Platform by JFrog Solutions Engineers Doron Meirfeld and Mansirman Singh

JFrog-artifactory-700x700


Migration from Docker Registry

https://github.com/jfrog/docker2artifactory to transition from V2 Docker registries to JFrog Artifactory. by Arturo Aparicio in San Jose, California.

Git LFS

https://github.com/jfrog-aparicio/git-lfs

User actions on sample repositories

All editions can search by Name, Archive, Property or Checksum values.

Artifactory Query Language (AQL) to extract data.

Annotate Artifacts with Searchable Properties Aggregate and Run Bulk Operations on Search Results

Move/copy/delete artifacts through the UI

https://github.com/jfrog/artifactory-user-plugins/tree/master/cleanup

Repository Replication

Paid editions of Artifactory can be replicated for redundancy.

Enterprise users can use Multi-push Replication

Web Start Application Hosting

Network Proxy setup

Define a network proxy to deliver vetted installers instead of public versions.


Usage from Build tools

GoCD, TFS, Azure DevOps, AWS OpsWorks, etc.

Create artifacts

On Demand Jar Signing are available to paid editions of Artifactory.

API

Paid editions of Artifactory can extend Artifactory with Groovy-based User Plugins.

JFrog X-Ray

JFrog’s Xray (2.X up) and scans binaries in Artifactory. Scanning references a vulnerability database that the local server downloads from JFrog. The database is about 16 GB and needs to be monitored for growth over time.

https://www.jfrog.com/confluence/display/XRAY/Installing+Xray

To scan packages, Xray downloads the entire artifact to its local disk So the Xray server needs a lot of disk space. At least 100 GB, or 200 GB to start.

The recommend Xray server minimum hardware is 8 cores CPU and 16 GB RAM.

https://support.jfrog.com/jFrogKnowledgePage#id=kA01r000000LwtmCAC Basic troubleshooting section at the end for the most commonly seen errors.

The JFrog VS Code Extension adds JFrog Xray scanning of npm project dependencies to your VS Code IDE.

Competition in vulnerability scanners

Software Composition Analysis:

JFrog the company

  • Glassdoor

Resources

https://www.vogella.com/tutorials/Artifactory/article.html

YouTube channel of videos include:

https://jfrog.com/blog/control-your-kubernetes-voyage-with-artifactory/ artifactory-flow-k8s.png

Social Community

Python client for Artifactory

REST API documentation refer to curl commands, but not sample coding, such as in Python, Go, NodeJs, etc.

Additional functionality can be added by custom programming code calling REST APIs.

https://github.com/jfrog/artifactory-scripts was last updated Sep 4, 2019

DevOpsHQ, a small team of Russians working on free tools, have developed https://github.com/devopshq/artifactory described at https://devopshq.github.io/artifactory/. It’s based on https://docs.python.org/3/library/pathlib.html Although it is marked as “Code Quality D”, it offers a good start.

The above was found in a search of “artifactory” throughout GitHub.com.

Plug-ins

https://github.com/jfrog/artifactory-user-plugins presented at https://www.jfrog.com/confluence/display/JFROG/User+Plugins


More on DevOps

This is one of a series on DevOps:

  1. DevOps_2.0
  2. ci-cd (Continuous Integration and Continuous Delivery)
  3. User Stories for DevOps
  4. Enterprise Software)

  5. Git and GitHub vs File Archival
  6. Git Commands and Statuses
  7. Git Commit, Tag, Push
  8. Git Utilities
  9. Data Security GitHub
  10. GitHub API
  11. TFS vs. GitHub

  12. Choices for DevOps Technologies
  13. Pulumi Infrastructure as Code (IaC)
  14. Java DevOps Workflow
  15. Okta for SSO & MFA

  16. AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
  17. AWS server deployment options
  18. AWS Load Balancers

  19. Cloud services comparisons (across vendors)
  20. Cloud regions (across vendors)
  21. AWS Virtual Private Cloud

  22. Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
  23. Azure Certifications
  24. Azure Cloud

  25. Azure Cloud Powershell
  26. Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
  27. Azure KSQL (Kusto Query Language) for Azure Monitor, etc.

  28. Azure Networking
  29. Azure Storage
  30. Azure Compute
  31. Azure Monitoring

  32. Digital Ocean
  33. Cloud Foundry

  34. Packer automation to build Vagrant images
  35. Terraform multi-cloud provisioning automation
  36. Hashicorp Vault and Consul to generate and hold secrets

  37. Powershell Ecosystem
  38. Powershell on MacOS
  39. Powershell Desired System Configuration

  40. Jenkins Server Setup
  41. Jenkins Plug-ins
  42. Jenkins Freestyle jobs
  43. Jenkins2 Pipeline jobs using Groovy code in Jenkinsfile

  44. Docker (Glossary, Ecosystem, Certification)
  45. Make Makefile for Docker
  46. Docker Setup and run Bash shell script
  47. Bash coding
  48. Docker Setup
  49. Dockerize apps
  50. Docker Registry

  51. Maven on MacOSX

  52. Ansible

  53. MySQL Setup

  54. SonarQube & SonarSource static code scan

  55. API Management Microsoft
  56. API Management Amazon

  57. Scenarios for load
  58. Chaos Engineering