Artifactory (with X-Ray security scan)

Privately store and retrieve binary artifacts vetted of vulnerabilities, instead of public DockerHub, PiPI, etc.

Overview

Artifactory stores binary format assets such as executable files from builds, virtual memory (container) images, graphic image files, etc. (rather than textual source code). It caches local binary files as a proxy to public repositories, which make them quicker to obtain and provides a way to provide security-vetted (whitelisted) versions.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Wikipedia categorizes it as a “software package management system”.

Artifactory works with another JFrog product: X-Ray, which identifies vulnerabilities in transitive dependencies calling 3rd-party open-source libraries.

User Guide: https://www.jfrog.com/confluence/display/RTF/Welcome+to+Artifactory

Competitors

Other Package Management systems (which store binary files) include Sonatype’s Nexus. Sonatype’s “Lifecycle” product competes with CI/CD projects such as Jenkins, GitHub Actions, etc.

Workflow by edition and license

Artifactory provides traceability of dependencies within components and images built as artifacts are promoted by Helm for use by Kubernetes.

Licensed editions integrates with JFrog Xray which performs “continuous” Security scans and Artifact impact analysis based on JFrog’s database of vulnerabilities. VIDEO

artifactory-flow-1044x546

A Virtual Repository can be defined in Artifactory to access multiple Bower registries aggregated under a single URL.

Both on-prem and cloud SaSS deployments are available.

Artifactory is licensed from JFrog at $14,400 per year for a single site and $29,500 for multiple (enterprise) sites. The money goes to support secure, clustered, High Availability registries.

See https://jfrog.com/open-source

Free OSS Artifactory editions fetch ONLY for transitive package managers (Java) Maven, Gradle, and Apache Ivy. But paid editions provide “universal support” for all major binary package formats:

artifactory-filetypes-1408x470

For Git provider source API to VCS (Version Control System) repositories:

Git LFS (Large File Service) on GitHub
GitLab
Bitbucket
Stash

For IaC (Infrastructure as Code) and virtualized images:

Docker (instead of DockerHub & Quay.io)
Vagrant
Chef Cookbooks
Puppet
Bower
P2 for Apache Karaf and other OSGi-based components (by bndtools) in the Eclipse/Equinox provisioning platform

For operating systems:

NuGet for Windows (Chocolatey client) as well as .NET components (DLL and EXE) and PowerShell Gallery instead of OneGet
Debian
CocoaPods package metadata (dependency) manager (Ruby gem) for Swift and Objective-C code running on iOS mobile devices, (instead of .podspec or .podspec.json files in GitHub)
ipk packages (resembling Debian’s dpkg) by Opkg client instead of the OpenWRT.org repository of Linux distributions for network routers (embedded devices).

For languages:

Maven instead of MavenCentral for Java jar, ear, war dependencies
PyPI for Python by pip and conda
SBT (Scala Built Tool) can be used for Java
NPM (Node Package Manager) for JavaScript
yum RPM and RubyGems for Ruby VIDEO
PHP Composer instead of Packagist.org
Conan for C/C++

File formats:

zip
tar.gz from Linux machines
7up???

https://github.com/jfrog/project-examples Sample projects for training and testing CI setup with Artifactory

https://github.com/jfrog/jfrog-data-generator Tools for generating and preloading JFrog products with data

https://github.com/jfrog/jfrog-idea-plugin https://jfrog.github.io/jfrog-idea-plugin/

HA (High Availability)

VIDEO Get Ahead of the Curve with JFrog Artifactory Enterprise

Artifactory Enterprise claims an SLA of “Five-nines Availability” and “Near-zero Maintenance Downtime” from Unlimited Server Scalability due to

High Availability (HA) Artifactory configuration consists of a cluster of several (Docker) active/active nodes behind a load balancer. All nodes reference a single MySQL/PostgreSQL database.

The Helm chart (using K8s 1.8+) is at https://hub.helm.sh/charts/jfrog/artifactory-ha

Storage configuration settings are defined in the binarystore.xml file.

Parameter artifactory.service.pool defines a single primary node handles jobs and tasks and not interrupted by inbound traffic. Inbound traffic is received by 2 member nodes which are load balanced.

Disaster Recovery Testing

To prepare for the ability to recover from a disaster, enterprise admins of Artifactory use Replicator to synchronize repositories between remote Artifactory instances.

Test DR.

Preparations

Based on VIDEO: Setup a Docker Registry in 5 minutes

Get and install license keys (perhaps for a ___ day trail).

Artifactory HA license?
Obtain DNS IP addresses for primary, second, and third nodes, plus load balancer.
Identify a place to store bootstrap files used to create Artifactory server instances.
Define and socialize artifact structure and naming conventions, such as this common four-part naming structure:
- Product or team name as the primary identifier of artifact ownership
- Technology, tool or package type being used
- Maturity level of package (such as development, staging, and release stages)
- Geographic locator to the physical topology of the artifact
Internally, each artifact is stored using a unique hash generated from the data in the file.
Assign access privileges according to projects or development teams.
Corporate policy on signing.

Installation

VIDEO: Installing JFrog Artifactory 5 High Availability Cluster - The Complete Walk through [9:23]

JFrog Mission Control (JFMC) is used to manage deployments

Download the bootstrap bundle tar.gz file or jfrog-artifactory-pro-5.1.0.zip.
Unzip the file.

The tomcat folder contains a web server.

See https://www.jfrog.com/confluence/display/RTF/Installing+Artifactory and https://www.jfrog.com/confluence/display/RTF/HA+Installation+and+Setup
Download the mysql-connector-java-5.1.40-bin.jar and copy it to tomcat/lib.
Environment variable $ARTIFACTORY_HOME defines the path to the folder installed.
Edit etc/ha-node.properties to change the default node.id from “art1”.
```
primary=true
```
Change the url from 192.168.59.186 to the one provided for the server.

The default port is 10001. This is the same for all nodes.
Copy
./bin/installService.sh installs the server on Linux machines.
./bin/installService.bat installs the server on Windows machines.

The Artifactory setup wizard should pop-up for manual configuration.

Configuration

QUESTION: How can this be automated?
Add license key obtained from JFrog.
Set admin password

PROTIP: Follow your organization’s password policies.
Configure corporate proxy settings.

Create from primary server file boostrap.bundle.tar.gz in folder etc:

curl -uadmin -X POST http://localhost:8081/artifactory/api/system/bootstrap_bundle

Store the bootstrap in Artifactory (I’m kidding because it doesn’t exist yet).
SCP to second and 3rd nodes.
Edit etc/ha-node.properties on the 2nd node.
```
primary=false
```
Repeat the above for the 3rd node.
[9:07] Store a new artifact.
Do “chaos engineering” by brining down a node to verify recovery.
Practice recovery from backup.

Enable GPG Signing
1. GPG Signing of components See https://www.jfrog.com/confluence/display/RTF/GPG+Signing

CI/CD Pipeline

[Webinar] JFrog Artifactory - The Easy One Sep 13, 2018 Artifactory Query Language (AQL)

Artifactory stores metadata in a relational database.

Jfrog API

https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API https://github.com/jfrog/artifactory-scripts/tree/master/REST-API-Examples

https://github.com/jfrog/artifactory-scripts Scripts for Artifactory (Usually, for REST API), community driven.

JFrog CLI on client

JFrog CLI optimizes both upload and download operations by skipping artifacts that already exist in their target location.

Instead of using installer from https://jfrog.com/getcli/

brew install jfrog-cli-go

The response reflects use of Artifactory’s bintray.com, JFrog’s Software Distribution as a Service (instead of Acakamai CDN, etc.):

==> Downloading https://homebrew.bintray.com/bottles/jfrog-cli-go-1.30.4.mojave.
==> Downloading from https://akamai.bintray.com/0f/0f348458539931885a3e0e6e90553
######################################################################## 100.0%
==> Pouring jfrog-cli-go-1.30.4.mojave.bottle.tar.gz
🍺  /usr/local/Cellar/jfrog-cli-go/1.30.4: 5 files, 18.2MB

See https://www.jfrog.com/confluence/display/CLI/JFrog+CLI for different authentication mechanisms that can be used.

Verify:
```
jfrog --version
```
```
jfrog version 1.30.4
```
Command syntax, such as:

View command help by the CLI alone:

jfrog

NAME:
jfrog - See https://github.com/jfrog/jfrog-cli-go for usage instructions.
USAGE:
jfrog [global options] command [command options] [arguments...]
VERSION:
1.30.4
COMMANDS:
rt Artifactory commands
bt Bintray commands
mc Mission Control commands
xr Xray commands
completion Generate autocomplete scripts
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--help, -h show help
--version, -v print the version
Environment Variables:
JFROG_CLI_LOG_LEVEL
[Default: INFO]
This variable determines the log level of the JFrog CLI.
Possible values are: INFO, ERROR, and DEBUG.
If set to ERROR, JFrog CLI logs error messages only.
It is useful when you wish to read or parse the JFrog CLI output and do not want any other information logged.
JFROG_CLI_OFFER_CONFIG
[Default: true]
If true, JFrog CLI prompts for product server details and saves them in its config file.
To avoid having automation scripts interrupted, set this value to false, and instead,
provide product server details using the config command.
JFROG_CLI_HOME_DIR
[Default: ~/.jfrog]
Defines the JFrog CLI home directory path.
JFROG_CLI_TEMP_DIR
[Default: The operating system's temp directory]
Defines the temp directory used by JFrog CLI.
JFROG_CLI_BUILD_NAME
Build name to be used by commands which expect a build name, unless sent as a command argument or option.
JFROG_CLI_BUILD_NUMBER
Build number to be used by commands which expect a build number, unless sent as a command argument or option.
JFROG_CLI_BUILD_URL
Sets the CI server build URL in the build-info. The "jfrog rt build-publish" command uses the value of this environment variable, unless the --build-url command option is sent.
JFROG_CLI_ENV_EXCLUDE
[Default: *password*;*secret*;*key*;*token*]
List of case insensitive patterns in the form of "value1;value2;...". Environment variables match those patterns will be excluded. This environment variable is used by the "jfrog rt build-publish" command, in case the --env-exclude command option is not sent.
CI
[Default: false]
If true, disables progress bar on the supporting commands.

Edit your ~/.bash_profile to define environment variables to configure.

Alternately, be prompted:

jfrog rt ping --url=http://my-rt-server.com/artifactory

To avoid this message in the future, set the JFROG_CLI_OFFER_CONFIG environment variable to false.
The CLI commands require the Artifactory URL and authentication details
Configuring JFrog CLI with these parameters now will save you having to include them as command options.
You can also configure these parameters later using the 'config' command.
Configure now? (y/n):

Helm charts

https://hub.helm.sh/charts/jfrog/artifactory-ha

Add jfrog repository:

helm repo add jfrog https://charts.jfrog.io

Install helm chart:

helm install jfrog/artifactory-ha –version 1.1.9

Regular server start-up

artifactory.sh starts the server on Linux machines.
artifactory.bat starts the server on Windows machines.

Access and Authentication

LDAP Authentication

Role-based authorization with teams and permissions

Active Directory, Atlassian Crowd and JIRA, OAuth (multiple providers)

Licensed versions can use SSO integration for NTLM, Kerberos, Etc.
Focused Email Notifications for Artifact Changes

Alternately, install using Docker on CentOS

AWS Cloud

Integrating JFrog Artifactory with AWS CodePipeline by Erin McGill | on 24 MAY 2018

artifactory-aws-746x414

Google Cloud

Artifactory can be used to create a hybrid solution by being a repository to bridge between Google Cloud Platform (GCP) and Google Kubernetes Engine (GKE)CI/CD pipeline on-prem, called Cloud Pro X.

Deploying JFrog Artifactory SaaS on Google Cloud Platform by JFrog Solutions Engineers Doron Meirfeld and Mansirman Singh

JFrog-artifactory-700x700

Migration from Docker Registry

https://github.com/jfrog/docker2artifactory to transition from V2 Docker registries to JFrog Artifactory. by Arturo Aparicio in San Jose, California.

Git LFS

https://github.com/jfrog-aparicio/git-lfs

All editions can search by Name, Archive, Property or Checksum values.

Artifactory Query Language (AQL) to extract data.

Annotate Artifacts with Searchable Properties Aggregate and Run Bulk Operations on Search Results

Move/copy/delete artifacts through the UI

https://github.com/jfrog/artifactory-user-plugins/tree/master/cleanup

Repository Replication

Paid editions of Artifactory can be replicated for redundancy.

Enterprise users can use Multi-push Replication

Web Start Application Hosting

Network Proxy setup

Define a network proxy to deliver vetted installers instead of public versions.

Usage from Build tools

GoCD, TFS, Azure DevOps, AWS OpsWorks, etc.

Create artifacts

On Demand Jar Signing are available to paid editions of Artifactory.

API

Paid editions of Artifactory can extend Artifactory with Groovy-based User Plugins.

JFrog X-Ray

JFrog’s Xray (2.X up) and scans binaries in Artifactory. Scanning references a vulnerability database that the local server downloads from JFrog. The database is about 16 GB and needs to be monitored for growth over time.

https://www.jfrog.com/confluence/display/XRAY/Installing+Xray

To scan packages, Xray downloads the entire artifact to its local disk So the Xray server needs a lot of disk space. At least 100 GB, or 200 GB to start.

The recommend Xray server minimum hardware is 8 cores CPU and 16 GB RAM.

https://support.jfrog.com/jFrogKnowledgePage#id=kA01r000000LwtmCAC Basic troubleshooting section at the end for the most commonly seen errors.

The JFrog VS Code Extension adds JFrog Xray scanning of npm project dependencies to your VS Code IDE.

Competition in vulnerability scanners

Software Composition Analysis:

Sonatype.com (which offers Artifactory competitor Nexus and scanner)
Aqua MicroScanner https://github.com/aquasecurity/microscanner
SNYK.io
Black Duck by Synopsys
Twistlock.com container and serverless security scanner
Veracode.com
Whitesource

JFrog the company

Glassdoor

Resources

https://www.vogella.com/tutorials/Artifactory/article.html

YouTube channel of videos include:

Introduction to Artifactory Feb 6, 2019 by Oren Ezer
2:31 Artifactory - Sharing Binaries the Smart Way!

https://jfrog.com/blog/control-your-kubernetes-voyage-with-artifactory/ artifactory-flow-k8s.png

Questions and Answers on StackOverflow tagged “Artifactory”
LinkedIn
Facebook
Instagram
https://flink.apache.org/

Python client for Artifactory

REST API documentation refer to curl commands, but not sample coding, such as in Python, Go, NodeJs, etc.

Additional functionality can be added by custom programming code calling REST APIs.

https://github.com/jfrog/artifactory-scripts was last updated Sep 4, 2019

DevOpsHQ, a small team of Russians working on free tools, have developed https://github.com/devopshq/artifactory described at https://devopshq.github.io/artifactory/. It’s based on https://docs.python.org/3/library/pathlib.html Although it is marked as “Code Quality D”, it offers a good start.

The above was found in a search of “artifactory” throughout GitHub.com.

Plug-ins

https://github.com/jfrog/artifactory-user-plugins presented at https://www.jfrog.com/confluence/display/JFROG/User+Plugins

More on DevOps

This is one of a series on DevOps:

Wilson Mar

Artifactory (with X-Ray security scan)

Overview

Competitors

Workflow by edition and license

HA (High Availability)

Disaster Recovery Testing

Preparations

Installation

Configuration

Enable GPG Signing

CI/CD Pipeline

Jfrog API

JFrog CLI on client

Helm charts

Regular server start-up

Access and Authentication

AWS Cloud

Google Cloud

Migration from Docker Registry

Git LFS

User actions on sample repositories

Repository Replication

Network Proxy setup

Usage from Build tools

Create artifacts

API

JFrog X-Ray

Competition in vulnerability scanners

JFrog the company

Resources

Python client for Artifactory

Plug-ins

More on DevOps

Wilson Mar

Artifactory (with X-Ray security scan)

Competitors

Workflow by edition and license

HA (High Availability)

Disaster Recovery Testing

Preparations

Installation

Configuration

Enable GPG Signing

CI/CD Pipeline

Jfrog API

JFrog CLI on client

Helm charts

Regular server start-up

Access and Authentication

AWS Cloud

Google Cloud

Migration from Docker Registry

Git LFS

User actions on sample repositories

Repository Replication

Network Proxy setup

Usage from Build tools

Create artifacts

API

JFrog X-Ray

Competition in vulnerability scanners

JFrog the company

Resources

Social Community

Python client for Artifactory

Plug-ins

More on DevOps

You might also enjoy (View all posts)