Privately store and retrieve binary artifacts vetted of vulnerabilities, instead of public DockerHub, PiPI, etc.
- Workflow by edition and license
- Disaster Recovery Testing
- CI/CD Pipeline
- Jfrog API
- JFrog CLI on client
- Helm charts
- Regular server start-up
- AWS Cloud
- Google Cloud
- Migration from Docker Registry
- Git LFS
- User actions on sample repositories
- Repository Replication
- Network Proxy setup
- Usage from Build tools
- Create artifacts
- JFrog X-Ray
- Competition in vulnerability scanners
- JFrog the company
- Social Community
- Python client for Artifactory
- More on DevOps
Artifactory stores binary format assets such as executable files from builds, virtual memory (container) images, graphic image files, etc. (rather than textual source code). It caches local binary files as a proxy to public repositories, which make them quicker to obtain and provides a way to provide security-vetted (whitelisted) versions.
NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.
Wikipedia categorizes it as a “software package management system”.
Artifactory works with another JFrog product: X-Ray, which identifies vulnerabilities in transitive dependencies calling 3rd-party open-source libraries.
User Guide: https://www.jfrog.com/confluence/display/RTF/Welcome+to+Artifactory
Workflow by edition and license
Artifactory provides traceability of dependencies within components and images built as artifacts are promoted by Helm for use by Kubernetes.
Licensed editions integrates with JFrog Xray which performs “continuous” Security scans and Artifact impact analysis based on JFrog’s database of vulnerabilities. VIDEO
A Virtual Repository can be defined in Artifactory to access multiple Bower registries aggregated under a single URL.
Both on-prem and cloud SaSS deployments are available.
Artifactory is licensed from JFrog at $14,400 per year for a single site and $29,500 for multiple (enterprise) sites. The money goes to support secure, clustered, High Availability registries.
Free OSS Artifactory editions fetch ONLY for transitive package managers (Java) Maven, Gradle, and Apache Ivy. But paid editions provide “universal support” for all major binary package formats:
For Git provider source API to VCS (Version Control System) repositories:
For IaC (Infrastructure as Code) and virtualized images:
- Docker (instead of DockerHub & Quay.io)
- Chef Cookbooks
- P2 for Apache Karaf and other OSGi-based components (by bndtools) in the Eclipse/Equinox provisioning platform
For operating systems:
- NuGet for Windows (Chocolatey client) as well as .NET components (DLL and EXE) and PowerShell Gallery instead of OneGet
- CocoaPods package metadata (dependency) manager (Ruby gem) for Swift and Objective-C code running on iOS mobile devices, (instead of .podspec or .podspec.json files in GitHub)
- ipk packages (resembling Debian’s dpkg) by Opkg client instead of the OpenWRT.org repository of Linux distributions for network routers (embedded devices).
- Maven instead of MavenCentral for Java jar, ear, war dependencies
- PyPI for Python by pip and conda
- SBT (Scala Built Tool) can be used for Java
- yum RPM and RubyGems for Ruby VIDEO
- PHP Composer instead of Packagist.org
- Conan for C/C++
- tar.gz from Linux machines
https://github.com/jfrog/project-examples Sample projects for training and testing CI setup with Artifactory
https://github.com/jfrog/jfrog-data-generator Tools for generating and preloading JFrog products with data
HA (High Availability)
VIDEO Get Ahead of the Curve with JFrog Artifactory Enterprise
Artifactory Enterprise claims an SLA of “Five-nines Availability” and “Near-zero Maintenance Downtime” from Unlimited Server Scalability due to
High Availability (HA) Artifactory configuration consists of a cluster of several (Docker) active/active nodes behind a load balancer. All nodes reference a single MySQL/PostgreSQL database.
The Helm chart (using K8s 1.8+) is at https://hub.helm.sh/charts/jfrog/artifactory-ha
Storage configuration settings are defined in the binarystore.xml file.
Parameter artifactory.service.pool defines a single primary node handles jobs and tasks and not interrupted by inbound traffic. Inbound traffic is received by 2 member nodes which are load balanced.
Disaster Recovery Testing
To prepare for the ability to recover from a disaster, enterprise admins of Artifactory use Replicator to synchronize repositories between remote Artifactory instances.
Based on VIDEO: Setup a Docker Registry in 5 minutes
Get and install license keys (perhaps for a ___ day trail).
Artifactory HA license?
Obtain DNS IP addresses for primary, second, and third nodes, plus load balancer.
Identify a place to store bootstrap files used to create Artifactory server instances.
Define and socialize artifact structure and naming conventions, such as this common four-part naming structure:
- Product or team name as the primary identifier of artifact ownership
- Technology, tool or package type being used
- Maturity level of package (such as development, staging, and release stages)
- Geographic locator to the physical topology of the artifact
Internally, each artifact is stored using a unique hash generated from the data in the file.
- Assign access privileges according to projects or development teams.
- Corporate policy on signing.
VIDEO: Installing JFrog Artifactory 5 High Availability Cluster - The Complete Walk through [9:23]
- JFrog Mission Control (JFMC) is used to manage deployments
- Download the bootstrap bundle tar.gz file or jfrog-artifactory-pro-5.1.0.zip.
Unzip the file.
The tomcat folder contains a web server.
See https://www.jfrog.com/confluence/display/RTF/Installing+Artifactory and https://www.jfrog.com/confluence/display/RTF/HA+Installation+and+Setup
Download the mysql-connector-java-5.1.40-bin.jar and copy it to tomcat/lib.
Environment variable $ARTIFACTORY_HOME defines the path to the folder installed.
Edit etc/ha-node.properties to change the default node.id from “art1”.
Change the url from 192.168.59.186 to the one provided for the server.
The default port is 10001. This is the same for all nodes.
- cp ../binarystore.xml etc/binarystore.xml
./bin/installService.sh installs the server on Linux machines.
./bin/installService.bat installs the server on Windows machines.
The Artifactory setup wizard should pop-up for manual configuration.
QUESTION: How can this be automated?
- Add license key obtained from JFrog.
Set admin password
PROTIP: Follow your organization’s password policies.
Configure corporate proxy settings.
Create from primary server file boostrap.bundle.tar.gz in folder etc:
curl -uadmin -X POST http://localhost:8081/artifactory/api/system/bootstrap_bundle
- Store the bootstrap in Artifactory (I’m kidding because it doesn’t exist yet).
SCP to second and 3rd nodes.
Edit etc/ha-node.properties on the 2nd node.
- Repeat the above for the 3rd node.
- [9:07] Store a new artifact.
- Do “chaos engineering” by brining down a node to verify recovery.
Practice recovery from backup.
Enable GPG Signing
- GPG Signing of components See https://www.jfrog.com/confluence/display/RTF/GPG+Signing
[Webinar] JFrog Artifactory - The Easy One Sep 13, 2018 Artifactory Query Language (AQL)
Artifactory stores metadata in a relational database.
https://github.com/jfrog/artifactory-scripts Scripts for Artifactory (Usually, for REST API), community driven.
JFrog CLI on client
JFrog CLI optimizes both upload and download operations by skipping artifacts that already exist in their target location.
Instead of using installer from https://jfrog.com/getcli/
brew install jfrog-cli-go
The response reflects use of Artifactory’s bintray.com, JFrog’s Software Distribution as a Service (instead of Acakamai CDN, etc.):
==> Downloading https://homebrew.bintray.com/bottles/jfrog-cli-go-1.30.4.mojave. ==> Downloading from https://akamai.bintray.com/0f/0f348458539931885a3e0e6e90553 ######################################################################## 100.0% ==> Pouring jfrog-cli-go-1.30.4.mojave.bottle.tar.gz 🍺 /usr/local/Cellar/jfrog-cli-go/1.30.4: 5 files, 18.2MB
See https://www.jfrog.com/confluence/display/CLI/JFrog+CLI for different authentication mechanisms that can be used.
jfrog version 1.30.4
Command syntax, such as:
View command help by the CLI alone:
NAME: jfrog - See https://github.com/jfrog/jfrog-cli-go for usage instructions. USAGE: jfrog [global options] command [command options] [arguments...] VERSION: 1.30.4 COMMANDS: rt Artifactory commands bt Bintray commands mc Mission Control commands xr Xray commands completion Generate autocomplete scripts help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --help, -h show help --version, -v print the version Environment Variables: JFROG_CLI_LOG_LEVEL [Default: INFO] This variable determines the log level of the JFrog CLI. Possible values are: INFO, ERROR, and DEBUG. If set to ERROR, JFrog CLI logs error messages only. It is useful when you wish to read or parse the JFrog CLI output and do not want any other information logged. JFROG_CLI_OFFER_CONFIG [Default: true] If true, JFrog CLI prompts for product server details and saves them in its config file. To avoid having automation scripts interrupted, set this value to false, and instead, provide product server details using the config command. JFROG_CLI_HOME_DIR [Default: ~/.jfrog] Defines the JFrog CLI home directory path. JFROG_CLI_TEMP_DIR [Default: The operating system's temp directory] Defines the temp directory used by JFrog CLI. JFROG_CLI_BUILD_NAME Build name to be used by commands which expect a build name, unless sent as a command argument or option. JFROG_CLI_BUILD_NUMBER Build number to be used by commands which expect a build number, unless sent as a command argument or option. JFROG_CLI_BUILD_URL Sets the CI server build URL in the build-info. The "jfrog rt build-publish" command uses the value of this environment variable, unless the --build-url command option is sent. JFROG_CLI_ENV_EXCLUDE [Default: *password*;*secret*;*key*;*token*] List of case insensitive patterns in the form of "value1;value2;...". Environment variables match those patterns will be excluded. This environment variable is used by the "jfrog rt build-publish" command, in case the --env-exclude command option is not sent. CI [Default: false] If true, disables progress bar on the supporting commands.
Edit your ~/.bash_profile to define environment variables to configure.
Alternately, be prompted:
jfrog rt ping --url=http://my-rt-server.com/artifactory
To avoid this message in the future, set the JFROG_CLI_OFFER_CONFIG environment variable to false. The CLI commands require the Artifactory URL and authentication details Configuring JFrog CLI with these parameters now will save you having to include them as command options. You can also configure these parameters later using the 'config' command. Configure now? (y/n):
Add jfrog repository:
helm repo add jfrog https://charts.jfrog.io
Install helm chart:
helm install jfrog/artifactory-ha –version 1.1.9
Regular server start-up
artifactory.sh starts the server on Linux machines.
artifactory.bat starts the server on Windows machines.
Access and Authentication
Role-based authorization with teams and permissions
Active Directory, Atlassian Crowd and JIRA, OAuth (multiple providers)
Licensed versions can use SSO integration for NTLM, Kerberos, Etc.
Focused Email Notifications for Artifact Changes
Alternately, install using Docker on CentOS
Integrating JFrog Artifactory with AWS CodePipeline by Erin McGill | on 24 MAY 2018
Artifactory can be used to create a hybrid solution by being a repository to bridge between Google Cloud Platform (GCP) and Google Kubernetes Engine (GKE)CI/CD pipeline on-prem, called Cloud Pro X.
Deploying JFrog Artifactory SaaS on Google Cloud Platform by JFrog Solutions Engineers Doron Meirfeld and Mansirman Singh
Migration from Docker Registry
https://github.com/jfrog/docker2artifactory to transition from V2 Docker registries to JFrog Artifactory. by Arturo Aparicio in San Jose, California.
User actions on sample repositories
All editions can search by Name, Archive, Property or Checksum values.
Artifactory Query Language (AQL) to extract data.
Annotate Artifacts with Searchable Properties Aggregate and Run Bulk Operations on Search Results
Move/copy/delete artifacts through the UI
Paid editions of Artifactory can be replicated for redundancy.
Enterprise users can use Multi-push Replication
Web Start Application Hosting
Network Proxy setup
Define a network proxy to deliver vetted installers instead of public versions.
Usage from Build tools
GoCD, TFS, Azure DevOps, AWS OpsWorks, etc.
On Demand Jar Signing are available to paid editions of Artifactory.
Paid editions of Artifactory can extend Artifactory with Groovy-based User Plugins.
JFrog’s Xray (2.X up) and scans binaries in Artifactory. Scanning references a vulnerability database that the local server downloads from JFrog. The database is about 16 GB and needs to be monitored for growth over time.
To scan packages, Xray downloads the entire artifact to its local disk So the Xray server needs a lot of disk space. At least 100 GB, or 200 GB to start.
The recommend Xray server minimum hardware is 8 cores CPU and 16 GB RAM.
https://support.jfrog.com/jFrogKnowledgePage#id=kA01r000000LwtmCAC Basic troubleshooting section at the end for the most commonly seen errors.
The JFrog VS Code Extension adds JFrog Xray scanning of npm project dependencies to your VS Code IDE.
Competition in vulnerability scanners
Software Composition Analysis:
- Sonatype.com (which offers Artifactory competitor Nexus and scanner)
- Aqua MicroScanner https://github.com/aquasecurity/microscanner
- Black Duck by Synopsys
- Twistlock.com container and serverless security scanner
JFrog the company
YouTube channel of videos include:
Introduction to Artifactory Feb 6, 2019 by Oren Ezer
Python client for Artifactory
REST API documentation refer to curl commands, but not sample coding, such as in Python, Go, NodeJs, etc.
Additional functionality can be added by custom programming code calling REST APIs.
https://github.com/jfrog/artifactory-scripts was last updated Sep 4, 2019
DevOpsHQ, a small team of Russians working on free tools, have developed https://github.com/devopshq/artifactory described at https://devopshq.github.io/artifactory/. It’s based on https://docs.python.org/3/library/pathlib.html Although it is marked as “Code Quality D”, it offers a good start.
The above was found in a search of “artifactory” throughout GitHub.com.
https://github.com/jfrog/artifactory-user-plugins presented at https://www.jfrog.com/confluence/display/JFROG/User+Plugins
More on DevOps
This is one of a series on DevOps:
- ci-cd (Continuous Integration and Continuous Delivery)
- User Stories for DevOps
- Git and GitHub vs File Archival
- Git Commands and Statuses
- Git Commit, Tag, Push
- Git Utilities
- Data Security GitHub
- GitHub API
- Choices for DevOps Technologies
- Pulumi Infrastructure as Code (IaC)
- Java DevOps Workflow
- AWS DevOps (CodeCommit, CodePipeline, CodeDeploy)
- AWS server deployment options
- Cloud services comparisons (across vendors)
- Cloud regions (across vendors)
- Azure Cloud Onramp (Subscriptions, Portal GUI, CLI)
- Azure Certifications
- Azure Cloud Powershell
- Bash Windows using Microsoft’s WSL (Windows Subsystem for Linux)
- Azure Networking
- Azure Storage
- Azure Compute
- Digital Ocean
- Packer automation to build Vagrant images
- Terraform multi-cloud provisioning automation
Hashicorp Vault and Consul to generate and hold secrets
- Powershell Ecosystem
- Powershell on MacOS
- Jenkins Server Setup
- Jenkins Plug-ins
- Jenkins Freestyle jobs
- Docker (Glossary, Ecosystem, Certification)
- Make Makefile for Docker
- Docker Setup and run Bash shell script
- Bash coding
- Docker Setup
- Dockerize apps
- Kubernetes Operators
- Threat Modeling
- API Management Microsoft
- Scenarios for load
- Chaos Engineering