Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

Don’t use this if you’re fond of SQL joins and static schemas

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

NOTICE: Below are my notes on Neo4j I have abandoned because I found the product to be too difficult to figure out for me, with not enough support nor documentation. The killer for me is not being able to get pass the password on initial use.

This tutorial is a hands-on introduction to install Neo4J, configure, create a database from commands,load data, etc. Rather than innudating you with facts and conceptual words to remember, commentary here is provided after you take some action, step-by-step. Like a guided walking tour. So you learn by doing. “PROTIP” flags unique or important information.

The contribution of this article is a maticulously sequenced presentation that curates a concise yet deep tidbits from the many resources about this topic.

Before reading this, read my notes on graph databases.

The great thing about Neo4j is that it can provide future flexibility with multi-platform data access without being locked in. This means if you want to use a different language (such as C#) next year, you can access the same database (using a plugin for that language). If your data volume scales massively, you can easily switch to a cloud-based Neo4j database. That’s not the case with Python data dictionaries. If data expands, you would need to either need to get a larger machine or rewrite to access databases.

There are several ways to use Neo4j:

If you’re on Windows, read this.

Neo4j and its Cypher language

Neo4j is a product from Neo Technology, Inc. since 2007.

neo4j.com started as neo4j.org but now Neo4j installers include a Enterprise edition which requires buying a license (unless the application built on top of it is also open-sourced) to unlock limitations, allowing for LDAP role-based and subgraph access control, lock manager, clustering, hot backups of Causal Clustering using Raft protocol, and monitoring. See https://neo4j.com/startup-program/

Flagship customers include* Walmart, eBay, Cisco, Citibank, ING, UBS, HP, CenturyLink, Telenor, TomTom, Telia, Comcast, Scripps Interactive Networks, The National Geographic Society, Airbus, Orange, AT&T, Verizon, DHS, US Army, Pitney Bowes, Vanguard, Microsoft, IBM, Thomson Reuters, Amadeus Travel, Caterpillar, Volvo, and many more.

Neo4j open-sourced (at github.com/neo4j/neo4j) their “Cypher” OpenCypher query language in 2015. Among their docs is a “Refcard” of commands that are declarative (the database figures out how to do the fetching).

Developer Certification

Neo currently sends you a certificate for free if you answer 80% of 80-questions in their 60 minute on-line exam (by classmarker.com). It covers these 105 points by subject category:

  • 44 - Cypher syntax, including creation and querying of data
  • 11 - Developer - Developing with Neo4j
  • 35 - Intro
  • 15 - Modeling (The property graph model)

After going through materials below, sign up and take the exam at neo4j.com/graphacademy/neo4j-certification. At this time, you can take as many tries as you want, so use it to prompt your research.

What about the blue t-shirt?


Visual diagrams and ASCII art

Neo4j databases can be defined as both visual diagrams and by ASCII art.

[23:50] into VIDEO:
neo4j-2350-loves-907x427.jpg

Each node can be labeled (such as entity type “Person”), have an identifier (id), and contain properties (attributes).

    (graphs)-[:ARE]->(everywhere)

Its relationships from the movie “Matrix”. [13:26]

MATCH path = (:Person)-[:ACTED_IN]->(:Movie)
   RETURN path


MATCH and RETURN are Cypher keywords that perform some action. Ironically, these are in all-caps but can be like “maTch” or “return” (are case-insensitive). However, others are case sensitive.

  • path is a variable.
  • :Movie is a node label.
  • :ACTED_IN is a relationship type.

  • Parentheses define nodes.

  • Square brackets provide details such as an action verb.

”->” arrows define one-way relationships between nodes, with Patterns drawn by connecting nodes and relationships.

In VIDEO: What are Graph Databases and Why should I care?, Dave Bechberger (@bechbd of Expero Labs and VIDEO: A Skeptic’s Guide to Graph Databases) explains that with Neo4j, one can easilly “travese” a graph with arbitrary hops such as “similar” to identify a recommandation:

neo4j-lunchbox-hops-866x365-48889

With Neo4j, one can “travese” a graph with arbitrary hops such as “similar” without the need to build foreign key joins or bridge tables.

PLUG: Moreover, as more relationships are added in Neo4j, performance is not degraded with joins.

Sample graph and code in legacy Console

http://console.neo4j.org is a legacy UI that displays a sample graph defined by this Cypher code:

create (Neo:Crew {name:'Neo'}), (Morpheus:Crew {name: 'Morpheus'}), (Trinity:Crew {name: 'Trinity'}), (Cypher:Crew:Matrix {name: 'Cypher'}), (Smith:Matrix {name: 'Agent Smith'}), (Architect:Matrix {name:'The Architect'}),
(Neo)-[:KNOWS]->(Morpheus), (Neo)-[:LOVES]->(Trinity), (Morpheus)-[:KNOWS]->(Trinity),
(Morpheus)-[:KNOWS]->(Cypher), (Cypher)-[:KNOWS]->(Smith), (Smith)-[:CODED_BY]->(Architect)

Note the commas separating commands.

neo4j-matrix-257x236.png

Query:

match (n:Crew)-[r:KNOWS*]-(m) where n.name='Neo' return n as Neo,r,m

Note spaces separating commands.

Use cases in new Sandbox

  1. In a Chrome browser, go to neo4jsandbox.com, which provides built-in guides and sample datasets for popular use cases for a limited time:

    Bloom visualization

  2. Select the Bloom “springy” UI for Visual Discovery (a good first one to view)

    Neo4j’s Bloom visualization is a web GUI that allows “near natural language” filter text to control what is visualized from the database (docs)

    VIDEO preview based on the Beers database.

    Near Natural Language Search in Bloom

    VIDEOS playlist

    Different “Perspectives” show different information depending on permissions level.

    VIDEO: Neo4j Bloom Data Visualization for everyone

    VIDEO: Ontologies in Neo4j: Semantics and Knowledge Graphs

Parent/Child in Sample Recommendations

In VIDEO: Graph Databases Will Change Your Freakin’ Life Nov 28, 2016, Ed Finkler (CTO of Cloud SaaS vendor GraphStory.com) presents this sample graph:
neo4j-child-of-484x177-27830

The arrow in the relationship line points from the “CHILD_OF” in the direction of the parent. A property of this relationship (named “Created”) is when the relationship was established (in “2002”).

Prior Cyber Console Online

Prior to the Sandbox was http://console.neo4j.org, still available.

Other Use cases in sandbox

Also on neo4jsandbox.com

Additional use cases from VIDEO: “Neo4j Top Use Cases”, etc:

  • Geography (Euler’s 7 bridges of Kunisberg)
  • Internet of Things(IoT)
  • fraud detection,
  • Artificial Intelligence
  • Knowledge graph
  • Centrality and Clustering
  • Master data


Docker Install

PROTIP: Alternately, rather than doing the above by hand, I recommend that you create and store in GitHub a shell script that does the above, then execute it a single command.

  1. Create or navigate to an account folder, such as:

    ~/gits/$GITHUB_USER
  2. Clone the whole repo, which creates the repo’s folder,

    git clone https://github.com/wilsonmar/pyneo.git
    cd pyneo
    
  3. Set permissions, then run neo4j-docker.sh cloned from GitHub:

    chmod +x neo4j-docker.sh
    ./neo4j-docker.sh
    
  4. Set permissions, then run it:

    chmod +x neo4j-docker.sh
    ./neo4j-docker.sh -a -v
    

    -a actually runs the docker run command.

  5. When you see this, you’re within the Docker container:

    root@db39751a2f11:/var/lib/neo4j#

    Optionally, you can exit the script here by typing exit.

  6. Set permissions, then run Neo4j’s Cyhper shell:

    cypher-shell -u neo4j -p test
    
  7. Type in Neo4j Cypher commands as described below.

    Optionally, you can exit here by typing :exit, the exit the Docker container with exit.


Alternately, there is a Docker image at https://hub.docker.com/_/neo4j

docker pull neo4j

Install on Mac using Homebrew

PROTIP: Its better to install on you local laptop, which does not expire, as Sandbox instances do after 10 days.

PROTIP: If you’re installing on your laptop, I recommend use of Homebrew for its ease-of-use, even though its version can be behind the official website.

  1. In a Terminal:

    Pre-requsite Java

  2. Install pre-requisite AdoptOpenJDK 8 with Homebrew Cask:

    PROTIP: Neo4j is multi-platform because it is written in Java.

    brew cask install adoptopenjdk8
    

    Response:

    ==> Tapping homebrew/cask-versions
    Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask-versions'...
    remote: Enumerating objects: 191, done.
    remote: Counting objects: 100% (191/191), done.
    remote: Compressing objects: 100% (188/188), done.
    remote: Total 191 (delta 13), reused 39 (delta 0), pack-reused 0
    Receiving objects: 100% (191/191), 82.30 KiB | 4.84 MiB/s, done.
    Resolving deltas: 100% (13/13), done.
    Tapped 160 casks (208 files, 321.3KB).
    ==> Downloading https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/downl
    ==> Downloading from https://github-production-release-asset-2e65be.s3.amazonaws
    ######################################################################## 100.0%
    ==> Verifying SHA-256 checksum for Cask 'adoptopenjdk8'.
    ==> Installing Cask adoptopenjdk8
    ==> Running installer for adoptopenjdk8; your password may be necessary.
    ==> Package installers may write to any location; options such as --appdir are i
    Password:
    installer: Package name is AdoptOpenJDK
    installer: Installing at base path /
    installer: The install was successful.
    🍺  adoptopenjdk8 was successfully installed!
    
  3. Verify:

    java -version
    

    Troubleshoot if you don’t see:

    openjdk version "1.8.0_232"
    OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09)
    OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode)
    

    To avoid Desktop database creation error

    Using Mac Finder:

  4. Navigate to the directory where the jre is installed, in my case it was in: C:\Program Files (x86)\Java\jre1.8.0_201\bin 2 - Within this directory create a folder called server 3 - Still in the bin directory, go to the client directory 4 - Copy all content from the client folder into the sever folder 5 - Restart Neo4J Desktop as administrator 6 - Try again to create the database

    Install server

  5. From any folder:

    brew search neo4j
    

    The response shows that there is both a cask (UI) and command-line neo4j avialable:

    ==> Formulae
    neo4j
    ==> Casks
    neo4j

    Cask install Desktop on Mac

  6. From any folder:

    brew cask info neo4j
    
    neo4j: 1.2.3
    https://neo4j.com/download/
    Not installed
    From: https://github.com/Homebrew/homebrew-cask/blob/master/Casks/neo4j.rb
    ==> Name
    Neo4j Desktop
    ==> Artifacts
    Neo4j Desktop.app (App)
    
  7. Install the latest version:

    brew cask install neo4j
    
    Updating Homebrew...
    ==> Downloading https://neo4j.com/artifact.php?name=neo4j-desktop-offline-1.2.3.
    ######################################################################## 100.0%
    ==> Verifying SHA-256 checksum for Cask 'neo4j'.
    ==> Installing Cask neo4j
    ==> Purging files for version 1.2.3 of Cask neo4j
    
  8. In the Mac’s Finder app, navigate to your /Applications folder. Scroll to see “Neo4j Desktop”.

    PROTIP: Here is where you’ll remove the program by right-clicking and selecting “Move to Trash”.

    Alternately, pinch 4 fingers together on the Mac’s Touchpad to swipe until you see the Neo4j logo.

  9. Click on the logo. You should see this:

    neo4j-desktop-797x599

    Enable offline mode

  10. Click the “gear” icon on the left menu.
  11. Check “Enable offline mode.

    Define Graph database

  12. Click the “Graph” panel.
  13. Click “Create a Local Graph”.
  14. Click “Graph” next to the database “sandwich” icon.
  15. Replace “Graph” with “Dr.Who” or other name for sample database.
  16. Click in the Set Pasword field and type a password (“password”).
  17. Click “Create”.

    Browser

  18. Click “Neo4j Browser” for a pop-up dialog:

    neo4j-desktop-browser-conn-failed-674x436

  19. Click the documentation icon for a list of Documents to read:

    neo4j-desktop-docs-265x576

  20. Next, work with databases.

Install Enterprise Desktop GUI

Alternately:

  1. https://neo4j.com/download
  2. Select Save file such as neo4j-desktop-offline-1.2.3.dmg.
  3. From Finder, invoke the installer.
  4. Authorize the Desktop App by copying and pasting the Desktop key and pasting it in the app’s Authorization field.
  5. Create a database per instructions.

    Install server CLI on Mac using Homebrew

    If you want programs making calls to a Neo4j server:

  6. From any folder, get information about it:

    brew info neo4j
    

    PROTIP: Notice in the response “Required: java = 1.8 ✔” that a JVM is a pre-requisite for Neo4j server:

    neo4j: stable 3.5.12
    Robust (fully ACID) transactional property graph database
    https://neo4j.com/
    /usr/local/Cellar/neo4j/3.5.12 (141 files, 163.6MB) *
      Built from source on 2019-11-28 at 00:27:37
    From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/neo4j.rb
    ==> Requirements
    Required: java = 1.8 ✔
    ==> Caveats
    To have launchd start neo4j now and restart at login:
      brew services start neo4j
    Or, if you don't want/need a background service you can just run:
      neo4j start
    ==> Analytics
    install: 1,953 (30 days), 5,351 (90 days), 18,320 (365 days)
    install-on-request: 1,736 (30 days), 4,684 (90 days), 15,963 (365 days)
    build-error: 0 (30 days)
  7. Install the latest version:

    brew install neo4j
    

    The response presents tips for starting the server:

    ==> Downloading https://neo4j.com/artifact.php?name=neo4j-community-3.5.12-unix.
    ######################################################################## 100.0%
    ==> Caveats
    To have launchd start neo4j now and restart at login:
      brew services start neo4j
    Or, if you don't want/need a background service you can just run:
      neo4j start
    ==> Summary
    🍺  /usr/local/Cellar/neo4j/3.5.12: 138 files, 163.6MB, built in 24 seconds
    
  8. Remember the path above for the forthcoming step:

    /usr/local/Cellar/neo4j/3.5.12
  9. Verify version:

    neo4j version</stro ng></pre>
    
    The response at time of this writing was:
    
    
    neo4j 3.5.12
    ### Configure Environment Variable PROTIP: The Summary response provides a hint of where Neo4j's binary is located.
  10. Create an environment in ~/.bash_profile to hold the curerent version:

    export NEO4J_VER="3.5"
    echo $NEO4J_VER
    
  11. Create an environment in ~/.bash_profile to hold a path to the goodies:

    export NEO4J_HOME="/usr/local/Cellar/neo4j/3.5.12/libexec/"
    echo $NEO4J_HOME
    ls $NEO4J_HOME
    

    The response should be:

    LICENSES.txt	bin		data		lib		plugins
    UPGRADE.txt	conf		import		logs		run
    

    Install utilities

    Install the Apoc neo4j-contrib library of community-contributed code:

  12. Identify the latest release in community-contributed jar from https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases

  13. Navigate to the plugins folder temporarily to obtain the url to the latest release to download, such as:

    pushd plugins
    curl -O https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/3.5.0.6/apoc-3.5.0.6-all.jar
    popd
    

    VIDEO: Neo4j APOC Utility Library HowTo Series

  14. Edit $NEO4J_HOME/conf/neo4j.conf
  15. Whitelist only specific procedures and functions (rather than all) to be loaded:

    dbms.security.procedures.whitelist=apoc.coll.*,apoc.load.*
  16. Save and exit the text editor.

    Streaming Graph Loading with Neo4j and APOC Triggers by David Allen

    APOC Articles by Mark Needham

    See https://medium.com/neo4j/efficient-neo4j-data-import-using-cypher-scripts-7d1268b0747

    Now skip to the Invoke section.

    Alternately:

    Manual download back version

    You may want to install a back version if you want to follow along an older tutorial: “Introduction to Graph Databases and Neo4j 2h Pluralsight video course by Microsoft MVP Roland Guijt (@RolandGuijt, rmgsolutions.nl) released February 5, 2015 while using Neo4j version 2.1.3 on Windows. So the UI has changed.

  17. In an internet browser, go to Neo4j’s Other Releases to download:

    https://neo4j.com/download/other-releases/

  18. Find the link for the version you want to use.
  19. Click the “Download” button for sign-ups
  20. Get the OReilly book.

    Manual install

    neo4j-desktop-?.?.?.dmg

  21. Click “Open” to “Are you sure you want to open it?”.
  22. Click “I Agree”.
  23. Login using your email or through social media.

    NOTE: It says Java 8 is downloaded if it doesn’t exist, but I got errors.

  24. Set the $NEO4J environment variable to point to where Neo4j is installed.

    Invoke

  25. For a menu, invoke the neo4j CLI executable without any parameters:

    neo4j
    

    The help response lists the sub-commands:

    Usage: neo4j { console | start | stop | restart | status | version }   
    
  26. Initially we don’t want/need a background service, so:

    neo4j start
    

    If you had installed using Homebrew:

    Active database: graph.db
    Directories in use:
      home:         /usr/local/Cellar/neo4j/3.5.12/libexec
      config:       /usr/local/Cellar/neo4j/3.5.12/libexec/conf
      logs:         /usr/local/var/log/neo4j
      plugins:      /usr/local/Cellar/neo4j/3.5.12/libexec/plugins
      import:       /usr/local/Cellar/neo4j/3.5.12/libexec/import
      data:         /usr/local/var/neo4j/data
      certificates: /usr/local/Cellar/neo4j/3.5.12/libexec/certificates
      run:          /usr/local/Cellar/neo4j/3.5.12/libexec/run
    Starting Neo4j.
    Started neo4j (pid 3336). It is available at http://localhost:7474/
    There may be a short delay until the server is ready.
    See /usr/local/var/log/neo4j/neo4j.log for current status.
    

    Alternately, to start neo4j now and automatically restart at login:

    brew services start neo4j
    

    Add sudo before the command if you get this error:

    Error: Permission denied @ rb_sysopen - /Users/wilson_mar/Library/LaunchAgents/homebrew.mxcl.neo4j.plist
    

    The expected response (at time of writing):

    Password:
    Warning: Taking root:admin ownership of some neo4j paths:
      /usr/local/Cellar/neo4j/3.5.12/bin
      /usr/local/Cellar/neo4j/3.5.12/bin/neo4j
      /usr/local/opt/neo4j
      /usr/local/opt/neo4j/bin
      /usr/local/var/homebrew/linked/neo4j
    This will require manual removal of these paths using `sudo rm` on
    brew upgrade/reinstall/uninstall.
    ==> Successfully started `neo4j` (label: homebrew.mxcl.neo4j)
    
  27. Verify:

    brew services list
    

    You should see:

    Name    Status  User Plist
    neo4j   started root /Library/LaunchDaemons/homebrew.mxcl.neo4j.plist
    
  28. Open in the default internet browser:

    open http://localhost:7474/browser/
    

    Restart server

    bash ./bin/neo4j restart
    
    Neo4j not running
    Starting Neo4j.
    Started neo4j (pid 29227). It is available at http://localhost:7474/
    There may be a short delay until the server is ready.
    See /usr/local/var/log/neo4j/neo4j.log for current status.
    

    Stop server

  29. To get the PID for the Neo4j process, widen the Terminal screen, then:

    ps -al
    

    Remember the PID on the line containing this CMD:

    /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java -cp /usr/local/Cellar/neo4j/3.5.12/libexec/plugins:/usr/l
  30. Stop it:

    brew services stop neo4j
    

    Edit Config for upgrade

  31. Within a Terminal, edit the configuration file using “subl” or substituting “vi”, “nano”, or your preferred text editor (Visual Studio Code, which is invoked by the code command):

    echo "NEO4J_HOME=$NEO4J_HOME"
    code $NEO4J_HOME/conf/neo4j.conf
    

    See a description of the dozens of keys at: https://neo4j.com/docs/operations-manual/3.5/reference/configuration-settings

  32. Find text “dbms.allow_upgrade=” within the file.
  33. Remove the # comment so your database upgrades automatically in case its built version is older than your current Neo4j version:

    dbms.allow_upgrade=true
    
  34. Set logging specifications: TODO:

    ???=true
    
  35. Change the password:

    call dbms.security.changePassword('Pa$$word1');
    

    Connect to Neo4j

    Database access not available. Please use to establish connection. There’s a graph waiting for you.

  36. Type the default username and password of “neo4j” and “neo4j”.
  37. Click “Connect”.

    BLAH

Graph database can store any kind of data using these concepts:

  • Nodes - graph data records
  • Relationships - connect nodes
  • Properties - named data values

https://diseaseknowledgebase.etriks.org/metabolic/browser/

### Query Logging

  1. Query logging must be enabled by setting the parameter to:

    dbms.logs.query.enabled=true

    To set all queries to be logged:

    dbms.logs.query.threshold=0

    Alternately, set a threshold for the number of seconds before logging, such as 7:

  2. There are additional logging parameters that are false by default:

    dbms.logs.query.parameter_logging_enabled=true
    dbms.logs.query.time_logging_enabled=true
    dbms.logs.query.allocation_logging_enabled=true
    dbms.logs.query.page_logging_enabled=true
    
  3. Save the file and exit the editor.

    Start Browser Console

    When your Neo4j instance is ready:

  4. Open the Neo4j Browser client:

    PROTIP: So you can quickly switch among the two using Command+Tab, open a browser different than the one used to display this tutorial (perhaps Firefox instead of Chrome), then go to the default URL:

    http://localhost:7474

    From https://neo4j.com/developer/docker-run-neo4j/

    docker run \
     --name testneo4j \
     -p7474:7474 -p7687:7687 \
     -d \
     -v $HOME/neo4j/data:/data \
     -v $HOME/neo4j/logs:/logs \
     -v $HOME/neo4j/import:/var/lib/neo4j/import \
     -v $HOME/neo4j/plugins:/plugins \
     --env NEO4J_AUTH=neo4j/test \
     neo4j:latest
    

    Alternately, when running within Docker Machine, open the Neo4j Browser client using your default browser. On a Mac:

    open http://$(docker-machine ip default):7474
    

    Ports (within Docker)

    PROTIP: By default the Docker image for Neo4j exposes three ports for remote access:

    docker run \
     --publish=7474:7474 --publish=7687:7687 \
     --volume=$HOME/neo4j/data:/data \
     --volume=$HOME/neo4j/logs:/logs \
     neo4j:3.2
     
    • 7474 for HTTP
    • 7473 for HTTPS
    • 7687 for Bolt

    Bolt Network Protocol

    PROTIP: Although Neo4j as a transactional http/https versioned endpoint, the Neo4j database (since v3) by default communicates using the Bolt binary protocol (for multi-cluster routing) based on Packetstream. Much like JDBC for relational databases, Bolt operates over a TCP connection or WebSocket. Built-in Auth and TLS. It’s defined at Bolt network protocol. Connection credentials are stored in your web browser, such as:

    from py2neo import Database
    db = Database("bolt://ws-10-0-1-64-32989.neo4jsandbox.com:443")

    To use the default bolt:

    default_db = Database()

    Confirm:

    default_db

    returns:

    <Database uri='bolt://localhost:7687'.

    Currently, Neo4j only supports one Graph per Database.

    Browser Commands

    Commands can be entered in the Editor field at the top which begins with a dollar sign in gray.

  5. Click the top command entry field to the right of the dollar sign, type the first letter of commands, a colon (:), for auto-completion list of most common commands:

    neo4j-commands-1206x800
    Click image for larger image pop-up.

    Dark theme color for commands

  6. Click the gear icon near the lower-left corner among menu icons.

    neo4j-console
    Click image for larger image pop-up.
  7. Click Theme Dark.
  8. Click the icon again to dismiss menu contents. (it’s a toggle)

    Play intro

  9. Click the icon that looks like a book (previously this was an i icon for information).
  10. Click “Getting Started”.

    Notice that these commands appear in the command field:

    :play intro
    

    Notice that the commands now have different colors.

  11. To submit the command, press Enter or click the arrow icon at the upper-right corner.

    The page says for multi-line commands to press Shift+Enter to enter multi-line mode, then press Ctrl+Enter instead of the arrow icon.

  12. Scroll down the page to see that the new response is added above the previous frame, as in a stack as a stream.
  13. Click the gray X to dismiss a content frame.
  14. Click “Operations Manual” under Useful Resources to pop up a new browser tab to the version installed:

    https://neo4j.com/docs/operations-manual.

    Monitoring limits

  15. Type command :sysinfo and press Enter for:

    neo4j-sysinfo-1530x596

    PROTIP: Execute this on a schedule to ensure that more space is allocated before need.

    Manage users and roles

    PROTIP: Passwords can be changed in the command line. See: https://neo4j.com/docs/operations-manual/3.2/reference/user-management-community-edition/#userauth-list-all-users-ce

    https://neo4j.com/docs/operations-manual/3.2/tutorial/role-based-access-control/

    The Enterprise edition of ???

    CALL dbms.security.listUsers()
    

    Change password

  16. Per documentation:

    neo4j-admin set-initial-password h6u4%kr
  17. http://boopathi.me/blog/reset-neo4j-graph-database-password/

  18. Notice in the sample console image that the password is blank.

  19. In the Password field, type the default “neo4j” (lower case).
  20. Type your own password. Twice.

Note in the response your user name is “neo4j”.


After desktop install:

Database failed to create: Error: Could not change password https://github.com/neo4j/neo4j/issues/11429

Database creation commands

Neo4j comes with several databases.

  1. Click “Write Code” and press Enter to invoke :play write code.

    neo4j-starter-650x208-37863

    Movie sample creation

  2. See instructions to build a relationship graph about among actors and directors:

    :play movie graph
    

    The “Create” pane appears with code that begins with command “CREATE”.

  3. Click the code for it to be posted in the command line.

  4. Click the full screen icon.

    neo4j-fullscreen-510x142-5114

  5. BLAH: Drag and drop items to arrange the graph to your asthetic taste.

    neo4j-movie-graph-1676x702-144758

    This is like ER (Entity-Relation) diagrams for SQL databases.

    PROTIP: Neo4J data is stored the same way as illustrated by the data model, whereas with SQL data is stored in separate tables joined together.

  6. The gray controls appear after you click on an object (the red colored dot labeled “The Polar Express”).

  7. Click the “X” in the wheel to delete the node.

    PLUG: Thus, Neo4j is naturally adaptive. Entities can be added dynamically, without schema migrations required in SQL databases.

  8. Click the database icon at the top of the left menu to view a list of

    • Node Labels
    • Relationship Types
    • Property Keys

    neo4j-db-info-734x978-42810

  9. Create an entry:

    CREATE (m:Movie {title:'Mystic River', released:2003})
    RETURN m
  10. On Create, time stamp: [40:40]

    MERGE (p:Person {name: 'Your Name'})
    ON CREATE p.created = timestamp(), p.updated = 0
    ON MATCH SET p.updated = p.updated + 1
    RETURN p.created, p.updated
  11. Modify an existing entry with a tagline: [37:11]

MATCH (m:Movie {title:'Mystic River'})
   SET m.tagline = 'We bury our sins here, Dave. We wash them clean.'
   RETURN m
  1. Create a relationship: [37:31]

    MATCH (m:Movie {title:'Mystic River'})
    MATCH (p:person {name: 'Kevin Bacon'})
    CREATE (p-[r:ACTED_IN {roles: ['Sean']}])->(m)
    RETURN p, r, m

    Northwind sample creation

    VIDEO:

    PROTIP: Relations are not “first-class citizens” in a relational database. But they are in graph databases.

    https://github.com/neo4j-contrib/developer-resources/tree/gh-pages/data/northwind

  2. Open the “Northwind” sample database which Microsoft provides with its SQL database.

    :play northwind graph
    

    This is a more complex database with data common to business accounting.

    LOAD CSV WITH HEADERS FROM "http://data.neo4j.com/northwind/products.csv" AS row
    CREATE (n:Product)
    SET n = row,
      n.unitPrice = toFloat(row.unitPrice),
      n.unitsInStock = toInteger(row.unitsInStock), n.unitsOnOrder = toInteger(row.unitsOnOrder),
      n.reorderLevel = toInteger(row.reorderLevel), n.discontinued = (row.discontinued <> "0")
    

</pre>

LOAD CSV WITH HEADERS FROM "http://data.neo4j.com/northwind/categories.csv" AS row
CREATE (n:Category)
SET n = row
CREATE INDEX ON :Product(productID)
CREATE INDEX ON :Category(categoryID)
CREATE INDEX ON :Supplier(supplierID)

### Indexes

Indexes are used to find the starting point for queries to traverse (where by contrast SQL uses indexes to look up rows in tables)

PROTIP: Cypher keywords ENDS, WITH, and CONTAINS are, as of v3, index-backed. Neo4j uses “index read adjacency” to make itself quicker to traverse nodes instead of slower index lookups in SQL.

To improve performance, indexes in Neo4j can be backed by Lucene index type.

  1. To schedule resampling of an index:

    db.resampleIndex(":Person(name)")
    

Load data

  1. First, stop the service:

    Stop Browser

  2. Open a Terminal window to issue command:

    neo4j stop
    

    The response:

    Stopping Neo4j.. stopped
    

    Now you can backup and dump the database.

  3. Switch to or open a Mac/Linux Terminal instance and:

    cd $NEO4J_HOME/data/Databases
    ls -al
    cd graph.db
    ls -al
    

    Notice graph.db.

    PROTIP: In the Neo4j world, a physical database consists of files stored under a folder named with a .db suffix. A “graph” references a physical Neo4j database that stores data.

    Backup default database

    See https://neo4j.com/docs/operations-manual/3.2/tools/dump-load/ for the various attributes to add.

    Download sample DrWho database

    With the server stopped:

  4. Open a new browser tab to download a zip file containing a sample database from:

    https://neo4j.com/developer/example-data

  5. Right-click to Save Link as… (download) the Jim Webber’s Doctor Who Data Set drwho.zip file to your Downloads folder.
  6. Unzip the file to create folder drwho.
  7. Look into the folder index.

    PROTIP: Neo4j uses Lucene to index, same as ElasticSearch and others.

  8. Copy the drwho folder within the Neo4j database folder:

    cp ~/Downloads/drwho  /usr/local/Cellar/neo4j/3.3.0/libexec/data/Databases
    ls
    

    Check db consistency

  9. See https://neo4j.com/docs/operations-manual/3.2/tools/consistency-checker/


    Use cases

    Recommendations can be made. You like Tom Hanks? Here are his other movies.

    More sophisticated versions of such a database are being used to detect fraud. See https://t.co/OMHaHOrYtq

    Video of use cases “What SQL had to process in batch can now be processed in real-time with Neo4j.”

    https://www.youtube.com/watch?v=lb90EBfAj0o

    https://maxdemarzi.com/2017/11/21/mutual-fund-benchmarks-with-neo4j/

    Graph Gists

    The developer community’s repository of sample data models and queries for a variety of use cases at portal.graphgist.org is showcased on the neo4j.com/GraphGists web page.

    Any developer can create a GraphGist. Reach out to devrel@neo4j.com.

    Building a graph of your data is fairly simple as the graph structure represents the real world much better than columns and rows of data.

    Mental Illnesses

    The Open Source Mental Illness Neo4j database is at: https://github.com/OSMIHelp/osmi-survey-graph

    VIDEO: datasets are easily accessible using the blue “Write Code” button under the “Jump into Code” section of the guides.

Cypher Query

https://www.opencypher.org/#resources

A big innovation by Neo4j is that it provides programmers with a flexible network structure of nodes and relationships rather than static SQL tables.

Cypher is a language akin to SQL.

Cypher Style Guide (for M08), a 10-page pdf covers Indentation and line breaks, Meta-characters, Casing, Patterns Spacing. The document presents this classical piece by Mark Needham: as a “sane query”:

MATCH (member:Member {name: 'Mark Needham'})
      -[:HAS_MEMBERSHIP]->()-[:OF_GROUP]->(:Group)-[:HAS_TOPIC]->(topic)
WITH member, topic, count(*) AS score
MATCH (topic)<-[:HAS_TOPIC]-(otherGroup:Group)
WHERE NOT (member)-[:HAS_MEMBERSHIP]->(:Membership)-[:OF_GROUP]->(otherGroup)
RETURN otherGroup.name, collect(topic.name), sum(score) AS score
ORDER BY score DESC

PROTIP: Do not use a semicolon at the end of the statement.

execution plan with EXPLAIN and PROFILE.

https://www.youtube.com/watch?v=1TSBXZMv6tc

https://neo4j.com/download-thanks-desktop/docs/developer-manual/current/cypher/#how-do-i-profile-a-query

Neo4j is considered among other “NOSQL” database tech that include Key-Value Stores, Column-Family Stores, and Document Databases. But these others use aggregate data models whereas graph databases such as Neo4j work with simple records and complex interactions.

Instead of SQL union statements. an example of code is:

(graphs)-[:ARE]->(everywhere)
   

Direction

Neo4j API allow developers to completely ignore relationship direction when querying the graph.

MATCH (boss)-[:MANAGES*0..3]->(sub),
      (sub)-[:MANAGES*1..3]->(report)
   WHERE boss.name = "John Doe"
   RETURN sub.name AS subordinate,
   count(report) AS Total
   

https://www.youtube.com/watch?v=NH6WoJHN4UA&index=3&list=PL9Hl4pk2FsvWM9GWaguRhlCQ-pa-ERd4U

Fast?

For many applications, Neo4j offers orders of magnitude performance advantage compared to relational DBs. Each index and join adds to the time needed.

  • Antlr grammar
  • EBNF (Extened Backus–Naur Form) metasyntax notation is used to make a formal description of a formal (computer programming) language . “Extended” refers to its pedigree from Niklaus Wirth’s BNF syntax which consists of some of the concepts, but with a different syntax and notation. See https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form

  • Railroad diagrams https://s3.amazonaws.com/artifacts.opencypher.org/M08/railroad/Cypher.html

  • Grammar specification
  • TCK (Technology Compatibility Kit) specification

Importing data

Michael Hungar, Neo4j Developer Evangelist, wrote

Michael started the https://github.com/neo4j-contrib “Apoc” add-on library for Neo4j neo4j-apoc-procedures. It provides hundreds of procedures and functions adding a lot of useful functionality. It is available in all Neo4j Sandboxes and in the Neo4j Cloud. It can also be installed with a single click in Neo4j Desktop. It’s described with videos at neo4j.com/labs/apoc and in the Developer Guide which references https://github.com/neo4j-contrib/developer-resources.

We build something similar to apoc.load.json just for YAML. You can create a GH issue there and ask for guidance.

VIDEO: Ambrose Leung at Microsoft Seattle wrote aka.ms/csv2graph program to load CSV files using API calls.

  1. Click the star icon on the menu to reveal the import area to drag files.

    neo4j-import-drop

    Only text files are dropped there, not images of Neo4j databases (.db files).

    Import your own data

    http://guides.neo4j.com/fundamentals/import.html

    Import data from CSV files using Cypher LOAD CSV command in the developer guide [49:12]

    [USING PERIODIC COMMIT]  // optional transaction batching
    LOAD CSV
    WITH HEADERS  // first header row as keys in "row" map
    FROM "url"
    AS row
    FIELDTERMINATOR ";"
    ...

    Polyglot

    Neo4j believes in polyglot persistence (multiple ways to store connected data), with columnar, tabular and document data stored elsewhere. The various types of data integrations possible with Neo4j is at: https://neo4j.com/developer/integration/

    For information about importing transactions into the database, see: https://neo4j.com/docs/operations-manual/3.2/tools/import/ https://neo4j.com/docs/operations-manual/3.2/tutorial/import-tool/

    The above links are for a specific version, so can becone outdated.

    https://neo4j.com/developer/guide-importing-data-and-etl

    cycli -f import-file.cypher
    

    [41:25]


Optimizing Cypher Queries

Optimizing Neo4j Cypher queries

Programming

A “hello world” example of a @userfunction:

WITH ['Hello','World'] as words
   RETURN apoc.test.join(words,' ')

==> “Hello World”

Example of making a call in Cypher:

CALL
   apoc.index.nodes('User','name:Brook*')
   YIELD node, score
   RETURN node.name, node.age, score

Language Extensions

To extend Cypher with user-defined procedures in JavaScript, C# .NET, Python, and Community drivers such as R, see https://neo4j.com/developer/language-guides

Python

pip install neo4j
pip install -r docs_requirements.txt
make -C docs html
   
  • git clone https://github.com/neo4j/neo4j-python-driver
  • Graph Databases in Python from 2012
  • https://github.com/neo4j/neo4j-python-driver

  1. Install Python driver:

    pip install neo4j-driver

    The response (at time of writing):

    Collecting neo4j-driver
      Downloading https://files.pythonhosted.org/packages/0e/96/bc81664d87975948713f7e4d3d4c3a21a3a6a813d03a161637573a587817/neo4j-driver-1.7.6.tar.gz
    Requirement already satisfied: neobolt~=1.7.15 in /usr/local/lib/python3.7/site-packages (from neo4j-driver) (1.7.16)
    Requirement already satisfied: neotime~=1.7.1 in /usr/local/lib/python3.7/site-packages (from neo4j-driver) (1.7.4)
    Requirement already satisfied: pytz in /usr/local/lib/python3.7/site-packages (from neotime~=1.7.1->neo4j-driver) (2019.1)
    Requirement already satisfied: six in ./Library/Python/3.7/lib/python/site-packages (from neotime~=1.7.1->neo4j-driver) (1.13.0)
    Building wheels for collected packages: neo4j-driver
      Building wheel for neo4j-driver (setup.py) ... done
      Created wheel for neo4j-driver: filename=neo4j_driver-1.7.6-cp37-none-any.whl size=32641 sha256=d9193b86dcfc42c27af73bdb026be1290521411e6e96678103fb8b3f0a4b8fa1
      Stored in directory: /Users/wilson_mar/Library/Caches/pip/wheels/8b/bf/07/48400007240b3dbb8bd336fdbba1a99c890144a3ba83fdfc38
    Successfully built neo4j-driver
    Installing collected packages: neo4j-driver
    Successfully installed neo4j-driver-1.7.6
    

Java coded queries

Neo4j provides Native server-side extensions in Java.

https://neo4j.com/blog/efficient-graph-algorithms-neo4j/

For large amounts of data, Cypher run time performance may be slower than Java API coding of traversals and writes. So do massive writes by writing a Java API program to read and query using parameterized Cypher queries.

Get the official drivers for Javascript, Java, .NET, and Python

https://neo4j.com/docs/developer-manual/current/drivers/#driver-get-the-driver

Additionally, the community has built a wide variety of other drivers in languages like PHP, Ruby, Go, Haskell and more.

https://neo4j.com/developer/language-guides/

https://neo4j.com/docs/developer-manual/current/procedures/

Awesome Procedures

APOC (Awesome Procedures on Cypher) are complex implementations that can’t be expressed directly on Cypher. They support data integration, graph algorithms, data conversion.

https://neo4j-contrib.github.io/neo4j-apoc-procedures/index32.html

refers to the code repository at:

https://github.com/neo4j-contrib/neo4j-apoc-procedures


Social

  1. community.neo4j.com provides weekly news and highlights popular community content.

  2. Sign up for events at neo4j.com/events/world/all

  3. Sign up for http://neo4j.com/slack at http://neo4j-users-slack-invite.herokuapp.com/

  4. Join https://groups.google.com/forum/#!forum/neo4j

  5. Subcribe to Neo4j’s YouTube channel and view videos.

  6. Click the icon at the bottom-left corner among menu icons.

    neo4j-console
    Click image for larger image pop-up.

  7. Q&A on http://stackoverflow.com/questions/tagged/neo4j

  8. Visit a Meetup group - https://www.meetup.com/Neo4j-Online-Meetup/

  9. Tweet to https://twitter.com/neo4j - #GraphViz #Neo4j #GraphDatabases

  10. Read https://neo4j.com/blog/

  11. Read https://en.wikipedia.org/wiki/Neo4j

Live in person

https://www.meetup.com/pro/neo4j/

Rockstars

Michael Hunger (@mesiri), Neo4j developer evangelist in Dresden, Germany

Jim Webber provides “koan” style tutorial presents a set of databases which have something not right, so students learn to fix things. Brilliant approach and a great learning tool:

Ryan Boyd (LinkedIn) a SF-based ex-Googler, now Neo4j Head of Developer Relations.

William Lyon (@lyonwj, lyonwj.com), software developer on the Developer Relations team, he works primarily on integrating Neo4j with other technologies, building demo apps, helping other developers build applications with Neo4j, and writing documentation. Applying an active learning algorithm for entity de-duplication in graph data.

  • https://www.slideshare.net/neo4j/building-a-graphql-service-backed-by-neo4j
  • https://github.com/neo4j-graphql/neo4j-graphql
  • https://grandstack.io (GraphQL, React, Apollo, Neo4j)

Emil Eifrem (@emileifrem, emil@neotechnologies.com), CEO

Mark Needham and Amy E. Hodler

  • DOWNLOAD O’Reilly BOOK: Graph Algorthms: Practical examples in Apache Spark and Neo4j
  • https://neo4j.com/docs/graph-algorithms/current/

Tim Ward (@jerrong, tiw@CluedIn.com)

Mats Rydberg living in Sweden

Johan Svensson, CTO

Philip Rathle, products VP

Neo4j, Inc. board of directors includes Rod Johnson (founder of the Spring Framework).

@GraphCurmudgeon

Rik van Bruggen

References and tutorials

https://neo4j.com/graphacademy/online-training/introduction-graph-databases

O’Reilly’s Graph Databases 211 page ebook from May 2015 with NeoLoad 2.2:

https://github.com/graphaware/neo4j-nlp Implementation of Microsoft Concept Graph

https://www.experfy.com/training/courses/an-introduction-to-neo4j#description $80 class

neo4j-hierarchy-graph-1250x476 From https://neo4j.com/blog/7-ways-data-is-graph/

quackit.com/neo4j/tutorial contains lots of ads, but is a nicely formatted tutorial if you don’t mind clicking after reading a few lines.

https://buff.ly/2w9PQFy The Top 13 Resources for Understanding Graph Theory & Algorithms

https://r.neo4j.com/2iSaBRi Geocoding #ParadisePapers Addresses in #Neo4j to Build Interactive Geographical Data Visualizations

https://www.slideshare.net/bachmanm/modelling-data-in-neo4j-plus-a-few-tips neo4j-property-graph-pulp-fiction.png

https://www.slideshare.net/KennyBastani/building-a-graph-based-analytics-platform?next_slideshow=1

  • https://www.youtube.com/watch?v=78r0MgH0u0w

  • https://www.youtube.com/watch?v=vJcxRjJ982k

  • https://www.youtube.com/watch?v=NO3C-CWykkY&index=4&list=PL9Hl4pk2FsvWM9GWaguRhlCQ-pa-ERd4U

Triplestore

Triplestore is based on the semantic web W3C RDF (Resource Definition Framework) for inter-connection

It is based on the subject - object - predicate triples where each property is a vertex (node).

An example is airline flight information VIDEO:

triplestore-flights-920x928.png

Graph databases are used by inference engines to discover relationship insights not easily found using traditional approaches. Thus, it is popular among (crime) forensic scientists.

Aggregate functions

VIDEO

Aggregates in Cypher do not require a “GROUP BY” keyword as in SQL.

A node and relationship are essentially a container of properties.

See the one-page Cyhper Refcard.

@UserAggregationFunctions need to keep state (the aggregates).

Timings / State

neo4j-timings-850x776

More info

https://github.com/elementsinteractive/flask-graphql-neo4j https://medium.com/elements/diving-into-graphql-and-neo4j-with-python-244ec39ddd94 makes use of docker-compose based on https://hub.docker.com/_/neo4j/

https://github.com/wgwz/flask-neo4j-demo by Kyle Lawlor https://github.com/wgwz/flask-py2neo

https://www.youtube.com/watch?v=78r0MgH0u0w Tips and Tricks for Graph Data Modeling Mar 25, 2015

VIDEO: Introduction to Graph Databases, Cypher, and Neo4j October 2018 by Roland Guijt (@RolandGuijt) in the Netherlands.

https://github.com/neo4j-examples

github.com/neo4j-examples contains several

https://diseaseknowledgebase.etriks.org/metabolic/browser/

http://boopathi.me/blog/category/neo4j/

Neo4j High Performance (Packt, March 2015) by Sonal Raj covers:

  • Query Neo4j using Cypher, and optimize your data model and queries to improve Cypher’s performance
  • Migrate from existing SQL stores and data import/export techniques
  • Explore the data modeling concepts and techniques associated with graph data in Neo4j
  • Develop applications with Neo4j to handle high volumes of data
  • Define how to develop an efficient architecture and transactions in a scalable way
  • Study the in-built graph algorithms for better traversals and discover Spring-Data-Neo4j
  • Look under the hood of Neo4j, covering concepts from the core classes in the source to the internal storage structure, caching, transactions, and related operations

More about Python

This is one of a series about Python:

  1. Python install on MacOS using Pyenv
  2. Python install on MacOS
  3. Python install on Raspberry Pi for IoT

  4. Test Python using Pytest BDD Selenium framework
  5. Test Python using Robot testing framework

  6. Python certifications
  7. Python tutorials
  8. Python coding notes
  9. Jupyter Notebooks provide commentary to Python

  10. Pulumi controls cloud using Python, etc.
  11. Microsoft Azure Machine Learning makes use of Python
  12. Testing AI uses Python code

  13. Python REST API programming using the Flask library
  14. Python coding for AWS Lambda Serverless programming
  15. Streamlit visualization framework powered by Python
  16. Web scraping using Scrapy, powered by Python
  17. Neo4j graph databases accessed from Python