Wilson Mar bio photo

Wilson Mar


Email me Calendar Skype call

LinkedIn Twitter Gitter Instagram Youtube

Github Stackoverflow Pinterest

What can possibly go wrong with robots smarter than humans?

US (English)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Cyrillic Russian   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


Here are my notes about technical aspects of how cars can drive themselves.

TAGS: #autonomousdriving #AI

Why AV (Autonomous Vehicles)

Part of the facination (and fear) about Artificial Intelligence (AI) is computers becoming better and faster than humans in many arenas.

Driving takes concentration and is tiring. Because human drivers can be inexperienced, drunk, too tired, too distracted, etc., at the current rate of progress on AVs (Autonomous Vehicles), eventually the cost of “human error” (1 accident every 100,000 miles) will be higher than misjudgements by computers controlling vehicles.

Then, governments, auto makers, insurance companies, and others will make it more difficult to own human-driven cars.

Also, as cars automate more, and human drivers have less to do, inattention becomes even more of an issue.

Over time, as AI take over the road, self-driving cars can travel faster than what people can safely handle (around 70 mph). This would create a market for mini-hotels such as Volvo’s 360c concept car unveiled in 2018* Without constant attention to driving, personal vehicles can be RVs with a desk, bed, toilet, frige, etc. Commute without stress. Travel between cities without going through security.

The race to AV

Every auto manufacturer has a self-driving car program. Among the 40 companies:

BaiduLincoln MKZChina
Waymo/GoogleVolvo XC90

Honda’s 2017 models and onward are built with self-driving features.

Levels of autonomy

(from the Society of Automotive Engineers):

  1. Driver Assistance - driver is fully engaged. Voice prompts.
  2. Partial Automation - lane keeping with cruise control.
  3. Conditional Automation - driver is ready to take over.
  4. High Automation - no controls for human use, operating within a geofence
  5. Full Automation - starting from without a geofence in a closed venue low-speed environment such as by minibuses, valet parking, delivery robots.

Shared rides

The high cost of computing power on purpose-built AVs forces amortization across many rides in taxis.

City governments may actually see lack of need for parking undermine revenue from cark parks and traffic tickets.

Uber has, since 2012, been offering free rides in driverless cars aroud Pittsberg and Chandler, AZ.

Cruise has been offering rides in San Francisco, California.

Las Vegas.

Uber and Lyft has dessicimated the jobs of human drivers. COVID-19 is making it even more dangerous to be a driver.

NOTE: On Teslas even the glovebox lock is controlled by the computer.

Zoox got the first California permit to transport passengers in self-driving cars, in 2018. Waymo in 2019. Uber in 2020.


  • https://www.technologyreview.com/s/604006/autox-has-built-a-self-driving-car-that-navigates-with-a-bunch-of-50-webcams/


Apple has not openly discussed their self-driving car program.

In 2016, Apple’s “Titan” program scaled back its 1,000 employee self-driving car platform.

A disclosure in 2018 states that 5,000 employees at Apple know about a self-driving car program in the company.

In April 2018, Apple hired Google’s former AI boss to run Siri and machine learning.

Alphabet (Google)

Alphabet (Google) holds a seven percent stake in Uber. Google also owns Waymo.

VIDEO: Chris Urmson, head of Google’s driverless car program, shares footage showing how cars see.


Baidu’s Apollo

Baidu is the Google of China, providing a search engine.

Apollo (@apolloplatform) Baidu’s self-titled “the world’s first production-ready” AV.

Apollo Valet Parking is due to launch in 2020.

Silver at Udacity created a free intro class using Baidu’s Apollo library at https://github.com/ApolloAuto/apollo

Baidu’s AV has 5 cameras and 12 ultrasonic radars. Processors onboard run Xilinx processors on Infineon chips.

DuerOS is Baidu’s conversational AI program with embedded AI speech and image recognition. *

(Apollo 2.5)


GM cars have “SuperCruise”


Rather than building vehicles, George Hotz, founder of Comma.ai in San Diego, has its $100 “Comma two” Android mobile app to provide self-driving capabilities by a CAN bus wire harness taping via ODB-II port on several recent models of cars: Acura RDX, Chrysler Pacifica, Honda Accord/CRV 2015+/Fit, Jeep Grand Cherokee, 2015+, Kia, Lexus CT/ES/IS/NX/RX, Subaru, Toyota Avalon/Camry/C-HR/Corolla/Highlander 2017+/Prius 2016+/RAV4, Volkswagon Golf 2015+). “The 2020 Corolla is the best car with OpenPilot. It has less lag”..

Comma’s “OpenDriver” software is open sourced, so it’s difficult to regulate by governments.

Heads-up Display (HUD) streams video for view online at https://my.comma.ai/cabana. It uses OpenStreetMap.

Comma’s access to ODB-II enables response to ABS (Anti-brake System) triggers.

VIDEO: The camera facing the driver detects whether the driver is paying attention to the road. There are also infrared LEDs on Comma’s windshield case to provide night-time driver monitoring.

  • VIDEO:
  • https://www.youtube.com/watch?v=2Veptye978c


Lyft’s Level5

Tesla Motors

Elon Musk became the wealthiest person in the world with 25% ownership in Tesla.

VIDEO: “Lidar is a fool’s errand”.



VIDEO: MIT 6.S094: Introduction to Deep Learning and Self-Driving Cars

BTW David Silver worked at Ford’s self-driving car program and is now teaching online Udacity’s hands-on Nanodegree programs on self-driving cars at the 4-month Intro and advanced Engineer (2 three-month terms). Students work on Udacity’s car named Nanna.

  • https://discussions.udacity.com/
  • Slack for students

Udacity is founded by Sabastian Thrun (from Sweden), the “father” of self-driving car. When he was a professor at Stanford, his team won the first DARPA Grand Challenge car race. He then joined Google.



Baidu uses the Surround Computer Vision Kit hardware and Responsibility Sensitivity Safety (RSS) model from Intel’s Mobileye.

ASUS GTX1080 GPU-A8G- Gaming GPU Card


Computers needs to be able to control the vehicle’s steering, throttle, and breaking systems to execute its planning.

So vehicles need to be equipped with by-wire systems: brake by-wire, steering by-wire, throttle by-wire and shift by-wire, etc.

Additional organizations work with the Autonomous Technology Certification Facility (ATCF)


Architecture of Processes

This 2017 TED Talk [9:10] by David Silver describes the various technologies necessary:


An updated diagram:


The eventual design for version 3.0 of Baidu’s design adds a “Guardian” component:

self-driving-apollo_3 0_699x365-33837

The “Canbus” is a Controller Area Network (CAN) which transfers data between devices without the assistance of a host computer. Attach a temperature sensor to the surface of the main IC chip on ESD CAN (an Altera FPGA chip) to monitor the surface temperature of the chip to make sure it is not overheating.

HMI (Human-Machine Interface)

An off-line demo without the expensive hardware can install and run on laptops. It’s kinda like the Grand Theft Auto game. It uses Baidu’s Python-based Apollo Dreamview visualization software running under Linux: Ubuntu 14.04. Apollo is based on Linux Kernel 4.4.32).

Cruse’s Rosbg.js is a node.js & browser compatible Node@10.x module for reading rosbag binary data files.



It needs a three-dimensional model (point cloud) of the road network, including the road, buildings, tunnels, etc. with road names and the speed limit for each stretch of road, traffic lights, and other traffic control information.

Apollo uses the OpenDRIVE map standard used by its competitors. Baidu has 300 survey vehicles to map all the highways in China.

[4:50] A particle filter, a sophisticated type of triangulation which calculates how far the vehicle is from various landmarks (street lights, traffic signs, manhole covers).


For a vehicle to “localize” itself to single-digit centimeter accuracy, it currently needs to use several technologies.

Self-driving cars need to figure out more precisely where it is in the world than what GPS (Global Positioning System) can provide. A GNSS (Global Navigation Satellite System) receiver needs at least 3 of 30 satellites to calulate its location (based on time of flight). Also, GPS updates every 10 seconds, which is too slow.

RTK (Real Time Kinematic) positioning uses ground stations to provide “ground truth” used to ensure GPS accuracy to 10 meters.

The Inertial Measurement Unit (IMU) consists of a 3-axis gyroscope and accelerometer. It updates at 1000 Hz (near real time). The system has to reconcile two XY coordinate frames: the vehicle and the map. In the 3D Gyroscope, the spin axis is set to the global coordinate system while the 3 gimbals rotate.


LiDARs today use 32 lasers and 1 or 2 million beams per second, and that a 64-laser system emitting 6.4 million beams a second would give superior vertical resolution and quicker refreshes. This would be better able to capture small, fast objects such as animals darting into the road.

Alex Lidow, CEO and cofounder of Efficient Power Conversion, a provider of the gallium nitride chips found in many modern lidars. https://backchannel.com/how-my-public-records-request-triggered-waymos-self-driving-car-lawsuit-1699ff35ac28#.vi4talr7i by @meharris


High defition (HD) maps use computer vision to recognize objects within images captured.

Classification, detection, segmentation. Perception using CNN (Convolutional Neural Networks) cameras, radar, LiDAR (Light Detection and Ranging System).

Deep (learning) Neural Networks are used to draw bounding boxes to identify which lane the car is using.


A RNN (Recurrent Neural Network) is used to project trajectories, Frenet coordinates on short and long time horizons.

Software creates waypoints that plot the plan.


Planning the expected route…

Analyzing the actual route traveled.