Dear friends,
In a recent letter, I noted that one difference between building traditional software and AI products is the problem of complex product specification. With traditional software, product managers can specify a product in ways that communicate clearly to engineers what to build — for example, by providing a wireframe drawing. But these methods don’t work for AI products.
For an AI product, among the most important parts of the specification are:
Consider the problem of how to build a self-driving car. We might decide the acceptable road conditions for autonomous operation and the acceptable rate of collisions with particular objects at various speeds (for example, gently bumping a traffic cone at five miles per hour every 1 million miles may be okay, but hitting a pedestrian at 20 miles per hour every 1,000 miles is not).
Or take reading electronic health records. What is an acceptable error rate when diagnosing a serious disease? How about the error rate when diagnosing a minor disease? What if human-level performance for a particular illness is low, so physicians tend to misdiagnose it, too? Specifying the metrics, and the dataset or data distribution on which the metrics are to be assessed, gives machine learning teams a target to aim for. In this process, we might decide how to define a serious versus a minor disease and whether these are even appropriate concepts to define a product around. Engineers find it convenient to optimize a single metric (such as average test-set accuracy), but it’s not usual for a practical specification to require optimizing multiple metrics.
Here are some ideas that I have found useful for specifying AI products.
I’ve found it very helpful to have sufficient data and a clear target specification for each slice. This isn’t always easy or even possible, but it helps the team advance toward a reasonable target.
Keep learning! Andrew
NewsLighter Traffic AheadTraffic signals controlled by AI are keeping vehicles rolling citywide. What’s new: Several U.S. cities are testing systems from Israel-based startup NoTraffic that promise to cut both commute times and carbon emissions, according to MotorTrend. The company plans to expand to 41 cities by the end of 2021. How it works: NoTraffic uses a combination of neural networks and other techniques to optimize intersections and coordinate traffic signals throughout a city. The system is outfitted to integrate with pavement sensors and connected-vehicle protocols.
Behind the news: Machine learning is combating congestion outside the U.S. as well.
Why it matters: Worldwide, congestion costs hundreds of billions of dollars in annual productivity, pollutes cities, and burdens the planet with greenhouse gases. AI-driven traffic control doesn’t eliminate those impacts, but it can take the edge off. We’re thinking: Many traffic lights already are geared to prioritize passage of emergency vehicles, for example by recognizing patterns of flashing lights — but networked sensors stand to improve traffic routing globally.
Behavioral Cloning ShootoutNeural networks have learned to play video games like Dota 2 via reinforcement learning by playing for the equivalent of thousands of years (compressed into far less time). In new work, an automated player learned not by playing for millennia but by watching a few days’ worth of recorded gameplay. What’s new: Tim Pearce and Jun Zhu at Cambridge University trained an autonomous agent via supervised learning to play the first-person shooter Counter Strike: Global Offensive (CS:GO) by analyzing pixels. The model reached an intermediate level of skill. Check out a video presentation here. Key insight: Reinforcement learning can be used to teach neural networks to play games that include a programming interface, which enables the model to explore all possible game states because gameplay proceeds much faster than real time. CS:GO lacks such an interface. An alternative is to learn from expert demonstrations, a technique known as behavioral cloning. Where such demonstrations are hard to collect, publicly broadcast matches can stand in. How it works: The system generated a representation of each video frame using a convolutional neural network and combined multiple representations using a convolutional LSTM. A linear layer decided what action to take per frame.
Results: Pitted against the game’s built-in medium-difficulty agent, which takes advantage of information that humans don’t have access to (such as the positions of all players), the author’s system came out on top. It achieved 2.67 kills per minute and 1.25 kills per death, compared to the built-in agent’s 1.97 kills per minute and 1.00 kills per death. Against human players in the top 10 percent, it didn’t fare so well. It achieved 0.5 kills per minute and 0.26 kills per death compared to the human average of 4.27 kills per minute and 2.34 kills per death Why it matters: Behavioral cloning is a viable alternative to reinforcement learning — within the limits of available expert demonstrations. The authors’ system even learned the classic gamer swagger of jumping and spinning while it reloaded. We’re thinking: We’re in the mood for a nonviolent round of Splatoon.
A MESSAGE FROM DEEPLEARNING.AILearn how to automate a natural language processing task in “Build, Train, and Deploy ML Pipelines Using BERT,” Course 2 in our new Practical Data Science Specialization. You'll build an end-to-end pipeline using Hugging Face’s optimized implementation of the state-of-the-art BERT algorithm using Amazon SageMaker Pipelines. Enroll now
Home Sweet AI-Appraised HomeReal estate websites helped turn automated real-estate assessment into a classic AI problem. The latest approach by a leader in the field gets a boost from deep learning. What’s new: Zillow developed a neural network that predicts the value of homes across the United States. The system narrowed the error between earlier estimates and actual selling prices by 1 percent, achieving a median error rate of 6.9 percent. In addition to making it available online, Zillow plans to use it to improve its own real estate business. How it works: Zillow’s Zestimate system previously employed roughly 1,000 separate non-machine-learning algorithms, each tailored to a different local market. The new network estimates the value of 104 million dwellings nationwide, updated as frequently as daily.
Behind the news: Zillow has been tweaking Zestimate since 2006. The new neural network grew from a hackathon in which 3,800 teams from 91 countries competed for a $1 million prize. The winning team used a combination of deep learning and other machine learning techniques. The company incorporates machine learning into other aspects of its business as well, Zillow vice president of AI Jasjeet Thind said in an interview for DeepLearning.AI’s Working AI series. For instance, the company is developing a natural language search system for parsing legal documents. Why it matters: Between inspections, negotiating a price, and filling out reams of paperwork, buying a home is a complex ordeal. A tool that helps buyers and sellers alike get a fair price could be a big help. We’re thinking: How much does a GPU rack add to the value of a home?
BugbotAn insect-sorting robot could help scientists grapple with the global biodiversity crisis. What’s new: An automated insect classifier sucks in tiny arthropods, classifies them, and maps their most important identifying features. It was developed by researchers at Karlsruhe Institute of Technology, Berlin Natural History Museum, Bavarian State Collection of Zoology, Sapienza University of Rome, and National University of Singapore. How it works: The bot integrates systems that transport insects in and out, snap photos of them, and process the images. A touch screen serves as the user interface and displays model output. The authors pretrained a VGG19 convolutional neural network on ImageNet and fine-tuned it using 4,325 images of insects plus augmentations.
Results: In testing, the system scored an average of 91.4 percent precision across all species — good but not up to the level of a human expert. Behind the news: This is just the latest use of AI in the time-consuming task of insect identification.
We’re thinking: If this system stopped working, someone would have to debug it.
A MESSAGE FROM DEEPLEARNING.AILearn how to counteract bias and unfair representations! Together with KPMG Ignition Tokyo and Machine Learning Tokyo, we invite you to “XAI: Learning Fairness with Interpretable Machine Learning” on July 8, 2021. Enroll now Work With Andrew Ng
Multiple Roles: Workera is a precision upskilling company that enables learners and organizations to identify, measure, interpret, and develop data and AI skills. We’re searching for a Head of Engineering, Head of Applied Science, IT & Security Manager, Software Engineer, Data Scientist, and Data Engineer. Check out all of our open roles on our career page. Apply here
VP of Marketing: Podcastle is looking for a marketer to shape its brand marketing system from the ground up. As an early member of our growing startup, you will play a strategic role in shaping the direction of the company. Apply here
Software Development Engineer (North America, Latin America): Landing AI is looking for a number of Software Development Engineers to develop scalable AI applications and deliver optimized inference software to the edge. A strong background in Docker, Kubernetes, network security, and cloud-based development is preferred. Apply here
Data Analyst (Latin America): Factored is looking for expert data analysts to analyze datasets, gain insights using descriptive modeling, and code nontrivial functions. A strong background in coding, specifically in Python or R, and excellent English skills are required. Apply here
Data Engineer (Latin America): Factored is looking for top data engineers with experience with data structures and algorithms, OS, computer networks and object-oriented programming. You must have experience in Python and excellent English skills. Apply here
Subscribe and view previous issues here.
Thoughts, suggestions, feedback? Please send to thebatch@deeplearning.ai. Avoid our newsletter ending up in your spam folder by adding our email address to your contacts list.
|