Dear friends,
How much math do you need to know to be a machine learning engineer? It’s always nice to know more math! But there’s so much to learn that, realistically, it’s necessary to prioritize. Here are some thoughts about how you might go about strengthening your math background.
For instance, in an earlier era of machine learning, linear algebra libraries for solving linear systems of equations (for linear regression) were immature. I had to understand how these libraries worked so I could choose among different libraries and avoid numerical roundoff pitfalls. But this became less important as numerical linear algebra libraries matured. Deep learning is still an emerging technology, so when you train a neural network and the optimization algorithm struggles to converge, understanding the math behind gradient descent, momentum, and the Adam optimization algorithm will help you make better decisions. Similarly, if your neural network does something funny — say, it makes bad predictions on images of a certain resolution, but not others — understanding the math behind neural network architectures puts you in a better position to figure out what to do.
Keep learning! Andrew
NewsAI Sees Race in X-RaysAlgorithms trained to diagnose medical images can recognize the patient’s race — but how? What’s new: Researchers from Emory University, MIT, Purdue University, and other institutions found that deep learning systems trained to interpret x-rays and CT scans also were able to identify their subjects as Asian, Black, or White. What they found: Researchers trained various implementations of ResNet, DenseNet, and EfficientNet on nine medical imaging datasets in which examples were labeled Asian, Black, or White as reported by the patient. In tests, the models reliably recognized the race, although their performance varied somewhat depending on the type of scan, training dataset, and other variables.
Behind the news: Racial bias has been documented in some medical AI systems.
Why it matters: The fact that diagnostic models recognize race in medical scans is startling. The mystery of how they do it only adds fuel to worries that AI could magnify existing racial disparities in health care. We’re thinking: Neural networks can learn in ways that aren’t intuitive to humans. Finding out how medical imaging algorithms learn to identify race could help develop less biased systems — and unlock other mysteries of machine learning. Sharper AttentionSelf-attention enables transformer networks to track relationships between distant tokens — such as text characters — in long sequences, but the computational resources required grow quadratically with input size. New work aims to streamline the process by rating each token’s relevance to the task at hand. What’s new: Sainbayar Sukhbaatar and colleagues at Facebook proposed Expire-Span, which enables attention to ignore tokens that aren’t useful to the task at hand. Key insight: Depending on the task, some tokens affect a model’s performance more than others. For instance, in predicting the sentiment of the sentence, “Then she cried,” “cried” is more important than “then.” By forgetting less relevant tokens, attention can process longer sequences with less computation. How it works: The authors modified a transformer’s attention layers. They trained the model in typical fashion to predict the next character in a sequence using the enwik8 dataset of text from English Wikipedia. Given the first token, it predicted the next. Then, using the first two tokens, it predicted the next, and so on.
Results: The authors evaluated Expire-Span based on total memory usage, training time per batch, and bits per byte (a measure of how well the model predicted the next token; lower is better). On enwik8, it achieved 1.03 bits per byte, while Adaptive-Span achieved 1.04 bits per byte and compressive transformer achieved 1.05 bits per byte. The authors’ model used 25 percent less GPU memory than the other two approaches (15GB versus 20GB and 21GB respectively). It also took less time to train (408ms per batch of 512 tokens compared to 483ms and 838ms). Why it matters: Forgetting the least relevant information enables transformers to process longer sequences in less time and memory. We’re thinking: Q: What do you do if a transformer forgets too much? A: Give it an Optimus Primer.
A MESSAGE FROM FACTOREDFactored, a sister company of DeepLearning.AI that helps ambitious Silicon Valley-based companies build data science teams, is partnering with rigorously vetted machine learning engineers, data engineers, and data analysts to work on your projects. Learn more To Bee or Not to BeeInsects that spread pollen to fruiting plants are in trouble. A possible alternative: Robots. What’s new: Farmers in Australia and the U.S. are using robots from Israeli startup Arugga Farming to pollinate greenhouse tomatoes, The Wall Street Journal reported. How it works: The system is designed for growing tomatoes, which self-pollinate when their pollen is stirred up by the beating of insect wings. Robots equipped with cameras, vision algorithms, and air compressors wheel themselves between rows of plants. When they recognize a flower that’s ready to produce fruit, they blast it with air to release its pollen.
Behind the news: A number of other companies are using AI-enabled robots to pollinate plants. Edete Precision Technologies has had success with almonds, and Bumblebee AI hopes to pollinate avocados, kiwis, and cocoa. Developed at West Virginia University, a robot called BrambleBee aims to pollinate blackberries, raspberries, and brambleberries. Why it matters: Robotic pollinators may prove to be an important technology outside of greenhouses. Climate change and habitat loss are ravaging Earth’s insect populations including bees. Meanwhile, such machines could be helpful to farmers: Bees are expensive to rent, they can spread plant diseases, and importing them is restricted in places such as Australia. We’re thinking: These robots are sure to generate a buzz. Fresh Funds for U.S. ResearchThe U.S. plans to build nearly a dozen new civilian AI research labs. What’s new: The U.S. National Science Foundation (NSF) committed $220 million to fund 11 National Artificial Intelligence Research Institutes, complementing seven other AI research institutes that were established last year. What’s happening: The NSF grants provide each institute about $20 million annually over five years. Some will receive additional funding from public and private partners such as the U.S. Department of Homeland Security, Amazon, and Intel. Their missions include:
Behind the news: The NSF funded an initial seven national AI institutes in September. Earlier, the U.S. had said it would spend $2 billion annually on AI over the next two years. Why it matters: Other governments spend much more on AI than the U.S., and this outlay is small in the scheme of national AI funding. However, the allocation and the goals to which it is being put suggest that the federal government recognizes AI’s importance to the U.S. economy and its potential to benefit the world at large. We’re thinking: U.S. government funding was critical to AI's rise. For example, the Defense Advanced Research Products Agency (DARPA) provided funds to both Andrew and Yann LeCun for deep learning research. We’re hopeful that these new programs will fund similarly valuable innovations. Work With Andrew Ng
Senior Software Development Engineers (Latin America): Landing Insight is looking for engineers to develop scalable AI applications and deliver optimized inference software. A strong background in cloud development and strong programming skills is preferred. Apply here
Software Development Engineer: Landing AI is looking for software development engineers to build scalable AI applications and deliver optimized inference software. A strong background in Docker, Kubernetes, infrastructure, network security, or cloud-based development is preferred. Apply in North America or Latin America.
Machine Learning Engineer (Customer Facing): Landing AI is looking for a machine learning engineer to work with internal and external engineers on novel models for customers. A solid background in machine learning and deep learning with proven ability to implement, debug, and deploy machine learning models is a must. Apply here
Head of Engineering: LandingInsight is looking for a head of engineering to manage and grow our world-class team. LandingInsight is developing a variety of products to assist ecommerce companies. Apply here
Learning Technologist: DeepLearning.AI seeks a technologist to guide and support learners across the platform. We’re looking for someone with a passion for online learning, teaching, and improving the learner experience. Apply here
Head of Engineering: Workera seeks a head of engineering to manage and grow its team. You will lead engineering execution and delivery, make Workera a rewarding place to work for engineers, and participate in company oversight. Apply here
Enterprise Lead Product Manager: Workera wishes to hire a lead product manager to develop its enterprise-grade precision upskilling platform and integrate it within the human resources and learning tech ecosystem. Apply here Subscribe and view previous issues here.
Thoughts, suggestions, feedback? Please send to thebatch@deeplearning.ai. Avoid our newsletter ending up in your spam folder by adding our email address to your contacts list.
|