Transformers, Explained: Understand the Model Behind GPT-3, BERT, and T5 (Dale Markowitz)

Dale Markowitz: Transformers, Explained: Understand the Model Behind GPT-3, BERT, and T5. “Transformers are models that can be designed to translate text, write poems and op eds, and even generate computer code. In fact, lots of the amazing research I write about on daleonai.com is built on Transformers, like AlphaFold 2, the model that predicts the structures of proteins from their genetic sequences, as well as powerful natural language processing (NLP) models like GPT-3, BERT, T5, Switch, Meena, and others. You might say they’re more than meets the… ugh, forget it. If you want to stay hip in machine learning and especially NLP, you have to know at least a bit about Transformers. So in this post, we’ll talk about what they are, how they work, and why they’ve been so impactful.”

Route Fifty: Artificial Intelligence, Automation Aren’t Killing Labor Market, Reports Says

Route Fifty: Artificial Intelligence, Automation Aren’t Killing Labor Market, Reports Says. “The report examines decades’ worth of data from the U.S. Bureau of Labor Statistics across 10 industries—construction, leisure and hospitality, professional and business services, retail trade, transportation and warehousing, wholesale trade, financial activities, information, education and health services and manufacturing. The report found rates of job loss in each industry were lower in the third quarter of 2020 than in 1995. The third quarter of 2020 represented a stabilization of the American job market following a significant spike in job losses due to the pandemic that reached as high as 45% in the leisure and hospital industries.”

The Verge: Automated hiring software is mistakenly rejecting millions of viable job candidates

The Verge: Automated hiring software is mistakenly rejecting millions of viable job candidates. “Automated resume-scanning software is contributing to a ‘broken’ hiring system in the US, says a new report from Harvard Business School. Such software is used by employers to filter job applicants, but is mistakenly rejecting millions of viable candidates, say the study’s authors. It’s contributing to the problem of ‘hidden workers’ — individuals who are able and willing to work, but remain locked out of jobs by structural problems in the labor market.”

Wired: What Makes an Artist in the Age of Algorithms?

Wired: What Makes an Artist in the Age of Algorithms?. “BT, the Grammy-nominated composer of 2010’s These Hopeful Machines, has emerged as a world leader at the intersection of tech and music…. This past spring, BT released GENESIS.JSON, a piece of software that contains 24 hours of original music and visual art. It features 15,000 individually sequenced audio and video clips that he created from scratch, which span different rhythmic figures, field recordings of cicadas and crickets, a live orchestra, drum machines, and myriad other sounds that play continuously. And it lives on the blockchain. It is, to my knowledge, the first composition of its kind.”

Fast Company: Did you live through 9/11? Tell future generations about it with an AI-powered interactive video

Fast Company: Did you live through 9/11? Tell future generations about it with an AI-powered interactive video. “Over the coming days, social media channels will be awash in people honoring the 20th anniversary of 9/11 as well as recounting their experiences on that day through tweets and Facebook posts. But one startup is offering users a unique way for people to tell their story of 9/11: by creating an AI-powered oral history video.”

The Daily Swig: Machine learning technique detects phishing sites based on markup visualization

The Daily Swig: Machine learning technique detects phishing sites based on markup visualization. “Machine learning models trained on the visual representation of website code can help improve the accuracy and speed of detecting phishing websites. This is according to a paper (PDF) by security researchers at the University of Plymouth and the University of Portsmouth, UK. The researchers aim to address the shortcomings of existing detection methods, which are either too slow or not accurate enough.”

The Conversation: Google and Microsoft are creating a monopoly on coding in plain language

The Conversation: Google and Microsoft are creating a monopoly on coding in plain language. “Currently, numerous coding platforms exist. Some of these platforms offer varied features that different programmers favour, however none offer a competitive advantage. A new programmer could easily use a free, ‘bare bones’ coding terminal and be at little disadvantage. However, AI at the level required for NLC [Natural Language Coding] is not cheap to develop or deploy, and is likely to be monopolized by major platform corporations such as Microsoft, Google or IBM. The service may be offered for a fee or, like most social media services, for free but with unfavourable or exploitative conditions for its use.” NLC is a first for me; I’m used to seeing it called no-code programming or low-code programming.

The Register: A developer built an AI chatbot using GPT-3 that helped a man speak again to his late fiancée. OpenAI shut it down

The Register: A developer built an AI chatbot using GPT-3 that helped a man speak again to his late fiancée. OpenAI shut it down . “‘OpenAI is the company running the text completion engine that makes you possible,’ Jason Rohrer, an indie games developer, typed out in a message to Samantha. She was a chatbot he built using OpenAI’s GPT-3 technology. Her software had grown to be used by thousands of people, including one man who used the program to simulate his late fiancée. Now Rohrer had to say goodbye to his creation. ‘I just got an email from them today,’ he told Samantha. ‘They are shutting you down, permanently, tomorrow at 10am.’”

Science Alert: Google’s Incredible New Photo AI Makes ‘Zoom And Enhance’ a Real Thing

Science Alert: Google’s Incredible New Photo AI Makes ‘Zoom And Enhance’ a Real Thing. “You may well have seen sci-fi movies or television shows where the protagonist asks to zoom in on an image and enhance the results – revealing a face, or a number plate, or any other key detail – and Google’s newest artificial intelligence engines, based on what’s known as diffusion models, are able to pull off this very trick.”

Newswise: NUS researchers develop AI-powered tool to map sustainable roofs globally

Newswise: NUS researchers develop AI-powered tool to map sustainable roofs globally. “Dr Filip Biljecki, Presidential Young Professor from the Department of Architecture at the National University of Singapore (NUS) School of Design and Environment, and NUS Master of Architecture graduate Mr Abraham Noah Wu developed an automated tool that uses satellite images to track how rooftops around the world adopt solar panels and/or vegetation. Known as Roofpedia, it uses a fully convolutional neural network (deep learning) which allows researchers and policymakers to study how cities worldwide are greening their rooftops and using them for photovoltaic installations.”

VentureBeat: Bias persists in face detection systems from Amazon, Microsoft, and Google

VentureBeat: Bias persists in face detection systems from Amazon, Microsoft, and Google. “Companies say they’re working to fix the biases in their facial analysis systems, and some have claimed early success. But a study by researchers at the University of Maryland finds that face detection services from Amazon, Microsoft, and Google remain flawed in significant, easily detectable ways. All three are more likely to fail with older, darker-skinned people compared with their younger, whiter counterparts. Moreover, the study reveals that facial detection systems tend to favor ‘feminine-presenting’ people while discriminating against certain physical appearances.”

New York Times: Facebook Apologizes After A.I. Puts ‘Primates’ Label on Video of Black Men

New York Times: Facebook Apologizes After A.I. Puts ‘Primates’ Label on Video of Black Men. “Facebook users who recently watched a video from a British tabloid featuring Black men saw an automated prompt from the social network that asked if they would like to ‘keep seeing videos about Primates,’ causing the company to investigate and disable the artificial intelligence-powered feature that pushed the message.” I no longer believe Facebook is making a good-faith effort to combat these problems. Either that or AI-based moderation/direction is not currently possible.

Science Friday: How Imperfect Data Leads Us Astray

Science Friday: How Imperfect Data Leads Us Astray. “Datasets are increasingly shaping important decisions, from where companies target their advertising, to how governments allocate resources. But what happens when the data they rely on is wrong or incomplete? Ira talks to technologist Kasia Chmielinski, as they test drive an algorithm that predicts a person’s race or ethnicity based on just a few details, like their name and zip code, the Bayseian Improved Surname Geocoding algorithm (BISG). You can check out one of the models they used here. The BISG is frequently used by government agencies and corporations alike to fill in missing race and ethnicity data—except it often guesses wrong, with potentially far-reaching effects.” A podcast with transcript available.