Pete Warden: How screen scraping and TinyML can turn any dial into an API

Pete Warden: How screen scraping and TinyML can turn any dial into an API. “I’ve already heard from multiple teams who have legacy hardware that they need to monitor, in environments as varied as oil refineries, crop fields, office buildings, cars, and homes. Some of the devices are decades old, so until now the only option to enable remote monitoring and data gathering was to replace the system entirely with a more modern version. This is often too expensive, time-consuming, or disruptive to contemplate. Pointing a small, battery-powered camera instead offers a lot of advantages. Since there’s an air gap between the camera and the dial it’s monitoring, it’s guaranteed to not affect the rest of the system, and it’s easy to deploy as an experiment, iterating to improve it.”

The Guardian: How technology unlocked the secretive power of ‘Queen’s consent’

The Guardian: How technology unlocked the secretive power of ‘Queen’s consent’. “Have you ever right-clicked on a webpage and pressed the ‘View Page Source’ button? You’ll see the HTML building blocks: the mark-up incantations used to build the page on your screen. The HTML focuses on presentation: what colour that text should be, how big that image should be, and so on. Web scraping is the art of transforming this semi-structured soup back into the structured data that produced it – in this case, who was speaking in which chamber at what time, and what did they say.”

ZDNet: Facebook link preview feature used as a proxy in website-scraping scheme

ZDNet: Facebook link preview feature used as a proxy in website-scraping scheme. “The technique consisted of using Facebook developer accounts to place calls to Facebook or Facebook Messenger API servers, requesting a link preview for pages a group wanted to scrape. Facebook would fetch the data, assemble it in a link preview, and return it to the data scrappers as an API response, ready to be ingested into the scrapper’s database.” Pretty sure they mean scrapers, but I’m not going to argue with ZDNet.

Washington Post: Chinese firm harvests social media posts, data of prominent Americans and military

Washington Post: Chinese firm harvests social media posts, data of prominent Americans and military. “Biographies and service records of aircraft carrier captains and up-and-coming officers in the U.S. Navy. Real-time tweets originating from overseas U.S. military installations. Profiles and family maps of foreign leaders, including their relatives and children. Records of social media chatter among China watchers in Washington. Those digital crumbs, along with millions of other scraps of social media and online data, have been systematically collected since 2017 by a small Chinese company called Shenzhen Zhenhua Data Technology for the stated purpose of providing intelligence to Chinese military, government and commercial clients, according to a copy of the database that was left unsecured on the Internet and retrieved by an Australian cybersecurity consultancy.”

Technical .ly: Volunteer data scrapers helped Philadelphia Lawyers for Social Equity preserve client court records

Technical .ly: Volunteer data scrapers helped Philadelphia Lawyers for Social Equity preserve client court records. “As the first state to implement the Clean Slate Law in 2018, Pennsylvania committed to sealing millions of criminal records. The law was enacted to remove educational and vocational disadvantages for people with eligible records, including those associated with certain misdemeanors and people found not guilty in court. While the law cleared barriers to housing, education and employment for individuals across the state, it indirectly created new technological barriers for Philadelphia Lawyers for Social Equity (PLSE).”

Towards Data Science: How to Scrape Tweets From Twitter

Towards Data Science: How to Scrape Tweets From Twitter. “This tutorial is meant to be a quick straightforward introduction to scraping tweets from Twitter in Python using Tweepy’s Twitter API or Dmitry Mottl’s GetOldTweets3. To provide direction for this tutorial I decided to focus on scraping through two avenues: scraping a specific user’s tweets and scraping tweets from a general text search.”

Hackaday: Think You Know cURL? Care To Prove It?

Hackaday: Think You Know cURL? Care To Prove It?. “Do you happen to remember a browser-based game ‘You Can’t JavaScript Under Pressure’? It presented coding tasks of ever-increasing difficulty and challenged the player to complete them as quickly as possible. Inspired by that game, [Ben Cox] re-implemented it as You Can’t cURL Under Pressure!”

Make Tech Easier: How to Use a Data-Scraping Tool to Extract Data from Webpages

Make Tech Easier: How to Use a Data-Scraping Tool to Extract Data from Webpages. “If you’re copying and pasting things off webpages and manually putting them in spreadsheets, you either don’t know what data scraping (or web scraping) is, or you do know what it is but aren’t really keen on the idea of learning how to code just to save yourself a few hours of clicking. Either way, there are a lot of no-code data-scraping tools that can help you out, and Data Miner’s Chrome extension is one of the more intuitive options.”

Codementor: How to Extract Google Maps Coordinates

Codementor: How to Extract Google Maps Coordinates. “Have you ever thought you can make money by knowing how many restaurants there are in a square mile? There is no free lunch, however, if you know how to use Google Maps, you can extract and collect restaurant’s GPS and store them in your own database. With that information on hand and some math calculations, you are off to creating a big data online service. In this article, I will show you how to quickly extract Google Maps coordinates with a simple and easy method.”

MakeUseOf: The Scrapestack API Makes It Easy to Scrape Websites for Data

MakeUseOf: The Scrapestack API Makes It Easy to Scrape Websites for Data. “Finding it time-consuming to visit all your favorite websites and read everything that matters? One solution is a web scraper, a software tool that gathers information you need from other sites. We’re going to look at the scrapestack API, a web scraping service that you can subscribe to. Once set up, you can use scrapestack to grab whatever data you want from other sites.”

Ars Technica: Web scraping doesn’t violate anti-hacking law, appeals court rules

Ars Technica: Web scraping doesn’t violate anti-hacking law, appeals court rules. “Scraping a public website without the approval of the website’s owner isn’t a violation of the Computer Fraud and Abuse Act, an appeals court ruled on Monday. The ruling comes in a legal battle that pits Microsoft-owned LinkedIn against a small data-analytics company called hiQ Labs.”