How To: The Next Big Thing in Smartphone Cameras Is Machine Learning & It's Already Here
As smartphones become more accessible, billions of people have come to depend on their features for daily life. One of the most important aspects these days is the camera. OEMs have been working for years to improve camera quality, and they seem to have finally figured it out — the best way to achieve DSLR-quality photos wasn't just with better sensors, but with better intelligence.Smartphones manufacturers are limited in the hardware department because of a lack of options. Many key components have only one or two suppliers, which leads to most flagship devices sharing the same parts. This has caused OEMs to start looking for other ways to improve photography, and three industry leaders have found that machine learning provided the answer they sought.
What Is Machine Learning?To put it as simply as possible, machine learning (ML) is a type of programming that lets computers improve their capabilities without further human interaction. A human will program basic components of the system, then allow the computer to refine itself based on parameters that were initially set by the human programmer.Think of machine learning like electronic evolution — we give the system all the basic tools it needs to get started, then the system refines itself through trial and error. Again, though, this is a simplified explanation. If you'd like a more detailed summary, I highly recommend watching this video.As it pertains to smartphone photography, the main thing you should know about machine learning is that devices can use this technology to refine their object recognition skills and better decipher patterns in images. Some of the methods used for machine learning. Image by Luis Serrano/YouTube However, many OEMs are marketing machine learning as artificial intelligence when it's merely a component of AI. Artificial Intelligence is a broad concept that describes machines which function entirely independently of human beings while managing to think like human beings. This includes the ability to learn independently, analyze their surroundings without assistance, and make independent decisions. The components that make up artificial intelligence. Image by Raj Ramesh/YouTube Smartphones are only just beginning to become efficient at machine learning, but are far away from incorporating all components of artificial intelligence. Their learned behavior comes from data provided during production, so their learning capabilities are limited or nonexistent after the devices have been shipped to consumers. However, this learned behavior does greatly assist with tasks performed by smartphones, including photography.With machine learning, datasets can be incorporated into a smartphone that "teach" the phone what a professional-style photo looks like. With this information, a device can analyze the photos you capture or the images being viewed by the lens, then adjust the image to mimic a pro-style photo. And to process all this, a piece of hardware called an "NPU" was born.
NPU: A New Type of ProcessorA Neural Network Processing Unit (NPU) is a component installed on or alongside the SoC that operates independently of the other parts (such as the CPU and GPU). The NPU is designed specifically to analyze and process machine learning data. While the CPU and the GPU can process this type of data, the NPU can process it faster and use less power doing so. This makes the entire SoC more power efficient, as all data is processed by the most efficient component.Before NPUs, some companies would use cloud computing to process machine learning data. However, this limits performance to the strength of the internet connection and prevents any processing when the web isn't available. With the push for smartphones in emerging markets where the internet isn't widely available (see Google's Android One program), NPUs will need to be included in phones' SoCs to bring machine learning to all markets.Eventually, these NPUs could become powerful enough to use the information provided by users to continue learning. By analyzing how you use your device, phones could better optimize themselves to your specific tendencies and usage and creep us closer to artificial intelligence.However, the way an NPU is used will depend on the manufacturer. As you will see, the three companies who added NPUs to their flagship phones this year utilize the component differently. That said, camera improvements seem to be the central theme so far.
1: Google's Pixel Visual Core: Machine Learning ISPGoogle has gone against the grain with their camera this year. Instead of deploying dual cameras like most other flagship devices, they opted for a single lens. To compensate for the lack of a second lens, they integrated machine learning into the Pixel 2's camera.Google Camera is the default camera app for Pixel devices, and its HDR+ feature has become a famous example of machine learning being used to improve smartphone photography. The app takes a large number of photos each time you press the shutter button, then uses machine learning to combine these photos in a way that results in better dynamic range.Prior to the Pixel 2 and 2 XL, the Google Camera app was dependent on Qualcomm's Hexagon DSP (Qualcomm's off-the-shelf NPU) to process any machine learning. However, this year's Pixel phones included a dedicated, Google-built NPU called the Pixel Visual Core. Google's Pixel Visual Core. Image via Google Essentially, Google is using its NPU as an ML-powered Image Signal Processor (ISP). The Google Camera app calls on the Pixel Visual Core to perform machine learning tasks that enhance photos and videos. HDR+ is just one example of Google's usage of machine learning, but the Pixel 2 packs several more ML-powered camera enhancements, including a portrait mode with only one lens.With the Android 8.1 Oreo update, the Pixel Visual Core has been unlocked for third-party apps. Now, any app that properly utilizes Android's Camera2 API can take advantage of the machine learning enhancements made by the Pixel Visual Core. Put simply, when you take a picture in an app like Instagram, Google's HDR+ algorithm and other machine learning enhancements will be applied to your photo as if you captured it with the default Google Camera app. Images captured with the Instagram app on the Pixel 2 XL before (left) and after (right) the Pixel Visual Core was activated for third-party apps. Images by Dallas Thomas/Gadget Hacks Thanks to machine learning, the Pixel 2 has become the standard for smartphone photography, besting even the iPhone X and Galaxy Note 8 in many areas. In order to achieve this, however, Google had to design their own NPU.Since the Pixel 2 uses Qualcomm's pre-made Snapdragon 835 SoC, adding a custom NPU would be difficult. Qualcomm already has the Hexagon DSP, which performs a similar task. Also, the Snapdragon 835 is the SoC of choice for all but a handful of flagship devices, so Qualcomm would prefer not to make special additions to their SoC just for one manufacturer. Therefore, Google opted to design its own NPU which operates both independently and in conjunction with the Qualcomm SoC.However, the greatest aspect of the Pixel Visual Core is its ability to perform such high-level computing five times faster than the CPU performs the same processes — and it does this while using 1/10th of the power. The addition of the Visual Core is how the Pixel 2 was able to surpass its competition's (and its predecessor's) photography performance.
2: Apple's Neural Engine: Depth Sensing & Face IDApple's decision to implement an NPU — known as the Neural Engine — was the result of a need. When Apple designed the new iPhone X with an edge-to-edge display, they knew that Touch ID would be affected. With no space to house the fingerprint scanner, Apple innovated by adding Face ID, a new method for secure authentication.However, to achieve security and reliability in a system that relies on facial recognition, an enormous amount of data needed to be collected and analyzed. According to Apple, each time an iPhone X user unlocks their device, thirty thousand dots align themselves along the user's face using infrared light. These dots create a map, which is than analyzed and processed by the Neural Engine to verify the user. Face ID is one the new features reliant on Apple's Neural Engine. Image by The Verge/YouTube The Neural Engine has two cores that perform real-time processing with a theoretical output of 600 billion operations per second. It is contained within the Image Signal Processor inside of Apple's acclaimed A11 Bionic SoC, and assists with everything from Face ID to photography. The Neural Engine is a part of the A11 Bionic chipset found in the iPhone 8, 8 Plus, and X. Image by Parameswaran B/YouTube For photography, the Neural Engine is used to achieve Portrait Mode on the front-facing camera. Instead of the dual lens available with the rear camera, Apple uses the information gathered from the various sensors that comprise Face ID and sends it to the NPU. The NPU analyzes the data and uses a system called TrueDepth to achieve the bokeh effect.The Neural Engine also analyzes various objects while taking a photo to improve image compression without distorting your photos. It even assists in tracking and focusing on subjects to help you take faster photos. In short, Apple is using their NPU in many of the same ways that Google used theirs, except Apple's is built into the SoC.
3: Huawei's NPU: An All-Around ApproachHuawei's approach to machine learning attempts to enhance everyday tasks such as security and overall performance. The Huawei-built Kirin 970 chipset has a dedicated NPU for several different machine learning computations, and has made its debut in the Huawei Mate 10 and Mate 10 Pro.The NPU is comprised of five "engines" that handle specific functions: A Vision Engine, Experience Engine, Performance Engine, Power Management Engine, and third-party App Engine. These engines are applied to improve efficiency in various tasks, such as lowering power consumption while using augmented reality and instantaneously translating text via Microsoft Translate. Huawei's NPU is a part of the Kirin 970 chipset. Image via Huawei Like Apple and Google, however, the Kirin 970's NPU is largely used to improve photography. There are two specific areas where machine learning helps: Before you take the shot and after you take the shot.Before an image is captured, the NPU provides real-time object recognition and scene adjustment. The NPU can help identify up to thirteen different scenes and objects as they appear on your display, and you'll be notified of this recognition with an icon in the bottom-left corner of your display (while in Auto Mode).Based on the objects and scene, the NPU will automatically configure the camera app with the optimal settings for the picture you're about to take. Then, after you take the photo, the image's colors, contrast, brightness, and other features will be automatically adjusted to provide professional quality photos. Ideally, Huawei wants users to not have fumble around menus to take great photos. Instead, the NPU handles all of that for you, so you can take professional style photos without any adjustment beforehand.Smartphone photography has gone through eras that have typically been defined by a big feature being adopted by major OEMs. First, we had higher megapixels, then optical image stabilization, secondary cameras, and optical zoom. Now, we're entering the machine learning era of smartphone photography, and the NPU is quite literally at the core of it all.How do you feel about this new era in smartphone photography? What do you predict the next one will be? Let us know in the comments below.Don't Miss: All the Best New Features Coming in the Android 8.1 Oreo UpdateFollow Gadget Hacks on Facebook, Twitter, YouTube, and Flipboard Follow WonderHowTo on Facebook, Twitter, Pinterest, and Flipboard
Cover image and screenshots by Jon Knight/Gadget Hacks
Polaroid's answer to the masculine-fueled GoPro comes in the form of a tiny family-friendly square, fittingly named the Polaroid CUBE. Starting at a very modest $99.99 , the water-resistant action camera comes in all different colors, shoots HD video at 1080p, allows users to take 6MP pictures, and supports a microSD card of up to 32GB.
How to Record Your Sleep Patterns with Samsung Health — No
Protect Apps with a Secret Volume Button Code on Android
DIY Laser Light Show [Instructables How-To] You can impress all your friends and fans at your next show/DJ night/dance performance with laser effects that are quite simple to implement and don
Laser Light Tutorial - YouTube
How to root Samsung Galaxy Note 9 SM-N9600 With Odin Tool we provide instructions with pictures to root Samsung Galaxy Note 9 SM-N9600 and we discuss errors of rooting and solutions. in this post, we use CF Root. (but you can use other Possible Options.) and included all possible root apps and other methods (with PC / without PC) to root your
How to Root Samsung Galaxy Devices with CF-Auto-Root in Odin?
The quickest and easiest way to calibrate your display is to stare at a number of test patterns and use your monitor's onscreen display (OSD) controls to adjust the contrast, brightness, color
HDTV: Adjust the picture for best quality - consumerreports.org
Create a recovery drive and a system image which you can use to easily restore if Windows does not boot or becomes corrupt or hardware failure. Details. First step, launch Backup and Restore. Press Windows + X then click Control Panel. In Control Panel, click Backup and Restore (Windows 7) Under Control Panel Home, click Create a system image
How To: Add a Power Menu to Your iPhone for Quick Reboots, Resprings, Safe Mode, & More How To: Boot Your Galaxy S9 or S9+ into Safe Mode Cydia 101: How to Re-Enable a Semi-Tethered Jailbreak Cydia 101: How to Respring Your iPhone Without Losing Jailbreak Each Time
How to Add a "Reboot into Recovery" Option to Your Nexus 5's
How To: Recover deleted files from recycle bin w/free software How To: Set a password for a new user in Windows 7 How To: Recover the Admin password on a Microsoft Windows 7 PC How To: Recover deleted files and lost data from disk crashes How To: Retrieve a lost Windows password easily
Forgot or need to Reset Joomla Admin Password? [How to
Today, we take a quick look at how you could turn your old or spare android phone or android tablet into an IP webcam/network security camera. What You Need to Turn Your Android Phone & Tablet into a Security Camera • One Android phone or tablet with camera • Charging cables • Security camera apps for android
Turn Your Webcam Into A Security Camera - Laptop Mag
How To: It's Raining Sweets and Treats! How to Make Your Own Pneumatic Candy Cannon How To: Make a PVC Hand Pump to Move Water, Compress Air, & Create Vacuum How To: Build a high-powered air cannon that shoots rolled up T-shirts How To: Make a Super Cheap One-Way Check Valve
A Simple Compressed Air Cannon Made from PVC Pipes! Powerful
How To: It's Raining Sweets and Treats! How to Make Your Own Pneumatic Candy Cannon How To: Make a DIY front surface mirror (cheap, small & circular) How To: Dance the lindy hop to a five beat swing How To: Connect your sprinkler system to the water line
Building a Steampunk Hand Cannon, Part 2: How to Make and
Install Google's Chrome Cleanup Tool for free; The Cleanup Tool searches for and removes malware like BrowseFox, Vitruvian, V-Bates, and many more. All you have to do is download the Cleanup Tool to your desktop (or wherever you'd like), run the program, and it will hunt for malicious programs on Chrome and remove any that it finds.
This Tool Will Make Your Buggy Chrome Browser Run Like New Again
We've all been there. Your phone is on your nightstand, but you're on the couch in the living room browsing the web on your Nexus 7. Then you hear it—a faint sound for a new text message notification from the bedroom.
How to Send & Receive Text Messages Right from Your Nexus 7
