Professional piano players spend countless hours at the keys, perfecting their craft. For people with smaller hands, this dedication can take a physical toll. Repeated stretching to reach distant keys in a chord can strain muscles and joints and may lead to tendonitis, carpal tunnel syndrome, and other injuries.Researchers at Stanford are working to understand the forces involved in the hand movements of elite-level piano players, and how those change with different hand sizes and keyboard widths. On Dec. 5 at SIGGRAPH Asia 2024, Stanford Engineering researchers presented an AI-trained model that can recreate the hand movements required to play complicated pieces of music and the physical stresses those hands are under. This is the first step in an effort to reduce the risk of long-term injuries in piano players and test solutions – such as narrower keyboards – that could make piano playing more equitable for everyone.“We would never expect a world-class athlete to compete with equipment that does not fit their body. Yet we ask pianists, particularly women, to adapt to a one-size-fits-all design that was never built with them in mind,” said<a href="https://profiles.stanford.edu/elizabeth-schumann" data-extlink="">&nbsp;Elizabeth Schumann</a>, the Billie Bennet Achilles Director of Keyboard Studies at the&nbsp;<a href="https://humsci.stanford.edu/" data-extlink="">School of Humanities and Sciences</a> and co-author on the paper. “We can use research like this to reshape the future of performance and make it more sustainable.”&nbsp;<h2>Capturing elite performance</h2>The modern piano keyboard was designed in the 19th century with the average European man in mind. Today, an estimated 87% of adult women and 24% of adult men have hands that are smaller than ideal for the standard piano. Using traditional methods to study the impacts of smaller hands would require following a cohort of piano players for decades, and it could still be difficult to quantify injury risks or test solutions. Instead, the researchers have recorded the hand movements of professional piano players and are using artificial intelligence to predict how different sizes of hands would move to play new music.Schumann and her colleagues recruited 15 elite-level pianists to play a total of 10 hours of music while cameras filmed their hands from every angle. The researchers couldn’t put any sensors on the players without potentially interfering with their performance, so they used advanced computer vision techniques to combine the videos and reconstruct the players’ hand motions in three dimensions. They used additional processing to make sure that those movements were perfectly synced with the audio.“The quality of data that we were able to achieve is unprecedented,” said&nbsp;<a href="https://profiles.stanford.edu/c-karen-liu" data-extlink="">Karen Liu</a>, a professor of computer science at Stanford and lead author on the paper. “It’s not just that the reconstructed 3D motions are accurate and clean; the quality of the performance itself is truly remarkable. We were able to capture diverse movements from a group of talented musicians performing at a high level – I have not seen a dataset like that in the past.”Roucheng Wang, a graduate student in Liu’s lab, and Pei Xu, a postdoctoral researcher, used this dataset to train a model to accurately generate new piano-playing data. When they gave the model sheet music it had never seen, such as Beethoven’s “Für Elise,” the model could provide the correct 3D hand movements needed to play the piece, using a simulated piano to generate sound.“I was just so stunned at how accurately this model could simulate elite-level technique,” Schumann said. “It’s really incredible.”<iframe width="640" height="360" src="https://www.youtube.com/embed/sGQNCEl7pJQ?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>When researchers gave the AI-trained piano playing model sheet music it had never seen, such as Beethoven’s “Für Elise,” the model could provide the correct 3D hand movements needed to play the piece. | Ruocheng Wang<h2 id="isPasted">From robot joints to real muscles</h2>This model is just the first step for Liu, Schumann, and their colleagues. At the moment, the model can physically simulate hand movements, but it doesn’t simulate the muscles and tendons that create those movements or the strain that they are subjected to.“We are not at the biomechanics level yet, where the actuation of the hands is generated by muscles – currently our model is more like a robot that directly generates torque at joints,” Liu said. “Eventually, our model needs to be able to predict the tension level of muscles and the potential of injury.”The researchers are currently working to add these layers to their model so that they can use it to help piano players – especially those with small hands – avoid injuries. They are also adapting their work to model the movements of other dexterous musicians. They have applied some of the same techniques to an existing dataset of hand motions of highly skilled guitar players. Their results, which were also presented recently at SIGGRAPH Asia 2024, showed that they could successfully model the distinct left- and right-hand movements needed to play a variety of both popular and classical music.“With high-quality data, we could model the 3D movements needed for other types of music performance as well,” Liu said. “This isn’t a model that is replacing people – we’re working with musicians to help understand and solve problems. This project is about advancing people, and AI is just a tool for that.”

Stanford researchers collected the first large-scale 3D hand motion dataset containing 10 hours and 153 pieces of piano music performed by 15 elite-level pianists, along with synchronized audio and key pressing events. | The Movement Lab

Researchers at Stanford Engineering have developed an AI-trained model to accurately recreate the hand movements of elite-level pianists and the physical stresses they endure while playing.

AI could help reduce injury risk in pianists

Optical sensors are the “eyes” of systems in industrial applications. However, just as with humans, in industrial applications a powerful peripheral system must ensure that image processing permanently functions reliably, without errors, and at the required performance level.As optical sensors become increasingly powerful, the number of applications in which they can be used is also increasing. Along with the associated quality management and documentation system, the monitoring and evaluation electronics in state-of-the art production and documentation systems produce a flood of data. This mass of data must be transmitted, collected, and processed reliably. The underlying technical structures of machine vision systems must remain fully functional even under demanding conditions, such as permanent mechanical stress due to shocks and vibrations. Data connectors in particular, which are important interfaces within the system, also have to meet these requirements so that there is no wear on the male and female contacts (cf. fretting corrosion). Otherwise, contact interruptions will lead to downtimes in production or process errors that require time-consuming and cost-intensive measures.Increasingly compact systems in smaller spaces, following the trend of miniaturization, pose additional challenges for device developers of machine vision systems. These include electromagnetic interference pulses that can influence the functionality of the individual components. The reduced structures within the system architecture also jeopardize stable system integrity. The increasingly powerful components, installed increasingly closer together, also bring with them a thermal problem that may not only have an impact on the operation of the sensors, but may also make device approval more difficult. Miniaturized interfaces and device-integrated connection technology that provide secure shielding and data transmission are active solutions to these potential problems. The optimized positioning of components on the printed circuit board in accordance with electromagnetic and thermal considerations during device development also ensures reliability. Simulations of heat generation, signal integrity, and impedance are helpful, especially when non-standard components are used.&nbsp;<h2>Standards for optimized image processing</h2>There are numerous transmission standards, including Camera Link, CoaXPress, GigE Vision, and USB 3, to suit the respective machine vision applications and associated interfaces. All factors must be coordinated and work reliably together to ensure optimized image processing.&nbsp;Phoenix Contact is synonymous with high quality in the field of industrial connection technology. The portfolio of the company, founded in 1923, includes tried and tested data connectors such as D-SUB, RJ45, USB, HDMI, coaxial, and fiber-optic connection technology.&nbsp;With its commitment to quality and innovation, Phoenix Contact is also an expert partner for new components, such as the Single Pair Ethernet (SPE), RJ45 Industrial (RJ45-IND), and the connection system for shielded and unshielded Finepitch series board-to-board connectors.RJ45 IndustrialThe RJ45 Industrial (RJ45-IND) product family was developed especially for industrial applications. This powerful product family has been further developed and optimized based on a large amount of experience gained with the standard RJ45 products in cooperation with customers from different industries. With this development, Phoenix Contact has created a product range featuring an optimized 360° shielding concept, for example.To counter electromagnetic interference, the housing shield springs and shield pins for the printed circuit board have been raised. This also improves the mechanical strength of the socket on the printed circuit board.The connection between RJ45-IND connectors or patch cables and RJ45-IND female printed circuit board (PCB) connectors has been decisively optimized. Shocks and vibrations are now counteracted by the so-called “perfect match” of the connector system. The development also included optimizing the contact depth of the female PCB connector from 8.3 m to 9 mm, thus enlarging the contact zone of the connector system. There have also been decisive improvements in the tolerances in the connection of the socket and plug, meaning that their contact connection is even better matched. On the connector side, the contact pressure was increased further through the inclusion of a compression spring.&nbsp;The temperature resistance of the female PCB connector has been improved through the use of a reinforced housing, now utilizing fiberglass technology. Temperatures from -40 up to +105 C are now possible without any problems.&nbsp;The RJ45-IND patch cables, which are suitable for drag chain applications and robots, are also particularly robust. A major advantage and unique selling point of the Phoenix Contact solution is the IDC fast connection, which enables the one-piece connector to be assembled quickly and easily without the use of a special tool.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732577673789-Bild+3_RJ45+IND+rechts+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Single Pair EthernetThe IEEE 802.3 task force has developed the transmission standards for Single Pair Ethernet (SPE), defining the standards 10Base-T1, 100Base-T1, 1000Base-T1, and MultiGigBASE-T1.&nbsp;The SPE standards enable data rates of 10 Mbps up to 1 Gbps at distances of 15 to 1,000 m. The MultiGigBASE-T1 standard goes so far as to enable data rates of up to 10 Gbps at distances of up to 15 m. The power transmission for the SPE standards 100BASE-T1 and 1000BASE-T1 are defined by the IEEE 802.3 bu standard (PoDL – Power over Data Lines).&nbsp;The load outputs of the end devices (power devices) are divided into 10 performance classes from 0.5 to 50 W.&nbsp;With regard to industrial image processing, SPE is an exciting field of application not just because of the potential to transmit data and power. In particular, the SPE portfolio from Phoenix Contact plays to its strengths here with its compact and 50 W consistent design. For example, the SPE-IP20 socket saves about 50 percent of space compared to a standardized RJ45 jack.&nbsp;Phoenix Contact also has a consistent SPE product portfolio in which all components have been matched and tested. This portfolio provides compact IP20 connectors (in accordance with IEC&nbsp;63171-2) and IP67-protected M8 connectors (in accordance with IEC&nbsp;63171-5). A brand new addition to the portfolio is the SPE-M12 hybrid connector from the Onepair series.The advantages of the SPE connection technology include the uniform mating face and the possibility to achieve a high degree of common part usage. This means that adapters are no longer necessary and no additional costs are incurred.Various SPE components are available for the developers of machine vision sensors. These include new semiconductors (SPE PHYs and PoDL controllers), SPE magnetics, connection technology in accordance with IEC 63171-2 and -5, and SPE-compatible cables in accordance with IEC 61156.&nbsp;Tip: With a membership in the System Alliance (<a href="http://www.singlepairethernet.com/">www.singlepairethernet.com/</a>), it is possible to gain detailed information about Single Pair Ethernet and become familiar with specific solutions for industrial applications.Board-to-board connectorsContact systems are to be preferred over simply designed male and female strips, in particular in demanding applications – such as for industrial image processing, for example. They are sophisticated yet at the same time robust.&nbsp;With its board-to-board connectors, the Finepitch product series from Phoenix Contact offers a professional contact system for space-saving PCB connections. Various numbers of positions as well as stack heights provide a great deal of flexibility for developers. The system features high-speed data transmission rates of up to 52&nbsp;Gbps. The innovative ScaleX double contacts (FP&nbsp;0,8) are designed as hermaphroditic, meaning that vibration-resistant connections can be created in very confined spaces.The shielded version available as an option also prevents electromagnetic interferences.As a valuable service offering, Phoenix Contact experts provide support during the entire design-in process as well as design-in support. This means that device developers of machine vision systems can quickly obtain preconfigured 2D/3D data for their CAD software.&nbsp;Phoenix Contact also provides individual thermal and housing simulations based on the finite element method (FEM). Free simulations for data and signal integrity complete the comprehensive service offering. With this support, customers can realize stable communication right from the start and throughout the entire development process.&nbsp;Phoenix Contact rounds off its services with a free sample service and reliable logistics.&nbsp;<h2>Summary&nbsp;</h2>The developers of modern machine vision systems must consider various important factors during the product development process to ensure that reliable and high-performance systems are ready for the market in the environment of high data transmission rates and miniaturization. The cost factor also plays a decisive role. Phoenix Contact is a reliable partner, providing both products and services. In addition to the proven data connectors and innovative product solutions – such as for SPE, for example – Phoenix Contact provides simulations that help to realize stable device communication and reliable systems with cost-efficiency. This creates a high-performance periphery for reliable industrial image processing.Written by:&nbsp;Dipl.-Ing. Verena Neuhaus, Product Management Data Connectors, Business Unit Field Device Connectors, Phoenix Contact GmbH &amp; Co. KG, Blomberg, GermanyThomas Schulze, M.Sc., Product Management Board-to-Board Connectors, Business Unit PCB Connectors, Phoenix Contact GmbH &amp; Co. KG, Blomberg, Germany

Optical sensors are the “eyes” of industrial systems, relying on robust peripherals for reliable image processing. As sensors grow more powerful, challenges like data management, mechanical stress, and thermal issues in compact designs demand innovative solutions.

Image data in industrial applications: Ensuring reliability and performance during transmission

A less wasteful way to train large language models, such as the GPT series, finishes in the same amount of time for up to 30% less energy, according to a new study from the University of Michigan.The approach could save enough energy to power <a href="https://www.eia.gov/energyexplained/use-of-energy/electricity-use-in-homes.php" target="_blank" rel="noreferrer noopener">1.1 million U.S. homes</a> in 2026, based on Wells Fargo’s projections of <a href="https://www.marketwatch.com/story/ai-could-demand-a-shocking-amount-of-electricity-check-out-this-chart-e91e306d" target="_blank" rel="noreferrer noopener">AI power demand</a>. It could also take a bite out of the International Monetary Fund’s prediction that data centers could account for <a href="https://www.imf.org/en/Blogs/Articles/2024/08/15/carbon-emissions-from-ai-and-crypto-are-surging-and-tax-policy-can-help" target="_blank" rel="noreferrer noopener">1.2% of the world’s carbon emissions by 2027</a>—and the <a href="https://theconversation.com/ais-excessive-water-consumption-threatens-to-drown-out-its-environmental-contributions-225854">water demands</a> that come with that energy use.Some experts say that these costs could be outweighed by environmental benefits. They argue that AI could be a <a href="https://news.un.org/en/story/2023/11/1143187" target="_blank" rel="noreferrer noopener">“game changer” for fighting climate change</a> by identifying ways to optimize supply chains and the grid, manage our energy needs, and improve research on climate change. Still, that doesn’t excuse squandering energy, and some of the power used to train AI has zero impact on training time and model accuracy.“Why spend something when there’s no point?” said <a href="https://eecs.engin.umich.edu/people/chowdhury-mosharaf/" target="_blank" rel="noreferrer noopener">Mosharaf Chowdhury</a>, an associate professor of computer science and engineering and the corresponding author of the study presented at the <a href="https://sigops.org/s/conferences/sosp/2024/index.html" target="_blank" rel="noreferrer noopener">30th Symposium on Operating Systems Principles</a> last Monday.“We can’t keep building bigger and bigger data centers because we won’t have the power to run them,” said Chowdhury. “If we can reduce the energy consumed by AI, we can reduce AI’s carbon footprint and cooling requirements and allow for more computation to fit within our current energy constraints.”The energy waste is created when AI training is unequally divided between GPUs, which are computer processors specialized for large data and graphics applications. Although it opens the door for waste, splitting the work is necessary for processing huge datasets.“AI models today are so large, they cannot fit inside a single computer processor,” said <a href="https://jaewonchung.me/about" target="_blank" rel="noreferrer noopener">Jae-Won Chung</a>, a doctoral student in computer science and engineering and the first author of the study. “They need to be divided into tens of thousands of processors to be trained, but dividing the models in perfectly equal sizes across all processors is practically impossible.”The training jobs are so difficult to evenly split up because some tasks need to be grouped together on the same processor—like how each installment of a book series will be grouped together in an organized shelf. Depending on how the tasks are grouped, some processors might get stuck with the AI-training equivalent of the Encyclopedia Britannica while others get assigned a fantasy trilogy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1731630851359-AI_waste_Infographic-768x1470.jpg" style="width: 100%px;" class="fr-fic fr-dib">Infographic credit: Rachel Voigt, Michigan Engineering.Because current training methods run each processor at top speed, processors with a lighter load will finish their calculations before other processors. This doesn’t speed up training, which isn’t complete until every processor finishes its job—but it is wasteful because faster calculations require more energy. In addition, problems such as faulty hardware or network delays create energy waste by slowing down a single processor’s computing speed.To save energy, the researchers developed a software tool, called&nbsp;<a href="https://ml.energy/zeus/research_overview/perseus/" target="_blank" rel="noreferrer noopener">Perseus</a>, that identifies a critical path, or a series of subtasks that will take the longest time to complete. Then, Perseus slows down processors that aren’t on the critical path so that they all finish their jobs around the same time—eliminating unnecessary power use.“Reducing the power cost of AI can have important implications for equitable AI access,” said Chowdhury. “If a country doesn’t have enough power to run a big model, they might need to use services from far away, or be stuck running smaller, less accurate models. This gap could further perpetuate disparity between different communities.”The team tested Perseus by training GPT-3, three other large language models and one computer vision model.Perseus is an open-sourced tool available as part of&nbsp;<a href="https://ml.energy/zeus/" target="_blank" rel="noreferrer noopener">Zeus</a>, a tool for measuring and optimizing AI energy consumption.&nbsp;

Smarter use of processor speeds saves energy without compromising training speed and performance.

Up to 30 percent of the power used to train AI is wasted. Here's how to fix it.

Manufacturing defect rates have risen with metal fabrication facilities facing unprecedented challenges in maintaining quality control amid increasing operational complexity.&nbsp;As production lines become more sophisticated and quality standards more stringent, traditional inspection methods must be revised for modern manufacturing demands.Matroid is transforming this landscape by democratizing computer vision technology, making advanced automation accessible to industry professionals without coding expertise. Through its innovative platform, Matroid enables manufacturing teams to implement sophisticated defect detection and process monitoring solutions that can be deployed within minutes rather than months.The impact is significant in welding applications, where Matroid's computer vision technology can evaluate multiple quality parameters simultaneously - from porosity and spatter to burn-through - with the precision of a weld engineering expert.&nbsp;This scalable approach to quality control is proving essential as manufacturers face mounting pressure to maintain precision while managing costs.We will explore in detail how Matroid's computer vision technology is changing the landscape of metal manufacturing, examining its technical capabilities, practical applications, and the tangible benefits it brings to production environments. &nbsp;For decision-makers, we'll look at how this technology sets new standards in automated quality control and process optimization.<h2 dir="ltr">Industry Challenges and Opportunities in Metal Manufacturing</h2>The metal manufacturing industry faces unprecedented pressures, with many organizations' quality control costs consuming <a href="https://www.weboccult.com/blog/computer-vision-for-quality-control-in-manufacturing-a-comprehensive-guide">15-20%</a> of annual sales revenue. Traditional manual inspection methods, while common, are increasingly inadequate for modern manufacturing demands, showing inconsistency rates increase due to operator fatigue and subjective assessments.According to the US. Census Bureau <a href="https://www.census.gov/library/stories/2021/09/manufacturing-continues-to-be-among-top-five-largest-employment-sectors.html">study</a> shows that production facilities struggle with a critical shortage of skilled workers, with nearly one-fourth of the manufacturing workforce aged 55 or older. This demographic shift and early retirement trends post-pandemic have created significant gaps in quality control expertise and operational efficiency.<h3 dir="ltr">The Role of Automation and Computer Vision</h3>Computer vision technology is emerging as a transformative solution, capable of analyzing images and video from any imaging technology, spanning resolutions from low (VGA) to ultra-high (gigapixel) and across various spectra, including visible light, infrared, and x-ray. &nbsp;This enables the detection of the smallest and most challenging defects in manufacturing processes, enhancing quality control and operational efficiency.Unlike traditional rule-based inspection systems, modern computer vision &amp; AI solutions can detect and classify multiple defect types simultaneously, from surface imperfections to welding anomalies.Automated inspection systems are valuable in welding applications, where they can evaluate multiple quality parameters - including porosity, spatter, burn-through, edge welds, and cold welds - with consistent precision.&nbsp;Computer vision &amp; AI address the skilled labor shortage and significantly improve inspection accuracy, with some implementations showing up to <a href="https://cloud.google.com/blog/products/ai-machine-learning/improve-manufacturing-quality-control-with-visual-inspection-ai">10x</a> improvement in defect detection compared to conventional methods, which leads to dramatic cost savings in cost per part and rework efforts and also improves production throughput.<h2 dir="ltr">Matroid's Vision and Technological Focus</h2>Founded in 2016 by Stanford professor and founding member of Databricks Reza Zadeh, Matroid has established itself as a leader in enterprise-level computer vision &amp; AI software platform, backed by <a href="https://growjo.com/company/Matroid">$33.5M</a> in funding from prominent investors, including NEA, Energize Ventures, and Intel Capital.&nbsp;The Matroid <a href="https://www.matroid.com/">platform</a> domain experts from manufacturing to rapidly deploy and adapt custom detectors that can leverage camera systems' imagery/ video types to detect and classify defects, monitor manual assembly operations, verify SOPs are followed, and provide continuous cycle-time analysis with unprecedented precision across various production environments.<h3 dir="ltr">Custom Detectors and Real-Time Monitoring</h3>Matroid's platform distinguishes itself through its no-code software to build and deploy detector systems, allowing manufacturing teams to develop specialized detection models through an intuitive point-and-click interface.&nbsp;These vision detectors can simultaneously evaluate multiple quality parameters in metal processing applications - from weld porosity and spatter to surface defects and assembly errors - while maintaining consistent accuracy across varying conditions.The platform's camera-agnostic system integrates seamlessly with existing infrastructure, supporting everything from standard industrial cameras to specialized equipment like borescopes, microscopes, and X-ray systems.&nbsp;Real-time alerts and analytics are delivered directly with production systems (PLCs, SCADA, MES, ERP) and mobile devices. This enables immediate response to the right people and systems at the right time to detect anomalies and maintain digital traceability throughout manufacturing.<h2 dir="ltr">How Matroid’s Computer Vision Powers Welding Applications</h2><iframe src="https://www.youtube.com/embed/20FpAGGAqmk?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" style="width: 640px; height: 460px;"></iframe> <h3 dir="ltr">Detecting Defects in Metal Manufacturing</h3>Matroid's computer vision enables quality control oversight across the entire metal manufacturing process. The <a href="https://www.matroid.com/automated-visual-anomoly-detection/">system</a> identifies structural defects at the casting stage, like cracks, porosity, cold flow, and hot tears.&nbsp;Within the stamping processes, Matroid detectors are used to replace manual visual inspection by automating the inspection of stamped parts for tears and splits. &nbsp;This gives near real-time feedback to production engineers on how the presses are performing and gives insights into potential adjustments to prevent further defects.&nbsp;In the final inspection, the platform performs comprehensive checks to ensure consistently high quality, effectively reducing returns and warranty claims.<h3 dir="ltr">Computer Vision in Welding</h3>Matroid's technology monitors operations such as spot welding, MIG welding, and continuous seam welding through real-time monitoring and instant defect detection. The system evaluates multiple quality parameters simultaneously, identifying common issues like porosity, spatter, cold welds, edge welds, and burn-through, just as a weld engineering expert would.&nbsp;The platform's deep learning capabilities enable it to detect defects as a person could through visual media (images/video). The solutions are designed for rugged &amp; dynamic manufacturing environments.&nbsp;<h3 dir="ltr">Root Cause Analysis and Real-Time Adjustments</h3>Matroid's <a href="https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control">solution</a> provides detailed defect classification and analysis through its integrated system. The system immediately alerts operators when issues are detected through direct integration with production systems, QMS, and mobile devices. &nbsp;Production teams can take corrective action when they have near real-time classification information and analytics; for example, when burn-through is detected on welds, the welding engineer can make more effective adjustments to robotic welding systems faster. &nbsp;This reduces rework and defect risks and improves production throughput.&nbsp;The platform operates 24/7, providing continuous monitoring and real-time insights into production operations. Through its historical data analysis capabilities, manufacturing teams can capture actionable insights and implement zero-defect initiatives.<h2 dir="ltr">Real-World Applications and Benefits&nbsp;</h2><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1731927333487-1731927333487.png" id="isPasted" class="fr-fic fr-dii"><h3 dir="ltr" id="isPasted">Traditional Methods vs. Computer Vision &amp; AI for Welding</h3>Traditional welding inspection methods, relying on manual or basic automated systems, provide only binary pass/fail results. These systems need help to meet modern manufacturing demands, where quality teams need detailed defect analysis and real-time process adjustments.&nbsp;Manual inspections introduce delays between defect occurrence and detection, while inspector fatigue and varying experience levels lead to inconsistent results. Computer vision &amp; AI solutions powered by deep learning technology transform this landscape by enabling real-time identification and classification of specific weld defects.&nbsp;Matroid's system automatically detects and categorizes issues such as porosity, burn-through, and spatter, immediately notifying weld engineers for rapid response. This immediate feedback loop allows manufacturing teams to quickly identify root causes and implement real-time process adjustments.The deployment of edge or on-premises installations ensures minimal latency for real-time monitoring while maintaining data security. This approach enables manufacturing teams to shift from reactive quality control to proactive process optimization.&nbsp;Through consistent quality assessment and automated defect classification, companies can reduce inspection time, minimize waste, and improve overall welding quality. The system's integration with existing production processes creates a seamless transition from traditional inspection methods to advanced computer vision &amp; AI solutions.<h2 dir="ltr">The Deployment of Matroid’s Solution</h2><h3 dir="ltr">Easy Integration and Deployment</h3>Matroid's computer vision &amp; AI platform integrates seamlessly with existing manufacturing infrastructure through multiple deployment options. The system can be implemented as downloadable software for on-premises installation or as a Cloud software service, allowing manufacturers to choose the deployment method that best suits their operational needs.&nbsp;This flexibility enables manufacturing teams to maintain data security while leveraging advanced detection capabilities. &nbsp;Matroid’s platform makes integration into various control systems, MES, QMS, ERP, and databases easy so that production can work with real-time data and has enhanced traceability of the products produced.&nbsp;<h3 dir="ltr">Compatibility with Multiple Camera Technologies</h3>The platform's camera-agnostic approach sets it apart in the manufacturing sector. Matroid works with existing and new vision systems, supporting any image-or-video-based camera regardless of resolution (VGA - giga-pixel) or spectra (visible, infrared, X-ray, etc.).This allows manufacturers to reuse their existing machine vision cameras and use the right imaging technology for the application at hand while maintaining the ability to detect visual defects. The system connects effortlessly with industrial cameras, supporting continuous 24/7 monitoring and real-time insights into manufacturing processes.<h2 dir="ltr">Future of Welding with Computer Vision &amp; AI</h2>Integrating computer vision technology in welding operations represents a significant leap forward in manufacturing quality control. Matroid's platform transforms traditional inspection methods by enabling real-time defect detection and detailed classification, which enables immediate process adjustments.&nbsp;By combining deep learning capabilities with enterprise-grade deployment options, manufacturers can achieve consistent quality while reducing operational costs. The platform's ability to empower domain experts in manufacturing, work with existing infrastructure, and support multiple camera technologies makes it a practical solution for facilities of all sizes.<h2 dir="ltr">Ready to make your welding operations efficient and reliable with advanced computer vision?</h2>Discover how Matroid's platform can enhance your quality control processes and drive real-time improvements. Visit Matroid's <a href="https://www.matroid.com/">website</a> to explore their innovative solutions for metal manufacturing.<h2 dir="ltr">References</h2><ol><li dir="ltr"><a href="https://info.matroid.com/visual-inspection-computer-vision-software">https://info.matroid.com/visual-inspection-computer-vision-software</a></li><li dir="ltr"><a href="https://www.dayjob.com.au/metal-fabrication-challenges-with-sustainability/">https://www.dayjob.com.au/metal-fabrication-challenges-with-sustainability/</a></li><li dir="ltr"><a href="https://www.matroid.com/automated-visual-anomoly-detection/">https://www.matroid.com/automated-visual-anomoly-detection/</a></li><li dir="ltr"><a href="https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control">https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control</a></li><li dir="ltr"><a href="https://www.weboccult.com/blog/computer-vision-for-quality-control-in-manufacturing-a-comprehensive-guide">https://www.weboccult.com/blog/computer-vision-for-quality-control-in-manufacturing-a-comprehensive-guide</a></li><li dir="ltr"><a href="https://cloud.google.com/blog/products/ai-machine-learning/improve-manufacturing-quality-control-with-visual-inspection-ai">https://cloud.google.com/blog/products/ai-machine-learning/improve-manufacturing-quality-control-with-visual-inspection-ai</a></li><li dir="ltr"><a href="https://www.matroid.com/automated-visual-anomoly-detection/">https://www.matroid.com/automated-visual-anomoly-detection/</a></li><li dir="ltr"><a href="https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control">https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control</a></li></ol>

Discover how computer vision is solving quality challenges in metal manufacturing through enhanced defect detection, proactive process monitoring, and flexible integration solutions.

AI-Powered Welding: How Computer Vision Solutions Drive Precision and Efficiency in Metal Manufacturing

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries. There are also full-sequence diffusion models like Sora, which convert words into dazzling, realistic visuals by successively “denoising” an entire video sequence.Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have proposed a simple change to the diffusion training scheme that makes this sequence denoising considerably more flexible.When applied to fields like computer vision and robotics, the next-token and full-sequence diffusion models have capability trade-offs. Next-token models can spit out sequences that vary in length. However, they make these generations while being unaware of desirable states in the far future — such as steering its sequence generation toward a certain goal 10 tokens away — and thus require additional mechanisms for long-horizon (long-term) planning. Diffusion models can perform such future-conditioned sampling, but lack the ability of next-token models to generate variable-length sequences.Researchers from CSAIL want to combine the strengths of both models, so they created a sequence model training technique called “Diffusion Forcing.” The name comes from “Teacher Forcing,” the conventional training scheme that breaks down full sequence generation into the smaller, easier steps of next-token generation (much like a good teacher simplifying a complex concept).<iframe width="640" height="360" src="https://www.youtube.com/embed/m4rhzR-h-JU?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Diffusion Forcing found common ground between diffusion models and teacher forcing: They both use training schemes that involve predicting masked (noisy) tokens from unmasked ones. In the case of diffusion models, they gradually add noise to data, which can be viewed as fractional masking. The MIT researchers’ Diffusion Forcing method trains neural networks to cleanse a collection of tokens, removing different amounts of noise within each one while simultaneously predicting the next few tokens. The result: a flexible, reliable sequence model that resulted in higher-quality artificial videos and more precise decision-making for robots and AI agents.By sorting through noisy data and reliably predicting the next steps in a task, Diffusion Forcing can aid a robot in ignoring visual distractions to complete manipulation tasks. It can also generate stable and consistent video sequences and even guide an AI agent through digital mazes. This method could potentially enable household and factory robots to generalize to new tasks and improve AI-generated entertainment.“Sequence models aim to condition on the known past and predict the unknown future, a type of binary masking. However, masking doesn’t need to be binary,” says lead author, MIT electrical engineering and computer science (EECS) PhD student, and CSAIL member Boyuan Chen. “With Diffusion Forcing, we add different levels of noise to each token, effectively serving as a type of fractional masking. At test time, our system can “unmask” a collection of tokens and diffuse a sequence in the near future at a lower noise level. It knows what to trust within its data to overcome out-of-distribution inputs.”In several experiments, Diffusion Forcing thrived at ignoring misleading data to execute tasks while anticipating future actions.When implemented into a robotic arm, for example, it helped swap two toy fruits across three circular mats, a minimal example of a family of long-horizon tasks that require memories. The researchers trained the robot by controlling it from a distance (or teleoperating it) in virtual reality. The robot is trained to mimic the user’s movements from its camera. Despite starting from random positions and seeing distractions like a shopping bag blocking the markers, it placed the objects into its target spots.To generate videos, they trained Diffusion Forcing on “Minecraft” game play and colorful digital environments created within Google’s <a href="https://github.com/google-deepmind/lab">DeepMind Lab Simulator</a>. When given a single frame of footage, the method produced more stable, higher-resolution videos than comparable baselines like a Sora-like full-sequence diffusion model and ChatGPT-like next-token models. These approaches created videos that appeared inconsistent, with the latter sometimes failing to generate working video past just 72 frames.Diffusion Forcing not only generates fancy videos, but can also serve as a motion planner that steers toward desired outcomes or rewards. Thanks to its flexibility, Diffusion Forcing can uniquely generate plans with varying horizon, perform tree search, and incorporate the intuition that the distant future is more uncertain than the near future. In the task of solving a 2D maze, Diffusion Forcing outperformed six baselines by generating faster plans leading to the goal location, indicating that it could be an effective planner for robots in the future.Across each demo, Diffusion Forcing acted as a full sequence model, a next-token prediction model, or both. According to Chen, this versatile approach could potentially serve as a powerful backbone for a “world model,” an AI system that can simulate the dynamics of the world by training on billions of internet videos. This would allow robots to perform novel tasks by imagining what they need to do based on their surroundings. For example, if you asked a robot to open a door without being trained on how to do it, the model could produce a video that’ll show the machine how to do it.The team is currently looking to scale up their method to larger datasets and the latest transformer models to improve performance. They intend to broaden their work to build a ChatGPT-like robot brain that helps robots perform tasks in new environments without human demonstration.“With Diffusion Forcing, we are taking a step to bringing video generation and robotics closer together,” says senior author Vincent Sitzmann, MIT assistant professor and member of CSAIL, where he leads the Scene Representation group. “In the end, we hope that we can use all the knowledge stored in videos on the internet to enable robots to help in everyday life. Many more exciting research challenges remain, like how robots can learn to imitate humans by watching them even when their own bodies are so different from our own!”<hr>Chen and Sitzmann wrote the paper alongside recent MIT visiting researcher Diego Martí Monsó, and CSAIL affiliates: Yilun Du, a EECS graduate student; Max Simchowitz, former postdoc and incoming Carnegie Mellon University assistant professor; and Russ Tedrake, the Toyota Professor of EECS, Aeronautics and Astronautics, and Mechanical Engineering at MIT, vice president of robotics research at the Toyota Research Institute, and CSAIL member. Their work was supported, in part, by the U.S. National Science Foundation, the Singapore Defence Science and Technology Agency, Intelligence Advanced Research Projects Activity via the U.S. Department of the Interior, and the Amazon Science Hub. They will present their research at NeurIPS in December.

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries.

Combining next-token prediction and video diffusion in computer vision and robotics

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/b921c890-089c-4f02-b05d-59a0c9850785?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">Episode Notes</h3>(0:50) - <a href="https://www.wevolver.com/article/ai-helps-detect-and-monitor-infrastructure-defects">Infrastructure Defect Detecting AI</a>Become a founding reader of our newsletter: <a href="http://read.thenextbyte.com/">http://read.thenextbyte.com/</a><hr><h2 data-attrid="title" data-local-attribute="d3bn" data-ved="2ahUKEwiBhLnmu4aAAxVmwjgGHcDWC38Q3B0oAXoECEgQEQ" id="isPasted">Transcript</h2>What's up, folks? As you know, cracks in buildings and roads and bridges can cause some serious damage if they're not found early enough. Today, we're talking about a team from EPFL that's gonna fix this problem using AI that can help detect cracks automatically and let us know when we need to do repairs. I think it's interesting, so let's jump into it.I'm Daniel, and I'm Farbod. And this is the NextByte Podcast. Every week, we explore interesting and impactful tech and engineering content from Wevolver.com and deliver it to you in bite sized episodes that are easy to understand, regardless of your background.&nbsp;Daniel:&nbsp;What's up friends? Like we said, today we're talking about the future of crack detection in infrastructure and why that's so important. We're not just talking about like cracks and buildings and roads and, sometimes it makes them look ugly or ivy can grow in them. But the problem is that they can cause serious damage if not found early enough. The traditional methods we have for inspecting and looking for these cracks are really slow, really costly, really hard to do in dangerous areas. And there's a lot of human error involved with it as well. So, we're talking about some awesome research here where they're using AI to try and help find and monitor cracks, but they're even taking kind of a unique AI approach as well that doesn't require detailed pixel by pixel labeling either. So, I'm pretty stoked about this one.Farbod:&nbsp;Dude, same. I feel like infrastructure improvement has been talked about for so long. I remember being in middle school and politicians were promising the next infrastructure build and how like, our infrastructure is aging and we need to replace it. Well, before this part, I was just kind of curious. I'm like, let's pull up some stats just about the US alone, right? I don't know if you knew this, but a freeway bridge inspection can take, can cost about $5,000 and take about eight hours to do. And that doesn't take into account the detour traffic causing people pains to get to work, et cetera, et cetera. And on top of that, the way we do grading of these cracks tends to be kind of subjective. So, the person that's doing it might grade it more, I don't know, liberally than the person that did last year. So that person might've thought things were going bad. This person says everything's good. So that kind of creates a weird situation where we can't track how bad it's getting super well.Daniel:&nbsp;Or can you trust the data that comes from these expectations? When you spend this much money and you spend this much time, you may not be even able to trust the data that comes out of it.Farbod: And to just make matters worse, there's something like one out of every three bridges in the US needs repair or replacement. So, one, you don't know if things are actually bad. Two, it seems like at least one third are kind of bad, maybe more. So not super comfortable about driving over a bridge at this point, but I saw something interesting about drones. Drones have had great improvement over the past couple of years. People were saying on average it's about 40% cheaper. So that makes it more affordable for local governments to take them on. But still, I'm guessing no one is super excited about shelling out over $3,000 to inspect every bridge they got. So yeah, sticky situation to be in.Daniel:&nbsp;Well, and just a quick throwback here. I would say like in the broader realm of infrastructure detection, both of our capstone projects, when we were graduating our undergrad program was related to using robotics and automation to try and detect issues with infrastructure before they progress to something that becomes a catastrophic failure. And that's not to say that we're experts in this topic at all. I think we were just at the tip of the iceberg here, but it is interesting to note that in separate graduating classes for about, and I both had this capstone project because it's such a big problem and we're still sitting here saying, maybe now we finally got some AI solution, which can help with this in a more robust manner, which is exciting for both of us, I think on a personal note, to see this field start to progress.Farbod: Dude, absolutely. Back when we were young men, which we're not anymore, at least my team, we were toying around with the idea of using AI, but even then, there wasn't a whole lot of excitement around it. Specifically, we were looking at corrosion, but now it looks like the game has completely shifted. We're going to EPFL for this episode, but if we can transition to the secret sauce, these folks are talking about how AI can actually completely remove the human element here and in the best way possible. We've been talking about subjectivity, how that can result in variance from person to person, and if a crack is getting better or worse. These folks are saying, hey, we can just train a very simple model that can tell us if there's a crack or if there's no crack, like binary. Once it learns how to detect the crack well, then it's just a matter of saying “cool, where are you seeing the crack and how is it changing over time?” Which in the world of like, you know, computer vision is a fairly simple and straightforward analysis. And the main application they're looking at for this is railroads, right? They're saying, railroads are used very often if there's cracks that grow that can cause catastrophic failures. So how do we maintain this and like monitor it on a regular basis? Well, they're like trains are passing through this bridges and stuff. What if they were the ones that were doing the analysis completely autonomously, no added costs, no added time, no requirement for shut down.Daniel: It's not sending people on harnesses to rappel from these. Honestly, you should just check on the link in the show notes, click for no other reason than to see the beautiful photo of this train on a railroad bridge going across a canyon in Switzerland where, and I wanted to confirm this, I wasn't 100% sure. The Matterhorn-Gothard bond network. I'm pretty sure Nellie and I took this train on our honeymoon, which is pretty cool.Farbod: Wow. That's special.Daniel:&nbsp;If not something very similar because all of Switzerland is full of beautiful trains and bridges and mountains, but I'm pretty sure we went on this exact railroad section where they're starting testing, but yeah, if for no other reason, check out this photo, but for me to get back to my main point, this is a bridge suspended several stories above the bottom of a canyon. It would cost a lot and you'd have to pay someone a lot to be able to visually inspect all of that manually, like as a person by hand. And what they're trying to do now is instead send a drone, but not in the form of the quadcopter drone that you think of probably when we say drone. The solution is, send a drone rail car, right? An instrumented rail car down this railroad and try and collect a bunch of data on, where are their cracks? Are they propagating? Are they staying the same? Which ones need to be addressed? Continuing to understand and map the ever evolving, let's say ecosystem around all these bridges and this infrastructure and understand when there's issues and indicate whether they need further inspection or whether they need immediate repair, et cetera.Farbod:&nbsp;Dude, totally agree. And the solution seems so simple when you say it out loud, but it really is, I don't know, I guess the best solutions are the simple ones. And what it made me think of is obviously this is great for the railroad system. But then we've been talking about like as a commuter that's going in a tunnel. That one tunnel in Boston I always hate it gives me the creeps when I'm going through it or when I'm going through bridges to Maryland and stuff like that. It would be great if there was something that could do the analysis there and in my mind, I'm like we're definitely a long way away from having autonomous vehicles doing these trips for us. But Uber and Lyft, they're drivers that are operating every single day that are doing these normal routes that people take to go to work and whatnot. I wonder if there was an opportunity there to say, hey, while you're doing this, mount this device on the back of your car so that whenever your GPS detects you're going from the beginning of the tunnel to the back of the tunnel, you do a scan and then that data uploads to a cloud somewhere and we can do analysis over time to see if we need to patch any cracks or what?Daniel: I'm like wondering if you read my mind here. I went to buses, because I'm like, the department of transportation in a city may also own the buses, right? So, they're in charge of the bridges. They're also in charge of the buses. Wonder if they can just put a bunch of instruments on top of a bus. In my mind, it's like the closest analogy to a rail car. Put a bunch of this infrastructure on top of a bus and this instrumentation, see if you can analyze the infrastructure around us using that same method as well. But I also want to jump a little bit into the specific AI methods they're using. And I'm not an expert here by any means, but I took some notes from the paper that I think are interesting. That there have previously tried to do AI classification of images, but it's too computationally intensive to justify the cost. So, if you were to use AI to try and evaluate millions of photos of an entire bridge. It may actually cost more than actually sending a human down there to do it. Because one of the things that I had to do is classify each pixel by pixel, trying to understand whether it was light or if it's dark and look at contrast to understand what is a crack, then it classifies a crack and that took a lot of computational energy. One of the things that they did is they, they train their AI ahead of time with images labeled as crack or no crack. And then it uses explainable AI is what they say is their unique method here to try and look at a new photo and circle, basically highlight a cracked area and create a map showing where cracks are and where they aren't. And essentially what it's doing is it's just taking photo after photo after photo and trying to trace the area around where it thinks the crack is. And over a period of time, it's trying to look at the same photo from the same location the last time this train came through and look at whether this crack got bigger or if it stayed the same. And if cracks are serially getting bigger and bigger and bigger from these images, it warrants sending someone down there, an expert down there to take a look at it. And one of the things that they mentioned is this cut the processing and labeling effort, as opposed to previous AI methods by about 90%, which makes it cheaper and faster and worthwhile to implement in an autonomous manner. It's not quite as accurate as these full AI models, but they said it can still help directionally track crack growth. And they think that they can hone it essentially and get rid of more of the area that they're experiencing right now. But essentially the game plan I guess is to go test this out in the field, get more learnings, understand more but in practice even if it has the same performance that it has today you can use this as kind of like a non-invasive constant monitoring where it looks at the rails, it looks at the ties, it looks at the retaining walls and tries to understand if there's anything wrong with the railroad. And you could have this on every train or one of every 10 trains passing through this railroad. Even though I think right now it's got 35% error, which again, it sounds high compared to most AI models, but if you're telling me, you can tell me 65% of the time, whether a crack's growing or not. If I've got two or three trains that pass through over a series of a week and show that this crack indicates that this crack may be growing, the playbook here is not to say, all right, let's shut down the bridge and tear it down. The playbook here could be, let's go send one of those expert inspectors that we used to have doing this job full time and had to shut down the railroad to do now that we know that there's something that's worth further evaluation, let's send one of those crack experts down there to understand what's going on and see whether it needs repair.Farbod: And this is a great example, at least in my opinion of complimentary automation, right? Like here's a task that was bad for people to do manually. And some places are like in America, it's just straight up not done at times. Now you have automation that's taking care of the tedious part. And then you have the human that's taking on the challenging part, which is, hey, we're like 90% sure there is a crack, please go out there and now characterize it and give us feedback on if it's catastrophic or if, you know, it's a simple crack that needs to be fixed, etc, etc. So, I feel like this is the best use of every technology or every resource, think of the human expert as a resource. And this is kind of going against the AI is going to get rid of all jobs mentality.Daniel: Well, and I think what we, the intended end here is to be able to do continuous monitoring, collect data all the time to understand which regions are susceptible to cracks, and then use that as a heat map essentially to show someone who's again, got expertise in this field to go, this is the area that you need to inspect. This is the area that you need to monitor. This is the area that may need potential repair as opposed to the other portions of track that maybe didn't need as much attention and they were getting it before because you had to go check every single mile of this thing manually. And maybe you used to only check one, every single mile of this thing manually, one out of every five years. And now we could have a train passing on it five times a day that tells us how the railroad's doing. So, I appreciate here, like you said, it's a complimentary solution between automation and humans. But it also gets us an immediate benefit, which is a lot more data, a lot more consistently on how our infrastructure is doing and when it needs to be repaired.Farbod: Totally agree, man. You want to do us a favor and wrap up the episode?Daniel:&nbsp;I will. What if we could stop big cracks in buildings and roads before they cause big problems, I'm thinking of bridges collapsing or bridges needing to be shut down because they need repair or rip, cars getting derailed. These are big problems and we're talking today about a new AI method that helps find and track these cracks automatically without needing a ton of expensive detailed data. Essentially, they're getting the job done with 90% less effort. That's what their new tech promises. And they use this system called explainable AI that spots cracks with simple image labels, saving time and money, keeping an eye on crack growth to let experts know where they need to address cracks and where we might need repairs in the future. I think it's a lot like, I don't know, Google Maps navigation or some sort of traffic heat map, but it tells us where most areas are prone to cracks and where we need repairs so that folks know where they need to focus their efforts.Farbod:&nbsp;Money.Daniel:&nbsp;Thanks, my dude.Farbod:&nbsp;You killed it, dude. All right, folks, thank you so much for listening. And as always, we'll catch you in the next one.Daniel:&nbsp;Peace.<hr>As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.To learn more about this show, please visit our<a href="https://www.wevolver.com/profile/the.next.byte">&nbsp;shows page</a>. By following the page, you will get automatic updates by email when a new show is published. Be sure to give us a follow and review on<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" target="_blank" rel="nofollow">&nbsp;Apple</a> podcasts,<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" target="_blank" rel="nofollow">&nbsp;Spotify</a>, and most of your favorite podcast platforms!--The Next Byte: We're two engineers on a mission to simplify complex science &amp; technology, making it easy to understand. In each episode of our show, we dive into world-changing tech (such as AI, robotics, 3D printing, IoT, &amp; much more), all while keeping it entertaining &amp; engaging along the way.

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems

Podcast: Saving America's Collapsing Infrastructure

Recently we launched new Edge Impulse functionality that shows just one of the powerful ways that LLMs can be utilized on the edge: <a href="https://www.edgeimpulse.com/blog/llm-knowledge-distillation-gpt-4o/" rel="noreferrer">using GPT-4o to train a 2,000,000x smaller model</a>, that runs directly on device.Our initial demo showed how this tool could be used to label images for binary classification (presence/absence) but there are more complex labeling tasks possible.In this post and video we explore how visual data from a satellite could be labeled both for cloud type and for coverage, generating two useful edge models from the same dataset very quickly:<iframe width="640" height="360" src="https://www.youtube.com/embed/iXdyrDrY9Zs?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;</iframe><h3 id="isPasted">The challenge of AI for remote applications</h3>Satellites are resource constrained — power, compute and storage must all be managed carefully as expansion is rarely possible. Another issue is data transfer to Earth, which can be costly and has bandwidth restrictions. Current LLMs can only run in large data centers due to their unwieldy size (billions of parameters) so to utilize their power, raw data must be sent over the air to be computed.Edge AI is perfectly positioned to address these issues. By developing a model that can run on the device where data collection occurs, you can reduce data transmission to just the desired insight. For example, if you’re only interested in seeing clear images but a particular satellite image is obscured by clouds, it doesn't need to be sent down to earth for inspection. These methods can also be useful for improving existing edge models — sending raw data for classifications on-device that return an “unsure” or “anomaly” result to an LLM for labeling can be a good way to improve model performance over time.<a href="https://form.jotform.com/242533284499365"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1728891710283-5+Rising+Trends+for+AI+Adoption+in+Manufacturing+%283%29.png" style="width: 100%px;" class="fr-fic fr-dib"></a><h3>Cloud type and coverage labeling examples</h3>To show the power of LLM labeling tools we have created two models from one raw dataset by designing two different LLM prompts for labeling. In the above video, I accessed an existing <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/BSETKZ" rel="noreferrer">Harvard cloud dataset</a>. Using ChatGPT, I can ask questions about the images in the dataset, such as what type of cloud is shown or how much of the frame is covered by clouds. The latest LLMs excel at this type of image analysis, but running this procedure has high latency, and incurs significant cloud costs, especially when in a production environment with many source cameras and a high frame-rate.Here's how we can create a smaller model to perform this same tasks, on-device:<h3>Project walkthrough</h3><ol><li>Data collection: I downloaded the open <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/BSETKZ" rel="noreferrer">Harvard cloud satellite-image dataset</a> (originally curated for cloud removal with spatiotemporal networks, a more complex task). Taking only the RGB images, I then uploaded them without labels into an Edge Impulse project. This unlabeled dataset forms the bases for both the cloud type classification and cloud coverage models we want to create.</li><li>Designing a prompt: I first experimented with labeling a few images in ChatGPT itself, noting the setting specific output options that worked best for each tasks. • Cloud Classification: My final prompt was “What cloud type is visible in this satellite picture? Respond only with “cirrus”, “cirrocumulus”, “cirrostratus”, “altocumulus”, “altostratus”, “nimbostratus”, “stratocumulus”, “stratus”, “cumulus”, “cumulonimbus”. If there is no cloud respond with "no cloud" or "unsure" if you're not sure. <img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1728891780952-Untitled--4--2.png" style="width: 100%px;" class="fr-fic fr-dib">•&nbsp;Cloud Coverage: My final prompt was “Look at this image from a satellite and respond with a percentage of how much of the image is obscured by clouds. Respond only with a percentage not including "%" or "unsure" if the image is corrupted, or it is not an image from a satellite.”</li></ol><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1728891803331-Untitled--5-.png" style="width: 100%px;" class="fr-fic fr-dib">3. Labeling with GPT-4o: Using the new transformation block "Label image data using GPT-4o" inside each Edge Impulse project, I asked GPT-4o to label the images with my prompt. The block discards any blurry or uncertain images, providing a clean dataset. It also provides natural language reasoning for every labeling decision — a powerful tool for debugging model performance.4. Model Training: For each model type, I designed an impulse with the desired output format. I used a Transfer Learning Image classifier block for the cloud classification task and a Regression block for the cloud coverage task: •&nbsp;Cloud Classification: I trained a small MobileNetV1&nbsp;classification model with these images. The model was only 1.6MB in size, significantly smaller than the original LLM. •&nbsp;Cloud Coverage: I trained a small convolutional neural network regression model with these images. The model was only 49.1K in size, significantly smaller than the original LLM.5. Deployment: This model could be deployed on MCU hardware from Cortex M4 upwards, for example an Arduino Nicla Vision. The model accurately detects cloud types on-device, in real-time, without requiring cloud services.<table><tbody><tr><td><h3>New whitepaper: Discover the 5 AI Trends Transforming Manufacturing</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1728891888961-1728891888961.png" class="fr-fic fr-dii">The whitepaper, "5 Rising Trends for AI Adoption in Manufacturing," offers key insights into how artificial intelligence is transforming manufacturing processes. It explores survey data from over 150 industry leaders, revealing that optimizing production and improving product quality are top goals driving AI adoption. The report also discusses the rise of AI solutions for worker safety, real-time insights to boost profitability, and the growing importance of internal expertise in implementing AI. Download the full whitepaper to gain a deeper understanding of these trends and how AI/ML is shaping the future of manufacturing.<a href="https://form.jotform.com/242533284499365" target="_blank" rel="nofollow">DOWNLOAD YOUR FREE COPY</a></td></tr></tbody></table><h3>Results</h3>These two models were created in an incredibly short time. Data labeling is often the most time-consuming and error-prone part of training a model for the edge. By utilizing LLMs you can create your first-pass model much more quickly, and the power of having natural language reasons for each label makes debugging model datasets even easier.LLMs are still flawed and don’t perform perfectly for every use-case, but they are a great tool to assist subject matter experts in developing models faster. As generative AI continues to accelerate in popularity and complexity these techniques will only get better.<h3>References</h3><ul><li>Jan Jongboom <a href="https://www.edgeimpulse.com/blog/llm-knowledge-distillation-gpt-4o/" rel="noreferrer">Bringing Large Language Models to the Edge with GPT-4o and NVIDIA TAO</a></li><li>Met Office <a href="https://www.metoffice.gov.uk/weather/learn-about/weather/types-of-weather/clouds/cloud-names-classifications" rel="noreferrer">Cloud names and classifications</a></li><li>Edge Impulse <a href="https://studio.edgeimpulse.com/public/430153/latest" rel="noreferrer">GPT-4o Cloud Type Experiment</a></li><li>Edge Impulse <a href="https://studio.edgeimpulse.com/public/417680/latest" rel="noreferrer">GPT-4o Cloud Coverage Experiment</a></li><li>Harvard Dataverse <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/BSETKZ" rel="noreferrer">Dataset From: Cloud Removal from Satellite Images using Spatiotemporal Generator Networks</a></li></ul><hr><h3 id="isPasted">Download the whitepaper: 5 AI Trends Transforming Manufacturing</h3><div data-message-author-role="assistant" data-message-id="a9ce89dd-4eb9-44c7-9d28-e3c7d10f7094" dir="auto" data-message-model-slug="gpt-4o"><div data-message-author-role="assistant" data-message-id="1c2c7603-add5-4aa4-84c9-81aa2c1371a2" dir="auto" data-message-model-slug="gpt-4o">Stay ahead of the curve on how AI is transforming businesses with the whitepaper "5 Rising Trends for AI Adoption in Manufacturing." The guide offers valuable insights to help leaders implement AI strategies that optimize production, improve product quality, and drive innovation. Download your free copy now:<iframe id="JotFormIFrame-242533284499365" title="Edge Impulse Form" allowtransparency="true" src="https://form.jotform.com/242533284499365" frameborder="0" style="min-width:100%;max-width:100%;height:900px;border:none;" scrolling="no" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp; </iframe></div></div>

Recently we launched new Edge Impulse functionality that shows just one of the powerful ways that LLMs can be utilized on the edge: using GPT-4o to train a 2,000,000x smaller model, that runs directly on device.

Using LLMs to Analyze and Label Satellite Imagery in Edge Impulse

More than 40,000 people have died on U.S. roads in each of the last three years. In an effort to eliminate these preventable accidents, The Department of Transportation has awarded The City of Pittsburgh, in partnership with Ding Zhao, $1.3 million to analyze Pittsburgh’s current transportation infrastructure and inform safety-first upgrades by the end of 2026.As part of the <a href="https://www.transportation.gov/grants/SS4A" rel="noopener" target="_blank">Biden-Harris Administration’s Bipartisan Infrastructure Law for the Safe Streets and Roads for All</a> (SS4A) program, the project will support a road safety audit, the development of a <a href="https://engage.pittsburghpa.gov/vision-zero" rel="noopener" target="_blank">Vision Zero</a>-focused streets design manual along with quick build projects and a Vision Zero ambassador program to eliminate traffic fatalities and increase equitable mobility.<a href="https://engineering.cmu.edu/directory/bios/zhao-ding.html">Ding Zhao</a>, director of the Safe AI Lab and an associate professor of <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank">mechanical engineering</a> at Carnegie Mellon University, will use large language models (LLMs) to analyze Pittsburgh’s streets, identify which intersections are the highest-risk for drivers and pedestrians, and work with city officials to test new street design improvements.“Large language models offer a very scalable, very powerful way to collect data and run simulations,” explained Zhao. “By collecting more data, more efficiently, we can be inclusive of rare events and vulnerable users.”<iframe width="640" height="360" src="https://www.youtube.com/embed/SDAoLGEE2CU?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-24="true" id="200260888" title="Simulation of Pittsburgh's city streets">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</iframe>Simulations of Pittsburgh’s streets will be used by Ding Zhao to identify intersections that pose the highest-risk for drivers and pedestrians. The team will work with city officials to test new street design improvements as part of the Safe Streets and Roads for All Program.The team will also use LLMs to analyze the results of their simulations, an approach Zhao says is novel to transportation and infrastructure.“We have a clear goal for this project and by working with the City of Pittsburgh we are guaranteeing real world impact,” said Zhao. “I’m excited to see how my research will positively affect our neighborhoods. If we can save even one person, I’ll be very proud.”<blockquote>I’m excited to see how my research will positively affect our neighborhoods. If we can save even one person, I’ll be very proud.<cite>Ding Zhao, Associate Professor, Mechanical Engineering</cite></blockquote>“We’re excited to partner with the research teams at Carnegie Mellon University as we work toward our goal of zero traffic fatalities,” said Mayor Ed Gainey. “To make our streets safer, we need to break down silos and build strong partnerships between the City of Pittsburgh and our private institutions. Together, we can drive the kind of innovation that makes a real difference in roadway safety.”Zhao, who has been working to advance AI in self-driving cars for nearly a decade, is also eager to explore how enhanced infrastructure can make the roads safer for semi and fully autonomous vehicles.The team will leverage on its in-house tool <a href="https://safebench.github.io/">SafeBench</a>, an open-sourced platform designed to assess and improve the safety of AI models based on a high-fidelity autonomous driving simulator. SafeBench provides a suite of standardized benchmarks and testing environments for evaluating how AI models perform in dynamic, safety-critical situations, such as autonomous driving or infrastructure planning. This tool can be integrated into projects like Pittsburgh’s transportation audit by offering a rigorous framework for analyzing AI-driven systems’ resilience under various conditions, including multimodal perturbations and rare events. By ensuring the safety and robustness of AI systems, SafeBench will help cities like Pittsburgh deploy cutting-edge AI technologies with confidence, minimizing risks and enhancing the overall safety of public infrastructure.<iframe width="640" height="360" src="https://www.youtube.com/embed/TnJxpTJ1OKQ?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-24="true" id="330679141" title="Scenarios for Perception">&nbsp;&nbsp;&nbsp;</iframe>“SafeBench allows us to identify potential safety issues in the AI-empowered systems early on, which helps improve their reliability as they are integrated into real-world applications,” said Haohong Lin, a Ph.D. student at Carnegie Mellon University working with Zhao.“I have been working on self-driving cars for 17 years,” Zhao said. “This project opens a new door for us to explore how streets can be modified to provide a safer environment for us all both today and in the future when humans will share the road with AVs.”

Ding Zhao is teaming up with the City of Pittsburgh to improve road safety and eliminate traffic fatalities using AI.

Safe AI for safer city streets

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/b921c890-089c-4f02-b05d-59a0c9850785?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Thanks to recent advances in artificial intelligence (AI), civil engineers can inspect large-scale infrastructure more efficiently and cost-effectively, while also monitoring the progression of damage severity over time. A team of EPFL researchers has demonstrated the feasibility of an AI-driven method for crack detection, growth and monitoring, and will soon test it on the railway section between Zermatt and Brig in Valais Canton.AI can help improve railway safety by enabling automated inspections of tracks, crossties, ballasts and retaining walls. Researchers at EPFL’s Intelligent Maintenance and Operations Systems (IMOS) Laboratory have developed an AI-driven method that improves the efficiency of crack detection in concrete structures. Their research, recently published in Automation in Construction, introduces a novel method that employs explainable artificial intelligence, or a form of AI which allows users to understand the basis of AI decisions.“We trained an algorithm to differentiate between images with and without cracks in concrete walls [a binary classification task] by feeding it hundreds of image samples from both categories. Then we asked the algorithm to highlight which pixels it used to make its decision,” says Florent Forest, a scientist at the IMOS lab and the study’s lead author. The algorithm successfully identified the pixels corresponding to cracks. “With our approach, users can feed the algorithm images taken over several years of a section of railway – or of any other kind of infrastructure that’s inspected regularly – and ask it to quantify the severity of cracks in walls and crossties over time. This helps infrastructure operators plan their maintenance more effectively,” he says.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI3NzY1MzQwMzA2LTRiZTNmODk0LmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Concrete crossties must be maintained by rail network operators. © iStock PhotosEnhanced inspectionsCurrently, railway operators regularly inspect the condition of infrastructure such as retaining walls by using predefined criteria, where grades are assigned by experienced inspectors. However, this process is often prone to subjective evaluations and makes it difficult to track changes over time, especially when different inspectors assess the same section of infrastructure at different points in time.Thanks to advancements in digitalization, railway operators can monitor track conditions using a specialized monitoring coach equipped with various measuring devices and side and floor cameras for the visual inspection of rails, concrete crossties and retaining walls. By using these AI-driven systems for damage severity quantification, the inspection process can be automated, making it more objective, accurate and easier to compare over time.The EPFL research team will test its method on sections of railway between Zermatt and Brig and between Brig and Disentis. These sections include a number of retaining walls of different shapes and materials, making the task significantly challenging for the algorithm. The team has already collected drone images, along with those from the monitoring coach, and will use its AI algorithm to assist the railway operator in monitoring its infrastructure more frequently and systematically.<hr>References<a href="https://doi.org/10.1016/j.autcon.2024.105497" target="_blank" rel="noopener">Florent Forest, Hugo Porta, Devis Tuia and Olga Fink, “From classification to segmentation with explainable AI: A study on crack detection and growth monitoring,” Automation in Construction, September 2024.</a>

The Matterhorn-Gotthard Bahn network links the plain to Zermatt. © iStock Photos

A team of EPFL researchers has demonstrated the feasibility of an AI-driven method for crack detection, growth and monitoring, and will soon test it on the railway section between Zermatt and Brig in Valais Canton.

AI helps detect and monitor infrastructure defects

Imagine having to straighten up a messy kitchen, starting with a counter littered with sauce packets. If your goal is to wipe the counter clean, you might sweep up the packets as a group. If, however, you wanted to first pick out the mustard packets before throwing the rest away, you would sort more discriminately, by sauce type. And if, among the mustards, you had a hankering for Grey Poupon, finding this specific brand would entail a more careful search.MIT engineers have developed a method that enables robots to make similarly intuitive, task-relevant decisions.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727764894016-MIT-Scene-Tasks-01-press_0.jpg" style="width: 100%px;" class="fr-fic fr-dib">From left to right: team members Lukas Schmid, Nathan Hughes, Dominic Maggio, Yun Chang, and Luca Carlone. Credit: Andy RyanThe team’s new approach, named Clio, enables a robot to identify the parts of a scene that matter, given the tasks at hand. With Clio, a robot takes in a list of tasks described in natural language and, based on those tasks, it then determines the level of granularity required to interpret its surroundings and “remember” only the parts of a scene that are relevant.In real experiments ranging from a cluttered cubicle to a five-story building on MIT’s campus, the team used Clio to automatically segment a scene at different levels of granularity, based on a set of tasks specified in natural-language prompts such as “move rack of magazines” and “get first aid kit.”The team also ran Clio in real-time on a quadruped robot. As the robot explored an office building, Clio identified and mapped only those parts of the scene that related to the robot’s tasks (such as retrieving a dog toy while ignoring piles of office supplies), allowing the robot to grasp the objects of interest.Clio is named after the Greek muse of history, for its ability to identify and remember only the elements that matter for a given task. The researchers envision that Clio would be useful in many situations and environments in which a robot would have to quickly survey and make sense of its surroundings in the context of its given task.“Search and rescue is the motivating application for this work, but Clio can also power domestic robots and robots working on a factory floor alongside humans,” says Luca Carlone, associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), principal investigator in the Laboratory for Information and Decision Systems (LIDS), and director of the MIT SPARK Laboratory. “It’s really about helping the robot understand the environment and what it has to remember in order to carry out its mission.”The team details their results in a <a href="https://dspace.mit.edu/handle/1721.1/157072" target="_blank">study appearing today</a> in the journal Robotics and Automation Letters. Carlone’s co-authors include members of the SPARK Lab: Dominic Maggio, Yun Chang, Nathan Hughes, and Lukas Schmid; and members of MIT Lincoln Laboratory: Matthew Trang, Dan Griffith, Carlyn Dougherty, and Eric Cristofalo.<h2>Open fields</h2>Huge advances in the fields of computer vision and natural language processing have enabled robots to identify objects in their surroundings. But until recently, robots were only able to do so in “closed-set” scenarios, where they are programmed to work in a carefully curated and controlled environment, with a finite number of objects that the robot has been pretrained to recognize.In recent years, researchers have taken a more “open” approach to enable robots to recognize objects in more realistic settings. In the field of open-set recognition, researchers have leveraged deep-learning tools to build neural networks that can process billions of images from the internet, along with each image’s associated text (such as a friend’s Facebook picture of a dog, captioned “Meet my new puppy!”).From millions of image-text pairs, a neural network learns from, then identifies, those segments in a scene that are characteristic of certain terms, such as a dog. A robot can then apply that neural network to spot a dog in a totally new scene.But a challenge still remains as to how to parse a scene in a useful way that is relevant for a particular task.“Typical methods will pick some arbitrary, fixed level of granularity for determining how to fuse segments of a scene into what you can consider as one ‘object,’” Maggio says. “However, the granularity of what you call an ‘object’ is actually related to what the robot has to do. If that granularity is fixed without considering the tasks, then the robot may end up with a map that isn’t useful for its tasks.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727764931766-MIT-Scene-Tasks-03-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">Clio quickly segments a scene such as a cluttered cubicle (pictured), and feeds the segments through an “information bottleneck” algorithm that focuses on the segments that only represent target objects relevant to the task at hand. Colored cubes represent segment regions that Clio has picked identified as task-relevant objects that a robot can focus on while ignoring the rest of the scene. Credit: Courtesy of the researchers<h2>Information bottleneck</h2>With Clio, the MIT team aimed to enable robots to interpret their surroundings with a level of granularity that can be automatically tuned to the tasks at hand.For instance, given a task of moving a stack of books to a shelf, the robot should be able to &nbsp;determine that the entire stack of books is the task-relevant object. Likewise, if the task were to move only the green book from the rest of the stack, the robot should distinguish the green book as a single target object and disregard the rest of the scene — including the other books in the stack.The team’s approach combines state-of-the-art computer vision and large language models comprising neural networks that make connections among millions of open-source images and semantic text. They also incorporate mapping tools that automatically split an image into many small segments, which can be fed into the neural network to determine if certain segments are semantically similar. The researchers then leverage an idea from classic information theory called the “information bottleneck,” which they use to compress a number of image segments in a way that picks out and stores segments that are semantically most relevant to a given task.“For example, say there is a pile of books in the scene and my task is just to get the green book. In that case we push all this information about the scene through this bottleneck and end up with a cluster of segments that represent the green book,” Maggio explains. “All the other segments that are not relevant just get grouped in a cluster which we can simply remove. And we’re left with an object at the right granularity that is needed to support my task.”The researchers demonstrated Clio in different real-world environments.“What we thought would be a really no-nonsense experiment would be to run Clio in my apartment, where I didn’t do any cleaning beforehand,” Maggio says.The team drew up a list of natural-language tasks, such as “move pile of clothes” and then applied Clio to images of Maggio’s cluttered apartment. In these cases, Clio was able to quickly segment scenes of the apartment and feed the segments through the Information Bottleneck algorithm to identify those segments that made up the pile of clothes.They also ran Clio on Boston Dynamic’s quadruped robot, Spot. They gave the robot a list of tasks to complete, and as the robot explored and mapped the inside of an office building, Clio ran in real-time on an on-board computer mounted to Spot, to pick out segments in the mapped scenes that visually relate to the given task. The method generated an overlaying map showing just the target objects, which the robot then used to approach the identified objects and physically complete the task.“Running Clio in real-time was a big accomplishment for the team,” Maggio says. “A lot of prior work can take several hours to run.”Going forward, the team plans to adapt Clio to be able to handle higher-level tasks and build upon recent advances in photorealistic visual scene representations.“We’re still giving Clio tasks that are somewhat specific, like ‘find deck of cards,’” Maggio says. “For search and rescue, you need to give it more high-level tasks, like ‘find survivors,’ or ‘get power back on.’ So, we want to get to a more human-level understanding of how to accomplish more complex tasks.”This research was supported, in part, by the U.S. National Science Foundation, the Swiss National Science Foundation, MIT Lincoln Laboratory, the U.S. Office of Naval Research, and the U.S. Army Research Lab Distributed and Collaborative Intelligent Systems and Technology Collaborative Research Alliance.

MIT’s Clio runs in real-time to map task-relevant objects in a robot’s surroundings, allowing the bot (Boston Dynamic’s quadruped robot Spot, pictured) carry out a natural language task (“pick up orange backpack”). Credit: Courtesy of the researchers

A new method called Clio enables robots to quickly map a scene and identify the items they need to complete a given set of tasks.

Helping robots zero in on the objects that matter

<h3 id="isPasted">4 Reasons to Create a Smart Tank Monitoring and Pump Control System</h3>#1. A smart solution allows operators to remotely manage tank levels and pump operations from anywhere, preventing common issues such as tank overflows, pump dry-running, and treatment process disruptions. The ability to monitor systems in real-time reduces the risk of costly emergencies by enabling early detection of potential problems, providing timely data for decision-making.#2.&nbsp;In terms of maintenance and usage, access to real-time data allows operators to optimize system performance. Data on pump operations and operating hours can be leveraged to support predictive maintenance, reducing downtime and extending the lifespan of equipment. This also ensures consistent water quality by making informed decisions based on continuous monitoring.#3.&nbsp;When based on versatile products like Arduino’s, inspired by open-source principles, a smart tank monitoring system is highly adaptable and scalable, allowing it to fit various industrial needs. It can be integrated with existing equipment and third-party systems, ensuring seamless incorporation into a facility’s operations. This flexibility makes it ideal for a range of industrial applications, from small-scale plants to complex systems.#4.&nbsp;By automating the monitoring process, the solution minimizes the need for frequent manual checks, reducing human error and cutting operational costs. Automated alerts and precise monitoring help maintain optimal chemical dosing and flow rates, while reducing any costs associated with equipment damage, process disruptions, and regulatory non-compliance.<h3>Technical Overview of the Solution</h3>A flexible and cost-effective solution for tank monitoring and pump control can be based on an easy-to-use micro PLC such as the <a href="https://www.arduino.cc/pro/hardware-arduino-opta/" target="_blank">Arduino Opta</a>. The <a href="https://docs.arduino.cc/tutorials/opta/tank-level-app-note/" target="_blank">complete tutorial is available on Arduino Docs</a>.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI2ODQ0ODQzNTk5LWVsZWN0cmljYWxfY29ubmVjdGlvbnMucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">In this case, the system employs a variety of sensors – such as float switches, ultrasonic transducers, and pressure transducers – to continuously track liquid levels. Data from flow meters is used to measure pump discharge rates and overall system flow. This information is then transmitted to a cloud-based SCADA&nbsp;(Supervisory Control and Data Acquisition) system for real-time access, allowing operators to visualize and manage tank levels, pump status, and system flow rates through an intuitive dashboard.Additionally, the system includes configurable alarms that send alerts for critical scenarios such as abnormal flow rates or high/low tank levels. These alarms can be customized and sent via multiple channels, including email and SMS, preventing unnecessary notifications and ensuring only critical alerts are triggered.The solution is compatible with a variety of sensors and supports industry-standard protocols, such as Modbus RTU over RS485 and Modbus TCP/IP.&nbsp;<h3>Adding Vision-Based Liquid Monitoring</h3>In situations where conventional sensors are not practical, a vision-based monitoring system can be deployed. A module such as the Arduino <a href="https://www.arduino.cc/pro/hardware-product-nicla-vision/" target="_blank">Nicla Vision</a>, which is equipped with a 2 MP color camera, has the intelligence to&nbsp;visually assess the current liquid level in a tank: applying so-called “blob detection”, the liquid’s surface can be identified and the tank’s fill level calculated. This data can be relayed to the control system to trigger appropriate actions, such as opening or closing valves.This vision-based approach offers a non-invasive method to monitor tank levels, making it suitable for systems where physical sensor installation is difficult or impossible. It can also be retrofitted to almost any existing system without significant modifications.<h3>Unlocking Smart Industrial Solutions with the Arduino Ecosystem</h3>The flexibility and adaptability of the Arduino platform make it an excellent choice for creating smart industrial systems, such as tank level monitoring and pump control solutions. Whether you're integrating traditional sensors or implementing innovative vision-based methods, Arduino’s open-source tools and broad compatibility with industry standards offer a versatile foundation for any project. Explore the Arduino ecosystem to find the right tools for your next industrial challenge, and see how it can accelerate your path to smarter, more efficient operations.

The need for advanced tank monitoring and pump control solutions is growing across industries – from water treatment to aquaculture to industrial utilities. Meanwhile, smart technology is transforming tank management for a variety of liquids, ensuring a streamlined process and minimizing risks.

Build Your Own Smart Industrial Tank Level Monitoring and Pump Control Solution

By achieving a means of driving machinery automatically, the Industrial Revolution changed the whole nature of production.It occurred in Great Britain between 1760 and 1830, then spread through Europe and the United States. This is now the First Industrial Revolution, or Industry 1.0. As far as the actual words go, the term "industrial revolution" was first used in 1799 and was taken up in the early 1800s.The second significant advance Industry 2 started around 1850, with electricity and large-scale industrial processing. It lasted until the second half of the 20th century.&nbsp;<h2 dir="ltr">Computer Age</h2>The third industrial revolution transformed our lives once we got into computers and electronic communications. We began using computers for everything, including the automation of production and assembly lines, although human involvement was still generally required for control. Information Technology (IT) became the next big thing. This step was retroactively called Industry 3.0. It saw a massive upswing in the industrial use of computers as human tasks were phased into automated ones. Programmable Logic Controllers (PLCs), IT and human-machine interfaces (HMIs) replaced humans in governing computerised processes.&nbsp;Since 2011, Industry 4.0 has encompassed many of the advanced systems we're familiar with. In Industry 4.0, most automated systems follow pre-set programming parameters. They can also streamline production processes by analysing and acting on collected data.The&nbsp;<a title="What Is IIoT?" href="https://www.rowse.co.uk/blog/post/what-is-iiot" target="_blank">industrial Internet of Things</a> (IoT), cloud computing, and cyber-physical systems (CPS) impact today's manufacturing processes. These technologies have hugely enhanced automation and global connectivity. The sequence of increased automation has also led us to AI and robots.<h2 dir="ltr">Industry 3.0 -&gt; 4.0</h2>The developments in industrial technology offer&nbsp;<a title="What Are The Benefits Of Industry 4.0?" href="https://www.rowse.co.uk/blog/post/benefits-of-industry-4" target="_blank">great benefits</a>, such as:<ul><li dir="ltr">Decentralised information</li><li dir="ltr">Real-time data collection</li><li dir="ltr">Big data analytics</li><li dir="ltr">AI and ML (machine learning)</li><li dir="ltr">Interoperability</li><li dir="ltr">Heightened flexibility</li></ul><h3 dir="ltr">Decentralised Information</h3>Many elements of the production process now contain sensors that collect a continuous stream of information. In Industry 3.0, each company's local servers usually store and manage all the data from an individual system. Only limited data sharing and integration took place across sectors. Industry 4.0, however, is just the opposite. Its interconnected platforms and systems allow data to be accessed in real time by multiple stakeholders.<h3 dir="ltr">Real-Time Data Collection</h3>Vast quantities of data are being gathered, processed and acted on in real-time. The availability of decentralised information and its immediate analysis has several advantages. It helps to optimise production and management processes, allows different entities to collaborate more effectively, and thus enables better decision-making. Strategies like predictive maintenance considerably reduce downtime.<h3 dir="ltr">Big Data Analytics</h3>Big data has had a significant impact on operational efficiency. We can now analyse and assess many aspects of production and business factors, such as customer feedback and returns. These and many other contributing factors can help to improve decision-making, anticipate future demands and reduce outages. Big data enables businesses the flexibility to adapt to current market demand.<h3 dir="ltr">AI And ML</h3>The sheer volume of intelligent data collected in today's processes is beyond the capacity of humans. The data analysts of Industry 3.0 have been replaced with Industry 4.0's AI and ML algorithms. Not only can these systems manage big data, but they can learn from its analysis how to improve all aspects of automation. This leads to greater productivity, better quality and overall efficiency. Industry 4.0 integrates automation with intelligence, allowing machines and systems to become more intelligent, autonomous, and adaptive.&nbsp;<h3 dir="ltr">Interoperability</h3>Industry 3.0 made significant advances in manufacturing technology, but its main focus was automating existing processes. It was also confined to individual businesses, each with its systems. Industry 4.0 is about collaboration and connectivity between stakeholders in a business ecosystem. These may include manufacturers, their customers, suppliers, and even the machines themselves. The intention is to promote a "collective intelligence" amongst entities who share their knowledge and insights to drive improvement and innovation. Thus, different value chain components collaborate in an ecosystem with complete transparency, seamless communication and interoperability.<h3 dir="ltr">Heightened Flexibility</h3>Industry 3.0 adopted the concept of lean production, saving money by automating and streamlining processes. Industry 4.0 is geared instead towards brilliant production, where decisions are made based on data rather than experience. Automated systems learn from the amassed data, optimising operations and making predictions without human intervention. New revenue streams are being created differently. This links systems to the product instead of the process. They are making production processes more flexible and responsive. This allows them to adapt better and faster to market conditions and changing demands.<h2 dir="ltr">So, How Does Industry 4.0 Differ From Industry 3.0?</h2>Industry 3.0 achieved limited technological advances in manufacturing, with computers, automation and PLCs. Industry 4.0 leverages far more advanced technologies through the IIoT, cloud computing,&nbsp;<a title="The Future Of Manufacturing" href="https://www.rowse.co.uk/blog/post/the-future-of-manufacturing" target="_blank">augmented reality</a>, and robotics. Add to these AI, ML and big data analytics, and you have real-time connectivity, autonomous systems, predictive maintenance and data-driven decision-making. In addition, the physical and virtual worlds can be merged to enhance many design and manufacturing processes.Industry 3.0 was all about automation in individual factories. It aimed to save money by improving efficiency and productivity. It optimised production lines by substituting machines and computers for human labour. Industry 4.0 has a much broader scope, integrating digital technologies into the value chain. Interconnected CPS can even manage distribution and customer interaction. This makes the digital transformation of the industry far more comprehensive. We're in a major shake-up of how things work not just processes but business models and value creation. As Industry 4.0 gathers momentum, we need new skills for the latest technologies. At this stage, we need digital capabilities to be integrated into every organisational aspect of a business.

To understand Industry 4.0 and Industry 3.0, we must go back to Industry 1.0, although it wasn't called that then. We learned about the Industrial Revolution in school and there was just one. 

Whats The Difference Between Industry 4.0 and Industry 3.0?

An autonomous robot – a base on wheels fitted with computer vision technology that snaps images of grapevines in near real-time – rolls down a long row of plants in a vineyard at Cornell AgriTech in Geneva, New York.The <a href="https://news.cornell.edu/stories/2022/11/autonomous-robots-help-modernize-grape-wine-industry" target="_blank">Cornell-developed “PhytoPatholoBot”</a> detects spectral signatures of three kinds of devastating grape diseases – powdery mildew, downy mildew and grapevine leafroll virus – leading to early management of pathogens without needing farmworkers to spot them. Ultimately, data collected by such robots will help to ground-truth NASA satellite data that could one day detect plant diseases from space.This was one of many research projects presented at a technology demonstration on Aug. 20, part of a three-day “Space for Ag Tour” by NASA leaders to better understand the remote sensing needs of Cornell AgriTech researchers, specialty crop growers and local stakeholders. NASA leadership visited Cornell’s main campus, Cornell AgriTech, sites in Niagara county and the Cornell Lake Erie Research and Extension Lab in Portland, New York.The tour was designed to foster two-way dialogue with growers and stakeholders to improve NASA Earth Science’s Earth-observing satellites and instruments for specialty crop agriculture and viticulture and foster a broader conversation about how to effectively translate research into practice.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1724648875297-0822_nasa1_0.jpg" style="width: 100%px;" class="fr-fic fr-dib">Yu Jiang poses with the PhytoPatholoBotas and UV light robot as part of the “Space for Ag Tour” at the Cornell Agritech campus. Ryan Young/Cornell University“My hope is that by having this Space for Ag Tour here, by showing them New York viticulture as well as what we have to offer here at Cornell, it can help grow the specialty crop footprint within NASA as well as their footprint in pest and disease research,” said&nbsp;<a href="https://cals.cornell.edu/kaitlin-gold" target="_blank">Katie Gold</a>, the event’s main organizer and an assistant professor and the Susan Ekert Lynch Faculty Fellow in the Plant Pathology and Plant-Microbe Biology Section of the School of Integrative Plant Science (SIPS) at Cornell AgriTech, which is in the College of Agriculture and Life Sciences.&nbsp;“I think there’s amazing potential for the satellite data that’s being collected both now and in the future with the eventual launch of the Surface Biology and Geology satellite to be applied to specialty crops,” said Gold, who is an expert on ground-based and remote plant disease sensing and applied grape pathology. The meetings were intended to raise awareness among NASA leaders and scientists of the unique challenges facing specialty crops, including pests and diseases, so they can make those connections in their own research and applications, she said. &nbsp;“For NASA leadership, it’s an opportunity as they think long-term about observational needs and priorities that helps them make the case for the observations that are necessary, from a scientific perspective, to make an impact on the different concerns that farmers have,” said Alyssa Whitcraft, a research professor in the Department of Geographical Sciences at the University of Maryland, whose work focuses on satellite remote sensing of agriculture. Whitcraft is the founding executive director of NASA Acres and co-founding deputy director of NASA Harvest, respectively the U.S. and international NASA consortia on food security and agriculture.&nbsp;Gold is the institutional lead for&nbsp;<a href="https://cals.cornell.edu/news/2023/05/bringing-nasa-technology-down-earth-agriculture" target="_blank">NASA Acres at Cornell</a>.As part of the demonstrations,&nbsp;<a href="https://cals.cornell.edu/yun-yang" target="_blank">Yun Yang</a>, assistant professor in SIPS’ Soil and Crop Sciences Section at Cornell AgriTech, described her NASA Acres-funded research to monitor plant water use using remote sensing, and understanding the impacts of agricultural practices on water resource sustainability.One of Yang’s research projects seeks to estimate water loss to evaporation and transpiration in California vineyards. Vineyard managers want to know how much and when to irrigate in spring based on groundwater depletion, which is a global issue, especially in California.&nbsp;Her team built an instrument to better monitor water fluxes on the ground, which provides validation for NASA Landsat satellite data that can be used to estimate water loss to evapotranspiration when farmers irrigate. By knowing how much water is used, based on a simple water balance, they can tell how much water is needed. Yang’s team has been developing a dashboard for users. “With this dashboard, we can really hand the technology we have to the growers,” Yang said. “And when they are in the field, we can easily check the information and help them manage irrigation.” &nbsp; &nbsp;Applied roboticist <a href="https://cals.cornell.edu/yu-jiang" target="_blank">Yu Jiang</a>, assistant professor in SIPS’ Horticulture Section at Cornell AgriTech, demonstrated the PhytoPatholoBot that he and doctoral student Ertai Liu developed with Gold with funding from NASA Acres and U.S. Department of Agriculture National Institute of Food and Agriculture VitisGen3.&nbsp;The PhytoPatholoBot eliminates background noise from cloud cover or sunlight to provide a constant, high-quality image and has autonomous navigation using GPS. The bot provides a readout per row of vines with red areas denoting higher infection and white areas showing lower infection, and coupled with AI, allows for near real-time inferences of disease that growers may use to focus their treatment efforts on plants that need it most.“We can scan and scale all the field, we are also generating the results that are ready to be realized by the growers, and hopefully, while working together to integrate this data stream back to NASA to see if we can train remote sensing models and make a bigger impact for small grower stakeholders,” Jiang said. Additionally, researchers can use data from the robots to train NASA airborne sensing better than humans, Gold said.&nbsp;The hope is to use robots to automate data collection, to make it possible to gather measurements for millions of acres.&nbsp;Following Jiang,&nbsp;<a href="https://cals.cornell.edu/sara-emery" target="_blank">Sarah Emery</a>, assistant professor of entomology at Cornell AgriTech, presented her work to develop risk maps for spotted lanternfly infestations, a growing threat to grape and other growers in New York.&nbsp;“We’ve been trying to understand how this insect pest is going to be moving across the landscape and develop a model to assess the risk to individual vineyards across New York state,” Emery said.&nbsp;In collaboration with the New York State Department of Agriculture and Markets, the model being developed will overlay observations of spotted lanternfly (90,000 sightings have been recorded across the state) with maps of vineyards and locations of its preferred host, tree of heaven. Eventually, they plan to create a public-facing system so that growers can input their information and get back a level of risk.&nbsp;The hope is to use hyperspectral data (which allows for the identification of objects and materials by analyzing their unique spectral signatures) from satellites to identify light reflectance patterns of non-crop plants across the landscape to create risk maps at finer and wider scales. Emery has teamed with Gold and doctoral student Jaclyn Eller for the project.&nbsp;Also,&nbsp;<a href="https://cals.cornell.edu/jason-londo" target="_blank">Jason Londo</a>,&nbsp;associate professor of fruit crop physiology and climate adaptation in SIPS Horticulture Section at Cornell AgriTech, described using hyperspectral and multispectral approaches to understanding plant stress in grapes and apples using drones. He hopes to one day use similar data from satellites, but needs better resolution to be able to detect stress on a tree-by-tree basis, he said.Other visiting NASA personnel included Karen St. Germain, division director of the Earth Science Division in the Science Mission Directorate at NASA.&nbsp;<a href="https://cals.cornell.edu/steven-wolf" target="_blank">Steven Wolf</a>, associate professor of natural resources and the environment in CALS, organized events at Cornell’s Ithaca campus.

Members of Yu Jiang’s team demonstrate the technology. Credits: Ryan Young/Cornell University

An autonomous robot – a base on wheels fitted with computer vision technology that snaps images of grapevines in near real-time – rolls down a long row of plants in a vineyard at Cornell AgriTech in Geneva, New York.

From space to farm: readying NASA satellites to help growers

As artificial intelligence models become increasingly prevalent and are integrated into diverse sectors like health care, finance, education, transportation, and entertainment, understanding how they work under the hood is critical. Interpreting the mechanisms underlying AI models enables us to audit them for safety and biases, with the potential to deepen our understanding of the science behind intelligence itself.Imagine if we could directly investigate the human brain by manipulating each of its individual neurons to examine their roles in perceiving a particular object. While such an experiment would be prohibitively invasive in the human brain, it is more feasible in another type of neural network: one that is artificial. However, somewhat similar to the human brain, artificial models containing millions of neurons are too large and complex to study by hand, making interpretability at scale a very challenging task.&nbsp;To address this, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers decided to take an automated approach to interpreting artificial vision models that evaluate different properties of images. They developed “MAIA” (Multimodal Automated Interpretability Agent), a system that automates a variety of neural network interpretability tasks using a vision-language model backbone equipped with tools for experimenting on other AI systems.“Our goal is to create an AI researcher that can conduct interpretability experiments autonomously. Existing automated interpretability methods merely label or visualize data in a one-shot process. On the other hand, MAIA can generate hypotheses, design experiments to test them, and refine its understanding through iterative analysis,” says Tamar Rott Shaham, an MIT electrical engineering and computer science (EECS) postdoc at CSAIL and co-author on a new&nbsp;<a href="https://arxiv.org/pdf/2404.14394.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">paper about the research</a>. “By combining a pre-trained vision-language model with a library of interpretability tools, our multimodal method can respond to user queries by composing and running targeted experiments on specific models, continuously refining its approach until it can provide a comprehensive answer.”The automated agent is demonstrated to tackle three key tasks: It labels individual components inside vision models and describes the visual concepts that activate them, it cleans up image classifiers by removing irrelevant features to make them more robust to new situations, and it hunts for hidden biases in AI systems to help uncover potential fairness issues in their outputs. “But a key advantage of a system like MAIA is its flexibility,” says Sarah Schwettmann PhD ’21, a research scientist at CSAIL and co-lead of the research. “We demonstrated MAIA’s usefulness on a few specific tasks, but given that the system is built from a foundation model with broad reasoning capabilities, it can answer many different types of interpretability queries from users, and design experiments on the fly to investigate them.”&nbsp;<h2>Neuron by neuron</h2>In one example task, a human user asks MAIA to describe the concepts that a particular neuron inside a vision model is responsible for detecting. To investigate this question, MAIA first uses a tool that retrieves “dataset exemplars” from the ImageNet dataset, which maximally activate the neuron. For this example neuron, those images show people in formal attire, and closeups of their chins and necks. MAIA makes various hypotheses for what drives the neuron’s activity:&nbsp;facial expressions, chins, or neckties. MAIA then uses its tools to design experiments to test each hypothesis individually by generating and editing synthetic images — in one experiment, adding a bow tie to an image of a human face increases the neuron’s response. “This approach allows us to determine the specific cause of the neuron’s activity, much like a real scientific experiment,” says Rott Shaham.MAIA’s explanations of neuron behaviors are evaluated in two key ways. First, synthetic systems with known ground-truth behaviors are used to assess the accuracy of MAIA’s interpretations. Second, for “real” neurons inside trained AI systems with no ground-truth descriptions, the authors design a new automated evaluation protocol that measures how well MAIA’s descriptions predict neuron behavior on unseen data.The CSAIL-led method outperformed baseline methods describing individual neurons in a variety of vision models such as ResNet, CLIP, and the vision transformer DINO. MAIA also performed well on the new dataset of synthetic neurons with known ground-truth descriptions. For both the real and synthetic systems, the descriptions were often on par with descriptions written by human experts.How are descriptions of AI system components, like individual neurons, useful? “Understanding and localizing behaviors inside large AI systems is a key part of auditing these systems for safety before they’re deployed — in some of our experiments, we show how MAIA can be used to find neurons with unwanted behaviors and remove these behaviors from a model,” says Schwettmann. “We’re building toward a more resilient AI ecosystem where tools for understanding and monitoring AI systems keep pace with system scaling, enabling us to investigate and hopefully understand unforeseen challenges introduced by new models.”<h2>Peeking inside neural networks</h2>The nascent field of interpretability is maturing into a distinct research area alongside the rise of “black box” machine learning models. How can researchers crack open these models and understand how they work? Current methods for peeking inside tend to be limited either in scale or in the precision of the explanations they can produce. Moreover, existing methods tend to fit a particular model and a specific task. This caused the researchers to ask: How can we build a generic system to help users answer interpretability questions about AI models while combining the flexibility of human experimentation with the scalability of automated techniques?One critical area they wanted this system to address was bias. To determine whether image classifiers displayed bias against particular subcategories of images, the team looked at the final layer of the classification stream (in a system designed to sort or label items, much like a machine that identifies whether a photo is of a dog, cat, or bird) and the probability scores of input images (confidence levels that the machine assigns to its guesses). To understand potential biases in image classification, MAIA was asked to find a subset of images in specific classes (for example “labrador retriever”) that were likely to be incorrectly labeled by the system. In this example, MAIA found that images of black labradors were likely to be misclassified, suggesting a bias in the model toward yellow-furred retrievers.Since MAIA relies on external tools to design experiments, its performance is limited by the quality of those tools. But, as the quality of tools like image synthesis models improve, so will MAIA. MAIA also shows confirmation bias at times, where it sometimes incorrectly confirms its initial hypothesis. To mitigate this, the researchers built an image-to-text tool, which uses a different instance of the language model to summarize experimental results. Another failure mode is overfitting to a particular experiment, where the model sometimes makes premature conclusions based on minimal evidence.“I think a natural next step for our lab is to move beyond artificial systems and apply similar experiments to human perception,” says Rott Shaham. “Testing this has traditionally required manually designing and testing stimuli, which is labor-intensive. With our agent, we can scale up this process, designing and testing numerous stimuli simultaneously. This might also allow us to compare human visual perception with artificial systems.”“Understanding neural networks is difficult for humans because they have hundreds of thousands of neurons, each with complex behavior patterns. MAIA helps to bridge this by developing AI agents that can automatically analyze these neurons and report distilled findings back to humans in a digestible way,” says Jacob Steinhardt, assistant professor at the University of California at Berkeley, who wasn’t involved in the research. “Scaling these methods up could be one of the most important routes to understanding and safely overseeing AI systems.”Rott Shaham and Schwettmann are joined by five fellow CSAIL affiliates on the paper: undergraduate student Franklin Wang; incoming MIT student Achyuta Rajaram; EECS PhD student Evan Hernandez SM ’22; and EECS professors Jacob Andreas and Antonio Torralba. Their work was supported, in part, by the MIT-IBM Watson AI Lab, Open Philanthropy, Hyundai Motor Co., the Army Research Laboratory, Intel, the National Science Foundation, the Zuckerman STEM Leadership Program, and the Viterbi Fellowship.

The automated, multimodal approach developed by MIT researchers interprets artificial vision models that evaluate the properties of images. Image: iStock

MAIA is a multimodal agent that can iteratively design experiments to better understand various components of AI systems.

Researchers advance automated interpretability in AI models

In a broad sense, "machine vision" refers to the utilization of visual information and optical information using digital technologies. However, because it is being introduced to production sites in the manufacturing industry, machine vision generally refers to digital image technologies utilized in automatic inspection and process automation in factories. Machine vision consists of the following:・Imaging section (cameras) ・Interface section (board/general-purpose interface) ・Processing engine section (computer) ・Processing control section (software)etc. The image data obtained by the imaging section is loaded into the processing engine via the interface section. In the processing engine section, the processing control section performs binarization*1, edge detection*2, and other forms of image processing. Next, it outputs the image data that has completed image processing to robots and industrial equipment.*1: Processing that converts the pixel values (threshold values) that are preset for the image to the two values of black and white. *2: Processing that detects the bright and dark areas in an image.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327121894-img-0001-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 1: Machine vision configurationMachine vision is used in automatic inspection, automatic recognition, and automatic measurement, etc. For example, automatic inspection and automatic recognition are very fast compared to manual inspection. Moreover, machine vision used in automatic measurement can accurately identify small value differences that humans cannot measure. In addition, linking machine vision to AI functions is expected to produce more intelligent uses going forward.<h2>Differences Between Machine Vision, Computer Vision, and Robot Vision</h2>Computer vision and robot vision are technologies that are similar to machine vision, but they each play different roles. Here, we explain how their roles differ.<h3>Differences Between Machine Vision and Computer Vision</h3>"Computer vision" is a technology that adds equipment control functions to machine vision functions. Computer vision can control various types of equipment based on data output by machine vision. For example, this would make it possible to combine the image data output by machine vision with CAD data generated by the computer to control processors or to adjust the conveyor transport speed based on image data that captured the progress of line work. In addition, computer vision has a broad scope of utilization that naturally includes smart factories as well as referee assistance systems and virtual replays in sports and projection mapping, etc. in entertainment.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327141371-img-0002-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 2: Example of computer vision utilization<h3 id="isPasted">Differences Between Machine Vision and Robot Vision</h3>"Robot vision" is a technology that adds machine vision to robot visual functions. Machine vision attaches cameras to various types of equipment to obtain image data, but robot vision attaches cameras to the end of robot arms to obtain image data. This enables the robot to operate based on visual information obtained by itself. For example, a robot that does not use robot vision can only execute certain operations programmed in advance. However, when a robot equipped with robot vision moves its robot arm closer to an object, it then sequentially identifies the target based on image data from the camera to execute operations. Because this type of operational control is possible, a robot equipped with robot vision is capable of detailed, precise, and flexible operations that cannot be specified with a program.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327158790-img-0003-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 3: Example of robot vision utilization<h2 id="isPasted">Example of Machine Vision Utilization in a Smart Factory</h2>Machine vision is used in various types of work in a smart factory. Here, we introduce an example of machine vision utilization.<h3>Reading Characters at Ultrahigh-speed – OCR (Optical Character Recognition/Reader)</h3>OCR is a technology that scans handwritten or printed characters with an image scanner or digital camera. Machine vision scans the 2D code, serial number, and date of manufacture, etc. printed on products. The scanned data is utilized in inspections of the print quality or whether the correct characters are printed as well as the screening of the next process, etc. In many cases, the products being scanned by OCR pass through the line at high speeds. Therefore, machine vision uses cameras that can capture images at high speed and image processing systems that are capable of high-speed processing.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327179168-img-0004+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 4: OCR on a manufacturing line<h3 id="isPasted">Scanning Shapes in 2D and 3D – Visual Inspections</h3>The utilization of machine vision is considered to be the most effective means of visual inspection on a manufacturing line. Because machine vision-based visual inspection is able to inspect the accuracy of products after cutting and molding and the mounting condition of components, etc. with high speed and accuracy, it is essential for improving production efficiency and maintaining quality. The surfaces of target objects can vary widely in terms of their glossiness and light absorption, etc. The lighting used in machine vision enables products with various surface conditions to be inspected by changing the method of illumination and the color of the light rays. In addition, 3D cameras equipped with a stereophonic*3, time-of-flight*4, and other measurement systems as well as structured light*5 and other imaging systems can also measure the depth.*3: Method that uses two cameras to perform triangulation and measure the distance to an object. *4: Method that measures the distance to an object according to the time it takes for light emitted by a light source to reflect off of the object and return to a light detector. *5: Lighting method that emits light with a specific shading pattern.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327197939-img-0005.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 5: Visual inspection with a 3D camera<h3 id="isPasted">Grabbing Specific Components – Picking</h3>"Picking" on a manufacturing line refers to the task of collecting (pickup) the necessary components and products. Utilizing machine vision makes it possible to identify and pick components with the specified color and shape even if components and products with various colors and shapes are mixed together in one case.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327216910-img-0006.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 6: Picking by a robot<h3 id="isPasted">Providing Visual Functions to Robots – Robot Operational Control</h3>In recent years, industrial robots have also been used in precision work due to their high operating precision. For example, they play an active role in installing, soldering, and welding microcomponents. However, controlling such work with programming requires high-precision positioning devices. With robot vision utilizing machine vision, high-precision positioning devices are not required for the robot to perform precise positioning with the image data from the camera mounted on the end of the robot arm.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327233332-img-0007.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 7: Installing microcomponents with a robot<h2 id="isPasted">Future of Machine Vision – AI Vision</h2>As various types of industrial equipment shift to using AI, one form of machine vision evolution that shows promise is "AI vision."<h3>What is AI Vision?</h3>AI vision is a system that equips lightweight AI chips in smartphones, etc. with cameras and camera functions to make inferences about captured image data. AI vision resembles computer vision in that it uses AI (artificial intelligence), but computer vision mainly makes inferences with a cloud AI service (Figure 8). On the other hand, AI vision equips an AI module in an edge device to make inferences, and the inference results are stored on the edge server. The edge server communicates with the cloud AI only when necessary. (Figure 9).<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327267580-1721327267580.png" class="fr-fic fr-dib">Therefore, AI vision has the following advantages over computer vision: ・Able to decrease the volume of communication with the cloud AI service ・Able to prevent external leaks of data collected by the sensors ・Superior responsiveness Furthermore, the cameras and smartphones used in AI vision are called "edge AI devices" while the method that makes inferences on the edge device is called "edge AI."<h3>What Technologies Are Required to Realize AI Vision?</h3>Although AI vision has the advantages described above, AI modules equipped on edge devices require the following types of functions. ・Superior energy conservation due to restrictions on battery capacity ・Ability to perform high-speed processing due to the responsiveness requirement ・Superior heat dissipation due to installation in a limited space More than anything, it is important that the AI modules be compact.<h2>Summary</h2>This article explained the topic of machine vision and the similar phrases of computer vision and robot vision as well as AI vision. Going forward, it is expected that machine vision and AI will evolve in a close relationship with the evolution in deep learning for image data. In addition, new AI systems and products are expected to be developed through the proper use of cloud AI and edge AI. These trends will likely become essential technologies for smart factory robots, automatic inspection, automatic recognition, and automatic measurement equipment that require visual information that is closer to the human eye and decision-making capabilities that exceed that of humans.

computer vision

Profiles