A less wasteful way to train large language models, such as the GPT series, finishes in the same amount of time for up to 30% less energy, according to a new study from the University of Michigan.The approach could save enough energy to power <a href="https://www.eia.gov/energyexplained/use-of-energy/electricity-use-in-homes.php" target="_blank" rel="noreferrer noopener">1.1 million U.S. homes</a> in 2026, based on Wells Fargo’s projections of <a href="https://www.marketwatch.com/story/ai-could-demand-a-shocking-amount-of-electricity-check-out-this-chart-e91e306d" target="_blank" rel="noreferrer noopener">AI power demand</a>. It could also take a bite out of the International Monetary Fund’s prediction that data centers could account for <a href="https://www.imf.org/en/Blogs/Articles/2024/08/15/carbon-emissions-from-ai-and-crypto-are-surging-and-tax-policy-can-help" target="_blank" rel="noreferrer noopener">1.2% of the world’s carbon emissions by 2027</a>—and the <a href="https://theconversation.com/ais-excessive-water-consumption-threatens-to-drown-out-its-environmental-contributions-225854">water demands</a> that come with that energy use.Some experts say that these costs could be outweighed by environmental benefits. They argue that AI could be a <a href="https://news.un.org/en/story/2023/11/1143187" target="_blank" rel="noreferrer noopener">“game changer” for fighting climate change</a> by identifying ways to optimize supply chains and the grid, manage our energy needs, and improve research on climate change. Still, that doesn’t excuse squandering energy, and some of the power used to train AI has zero impact on training time and model accuracy.“Why spend something when there’s no point?” said <a href="https://eecs.engin.umich.edu/people/chowdhury-mosharaf/" target="_blank" rel="noreferrer noopener">Mosharaf Chowdhury</a>, an associate professor of computer science and engineering and the corresponding author of the study presented at the <a href="https://sigops.org/s/conferences/sosp/2024/index.html" target="_blank" rel="noreferrer noopener">30th Symposium on Operating Systems Principles</a> last Monday.“We can’t keep building bigger and bigger data centers because we won’t have the power to run them,” said Chowdhury. “If we can reduce the energy consumed by AI, we can reduce AI’s carbon footprint and cooling requirements and allow for more computation to fit within our current energy constraints.”The energy waste is created when AI training is unequally divided between GPUs, which are computer processors specialized for large data and graphics applications. Although it opens the door for waste, splitting the work is necessary for processing huge datasets.“AI models today are so large, they cannot fit inside a single computer processor,” said <a href="https://jaewonchung.me/about" target="_blank" rel="noreferrer noopener">Jae-Won Chung</a>, a doctoral student in computer science and engineering and the first author of the study. “They need to be divided into tens of thousands of processors to be trained, but dividing the models in perfectly equal sizes across all processors is practically impossible.”The training jobs are so difficult to evenly split up because some tasks need to be grouped together on the same processor—like how each installment of a book series will be grouped together in an organized shelf. Depending on how the tasks are grouped, some processors might get stuck with the AI-training equivalent of the Encyclopedia Britannica while others get assigned a fantasy trilogy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1731630851359-AI_waste_Infographic-768x1470.jpg" style="width: 100%px;" class="fr-fic fr-dib">Infographic credit: Rachel Voigt, Michigan Engineering.Because current training methods run each processor at top speed, processors with a lighter load will finish their calculations before other processors. This doesn’t speed up training, which isn’t complete until every processor finishes its job—but it is wasteful because faster calculations require more energy. In addition, problems such as faulty hardware or network delays create energy waste by slowing down a single processor’s computing speed.To save energy, the researchers developed a software tool, called&nbsp;<a href="https://ml.energy/zeus/research_overview/perseus/" target="_blank" rel="noreferrer noopener">Perseus</a>, that identifies a critical path, or a series of subtasks that will take the longest time to complete. Then, Perseus slows down processors that aren’t on the critical path so that they all finish their jobs around the same time—eliminating unnecessary power use.“Reducing the power cost of AI can have important implications for equitable AI access,” said Chowdhury. “If a country doesn’t have enough power to run a big model, they might need to use services from far away, or be stuck running smaller, less accurate models. This gap could further perpetuate disparity between different communities.”The team tested Perseus by training GPT-3, three other large language models and one computer vision model.Perseus is an open-sourced tool available as part of&nbsp;<a href="https://ml.energy/zeus/" target="_blank" rel="noreferrer noopener">Zeus</a>, a tool for measuring and optimizing AI energy consumption.&nbsp;

Smarter use of processor speeds saves energy without compromising training speed and performance.

Up to 30 percent of the power used to train AI is wasted. Here's how to fix it.

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries. There are also full-sequence diffusion models like Sora, which convert words into dazzling, realistic visuals by successively “denoising” an entire video sequence.Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have proposed a simple change to the diffusion training scheme that makes this sequence denoising considerably more flexible.When applied to fields like computer vision and robotics, the next-token and full-sequence diffusion models have capability trade-offs. Next-token models can spit out sequences that vary in length. However, they make these generations while being unaware of desirable states in the far future — such as steering its sequence generation toward a certain goal 10 tokens away — and thus require additional mechanisms for long-horizon (long-term) planning. Diffusion models can perform such future-conditioned sampling, but lack the ability of next-token models to generate variable-length sequences.Researchers from CSAIL want to combine the strengths of both models, so they created a sequence model training technique called “Diffusion Forcing.” The name comes from “Teacher Forcing,” the conventional training scheme that breaks down full sequence generation into the smaller, easier steps of next-token generation (much like a good teacher simplifying a complex concept).<iframe width="640" height="360" src="https://www.youtube.com/embed/m4rhzR-h-JU?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Diffusion Forcing found common ground between diffusion models and teacher forcing: They both use training schemes that involve predicting masked (noisy) tokens from unmasked ones. In the case of diffusion models, they gradually add noise to data, which can be viewed as fractional masking. The MIT researchers’ Diffusion Forcing method trains neural networks to cleanse a collection of tokens, removing different amounts of noise within each one while simultaneously predicting the next few tokens. The result: a flexible, reliable sequence model that resulted in higher-quality artificial videos and more precise decision-making for robots and AI agents.By sorting through noisy data and reliably predicting the next steps in a task, Diffusion Forcing can aid a robot in ignoring visual distractions to complete manipulation tasks. It can also generate stable and consistent video sequences and even guide an AI agent through digital mazes. This method could potentially enable household and factory robots to generalize to new tasks and improve AI-generated entertainment.“Sequence models aim to condition on the known past and predict the unknown future, a type of binary masking. However, masking doesn’t need to be binary,” says lead author, MIT electrical engineering and computer science (EECS) PhD student, and CSAIL member Boyuan Chen. “With Diffusion Forcing, we add different levels of noise to each token, effectively serving as a type of fractional masking. At test time, our system can “unmask” a collection of tokens and diffuse a sequence in the near future at a lower noise level. It knows what to trust within its data to overcome out-of-distribution inputs.”In several experiments, Diffusion Forcing thrived at ignoring misleading data to execute tasks while anticipating future actions.When implemented into a robotic arm, for example, it helped swap two toy fruits across three circular mats, a minimal example of a family of long-horizon tasks that require memories. The researchers trained the robot by controlling it from a distance (or teleoperating it) in virtual reality. The robot is trained to mimic the user’s movements from its camera. Despite starting from random positions and seeing distractions like a shopping bag blocking the markers, it placed the objects into its target spots.To generate videos, they trained Diffusion Forcing on “Minecraft” game play and colorful digital environments created within Google’s <a href="https://github.com/google-deepmind/lab">DeepMind Lab Simulator</a>. When given a single frame of footage, the method produced more stable, higher-resolution videos than comparable baselines like a Sora-like full-sequence diffusion model and ChatGPT-like next-token models. These approaches created videos that appeared inconsistent, with the latter sometimes failing to generate working video past just 72 frames.Diffusion Forcing not only generates fancy videos, but can also serve as a motion planner that steers toward desired outcomes or rewards. Thanks to its flexibility, Diffusion Forcing can uniquely generate plans with varying horizon, perform tree search, and incorporate the intuition that the distant future is more uncertain than the near future. In the task of solving a 2D maze, Diffusion Forcing outperformed six baselines by generating faster plans leading to the goal location, indicating that it could be an effective planner for robots in the future.Across each demo, Diffusion Forcing acted as a full sequence model, a next-token prediction model, or both. According to Chen, this versatile approach could potentially serve as a powerful backbone for a “world model,” an AI system that can simulate the dynamics of the world by training on billions of internet videos. This would allow robots to perform novel tasks by imagining what they need to do based on their surroundings. For example, if you asked a robot to open a door without being trained on how to do it, the model could produce a video that’ll show the machine how to do it.The team is currently looking to scale up their method to larger datasets and the latest transformer models to improve performance. They intend to broaden their work to build a ChatGPT-like robot brain that helps robots perform tasks in new environments without human demonstration.“With Diffusion Forcing, we are taking a step to bringing video generation and robotics closer together,” says senior author Vincent Sitzmann, MIT assistant professor and member of CSAIL, where he leads the Scene Representation group. “In the end, we hope that we can use all the knowledge stored in videos on the internet to enable robots to help in everyday life. Many more exciting research challenges remain, like how robots can learn to imitate humans by watching them even when their own bodies are so different from our own!”<hr>Chen and Sitzmann wrote the paper alongside recent MIT visiting researcher Diego Martí Monsó, and CSAIL affiliates: Yilun Du, a EECS graduate student; Max Simchowitz, former postdoc and incoming Carnegie Mellon University assistant professor; and Russ Tedrake, the Toyota Professor of EECS, Aeronautics and Astronautics, and Mechanical Engineering at MIT, vice president of robotics research at the Toyota Research Institute, and CSAIL member. Their work was supported, in part, by the U.S. National Science Foundation, the Singapore Defence Science and Technology Agency, Intelligence Advanced Research Projects Activity via the U.S. Department of the Interior, and the Amazon Science Hub. They will present their research at NeurIPS in December.

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries.

Combining next-token prediction and video diffusion in computer vision and robotics

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/b921c890-089c-4f02-b05d-59a0c9850785?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">Episode Notes</h3>(0:50) - <a href="https://www.wevolver.com/article/ai-helps-detect-and-monitor-infrastructure-defects">Infrastructure Defect Detecting AI</a>Become a founding reader of our newsletter: <a href="http://read.thenextbyte.com/">http://read.thenextbyte.com/</a><hr><h2 data-attrid="title" data-local-attribute="d3bn" data-ved="2ahUKEwiBhLnmu4aAAxVmwjgGHcDWC38Q3B0oAXoECEgQEQ" id="isPasted">Transcript</h2>What's up, folks? As you know, cracks in buildings and roads and bridges can cause some serious damage if they're not found early enough. Today, we're talking about a team from EPFL that's gonna fix this problem using AI that can help detect cracks automatically and let us know when we need to do repairs. I think it's interesting, so let's jump into it.I'm Daniel, and I'm Farbod. And this is the NextByte Podcast. Every week, we explore interesting and impactful tech and engineering content from Wevolver.com and deliver it to you in bite sized episodes that are easy to understand, regardless of your background.&nbsp;Daniel:&nbsp;What's up friends? Like we said, today we're talking about the future of crack detection in infrastructure and why that's so important. We're not just talking about like cracks and buildings and roads and, sometimes it makes them look ugly or ivy can grow in them. But the problem is that they can cause serious damage if not found early enough. The traditional methods we have for inspecting and looking for these cracks are really slow, really costly, really hard to do in dangerous areas. And there's a lot of human error involved with it as well. So, we're talking about some awesome research here where they're using AI to try and help find and monitor cracks, but they're even taking kind of a unique AI approach as well that doesn't require detailed pixel by pixel labeling either. So, I'm pretty stoked about this one.Farbod:&nbsp;Dude, same. I feel like infrastructure improvement has been talked about for so long. I remember being in middle school and politicians were promising the next infrastructure build and how like, our infrastructure is aging and we need to replace it. Well, before this part, I was just kind of curious. I'm like, let's pull up some stats just about the US alone, right? I don't know if you knew this, but a freeway bridge inspection can take, can cost about $5,000 and take about eight hours to do. And that doesn't take into account the detour traffic causing people pains to get to work, et cetera, et cetera. And on top of that, the way we do grading of these cracks tends to be kind of subjective. So, the person that's doing it might grade it more, I don't know, liberally than the person that did last year. So that person might've thought things were going bad. This person says everything's good. So that kind of creates a weird situation where we can't track how bad it's getting super well.Daniel:&nbsp;Or can you trust the data that comes from these expectations? When you spend this much money and you spend this much time, you may not be even able to trust the data that comes out of it.Farbod: And to just make matters worse, there's something like one out of every three bridges in the US needs repair or replacement. So, one, you don't know if things are actually bad. Two, it seems like at least one third are kind of bad, maybe more. So not super comfortable about driving over a bridge at this point, but I saw something interesting about drones. Drones have had great improvement over the past couple of years. People were saying on average it's about 40% cheaper. So that makes it more affordable for local governments to take them on. But still, I'm guessing no one is super excited about shelling out over $3,000 to inspect every bridge they got. So yeah, sticky situation to be in.Daniel:&nbsp;Well, and just a quick throwback here. I would say like in the broader realm of infrastructure detection, both of our capstone projects, when we were graduating our undergrad program was related to using robotics and automation to try and detect issues with infrastructure before they progress to something that becomes a catastrophic failure. And that's not to say that we're experts in this topic at all. I think we were just at the tip of the iceberg here, but it is interesting to note that in separate graduating classes for about, and I both had this capstone project because it's such a big problem and we're still sitting here saying, maybe now we finally got some AI solution, which can help with this in a more robust manner, which is exciting for both of us, I think on a personal note, to see this field start to progress.Farbod: Dude, absolutely. Back when we were young men, which we're not anymore, at least my team, we were toying around with the idea of using AI, but even then, there wasn't a whole lot of excitement around it. Specifically, we were looking at corrosion, but now it looks like the game has completely shifted. We're going to EPFL for this episode, but if we can transition to the secret sauce, these folks are talking about how AI can actually completely remove the human element here and in the best way possible. We've been talking about subjectivity, how that can result in variance from person to person, and if a crack is getting better or worse. These folks are saying, hey, we can just train a very simple model that can tell us if there's a crack or if there's no crack, like binary. Once it learns how to detect the crack well, then it's just a matter of saying “cool, where are you seeing the crack and how is it changing over time?” Which in the world of like, you know, computer vision is a fairly simple and straightforward analysis. And the main application they're looking at for this is railroads, right? They're saying, railroads are used very often if there's cracks that grow that can cause catastrophic failures. So how do we maintain this and like monitor it on a regular basis? Well, they're like trains are passing through this bridges and stuff. What if they were the ones that were doing the analysis completely autonomously, no added costs, no added time, no requirement for shut down.Daniel: It's not sending people on harnesses to rappel from these. Honestly, you should just check on the link in the show notes, click for no other reason than to see the beautiful photo of this train on a railroad bridge going across a canyon in Switzerland where, and I wanted to confirm this, I wasn't 100% sure. The Matterhorn-Gothard bond network. I'm pretty sure Nellie and I took this train on our honeymoon, which is pretty cool.Farbod: Wow. That's special.Daniel:&nbsp;If not something very similar because all of Switzerland is full of beautiful trains and bridges and mountains, but I'm pretty sure we went on this exact railroad section where they're starting testing, but yeah, if for no other reason, check out this photo, but for me to get back to my main point, this is a bridge suspended several stories above the bottom of a canyon. It would cost a lot and you'd have to pay someone a lot to be able to visually inspect all of that manually, like as a person by hand. And what they're trying to do now is instead send a drone, but not in the form of the quadcopter drone that you think of probably when we say drone. The solution is, send a drone rail car, right? An instrumented rail car down this railroad and try and collect a bunch of data on, where are their cracks? Are they propagating? Are they staying the same? Which ones need to be addressed? Continuing to understand and map the ever evolving, let's say ecosystem around all these bridges and this infrastructure and understand when there's issues and indicate whether they need further inspection or whether they need immediate repair, et cetera.Farbod:&nbsp;Dude, totally agree. And the solution seems so simple when you say it out loud, but it really is, I don't know, I guess the best solutions are the simple ones. And what it made me think of is obviously this is great for the railroad system. But then we've been talking about like as a commuter that's going in a tunnel. That one tunnel in Boston I always hate it gives me the creeps when I'm going through it or when I'm going through bridges to Maryland and stuff like that. It would be great if there was something that could do the analysis there and in my mind, I'm like we're definitely a long way away from having autonomous vehicles doing these trips for us. But Uber and Lyft, they're drivers that are operating every single day that are doing these normal routes that people take to go to work and whatnot. I wonder if there was an opportunity there to say, hey, while you're doing this, mount this device on the back of your car so that whenever your GPS detects you're going from the beginning of the tunnel to the back of the tunnel, you do a scan and then that data uploads to a cloud somewhere and we can do analysis over time to see if we need to patch any cracks or what?Daniel: I'm like wondering if you read my mind here. I went to buses, because I'm like, the department of transportation in a city may also own the buses, right? So, they're in charge of the bridges. They're also in charge of the buses. Wonder if they can just put a bunch of instruments on top of a bus. In my mind, it's like the closest analogy to a rail car. Put a bunch of this infrastructure on top of a bus and this instrumentation, see if you can analyze the infrastructure around us using that same method as well. But I also want to jump a little bit into the specific AI methods they're using. And I'm not an expert here by any means, but I took some notes from the paper that I think are interesting. That there have previously tried to do AI classification of images, but it's too computationally intensive to justify the cost. So, if you were to use AI to try and evaluate millions of photos of an entire bridge. It may actually cost more than actually sending a human down there to do it. Because one of the things that I had to do is classify each pixel by pixel, trying to understand whether it was light or if it's dark and look at contrast to understand what is a crack, then it classifies a crack and that took a lot of computational energy. One of the things that they did is they, they train their AI ahead of time with images labeled as crack or no crack. And then it uses explainable AI is what they say is their unique method here to try and look at a new photo and circle, basically highlight a cracked area and create a map showing where cracks are and where they aren't. And essentially what it's doing is it's just taking photo after photo after photo and trying to trace the area around where it thinks the crack is. And over a period of time, it's trying to look at the same photo from the same location the last time this train came through and look at whether this crack got bigger or if it stayed the same. And if cracks are serially getting bigger and bigger and bigger from these images, it warrants sending someone down there, an expert down there to take a look at it. And one of the things that they mentioned is this cut the processing and labeling effort, as opposed to previous AI methods by about 90%, which makes it cheaper and faster and worthwhile to implement in an autonomous manner. It's not quite as accurate as these full AI models, but they said it can still help directionally track crack growth. And they think that they can hone it essentially and get rid of more of the area that they're experiencing right now. But essentially the game plan I guess is to go test this out in the field, get more learnings, understand more but in practice even if it has the same performance that it has today you can use this as kind of like a non-invasive constant monitoring where it looks at the rails, it looks at the ties, it looks at the retaining walls and tries to understand if there's anything wrong with the railroad. And you could have this on every train or one of every 10 trains passing through this railroad. Even though I think right now it's got 35% error, which again, it sounds high compared to most AI models, but if you're telling me, you can tell me 65% of the time, whether a crack's growing or not. If I've got two or three trains that pass through over a series of a week and show that this crack indicates that this crack may be growing, the playbook here is not to say, all right, let's shut down the bridge and tear it down. The playbook here could be, let's go send one of those expert inspectors that we used to have doing this job full time and had to shut down the railroad to do now that we know that there's something that's worth further evaluation, let's send one of those crack experts down there to understand what's going on and see whether it needs repair.Farbod: And this is a great example, at least in my opinion of complimentary automation, right? Like here's a task that was bad for people to do manually. And some places are like in America, it's just straight up not done at times. Now you have automation that's taking care of the tedious part. And then you have the human that's taking on the challenging part, which is, hey, we're like 90% sure there is a crack, please go out there and now characterize it and give us feedback on if it's catastrophic or if, you know, it's a simple crack that needs to be fixed, etc, etc. So, I feel like this is the best use of every technology or every resource, think of the human expert as a resource. And this is kind of going against the AI is going to get rid of all jobs mentality.Daniel: Well, and I think what we, the intended end here is to be able to do continuous monitoring, collect data all the time to understand which regions are susceptible to cracks, and then use that as a heat map essentially to show someone who's again, got expertise in this field to go, this is the area that you need to inspect. This is the area that you need to monitor. This is the area that may need potential repair as opposed to the other portions of track that maybe didn't need as much attention and they were getting it before because you had to go check every single mile of this thing manually. And maybe you used to only check one, every single mile of this thing manually, one out of every five years. And now we could have a train passing on it five times a day that tells us how the railroad's doing. So, I appreciate here, like you said, it's a complimentary solution between automation and humans. But it also gets us an immediate benefit, which is a lot more data, a lot more consistently on how our infrastructure is doing and when it needs to be repaired.Farbod: Totally agree, man. You want to do us a favor and wrap up the episode?Daniel:&nbsp;I will. What if we could stop big cracks in buildings and roads before they cause big problems, I'm thinking of bridges collapsing or bridges needing to be shut down because they need repair or rip, cars getting derailed. These are big problems and we're talking today about a new AI method that helps find and track these cracks automatically without needing a ton of expensive detailed data. Essentially, they're getting the job done with 90% less effort. That's what their new tech promises. And they use this system called explainable AI that spots cracks with simple image labels, saving time and money, keeping an eye on crack growth to let experts know where they need to address cracks and where we might need repairs in the future. I think it's a lot like, I don't know, Google Maps navigation or some sort of traffic heat map, but it tells us where most areas are prone to cracks and where we need repairs so that folks know where they need to focus their efforts.Farbod:&nbsp;Money.Daniel:&nbsp;Thanks, my dude.Farbod:&nbsp;You killed it, dude. All right, folks, thank you so much for listening. And as always, we'll catch you in the next one.Daniel:&nbsp;Peace.<hr>As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.To learn more about this show, please visit our<a href="https://www.wevolver.com/profile/the.next.byte">&nbsp;shows page</a>. By following the page, you will get automatic updates by email when a new show is published. Be sure to give us a follow and review on<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" target="_blank" rel="nofollow">&nbsp;Apple</a> podcasts,<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" target="_blank" rel="nofollow">&nbsp;Spotify</a>, and most of your favorite podcast platforms!--The Next Byte: We're two engineers on a mission to simplify complex science &amp; technology, making it easy to understand. In each episode of our show, we dive into world-changing tech (such as AI, robotics, 3D printing, IoT, &amp; much more), all while keeping it entertaining &amp; engaging along the way.

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems

Podcast: Saving America's Collapsing Infrastructure

More than 40,000 people have died on U.S. roads in each of the last three years. In an effort to eliminate these preventable accidents, The Department of Transportation has awarded The City of Pittsburgh, in partnership with Ding Zhao, $1.3 million to analyze Pittsburgh’s current transportation infrastructure and inform safety-first upgrades by the end of 2026.As part of the <a href="https://www.transportation.gov/grants/SS4A" rel="noopener" target="_blank">Biden-Harris Administration’s Bipartisan Infrastructure Law for the Safe Streets and Roads for All</a> (SS4A) program, the project will support a road safety audit, the development of a <a href="https://engage.pittsburghpa.gov/vision-zero" rel="noopener" target="_blank">Vision Zero</a>-focused streets design manual along with quick build projects and a Vision Zero ambassador program to eliminate traffic fatalities and increase equitable mobility.<a href="https://engineering.cmu.edu/directory/bios/zhao-ding.html">Ding Zhao</a>, director of the Safe AI Lab and an associate professor of <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank">mechanical engineering</a> at Carnegie Mellon University, will use large language models (LLMs) to analyze Pittsburgh’s streets, identify which intersections are the highest-risk for drivers and pedestrians, and work with city officials to test new street design improvements.“Large language models offer a very scalable, very powerful way to collect data and run simulations,” explained Zhao. “By collecting more data, more efficiently, we can be inclusive of rare events and vulnerable users.”<iframe width="640" height="360" src="https://www.youtube.com/embed/SDAoLGEE2CU?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-24="true" id="200260888" title="Simulation of Pittsburgh's city streets">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</iframe>Simulations of Pittsburgh’s streets will be used by Ding Zhao to identify intersections that pose the highest-risk for drivers and pedestrians. The team will work with city officials to test new street design improvements as part of the Safe Streets and Roads for All Program.The team will also use LLMs to analyze the results of their simulations, an approach Zhao says is novel to transportation and infrastructure.“We have a clear goal for this project and by working with the City of Pittsburgh we are guaranteeing real world impact,” said Zhao. “I’m excited to see how my research will positively affect our neighborhoods. If we can save even one person, I’ll be very proud.”<blockquote>I’m excited to see how my research will positively affect our neighborhoods. If we can save even one person, I’ll be very proud.<cite>Ding Zhao, Associate Professor, Mechanical Engineering</cite></blockquote>“We’re excited to partner with the research teams at Carnegie Mellon University as we work toward our goal of zero traffic fatalities,” said Mayor Ed Gainey. “To make our streets safer, we need to break down silos and build strong partnerships between the City of Pittsburgh and our private institutions. Together, we can drive the kind of innovation that makes a real difference in roadway safety.”Zhao, who has been working to advance AI in self-driving cars for nearly a decade, is also eager to explore how enhanced infrastructure can make the roads safer for semi and fully autonomous vehicles.The team will leverage on its in-house tool <a href="https://safebench.github.io/">SafeBench</a>, an open-sourced platform designed to assess and improve the safety of AI models based on a high-fidelity autonomous driving simulator. SafeBench provides a suite of standardized benchmarks and testing environments for evaluating how AI models perform in dynamic, safety-critical situations, such as autonomous driving or infrastructure planning. This tool can be integrated into projects like Pittsburgh’s transportation audit by offering a rigorous framework for analyzing AI-driven systems’ resilience under various conditions, including multimodal perturbations and rare events. By ensuring the safety and robustness of AI systems, SafeBench will help cities like Pittsburgh deploy cutting-edge AI technologies with confidence, minimizing risks and enhancing the overall safety of public infrastructure.<iframe width="640" height="360" src="https://www.youtube.com/embed/TnJxpTJ1OKQ?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-24="true" id="330679141" title="Scenarios for Perception">&nbsp;&nbsp;&nbsp;</iframe>“SafeBench allows us to identify potential safety issues in the AI-empowered systems early on, which helps improve their reliability as they are integrated into real-world applications,” said Haohong Lin, a Ph.D. student at Carnegie Mellon University working with Zhao.“I have been working on self-driving cars for 17 years,” Zhao said. “This project opens a new door for us to explore how streets can be modified to provide a safer environment for us all both today and in the future when humans will share the road with AVs.”

Ding Zhao is teaming up with the City of Pittsburgh to improve road safety and eliminate traffic fatalities using AI.

Safe AI for safer city streets

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/b921c890-089c-4f02-b05d-59a0c9850785?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Thanks to recent advances in artificial intelligence (AI), civil engineers can inspect large-scale infrastructure more efficiently and cost-effectively, while also monitoring the progression of damage severity over time. A team of EPFL researchers has demonstrated the feasibility of an AI-driven method for crack detection, growth and monitoring, and will soon test it on the railway section between Zermatt and Brig in Valais Canton.AI can help improve railway safety by enabling automated inspections of tracks, crossties, ballasts and retaining walls. Researchers at EPFL’s Intelligent Maintenance and Operations Systems (IMOS) Laboratory have developed an AI-driven method that improves the efficiency of crack detection in concrete structures. Their research, recently published in Automation in Construction, introduces a novel method that employs explainable artificial intelligence, or a form of AI which allows users to understand the basis of AI decisions.“We trained an algorithm to differentiate between images with and without cracks in concrete walls [a binary classification task] by feeding it hundreds of image samples from both categories. Then we asked the algorithm to highlight which pixels it used to make its decision,” says Florent Forest, a scientist at the IMOS lab and the study’s lead author. The algorithm successfully identified the pixels corresponding to cracks. “With our approach, users can feed the algorithm images taken over several years of a section of railway – or of any other kind of infrastructure that’s inspected regularly – and ask it to quantify the severity of cracks in walls and crossties over time. This helps infrastructure operators plan their maintenance more effectively,” he says.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI3NzY1MzQwMzA2LTRiZTNmODk0LmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Concrete crossties must be maintained by rail network operators. © iStock PhotosEnhanced inspectionsCurrently, railway operators regularly inspect the condition of infrastructure such as retaining walls by using predefined criteria, where grades are assigned by experienced inspectors. However, this process is often prone to subjective evaluations and makes it difficult to track changes over time, especially when different inspectors assess the same section of infrastructure at different points in time.Thanks to advancements in digitalization, railway operators can monitor track conditions using a specialized monitoring coach equipped with various measuring devices and side and floor cameras for the visual inspection of rails, concrete crossties and retaining walls. By using these AI-driven systems for damage severity quantification, the inspection process can be automated, making it more objective, accurate and easier to compare over time.The EPFL research team will test its method on sections of railway between Zermatt and Brig and between Brig and Disentis. These sections include a number of retaining walls of different shapes and materials, making the task significantly challenging for the algorithm. The team has already collected drone images, along with those from the monitoring coach, and will use its AI algorithm to assist the railway operator in monitoring its infrastructure more frequently and systematically.<hr>References<a href="https://doi.org/10.1016/j.autcon.2024.105497" target="_blank" rel="noopener">Florent Forest, Hugo Porta, Devis Tuia and Olga Fink, “From classification to segmentation with explainable AI: A study on crack detection and growth monitoring,” Automation in Construction, September 2024.</a>

The Matterhorn-Gotthard Bahn network links the plain to Zermatt. © iStock Photos

A team of EPFL researchers has demonstrated the feasibility of an AI-driven method for crack detection, growth and monitoring, and will soon test it on the railway section between Zermatt and Brig in Valais Canton.

AI helps detect and monitor infrastructure defects

Imagine having to straighten up a messy kitchen, starting with a counter littered with sauce packets. If your goal is to wipe the counter clean, you might sweep up the packets as a group. If, however, you wanted to first pick out the mustard packets before throwing the rest away, you would sort more discriminately, by sauce type. And if, among the mustards, you had a hankering for Grey Poupon, finding this specific brand would entail a more careful search.MIT engineers have developed a method that enables robots to make similarly intuitive, task-relevant decisions.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727764894016-MIT-Scene-Tasks-01-press_0.jpg" style="width: 100%px;" class="fr-fic fr-dib">From left to right: team members Lukas Schmid, Nathan Hughes, Dominic Maggio, Yun Chang, and Luca Carlone. Credit: Andy RyanThe team’s new approach, named Clio, enables a robot to identify the parts of a scene that matter, given the tasks at hand. With Clio, a robot takes in a list of tasks described in natural language and, based on those tasks, it then determines the level of granularity required to interpret its surroundings and “remember” only the parts of a scene that are relevant.In real experiments ranging from a cluttered cubicle to a five-story building on MIT’s campus, the team used Clio to automatically segment a scene at different levels of granularity, based on a set of tasks specified in natural-language prompts such as “move rack of magazines” and “get first aid kit.”The team also ran Clio in real-time on a quadruped robot. As the robot explored an office building, Clio identified and mapped only those parts of the scene that related to the robot’s tasks (such as retrieving a dog toy while ignoring piles of office supplies), allowing the robot to grasp the objects of interest.Clio is named after the Greek muse of history, for its ability to identify and remember only the elements that matter for a given task. The researchers envision that Clio would be useful in many situations and environments in which a robot would have to quickly survey and make sense of its surroundings in the context of its given task.“Search and rescue is the motivating application for this work, but Clio can also power domestic robots and robots working on a factory floor alongside humans,” says Luca Carlone, associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), principal investigator in the Laboratory for Information and Decision Systems (LIDS), and director of the MIT SPARK Laboratory. “It’s really about helping the robot understand the environment and what it has to remember in order to carry out its mission.”The team details their results in a <a href="https://dspace.mit.edu/handle/1721.1/157072" target="_blank">study appearing today</a> in the journal Robotics and Automation Letters. Carlone’s co-authors include members of the SPARK Lab: Dominic Maggio, Yun Chang, Nathan Hughes, and Lukas Schmid; and members of MIT Lincoln Laboratory: Matthew Trang, Dan Griffith, Carlyn Dougherty, and Eric Cristofalo.<h2>Open fields</h2>Huge advances in the fields of computer vision and natural language processing have enabled robots to identify objects in their surroundings. But until recently, robots were only able to do so in “closed-set” scenarios, where they are programmed to work in a carefully curated and controlled environment, with a finite number of objects that the robot has been pretrained to recognize.In recent years, researchers have taken a more “open” approach to enable robots to recognize objects in more realistic settings. In the field of open-set recognition, researchers have leveraged deep-learning tools to build neural networks that can process billions of images from the internet, along with each image’s associated text (such as a friend’s Facebook picture of a dog, captioned “Meet my new puppy!”).From millions of image-text pairs, a neural network learns from, then identifies, those segments in a scene that are characteristic of certain terms, such as a dog. A robot can then apply that neural network to spot a dog in a totally new scene.But a challenge still remains as to how to parse a scene in a useful way that is relevant for a particular task.“Typical methods will pick some arbitrary, fixed level of granularity for determining how to fuse segments of a scene into what you can consider as one ‘object,’” Maggio says. “However, the granularity of what you call an ‘object’ is actually related to what the robot has to do. If that granularity is fixed without considering the tasks, then the robot may end up with a map that isn’t useful for its tasks.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727764931766-MIT-Scene-Tasks-03-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">Clio quickly segments a scene such as a cluttered cubicle (pictured), and feeds the segments through an “information bottleneck” algorithm that focuses on the segments that only represent target objects relevant to the task at hand. Colored cubes represent segment regions that Clio has picked identified as task-relevant objects that a robot can focus on while ignoring the rest of the scene. Credit: Courtesy of the researchers<h2>Information bottleneck</h2>With Clio, the MIT team aimed to enable robots to interpret their surroundings with a level of granularity that can be automatically tuned to the tasks at hand.For instance, given a task of moving a stack of books to a shelf, the robot should be able to &nbsp;determine that the entire stack of books is the task-relevant object. Likewise, if the task were to move only the green book from the rest of the stack, the robot should distinguish the green book as a single target object and disregard the rest of the scene — including the other books in the stack.The team’s approach combines state-of-the-art computer vision and large language models comprising neural networks that make connections among millions of open-source images and semantic text. They also incorporate mapping tools that automatically split an image into many small segments, which can be fed into the neural network to determine if certain segments are semantically similar. The researchers then leverage an idea from classic information theory called the “information bottleneck,” which they use to compress a number of image segments in a way that picks out and stores segments that are semantically most relevant to a given task.“For example, say there is a pile of books in the scene and my task is just to get the green book. In that case we push all this information about the scene through this bottleneck and end up with a cluster of segments that represent the green book,” Maggio explains. “All the other segments that are not relevant just get grouped in a cluster which we can simply remove. And we’re left with an object at the right granularity that is needed to support my task.”The researchers demonstrated Clio in different real-world environments.“What we thought would be a really no-nonsense experiment would be to run Clio in my apartment, where I didn’t do any cleaning beforehand,” Maggio says.The team drew up a list of natural-language tasks, such as “move pile of clothes” and then applied Clio to images of Maggio’s cluttered apartment. In these cases, Clio was able to quickly segment scenes of the apartment and feed the segments through the Information Bottleneck algorithm to identify those segments that made up the pile of clothes.They also ran Clio on Boston Dynamic’s quadruped robot, Spot. They gave the robot a list of tasks to complete, and as the robot explored and mapped the inside of an office building, Clio ran in real-time on an on-board computer mounted to Spot, to pick out segments in the mapped scenes that visually relate to the given task. The method generated an overlaying map showing just the target objects, which the robot then used to approach the identified objects and physically complete the task.“Running Clio in real-time was a big accomplishment for the team,” Maggio says. “A lot of prior work can take several hours to run.”Going forward, the team plans to adapt Clio to be able to handle higher-level tasks and build upon recent advances in photorealistic visual scene representations.“We’re still giving Clio tasks that are somewhat specific, like ‘find deck of cards,’” Maggio says. “For search and rescue, you need to give it more high-level tasks, like ‘find survivors,’ or ‘get power back on.’ So, we want to get to a more human-level understanding of how to accomplish more complex tasks.”This research was supported, in part, by the U.S. National Science Foundation, the Swiss National Science Foundation, MIT Lincoln Laboratory, the U.S. Office of Naval Research, and the U.S. Army Research Lab Distributed and Collaborative Intelligent Systems and Technology Collaborative Research Alliance.

MIT’s Clio runs in real-time to map task-relevant objects in a robot’s surroundings, allowing the bot (Boston Dynamic’s quadruped robot Spot, pictured) carry out a natural language task (“pick up orange backpack”). Credit: Courtesy of the researchers

A new method called Clio enables robots to quickly map a scene and identify the items they need to complete a given set of tasks.

Helping robots zero in on the objects that matter

<h3 id="isPasted">4 Reasons to Create a Smart Tank Monitoring and Pump Control System</h3>#1. A smart solution allows operators to remotely manage tank levels and pump operations from anywhere, preventing common issues such as tank overflows, pump dry-running, and treatment process disruptions. The ability to monitor systems in real-time reduces the risk of costly emergencies by enabling early detection of potential problems, providing timely data for decision-making.#2.&nbsp;In terms of maintenance and usage, access to real-time data allows operators to optimize system performance. Data on pump operations and operating hours can be leveraged to support predictive maintenance, reducing downtime and extending the lifespan of equipment. This also ensures consistent water quality by making informed decisions based on continuous monitoring.#3.&nbsp;When based on versatile products like Arduino’s, inspired by open-source principles, a smart tank monitoring system is highly adaptable and scalable, allowing it to fit various industrial needs. It can be integrated with existing equipment and third-party systems, ensuring seamless incorporation into a facility’s operations. This flexibility makes it ideal for a range of industrial applications, from small-scale plants to complex systems.#4.&nbsp;By automating the monitoring process, the solution minimizes the need for frequent manual checks, reducing human error and cutting operational costs. Automated alerts and precise monitoring help maintain optimal chemical dosing and flow rates, while reducing any costs associated with equipment damage, process disruptions, and regulatory non-compliance.<h3>Technical Overview of the Solution</h3>A flexible and cost-effective solution for tank monitoring and pump control can be based on an easy-to-use micro PLC such as the <a href="https://www.arduino.cc/pro/hardware-arduino-opta/" target="_blank">Arduino Opta</a>. The <a href="https://docs.arduino.cc/tutorials/opta/tank-level-app-note/" target="_blank">complete tutorial is available on Arduino Docs</a>.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI2ODQ0ODQzNTk5LWVsZWN0cmljYWxfY29ubmVjdGlvbnMucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">In this case, the system employs a variety of sensors – such as float switches, ultrasonic transducers, and pressure transducers – to continuously track liquid levels. Data from flow meters is used to measure pump discharge rates and overall system flow. This information is then transmitted to a cloud-based SCADA&nbsp;(Supervisory Control and Data Acquisition) system for real-time access, allowing operators to visualize and manage tank levels, pump status, and system flow rates through an intuitive dashboard.Additionally, the system includes configurable alarms that send alerts for critical scenarios such as abnormal flow rates or high/low tank levels. These alarms can be customized and sent via multiple channels, including email and SMS, preventing unnecessary notifications and ensuring only critical alerts are triggered.The solution is compatible with a variety of sensors and supports industry-standard protocols, such as Modbus RTU over RS485 and Modbus TCP/IP.&nbsp;<h3>Adding Vision-Based Liquid Monitoring</h3>In situations where conventional sensors are not practical, a vision-based monitoring system can be deployed. A module such as the Arduino <a href="https://www.arduino.cc/pro/hardware-product-nicla-vision/" target="_blank">Nicla Vision</a>, which is equipped with a 2 MP color camera, has the intelligence to&nbsp;visually assess the current liquid level in a tank: applying so-called “blob detection”, the liquid’s surface can be identified and the tank’s fill level calculated. This data can be relayed to the control system to trigger appropriate actions, such as opening or closing valves.This vision-based approach offers a non-invasive method to monitor tank levels, making it suitable for systems where physical sensor installation is difficult or impossible. It can also be retrofitted to almost any existing system without significant modifications.<h3>Unlocking Smart Industrial Solutions with the Arduino Ecosystem</h3>The flexibility and adaptability of the Arduino platform make it an excellent choice for creating smart industrial systems, such as tank level monitoring and pump control solutions. Whether you're integrating traditional sensors or implementing innovative vision-based methods, Arduino’s open-source tools and broad compatibility with industry standards offer a versatile foundation for any project. Explore the Arduino ecosystem to find the right tools for your next industrial challenge, and see how it can accelerate your path to smarter, more efficient operations.

The need for advanced tank monitoring and pump control solutions is growing across industries – from water treatment to aquaculture to industrial utilities. Meanwhile, smart technology is transforming tank management for a variety of liquids, ensuring a streamlined process and minimizing risks.

Build Your Own Smart Industrial Tank Level Monitoring and Pump Control Solution

By achieving a means of driving machinery automatically, the Industrial Revolution changed the whole nature of production.It occurred in Great Britain between 1760 and 1830, then spread through Europe and the United States. This is now the First Industrial Revolution, or Industry 1.0. As far as the actual words go, the term "industrial revolution" was first used in 1799 and was taken up in the early 1800s.The second significant advance Industry 2 started around 1850, with electricity and large-scale industrial processing. It lasted until the second half of the 20th century.&nbsp;<h2 dir="ltr">Computer Age</h2>The third industrial revolution transformed our lives once we got into computers and electronic communications. We began using computers for everything, including the automation of production and assembly lines, although human involvement was still generally required for control. Information Technology (IT) became the next big thing. This step was retroactively called Industry 3.0. It saw a massive upswing in the industrial use of computers as human tasks were phased into automated ones. Programmable Logic Controllers (PLCs), IT and human-machine interfaces (HMIs) replaced humans in governing computerised processes.&nbsp;Since 2011, Industry 4.0 has encompassed many of the advanced systems we're familiar with. In Industry 4.0, most automated systems follow pre-set programming parameters. They can also streamline production processes by analysing and acting on collected data.The&nbsp;<a title="What Is IIoT?" href="https://www.rowse.co.uk/blog/post/what-is-iiot" target="_blank">industrial Internet of Things</a> (IoT), cloud computing, and cyber-physical systems (CPS) impact today's manufacturing processes. These technologies have hugely enhanced automation and global connectivity. The sequence of increased automation has also led us to AI and robots.<h2 dir="ltr">Industry 3.0 -&gt; 4.0</h2>The developments in industrial technology offer&nbsp;<a title="What Are The Benefits Of Industry 4.0?" href="https://www.rowse.co.uk/blog/post/benefits-of-industry-4" target="_blank">great benefits</a>, such as:<ul><li dir="ltr">Decentralised information</li><li dir="ltr">Real-time data collection</li><li dir="ltr">Big data analytics</li><li dir="ltr">AI and ML (machine learning)</li><li dir="ltr">Interoperability</li><li dir="ltr">Heightened flexibility</li></ul><h3 dir="ltr">Decentralised Information</h3>Many elements of the production process now contain sensors that collect a continuous stream of information. In Industry 3.0, each company's local servers usually store and manage all the data from an individual system. Only limited data sharing and integration took place across sectors. Industry 4.0, however, is just the opposite. Its interconnected platforms and systems allow data to be accessed in real time by multiple stakeholders.<h3 dir="ltr">Real-Time Data Collection</h3>Vast quantities of data are being gathered, processed and acted on in real-time. The availability of decentralised information and its immediate analysis has several advantages. It helps to optimise production and management processes, allows different entities to collaborate more effectively, and thus enables better decision-making. Strategies like predictive maintenance considerably reduce downtime.<h3 dir="ltr">Big Data Analytics</h3>Big data has had a significant impact on operational efficiency. We can now analyse and assess many aspects of production and business factors, such as customer feedback and returns. These and many other contributing factors can help to improve decision-making, anticipate future demands and reduce outages. Big data enables businesses the flexibility to adapt to current market demand.<h3 dir="ltr">AI And ML</h3>The sheer volume of intelligent data collected in today's processes is beyond the capacity of humans. The data analysts of Industry 3.0 have been replaced with Industry 4.0's AI and ML algorithms. Not only can these systems manage big data, but they can learn from its analysis how to improve all aspects of automation. This leads to greater productivity, better quality and overall efficiency. Industry 4.0 integrates automation with intelligence, allowing machines and systems to become more intelligent, autonomous, and adaptive.&nbsp;<h3 dir="ltr">Interoperability</h3>Industry 3.0 made significant advances in manufacturing technology, but its main focus was automating existing processes. It was also confined to individual businesses, each with its systems. Industry 4.0 is about collaboration and connectivity between stakeholders in a business ecosystem. These may include manufacturers, their customers, suppliers, and even the machines themselves. The intention is to promote a "collective intelligence" amongst entities who share their knowledge and insights to drive improvement and innovation. Thus, different value chain components collaborate in an ecosystem with complete transparency, seamless communication and interoperability.<h3 dir="ltr">Heightened Flexibility</h3>Industry 3.0 adopted the concept of lean production, saving money by automating and streamlining processes. Industry 4.0 is geared instead towards brilliant production, where decisions are made based on data rather than experience. Automated systems learn from the amassed data, optimising operations and making predictions without human intervention. New revenue streams are being created differently. This links systems to the product instead of the process. They are making production processes more flexible and responsive. This allows them to adapt better and faster to market conditions and changing demands.<h2 dir="ltr">So, How Does Industry 4.0 Differ From Industry 3.0?</h2>Industry 3.0 achieved limited technological advances in manufacturing, with computers, automation and PLCs. Industry 4.0 leverages far more advanced technologies through the IIoT, cloud computing,&nbsp;<a title="The Future Of Manufacturing" href="https://www.rowse.co.uk/blog/post/the-future-of-manufacturing" target="_blank">augmented reality</a>, and robotics. Add to these AI, ML and big data analytics, and you have real-time connectivity, autonomous systems, predictive maintenance and data-driven decision-making. In addition, the physical and virtual worlds can be merged to enhance many design and manufacturing processes.Industry 3.0 was all about automation in individual factories. It aimed to save money by improving efficiency and productivity. It optimised production lines by substituting machines and computers for human labour. Industry 4.0 has a much broader scope, integrating digital technologies into the value chain. Interconnected CPS can even manage distribution and customer interaction. This makes the digital transformation of the industry far more comprehensive. We're in a major shake-up of how things work not just processes but business models and value creation. As Industry 4.0 gathers momentum, we need new skills for the latest technologies. At this stage, we need digital capabilities to be integrated into every organisational aspect of a business.

To understand Industry 4.0 and Industry 3.0, we must go back to Industry 1.0, although it wasn't called that then. We learned about the Industrial Revolution in school and there was just one. 

Whats The Difference Between Industry 4.0 and Industry 3.0?

An autonomous robot – a base on wheels fitted with computer vision technology that snaps images of grapevines in near real-time – rolls down a long row of plants in a vineyard at Cornell AgriTech in Geneva, New York.The <a href="https://news.cornell.edu/stories/2022/11/autonomous-robots-help-modernize-grape-wine-industry" target="_blank">Cornell-developed “PhytoPatholoBot”</a> detects spectral signatures of three kinds of devastating grape diseases – powdery mildew, downy mildew and grapevine leafroll virus – leading to early management of pathogens without needing farmworkers to spot them. Ultimately, data collected by such robots will help to ground-truth NASA satellite data that could one day detect plant diseases from space.This was one of many research projects presented at a technology demonstration on Aug. 20, part of a three-day “Space for Ag Tour” by NASA leaders to better understand the remote sensing needs of Cornell AgriTech researchers, specialty crop growers and local stakeholders. NASA leadership visited Cornell’s main campus, Cornell AgriTech, sites in Niagara county and the Cornell Lake Erie Research and Extension Lab in Portland, New York.The tour was designed to foster two-way dialogue with growers and stakeholders to improve NASA Earth Science’s Earth-observing satellites and instruments for specialty crop agriculture and viticulture and foster a broader conversation about how to effectively translate research into practice.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1724648875297-0822_nasa1_0.jpg" style="width: 100%px;" class="fr-fic fr-dib">Yu Jiang poses with the PhytoPatholoBotas and UV light robot as part of the “Space for Ag Tour” at the Cornell Agritech campus. Ryan Young/Cornell University“My hope is that by having this Space for Ag Tour here, by showing them New York viticulture as well as what we have to offer here at Cornell, it can help grow the specialty crop footprint within NASA as well as their footprint in pest and disease research,” said&nbsp;<a href="https://cals.cornell.edu/kaitlin-gold" target="_blank">Katie Gold</a>, the event’s main organizer and an assistant professor and the Susan Ekert Lynch Faculty Fellow in the Plant Pathology and Plant-Microbe Biology Section of the School of Integrative Plant Science (SIPS) at Cornell AgriTech, which is in the College of Agriculture and Life Sciences.&nbsp;“I think there’s amazing potential for the satellite data that’s being collected both now and in the future with the eventual launch of the Surface Biology and Geology satellite to be applied to specialty crops,” said Gold, who is an expert on ground-based and remote plant disease sensing and applied grape pathology. The meetings were intended to raise awareness among NASA leaders and scientists of the unique challenges facing specialty crops, including pests and diseases, so they can make those connections in their own research and applications, she said. &nbsp;“For NASA leadership, it’s an opportunity as they think long-term about observational needs and priorities that helps them make the case for the observations that are necessary, from a scientific perspective, to make an impact on the different concerns that farmers have,” said Alyssa Whitcraft, a research professor in the Department of Geographical Sciences at the University of Maryland, whose work focuses on satellite remote sensing of agriculture. Whitcraft is the founding executive director of NASA Acres and co-founding deputy director of NASA Harvest, respectively the U.S. and international NASA consortia on food security and agriculture.&nbsp;Gold is the institutional lead for&nbsp;<a href="https://cals.cornell.edu/news/2023/05/bringing-nasa-technology-down-earth-agriculture" target="_blank">NASA Acres at Cornell</a>.As part of the demonstrations,&nbsp;<a href="https://cals.cornell.edu/yun-yang" target="_blank">Yun Yang</a>, assistant professor in SIPS’ Soil and Crop Sciences Section at Cornell AgriTech, described her NASA Acres-funded research to monitor plant water use using remote sensing, and understanding the impacts of agricultural practices on water resource sustainability.One of Yang’s research projects seeks to estimate water loss to evaporation and transpiration in California vineyards. Vineyard managers want to know how much and when to irrigate in spring based on groundwater depletion, which is a global issue, especially in California.&nbsp;Her team built an instrument to better monitor water fluxes on the ground, which provides validation for NASA Landsat satellite data that can be used to estimate water loss to evapotranspiration when farmers irrigate. By knowing how much water is used, based on a simple water balance, they can tell how much water is needed. Yang’s team has been developing a dashboard for users. “With this dashboard, we can really hand the technology we have to the growers,” Yang said. “And when they are in the field, we can easily check the information and help them manage irrigation.” &nbsp; &nbsp;Applied roboticist <a href="https://cals.cornell.edu/yu-jiang" target="_blank">Yu Jiang</a>, assistant professor in SIPS’ Horticulture Section at Cornell AgriTech, demonstrated the PhytoPatholoBot that he and doctoral student Ertai Liu developed with Gold with funding from NASA Acres and U.S. Department of Agriculture National Institute of Food and Agriculture VitisGen3.&nbsp;The PhytoPatholoBot eliminates background noise from cloud cover or sunlight to provide a constant, high-quality image and has autonomous navigation using GPS. The bot provides a readout per row of vines with red areas denoting higher infection and white areas showing lower infection, and coupled with AI, allows for near real-time inferences of disease that growers may use to focus their treatment efforts on plants that need it most.“We can scan and scale all the field, we are also generating the results that are ready to be realized by the growers, and hopefully, while working together to integrate this data stream back to NASA to see if we can train remote sensing models and make a bigger impact for small grower stakeholders,” Jiang said. Additionally, researchers can use data from the robots to train NASA airborne sensing better than humans, Gold said.&nbsp;The hope is to use robots to automate data collection, to make it possible to gather measurements for millions of acres.&nbsp;Following Jiang,&nbsp;<a href="https://cals.cornell.edu/sara-emery" target="_blank">Sarah Emery</a>, assistant professor of entomology at Cornell AgriTech, presented her work to develop risk maps for spotted lanternfly infestations, a growing threat to grape and other growers in New York.&nbsp;“We’ve been trying to understand how this insect pest is going to be moving across the landscape and develop a model to assess the risk to individual vineyards across New York state,” Emery said.&nbsp;In collaboration with the New York State Department of Agriculture and Markets, the model being developed will overlay observations of spotted lanternfly (90,000 sightings have been recorded across the state) with maps of vineyards and locations of its preferred host, tree of heaven. Eventually, they plan to create a public-facing system so that growers can input their information and get back a level of risk.&nbsp;The hope is to use hyperspectral data (which allows for the identification of objects and materials by analyzing their unique spectral signatures) from satellites to identify light reflectance patterns of non-crop plants across the landscape to create risk maps at finer and wider scales. Emery has teamed with Gold and doctoral student Jaclyn Eller for the project.&nbsp;Also,&nbsp;<a href="https://cals.cornell.edu/jason-londo" target="_blank">Jason Londo</a>,&nbsp;associate professor of fruit crop physiology and climate adaptation in SIPS Horticulture Section at Cornell AgriTech, described using hyperspectral and multispectral approaches to understanding plant stress in grapes and apples using drones. He hopes to one day use similar data from satellites, but needs better resolution to be able to detect stress on a tree-by-tree basis, he said.Other visiting NASA personnel included Karen St. Germain, division director of the Earth Science Division in the Science Mission Directorate at NASA.&nbsp;<a href="https://cals.cornell.edu/steven-wolf" target="_blank">Steven Wolf</a>, associate professor of natural resources and the environment in CALS, organized events at Cornell’s Ithaca campus.

Members of Yu Jiang’s team demonstrate the technology. Credits: Ryan Young/Cornell University

An autonomous robot – a base on wheels fitted with computer vision technology that snaps images of grapevines in near real-time – rolls down a long row of plants in a vineyard at Cornell AgriTech in Geneva, New York.

From space to farm: readying NASA satellites to help growers

As artificial intelligence models become increasingly prevalent and are integrated into diverse sectors like health care, finance, education, transportation, and entertainment, understanding how they work under the hood is critical. Interpreting the mechanisms underlying AI models enables us to audit them for safety and biases, with the potential to deepen our understanding of the science behind intelligence itself.Imagine if we could directly investigate the human brain by manipulating each of its individual neurons to examine their roles in perceiving a particular object. While such an experiment would be prohibitively invasive in the human brain, it is more feasible in another type of neural network: one that is artificial. However, somewhat similar to the human brain, artificial models containing millions of neurons are too large and complex to study by hand, making interpretability at scale a very challenging task.&nbsp;To address this, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers decided to take an automated approach to interpreting artificial vision models that evaluate different properties of images. They developed “MAIA” (Multimodal Automated Interpretability Agent), a system that automates a variety of neural network interpretability tasks using a vision-language model backbone equipped with tools for experimenting on other AI systems.“Our goal is to create an AI researcher that can conduct interpretability experiments autonomously. Existing automated interpretability methods merely label or visualize data in a one-shot process. On the other hand, MAIA can generate hypotheses, design experiments to test them, and refine its understanding through iterative analysis,” says Tamar Rott Shaham, an MIT electrical engineering and computer science (EECS) postdoc at CSAIL and co-author on a new&nbsp;<a href="https://arxiv.org/pdf/2404.14394.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">paper about the research</a>. “By combining a pre-trained vision-language model with a library of interpretability tools, our multimodal method can respond to user queries by composing and running targeted experiments on specific models, continuously refining its approach until it can provide a comprehensive answer.”The automated agent is demonstrated to tackle three key tasks: It labels individual components inside vision models and describes the visual concepts that activate them, it cleans up image classifiers by removing irrelevant features to make them more robust to new situations, and it hunts for hidden biases in AI systems to help uncover potential fairness issues in their outputs. “But a key advantage of a system like MAIA is its flexibility,” says Sarah Schwettmann PhD ’21, a research scientist at CSAIL and co-lead of the research. “We demonstrated MAIA’s usefulness on a few specific tasks, but given that the system is built from a foundation model with broad reasoning capabilities, it can answer many different types of interpretability queries from users, and design experiments on the fly to investigate them.”&nbsp;<h2>Neuron by neuron</h2>In one example task, a human user asks MAIA to describe the concepts that a particular neuron inside a vision model is responsible for detecting. To investigate this question, MAIA first uses a tool that retrieves “dataset exemplars” from the ImageNet dataset, which maximally activate the neuron. For this example neuron, those images show people in formal attire, and closeups of their chins and necks. MAIA makes various hypotheses for what drives the neuron’s activity:&nbsp;facial expressions, chins, or neckties. MAIA then uses its tools to design experiments to test each hypothesis individually by generating and editing synthetic images — in one experiment, adding a bow tie to an image of a human face increases the neuron’s response. “This approach allows us to determine the specific cause of the neuron’s activity, much like a real scientific experiment,” says Rott Shaham.MAIA’s explanations of neuron behaviors are evaluated in two key ways. First, synthetic systems with known ground-truth behaviors are used to assess the accuracy of MAIA’s interpretations. Second, for “real” neurons inside trained AI systems with no ground-truth descriptions, the authors design a new automated evaluation protocol that measures how well MAIA’s descriptions predict neuron behavior on unseen data.The CSAIL-led method outperformed baseline methods describing individual neurons in a variety of vision models such as ResNet, CLIP, and the vision transformer DINO. MAIA also performed well on the new dataset of synthetic neurons with known ground-truth descriptions. For both the real and synthetic systems, the descriptions were often on par with descriptions written by human experts.How are descriptions of AI system components, like individual neurons, useful? “Understanding and localizing behaviors inside large AI systems is a key part of auditing these systems for safety before they’re deployed — in some of our experiments, we show how MAIA can be used to find neurons with unwanted behaviors and remove these behaviors from a model,” says Schwettmann. “We’re building toward a more resilient AI ecosystem where tools for understanding and monitoring AI systems keep pace with system scaling, enabling us to investigate and hopefully understand unforeseen challenges introduced by new models.”<h2>Peeking inside neural networks</h2>The nascent field of interpretability is maturing into a distinct research area alongside the rise of “black box” machine learning models. How can researchers crack open these models and understand how they work? Current methods for peeking inside tend to be limited either in scale or in the precision of the explanations they can produce. Moreover, existing methods tend to fit a particular model and a specific task. This caused the researchers to ask: How can we build a generic system to help users answer interpretability questions about AI models while combining the flexibility of human experimentation with the scalability of automated techniques?One critical area they wanted this system to address was bias. To determine whether image classifiers displayed bias against particular subcategories of images, the team looked at the final layer of the classification stream (in a system designed to sort or label items, much like a machine that identifies whether a photo is of a dog, cat, or bird) and the probability scores of input images (confidence levels that the machine assigns to its guesses). To understand potential biases in image classification, MAIA was asked to find a subset of images in specific classes (for example “labrador retriever”) that were likely to be incorrectly labeled by the system. In this example, MAIA found that images of black labradors were likely to be misclassified, suggesting a bias in the model toward yellow-furred retrievers.Since MAIA relies on external tools to design experiments, its performance is limited by the quality of those tools. But, as the quality of tools like image synthesis models improve, so will MAIA. MAIA also shows confirmation bias at times, where it sometimes incorrectly confirms its initial hypothesis. To mitigate this, the researchers built an image-to-text tool, which uses a different instance of the language model to summarize experimental results. Another failure mode is overfitting to a particular experiment, where the model sometimes makes premature conclusions based on minimal evidence.“I think a natural next step for our lab is to move beyond artificial systems and apply similar experiments to human perception,” says Rott Shaham. “Testing this has traditionally required manually designing and testing stimuli, which is labor-intensive. With our agent, we can scale up this process, designing and testing numerous stimuli simultaneously. This might also allow us to compare human visual perception with artificial systems.”“Understanding neural networks is difficult for humans because they have hundreds of thousands of neurons, each with complex behavior patterns. MAIA helps to bridge this by developing AI agents that can automatically analyze these neurons and report distilled findings back to humans in a digestible way,” says Jacob Steinhardt, assistant professor at the University of California at Berkeley, who wasn’t involved in the research. “Scaling these methods up could be one of the most important routes to understanding and safely overseeing AI systems.”Rott Shaham and Schwettmann are joined by five fellow CSAIL affiliates on the paper: undergraduate student Franklin Wang; incoming MIT student Achyuta Rajaram; EECS PhD student Evan Hernandez SM ’22; and EECS professors Jacob Andreas and Antonio Torralba. Their work was supported, in part, by the MIT-IBM Watson AI Lab, Open Philanthropy, Hyundai Motor Co., the Army Research Laboratory, Intel, the National Science Foundation, the Zuckerman STEM Leadership Program, and the Viterbi Fellowship.

The automated, multimodal approach developed by MIT researchers interprets artificial vision models that evaluate the properties of images. Image: iStock

MAIA is a multimodal agent that can iteratively design experiments to better understand various components of AI systems.

Researchers advance automated interpretability in AI models

In a broad sense, "machine vision" refers to the utilization of visual information and optical information using digital technologies. However, because it is being introduced to production sites in the manufacturing industry, machine vision generally refers to digital image technologies utilized in automatic inspection and process automation in factories. Machine vision consists of the following:・Imaging section (cameras) ・Interface section (board/general-purpose interface) ・Processing engine section (computer) ・Processing control section (software)etc. The image data obtained by the imaging section is loaded into the processing engine via the interface section. In the processing engine section, the processing control section performs binarization*1, edge detection*2, and other forms of image processing. Next, it outputs the image data that has completed image processing to robots and industrial equipment.*1: Processing that converts the pixel values (threshold values) that are preset for the image to the two values of black and white. *2: Processing that detects the bright and dark areas in an image.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327121894-img-0001-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 1: Machine vision configurationMachine vision is used in automatic inspection, automatic recognition, and automatic measurement, etc. For example, automatic inspection and automatic recognition are very fast compared to manual inspection. Moreover, machine vision used in automatic measurement can accurately identify small value differences that humans cannot measure. In addition, linking machine vision to AI functions is expected to produce more intelligent uses going forward.<h2>Differences Between Machine Vision, Computer Vision, and Robot Vision</h2>Computer vision and robot vision are technologies that are similar to machine vision, but they each play different roles. Here, we explain how their roles differ.<h3>Differences Between Machine Vision and Computer Vision</h3>"Computer vision" is a technology that adds equipment control functions to machine vision functions. Computer vision can control various types of equipment based on data output by machine vision. For example, this would make it possible to combine the image data output by machine vision with CAD data generated by the computer to control processors or to adjust the conveyor transport speed based on image data that captured the progress of line work. In addition, computer vision has a broad scope of utilization that naturally includes smart factories as well as referee assistance systems and virtual replays in sports and projection mapping, etc. in entertainment.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327141371-img-0002-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 2: Example of computer vision utilization<h3 id="isPasted">Differences Between Machine Vision and Robot Vision</h3>"Robot vision" is a technology that adds machine vision to robot visual functions. Machine vision attaches cameras to various types of equipment to obtain image data, but robot vision attaches cameras to the end of robot arms to obtain image data. This enables the robot to operate based on visual information obtained by itself. For example, a robot that does not use robot vision can only execute certain operations programmed in advance. However, when a robot equipped with robot vision moves its robot arm closer to an object, it then sequentially identifies the target based on image data from the camera to execute operations. Because this type of operational control is possible, a robot equipped with robot vision is capable of detailed, precise, and flexible operations that cannot be specified with a program.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327158790-img-0003-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 3: Example of robot vision utilization<h2 id="isPasted">Example of Machine Vision Utilization in a Smart Factory</h2>Machine vision is used in various types of work in a smart factory. Here, we introduce an example of machine vision utilization.<h3>Reading Characters at Ultrahigh-speed – OCR (Optical Character Recognition/Reader)</h3>OCR is a technology that scans handwritten or printed characters with an image scanner or digital camera. Machine vision scans the 2D code, serial number, and date of manufacture, etc. printed on products. The scanned data is utilized in inspections of the print quality or whether the correct characters are printed as well as the screening of the next process, etc. In many cases, the products being scanned by OCR pass through the line at high speeds. Therefore, machine vision uses cameras that can capture images at high speed and image processing systems that are capable of high-speed processing.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327179168-img-0004+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 4: OCR on a manufacturing line<h3 id="isPasted">Scanning Shapes in 2D and 3D – Visual Inspections</h3>The utilization of machine vision is considered to be the most effective means of visual inspection on a manufacturing line. Because machine vision-based visual inspection is able to inspect the accuracy of products after cutting and molding and the mounting condition of components, etc. with high speed and accuracy, it is essential for improving production efficiency and maintaining quality. The surfaces of target objects can vary widely in terms of their glossiness and light absorption, etc. The lighting used in machine vision enables products with various surface conditions to be inspected by changing the method of illumination and the color of the light rays. In addition, 3D cameras equipped with a stereophonic*3, time-of-flight*4, and other measurement systems as well as structured light*5 and other imaging systems can also measure the depth.*3: Method that uses two cameras to perform triangulation and measure the distance to an object. *4: Method that measures the distance to an object according to the time it takes for light emitted by a light source to reflect off of the object and return to a light detector. *5: Lighting method that emits light with a specific shading pattern.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327197939-img-0005.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 5: Visual inspection with a 3D camera<h3 id="isPasted">Grabbing Specific Components – Picking</h3>"Picking" on a manufacturing line refers to the task of collecting (pickup) the necessary components and products. Utilizing machine vision makes it possible to identify and pick components with the specified color and shape even if components and products with various colors and shapes are mixed together in one case.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327216910-img-0006.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 6: Picking by a robot<h3 id="isPasted">Providing Visual Functions to Robots – Robot Operational Control</h3>In recent years, industrial robots have also been used in precision work due to their high operating precision. For example, they play an active role in installing, soldering, and welding microcomponents. However, controlling such work with programming requires high-precision positioning devices. With robot vision utilizing machine vision, high-precision positioning devices are not required for the robot to perform precise positioning with the image data from the camera mounted on the end of the robot arm.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327233332-img-0007.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 7: Installing microcomponents with a robot<h2 id="isPasted">Future of Machine Vision – AI Vision</h2>As various types of industrial equipment shift to using AI, one form of machine vision evolution that shows promise is "AI vision."<h3>What is AI Vision?</h3>AI vision is a system that equips lightweight AI chips in smartphones, etc. with cameras and camera functions to make inferences about captured image data. AI vision resembles computer vision in that it uses AI (artificial intelligence), but computer vision mainly makes inferences with a cloud AI service (Figure 8). On the other hand, AI vision equips an AI module in an edge device to make inferences, and the inference results are stored on the edge server. The edge server communicates with the cloud AI only when necessary. (Figure 9).<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327267580-1721327267580.png" class="fr-fic fr-dib">Therefore, AI vision has the following advantages over computer vision: ・Able to decrease the volume of communication with the cloud AI service ・Able to prevent external leaks of data collected by the sensors ・Superior responsiveness Furthermore, the cameras and smartphones used in AI vision are called "edge AI devices" while the method that makes inferences on the edge device is called "edge AI."<h3>What Technologies Are Required to Realize AI Vision?</h3>Although AI vision has the advantages described above, AI modules equipped on edge devices require the following types of functions. ・Superior energy conservation due to restrictions on battery capacity ・Ability to perform high-speed processing due to the responsiveness requirement ・Superior heat dissipation due to installation in a limited space More than anything, it is important that the AI modules be compact.<h2>Summary</h2>This article explained the topic of machine vision and the similar phrases of computer vision and robot vision as well as AI vision. Going forward, it is expected that machine vision and AI will evolve in a close relationship with the evolution in deep learning for image data. In addition, new AI systems and products are expected to be developed through the proper use of cloud AI and edge AI. These trends will likely become essential technologies for smart factory robots, automatic inspection, automatic recognition, and automatic measurement equipment that require visual information that is closer to the human eye and decision-making capabilities that exceed that of humans.

In a broad sense, "machine vision" refers to the utilization of visual information and optical information using digital technologies.

The Eyes of Smart Factories - What is Machine Vision?

In this episode, we discuss how 2 Carnegie Mellon University graduate students developed an AI system capable of giving you feedback on your interview performance in real time!

Podcast: Meet the AI That Helps You Ace Job Interviews

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/d97ba401-cff2-4367-adb7-043c01d4a88a?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>When Haoguang Cai and Khem Agarwal decided they wanted to work on a project to present at the annual IEEE Rising Stars conference, they knew they wanted to build something that would be of interest and use to the Institute for Electrical and Electronics Engineers audience of students and young technical professionals.The <a href="https://www.sv.cmu.edu/index.html" rel="noopener" target="_blank">Silicon ValleyOpens in new window</a> graduate students, who care about making AI that benefits people, came up with an idea that achieved both goals. They knew that IEEE Rising Star attendees are focused on career opportunities so they set to work devising a system that could help job seekers improve their performance and become more confident in job interviews.Cai and Agarwal also knew that many of their fellow students practiced interviewing with friends or colleagues. But when mock interview options were exhausted or not available, they still wanted more practice and helpful feedback.The system that the pair developed and later demonstrated at the Las Vegas conference analyzes a user’s facial and speech expression to detect what emotions they are conveying and tracks their gaze or eye motion to detect where they are focusing their attention. Feedback in both real time and after the interview session gives users insight as to how they are likely being perceived during an interview.“When I showed the live demo on stage, everyone wanted to try it. They really saw its potential and lined up after the workshop to try it and learn how we did it,” said Cai.<blockquote>When I showed the live demo on stage, everyone wanted to try it. They really saw its potential and lined up after the workshop to try it and learn how we did it.<cite>Haoguang Cai, MSSM Student, SV - III</cite></blockquote>The project originated in their Software Engineering Management class. Their instructor, <a href="https://engineering.cmu.edu/directory/bios/fang-catherine.html" target="_blank">Catherine Fang</a>, was scheduled to speak at the January 2024 conference. The distinguished service professor is an entrepreneur who has co-founded two companies, holds multiple U.S. patents, has worked for LinkedIn, Yahoo, and Sun Microsystems, and serves as an associate editor for the IEEE reliability society.“I’m so proud of all of our students, but this team really stood out. They have a very solid technical and product management background. Their presentation was a big success,” said Fang.<blockquote>I’m so proud of all of our students, but this team really stood out. They have a very solid technical and product management background. Their presentation was a big success.<cite>Catherine Fang, Distinguished Service Professor, SV - III</cite></blockquote>She added that emotion detection technology is an emerging trend with many useful consumer applications that she has seen demonstrated at other big conferences.Cai and Agarwal, who will both earn their Master of Science in Software Management (MSSM) from the Integrated Innovation Institute in December 2024, knew how to program the machine learning model to identify and measure facial expressions and eye tracking but they needed a data set with a large number of labelled images that represented a diverse set of people including those of varying color, age, and gender.They contacted Hume AI, a research lab and technology company that uses artificial intelligence to measure emotional expression, which was able to provide access to their extensive data set through use of their Application Programming Interface (API). APIs are commonly used in software development to enable integration between different systems or services for applications such as accessing data from a remote server.Jeremy Hadfield, who is a technical product manager at Hume, has supported the pair through the development of their emotion detection system. He was recently a guest at a campus event.Cai, who hopes to complete his transition from a machine learning engineer to product manager in the AI field after graduation, will intern with Hume AI this summer. Agarwal is planning for a career that will enable him to use AI to create virtual reality learning tools.They both agree that the challenge and extra work to build and demonstrate their emotion detection system was a worthwhile endeavor and that it only furthered their desire to use AI to build products that will help people.

Face expression analysis that users can view in real time while answering interview questions indicate the emotion they are conveying.

Silicon Valley students use AI to help job seekers improve performance and build confidence.

Emotion detection system puts a smile on their face

In this episode, we discuss research coming out of the Karlsruhe Institute of Technology to detect the emotions of tennis players better than humans can!

Podcast: Catching Feelings with 69 percent Accuracy: AI Reads Emotions in Sports

Boosting the performance of solar cells, transistors, LEDs, and batteries will require better electronic materials, made from novel compositions that have yet to be discovered.To speed up the search for advanced functional materials, scientists are using AI tools to identify promising materials from hundreds of millions of chemical formulations. In tandem, engineers are building machines that can print hundreds of material samples at a time based on chemical compositions tagged by AI search algorithms.But to date, there’s been no similarly speedy way to confirm that these printed materials actually perform as expected. This last step of material characterization has been a major bottleneck in the pipeline of advanced materials screening.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1719275773623-MIT_Accelerating-Materials-Vickmark-02-PRESS.jpg" style="width: 100%px;" class="fr-fic fr-dib">To speed up the search for advanced functional materials, scientists are using a new computer vision approach to identify new solar cell materials from hundreds of millions of chemical formulations. Credit: Bryce VickmarkNow, a new computer vision technique developed by MIT engineers significantly speeds up the characterization of newly synthesized electronic materials. The technique automatically analyzes images of printed semiconducting samples and quickly estimates two key electronic properties for each sample: band gap (a measure of electron activation energy) and stability (a measure of longevity).The new technique accurately characterizes electronic materials 85 times faster compared to the standard benchmark approach.The researchers intend to use the technique to speed up the search for promising solar cell materials. They also plan to incorporate the technique into a fully automated materials screening system.“Ultimately, we envision fitting this technique into an autonomous lab of the future,” says MIT graduate student Eunice Aissi. “The whole system would allow us to give a computer a materials problem, have it predict potential compounds, and then run 24-7 making and characterizing those predicted materials until it arrives at the desired solution.”“The application space for these techniques ranges from improving solar energy to transparent electronics and transistors,” adds MIT graduate student Alexander (Aleks) Siemenn. “It really spans the full gamut of where semiconductor materials can benefit society.”Aissi and Siemenn detail the new technique in a <a href="https://www.nature.com/articles/s41467-024-48768-2" target="_blank">study appearing today</a> in Nature Communications. Their MIT co-authors include graduate student Fang Sheng, postdoc Basita Das, and professor of mechanical engineering Tonio Buonassisi, along with former visiting professor Hamide Kavak of Cukurova University and visiting postdoc Armi Tiihonen of Aalto University.<h2>Power in optics</h2>Once a new electronic material is synthesized, the characterization of its properties is typically handled by a “domain expert” who examines one sample at a time using a benchtop tool called a UV-Vis, which scans through different colors of light to determine where the semiconductor begins to absorb more strongly. This manual process is precise but also time-consuming: A domain expert typically characterizes about 20 material samples per hour — a snail’s pace compared to some printing tools that can lay down 10,000 different material combinations per hour.“The manual characterization process is very slow,” Buonassisi says. “They give you a high amount of confidence in the measurement, but they’re not matched to the speed at which you can put matter down on a substrate nowadays.”To speed up the characterization process and clear one of the largest bottlenecks in materials screening, Buonassisi and his colleagues looked to computer vision — a field that applies computer algorithms to quickly and automatically analyze optical features in an&nbsp;image.“There’s power in optical characterization methods,” Buonassisi notes. “You can obtain information very quickly. There is richness in images, over many pixels and wavelengths, that a human just can’t process but a computer machine-learning program can.”The team realized that certain electronic properties — namely, band gap and stability — could be estimated based on visual information alone, if that information were captured with enough detail and interpreted correctly.With that goal in mind, the researchers developed two new computer vision algorithms to automatically interpret images of electronic materials: one to estimate band gap and the other to determine stability.The first algorithm is designed to process visual data from highly detailed, hyperspectral images.“Instead of a standard camera image with three channels — red, green, and blue (RBG) — the hyperspectral image has 300 channels,” Siemenn explains. “The algorithm takes that data, transforms it, and computes a band gap. We run that process extremely fast.”The second algorithm analyzes standard RGB images and assesses a material’s stability based on visual changes in the material’s color over time.“We found that color change can be a good proxy for degradation rate in the material system we are studying,” Aissi says.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1719275806756-MIT_Accelerating-Materials-Vickmark-03-PRESS.jpg" style="width: 100%px;" class="fr-fic fr-dib">MIT professor of mechanical engineering Tonio Buonassisi (left) is a co-author on the paper outlining the new technique for identifying advanced materials. Credit: Bryce Vickmark<h2>Material compositions</h2>The team applied the two new algorithms to characterize the band gap and stability for about 70 printed semiconducting samples. They used a robotic printer to deposit samples on a single slide, like cookies on a baking sheet. Each deposit was made with a slightly different combination of semiconducting materials. In this case, the team printed different ratios of perovskites — a type of material that is expected to be a promising solar cell candidate though is also known to quickly degrade.“People are trying to change the composition — add a little bit of this, a little bit of that — to try to make [perovskites] more stable and high-performance,” Buonassisi says.Once they printed 70 different compositions of perovskite samples on a single slide, the team scanned the slide with a hyperspectral camera. Then they applied an algorithm that visually “segments” the image, automatically isolating the samples from the background. They ran the new band gap algorithm on the isolated samples and automatically computed the band gap for every sample. The entire band gap extraction process process took about six minutes.“It would normally take a domain expert several days to manually characterize the same number of samples,” Siemenn says.To test for stability, the team placed the same slide in a chamber in which they varied the environmental conditions, such as humidity, temperature, and light exposure. They used a standard RGB camera to take an image of the samples every 30 seconds over two hours. They then applied the second algorithm to the images of each sample over time to estimate the degree to which each droplet changed color, or degraded under various environmental conditions. In the end, the algorithm produced a “stability index,” or a measure of each sample’s durability.&nbsp;As a check, the team compared their results with manual measurements of the same droplets, taken by a domain expert. Compared to the expert’s benchmark estimates, the team’s band gap and stability results were 98.5 percent and 96.9 percent as accurate, respectively, and 85 times faster.“We were constantly shocked by how these algorithms were able to not just increase the speed of characterization, but also to get accurate results,” Siemenn says. &nbsp;“We do envision this slotting into the current automated materials pipeline we’re developing in the lab, so we can run it in a fully automated fashion, using machine learning to guide where we want to discover these new materials, printing them, and then actually characterizing them, all with very fast processing.”This work was supported, in part, by First Solar.&nbsp;

MIT graduate students Eunice Aissi, left, and Alexander Siemenn, have developed a technique that automatically analyzes visual features in printed samples (pictured) to quickly determine key properties of new and promising semiconducting materials. Credit: Bryce Vickmark

computer vision

Profiles