Rise of Engineering GPTs

Various industries are developing their own large language models for specific engineering applications. While AI has been used for medical systems for several years, and ChatGPT has been used for chip design, connection with Wolfram analysis and even robot control, LLMs and GPTs are now being used for telecoms, optoelectronics, and even power grid designs.

TelecomGPT, OptoGPT, and ‘GridGPT’ have all been developed using LLMs to help engineers. OptoGPT, developed by University of Michigan engineers, uses the computer architecture underpinning ChatGPT to work backward from desired optical properties to the material structure that can provide them. The new algorithm designs optical multilayer film structures—stacked thin layers of different materials—that can serve a variety of purposes.

OptoGPT produces designs for multilayer film structures within 0.1 seconds, almost instantaneously. In addition, OptoGPT's designs contain six fewer layers on average compared to previous models, making them easier to manufacture. "Designing these structures usually requires extensive training and expertise as identifying the best combination of materials, and the thickness of each layer, is not an easy task," said L. Jay Guo, professor of electrical and computer engineering at the University of Michigan.

To automate the design process for optical structures, the research team tailored a transformer architecture to treat materials at a certain thickness as words, also encoding their associated optical properties as inputs. Seeking out correlations between these "words," the model predicts the next word to create a "phrase"—in this case, a design for an optical multilayer film structure that achieves the desired property such as high reflection.

SoundHound adds ChatGPT to European vehicles. Researchers tested the new model's performance using a validation dataset containing 1,000 known design structures including their material composition, thickness, and optical properties. When comparing OptoGPT's designs to the validation set, the difference between the two was only 2.58%, lower than the closest optical properties in the training dataset at 2.96%.

Taking analysis a step further, the researchers used a statistical technique to map out associations that OptoGPT makes. When mapped in a 2D space, materials cluster by type such as metals and dielectric materials. All dielectrics, including semiconductors, converge upon a central point as the thickness approaches 10 nanometers. Known as an inverse design algorithm, OptoGPT offers more flexibility than previous approaches, enabling researchers and engineers to design optical multilayer film structures for a wide breadth of applications.

A study by Na Li, Professor of Electrical Engineering and Applied Mathematics at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) suggests that LLMs could play an important role in co-managing some aspects of the grid, including emergency and outage response, crew assignments, and wildfire preparedness and prevention. However, security and safety concerns need to be addressed before LLMs and GPTs can be deployed in the field.

“There is so much hype with large-language models, it’s important for us to ask what LLMs can do well and, perhaps more importantly, what they can’t do well, at least not yet, in the power sector,” said Le Xie, Professor of Electrical & Computer Engineering at Texas A&M University. “The best way to describe the potential of LLMs in this sector is as a co-pilot. It’s not a pilot yet — but it can provide advice, a second opinion, and very timely responses with very few training data samples, which is really beneficial to human decision making.”

The research team, which included engineers from Houston-based energy-provider CenterPoint Energy and grid-operator Midcontinent Independent System Operator, used GPT models to explore the capabilities of LLMs in the energy sector and identified both strengths and weaknesses. Challenges to implementing LLMs in the energy sector include the lack of grid-specific data to train the models and the need for safety guardrails.

Another issue is the lack of safety guardrails. The power grid, like autonomous vehicles, needs to prioritize safety and incorporate large safety margin when making real-time decisions. LLMs and GPTs also need to get better about providing reliable solutions and transparency around their uncertainties, said Li. “We want foundational LLMs to be able to say ‘I don’t know’ or ‘I only have 50% certainty about this response’, rather than give us an answer that might be wrong,” said Li.

“As engineers, we want to highlight these limitations because we want to see how we can improve them,” said Li. “Power system engineers can help improve security and safety guarantees by either fine-tuning the foundational LLM or developing our own foundational model for the power systems. One exciting part of this research is that it is a snapshot in time. Next year or even sooner, we can go back and revisit all these challenges and see if there has been any improvement.”

A framework for building LLMs for 6G telecoms systems has been developed at Khalifa University in Abu Dhabi. Current mainstream LLMs generally lack the specialized knowledge in the telecom domain. Researchers have proposed a pipeline to adapt any general-purpose LLMs to a telecom-specific LLMs by collecting telecom-specific pretrain dataset, instruction dataset, preference dataset to perform continual pre-training, instruct tuning, and alignment tuning respectively.

They developed three benchmarks, Telecom Math Modeling, Telecom Open QnA, and Telecom Code Tasks. These provide an evaluation of the capabilities of LLMs and GPTs including math modeling, Open-Ended question answering, code generation, infilling, summarization, and analysis in the telecom domain. The resulting fine-tuned LLM TelecomGPT outperforms state-of-the-art LLMs in Telecom Math Modeling benchmark significantly and achieves comparable performance in various evaluation benchmarks.