The Need for Situational Awareness in IoT Applications

Many IoT and other streaming-style applications have evolved beyond simple dashboarding to become situationally aware. They can analyze the streams of IoT data and detect situations of interest which can then be presented by the system to a user in real-time. The next significant step for these applications is to integrate situational information with Generative AI to further enhance the productivity and situational awareness of knowledge workers, such as technicians and clinicians.

In the evolving landscape of IoT (Internet of Things) applications across diverse sectors such as agriculture, manufacturing, urban management, and healthcare, the challenge of effectively utilizing situational awareness is paramount. These applications rely on real-time data and context-aware information for critical decision-making and efficient response to dynamic conditions. However, a significant gap exists in harnessing this data effectively due to the overwhelming volume and complexity of information, the need for extensive experience and institutional knowledge to interpret it, and limitations in current technological tools.

Situational awareness in IoT (Internet of Things) applications involves leveraging real-time data and context-aware information to help users make informed decisions and take appropriate actions in real-time, in response to critical conditions or circumstances. These applications play a crucial role in various industries and scenarios where timely and accurate information is vital for optimizing processes, enhancing safety, and improving efficiency.

Here are some examples of situationally aware IoT applications:

• Smart Agriculture

Situationally aware IoT applications in agriculture utilize sensors, drones, and weather data to monitor soil conditions, crop health, and weather patterns. This enables farmers to make data-driven decisions regarding irrigation, pest control, and harvesting, ensuring optimal crop yields and resource utilization.

• Industrial Manufacturing

IoT devices in manufacturing plants collect real-time data on machine performance, production rates, and product quality. Situationally aware systems can detect anomalies or equipment failures, triggering maintenance alerts or adjusting production parameters to minimize downtime and maximize output.

• Smart Cities

In urban environments, IoT sensors and cameras provide data on traffic congestion, air quality, and waste management. Situationally aware applications can reroute traffic, optimize public transportation schedules, and manage energy consumption based on changing conditions to enhance the quality of life in cities.

• Healthcare

In healthcare settings, IoT devices, such as wearable health monitors and smart medical equipment, collect patient data. Situationally aware healthcare systems can analyze this data to provide real-time health alerts to medical professionals, improving patient care and response times.

The Challenge of Domain Expertise

All of these applications share a common challenge: even when they detect situations of interest, knowledge workers require a substantial amount of experience and institutional knowledge to interpret the presented information and to decide how to respond. The body of knowledge and data that technicians and clinicians need to manage is extensive and continually expanding. Some of this information is locked in systems that are difficult to access or that respond with an information overload that needs to be filtered in order to be digestible, leading to people ignoring useful sources of information. Also with an aging workforce a lot of institution knowledge if being lost but capturing that information in knowledge bases is not sufficient as this further increases the information overload on inexperienced knowledge workers.

Generative AI tools like ChatGPT and other chatbots are widely used by knowledge workers today, but they exhibit several limitations:

• These tools lack situational awareness and do not possess knowledge of the current status of a time-critical situation to be monitored by IoT (think of a contested battlespace or patient health in an ICU).
• They do not have access to historical or proprietary information that knowledge workers may possess and require to make a decision (think patient medical records).
• Most LLMs were trained on data from around 2021 and may potentially be outdated.
• LLMs are susceptible to hallucinations, especially when trained on conflicting data or when data was missing from the original training dataset. Fine-tuning these models is possible but is a highly specialized, costly, and time-consuming process.
• Constructing the appropriate prompt to extract the most from the LLM is a complex undertaking.

Overall, while Chatbots are impressive, they have limited utility for knowledge workers when they need to respond to work-related activities. Like search engines in the 90s, users must learn how to phrase questions effectively. If the required information is not available, these tools offer limited assistance, and users must combine existing knowledge with search engine results to locate the necessary information.

Situational Awareness Requires a New Data Architecture

Integrating situationally aware IoT-style applications with LLMs, (and utilizing techniques like RAG (Retrieval Augmented Generation), Auto Prompting, automatic inclusion of historic information into prompts, and incorporating features like React to inject task-specific actions into LLMs), enables systems that manage situational awareness to begin to address all the aforementioned issues and enhance the productivity of knowledge workers.

• Auto Prompting

Instead of users manually initiating interactions with an LLM via a Chatbot-like ChatGPT, Auto Prompting can, within the system, automatically generate responses to questions that users would naturally want to know next, based on situational awareness and a prompt template. For example, when a situation is detected, the system can instruct the LLM to generate responses to predefined questions without requiring user input.

• RAG (Retrieval Augmented Generation)

RAG is a framework for retrieving information from external knowledge bases, thereby providing the model with external sources of information. RAG offers several advantages, such as ensuring access to the most up-to-date, reliable facts and enabling users to verify the model’s claims for accuracy and trustworthiness. This helps prevent hallucinations and allows LLMs to answer questions related to subjects or information they were not originally trained on.

• Combining Historic Information

An LLM lacks awareness of the real time specifics of any given situation of interest, which the situationally-aware application captures. This information can be automatically added by the application to the prompt, potentially fine-tuning and directing information retrieval in RAG to be more specific to the current situation. For instance, if a machine’s air filter was changed two weeks ago, information about the correct placement and fitting of the air filter may be useful when generating diagnostic steps.

Conclusion

When all of these capabilities are combined, we can provide knowledge workers with comprehensive, reliable information automatically, rather than merely presenting raw data for analysis. Additionally, knowledge workers can interact with this information through chatbots that can now provide answers based on the latest information. Furthermore, these chatbots can be updated with new information as it becomes available, enabling them to act proactively. The horizon includes the possibility of systems akin to Jarvis in Iron Man.