Platform for Situated Intelligence
Platform for Situated Intelligence is an open, extensible framework that enables the enables the development, fielding and study of situated, integrative-AI systems.
In recent years, we have seen significant progress with machine learning techniques on various perceptual and control problems. At the same time, building end-to-end, multimodal, integrative-AI systems that leverage multiple technologies and act autonomously or interact with people in the open world remains a challenging, error-prone and time-consuming engineering task. Numerous challenges stem from the sheer complexity of these systems and are amplified by the lack of appropriate infrastructure and development tools.
The Platform for Situated Intelligence project aims to address these issues and provide a basis for developing, fielding and studying integrative-AI systems. The platform consists of three layers.
The Runtime layer provides a parallel programming model centered around temporal streams of data, and enables easy development of components and applications using .NET, while retaining the performance properties of natively written, carefully tuned systems.
A set of Tools and APIs enable multimodal data visualization, processing and analytics. The image below shows Platform for Situated Intelligence studio, i.e., the multimodal data visualization tool.
Finally, an open ecosystem of Components encapsulate various AI technologies and allow for quick compositing of integrative-AI applications.
For more information about the goals of the project, the types of systems that you can build using it, and the various layers see Platform for Situated Intelligence Overview.
Using and Building
Platform for Situated Intelligence is built on the .NET Framework. Large parts of it are built on .NET Standard and therefore run both on Windows and Linux, whereas some components are specific and available only to one operating system (for instance the Kinect sensor component is available only for Windows.)
You can build applications based on Platform for Situated Intelligence either by leveraging nuget packages, or by cloning and building the code. Below are instructions:
Brief Introduction. To get started with using Platform for Situated Intelligence, the Brief Introduction page provides a guided walk-through for some of the main concepts in \psi. It shows how to create a simple program, describes the core concept of a stream, and explains how to transform, synchronize, visualize, persist to and replay data from disk. We recommend that you work through the examples in this tutorial to familiarize yourself with these core concepts.
Samples. After going through this first brief tutorial, it may be helpful to look through the set of Samples provided. While some of the samples address specialized topics such as how to leverage speech recognition components or how to bridge to ROS, reading them will give you more insight into programming with \psi.
In-depth Topics. Additional, in-depth information is provided in a set of Topics that dive into more detail on various aspects of the framework including basic stream operators, synchronization, writing new components, delivery policies, remoting, interop etc.
Components. Additional useful information regarding available packages and components can be found in the NuGet packages list and in the component list pages. The latter page also has pointers to other repositories by third parties containing other \psi components.
Like the rest of the codebase, the documentation is still under construction and in various phases of completion. If you need further explanation in any of these areas, please open an issue, label it
documentation, as this will help us target our documentation development efforts to the highest priority needs.
Platform for Situated Intelligence is currently in beta and various aspects of the platform are at different levels of completion and robustness. There are probably still bugs in the code and we will likely be making breaking API changes. We plan to continuously improve the framework and we encourage the community to contribute.
If you find a reproducible bug or if you would like to request a new feature or additional documentation, please file an issue on the github repo. If you do so, please make sure a corresponding issue has not already been filed. Use the
bug label when filing issues that represent code defects, and provide enough information to reproduce. Use the
feature request label to request new features, and use the
documentation label to request additional documentation.
We hope the community can help improve and evolve Platform for Situated Intelligence. If you plan to contribute to the codebase, please read the Contributing Guidelines document. It describes how the source code is organized and things you need to know before making any source code changes.
Who is Using
Platform for Situated Intelligence is currently being used in a number of industry and academic research labs, including (but not limited to):
- in the Situated Interaction project, as well as other research projects at Microsoft Research.
- in the MultiComp Lab at Carnegie Mellon University.
- in the Speech Language and Interactive Machines research group at Boise State University.
- in the Qualitative Reasoning Group, Northwestern University.
Platform for Situated Intelligence is available under an MIT License.
We would like to thank our internal and external early adopters for the feedback provided during the alpha testing period, including (but not limited to): Daniel McDuff and Kael Rowan at MSR, Chirag Raman and Louis-Phillipe Morency in the MultiComp Lab at CMU, as well as adopters in the SLIM research group at Boise State and in the Qualitative Reasoning Group at Northwestern University.