(working title) Meaningful data visualizations don’t visualize the actual data

Prompt: design a novel mode of engagement with data, using sensor information from a physical environment

Immediately, my mind headed in the direction of spacial data visualization. However, with the data being environmental sensor information, I wonder:

Rather than visualizing what can already be felt, how can the data be reinterpreted for another level of application?

I ask this question as a designer with the drive of creatively and artistically using this new tool at hand (sensor data). On the other hand, as a researcher, it turns out this question is actually what drove the technical development of the sensor.

Background on ubiquitous computing: “Enchanted Objects” by David Rose

Enchanted objects are a method of moving towards ubiquitous computing, where technology is seamlessly and screen-lessly embedded into our physical environment. They are ordinary things with extraordinary, magic-resembling, capabilities — often semantic to the object itself, and often objects that run in our “cultural bloodstream”. They satisfy human wants:

1. All-knowing. Having data that’s unavoidable is a way of changing people’s behaviors — in a predictable way.
2. Desire to connect to people.
3. Safekeeping
4. Effortless travel

“I want you to make enchanted objects.” This is how:

Background on mites.io

mites.io was developed by the FigLab at CMU. It is a single camera-less, plug-and-play sensor package that can capture 12 multimodal sensing modalities. Use cases for these types of technology span to context-awareness and activity labeling, and contribute towards a future of ubiquitous computing. At a high level, the crux of this type of technology development is acquiring the greatest and most useful amount of sensing information at the lowest cost to privacy concerns. This is especially made possible through ML, which takes the data collected from simply being sensed, to classified, and then labeled.

(directly) Related works that I referred to include:

• Ubicoustics
• Listen Learner

Their use cases: making human valuable, (sometimes hard and emotional) decisions, logically.

human-ai information -> make the sensor data AI,

• humans are part of the ai system
• helping the environment learn that it is getting tracked
• making workspaces more appealing

My first round of questions/thoughts

The existing research uses a combination of sensor data to make assumptions on the activities/stories of the space.

precursor: [[What does data visualization “typically” look like? (esp. displayed within a space). Visually (and trend wise), how can we expect/predict it to evolve? Further so, how can data viz evolve with new types of information—provided by new types of technology?]]

• What are the super interesting stories that can be told through remixing/reinterpreting the sensor data?
• How can context-awareness be physicalized?
• (thinking in the direction of robotics) How can a designerly approach bridge the gap between sensing and labeling, and assumptions and decision-making?

extra credit: How can the results of this project better guide further technical development of context-aware sensing technologies?

One last non-leading (maybe somewhat leading) curiosity of mine: Enchanted objects are obvious interfaces for technology. I wonder how this sensor research has to relate with “interfaces”. Can sensing technology be an interface? And is there (could there be??!!) a connection between enchanted objects and context-aware sensing?

• Could the sensed data inform actions taken by objects, thus making them enchanted?

Exiting the research lab, and returning to the design studio

This project will be guided by human behavior, as well as the detail of it being useful in a shared, public space.

This begs the next question, of comparing IoT in the home vs. shared public spaces.

And it forces me to reconsider other questions such as when asking “How can a space remember occupants?”, we must also ask: What occupants would a public, shared space need to remember? What could this type of space do with a memory?

Most applications of IoT and ML sensing relates to learning and tracking to find patterns to then make educated assumptions. But, patterns are easy to find in private spaces (full of routines), compared to public ones which are less predictable and more often changing. Even when patterns are found, they should tell meaningful information.

(What makes information meaningful?)

• the environment becomes the enchanted object. the environment is learning, reacting (maybe even emotionally), changing. the same way people change their habits when they are aware of their data, can environments change too?
• environments becoming sentient, resembling sentience
• environment as an interface
• feedback: how can you know when/if an environment is communicating with you?

01/24 Next steps: Draft out “stories” this data can tell, think about ways an environment as a whole can be prototyped.

• passing by, how it fits into your day (that screen that you makes you into particles)
• make people want to be in the space more/enjoy being in the space more
• what do the sensors interact with, the objects that people bring in? brightness -> laptops

Consulting Daragh

• spooky technology in a public, shared space vs. private space. types of routines?
• stories that assumptions based on data can tell
• sentient architecture
• ways that environments become interfaces

01/25 I’m Deciding to Lean In

anthropomorphizing a physical environment: how does the space feel being tracked?

• for something to be truly AI, it must be emotional. conscious of its own existence, of its mortality.
• - able to sense and make decisions and understand consequences

What to make: open a conversation between building, sensor, people

• stickers to communicate with the building, that “intercept”/hack that certain sensor, to make it seem like something needs to change
• ie. needing a light open in the daytime
• people become frustrated with buildings and change them when they don’t give you what you need

building a relationship between AI and human, making sensor data intelligent.

dump:

• series of objects within the building that become smart. think: encanto, beauty and the beast, monster house
• materials of building itself become smart. think sentient materials
• a robot, one thing that is an anthropomorphized representation of the building
• stickers or posters/AR
• building becomes smart building. turns lights on/off and shit