Making an AR stethoscope in the form of a Y2K walkman that kind of acts like a metal detector and xray too.
Week Zero: Assigned readings and precedents:
Sonic Arcade: I think that these are great gallery and art pieces, but I would be interested in seeing real world application of these technologies or scalability.
Bloom: the common AR headset problem where the hand gesture interaction feels unnatural to outside users
Magic Leap: I wonder how the experience feels without tactile feedback.
Towards a Framework for Designing Playful Gustosonic Experiences:
- “design sound ambiguity to direct attention back to ____”
- design the experience as “easy fun”
- cognitive immersion describes how players figure things out but focus more on surreal things
- behavioural immersion refers to players doing particularly enjoyable behaviours in their own manner
Week One:
So basically I was in my spooky tech class last Thursday morning doing crits and this one group next to me connected a color detector & proximity sensor to a speaker so that depending on the color it faced, a different recorded sound played. They did like no fabrication so it was a bare sensor on a breadboard with a speaker wired to it. I watched the crit lady try using it and she held the speaker really close to her ear and the entire breadboard up to the colors and it looked like a stethoscope/solocup phone.
Seeing the accidental interaction with this super lowfi prototype really inspired me when thinking about the concept of augmenting reality through sound — I was like, “can I create a stethoscope to allow someone to hear what isn’t there? to hear into another dimension?”
I think hearing into another dimension is a novel take on AR.
Research question:
- tapping into another dimension
- messages hidden, hidden SOUL, hidden HEART
- messages of a multi-dimension
- gifting
Precedents: https://www.youtube.com/@ELECTRONICOSFANTASTICOS_YT
High level project description: embedding/decoding sound into patterns
The stethoscope:
- running against something dimensional
- stethoscope: tells you about the heartbeat
- metal detector: tells you when you’re close to treasure
Gifts that are mailed/wrapped:
- teddy bear
- bouquet
- wrapping paper
- blanket
Entice, Enter, Engage, Exit, Extend
Research Questions:
Most people when thinking about augmented reality see superimposing visuals onto surfaces. What about superimposing sounds?
- How would sound augment reality differently than visuals? What are the affordances (and limitations) of sound?
How could a tangible user interface that forces a user to be physically connected to an object increase the intimacy and challenge the relationship they have with the object?
Should we care about:
- How does the sound get there?
- How does the stethoscope get there?
- → Not typically thought about for AR…
Largest range possible:
- small to large
- 2D to 3D
- absurd to expected
- sentimental to common
[emily, upload pic of chameleon note paper]
Week Two: Early-stage Prototyping
Form:
- round body, contains hardware and allows wires to wrap around
- over vs. in ear headphones
- ergonomic, hand-grasp-sized sensing piece
Deciding on the components:
- sensor for rotation (scrubbing)
- scrubbing through sound file
- sensor for pressure (volume)
- tagging
Rotation
First attempt: understanding compass vs. accelerometer vs. gyroscope
Using GY-271 compass — not very reliable.
Second attempt: Switching to MPU 6050 https://www.electronicshub.org/getting-started-arduino-mpu6050/
Print only yaw value gives orientation (vs. rotation). ← wow, the power of using the right words. yayy
Sound Scrubbing
gia, insert p5 sketch
p5 sketch: scrubbing + volume
Sound update: we will serial communicate between Arduino and p5.js sketch, which will do all the sound manipulation and
Week Three: Early-stage Prototyping (continued)
Tagging
Using RFID reader and stickers.
Headphones/Fabrication
You are attached. Your torso is like the string.
Differentiate form between hearing and sensing piece so user knows which is which.
Should we put in an LED that also signals that you have reached the sound?
<<
Connecting chatgpt?
- arduino → p5 sketch → chatgpt → text to speech generator → .wav file → download onto computer → back to p5 sketch
- chatGPT would be the object itself talking to user, recording would be human-to-human communication
>>
Form:
- squishable quality? / amount of squish
Week Four: tech prototyping + material explorations
Tags
#include <SPI.h>
#include <Wire.h>
#include <MFRC522.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
// Mifare TC522 settings
#define SS_PIN 12
#define RST_PIN 13
MFRC522 mfrc522(SS_PIN, RST_PIN); // Create MFRC522 instance.
int tagValue;
int mapped_tagValue;
void setup() {
Serial.begin(115200);
// Serial.println("\n\nRFID Reader 1.0 - first prototype");
// initDisplay();
// Initialize SPI
SPI.beginTransaction(SPISettings(1000000, MSBFIRST, SPI_MODE0)); //set parameters
SPI.begin(); // init SPI
delay(10);
mfrc522.PCD_Init(); // Init MFRC522 card
// printReceivedText( "Calling OpenAI…" );
// Serial.println("ASK_OPENAI");
}
void loop() {
// Look for new cards
if ( mfrc522.PICC_IsNewCardPresent()) {
// Serial.println("Card present");
// Select one of the cards
if ( mfrc522.PICC_ReadCardSerial()) {
// Serial.println("Data available");
// Serial.print("UID tag :");
String content = "";
byte letter;
for (byte i = 0; i < mfrc522.uid.size; i++) {
// Serial.print(mfrc522.uid.uidByte[i] < 0x10 ? " 0" : " ");
// Serial.print(mfrc522.uid.uidByte[i], HEX);
content.concat(String(mfrc522.uid.uidByte[i] < 0x10 ? " 0" : " "));
content.concat(String(mfrc522.uid.uidByte[i], HEX));
}
// Serial.println();
// Serial.print("Message :");
content.toUpperCase();
if (content.substring(1) == "21 2A F6 F6") { //change here the UID of the card/cards that you want to give access
// tagValue = 1;
// mapped_tagValue = map(tagValue, 0, 1023, 0, 255);
Serial.print("bear"); //bear
Serial.println();
delay(100);
// Serial.write(mapped_tagValue);
//Serial.print(String(mapped_tagValue));
}
else if (content.substring(1) == "F1 44 F7 F6") { //change here the UID of the card/cards that you want to give access
// tagValue = 200;
// mapped_tagValue = map(tagValue, 0, 1023, 0, 255);
Serial.print("toaster"); //toaster
Serial.println();
delay(100);
// Serial.write(mapped_tagValue);
//Serial.print(String(mapped_tagValue));
}
else {
Serial.println(" Access denied");
Serial.println();
delay(3000);
}
// // Dump debug info about the card. PICC_HaltA() is automatically called.
// mfrc522.PICC_DumpToSerial(&(mfrc522.uid));
}
}
}
Arduino → p5.js sketch
My code so far: https://editor.p5js.org/emilyliu014/sketches/8GFug-XXu
Further reference: https://docs.google.com/document/d/1-TtSOvSFKvi71kwfjSZ0pF9OHeZ4ODIX50imVP6hk0M/edit?usp=sharing
(thanks Daniel)
note: try mapping.
04.18 ~6pm update: tried mapping. it still becomes random numbers in the p5 GUI thing.
04.19 1:30pm: look at this https://forum.arduino.cc/t/receiving-random-numbers-from-serial-read/700957/6
chatGPT → p5.js sketch
My code so far: https://editor.p5js.org/emilyliu014/sketches/iXdsHPMt1
Reference this project for code maybe: http://ideate.xsead.cmu.edu/gallery/projects/sherlock
material exploration
vibe/story:
alien-like. arrived from another planet.
interdimensional. deep time
discovery aspect:
mix between found object and alien
research question development (04.22)
the prompt is about augmenting reality through sound.
Augmented reality as we know it superimposes new layers of information. What about information that was always there, that just hasn’t been tapped into?
If the world had more stories to tell, what would they be? How would they be found? What would unlock them? And how would the stories want to be told?
Strategic insight:
Observation: started off inspired by the forms of “magical” listening devices, like cans connected by string. (how does string carry sound??)
the device is like a portal, what are the possibilities it is opening? translation machine
Observation: devices physically connecting you to sound vs. current day where everything is connected by bluetooth. how does the experience change if it’s a physical (vs. digital) connection. a lot of AR devices are also wearables
how do our relationships with stories and experiences with our environments change when they are ephemeral? what are ways mix-modality can make experiences more ephemeral?
Observation: people are used to connecting to other people remotely physically/digitally . used to taking in mobile vs. stationary information.
How people are accustomed to listening to stories (status quo) vs. ______
Approach:
type of information: not prerecorded, live and generated information. new type of story.
- taking pre-existing information, assembling it in a certain way.
- way of experiencing historical data
- empowering things to tell their own(?) stories → degree of sentience, appearing sentient?
an extra-terrestrial device that lets you listen in to the “voices” of objects and spaces (found at their “heart” → voice box?).
this object is found/discovered. it looks like it was dropped onto our planet from another similar planet.
- has perfections found in nature. logarithmic spiral. and ergonomic nature. meant to be held and used by humans. → way for the world to talk to you. almost like a gift.
- blinks to beckon you and glows to show its “aliveness” and responsiveness to your presence
technical development update: switched to processing. life is so much better now.
OpenAI API to Processing
import processing.net.*;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
void setup() {
String OPENAI_API_KEY = " "; // replace with your actual API key
String url = "https://api.openai.com/v1/completions";
String model = "text-davinci-003";
String prompt = "Tell me this is a sample prompt.";
double temperature = 0.7;
int maxTokens = 256;
double topP = 1;
double frequencyPenalty = 0;
double presencePenalty = 0;
// Build the request body JSON string
String requestBody = String.format("{\"model\": \"%s\", \"prompt\": \"%s\", \"temperature\": %s, \"max_tokens\": %s, \"top_p\": %s, \"frequency_penalty\": %s, \"presence_penalty\": %s}",
model, prompt, temperature, maxTokens, topP, frequencyPenalty, presencePenalty);
// Create HttpRequest
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + OPENAI_API_KEY)
.POST(HttpRequest.BodyPublishers.ofString(requestBody))
.build();
String responseBody = "";
// Send the request and get the response
try {
HttpClient client = HttpClient.newHttpClient();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.body());
responseBody = response.body();
} catch (IOException | InterruptedException | IllegalArgumentException | SecurityException e) {
e.printStackTrace();
}
// Extract the "text" field from the response body
String text = "";
if (!responseBody.isEmpty()) {
int startIndex = responseBody.indexOf("\"text\":") + 8;
int endIndex = responseBody.indexOf("\",", startIndex);
text = responseBody.substring(startIndex, endIndex);
}
// Print the text
System.out.println(text);
exec("say", text.substring(4));
}Final Processing code. Taking in serial input (RFID tag) to choose prompt. TTS reading it out loud. The story ends when tag leaves, and resets when on a tag again.
import processing.serial.*;
processing.serial.Serial myPort;
String myString = null;
int nl = 10;
float inData; // Data received from Serial port
float inDataLastFrame;
import processing.net.*;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
boolean shouldSendPrompt = true;
import java.lang.Process;
Process speechProcess;
void setup() {
String portName = "/dev/cu.usbserial-02898BB7"; //change to match your port
myPort = new processing.serial.Serial(this, portName, 115200);
}
void draw() {
if ( myPort.available() > 0) { // If data is available,
myString = myPort.readStringUntil(nl);
//inData = myPort.readStringUntil('\n'); // read it and store it in val
//
if (myString != null) {
inData = float(myString);
} else {
println("no data recieved");
}
if ((int)inData != (int)inDataLastFrame) {
println((int)inData);
if ((int)inData == 0) {
println("stop");
if (speechProcess != null) {
speechProcess.destroy();
}
}
if (shouldSendPrompt) {
if ((int)inData == 1) {
//println("1");
sendPromptToOpenAI("in less than 100 words tell me a story about a girl named Shannon");
}
if ((int)inData == 2) {
//println("2");
sendPromptToOpenAI("in less than 100 words tell me a story about a girl named Gia");
}
shouldSendPrompt = false;
}
shouldSendPrompt = true;
}
//print((int)inData);
}
inDataLastFrame = inData;
}
void sendPromptToOpenAI(String prompt) {
String OPENAI_API_KEY = " "; // replace with your actual API key
String url = "https://api.openai.com/v1/completions";
String model = "text-davinci-003";
double temperature = 0.7;
int maxTokens = 256;
double topP = 1;
double frequencyPenalty = 0;
double presencePenalty = 0;
// Build the request body JSON string
String requestBody = String.format("{\"model\": \"%s\", \"prompt\": \"%s\", \"temperature\": %s, \"max_tokens\": %s, \"top_p\": %s, \"frequency_penalty\": %s, \"presence_penalty\": %s}",
model, prompt, temperature, maxTokens, topP, frequencyPenalty, presencePenalty);
println(requestBody);
// Create HttpRequest
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + OPENAI_API_KEY)
.POST(HttpRequest.BodyPublishers.ofString(requestBody))
.build();
//Initialize text variable
String responseBody = "";
// Send the request and get the response
try {
HttpClient client = HttpClient.newHttpClient();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.body());
responseBody = response.body();
}
catch (IOException | InterruptedException | IllegalArgumentException | SecurityException e) {
e.printStackTrace();
}
// Extract the "text" field from the response body
String text = "";
if (!responseBody.isEmpty()) {
int startIndex = responseBody.indexOf("\"text\":") + 8;
int endIndex = responseBody.indexOf("\",", startIndex);
text = responseBody.substring(startIndex, endIndex);
}
// Print the text
System.out.println(text);
speechProcess = exec("say", text.substring(4));
shouldSendPrompt = true;
}
void stop() {
if (speechProcess != null) {
speechProcess.destroy();
}
}https://vimeo.com/827094200?share=copy
Physical Form
