#UX Design Research
#Prototyping
#Javascript
#Computer Vision
#Accessibility
Major Studio course at MFA Design & Technology at Parsons School of Design
Figma/FigJam, Javascript, p5.js, handsfree.js
Developed during a core studio course at Parsons School of Design, this project aims to expand digital communication, specifically video call communication. This project goes through extensive research, idea conception, prototyping, and a working code implementation involving hand tracking with handsfree.js.
Computer-mediated communication, or any socialization that occurs over social media or the internet, is an inclusive & comforting space for many adults with ASD. This is because it requires less on-the-spot decoding of social information, since a person can take time to process and respond to a text message or DM.
Video calls can be alienating and distressing for autistic participants. This is because video calls remove any potential for cue assistance through bodily gestures. Furthermore, lack of feedback can cause misunderstandings, resulting in social & conversational insecurity.
Shoulder-up video calls lack visual and bodily cues that are often helpful in decoding conversations.
Rather than further alienating disadvantaged groups with temporary assistive technologies that may not be adopted by everyone, video calls are already a successful and widely used method of communication.
Instead, the video call system itself should be improved, and conversational hand gestures should be reintigrated for cue assistance.
It is important for AT to not be alienating. Oftentimes, this occurs from unwanted visibility resulting from use of AT in public settings. By making the system open to everyone, it fosters empathy in communication between groups of people.
79.6% of adults with ASD prefer social media for socialization and find it to be a more comfortable way to communicate with others. (source)
1. Purposely design for adults with ASD in mind
Currently, assistance widely focuses on children and those caring for them. As a result, it implies that adults with ASD require less support.
2. Actively design againt the infantilization of ASD
Infantilization occurs when people are treated like children and expected to be naive or "pure". This also occurs when someone addresses an accompanying person rather than acknowledging a disabled adult themselves.
3. Design for the target user with the potential to improve it for everyone
Many well-designed interventions have the ability to assist more than just their target group (ex. subtitles, curb cuts, text-to-911...)
4. Intervene digitally to holistically empower offline socialization
Once comfortable with de-codified online communication, people can carry that with them into real life.
Online Survey
An online survey was shared with classmates and acquaintances. The purpose was discover what messages people would want to be able to gesture and people's perceptions of different hand gestures.
1. Testing Physical Cues
Low-tech experiment with neurodivergent & neurotypical individuals to analyze how gestures can provide cue assistance.
Conclusion:
2. Technology Prototype with Handsfree.JS
Test the technology that could add hand gestures & conversational cue gestures into video calls.
Conclusion:
3. Figma Initial Wireframes
Figma was used to ideate where and how gesture integration could be included in the UI. I settlef on having a hand icon at the top right so it is grouped with the video call view type button.
EDIT: Since then, Discord's UI has changed. As of 2024, if I were to redo this project, I would include it at the bottom.
Try using it directly from the fullscreen p5.js viewer >>>
In order to use, do the following:
1. Allow webcam access to the browser
2. Click the hand button at the bottom on (it will turn green)
3. For more info, click the carrot button directly under the hand
4. To turn it off, select the hand button again
If I were to design further:
