Accessibility Copilot
Microsoft Hackathon 2024
Our hackathon team created a vision for an Accessibility Copilot for Figma as a plug-in for designers to introduce accessibility in the early stages of their work.
Accessibility Copilot
Our project is driven by a passion for improving accessibility in digital design, enhancing the user experience for people with disabilities, and innovating with design tools and technology.
We are creating a tool within the Flubert Figma Plugin to help annotate designs for accessibility, powered by AI. This will help to clearly communicate accessibility requirements downstream to Engineers. In the future, we hope to add further capabilities to Flubert, such as accessibility testing, validation, and AI co-design for accessibility, to make a significant impact on the design community.
PRODUCT DESIGN @ MICROSOFT
PROCESS SUMMARY
Quick hackathon sprints: This project was part of the 2024 Microsoft Security org hackathon and the 2024 Microsoft global hackathon. We received 2nd place in the hack for diversity & inclusion category.
TEAM
Head of accessibility — Michael Fairchild, Mallika Meiyappan
Product Designers — Amy Zhang, JaneLynn Chan, Solomon Abbey, Hugo Palomares
Engineers — Tony Chan
Final video submission for Accessibility Copilot in Flubert created by me. Click bottom right icon to enable sound.
SCOPING THE PROJECT
Executive challenge
For our Hackathon, we chose the Ability Hack Executive Challenge, which invites innovative projects centered on accessibility and disability-related ideas. Our team, mostly comprised of product designers and accessibility leads, was especially passionate about the idea of including the differently abled into every design we create.
The challenge encompasses six key areas—Vision, Hearing, Neurodiversity, Speech, Mobility, and Mental Health—allowing for a broad exploration of the disability experience. We were allowed the flexibility of enhancing existing products or develop entirely new solutions. Our ultimate goal was to foster creativity and support impactful initiatives in this vital space.
How would you update this design to be more accessible?
SPRING HACKATHON 2024
Co-design, Annotate, Validate
For our first hackathon, our team focused on creating a new Figma plug-in that was able to help with all things accessibility, including annotation, validation, and an open-prompting co-designer tool. This accessibility Copilot, or A11y Co-designer, focused on helping designers identify all possible accessibility concerns with their designs in one quick scan.
A11Y CO-DESIGNER
Open-prompting tool to help designers scan their current designs and quickly identify any accessibility concerns.
ANNOTATE
Helping auto-generate accessibility annotations like screen-reader labels and focus order tab stops.
VALIDATE
When designers are done designing, they can use the validate tool to check their color contrast and make sure they are meeting the accessible design principles.
A11y Co-designer
The Co-designer was our main focus for this project, as we believed it to be the most revolutionary part of the plug-in proposal. This Copilot-powered tool would allow users to upload their own custom best practices into a knowledge base, scan their Figma designs directly into the plug-in, and Co-designer would provide an easy-to-follow list of suggestions in a popover format.
Furthermore, this powerful tool would be able to create new updated versions of your designs with the suggested updates directly in your file.
KNOWLEDGE BASES
The Co-designer tool would include a knowledge base where designers could upload their design systems and accessibility best practices.
SCAN AND GENERATE
The value proposition of A11y Co-designer is its ability to generate accessibility suggestions for your designs using AI. It can also generate an "accessibility moodboard" with updated designs that better fit accessibility best practices.
SUGGESTION POPOVERS
The suggestions for your design frame appear in the plug-in, but also as popovers on your Figma workspace. This allows the plug-in to locationally pinpoint where the accessibility opportunity is.
NO HEADING DETECTED
A11y Co-designer noticed that this design is missing a header, which makes the navigation and understanding of this page unclear.
NO FIELD LABELS
The fields on this design are missing labels, which would be confusing once the user enters information into the fields.
ACCESSIBILITY MOODBOARD
GENERATING NEW OPTION
A11y Co-designer can generate improved, more accessible designs for you based on your original design.
You can ask Co-designer to combine design elements from different options into a new design.
APPLY UPDATES
UPDATES APPLIED
Co-designer will replace your original design with a vector version of its approved AI-generated option.
You can ask Co-designer to apply any of its generated designs to your original design.
Annotate and validate
Annotation and validation are important steps in the design process for accessibility. After Co-designer helps you create a mostly-accessible design, you can use the annotate feature to generate a focus order table and the validate feature to check your knowledge of accessibility principles.
FALL HACKATHON 2024
Flubert plug-in
For the second hackathon we explored ways to continue this project in small, realistic steps and focused on how we could get this project to reality.
​
We learned that there is currently a Microsoft Fluent design plug-in called Flubert, aimed to help designers using the Fluent design system. This was the perfect place to introduce our accessibility annotation feature, as Flubert's current design integrates Copilot to help users during the Figma design phase.
Flubert Figma Plug-in
Annotations with ease
For this hackathon project, we focused on perfecting the annotation flow and integrating it seamlessly within Flubert. According to our design, this tool will be able to generate an annotation table customized to your frame.
SCANNING FOR ANNOTATIONS
You can select a frame that you want Flubert to scan and create auto-annotations.
EDIT IN FLUBERT
You can ask Flubert to directly make changes to your annotation table with open-prompting, or make changes in your table yourself.
GENERATED ANNOTATION TABLE
The editable annotation table is auto-generated and the corresponding stamp card is created, allowing users to drag and drop onto their design.
REFRESH TABLE
After prompting in Flubert, your annotation table is refreshed with your changes.
Watch the full video walkthrough of Flubert for more insight into how this works!
TAKEAWAYS
Outcomes
Our hackathon team won second place in the spring security hackathon in the Hack for Diversity & Inclusion category with our Accessibility Copilot proposal. We plan to continue working on this project with the Flubert development team and hopefully turn our vision into reality!
SHORT 1-WEEK SPRINT
It was challenging to work on a 1-week sprint timeline due to the nature of the Hackathon time constraints. We had to scope our project down to a manageable workload, which required us to test our project management skills.
VIDEO CREATION
I was tasked to create the final video for our spring hackathon, which meant I needed to create a compelling script, include all the design details that made our project special, and learn how to best edit the video.
Congrats! You made it to the end of this page.
Hope you enjoyed coming along for the ride, and I truly appreciate you taking the time to check out my work! You can reach me at amyzh425@gmail.com — I'd love to chat with you.