Amazon Bedrock model distillation
A Zero-to-One innovation product- Empowering customers to scale their Gen AI applications by optimizing price and performance for their specific use-cases.
Company: Amazon web service (aws)
My Role
I executed as the User Experience design lead for this Zero - to -One innovation product from inception to launch- Bedrock Model Distillation. This product previewed in re:Invent 2024, as an opening key notes of AWS CEO Matt Garman and Amazon CEO Andy Jassy.
My role for this project included:
Design Vision and Strategy:
Driving UX-Product alignment with S-team (VPs and Senior Directors). Articulated design vision and approach to senior leadership.
Contributing in defining product requirements based on user research, influencing science team’s innovation and engineering team’s feasibility with rapid prototyping and iterating from concept to launch.
Cross-functional Collaboration:
Aligned with partner teams to identify and aggregate disparate qualitative and quantitative data to inform design solutions at the product level.
Collaborated effectively with partner teams to mitigate risks and manage dependencies. Decoupled complex dependencies on design and tech decisions.
Design Leadership and Execution:
Demonstrated high judgement trade-off decisions for the product, incorporating system limitations, scaling factors, edge cases, and architectural decisions.
Success Metrics
Model Distillation, will help reduce compute cost by 75% and accelerate model efficiency by 500% for Million plus users. Achieved adoption of 30K+ distillation jobs in 6 weeks post-launch.
AWS LAUNCH ANNOUNCEMENT
Amazon Bedrock Model Distillation Preview
Project timeframe
3 months(September 2024- November 2024).
Launched: Preview December 3 2024.
PLATFORM
Responsive, Enterprise product.
Team
Product Manager, Data Scientist, Front-end developers, Backend developers, Legal, UX copywriter.
UX Tools Used
Figma, Figjam, Usertesting.com
What is Model Distillation?
Bedrock Model Distillation enables its customers to use smaller, faster, more cost-effective models that deliver use-case specific accuracy that is comparable to the most models in Bedrock. Amazon Bedrock is a service for building generative AI applications on Amazon’s cloud computing platform, Amazon Web Service
CONTEXT
In the current landscape, customers have access to powerful and highly accurate Foundation Models (FMs) for their Generative AI (Gen AI) applications. However using these FMs can be resource-intensive, and their high latency may not be suitable for real-time applications like virtual assistants or chatbots, where slow responses can frustrate users.
PROPOSED SOLUTION
To bridge this gap, we launched a new primitive offering in preview for model customization in Bedrock called Model Distillation. Distillation is the process of transferring knowledge from a more capable model (teacher) to a less capable one (student) with the goal to make the faster and cost-efficient student model as performant as the teacher for a specific use-case.
Value Proposition
Efficiency
Cost Optimization
Advance customization
Ease of use
Target users
Target customers are those who are looking for ways to scale their Gen AI applications across millions of users, and leverage powerful Foundation Models (FMs) through model customization while optimizing for efficiency, cost, and performance in their Gen AI applications.
the design process
I led the daily/weekly meeting with my stakeholders including Product Manager (PM), Data Science team, Engineering team through the design process.
Empathize and Discover: I teamed with Product Manager, Data Scientist, and legal team to strategize accurate and deep intuitive understanding of the customer needs and translate concepts into features that addressed user’s information needs and behavior. I created journey maps, user flows, and task analysis during early design phase.
User Interviews: I conducted customer interviews to gather and understand their requirements and constraints. I collaborated with Product Manager and various customers to gain insight into user behavior and expectations. I was rapidly prototyping and iterating based on customer feedback and stakeholder feedback.
Hunt for Data Source and Utilize Metrics and collaborate with diverse teams: I collaborated with engineering team to get adoption metrics for existing “custom models” feature in Bedrock to understand how customers are using current methods of Fine-tuning and Continued pre-training and how we could improve the flow in the new feature. I collaborated with partner teams - Model Evaluations, Inference, Provision Throughput to understand and propose how Model Distillation would surface in these areas.
Strategy, Vision, Ideation and Iteration: I facilitated design workshops with the stakeholders to discover early stage concept designs which were used to customer feedback in user interview. I participated in S-team stakeholder discussion to push back for customer needs and request for certain features that required me to give my input on how we would define pricing for this product. I gave several UX design reviews to UX Design Review (part of UX Review process at AWS), to S-team leaders at every stage of design process to gain alignment and validation from various cross-functional teams.
Testing and Validation : I tested and validated the designs with customers early and often during various stages of the design process to craft the final product. This helped the product take its final shape.
Model Distillation Flow Chart
Custom Models - Distillation Method
Teacher -Student Model Details
Synthetic Data Generation
Distillation Output Metrics
Distillation Job created
Teacher-Student model exploration
Speed vs Cost Exploration
Demo Video
Bedrock Model Distillation Demo Video
Challenges & Lessons learned
Model Distillation project had high visibility of the Senior leadership team members (AWS CEO, VP, and Directors). Since its inception, the project went fast while defining the product and rapid prototype iterations happening simultaneously. This project had various pit falls and last minute pivots with regards to Legal, Data Science team’s invention, Engineering team’s feasibility, re:Invent time constraints, and customer needs. UX Designs were being approved and then needed to be changed frequently. I adapted quickly and helped the teams to move faster taking End-to end ownership of the product. I learned to quickly collaborate with all cross-functional teams earning their trust and delivering the product on time with quality and grit.
