Flashield's Blog

Just For My Daily Diary

Flashield's Blog

Just For My Daily Diary

05.exercise-model-cards【练习:模型卡】

This notebook is an exercise in the AI Ethics course. You can reference the tutorial at this link.


In the tutorial, you learned how to use model cards. In this exercise, you'll sharpen your understanding of model cards by engaging with them in a couple of scenarios.

在本教程中,您学习了如何使用模型卡。在本练习中,您将通过在几个场景中使用模型卡来加深对模型卡的理解。

Introduction

简介

Run the next code cell to get started.

运行下一个代码单元以开始使用。

from learntools.core import binder
binder.bind(globals())
from learntools.ethics.ex5 import *

Scenario A

场景 A

You are the creator of the Simple Zoom video editing tool, which uses AI to automatically zoom the video camera in on a presenter as they walk across a room during a presentation. You are launching the Simple Zoom tool and releasing a model card along with the tool, in the interest of transparency.

您是 Simple Zoom 视频编辑工具的创建者,该工具使用人工智能自动将摄像机拉近到演示者身上,当演示者在演示过程中走过房间时。为了提高透明度,您正在推出 Simple Zoom 工具,并随工具一起发布模型卡。

1) Audience

1) 受众

Which audiences should you write the model card for? Once you have thought about your answer, check it against the solution below.

您应该为哪些受众编写模型卡?一旦您想好了答案,请将其与下面的解决方案进行核对。

# Check your answer (Run this code cell to receive credit!)
q_1.check()

Solution: Model cards should be written for the groups that are most likely to read it. For Simple Zoom, such groups probably include people using the tool to record videos, organizations seeking to adopt the tool, IT and audio-visual teams and agencies, computer vision researchers, policymakers and developers of similar AI systems. Given how broad this group is, your team can only assume a basic knowledge of video recording terms throughout the model card.

模型卡应该针对最有可能阅读它的群体来编写。对于 Simple Zoom ,这些群体可能包括使用该工具录制视频的人、寻求采用该工具的组织、IT 和视听团队和机构、计算机视觉研究人员、政策制定者和类似 AI 系统的开发人员。鉴于这个群体的广泛性,您的团队只能假设他们对整个模型卡中的视频录制术语有基本的了解。

Scenario B

场景 B

You are the product manager for Presenter Pro, a popular video and audio recording product for people delivering talks and presentations. As a new feature based on customer demand, your team has been planning to add the AI-powered ability for a single video camera to automatically track a presenter, focusing on them as they walk across the room or stage, zooming in and out automatically and continuously adjusting lighting and focus within the video frame.

您是 Presenter Pro 的产品经理,这是一款供人们进行演讲和演示的热门视频和音频录制产品。作为一项基于客户需求的新功能,您的团队一直计划添加由 AI 提供支持的单个摄像机自动跟踪演示者的功能,在演示者走过房间或舞台时聚焦他们,自动放大和缩小,并不断调整视频帧内的灯光和焦点。

You are hoping to incorporate a different company’s AI tool (called Simple Zoom) into your product (Presenter Pro). To determine whether Simple Zoom is a good fit for Presenter Pro, you are reviewing Simple Zoom’s model card.

您希望将另一家公司的 AI 工具(称为 Simple Zoom )整合到您的产品( Presenter Pro )中。为了确定 Simple Zoom 是否适合 Presenter Pro ,您正在查看 Simple Zoom 的模型卡。

2) Intended Use

2) 预期用途

The Intended Use section of the model card includes the following bullets:

模型卡的 预期用途 部分包括以下内容:

  • Simple Zoom is intended to be used for automatic zoom, focus, and lighting adjustment in the real-time video recording of individual presenters by a single camera
  • Simple Zoom 旨在用于单个摄像机实时录制单个演示者的视频时自动缩放、对焦和调整灯光
  • Simple Zoom is not suitable for presentations in which there is more than one presenter or for presentations in which the presenter is partially or fully hidden at any time
  • Simple Zoom 不适用于有多个演示者的演示,也不适用于演示者在任何时候部分或全部隐藏的演示

As a member of the team evaluating Simple Zoom for potential integration into Presenter Pro, you are aware that Presenter Pro only supports one presenter per video.

作为评估 Simple Zoom 是否可能集成到 Presenter Pro 的团队成员,您知道 Presenter Pro 每个视频仅支持一个演示者。

However, you are also aware that in some Presenter Pro customers use large props in their presentation videos. Given the information in the Intended Use section of the model card, what problem do you foresee for these customers if you integrate Simple Zoom into Presenter Pro? What are some ways in which you could address this issue?

但是,您也知道,在某些 Presenter Pro 客户中,他们的演示视频中使用了大型道具。根据模型卡预期用途部分中的信息,如果您将 Simple Zoom 集成到 Presenter Pro 中,您预见到这些客户会遇到什么问题?您可以通过哪些方式解决这个问题?

# Run this code cell to receive a hint
q_2.hint()

Hint: There is a risk that the large props could partially or fully hide the presenter.

大型道具可能会部分或全部遮挡演示者。

# Check your answer (Run this code cell to receive credit!)
q_2.check()

Solution: Since Simple Zoom is not suitable for presentations in which the presenter is partially or fully hidden at any time, it might not work well in a presentation in which the presenter uses a large object, because the object could partially or fully hide the presenter. There are many potential ways to address this issue. For example, your team could reach out to the Simple Zoom team to assess the potential risks and harms of using Simple Zoom with props. As another example, your team could eventually add a message in the Presenter Pro user interface explaining that the Simple Zoom feature should not be used in presentations that use props.

由于 Simple Zoom 不适用于演示者在任何时候都处于部分或完全隐藏状态的演示,因此它可能在演示者使用大型物体的演示中效果不佳,因为该物体可能会部分或完全隐藏演示者。有很多潜在的方法可以解决这个问题。例如,您的团队可以联系 Simple Zoom 团队,以评估使用道具的 Simple Zoom 的潜在风险和危害。再举一个例子,您的团队最终可以在 Presenter Pro 用户界面中添加一条消息,解释不应在使用道具的演示中使用简单缩放功能。

3) Factors, Evaluation Data, Metrics, and Quantitative Analyses

3) 因素、评估数据、指标和定量分析

We'll continue with Scenario B, where you are the product manager for Presenter Pro. Four more sections of the model card for Simple Zoom are described below.

我们将继续讨论 场景 B ,您是 Presenter Pro 的产品经理。下面将介绍 Simple Zoom 模型卡的另外四个部分。

Factors: The model card lists the following factors as potentially relevant to model performance:

因素:模型卡列出了以下可能与模型性能相关的因素:

  • Group Factors
  • 群体因素
    • Self-reported gender
    • 自我报告的性别
    • Skin tone
    • 肤色
    • Self-reported age
    • 自我报告的年龄
  • Other Factors
  • 其他因素
    • Camera angle
    • 摄像机角度
    • Presenter distance from camera
    • 演示者与摄像机的距离
    • Camera type
    • 摄像机类型
    • Lighting
    • 照明

Evaluation Data: To generate the performance metrics reported in the Quantitative Analysis section (discussed below), the Simple Zoom team used an evaluation data set of 500 presentation videos, each between two and five minutes long. The videos included both regular and unusual presentation and recording scenarios and included presenters from various demographic backgrounds.

评估数据:为了生成定量分析部分(如下所述)中报告的性能指标,Simple Zoom 团队使用了 500 个演示视频的评估数据集,每个视频时长为 2 到 5 分钟。这些视频包括常规和不寻常的演示和录制场景,并包括来自不同人口背景的演示者。

Metrics: Since Simple Zoom model performance is subjective (involving questions like whether a zoom is of appropriate speed or smoothness; or whether a lighting adjustment is well-executed), the Simple Zoom team tested the tool’s performance by asking a diverse viewer group to view Simple Zoom’s output videos (using the evaluation dataset’s 500 videos as inputs). Each viewer was asked to rate the quality of video editing for each video on a scale of 1 to 10, and each video’s average rating was used as a proxy for Simple Zoom’s performance on that video.

指标:由于 Simple Zoom 模型性能具有主观性(涉及诸如缩放速度或平滑度是否合适;或者灯光调整是否执行良好等问题),Simple Zoom 团队通过要求不同的观看者群体观看 Simple Zoom 的输出视频(使用评估数据集的 500 个视频作为输入)来测试该工具的性能。要求每位观看者以 1 到 10 的等级对每个视频的视频编辑质量进行评分,并且每个视频的平均评分用作 Simple Zoom 在该视频上的性能的代理。

Quantitative Analyses: The quantitative analyses section of the model card includes a brief summary of performance results. According to the summary, the model generally performs equally well across all the listed demographic groups (gender, skin tone and age).

定量分析:模型卡的定量分析部分包括性能结果的简要摘要。根据摘要,该模型在所有列出的人口统计群体(性别、肤色和年龄)中通常表现同样出色。

The quantitative analyses section also includes interactive graphs, which allow you to view the performance of the Simple Zoom tool by each factor and by intersections of ‘Group’ and ‘Other’ factors.

定量分析部分还包括交互式图表,您可以通过这些图表查看 Simple Zoom 工具按每个因素以及“组”和“其他”因素的交集所呈现的性能。

As a member of the team evaluating Simple Zoom for potential integration into Presenter Pro, what are some questions you might be interested in answering and exploring via the interactive graphs?

作为评估 Simple Zoom 工具是否可能集成到 Presenter Pro 的团队成员,您可能有兴趣通过交互式图表回答和探索哪些问题?

# Check your answer (Run this code cell to receive credit!)
q_3.check()

Solution: There are many possible answers to this question. For example, you may want to check that the model’s equal performance across demographic groups (gender, skin tone and age) remains equal across different camera angles, distances from camera, camera types and lighting conditions. As another example, you may want to know how well the model performs from the specific camera angles that Production Pro customers most commonly use.

这个问题有很多可能的答案。例如,您可能想检查模型在不同人口统计群体(性别、肤色和年龄)中的相同表现是否在不同相机角度、与相机的距离、相机类型和照明条件下保持相同。再举一个例子,您可能想知道模型在 Production Pro 客户最常使用的特定相机角度下的表现如何。

Conclusion

结论

Congratulations! You have reached the end of the AI Ethics course. Once you have viewed all of the tutorials and successfully answered the questions in the exercises, you will receive a completion certificate to celebrate your hard work.

恭喜!您已完成 AI 伦理课程。一旦您查看了所有教程并成功回答了练习中的问题,您将获得结业证书以庆祝您的辛勤工作。

If you enjoyed the AI Ethics course, we recommend the Machine Learning Explainability course as a next step. To prepare for the content in this course, you should have some background in python (which you can get from the Python course) and machine learning (which you can get from the Intro to Machine Learning course).

如果您喜欢 AI 伦理课程,我们建议您下一步学习 机器学习可解释性 课程。为了准备本课程的内容,您应该具备一些 Python 背景知识(您可以从 Python 课程中获得)和机器学习背景知识(您可以从 机器学习简介 课程中获得)。


05.exercise-model-cards【练习:模型卡】

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top