WebKit 贡献者会议 2022¶ WebKit 贡献者会议 2022 年 11 月 9-10 日 日程¶ 周三会议¶ 演讲 Eric Meyer：WebKit 2022-2023 Don Olmstead：索尼 WebKit 2023 优先事项 Alex Christensen：站点隔离 Alan Baradlay：LFC/IFC 有哪些新功能 Nikolas Zimmerman：新的基于图层的 SVG 引擎状态 周四会议¶ 演讲 Anne van Kesteren：标准立场 Apple WebKit 目标（已取消） Youenn Fablet：WebCodecs Jonathan Bedard：WebKit SCM 现状 Jani Hautakangas：面向 Android 的 WPE WebKit Basuke Suzuki：JavaScript 预编译字节码评估 Mikhail R. Gadelha：JSC 上的 clang-tidy Jen Simmons：互操作性 2022/2023 Patrick Angle：WebDriver Simon Fraser：将 GPU 进程引入 macOS Jon Davis：改善贡献者入门体验 演讲¶ 周三¶ WebKit 2022-2023 (Igalia)¶ 备注¶ Igalia 是一家拥有 128 名员工的分布式公司，专注于开源 8 个主要领域，从浏览器到云计算。专注于所有使用网络的设备和器具 今年投资了 Wolvic VR 浏览器 主要还是专注于 Linux 和 Android 设备 回顾 2022 年 2021 年，Igalia 的 WebKit 贡献约占 16%，Apple 占 77%，Sony 占 4% 2022 年，Igalia 约占 13%，Apple 占 79%，Sony 占 3% 但其他贡献者的比例从 2% 上升到 4%，所以个人贡献者更多了 Igalia 的许多贡献来自 WPE WebKit 在开发分支上启用了 Angle 开始研究 GPU 进程和 WebGPU，需要一些缓冲区共享技术才能工作 重构滚动代码，使 WPE 和 GTK 中的滚动更流畅 动画帧垂直同步分析 WebRTC 的 GStreamer 后端已上游 MSE 和 EME 的工作已上游 正在为 WebRTC 开发云游戏 Igalia 最大的工作领域之一是面向 Android 的 WPE 驱动因素是如此多的嵌入式设备运行 Android 完成了部分 API 的 Android WebView 兼容性、媒体播放的硬件加速（仅解码，不编码）、PSON（对安全性很重要）、全屏支持、64 位 ARM 目标支持，以及大量重构 JavaScriptCore 工作：离线程编译，32 位平台相关工作，包括 WASM 信号内存 Web 平台工作：GamePad API（用于需要支持嵌入式设备游戏的客户）、HTML 交互式表单验证、基于图层的 SVG 引擎、WebSpeech API（仍在进行中）、:focus-visible、ARIA 属性反射 QA 工作：2 个新的 Ubuntu 22 机器人，用于使用 clang 构建的新机器人，树莓派机器人 互操作性 2022（专注于改进互操作性的跨浏览器项目）：截至 11 月 7 日星期一，Safari 处于领先地位（年初时垫底）。所有浏览器在这一年中都取得了显著进步，从约 60% 提高到约 80%。许多贡献不仅仅来自 Igalia 互操作性 2022 工作：级联层、表单元素、滚动、子网格测试 转向 2023 年计划 面向 Android 的 WPE：保持与最新 WPE WebKit 的同步（应该很简单）；实现缺失的 API；支持 WebDriver、WebInspector、WebXR；错误修复和性能改进（对嵌入式设备，尤其是低能耗设备很重要） 图形：2D 渲染性能改进（客户要求，他们希望在触摸屏上获得流畅的界面，在设置过程中显示视频等）；WebGL2 支持；基于 ANGLE 的 GPU 进程；修复影响 WPE 在基准测试中正确结果的问题；Vulkan 和 WebGPU 支持的投机性计划 多媒体：WebRTC GStreamer 后端；改进 MSE 逐出算法使其更具侵略性；视频画中画 (PIP) 支持（取决于 GPU 进程的进展）；使用 playbin3 进行媒体播放 JavaScriptCore：支持 32 位 FTL 层；支持 32 位 LOL JIT 层；将 JSC 移植到 RISC-V；调查 ARM64e 上 PAC 失败的运行时 sanitizer；时间 API 支持——正在制定跨浏览器规范 12:30 PM Web 平台：将基于图层的 SVG 引擎上游（历史上，SVG 和 MathML 都有自己的渲染引擎，希望与 HTML 统一）；GTK + WPE 的 WebSpeech API 和 ImageBitmap（对嵌入式设备有用）；WPE 端口的 DeviceOrientation（对手持设备有益）；WebXR 实现（取决于测试基础设施） 全面支持 GTK4 上的辅助功能 (a11y); HTTP/3; WebExtensions API QA：改进对 ARMv7 和 ARM64 上的 Flatpak 的支持；OSS-Fuzz 中的 WebKitGTK/WPE；WebKitSearch 构建机器人；WPE 安全机器人 问答¶ Geoff 提问：关于基于图层的 SVG 引擎，是什么让 Igalia 对这项工作感兴趣？ Brian 回答：我们为此做了一个播客。KSVG 的原创者是 Igalia 人，他们了解所有的缺点。我会在 Slack 中分享播客链接。这对嵌入式设备的效率很重要，并且能够实现否则不可能实现的功能 https://www.igalia.com/chats/Igalia-Chats-Niko-SVG-WPE Niko 回答：你很快就会听到更多。Igalia 有兴趣减少渲染时间，尤其是在低端 CPU 上。我们正在软件上进行渲染，所以如果你能进行更多硬件加速，你就会成功，尤其是在嵌入式设备上 Tim 提问：Igalia 在 content-visibility 和类似属性方面的活动。为什么 Igalia 会感兴趣？ Brian 回答：我没有答案说明为什么它有趣。它就是有趣：:slightly_smiling_face Cathie 回答：`content-visibility` 有利于性能，并提供了多种方式让开发者决定哪些部分优先渲染或跳过渲染 Jonathan 提问：我们如何确保不会破坏其他平台的构建？ Simon 提问：WebXR 和 GPU 进程结合起来怎么样？我们 Apple 认为一些 WebXR 代码必须在 GPU 进程中运行（编辑过） Brian 回答：我觉得这听起来很有道理 Alex 回答：仍处于早期阶段，所以不确定，但是的，我们想尝试在 GPU 进程中完成它（编辑过） 索尼 WebKit 2023 优先事项¶ 备注¶ PS5 发布两周年 将 WebKit 用于媒体应用 游戏开发者将其用作库 计划涵盖 1-5 年 WinCairo 端口 移除 WebKitLegacy 支持 受测试阻碍 需要更新测试配置 网络后端 为 cURL 后端添加 HTTP/3 支持 cURL 内部支持此项的动向 尚未完全正常工作 已完成一些初步工作，例如在 Web Inspector 中显示请求 正在研究 cURL WebSockets API JavaScriptCore 旨在减少启动时间 利用字节码 将在未来的演示中扩展 Temporal API Record & Tuple WPE PlayStation 后端 与 Igalia 的联合努力 正在研究使用 WPE 渲染器 媒体支持 游戏手柄支持 媒体 历史上 PlayStation 特定的代码未上游，有兴趣改进这一点 MSE 关于 Web Inspector 中“媒体”选项卡的想法 GPU 进程 称为“Web Compositor” 异步合成 异步滚动 视频渲染 希望为 PlayStation 启用，目前已在 Windows 构建中启用 WebGPU 使 Dawn + WebGPU 正常工作 在 PlayStation 硬件上移植 Dawn Dawn 是 Google 的 ANGLE 替代品 现代 2D 渲染 修复 TextureMapper 错误 尝试使用 WebGPU API 来实现 TextureMapper 与 GL 不同 速度提升 问答¶ Alex 提问：关于 WebGPU 原型与 Cairo 端口的关系？ 答：正在试验中，不一定是最终计划。 站点隔离¶ 站点隔离是使浏览器更安全的下一步，其实现是重大的架构转变。 备注¶ 过去：单进程浏览器 / WebKitLegacy 稳定性：一个标签页崩溃，所有都崩溃 安全性：无沙盒 性能：一个线程应该足以运行 10 个网站 响应性：那个线程永远不会挂起，对吧？ iOS 1.0 引入了 WebThread 当前多进程架构 / WebKit2 多个严格沙盒化的网页内容进程 网络/存储进程 GPU 进程 UI 进程 分区存储 IndexedDB / 缓存 / 其他持久存储按源进行分区 如果 a.com 和 b.com 都嵌入 example.com，那么 example.com 的缓存数据将分别存储在 a.com 和 b.com 的分区中 为什么需要站点隔离？ Meltdown/Spectre 漏洞：JS 现在可以读取任意数据 部分缓解措施 Cross-Origin-Embedder-Policy, Cross-Origin-OpenerPolicy 禁用 SharedArrayBuffer 和 performance.now 的精度 漏洞利用读/写/执行小工具可以发送恶意消息；它能达到什么目的？ 数据隔离：已在开源中 在允许访问 Cookie 之前检查 firstPartyForCookies 是否合理 window.open 仍然可以将多个第一方置于同一进程中 站点隔离 将跨源 iframe 放在与其父级不同的进程中 每个进程只能读取一个数据分区；理想情况下，该分区永远不会是消息参数 Spectre 攻击无法读取任何有趣的内容 新的跨源 iframe 加载流程 decidePolicyForNavigationAction - 告诉旧进程导航很可能在新进程中继续 当导航提交时，告诉旧进程已发生，在帧树中从 LocalFrame 切换到 RemoteFrame 当新进程中加载完成时，告诉旧进程已发生，然后可以触发 on-load 使所有 FrameTree 遍历使用 IPC 来拉取或推送信息 其他需要考虑的事项 屏幕绘制/合成 可访问性树需要了解进程外 iframe 性能 - 进程不是免费的 Web 检查器 - 需要能够检查进程外状态 带有跨源 iframe 的打印 任何与帧相关的内容，必须至少审计 200 个帧使用案例 注入包：许多 API 不适用于进程外 iframe InjectedBundlePagePolicyClient：允许您在不询问 UI 客户端的情况下进行导航 WKBundleFrameCopyChildFrames：API 必须被移除或更改，在隔离环境中无法以相同方式返回所有帧的副本 问答¶ Jean-Yves 提问：完成这项工作的预计时间表是多久？过去的经验表明这将需要很长时间才能完成。 答：我们知道这是一项巨大的努力，将需要一年以上的时间。 Cameron 提问：其他浏览器已决定在移动设备上，由于性能限制，为每个源设置一个进程不切实际。WebKit 会受到同样的限制吗？ 答：我们知道这一点，并且我们将不得不测量开销以帮助做出决策。我们仍处于早期阶段，只是在努力让某些东西能够运行。（编辑过） Brian Kardell 提问：WebKit 是首批实现分区存储的浏览器之一。在该功能引入期间，是否出现了许多问题？ Alex 回答：Web 标准中有可用的跨源通信通道。我们决定不让 WebKit 中的跨源跟踪变得容易，这是出于隐私目的，尽管这种跟踪有一些合法的用途。 John 回答：分区存储的隐私边界是站点而不是源。最初在实现存储分区时，我们也想对 Cookie 进行分区，但破坏性太大。直到 2017 年才实现分区 Cookie，直到今年才有一个 Web 标准来解决这个问题。 LFC 有哪些新功能¶ 今年添加到 LFC 的内联和弹性布局功能——也即 WebKit 行布局的进展程度。 备注¶ LFC，快速回顾 布局格式化上下文 重新实现和重新设计 WebKit 渲染引擎的倡议（类似于 Blink 的 layout ng 工作） 专注于内联格式化上下文和弹性格式化上下文 IFC Webkit 维护着 2 个相互独立的内联布局实现（传统和现代） 传统模式始终启用，现代模式则通过运行时标志控制 现在已在 Safari 中启用多年 当两者都启用时，我们必须在布局期间做出某个决定，是使用现代模式还是传统模式 决定在块级别做出，以确定 IFC 是否可以处理内容 示例 第二个块容器必须使用传统模式，因为该块具有 IFC 不支持的属性 截至 10 月 8 日主干分支，有多少内容通过 IFC 一些文本量大的页面 100% 使用 IFC，完全不使用传统模式 其他一些页面，覆盖率可能会降至 50% 以下 line-clamp 是导致百分比下降的一个“罪魁祸首” IFC，进展 在禁用 IFC 的 WPT 中有许多测试失败 即使是一些旧的 bug 报告也在启用 IFC 后得到修复 IFC，扩展覆盖范围，新功能 继续向 IFC 添加新功能，使其达到与传统模式相同的水平，以便我们能够移除传统模式 使用 IFC 处理双向内容取得了许多进展 在处理内联布局时，您不必关心行方向，因为一切都使用逻辑尺寸 传统上，WebKit 有一个首选宽度代码路径，它基本上会运行两次布局 这两个代码路径可能返回不同的值，从而导致意外的溢出或换行 新的浮动布局应该更容易理解和维护 FFC，新的弹性布局实现 基于 LFC 设计原则重写弹性布局 目前未启用，因为我们需要进行集成工作 需要方法与现有渲染树架构集成 新的基于图层的 SVG 引擎状态¶ 详细了解 WebKit 中基于图层的 SVG 引擎的当前状态。 备注¶ 总部设在德国——2001 年开始为 KHTML 贡献代码，与 Rob Buis（也是 Igalia 成员）共同创立了 ksvg，并随着它进入 webkit 而继续发展 一段时间以来一直专门从事 webkit 工作 LBSE 是什么？自 2019 年以来，Igalia 在 WebKit 中开发的基于布局的 SVG 引擎 旨在解决存在 15 年以上的架构问题 允许我们使用硬件加速和有用的 CSS 功能 2021 年 10 月的 POC 补丁通过了所有现有测试，像素完美 性能与 motion mark 中的传统引擎相当——有些测试快得多，有些则慢一些……但原则上我们证明了这是可能的，并且不是性能问题 如何实现？在重新设计中协调系统决策与 CSS，在 RenderLayer 中尽可能多地重用代码，而无需进行 SVG 特定更改 自 2021 年以来的演变：“最终”版本的原型是旧 SVG 引擎的直接替代品，但它不是一种上游化方式——这将是一个巨大的补丁。这是要么全有要么全无，无法合并部分——一次性更改太大 我们制定了一个上游计划，使其发展并以原子化方式提交各个部分，所有代码都通过编译时标志控制 不幸的是，这意味着大量的手动返工，基本上相当于又一次重写……但是.. 有一个主错误报告 90738“协调 HTML 和 SVG 渲染” 89 个补丁已提交，75% 已上游 我们去年 11 月开始添加运行时标志 直到今年 1 月，我们才开始打下真正的基础 在此过程中，修复了一些有趣且重要的上游错误——其中一些已存在十年之久，例如透视不应受 transform-origin 影响 到 4 月底，我们开始处理 SVG 文本 然后我们添加了所有基本形状，接着是 viewbox，然后是合成——这还修复了一个关于通过 CSS 变换放大对象的 10 多年旧的错误 (这让我们度过了七月) 8 月，我们启用了对“defs”和外部对象以及图像元素的支持，并启用了剩余的形状（折线、多边形、线） 9 月，我们修复了一些关于合成、设备像素对齐和元素合成时像素对齐的重大错误 我们为渲染树转储启用了亚像素精度 最近（十月）关于规模的协商 + RenderSVGRoot, fixed SVGImage container size propagations , assured HTML documents create a new formatting context - also fixes a 15+ year old bug! panning and zooming was enabled, there is 1 important interop deviation we need to look into. we fixed transform support for svg elements so… where do we stand now? when we said 75%, most of the core layout is upstreamed… All the render layer, etc. So, roughly 95% of it is done - there is a patch pending review to fix wrong answers to some DOM APIs - will fix a number of issues once this is bug we can go back to the list of open bugs, we are missing patterns, gradients, masking, I think we will have a patch in the next few weeks - some of this is solved downstream that is the short term list - but the long term plan involves more performance work and making sure that all of the layout test pass There are optimization that we can do to make it faster, we will be dealing with adding those to reduce the overhead and only construct layers if necessary and show that it is not then a massive overhead while we said we are 75% completed, we believe that we can hit the goal in 2023 and show it complete and fast (showing some video/demos) (key here is looking at cpu use and timing is really improved) (room gasps audibly, but they are all muted) (shows smooth 3d transformation possible with lbse vs vanilla safari with good performance and even panning and zooming while it is happening) (shows many individual transforms on individual parts in 3d) (there is no flashing or flickering, it’s very smooth) (shows the layers - wow there are a lot) (242 layers…. you can inspect them to see which parts.. and we are now enabling the compositing / repaint and you can see that we are not increasing the painting count for these objects. You can see that sometimes the count changes because we occasionally do need to repaint, but it is only on a small portion of the image) (comparing to the old implementation - there are 0 layers, we are just repainting the whole document every time - and it is less capable too) we need to find a tradeoff and conditions which just traverse the render tree and children and don’t create all the layers - to reduce the layers when they are not necessary - and then we can really have the best of all worlds There is another demo, but I ran long so I will just share the link and you can watch it yourself Q&A¶ Q from cam “how sure are you that these optimizations will improve performance” A I have done some things and I have high expectations that it will greatly improve - one for example was real bad - 25% regression and this makes a huge difference if you do that Q from cam - (something about pixel precision that are scaled up and down) A I shared some things just today that are pixel tests and it surprised me that it is legacy that actually had more problems. we correct for offsets on painting and we actively take it into account. All of the basic stuff is working, but there may still be some issues Q from Ahmad “before we roll this out, is there a plan to put it into tech preview where it is easier for users to test” A Yes, we would love that. I was approached about this a few weeks ago, yes, we would love any help at all to help with these Ahmad “It would definitely help me close some old bugs” Q from Said “I went through a lot of patches trying to understand the new changes, but I couldn’t understand what you are trying to do. My question is is there a way you can write some documentation about the changes and the difference between them” A Yes, indeed - I published a really lengthy post about it before we started upstreaming. It is out of date, but yes I think that kind of document is really useful. I agree I’m happy to do that, I’ve been kind of working on this in isolation for a few years which isn’t my preference, but that kind of thing is what happens - I am really happy when people are reviewing and comparing and asking these questions. What I was trying to do here was trying to break down these commits into reviewable chunks, I hope that we are now at a point where all of the readme notes to help the planning are gone and we can do that Said yes, I’m not asking for an explanation of every line, just the interesting changes Q from said about foreign object - is there a way to enable this only for that initially, we need that A yes, I had a similar idea even Thursday¶ Standards Positions¶ Learn about WebKit’s standards-positions repository and how you can help shape the web platform through standards. Notes¶ Anne, working on web standards for ~18 years. Brief overview of current standards positions. Standards positions: WebKit's current position on standards Positions are not guarantees on what WebKit will implement They are a simple/effective way to share the web Everyone should feel empowered to speak up and outline their thoughts on standards proposals Please add labels in the positions repo When a position is decided, please give additional time to allow others to provide their thoughts/comments/feedback (at least 1-2 weeks) especially around holidays. When should a position be declared? WebKit is asked for a position, an issue is opened, and there is discussion within the WebKit community about whether WebKit supports (or not) the proposal To what extent does "imminent implementation" affect positions? It doesn't really relate to implementations, but we may have a position on a proposal that we are implementing - but there may be situations where we oppose a proposal but we implement it anyway. "Standard" - Should we make a meaningful distinction between "proposals" and "standards"? The repository handles both standards and proposals, but the name of the repo is a little confusing. Is this the place where WebKit can announce its position on a proposal? Yes This should not be "Apple's" position, it should be Webkit's - positions as a whole Ideally, this repository will help reduce the opaqueness of why features are not available in a WebKit-based browser (like Safari), and it's not necessarily that "Apple hates the feature and silently will never implement it" WebCodecs¶ The ongoing WebKit work towards WebCodecs support. Notes¶ Recent work implementing video hardware codecs WebCodecs is a new API. In development. Already in STP Interest from Zoom because Zoom is not compatible with WebRTC Also interest from web apps for video editing WebCodecs also useful for computer vision applications Why use WebCodecs vs existing APIs? Existing API designed to make playing media easy for web developers. Hands off. WebCodecs is lower level. More knobs for specific use cases Based on Decoder/Encoder VideoFrame is a central object for API API objects are thin wrappers for native OS objects for performance This makes it harder to use potentially Need to close VideoFrame objects manually, should not rely on GC Closing VideoFrame directly closes all references. Use ReadableStream to share VideoFrames VP8 implemented in software All software codecs implemented in web process H.264 implemented in hardware All hardware codecs implemented in GPU process Frames and codecs may reside in either web process or gpu process so may involve IPC macOS/iOS ports - implementation almost finished VideoFrame cannot be sent out of process Potential next steps: more GPU optimizations (e.g. software encoders), support in more ports, more codecs (HEVC, ImageDecoder, Audio), integrating with existing APIs (OffscreenCanvas, WebGPU, ReadableStream/MediaStreamTrack transform) Q&A¶ Q from Michael Catanzaro: could be easier for compliance/legal if more clarity around when/where codecs are implemented A: WebCodecs only exposes what platform already supports. All infra is abstract. All Cocoa specific for GPU codecs Follow up from Michael Catanzaro: Presence of code in repo anywhere can sometimes be legal issue. Red Hat: hardware decoders can be illegal to expose based on licensing (edited) Q from Simon Fraser: Does everything apply to both iOS/macOS? Work with IOKit blocking? A: May have different impl. wrt canvas. Should work but may need to make some changes wrt IOKit blocking IOSurfaces State of WebKit SCM¶ A short overview of how we’re managing our repository in GitHub and what we’re looking to work on this year. Notes¶ Followup to last year's GitHub migration talk What have we done Moved the repository to GitHub Decomissioning svn.webkit.org not moving all subversion branches Deleted ChangeLogs Migrated to identifiers Integrated into commits Used in bugs Quite effective for bisection Pull requests Better review interface Before, patch review was not compatible with every type of change, now it is ~250 PRs a week Early Warning System (EWS) Supports GitHub PRs Part of making EWS support GitHub, means that branches are supported too Security Pull Requests Supported through WebKit/WebKit-security Big difference between security PRs and security bugs: Security PRs are available to everyone in the security group, bug access is more limited Complications with landing to make it hard to make mistakes Use -v flag on git-webkit to get info about setup git-webkit pr --remote security to create Security Pull Request Git and GitHub can't secure specific branches, but can secure entire repositories Hence, WebKit-security is a mirror WebKit/metadata/git_config_extension contains project configuration settings git-webkit has the idea of source remotes, there can be any number of them Can be used to create remotes only visible to a particular company Tooling also supports bitbucket Faster EWS Reducing retries by consulting results.webkit.org If a test fails in EWS, check if the test is currently failing on main, if it is, ignore failure Somewhat dangerous, does allow us to pile on bugs for a specific test failure (e.g. flaky failure becomes a full failure) Patch Review Haven't deprecated patch review, no immediate plans to Would deprecate if it became difficult to maintain Tied closely to Bugzilla Bugzilla Desire to migrate to GitHub issues Don't think the project is ready to move to GitHub issues Concern is that we will get an influx of GitHub issues, not able to triage Ongoing discussions Multi-commit Pull Requests Atomic commits are really important Each commit needs to independently revertible Auto squashing is hard, how to get the commit message? Merge commits are hard to understand Commit Signing Would require merge commits Would make merge-queue more complicated Documentation Intention to wind down the Subversion Wiki Candidate replacements: GitHub Wiki, GitHub Pages, automatically deployed website Leaning towards automatically deployed website – driven by Brandon Stewart Q&A¶ Q from Basuke Suzuki: Bugzilla is low priority, can you add automatic comments back to the Bugzilla? (edited) Jonathan: How are you uploading your PRs? It should make a comment. Basuke: I do it manually Jonathan: Use the script to get automatic comments (edited) Q from calvaris: Is there a reason why you cannot install ccache on the bots? Jonathan: I am not sure, not super familiar with ccache. Is that what helps us switch branches? calvaris: It's a cache for compilation. Speeds up compilation. Can potentially reduce time to land a patch. Jonathan: Need to talk to Elliot Williams about build speed improvements Jonathan: merge-queue builds on tip-of-tree, bot is picking up more content than your change Jonathan: Have had a lot of build / performance issues, are looking into them. Will see if ccache is being investigated Q from Ahmad: Should modifications to contributors.json or other small contributions trigger EWS? Jonathan: Need to do work on this, can be more intelligent Jonathan: There are some checks to make sure a change applies to an area before running EWS Jonathan: Changes may affect more places that you expect Jonathan: If you're making changes to Tools/Scripts you might think it doesn't affect EWS, but it does Jonathan: We should be smarter, talk to Aakash and I Aakash: Agree with Jonathan Q from Ahmad: Is bugs.webkit.org a good place to raise issues with EWS / GitHub tooling? Jonathan: Yes – Slack Aakash or I to get a quicker response to bugs, consider importing into radar (edited) WPE WebKit for Android¶ WPE Android: Improving the Web landscape in Android devices. Quick 5-min introduction to Igalia’s WPE on Android effort. Notes¶ WPE android started as research project in 2017, in development since, goal is to provide a WebView on Android that uses WPE as an engine. API is designed to line up with Android System WebView to make development easier. No need to introduce new WebKit port - uses existing public API. Supports multiple architectures, integrates into the Android main loop, requires an Android-specific SharedMemory implementation. Supports hardware accelerated media playback, fullscreen, cookies. At some point in the future, support for WebDriver and WebInspector is planned. Also planned is being able to provide WPEView as the default WebView on Android, integrating with the existing Android API Demo shows WebKit WPE view working on Samsung S20 FE, including user-agent and WebGL support. Very cool! (discussion and questions taking place on Slack: https://webkit.slack.com/archives/C01B9F8N0JZ/p1668101419122349?thread_ts=1668101280.148769&cid=C01B9F8N0JZ) JavaScript Precompiled Bytecode Evaluation¶ Investigation for the precompiled bytecode generation to minimize the cost of JS evaluation. Notes¶ Report on investigation about precompiled bytecode evaluation. This is just early investigation. We do not have anything yet, but we would like help if we get stuck so that's why we are reporting. Precompiled bytecode: We use JSC in our system apart from the web browser. Of course, source code is text, and source is stored somewhere and loaded and compiled each time. If we can achieve that cost at build time this would be big benefit. Not for the web, just JSC standalone context. Came from blog post on new bytecode format and how that it is cacheable and so this seemed promising. Apple implemented bytecode cache a few years ago. We can use --diskCachePath option with JSC. The first evaluation is the same, but it looks in the cache. This just uses disk and requires original source. Which is not our usecase. What is in the cache file? The entire source code is stored inside the file. Other information is stored, we say bytecode but it contains additional information. There are two timings for bytecode generation, before the execution and during the execution there are many modifications happening. What we did. The function metadata is not necessary as it will be generated and is large and our goal is boot time speed up. The large size of this causes high I/O cost which kills the benefit. Also, bytecode cache system is good but doesn't fit our purpose, extra I/O cost, first source is loaded then cache is checked. Added an option to jsc to generate the byte code file. From there, we need to accept this as source, there's a notion of SourceProvider which we used to parse the bytecode file. Added functions to get information from the cached information mostly in CachedTypes. Result: Performance is significant, 1/2 to 1/3 time for initial evaluation. Very first line throws exception so most time is compilation/prep time. For 900kb source time was 107ms->47ms, 6M file 365 ms-> 103ms. For file size, it contains original source but not too big. Extra cost (~20%) was acceptable for our use case. At first memory usage increased, at first didn't know why, but then found that source code and instruction stream is duplicated. That was the result, so we have more to do. We may need an API as the execute script takes a string. We could improve memory by using source and instruction stream in-place. May be able to use same compiler on same architecture. We believe this may be useful to others, so looking for help. I have many questions, so please give me feedback. Q&A¶ Q (Mark Lam): Regarding API, is the only way parsing string? . A: There's two, one for generation and one for taking it. Q: Can you use map the file? Around the memory footprint. A: It might be possible, Q: Does the bytecode format (instruction stream) lend itself into memory mapping? A: Current just allocates memory, but instruction stream should be able to. Q: On PC, what does architecture dependency? A: Endianness, can be different if compiler is changed, cache entry is copied into the file so the layout of the memory of the class must be the same (including alignment). (Ben Niham): I don't think this disk cache is in use for Safari but there might be some internal use case. There might be a private API for some case. (Basuke) Disk cache is also used in same boot session or if binary is rebuilt, it won't be used. (Mark Lam) We do have some internal cases. We don't use across boot sessions for security. (Basuke) For this I disabled that with fixed info. (Yusuke) Measured result looks following, would like to followup Mark's question around memory usage. Would suggest mmap from the file. Usually footprint doesn't count this memory. Some data would need to be regenerated, but some could be used fixed. Another thing is following up why why cache on disk, we don't have a precompiled file. Q (Basuke): After evaluation what is purpose of source code A? (Yusuke): If bytecode cache is not enabled, we need source code even if entirely done. We need a string right now for bytecode cache today, error handling case. I don't think this is mandatory for bytecode cache, we might need to do plumbing but it should be possible. (Basuke) Right now looks like main use-case is for generating hash for the code cache (Yusuke) If we are using a pre-generated bytecode cache file, right now it probably doesn't work, but if we do some work, I don't think there's a super large blocking issue that would avoid purging the source string. (Mark Lam) Function.toString is another case. Have you tried to apply this to the builtins as we build those into the binary. Q (Basuke) Entire instruction stream is generated at beginning? A (Mark Lam) No. (Yusuke) For one function, but not for inner functions for outer function. Current bytecode cache, every time each calling function build up bytecode. If we want to forcibly generate all instruction stream we need to do so. If you made something that forced generation of all functions then you wouldn't need the source, but we have different instruction stream for constructor or other function so each function could have 2 instruction streams when called normally or from new. Which might generate a lot of unused code. (Mark Lam) There's a JSC option for eagerly generate bytecode. (Yusuke) Given the result, the hybrid result seems to be good. If previous run didn't generate a function then generate it from source will probably work. If eager generates too much code. (Alex) In a similar case, I store bytecode and source so that if I change the bytecode it can generate it for a reason like that. (Basuke) This may help for cases on upgrading. (Mark Lam) - The inspector will cause throwing away of bytecode. clang-tidy on JSC¶ What worked and what didn’t when applying clang-tidy to JSC, and starting a broader discussion on using it along with other tools in JSC. Notes¶ This talk will cover what worked and what didn't in my experiment, but I hope to start a discussion on using clang-tidy on JSC. clang-tidy is an extensible framework to detect programmer errors on the source code. it can tell you about things to fix/improve, and even do the changes for you automatically. I did run a few passes related to improved readability and modernization on the JSC/WTF/bmalloc source code. There were some small things I could not fix. For instance some constants are defined as macros, so could not be moved to move inits. There were some quirks with enums in the initializer list. Etc. I found some issues that the compiler did not warn us about, like float narrowing. It was a massive patch in the end, and only stayed upstream for 3 weeks 😞 Some bots broke: clang GTK debug builds, a SIGILL during WPE startup, watchOS regressions. Q&A¶ Questions: which checks should be used? Smaller patches and different PRs? CI integration? Run clang-tidy automatically every week/month/whatever? (Mark) Instead of landing a mega patch land different patches. Land non-controversial things first. The rest can go after that. (Mark) On CI. clang-tidy should not be turned on automatically unless we are 100% sure that it won't introduce bugs. (Jonathan) I want to second the smaller changes comment. That way it's less of a big deal to revert it. Especially for watchOS, where good testing CI is difficult to do and will remain difficult to do. (Jonathan) If we are concerned about clang-tidy generating bugs, maybe we should do it locally with PR upload. The same way than style checker works? But let developers override it if the suggestions clang-tidy does happen to be wrong. (Nikolas) I'd say the bug is not really clang-tidy's, it was a compiler bug. The one that happened in WPE. (Nikolas) I agree about splitting the mega patch in smaller patches too, otherwise it's very hard to dissect. (Don) I also agree with smaller patches. Also a comment: if something in WTF is not used by JSC then it won't be caught by clang-tidy. Maybe we could create a dummy file that includes every header, to force us to go through everything. These sanitizers helps us to improve our tooling in general. We could create special rules for "WebKit-isms". (edited) Interop 2022 + 2023¶ A look at Interop 2022, how it’s scored and how to help. And at what’s being planned for Interop 2023. Notes¶ Interop is part of the web-platform-tests (WPT) project Interop '22 launched in March '22, with participants from Google, Mozilla, Microsoft, Apple, Igalia, Bocoup. Scores were mid-70s in March for all browsers. Across the year, things have slowly progressed over the year, implementing features and fixing bugs. The goal, of course, is to make it possible to use these features across all browsers — make them all interoperable. Scores in experimental browsers are 85–91, we're leading unlike Jan. "Investigation" areas are research areas for "where the specs, tests, infrastructure, etc. are in a palce where we can make these features interoperable" Scores for investigation areas are reported back by the investigation groups, based on their instinct. Overall investigation is ~32%, which brings the rest of the score down. 60% of the score is 2022 Focus Areas, 30% is 2021 Focus Areas, 10% is Investigation Areas. So what tests is this? So there are focus areas—[lists them]—most of these are CSS, but were chosen because this is where these are the ones with the largest impact for web developers. Clicking on focus area names takes you over to the results, run on WPT infrastructure. Can click through to results, and find the failures, and fix them in WebKit/WPT. Things are included in Interop by them being labelled; there are many more tests in WPT than there are in Interop. We can add other browsers—including WebKitGTK—and see the results of that too, and not just STP. The Interop project is being managed on the GitHub repo. If we go into the 2023 directory we can see a README that gives a lot more detail than I'm going to. In Sep–Oct, we got submissions for feature proposals, but that's now come and gone. We're now in the point where we're deciding what the positions of the six organizations are. And then make a determination from that. A single exclusion just excludes this. Will launch in January, scored somehow, which we keep discussing. Got a lot of proposals for 2023—83, after 10 in 2022, and 5 in 2021. Partly because the project has got a lot more attention now, and a lot more attention from web developers and a lot more proposals! Choosing proposals is partly a question of prioritizing, what things we think are important to developers. If you have opinions—if you were at Apple/Igalia people to people in your org, for everyone else speak to those at Apple (Jen, Tim, Sam), or Igalia (Eric, Brian) WebDriver¶ What is WebDriver, why does it matter, what have we improved this year, and what’s left to do? Notes¶ WebDriver is standardize remote interface for introspection and control of web browser Automate web browsers from separate process Safari implements this in 3 parts safaridriver, WebKit, and Safari Easy cross browsers testing. WPT.fyi WebDriver Results: Safari lags behind currently Resolved issues around dismissing user prompts Consistent session creation/deletion Auditing code against the specWhat's next? Actions is a key area Eliminate remaining time outs Out of order processing Q&A¶ Brent: I was curious about moving test infrastructure into WebDriver. Patrick: Anything is possible. Key problem is the reliability. Brent: What do other engine implementors do? Tim: Safari is closed source. Firefox and Chromium is open source. Brent: Do other browser use WebDriver or do they use something else? (edited) Tim: Firefox use WebDriver Alex: Chromium use WebDriver and other things Jonathan: Some test do things that we don't want others to do in WebDriver. Performance matters to us. Bringing the GPU Process to macOS¶ GPU Process status and plans for the next year. Notes¶ I'm going to talk about bring the GPU process to mac os goal of security Security is the only reason code that access devices, devices on the chip, moved to the gpu process GPU is primary, but also cameras and stuff tightens the sandbox one of the goals it to keep risky code in the Web Content Process Also consider image and pdf decoding to be risky compilated formats with old code that can be more easily exploited Sandboxing - what is it? it's a thing we use on Apple Platforms to limit the functionality of a process allows you to describe what a process is allowed to access. sandboxing is 'action at a distance' - at a level that is far below the API level this is fragile, we can block a call that we shouldn't be if someone changes code in another framework, that can cause issues. sandboxing is incredibly powerful, our strongest weapon in the security race this has been recognized in the industry - hexacon do the same that we did for iOS for macOS What have to moved? 2d canvas - two years ago Also media WebGL on iOS in moved this year and DOM rendering is also been moved Then we do do IOKit blocking, so we succeeded in doing that for iOS 16 We're working on that for this year the reason we did iOS first is because it was easier, we had an existing model that made it easier to do DOM rendering also mostly works, because 90% is shared with iOS there are small things like form controls that we need to do some custom work for mac webrx and webgpu are not ready yet there's a lot of interops between these, and all the code paths need to work we need to avoid bouncing between the GPU and the web process iOS already used UI-side compositing it's a confusing term when we say 'compositing' we're misusing the term it's not the thing that runs the shaders together we're creating the core animations layers in the UIprocess, not in the webprocess if we want to cut off IOKit, we can't use those in the WebContent process all IOS surfaces access has to be in the GPU process We can't use CA backing store in the web content process DOM rendering has to all be done in the GPU process This need to be transparent to most of the code in WebCore most of the code still uses graphics context, still uses image buffer there are things we need to do special things for images, pdfs and fonts We can encode the information about the filters to do the work in the gpu process web content creates image buffers, in the uiprocess model, we explicitly create image buffers we have a implementation that encodes into display list commands in the gpu process we have a concrete image buffer we transit the ioSurfaces through the webprocess in an opaque manner graphic context was made into a pure virtual class two years ago we also did some work with display lists that helped we subclass it as a display List Recorder, which just records the commands we subclass that as the remote proxy that knows how to replay those commands We do the same thing for images If DOM rendering is enabled, you'd get a graphics context to draw into, but it's really recoding and sending them to the GPU process under the hood in order to give us a context class that holds state related to the rendering, we have a proxy in the web content process These can IPC with each other that will manage a collection of image buffers that will interact with some remote image buffers per webpage context holds collections of object key by rendering resources identifier everything is referenced by a resource identifier we may get drawing commands that might reference images or fonts if we get a command to paint an image, the first thing we do, is we decode the image into a buffer, that image buffer is held in the image head, and the GUP process referenced that shared identifier same with fonts, we only want to run some of the content in the webprocess the fonts themselves, we share that data with the GPU process only runs the code that gets the glyf outlines and runs them Do the shaping code once, and then run it each time it's needed the scope of the GPU process, one per UIProcess using WebKit multiple webpages to share a GPU process each remote rendering backend runs separately a GPU process pre webpage is too much, so just one per UIProcess We already had remote layer remote rendering is the GPU process a lot of renaming is need to clean this up the main class that makes drawing work is in the UIprocess How does the rendering work for that particular platform? part of this work, the mac will start using RemoteLayerTreeDrawingArea, get this working for all the mac bits The page content consists of a set of tiles, we create additional later for animations each of those layers has a remote layer backing store one or more buffers that as iOSurface backed incremental painting each time the page changes the internals are different, but it's our way of hooking up between those two processes important that all updates come atomically to the process by changes, I mean any geometry changes everything comes in a transation and is applies in the UIprocss all in one go push the changes onto core animation layers makes the mac code more similar to iOS scrolling is different, on iOS, UIScrollView we use on the mac, we decided we didn't want to use app kit because it doesn't match as well we need support for subscrollers, we have to be able to support everything on the subscrollers, if you try and use an NSScrollView to do all of that it, doesn't really work as well for the mac, we will be bringing a lot of the existing code to the UIprocess we will need to do some work to have a scrolling thread in the UIProcess we need to implement pinch-zoom we can do it better with UI-side compositing this will be smoother in the UIProcess we also need to deal with color spaces we need to make sure that video full screen works, etc. Goal is to use IOKit blocking implication for clients who use injected bundles if you turn on IOKit blocking, all that code is affected trying to reduce and remove injected bundle code pdfs - we rely on PDF kit now, but might try something else image decoding - risky so run in the webprocess need to make sure that hardware image decoders work but we've turned them off for now, even though they are more performant testing - see if there's an impact on our benchmarks don't ship a regression in motion Mark we have to find optimizations to make up for our slowdown because of the IPC We also have layout tests with UISide compositing, this can get tricky the mac will tell us if things are getting blocked by IOKit layout test bot, running very early UISide compositing code lots of crashes currently all the impacts of turning this on how can you test this you can in minibrowers Enable DOM render, enable GPU for WebGL, use UI-Side compositing if you have all these, we will block IOKit, and see how it works basic stuff works, you can load webpages and click on links only main-frame scrolling works, nothing for subscrollers Q&A¶ Ujwal: are there any performance numbers on moving surface between processes Simon: this is not a performance hit, it's basically free. Get an put image data is slow content that does a lot of get and put image data can be slower also applies to webGL, getting the current error state getting images to the GPU process uses shared memory same with fonts a lot of our performance work was to do with throughput and latency you want to optimize throughput and reduce latency get enough data to the GPU process to allow it to start rendering you need to get things across the divide soon so that you can start executing in the GPU process quickly have the drawing commands be as small as possible is important get the data through as soon as you can get and get image data is the only really bottleneck but we've used tomse shard memory for that Nikolas: Wondering about several design decisions: Web rendering backend and remote layers more wondering about starting this on iOS you're not using NSScroll View implications for the future. what are your visions/plans if you could change all ports The non-UISide Compositing Simon: the goal is for WebCore code to not care what we do. Really the WebKit code is where we care about the differences between platforms What we'd like to do is we'd like to share more code in WKWebView with mac there's a legacy bit in WKView, but I do't know if we'll have time for that you can imaging a world where we can share all the viewport code in terms of legacy, we still have to support UIWebView and WebView, so those still work in the old way, so that won' t change soon the maintenance level is up in the UIWe we will be able to support both paths SVG doesn't need to care Nikolas: I'm not asking about that, it's equal if not worse in some thing. We often ran into issues for WPE that we couldn't cover with automatic testing something is broken in certain combinations We are in the process of thinking about GPU process for WPE we want to go for a model that has your support that you think it's safe for the future so we can maximize sharing code reducing breakage that's why I'm asking Simon: We did a good thing with Sharing the scrolling tree good position there about testing - with scrolling tree you can only scroll and pretend that the webprocess sis unresponsive you hav to do the correctly UIScriptController methods that helps us to do the UIProcess scroll you can do that in a way that you can do the UIProcess scroll there are quite a few test that can do this now that I think about that, they live in fast/scrolling/iOS part of the story here is we should make sure that the test work well there's more we could do with UIScript controller Improving Contributor Onramps¶ Improving Contributor Onramps Notes¶ Getting started in intimidating ... confusing ... overwhelming Ideas for improvement Rework our guides, make better use of existing resources. Audit "Getting Started" guide: Areas that are lacking or causing stumbling blocks? Got someone interested in helping with this. "Contributing Code" page and flow chart needs updating, in particular wrt git-based workflow (was svn). Also mention WPT in "Testing Contributions". Add "GoodFirstBug" keyword to bug reports, and link to our guides. Also link to guide from Slack channel descriptions&titles. Build new onramps. E.g.: New how-to-contribute videos. Code-along videos showing implementation, testing, and landing. Ok to show someone tripping up in the process, to reassure audience that they're not alone in struggling at times -- good encouragement! Build new, more focused, guides. E.g.: How to contribute to Web Inspector (so adding tooling when working on new web platform capability). Facilitate community onramps. E.g.: One-a-quarter office hours Q&A, invite Slack folks to scheduled events. Getting new contributors used to asking Qs to folks in the community.