I recently noticed the following message in Sentry’s pip installation step: Upon some investigation, I noticed that the package was not being installed. After making , I can now guarantee that our development environment installs it by default and it’s given us about 40–50% speed gain.wheelsome changes The screenshot above shows the steps from two different Github workflows; it installs…
Mozilla Release Engineering
https://planet.mozilla.org/releng/ · 54 posts · history since 2014 · active
18 Feb 2021
3 Feb 2021
Soon after Big Sur came out, I received my new work laptop. I decided to upgrade to it. Unfortunately, I quickly discovered that the Python set up needed for required some changes. Since it took me a bit of time to figure it out I decided to document it for anyone trying to solve the same problem.Sentry If you are…
19 Jan 2021
I’m happy to announce that at the end of 2020 I joined Sentry.io as their second Developer Productivity engineer \o/ I’m excited to say that it’s been a great fit and that I can make use of most of the knowledge I’ve gained in the last few years. I like the ambition of the company and that they like to…
31 Dec 2020
The summer of 2020 marked the end of 12 years of working for Mozilla. My career with Mozilla began with an internship during the summer of 2008 when I worked from Building K in 1981 Landings Drive, Mountain View, CA. Writing this post is hard since Mozilla was such a great place to work at, not only for its altruistic…
15 Jul 2020
If you use Treeherder on repositories that are not Try, you might have used the backfill action. The backfill action takes a selected task and schedules it nine pushes back from the selected task. You can see the original work and the follow up .herehere In the screenshot above you can see that the task mdaturned orange (implying that it…
3 Jul 2020
In I describe the feature. In this post I will explain how it came about.Filter Treeherder jobs by test or manifest path I want to highlight the process between a conversation and a deployed feature. Many times, it is an unseen part of the development process that can be useful for contributors and junior developers who are trying to grow…
At the beginning of this year we landed a new feature on Treeherder. This feature helps our users to filter jobs using test paths or manifest paths. This feature is useful for developers and code sheriffs because it permits them to determine whether or not a test that fails in one platform configuration also fails in other ones. Previously, this…
15 May 2020
In the last few months I’ve worked with contributors who wanted to be selected to work on Treeherder during this year’s . The initial proposal was to improve various Treeherder developer ergonomics (read: make Treeherder development easier). I’ve had three very active contributors that have helped to make a big difference (in alphabetical order): , and .Google Summer of CodeShubhamShubhankSuyash…
31 Jan 2020
For over a decade Mozilla has been using to publicly chat with anyone interested to join the community. Recently, we’ve launched a replacement for it by creating a . I will be focusing on simply documenting what the process looks like to join in as a community member (without an LDAP account/Mozilla email address). For the background of the process…
30 Jan 2020
Web performance issue — reoccurrence In June we discovered that Treeherder’s UI slowdowns were due to database slow downs (For full details you can read ). After a couple of months of investigations, we did various changes to the RDS set up. The changes that made the most significant impact were doubling the DB size to double our IOPS cap…
24 Oct 2019
The 3-minute-video Context Taskgraph, the tour Introduction to taskgraph What’s next? Further reading Need any help? Special thanks Comments See the , if it doesn’t display up here.video there It’s a fairly common practice to build and test every time someone makes a change in the code. In the industry, we call this process “Continuous Integration” (CI). Another good practice…
13 Jun 2017
The Release Engineering team fully-automated the publication of Firefox for Android in version 53.0. Let’s see what was already there and how things have changed since version 53.0. The initial publication was the first big step of the project. The next big one will be to expand the release workflow to simplify .what percentage of the user base is getting…
12 Jun 2017
The Release Engineering team fully-automated the publication of Firefox for Android in version 53.0. Let’s see what was already there and how things have changed since version 53.0. This part is more oriented to personal takeaways and a couple of questions that remain unanswered. A few checks done in pushapk_scriptworker are actually because of previous errors. The number of locales…
7 Jun 2017
The Release Engineering team fully-automated the publication of Firefox for Android in version 53.0. Let’s see what was already there and how things have changed since version 53.0. Mozilla, and more specifically the Release Engineering team, uses to implement the Firefox release workflow. The workflow can be summed up as:Taskcluster Each step is defined by its own set of tasks.…
6 Jun 2017
The Release Engineering team fully-automated the publication of Firefox for Android in version 53.0. Let’s see what was already there and how things have changed since version 53.0. This is true for desktop (Windows, Linux, Mac) and Android. However, we don’t ship that often to every user. We have different channels, receiving updates at different frequencies: You may have heard,…
17 May 2017
I moved my blog to WordPress. New location is here https://kimmoir.blog/
30 Sept 2016
The conference started off with Anna Lambert of Shopify welcoming everyone to the conference. The first speaker was Atlee Clark, Director of App and Developer relations at Shopify who discussed the wheel of diversity. The wheel of diversity is a way of mapping the characteristics that you're born with (age, gender, gender expression, race or ethnicity, national origin, mental/physical ability),…
29 Jul 2016
Last night, I attended my first Ottawa Python Authors Meetup. It was the first time that I had attended despite wanting to attend for a long time. (Mr. Releng also works with Python and thus every time there's a meetup, we discuss who gets to go and who gets to stay home and take care of little Releng. It depends…
I received this very kind email in my inbox this morning. "David Williams has expired your commit rights to the eclipse.platform.releng project. The reason for this change is: We have all known this day would come, but it does not make it any easier. It has taken me four years to accept that Kim is no longer helping us with…
3 Jun 2016
The The workshop will be held November 18, 2016 in Seattle. It will be held in conjunction with . (Foundations of Software Engineering ACM conference) If you've done something like We'd like to encourage people new to speaking to apply, as well as those from underrepresented groups in tech. We'd love to hear from some new voices and new companies…
2 Jun 2016
Last week I attended . It was my first time attending a DevOpsDays event and it was quite interesting. It was held at CBC's Glenn Gould studios which is a quick walk from the Toronto Island airport where I landed after an hour flight from Ottawa. This blog post is an overview of some of the talks at the conference.…
13 May 2016
We're delighted to have Francis Kang and Connor Sheehan join the Mozilla release engineering team as summer interns. Francis is studying at the University of Toronto while Connor attends McMaster University in Hamilton, Ontario. We'll have another intern (Anthony) join us later on in the summer who will be working from our San Francisco office. Francis and Connor will be…
7 Mar 2016
It was a busy week with many releases in flight, as well as preparation for running beta 1 with release promotion next week. We also are in the process of adding more capacity to certain test platform pools to lower wait times given all the new e10s tests that have been enabled. The releases calendar is getting busier as we…
29 Feb 2016
It was a busy week for release engineering as several team members travelled to the Vancouver office to sprint on the release promotion project. The goal of the release promotion project is to promote continuous integration builds to release channels, allowing us to ship releases much more quickly. Improve Release Pipeline: Improve CI Pipeline: Release: Operational: Chris, Jordan, Callek (remotely),…
8 Jan 2016
Before I begin this post a few caveats: I'm kind of a resume and interview nerd. I like helping friends fix their resumes and write amazing cover letters. In the past year I've helped a few (non-Mozilla) friends fix up their resumes, write cover letters, prepare for interviews as they search for new jobs. This post will discuss some things…
10 Dec 2015
You may have noticed that Windows has had no updates for Nightly for the last week or so. We’ve had a few issues with signing the binaries as part of moving from a SHA-1 certificate to SHA-2. This needs to be done because Windows won’t accept SHA-1 signed binaries from January 1 2016 (this is tracked in ).bug 1079858 Updates…
24 Nov 2015
November 13th, I attended the USENIX Release Engineering Summit in Washington, DC. This summit was along side the larger LISA conference at the same venue. Thanks to Dinah McNutt, Gareth Bowles, Chris Cooper, Dan Tehranian and John O'Duinn for organizing. I gave two talks at the summit. One was a long talk on how we have scaled our Android testing…
16 Nov 2015
The primary way to download Firefox is at , but Mozilla’s Release Engineering team has also maintained directories likewww.mozilla.org https://ftp.mozilla.org/pub/firefox/releases/latest/ to provide a stable location for scripted downloads. There are similar links for and . Read on to learn how these directories have changed, and how you can continue to download the latest releases.betasextended support releases for organisations Until recently…
23 Oct 2015
Due to with the new ftp server we’ve had to disable updates fora bug They’ll resume just as soon as we can get the fix landed. Updates are re-enabled, thanks to Mike Shal for the fix.Update (Oct 25th): Firefox for Android Nightly Firefox for Android Aurora
21 Oct 2015
Today we started serving an important set of directories on ftp.mozilla.org using Amazon S3, more details on that over in . Some configuration changes landed in the tree to make that happen.the newsgroups Please rebase your try pushes to use revision or later, currently on mozilla-inbound. Otherwise your builds will fail to upload, which means they won’t run any tests.…
25 Sept 2015
In September, Mozilla release engineering started experiencing high pending counts on our test pools, notably Windows, but also Linux (and consequently Android). High pending counts mean that there are thousands of jobs queued to run on the machines that are busy running other jobs. The time developers have to wait for their test results is longer than ideal. Usually, pending…
17 Jul 2015
This spring, I took several online courses on the topic of data science. I became interested in expanding my skills in this area because as release engineers, we . I wanted to learn new tools to extract useful information from the . This xckd reminded me of the challenges of managing our buildfarm somedays :-) I took three courses from…
16 Jun 2015
In an , I wrote how we had reduced the amount of test jobs that run on two branches to allow us to scale our infrastructure more effectively. We run the tests that historically identify regressions more often. The ones that don't, we skip on every Nth push. We now have data on how this reduced the number of jobs…
12 Jun 2015
Here's May 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an or as a . The number of pushes decreased from those recorded in the previous month (8894) with a total of 8363. HTML page json file Trends Highlights General Remarks Records 8363 pushes 270 pushes/day (average) Highest number of pushes/day:…
1 May 2015
Here's April 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an or as a . The number of pushes decreased from those recorded in the previous month with a total of 8894. This is due to the fact that gaia-try is managed by and thus these jobs don't appear in the…
28 Apr 2015
Releng 2015 will take place in concert with ICSE in Florence, Italy on May 19, 2015. The program is now . Register available here! by ©, Creative Commons by-nc-sa 2.0 via romana in firenze pinomoscato
Running a large continuous integration farm forces you to deal with many dynamic inputs coupled with capacity constraints. The . People add more tests. We build and test on a new platform. If the number of machines available remains static, the computing time associated with a single push will increase. You can scale this for platforms that you build and…
21 Apr 2015
ftp.mozilla.org has been around for a long time in the world of Mozilla, dating back to original source release in 1998. Originally it was a single server, but it’s grown into a cluster storing more than 60TB of data, and serving more than a gigabit/s in traffic. Many projects store their files there, and there must be a wide range…
15 Apr 2015
Here's March 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an or as a . The number of pushes increased from those recorded in the previous month with a total of 10943. HTML page json file Trends Highlights General Remarks Records 10943 pushes 353 pushes/day (average) Highest number of pushes/day: 579…
20 Mar 2015
We migrated most of our Mac OS X 10.8 (Mountain Lion) test machines to 10.10.2 (Yosemite) this quarter. This project had two major constraints: 1) Use the existing hardware pool (~100 r5 mac minis) 2) Keep wait times sane. (The machines are constantly running tests most of the day due to the distributed nature of the Mozilla community and this…
17 Mar 2015
Here's February's 2015 monthly analysis of the pushes to our Mozilla development trees. You can load the data as or as a .Although February is a shorter month, the number of pushes were close to those recorded in the previous month. We had a higher average number of daily pushes (358) than in January (348). 10015 pushes358 pushes/day (average)Highest number…
25 Feb 2015
The was published yesterday. (Download ). This issue focuses on the current state of release engineering, from both an industry and research perspective. Lots of exciting work happening in this field! I'm interviewed in the , along with of Facebook and of Google. Interesting discussions on the current state of release engineering at organizations that scale large number of builds…
23 Feb 2015
FileMerge is a nice diff and merge tool for OS X, and I use it a lot for larger code reviews where lots of context is helpful. It also supports intra-line diff, which .comes in pretty handy However in recent releases, at least in v2.8 which comes as part of XCode 6.1, it assumes you want to be merging and…
13 Feb 2015
Here's January 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an or as a . We're back to regular volume after the holidays. Also, it's really cold outside in some parts of the of the Mozilla world. Maybe committing code > going outside. 10798 pushes 348 pushes/day (average) Highest number of…
21 Jan 2015
Just a reminder that submissions for the Releng 2015 conference are due this Friday, January 23. It will be held on . If you've done recent work like we'd love to hear from you. Please consider submitting a talk!In addition, if you have colleagues that work in this space that might have interesting topics to discuss at this workshop, please…
16 Jan 2015
Mozilla’s Release Engineering team has been through several major iterations of our “release automation”, which is how we produce the bits for Firefox betas and releases. With each incarnation, the automation has become more reliable, supported more functionality, and end-to-end time has reduced. If you go back a few years to Firefox 2.0 it took several days to prepare 40…
8 Jan 2015
Here's December 2014's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an or as a . There was a low number of pushes this month. I expect this is due to the Mozilla all-hands in Portland in early December where we were encouraged to meet up with other teams instead of coding…
9 Sept 2014
ZNC is great for having a persistent IRC connection, but it’s not so great when the IRC server or network has a blip. Then you can end up failing to rejoin with over and over again. The way to fix this is to limit the number of channels ZNC can connect to simultaneously. In the Web UI, you change ‘Max…
1 Sept 2014
We’ve removed the rsync modules and today, after a few months ago and hearing no objections. Those modules were previously used to deliver Firefox and other Mozilla products to end users via a network of volunteer mirrors but we now use content delivery networks (CDN). If there’s a use case we haven’t considered then please get in touch in the…
30 Jun 2014
Mozilla Release Engineering provides some of the Buildbot continuous integration system, which can be useful to check how many jobs are currently running versus pending. There are graphs of the last 24 hours broken out in various ways – for example compilation separate from tests, compilation on try and everything else. This data also feeds into the pending queue on…
22 May 2014
This post is password protected. You must visit the website and enter the password to continue reading.
29 Apr 2014
from Release Engineering work week in Portland, 29 April 2014Video of this presentation So in Part 1 and 2, we saw how Buildbot tegra and panda masters can assign jobs to Buildbot slaves, and that these slaves run on foopies, and that these foopies then connect to the SUT Agent on the device, to deploy and perform the tests, and…
from Release Engineering work week in Portland, 29 April 2014Video of this presentation By design, Buildbot masters require a Buildbot slave to perform any job. For example, if we have a Windows slave for creating Windows builds, we would expect to run a Buildbot slave on the Windows machine, and this would then be assigned tasks from the Buildbot master,…
26 Apr 2014
from Release Engineering work week in Portland, 29 April 2014Video of this presentation So first key point unveiled: Second key point: To answer this, first remember the wide variety of builds and tests we perform: The answer is: Notice: So as you see, a panda is a more serious piece of kit than a tegra. Think of a tegras as…