Yaraslau Kurmyza

Firefox-CI 502 errors post mortem

Fri, 15 Mar 2024 00:00:00 +0000

Below is a multi-month investigation of 502 GLB errors. Scroll down to summary for the tl;dr

Firefox-CI 502s post mortem

Firefox-CI (a Taskcluster deployment) is a distributed system that orchestrates millions of tasks daily, making sure they are being scheduled, executed and resolved. It was built in such a way that any operation could be retried in case of a server failure.

However, if you have large amounts of errors, it might make the system much slower, increase execution time of tasks and introduce unnecessary delays and disruptions.

Firefox-CI runs on Google Cloud Platform in Kubernetes and uses Google Load Balancer.

Timeline

After noticing a big amount of 5xx daily errors and were able to find the root causes and reduce those from 40k-50k/day to zero.

Almost all of those 502 errors were sent with a timeout (30s - 120s), which means that clients had to wait around 75s on average before retrying failed calls.

That means that we could save daily on average 45.000 * 75s = 3.375.000s (~930 hours).

Investigation

While identifying an issue with workers, which would shut themselves down while running a task, I noticed that a few API calls returned 502 errors around the same time.

#6681 #6682

One particular issue with 502 is that it was happening on the Load Balancer and client side. But on the services side, logs suggested that the call executed successfully and returned some payload.

Flow is:

Worker calling queue.claimWork()
Queue service handles the request and sends back response with the payload
Response is lost at transmission somewhere between the service and Load Balancer
Worker gets a 502 error and keeps retrying the call.

Note: That means that the system thinks that worker received a response, but in reality, it was lost somewhere in the network stack.

Client libraries can handle server-side errors, but the problem with those is that it can take a random amount of time to see that error, usually between 30s-120s.

Schematically, network stack in cloud infrastructure consists of:

API server (node.js application) running on a kubernetes POD
Kubernetes service that exposes this particular pod
Ingress service that proxies all incoming calls to the underlying services
Load Balancer’s backend that is running on instance groups (k8s nodes)
Google Load Balancer that accepts HTTP requests from the world and passes down to services

This leads to an assumption that calls are “getting lost” somewhere between 1-5 elements of this chain.

Question: where exactly? (Spoiler: everywhere)

Checking logs showed that those 502 errors happen frequently and in extremely large quantities!

Somewhere around 40.000 - 50.000 errors a day!

Analysis of existing errors revealed some patterns:

Most of the 5xx errors are 502s (>95%)
502 errors have different types of errors:
1. response_sent_by_backend
2. backend_timeout
3. failed_to_connect_to_backend
4. backend_connection_closed_before_data_sent_to_client
Errors were usually seen in spikes, some of them happened daily around the same time

Suspect #1 - keepalive timeouts

Based on types of errors, one in particular stood out - backend_connection_closed_before_data_sent_to_client which contributed to more than 85% of failures.

GCP’s troubleshooting guide hinted that keepalive timeout might be out of sync between GLB, ingress and services.

GLB has fixed keepalive timeout of 600s.

Nginx’s (ingress) keepalive timeout was not set, as well as service’s.

That means that GLB can keep connections open and send requests after a connection was being closed by ingress or service.

Patch was applied to set bigger keepalive timeouts for the nginx ingress service.

Similar change was applied on the services side: #6692 to make server keep connection to ingress open longer. #5267 was added to set some numbers for the deployment.

Keepalive timeout tweaks were helping - number of errors dropped from 40-50k a day to 4-5k a day

Backend_connection_closed_before_data_sent_to_client errors were mostly gone, and new target was to find out what causes remaining timeouts.

Suspect #2 - pod autoscaling

Once the majority of errors disappeared, it became easier to spot a few irregularities in those errors - there were spikes.

And those spikes were happening during pod autoscaling:

FirefoxCI cluster runs few HPA (Horizontal Pod Autoscalers) for ingress and queue, auth services to scale pods up and down based on demand (CPU utilization).

Those events happened frequently, which led to the termination of a big number of pods.

Also it was quite easy to verify - disabling HPA showed that the number of errors was much lower.

Suspect #3 - graceful termination

Pods being terminated is a usual thing, and they must be ready, at least in theory, to handle terminations gracefully.

When kubernetes tells pod that it needs to be terminated (pod autoscaler requests this), it would send a SIGTERM signal to the container and would wait at most terminationGracePeriodSeconds, before it would be forced killed.

Kubernetes also allows running custom handlers when POD is about to be terminated. lifecycle.preStop hook is being invoked before SIGTERM is sent.

Flow here is:

HPA requests deployment to be scale down (i.e. we don’t need 20 pods of this service, only 15)
Kubernetes decides which pods to terminate
lifecycle.preStop hook is called
SIGTERM signal is sent to container
Kubernetes waits for pod to finish up to terminationGracePeriodSeconds
Pod is being terminated

Taskcluster up until this point didn’t define any custom preStop hook, or didn’t handle SIGTERM gracefully, but rather just stopped the process, dropping all open connections.

Nginx ingress handled it in some way, but that was not good enough, so few fixes were made:

#6717 added SIGTERM listener in services in attempt to let existing connections finish before process is being killed
#6718 added lifecycle.preStop routine which is simply a sleep 30s at that point.
ingress preStop hook improved to wait for all processes to finish serving requests:

Sleep XXs importance is as follows: Kubernetes controls the flow of incoming requests by using network endpoints. During lifecycle events (creation, termination), it would create or remove network endpoints for those pods. Due to the async nature of kubernetes events, during pod termination, it is not guaranteed that new requests will not be sent to a stopping pod right away. When we add sleep xx to the preStop hook, we make sure that no requests are lost before the network endpoint finally disconnects that pod from new traffic. Existing connections would not be affected, and server would finish serving them after receiving the SIGTERM signal.

Nginx’s stopping procedure became:

sleep 5; nginx -s quit; while pgrep -x nginx; do sleep 1; done

This was necessary because nginx spawns worker processes, and when main processes receive SIGTERM, it would not stop immediately, it would only notify workers. Checking if nginx processes are still running is a way to know that open connections are still being served

Things start to look better after those fixes, total number of daily errors dropped even more:

Further steps

At this point it became really hard to figure out the root cause of the remaining errors. There were few assumptions:

Keepalive timeouts were too big or to small
preStop sleep time was too high or too low
GLB or backends configuration is not optimal
HorizontalPodAutscaler might be killing too many pods at once

More observations

HPA would scale down very fast and kill too many pods at once.
Containers (pods) are being restarted

Suspect #4 - container restarts

Noticing that some spikes occur at a given time of the day and seeing the number of container restarts it became clear that this is an important factor, too.

Turned out, that it was “by design” - #1727

Assumption in the past was that containers should not be running for too long, and were forced to be restarted daily.

#6759 fixes were made to address that.

Side quest achievement - websocktunnel and livelog

Looking at container restarts and lost connections, we were able to make a conclusion that websocktunnel might be suffering from similar issues. #6563

Websocktunnel is being used to stream live logs from running tasks to the browser. Due to some memory leaks, that container would be restarted, and due to some other bugs in worker implementation, it would not properly connect back to the tunnel. Which means long-running tasks (like machine learning for translations that run for days) would be running without logs being visible.

Increasing memory limits helped to avoid restarting containers too often, and made livelogs more stable. Restarts went down from multiple a day to once in 10-15 days.

More tweaks

More kubernetes configuration and application fixes were added in #6760

Suspect #5 - node autoscaling

It became apparent that the majority of remaining 502s are happening during cluster scaling events.

GCP allows us to configure the cluster to have a variable number of nodes in the cluster, based on the current load.

Observing logs revealed that this is happening quite frequently. Nodes are being added and then removed. During node draining and removal, more connections were still being lost.

Few weeks of experimenting with timeouts, keepalives, grace termination periods and sleep delays couldn’t help reduce the number of errors.

At this point I decided to disable this feature, and keep cluster running a fixed number of nodes. Cost savings from autoscaling were not substantial for FxCI, as it was mostly single node being added and removed during the day.

Even GCP documentation suggests disabling this feature where dropped connections should be an issue.

By disabling node autoscaling we were able to get rid of most of the 502s which means that clients would not be affected by random 30s-120s delays, and be forced to retry api calls. Which makes the whole system more reliable and resilient.

Node autoscaling fix

As the last piece of the puzzle, there were still few things we could try. One of them was container-native load balancing feature that allows GLB to send traffic directly to the pod which should receive the traffic, skipping the node instance group.

This works on the service and ingress levels and to let Google Kubernetes Engine to know that some particular service should use Network Endpoint Groups (NEGs) and route traffic to pod directly.

To enable this feature service should have this annotation:

cloud.google.com/neg: '{"ingress": true}'

Once this is done, Load Balancer should start sending traffic directly to nginx pods, so in theory, when node scaling events happen, GLB would be aware which pods are healthy and which are not, and avoid sending traffic to the terminating pods.

After 24h there were almost a hundred of scale up/down events, and zero related 502s

Some final tweaks

Experimentally it was proven that following combination provided best results for FirefoxCI:

Pod’s terminationGracePeriodSeconds: 210
Pod’s preStop sleep: 120
GLB’s backend configuration: connectionDraining.drainingTimeoutSec: 60
Pods are being killed too fast during downscaling

https://github.com/taskcluster/taskcluster/pull/6824

Summary

Major issues and findings were:

Application was not prepared to be killed gracefully
Nginx ingress was not waiting long enough for existing connections to be drained
GLBs configuration didn’t have draining timeout set
Autoscaling of pods and nodes were not configured optimally
1. Network Endpoint Groups were not configured to use container-native load balancing
Lack of proper E2E monitoring - application thinks the response was successful but client sees 5xx errors instead
Containers were forced-killed daily for no reason

ES modules. Migration of a large codebase

Tue, 17 Oct 2023 00:00:00 +0000

Taskcluster has a large codebase that is written in nodejs and consists of many components: database layer, internal libraries shared among microservices, and services themselves.

They all use shared npm dependencies.

To keep project up-to-date, we use dependabot. However, more and more packages switch to ESM-only distributions, which makes it impossible to use in commonjs projects.

ES Modules can import both ESM and CJS modules, but CJS can only import CJS.

CJS vs ESM

CommonJS modules usually look like this:

// package.json
{
  "main": "index.js"
}

// index.js
const { foo } = require('./foo');
console.log(foo);

// foo.js
exports.foo = 'foo';

ESM modules look like this:

// package.json
{
  "type": "module",
  "exports": "./index.js"
}

// index.js
import { foo } from './foo.js';
console.log(foo);

// foo.js
export const foo = 'foo';

Migration

In case of the example above, migration is straightforward:

Change or add "type": "module" to package.json
Replace all const x = require(y) with import x from y
1. Make sure that files are referenced with .js extension: import x from './y.js'
2. In case of folder import x from './y/index.js'
Replace all exports.x = y with export const x = y
Check possible problems with default vs named exports.
Replace __filename, __dirname with import equivalents
Replace dynamic require(x) with await import(x)
Replace require('./data.json') with type assertions or fs.readFile + JSON.parse
Replace module.parent usage

If you have a larger codebase you could follow my steps from taskcluster migration PR.

Bulk replace all imports

There’s a great but abandoned utility cjs-to-es6 which would help you to run the initial migration.

npx cjs-to-es6 ./services
npx cjs-to-es6 ./internal
// etc

It would do pretty good job already by replacing all require and exports with import and export.

After that you would still need to go through all files and fix remaining issues:

it doesn’t know if imported module is file or folder, so adding .js or /index.js would be up to you to fix
it doesn’t rewrite all module.exports or exports.xx magic that might be happening in the code
it doesn’t replace require() in the middle of your code with dynamic import()

Named vs default exports

With Taskcluster internal libraries we kept both default and named imports to minimize the impact of the migration.

However, for new code, named exports should be preferred, as it is easier to understand what is being imported.

Consider default export example:

// foo.js
export default (arg) => {
  return arg + 1;
}

// index.js
import foo from './foo.js';
foo(1);

// other.js
import incrementor from './foo.js';
incrementor(1);

If you want to refactor you would run into issue, because different files might have given different name for the default export and you would need to update all of them.

export const incrementor(arg) => {
  return arg + 1;
}

// index.js
import { incrementor } from './foo.js';
incrementor(1);

// other.js
import { incrementor } from './foo.js';
incrementor(1);

Would make search and replace easier.

Fixing dynamic exports

Taskcluster used a lot of magic in testing library to mock dependencies for various tests.

Something like this:

// helper.js

exports.withServer = () => {
  setup('server setup', () => {
    exports.server = new ApiClient();
  });
  teardown('server teardown', () => {
    exports.server = null;
  });
}

// client_test.js
const helper = require('./helper.js');

suite('serverTest', () => {
  helper.withServer();

  test('has server', async () => {
    assert.ok(helper.server);
  });
})

This kind of magic will not work with ESM as they strictly require to have exports to be statically analyzable.

So one way to fix this issue is to return an object that would be modified by those helper functions, or make such functions return unique object every time.

// helper.js
// Approach 1

const helper = {};
export default helper;

helper.withServer = () => {
  setup('server setup', () => {
    helper.server = new ApiClient();
  });
  teardown('server teardown', () => {
    helper.server = null;
  });
}

This way in your tests you could import helper and use it as before.

// helper.js
// Approach 2

export const withServer = () => {
  const helper = {};
  setup('server setup', () => {
    helper.server = new ApiClient();
  });
  teardown('server teardown', () => {
    helper.server = null;
  });
  return helper;
}

// client_test.js
import { withServer } from './helper.js';

suite('serverTest', () => {
  const helper = withServer();

  test('has server', async () => {
    assert.ok(helper.server);
  });
})

Both approaches are fine, but in case of Taskcluster it was easier to go with approach 1.

Importing json files

Node.js allows to use import assertions since v17, but it is still marked as experimented and you will likely see errors like this:

(node:40419) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time

But you can still use them like this;

import jsonSchemaDraft06 from 'ajv/lib/refs/json-schema-draft-06.json' assert { type: 'json' };

Alternatively, to avoid seeing warnings you could just use fs to read file:

const jsonSchemaDraft06 = JSON.parse(fs.readFileSync('ajv/lib/refs/json-schema-draft-06.json', 'utf8'));

`dirname`, `filename`

Are gone, so if you need them, use import.meta instead:

const __filename = new URL('', import.meta.url).pathname;
const __dirname = new URL('.', import.meta.url).pathname;

`module.parent` usage

Also no longer available and if you want to know if your module was imported or executed directly you could check same import.meta.url:

// before
if (!module.parent) {
  // file was executed directly
  load.crashOnError(process.argv[2]);
}

// after
import { fileURLToPath } from "url";
if (process.argv[1] === fileURLToPath(import.meta.url)) {
  load.crashOnError(process.argv[2]);
}

Conclusion

Migration to ESM is not that hard, but it is not as easy as just adding "type": "module" to package.json.

TIL: Prometheus Conference Berlin 2023

Sat, 30 Sep 2023 00:00:00 +0000

I didn’t attend PromCon Berlin 2023 in person, but I watched streams on the youtube.

Below are some new (for me) things, that are worth noting:

Perses is a dashboard for Prometheus (and later other sources) metrics
OpenTelemetry is a standard for metrics, traces and logs
Beyla automated application metrics with eBPF and Prometheus (more below)
Kepler Kubernetes Efficient Power Level Exporter
Platypus attack. Power-side channel attacks on Intel CPUs. Guessing CPU instructions and operands by their memory consumption, and ability to extract crypto keys
Cloud Carbon Footprint tool to estimate energy use and carbon emissions from public cloud usage
CNCF Mentoring Initiatives if anyone wants to do contribute and need help
Autometrics add SLOs like success rate, latency, attach to your code and let it do the restore

Few takeaways:

create a tool (prometheus) and everyone would try to find an application for it, even if it is not always the right tool. Common issue.- create a tool and everyone would build 100x more tools around it. No wonder it’s so hard to figure out what’s good
measuring code metrics from inside the code is not right, as it doesn’t include kernel, IO, network timing and client side issues

Beyla

Grafana’s beyla is a “eBPF-based auto-instrumentation of HTTP and HTTPS services”

This is a quite novel and interesting approach to collect metrics from code without even touching the code. It does so by using eBPF. My understanding of eBPF is that it is a set of kernel calls that allow user-space applications to get notified on various events that are happening inside kernel. For example listen to all tcp_connect calls. It is also language (and version) specific though, intercepting those calls alone is not enough, you need to know memory layout to get list of arguments and context of that call.

Beyla mainly supports GO code and some generic HTTP, HTTPS calls for other languages and can provide basic RED metrics (Request, Errors, Duration)

It can send later all this events to prometheus directly. User-space monitoring program requires CAP_SYS_ADMIN privileges.

GreenOps

It is really hard to measure impact of data centers, and cloud services provide limited carbon monitoring.

Kepler Kubernetes-based Efficient Power Level Exporter. It uses RAPL (Intel-based Running Average Power Limit) feature that reports energy consumption. Which later uses eBPF to attribute power usage to process and then to pod. So you could later aggregate by namespace/deployment/service.

Downside is that RAPL not accessible in VMs and is the source of fantastic vulnerability discovered in Platypus attack.

Platypus attack is a brilliant and creative way to sample power usage and associate it with unique processor instruction (different instructions consume different amount of power). It does also allow to get the Hamming distance of its operands (number of non-zero bits). Having this info it is “really easy” to extract information like crypto keys from memory. Crazy.

If you want to measure your impact you could use Cloud Carbon Footprint to get those estimates. Here’s demo

Autometrics

Autometrics look interesting too. It is a micro-framework which uses decorators to wrap code and extract metrics automatically. Spec here

What I find interesting is that it allows to provide objectives (SLOs) in form of success rate, latency. Besides that, it generates promql queries to query those metrics in provided Grafana dashboards.

“Auto” in name might be misleading, because it still requires adding a library, making changes to code and providing a config.

One year at mozilla

Sun, 15 Jan 2023 00:00:00 +0000

My last post on this topic one month at Mozilla was posted as the title suggests when I just started.

My contribution at that point was a single pull request as I was struggling to understand what is happening and how things work.

Since then, I was able to do a bit more and here is a brief summary of my first year at Mozilla:

submitted 119 PRs in taskcluster project.
created 40 issues
helped shipping 3 major versions to Firefox-CI and 75 minor + patch releases
fixed worker-manager bug that drove everyone crazy
sent first patch to Firefox (1807667)
attended or lead 27 Taskcluster weekly Community meetings
improved developer experience
was on approx 350+ calls (according to Raycast wrapped)
met with 800+ mozillians at the annual All-Hands
visited Berlin office 10+ times
ordered 25+ books of which managed to finish maybe 5 for now
learned truly a lot from the people who I work with or who worked there before me

Overall, I feel like this was a very productive year for me, despite all my fears and concerns at the beginning, when I was trying to figure things out.

That, of course, came with big support from the people I work with, big thanks and credits to:

Pete Moore great patience and discussions about the golden age of 8-bit computing
Matt Boris mastering dependabot and node/go upgrades
Benson Wong guiding me on what to focus on
Mathieu Leplatre reminded me how to use mercurial to contribute to Firefox
Graham Beckley for sharing thoughtful articles, ideas and music
Bryan Sieber for being goal-oriented

There are also many people outside of my team who influenced me in a positive way:

Bastien Orivel from community who contributes quite a lot to the Taskcluster project
Dustin Mitchell is a former Mozilla employee who keeps writing code even when he sleeps
Jonas Finnemann Jensen was one of the authors of Taskcluster, created few cool things while at Mozilla
whole RelEng team including Aki Sasaki, Andrew Halberstadt, Johan Lorenzo who support my team and provide valuable insights to the Taskcluster project.
Michelle for trying to deploy Taskcluster the “hard way”

I had to spend some time to collect various bugs and issues that were important at that time. Here is that list. This list is huge, so we had to prioritize and decide what to work on. Below I list few achievements that I was able to accomplish.

When I joined Taskcluster project it was mostly in a maintenance mode. Most of the original team left after unfortunate round of layoffs or its aftermath few years ago. The knowledge of the system was mostly lost or existed in some pieces in numerous places. Firefox CI which is the main deployment of Taskcluster that runs thousands of millions of tasks was not updated for many months and was haunted by some bugs that made whole service feel pretty buggy and slow.

Taskcluster consists of several micro-services that exist in the same monorepo and it had few issues that a new person joining this project might have:

You could only run one or few micro-services locally while you do your development. You need to start each one manually, do some changes, add some tests, hope for the best and push.
It was not possible to test your changes on some environment before you make a release. So often we would need to deploy release to be able to test it, and then ship few hotfixes right after.
Dependabot sent many PRs to update UI packages, but no one knew if those will work because tests were not present

UI tests

To gain more confidence in frontend dependency upgrades I’ve switched testing from mocha to jest and improved coverage for most of the utils functions and few components. I also added e2e tests with cypress. But the problem with E2E was that at this time I didn’t know how the system works or what to test, so it really covered just a simple case, and is still not running as part of CI jobs.

Having more tests helped to be slightly more confident in the merged dependency upgrades.

Dev deployment

Although Taskcluster had everything it needed documented and had the right set of tools, it was still quite tricky to deploy your own Taskcluster instance in the Kubernetes cluster.

Pre-conditions where to set up RabbitMQ instance and get the credentials, create Postgres DB instance in Cloud SQL, provision static IP and load balancer, manually issue certificate (which will later expire and you’d have to upload renewed one manually). Next you would have to manually configure DNS entries in Route53 to point to your dev deployment

Then, having all those necessary credentials you could yarn run dev:init that will ask few questions and generate big dev-config.yml containing all necessary values for the kubernetes templates that would later be deployed with yarn run dev:apply command. If you made a mistake, you’d have to repeat that process again, and some config steps were not obvious.

In order to simplify this I did few improvements:

patched config tool and documentation on how to properly generate dev-config.yml
added new utility tools: dev:ensure:db, dev:ensure:rabbit to allow re-running some config steps separately, without the need to run whole config again
tested and documented how to install both rabbit and postgres into the cluster with helm to be able to script all those steps without the need to use external services. In the end you could run one script to install rabbit, postgres and use those for the deployment

Still, the problem was to provision static IP, Load Balancer and create certificates manually. For this I’ve added ingress-nginx support and cert-manager to use LetsEncrypt to provision certificates automatically.

I was also able to configure a wildcard DNS to point to our single static IP on dev cluster so anyone could deploy his dev instance easily.

Major benefit was that we no more had to manually provision static IP, certificate and load balancer. By getting rid of GLB and using nginx-ingress we got more flexibility during deployments and re-deployments and saved time (no need to wait 5-10 minutes for GLB to pick up “healthy backends”). Provisioning of certificates was automated with adding few annotations and metadata to the config files.

Worker Manager “slowness”

No argue that Taskcluster is a very complex system that went through a lot of changes to get where it is today. One of the problems was to run thousands of tasks for any change you make in Firefox source code. Having multiple contributors results in tens and hundreds of thousands of tasks that needs to be executed somewhere.

Those tasks are executed by workers (docker-worker or generic-worker) that are running on any virtual or physical device. And the best way to do this was to create those machines on demand, based on the load.

Initial attempts were aws-provisioner) and ec2-provisioner. Which were later replaced by worker-manager.

Taskcluster has worker pools. Each worker pool belongs to provider (AWS, Azure, GCP, static) and specifies how new instances should be created (AMI, config, timeouts, etc). Each task is being claimed by a worker from specified worker pool (i.e. I want taskA to run on google/ubuntu-18.04 worker pool).

This way worker-manager can constantly check every worker pool to see if it has new tasks. Then using some simple estimations it can decide if new instances are needed, or existing capacity is enough. If new instances are needed, it will provision up to maxCapacity instances. Instances will be launched, workers would connect to Taskcluster and start executing those tasks, shutting themselves down once there are no new tasks.

Every once in a while, worker-manager would scan all workers that it started to see if they are still fine or need to be shut down.

So essentially, worker-manager consists of provisioner loop (deciding how many instances to start) and worker-scanner loop (checking running instances status, stopping).

All works good … before the loop times out:

The red dots above says that the worker-scanner loop was timing out after about 2.5 hours. This means that not all of the workers could been checked, so system doesn’t know what state the worker is or to even know the exact amount of running capacity.

Digging into the the timeouts I’ve discovered that most of those delays come from one single provider (starts with A and ends with zure). And when you do thousands and thousands of calls daily you would inevitably run into all sorts of issues. But somehow it was only Azure.

How many API calls do you need to start new virtual machine? AWS - 1, GCP - 1, Azure - 3+! Yes, at least 3 calls to create a virtual machine (provision IP, provision NIC, provision Disks (optional), provision VM). Not to mention that Azure API is not very reliable.. calls take different amount of time to complete and sometimes hang indefinitely, which made us implement few timeouts to kill the calls that take too long (speaking of 10m+ here). And to add to this, provisioning of a single resource (pi/nic/disk/vm) is done in async manner, which means you need to: 1. request resource and get operation id 2. query if the operation with given id completed. So to provision IP/NIC/VM you need at least 6+ API calls. (Why Azure? WHY?)

Let’s get back to the worker-scanner who tries to do as much as possible as fast as possible. What happens is that it goes through every worker pool and checks all workers within that pool. When it gets to the Azure pools and some calls begin to take longer it cannot process other pools, times out and dies. Simply making this process parallel wouldn’t help, because you can easily hit rate limits and it would be even a bigger issue. Having single loop helps to control backoff timeouts and retries for cloud api calls easier.

So all of those timed out loops led to few workers being created on time, which led to stuck builds and hours and hours of waiting time to deliver anything. And if worker fails to start within a defined timeframe it will be killed. No wonder everyone was frustrated and overall impression was quite bad, but no one could figure out why everything is so slow.

There were few different solutions proposed in the past to this problem, like combining scanner and provisioner into a single loop. But this still would have an issue with timing out azure calls.

Without understanding how to fix this issue at the time, my first take was to separate bad providers into own isolated loop, that I could examine separately to figure out how to fix its slowness. So instead of merging both loops, I split it even more with creating separate azure scanner job.

Next small improvements where to realize that not all workers need Public IP, so we could skip IP provisioning and reduce total number of calls. Here we go, skipPublicIP was added. Some calls take forever? Okay, let’s time cap it at 10min, which is insanely too much anyways. And then optimize it a bit by requesting first resource from the provisioner loop immediately.

I’ve added few diagrams in the docs to display whole process. Check how insanely complex Azure checkWorker is.

Huh, alright.. so what good or bad did those changes do?

Turned out, that as soon as Azure pools were kicked out of the main loop, it automatically solved all the issues! Miracle. Worker scanner was not timing out anymore (or at least did it way less often), which means that AWS and GCP workers could start instantly now and serve incoming numbers of tasks. But also isolated Azure loop was seeing much less timeouts and was working more predictably.

After few weeks everyone who considered Taskcluster slow and buggy suddenly forgot about those issues. Tasks were being run on time, birds were signing, grass was green and sun was bright..

I was just lucky to be able to solve this issue in a relatively simple manner. Understanding and getting down to that solution took me few handful of weeks though.

Continuous Deployment

Ok, so as the main problem was solved, it was time to continue solving the problem of testing before the release is done. For this I experimented with GCP’s Cloud Build and it proved to be working great. Each new commit to main branch triggered build job in Cloud Build. New docker image would be built and stored in Cloud Registry.

Having up-to-date docker image for the main branch means that we can now use it to deploy to our dev environment. Using previously implemented changes it was trivial to create new environment https://dev.alpha.taskcluster-dev.net/ that will use ingress-nginx, cert-manager and deploy change there.

Later Matt Boris helped to create cloudbuild.yaml and put steps definitions in there. It could build new image, deploy it, remove older images as we no longer need them and run smoketests.

This was a huge leap forward because it allowed us to experiment and test features even before they would be released. And this happens within 10-15 minutes after branch gets merged to main.

Developer experience

Having own dev environment was already a big win, but still, it required merging your changes into main branch and waiting for the build to finish. So I started to build a docker-compose approach, that would allow to run all or required services locally and develop all or many services at once. This promised to be a huge win, because it would eliminate the need to build and deploy at all. You could just run everything locally.

However, it took more than one month to get there. Because every service has its own set of configuration options, own client ids, own credentials to access database, queue, apis, it required a lot of trial and error attempts to make it right. Huge credits go to Bastien for his contributions. He was able to provide valuable comments and patches to make things work together. Setting up local S3 storage with minio, showing few tricks how to use healthcheck conditions properly (service_healthy, service_completed_successfully), and how to set up minio and ingress properly.

Still, to run everything locally you’d need a lot of CPU and memory, as in addition to running main services, there were tens of background jobs. Sure, you don’t need all of them to run locally, but also I had no idea how to make services optional in docker-compose.yml so they wouldn’t be starting with docker compose up. Luckily it was a solved problem, as I discovered profiles, which I described in this post

So you could easily start just the API workers, and additionally include some background jobs with profiles as needed.

Next Matt Boris helped to add docker-compose.dev.yml and necessary yarn scripts so we could run some services in production mode and some in development.

By adding -devel image with nodemon it was possible to map local volumes and make dev containers to reload once the source code changed. (This is still an issue for UI though, as webpack inside docker works extremely slow, so running UI natively is a better approach for now)

This allowed to have full local development environment. Which included the UI, where you could login, create tasks, do everything as in production environments, call API endpoints and run tasks.

Running tasks locally proved to be a harder task, as it required containerizing generic-worker and bundling all its dependencies. In the end it was possible to make it running along with the services using same docker compose up.

Having those pieces of puzzle in place, made it possible for other teams to contribute easier. Andrew Halberstadt was able to implement changes to the github service using ngrok to forward webhooks to the local services he was running with docker compose.

Metrics and visualisations

Alright, having those problems solved I was able to have some fun and add few missing features.

Dashboard

Taskcluster is big but you don’t know what’s happening inside. It is a distributed system and you don’t know that something is happening, or know what exactly is happening there.

I wanted to be able to see a few stats. So I added some stats to the dashboard:

Community TC:

Firefox CI:

Good, it is helpful to see how many pending tasks you have now, or how many workers were requested, or are running at the moment.

Timing

Let’s move on to the tasks. As I mentioned somewhere above, each push to mozilla-central can trigger thousands of builds. Each single push results in a Task Group being created that consists of all the tasks.

If you opened Task Group page before you would see how many tasks are in Completed state, how many Failed, how many Running. But you were not able to see which task is the longest to run? Or how long does it take to finish some task without going into each task individually.

Okay, so I want to open the Task Group page and see how long did it take for each task to complete, what was the longest one, how long it took the whole graph to resolve?

This was the first take:

I say “first” because the information presented here only gives answers to “how long it took to resolve”. Ideally it should also include information on how long do tasks need to wait before they are being executed. If tasks depend on each other it would be good to show who is waiting the longest, and few other things.

With this information you can already understand a lot. Sort by duration and see how some windows tests run for two hours. See the distribution of task times on a graph. Hover quickly to see which tasks were failing and how long it took for all tasks to complete.

It revealed that some groups might have more than a thousand compute hours to resolve. That is a lot!

Firefox Remote Settings tests

This one is rather funny and is a good example that you should always fix the root cause of the issue.

I’ve discovered that Taskcluster UI serves millions of calls a week to the unknown URLs: /api/remote-settings-dummy/v1 .. which did not belong to Taskcluster. And because default backend is ui which runs nginx that serves all requests to index.html which respond with 200 to all requests.

My first thought was to add 404 response status on nginx level to all endpoints that don’t belong there. But this will not fix the problem, calls would still be made. Luckily, Pete discovered that all those URLs are defined in the Firefox source code. Test profiles used http://localhost/api/remote-settings-dummy* endpoints. Somebody wanted to avoid doing real network calls in tests and thought that this would be enough.

However, the way that docker-worker and generic-worker work, they expose taskcluster-proxy on http://taskcluster host that enables easy way for tasks to communicate with Taskcluster (to fetch secrets, artifacts and create new tasks). However, this domain is mapped to 127.0.0.1 by default. And what happened that all http://localhost/ requests were also served by this proxy and forwarded back to the Taskcluster deployment.

That was my chance after almost a year in Mozilla to finally contribute to Firefox. Luckily, main developer of Remote Settings, Mathieu Leplatre was from my team, and he was happy to help me to submit my patch. Was a cool experiment, as the last time I used mercurial was around 2010 or so.

Here was this patch that triggered more discussions from the reviewers and I was able to patch all places and make sure things work as expected after those changes. So instead of changing two files as I initially thought, I had to change a bunch of related files and updates few tests. This was during the Christmas holidays, so it took a bit longer. Thanks to Rob Wu for being patient with the reviews.

The fix landed into 110.0a1 release.

Few days later I started to observe less and less calls to Taskcluster UI, and eventually numbers went from millions a week to just few thousands. There are still some older branches people develop against, it will take time for everyone to fetch latest changes. But to be fair, I have no idea how many developers work on Firefox source code, because you can checkout clean branch, apply few changes, and be behind 100+ commits just few hours later. Lots of things happening there.

Bonus win here came few days later as someone reported 7%..11% improvement in test times. Ha! We removed network calls and now the test would finish 6s..16s faster. Awesome. Considering absolute numbers of those tests, total gain would be quite impressive. Possibly few hours to days of compute resources saved daily or weekly.

What could be done better?

Of course, mentioning just the good parts wouldn’t be fair and would mean that there were no downsides or disappointments. I’ll list some of them here.

It took me way longer to understand how Taskcluster works, and I still don’t fully understand it. It took me about two month to find to start delivering valuable patches.

I didn’t write nearly enough blog posts as I was hoping to. I had plans to write series of posts describing how Taskcluster works. So this is the first post, but hopefully not the last one on the topic.

I’ve started with E2E tests but didn’t manage to finish those. And they are not being used at the moment. Should find some time and cover more workflows and then make it run on each build.

I saw few gaps in documentation and was planning to extend it with more topics, but only managed to cover one tutorial: how to run tc locally. Improving search functionality of the documentation, along with providing client library code examples are in my future plans.

I planned to release a TUI utility to manage Taskcluster deployments. Something like k9s but for taskcluster. Being able to navigate through worker pools, workers, scopes, roles, tasks. This could be a fun tool, I just need to find more time to continue working on it.

Few times I was trying to fix something without understanding or seeing the bigger picture, which lead to increased complexity of the system. Luckily I have great team members who are willing to tell me about the alternatives.

Taskcluster’s codebase is quite old, and was created at times where ESM modules didn’t exist yet. Now, more and more libraries are switching to ESM-only packages, and it is impossible to upgrade those. This is something that we’ll need to look into and make necessary changes to the monorepo to add support to ESM modules.

I’ve also experimented with adding Typescript support to it, but didn’t have time to finish those experiments.

Another big issue is that ui uses neutrinojs that was created by great folks in Mozilla some time ago, but, unfortunately, it is no longer actively maintained. And this prevents us from doing some upgrades and changes. It is also based on older versions of webpack which are very slow in making builds and in local development. It is especially painful to build static components inside docker. The plan is here to migrate to vite and get rid of neutrino, but it wasn’t finished yet. WIP PR open since August’22.

And of course OKRs. I have a privilege on this project to decide myself what to work on, to set my own goals. Some OKRs where a hit, some were a miss. As always, it is better to under-promise and over-deliver.

And also I’ve broken few builds, which blocked few teams and made few releases roll back.

Stay tuned :)

Docker compose profiles

Sun, 14 Aug 2022 00:00:00 +0000

So you have too many (micro-) services..

It is quite common to have docker compose for your local development. Sometimes even for production. But as your project grows you have more and more services and containers. It takes a lot of resources to run them at once.

To prevent docker compose from starting all services, you can use service profiles for this.

Official documentation contains some examples, but I will provide another real-world example below.

Yet another web app example

Let’s say our application consists of the frontend application, backend API, database, two background workers and one cronjob:

services:
  db:
    image: postgres

  frontend:
    image: your-app/frontend

  backend:
    image: your-app/backend
    command: api

  backend-worker-1:
    image: your-app/backend
    command: worker-1

  backend-worker-2:
    image: your-app/backend
    command: worker-2

  cronjob:
    image: your-app/backend
    command: cronjob

So if you run docker compose up -d now, it will start all 6 services at once. Quite possible, you only wanted to start backend and frontend.

You can introduce profiles for the services that don’t need to start:

  backend-worker-1:
    image: your-app/backend
    command: worker-1
++  profiles: ["backend", "worker"]

  backend-worker-2:
    image: your-app/backend
    command: worker-2
++  profiles: ["backend", "worker"]

  cronjob:
    image: your-app/backend
    command: cronjob
++  profiles: ["backend", "cron"]

With those changes when you run docker compose up -d, those extra services will no longer start. Only db, frontend, backend will.

If you want to run those services, you have three options:

docker compose up -d backend-worker-2 this will auto-enable profile and start specific service
docker compose --profile worker up -d will start both workers, as both of them have this profile. Keep in mind, profile argument is positional, you cannot use it after up command.
COMPOSE_PROFILES=backend docker compose up -d will start all backend workers and cronjob using env var.

I’ve noticed that second approach is quite tedious. You need to remember which profile you used. This will be a problem once you try to stop all containers with docker compose down and you will notice that those extra workers didn’t stop.

Note: if you use networks, you might additionally see this error: failed to remove network network_name: Error response from daemon: error while removing network: network network_name id f3f6e7fc5 has active endpoints.

This is because down command didn’t know about those extra profiles. To solve this you need to call it with exactly same profile names: docker compose --profile worker down.

And with environmental variable you can do:

export COMPOSE_PROFILES=backend
docker compose up -d # will start all
# ...
docker compose down  # this will stop all according to COMPOSE_PROFILES

Of course, this only works in the same terminal session, or if you have this variable set globally.

We started using profiles in my current project - Taskcluster because of the big amount of services.

Currently Taskcluster’s docker-compose.yml defines more than 50 services, most of which are periodic cronjobs or background workers that are not needed during local development.

First month at mozilla

Mon, 14 Feb 2022 00:00:00 +0000

It’s been one month since I’ve joined Mozilla as a software engineer. I am part of the system engineering team that handles taskcluster.

Taskcluster is crucial for Mozilla projects. It orchestrates thousands of builds, tests and release tasks daily. It deserves a separate post, which I will do next.

It took me exactly 1 month to contribute.

There are things that are new to me, and I would like to share some of those:

Many people work in Mozilla long years

5-7 years is the usual number. Quite often you meet people that have worked 10+ years. But there are those who have worked since the beginning.

It would be good for me to find out what motivates them ;)

Working on open-sourced projects makes lots of people open, too

They discuss things they are working on and problems they have.

I admire how John Whitlock keeps his dairy public where he describes all the things he’s working on.

Would be great to follow the same principles and share my progress, as well.

Importance to self-organize

Being remote demands you are able to organize yourself:

create your own schedule and stick to it. Free to work evenings/mornings/combination of both with long pause in the afternoon
keep detailed log of all activities (meetings, tasks you are working on, things you learned, etc)
make your own notes of every meeting. It helps to revisit them, so you have a better understanding and make sure you don’t miss something.
write down people’s names, what they are doing and how you can help each other. There are way too many people to remember everything
write plans/goals for the next day, next week, next month, …
dump all your ideas at the end of the day, things that you didn’t finish, or would like to work on, and use it a plan next day
keep a list of all the things you discover, everything that you can improve, fix, change or remove. Revisit and check if you can execute on those
review past meeting notes to spot forgotten action items

Time is relative

If you try to schedule one on one with people, some will be happy to talk to you tomorrow, some next week, some next month. Urgency is usually not a thing compared to the startup world.

Community involvement

For Mozilla the community is the integral part of their being. In my Taskcluster project we have weekly community meetings. It gives an excellent opportunity to talk to people outside of Mozilla who use the same project. Give them status updates on the progress. Take feedback and listen to their opinions.

Those meetings happen even if there is only one person joining. Which is great!

Mozilla originals

Turns out there are many wonderful projects that are being developed or originated at Mozilla:

bugzilla .. as old as Firefox or even older. Fun fact, JIRA is coming from Gojira, Japanese for Godzilla.
sops modern way of managing secrets which can check-in to vcs
letsencrypt standard nowadays for HTTPS certificates
json-e extended json to convert boring yaml into dynamic yaml

I’ll keep discovering and extending this list.

Contribute - Codetribute

If you feel like contributing but don’t know where to start, there is a great page: Codetribute

You can choose your preferred language, pick a bug/task and improve something.

Docker Desktop alternatives for Mac with M1 chips

Mon, 01 Nov 2021 00:00:00 +0000

Docker Desktop is not the only way to run Docker on Mac with M1 chips (as of November 2021, updated July 2023).

Important thing to know and understand is that your docker images must have either aarch64 (arm64) or x86 (amd64) architecture. Afaik, you can mix both to some extent, but only with Docker Desktop.

I experimented with alternatives, and they are quite nice to work with, but not everyone knows about them.

OrbStack

I’ve learned about OrbStack in June 2023. It is a similar to WSL2 approach to run lightweight Linux virtual machine on M1 mac. It integrates itself into command line and gives you both docker support and ability to run linux virtual machine.

You can use orb or orbctl to manage VMs, or use docker right away.

$ docker run -it -p 80:80 docker/getting-started

UI itself is minimalistic, yet powerful and extremely fast. It can ran 20+ taskcluster containers easily, and CPU usage is minimal, comparing to Docker Desktop.

As of 2023 OrbStack is my favourite choice on M1/M2 Macs due to it’s low footprint and speed.

Multipass

Easiest way to launch Ubuntu VMs is by using multipass from Canonicial.

Installation is trivial: brew install --cask multipass. After this you can create and run as many VMs as you want, provided you have enough CPU and memory.

List available VMs:

$ multipass ls

Name                    State             IPv4             Image
primary                 Stopped           --               Ubuntu 20.04 LTS

Primary should be installed by default, if not one can run multipass launch --name primary --cpus 2 --mem 4G --disk 20G.

If you see something like Launch failed: Operation canceled you can run according to this comment:

sudo launchctl stop com.canonical.multipassd
sudo launchctl start com.canonical.multipassd

Controlling it’s state is as easy as multipass start, multipass stop, to enter multipass shell.

Cool thing about it, is that can be used as a default linux environment for development, not just for Docker.

It mounts automatically your home folder to /home/ubuntu/Home using sshfs, so you can run commands inside, but edit files on the host machine.

To get docker up and running:

$ sudo apt update
$ sudo apt install -y docker.io docker-compose
# to be able to run docker without sudo
$ sudo usermod -aG docker $USER

After that you can go and run your containers as you did so far. If you want to do it from the host machine, you’d need to set up docker context.

On your host machine you should have docker binary either from Docker Desktop (which you don’t need to start), from brew, or other source. Docker can fire commands against docker daemon that runs inside ubuntu machine. To do so, it needs ssh access inside the machine, it will need ssh ubuntu@IP (IP can be seen in multipass ls output).

Make sure to add your ssh key to the ubuntu machine, so you can access it from the host machine.

# on host
$ cat ~/.ssh/id_rsa.pub | pbcopy

# inside ubuntu vm
echo '<your_key>' >> ~/.ssh/authorized_keys

With that done, you can add docker context on your host machine

# Create context
$ docker context create multipass --docker "host=ssh://ubuntu@192.168.64.7"
# Switch to it
$ docker context use multipass
# Run container
$ docker run --rm -it alpine /bin/sh

# In case previous command fails with
# `exec user process caused: exec format error`, you'd need to specify platform:
$ docker run --rm -it --platform linux/arm64/8 alpine /bin/sh

As far as I understand, ports are not forwarded automatically, so you need to forward them manually, or just use vm IP instead of localhost, i.e: http://192.168.64.7:5000

$ time multipass start
multipass start  0.03s user 0.03s system 0% cpu 19.318 total

$ time mp stop
multipass stop  0.02s user 0.02s system 3% cpu 1.219 total

Lima

Lima launches Linux virtual machines with automatic file sharing, port forwarding, and containerd.

Fast and lightweight, you are not limited to Ubuntu. Can be installed with brew install lima.

Similar workflow - create one or more VMs, and then run containers inside them, or just use them as a default linux environment.

# launch linux
$ limactl start

# shell into
$ lima

This will open an editor to specify the configuration for the new VM, like architecture, mounts, ports, etc.

There are plenty of examples how to configure for different use-cases (k3s, k8s, alpine, centos, etc)

Can run x86 with qemu emulation, but is rather slow. While aarch64 feels like native.

$ time limactl start

limactl start  0.02s user 0.03s system 0% cpu 22.145 total

$ time limactl stop

limactl stop  0.01s user 0.02s system 3% cpu 1.069 total

Podman machine

Podman is a daemonless container runtime for Linux.

It claims to be a drop-in replacement for docker, but on Mac you obviously need an underlying linux VM running.

Podman machine is a tool to create and manage VMs for Podman. (This was introduced recently, as a replacement for podman-machine). Naming, is hard… remember.

$ podman machine init
$ podman machine start
$ podman machine ssh # but you don't have to, you can stay on host and run `podman` as `docker

One thing to note is that you don’t have a docker-compose equivalent, althought there is such thing as podman-compose. I believe this is because podman is trying to be closer to kubernetes and provides k8s compatible configs and pods. You can create pod, and attach multiple containers to it, which will share fs, ports, etc. Pretty much as pod in k8s.

$ time podman machine start

podman machine start  0.02s user 0.03s system 0% cpu 14.526 total

$ time podman machine stop
time podman machine

podman machine stop  0.01s user 0.02s system 27% cpu 0.110 total

Docker desktop

Yep, rather slow, compared to lima and multipass, but can run both x86 and aarch64 images simultaneously.

Upd 2023: Seems like Docker Desktop was seriously improved and provides significantly better performance, also UI gives much more options now for monitoring, viewing files and seamlessly executing in containers.

UTM

This is a GUI tool to run VM’s on Mach. Not limited to linux, can run Windows as well.

You can launch favourite linux distro.

Personaly, I don’t consider this as a good alternative, as you it provides less integrations with docker and command line.

My done is done done

Thu, 10 Jun 2021 00:00:00 +0000

It is difficult to imagine an IT company nowadays that doesn’t speak scrums. Every job description you read, every company you talk to will tell you that they are doing “sprints” and that they are “mostly agile”.

And unless you work alone or in a very small group, chances are you have a lot of participants. Each feature will go through the hands of a bunch of people during its lifetime. Let’s take this example:

The flow

Stakeholders or business will come up with feature idea
Product owner or project manager will gather all requirements and will come up with development/release plan
UX/UI expert will do analysis and initial design based on the requirements they have
Developers will take the requirements, prototype, design and will work on a feature, deploy it to a staging environment
Different developers would normally do a code review of the introduced changes
QA will check if feature works according to initial requirements and UX rules
Developers will fix the bugs would there be any
Product owner together with stakeholders will do their business acceptance
Someone (developer, release manager, lead dev) will release this feature to prod
Plus ideally providing all features with full test coverage - unit/integration/e2e tests

This is a more or less happy flow for small features that don’t have many dependencies and don’t need to be broken down into smaller features. This is also a scenario where developers are cross-functional and can do everything (backend/frontend/devops) without making this process even more complicated. Developing complex features usually takes more teams, more coordination between those teams and more communication. I will leave it out for now.

My done is done done

Now if you ask participants, what they think when they are done, you would probably hear many different things:
“I finished working on mockups and design, I’m done“
“I’m done with development of this feature”,
“I’m done testing, sent it back to developers”,
“I’m done reviewing the ticket”,

And it is kinda normal nowadays. Projects are big, there are many features to work on, constant time pressure, etc, etc. Often it comes down to fire and forget scenarios.

Problems

This kind of attitude often results in:

too many features are developed at the same time (different participants)
features rarily meet deadlines
tickets are being underestimated
lack of understanding the scope of the ticket
missed sprint goals
slow development/release pace
forgotten/abandoned merge requests or PRs that stay “forever” in open state

Definition of done

For most of the features definition of done will look like the flow above. The final done would be a release to prod and closing of the ticket.

Each team member should:

have a clear understanding of what done is
be responsible for the success of the whole team
be able to follow feature through all steps
be able to communicate problems

Of course, different members may join at different stages of feature development lifecycle. But it is still important to communicate goals, deadlines and requirements well enough that everybody could understand what their role is.

In ideal flow no one will say “it’s done” until feature is released and ticket is closed.

Docker called me

Thu, 03 Jun 2021 00:00:00 +0000

Story about me giving feedback to Docker.

Docker

I’ve been using docker since 2014. It is hard to imagine nowadays development and deployment without the containers.

Using it on non-linux machines, especially on MacOS was always somewhat problematic. Running out of disk space, fans spinning like crazy, networking peculiarities. Evolution of tooling and introduction of Docker Desktop improved things a lot. No more self-managed virtual machines, no more Vagrant provisioning. Start the desktop app and start using docker, magic.

Docker Developer Preview Program

This year I gave in and finally bought my first personal laptop since 2010. Because I was using laptops from work for my personal stuff from time to time. But the magic of M1 silicon was too attractive to resist it.

With M1 came new problems, especially with x86 and arm64 images and ability to run Docker at all. But it didn’t take long to roll out the first public betas for Developer Preview, so I decided to join this program.

I wanted to get new releases faster and communicate M1 specific issues. So I joined a Docker Community Slack.

There is a small form to enroll into Docker Developer Preview Program. One of the questions is “What one feature would you add to Desktop?”

I realised that it would be nice to add Volume management and disk space management into the desktop app.

The call

Few months later I received an email from Dieu Cao who is a Senior Product Management Director at Docker, Inc. She is interested in discussing the Volume Management I’ve written about in a feedback form.

This was a nice surprise and the first time any software vendor of this size came back to me. It feels good, when someone is interested in hearing feedback from you.

Zoom call consisted of several participants. Besides Dieu, I’ve got a chance to speak to Djordje Lukic (Staff Software Engineer), Usha Mandya (Senior Technical Writer) and Trung Nguyen (Software Engineer).

Although I didn’t have much time to prepare for the call, I was able to describe my thoughts and pain points we have using Docker Desktop. I mentioned that one of the bigger issues on MacOS is that disk space tends to disappear quite fast. Not all IT staff in my company are tech-savvy and know how to fix issues. Like executing several CLI commands to free up space:

# analyse disk usage
docker system df -v
# clean what can be cleaned
docker system prune
# drop unused containers
docker ps -a | grep Exited | cut -d '\'' '\\'' -f 1 | xargs docker rm
# then a bit more with images
docker rmi $(docker images --filter "dangling=true" -q --no-trunc)
# and then even more
docker volume prune

And while you already can see images and containers in Desktop App, you can not do the same with Volumes. And volumes tend to grow in size, and in case of our application, we have a lot of those. So disk space ends fast. Having this feature inside the app would make it much easier to manage and free-up space when it’s needed.

I also mentioned that compose integration can be improved. Our application consists of 25 services. Orchestrating them in the Desktop App is not always an easy task. Sometimes you get cannot execute docker-compose script . Sometimes some Environment variables are missing or mounts are empty.

Dieu said that they’ve been working on Volume Management already. And they wanted to share it with me already to hear my feedback. How awesome! This feature is due to be released next week, and I have an pleasure to review it and give feedback 😏

So I’ve got a chance to sneak peak new features in action, leave my comments and also learn something new for myself. They described some extra functionality related to volumes that I’d never thought about before.

That was a surprisingly great experience for me, and I hope my feedback was helpful to guys and girls from Docker.

Takeaways

Very positive experience and solid approach from the Docker team. Reaching out to the primary audience to discuss the major pain points is the right attitude.

I wish more companies would be that open and willing to accept ideas and suggestions.

Controlling Phillips HUE lights with Arduino

Tue, 28 Jul 2020 00:00:00 +0000

Home improvement project with Arduino and HUE lights.

Controlling Phillips HUE lights with Arduino

I have a bunch of HUE lamps at home that are mainly controlled from the app.

Playing with Arduino, I thought that I can easily add some motion detection and remote control for some of the lamps.

Hardware

What we have is:

Arduino Nano 33 IoT which comes with WiFi and Bluetooth
Infrared Motion Sensor (HC-SR501)
Phillips HUE Bridge API integration

Hue Bridge API

One has to obtain new access token in order to have access to the lights API.

Once the key is obtained, lights can be turned on and off by doing PUT requests to the bridge:

curl -X PUT 'http://YOUR-BRIDGE-IP/api/YOUR-API-TOKEN/lights/3/state'  -d '{"on": true, "sat": 54, "bri": 130 ,"hue": 33}'

It is possible to change color, brightness and on/off state. For our scenario, we are only interested in {"on": true} or {"on": false}

Assembly

PIR sensor has 3 connections:

VCC -> +5V
GND -> GND
OUT -> 2 (2nd pin in my case)

Note about +5V which was required by sensor: Arduino Nano 33 IoT have +5V disabled by default (!). And it can only be operated if powered by USB. In order to get this +5V on the pin, one need to solder (connect) two VUSB connectors next to it.

Code

Setup phase consists of setting up pins and connecting to WiFi.

#define SENSOR_PIN 2
#define SECRET_SSID "home-ssid"
#define SECRET_PASS "home-pass"

int status = WL_IDLE_STATUS;

char ssid[] = SECRET_SSID;
char pass[] = SECRET_PASS;

int sensorValue = 0;
int oldValue = 0;

WiFiClient client;

void setup() {
  Serial.begin(9600);
  // When debugging locally following line can be uncommented, so we can see whole output from the device
  // while (!Serial);

  Serial.println("Setup pins");
  pinMode(SENSOR_PIN, INPUT);
  pinMode(LED_BUILTIN, OUTPUT);

  // check for the WiFi module:
  if (WiFi.status() == WL_NO_MODULE) {
    Serial.println("Communication with WiFi module failed!");
    // don't continue
    while (true);
  }

  String fv = WiFi.firmwareVersion();
  if (fv < WIFI_FIRMWARE_LATEST_VERSION) {
    Serial.println("Please upgrade the firmware");
  }

  // attempt to connect to Wifi network:
  while (status != WL_CONNECTED) {
    Serial.print("Attempting to connect to SSID: ");
    Serial.println(ssid);
    status = WiFi.begin(ssid, pass);
    // wait 10 seconds for connection:
    delay(5000);
  }
  Serial.println("Connected to wifi");
}

Once connected, we can listen for the sensor value to change:

void loop() {
  sensorValue = digitalRead(SENSOR_PIN);

  if (sensorValue == HIGH) {
    if (oldValue == LOW) {
     Serial.println("State changed to: High :)");
     digitalWrite(LED_BUILTIN, HIGH);
     sendCommand(true);
    }
  } else {
    if (oldValue == HIGH) {
      digitalWrite(LED_BUILTIN, LOW);
      Serial.println("State changed to: Low :(");
      sendCommand(false);
    }
  }
  oldValue = sensorValue;
  delay(50);  // Not really necessary, but some delay wouldn't hurt
}

Last thing is to actually send a HTTP request to the bridge to turn on/off the lamp:

IPAddress server(192,168,2,225); // IP of the bridge
const int port = 80;
const char apiUrl[] = "PUT /api/TOKEN/lights/14/state HTTP/1.1"; // URL with the lamp id to control

void sendCommand(bool isOn) {
  String content = "{\"on\": false}";
  if (isOn == true) {
    Serial.println("Turning lamp on");
    content = "{\"on\": true}";
  } else {
    Serial.println("Turning lamp off");
  }

  client.stop(); // if previous request is active, it can be stopped

  if (client.connect(server, port)) {
    Serial.println("Connected to server");
    client.println(apiUrl);
    client.println("Accept: application/json");
    client.println("Content-Type: application/json");
    client.println("Content-length: " + String(content.length()));
    client.println();
    client.println(content);
    Serial.println("Sent: " + content);
  } else {
    Serial.println("Connection failed");
  }
}

And that should be it.

Notes

PIR sensor turned out to be quite a stubborn one. It has two dials to control sensitivity and delay, and it was tricky to find the proper combination that would work in my room.
Code is missing the check for the daylight, as it will make no sense to run it during the day. It can either require additional photo sensor, to see if it is bright enough in order not to switch lamp on.
As the next step, It would probably be better to connect to the MQTT and send messages there, because:
- It would make this client simpler, and would save us from talking to the bridge directly.
- external listeners could use additional information to control the lights
- it could all be connected to the Home Assistant for extra fun, and tracked history of events

Updated version with MQTT

To address the issues and shortcomings of direct communication with HUE Bridge, I’ve changed the way how events are being handled. I push those to the MQTT which is running on a home server, driven by Raspberry Pi 4.

MQTT and Home Assistant are running with docker:

version: '3'

services:
  home-assistant:
    image: homeassistant/home-assistant:stable
    environment:
      - TZ=Europe/Berlin
    ports:
      - 8123:8123
    volumes:
      - ./config:/config

  mqtt:
    image: eclipse-mosquitto:1.6
    ports:
      - 1883:1883
      - 9001:9001

Code now looks like this:

#include <SPI.h>
#include <WiFiNINA.h>
#include <ArduinoMqttClient.h>

#define SENSOR_PIN 2

#define SECRET_SSID "ssid"
#define SECRET_PASS "pass"

char ssid[] = SECRET_SSID;
char pass[] = SECRET_PASS;

WiFiClient client;
MqttClient mqttClient(client);

// sensor values
int sensorValue = 0;
int oldValue    = 0;

const char broker[]     = "192.168.55.55";  # IP of the MQTT
int        port         = 1883;
const char topicWrite[] = "sensors/event";  #
const char sensorId[]   = "s1";


void setup() {
  Serial.begin(9600);

  Serial.println("Setup pins");
  pinMode(SENSOR_PIN, INPUT);
  pinMode(LED_BUILTIN, OUTPUT);

  // check for the WiFi module
  if (WiFi.status() == WL_NO_MODULE) {
    Serial.println("Communication with WiFi module failed!");
    while (true);
  }

  String fv = WiFi.firmwareVersion();
  if (fv < WIFI_FIRMWARE_LATEST_VERSION) {
    Serial.println("Please upgrade the firmware");
  }

  // attempt to connect to Wifi network:
  while (WiFi.begin(ssid, pass) != WL_CONNECTED) {
    Serial.println("Attempting to connect to SSID: " + String(ssid));
    delay(5000);
  }
  Serial.println("Connected to wifi");
  printWifiStatus();

  Serial.print("Attempting to connect to the MQTT broker: ");
  Serial.println(broker);

  if (!mqttClient.connect(broker, port)) {
    Serial.print("MQTT connection failed! Error code = ");
    Serial.println(mqttClient.connectError());
    while (1);
  }

  Serial.println("We are connected to the MQTT broker!");
  Serial.println();
  Serial.println("Awaiting orders");
}

void loop() {
  // call poll() regularly to allow the library to send MQTT keep alives which
  // avoids being disconnected by the broker
  mqttClient.poll();

  sensorValue = digitalRead(SENSOR_PIN);

  if (sensorValue == HIGH) {
    if (oldValue == LOW) {
     Serial.println("State changed to: High :)");
     digitalWrite(LED_BUILTIN, HIGH);
     sendCommand(true);
    }
  } else {
    if (oldValue == HIGH) {
      digitalWrite(LED_BUILTIN, LOW);
      Serial.println("State changed to: Low :(");
      sendCommand(false);
    }
  }
  oldValue = sensorValue;

  delay(50);
}

void sendCommand(bool isOn) {
  mqttClient.beginMessage(topicWrite);
  if (isOn == true) {
    mqttClient.print("ON");
  } else {
    mqttClient.print("OFF");
  }
  mqttClient.print("|");
  mqttClient.print(sensorId);
  mqttClient.endMessage();
}

void printWifiStatus() {
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your board's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.print(rssi);
  Serial.println(" dBm");
}

I’ve chose to publish events to the sensor/event with two messages ON|s1 and OFF|s1. s1 is just in case I’d add more later, and plain text for simplicity. It could have also been a separate queue for on/off.

Now, when the sensor is only sending messages to the queue, we can do the heavy part on the Home Assistant side.

For this I used built-in Automation tools, adding two listeners, one that listens to the MQTT message and turns lamp on, and the one that turns lights off.

With the automation it became possible to add conditions, like execute the commands only during specific time (night), or using sunrise/sunset events for my location. Also I was able to add the delay for turning lamps off, to prevent it from flickering back and force too often.

Automation config:

- id: '1596129224206'
  alias: Turn on  the lights on MQTT
  description: ''
  trigger:
  - payload: ON|s1
    platform: mqtt
    topic: sensors/event
  condition:
  - after: '21:30'
    before: 06:00
    condition: time
  action:
  - brightness_pct: 70
    device_id: 289e562dddbc4af783c23cdcc4a8b9cd
    domain: light
    entity_id: light.esszimmer2_lamp
    type: turn_on
- id: '1596130596060'
  alias: Turn off the lights on MQTT
  description: ''
  trigger:
  - payload: OFF|s1
    platform: mqtt
    topic: sensors/event
  condition:
  - after: '21:30'
    before: 06:05
    condition: time
  action:
  - delay: 00:00:20
  - device_id: 289e562dddbc4af783c23cdcc4a8b9cd
    domain: light
    entity_id: light.esszimmer2_lamp
    type: turn_off

Yaraslau Kurmyza

Firefox-CI 502 errors post mortem

Firefox-CI 502s post mortem

Timeline

Investigation

Suspect #1 - keepalive timeouts

Suspect #2 - pod autoscaling

Suspect #3 - graceful termination

Further steps

More observations

Suspect #4 - container restarts

Side quest achievement - websocktunnel and livelog

More tweaks

Suspect #5 - node autoscaling

Node autoscaling fix

Some final tweaks

Summary

ES modules. Migration of a large codebase

CJS vs ESM

Migration

Bulk replace all imports

Named vs default exports

Fixing dynamic exports

Importing json files

__dirname, __filename

module.parent usage

Conclusion

TIL: Prometheus Conference Berlin 2023

Beyla

GreenOps

Autometrics

One year at mozilla

Most notable achievements

UI tests

Dev deployment

Worker Manager “slowness”

Continuous Deployment

Developer experience

Metrics and visualisations

Dashboard

Timing

Firefox Remote Settings tests

What could be done better?

Docker compose profiles

So you have too many (micro-) services..

Yet another web app example

First month at mozilla

Many people work in Mozilla long years

Working on open-sourced projects makes lots of people open, too

Importance to self-organize

Time is relative

Community involvement

Mozilla originals

Contribute - Codetribute

Docker Desktop alternatives for Mac with M1 chips

OrbStack

Multipass

Lima

Podman machine

Docker desktop

UTM

My done is done done

The flow

My done is done done

Problems

Definition of done

Docker called me

Docker

Docker Developer Preview Program

The call

Takeaways

Controlling Phillips HUE lights with Arduino

Controlling Phillips HUE lights with Arduino

Hardware

Hue Bridge API

Assembly

Code

Notes

Updated version with MQTT

`dirname`, `filename`

`module.parent` usage