Closed Bug 1329282 Opened 7 years ago Closed 6 years ago

Deploy QEMU engine to build docker images

Categories

(Taskcluster :: Workers, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jonasfj, Assigned: jonasfj)

References

(Blocks 1 open bug)

Details

Attachments

(7 files)

Basically the plan to replace dind-service is to use a VM-per-task, and run docker within the VM.

That way we can upgrade docker and avoid all the security concerns of running dind.
Summary: Deploy QEMU engine to builder docker images → Deploy QEMU engine to build docker images
First stage for this is getting QEMU images in-tree:
 - format for declaration of QEMU images in-tree
 - base image for building QEMU images in-tree (the bottom turtle)
 - task-kind for building in-tree QEMU images on-push
 - mach commands for working qemu:
     setup                 Setup environment using QEMU
     build                 Build QEMU image
     images                List cached QEMU images
     pull                  Pull image into local cache
     push                  Push image from local cache to S3
     export                Export image from local cache
     import                Import image from file to local cache
     run                   Run command inside VM
     shell                 Start interactive shell inside VM
     purge                 Purge images from local cache
     clear-cache           Clear blob cache used for image builds
Assignee: nobody → jopsen
Attachment #8907093 - Flags: review?(dustin)
Attachment #8907094 - Flags: review?(dustin)
Attachment #8907095 - Flags: review?(dustin)
Attachment #8907096 - Flags: review?(dustin)
Attachment #8907097 - Flags: review?(dustin)
In preparation of landing this, I have granted:
  auth:aws-s3:read-write:public-qemu-images/repository/hg.mozilla.org/mozilla-central/*
to moz-tree:level:3

This will allow people with commit level 3 to use |mach qemu push <image-name>| to upload manually built QEMU images.
Notably, this is useful for updating the bottom turtle.
Comment on attachment 8907093 [details]
Bug 1329282 - mach qemu commands

https://reviewboard.mozilla.org/r/178808/#review184440

A few minor notes, but I really like this.  Very polished!  I'm adding :gps too for the mach bits.  I wonder if this shouldn't be a different Python package entirely (so, taskcluster/qemu/mach_commands.py, and move taskcluster/taskgraph/qemu -> taskcluster/qemu.  I'm not sure, and maybe it doesn't matter.

::: taskcluster/taskgraph/qemu/buildimage.py:62
(Diff revision 1)
> +        # Download from tooltool
> +        if 'tooltool' in source:
> +            # TODO: Figure out how to call tooltool programmatically, it can't
> +            #       be right that it there is no way to just call:
> +            #         tooltool(digest, algorithm, size, targetFile, url=...)
> +            #       Or something like that would really nice.

:glandium has done some work with this..

::: taskcluster/taskgraph/qemu/httpcache.py:34
(Diff revision 1)
> +
> +
> +def fetch_file(url, targetFile, sha256=None, description=None):
> +    """ Download url to targetFile validating that content hash matches ''sha256''"""
> +    description = description or url
> +    for i in range(0, 8):

there's a `retry` lib in `python/third-party` that can do retries.

::: taskcluster/taskgraph/qemu/imagehash.py:62
(Diff revision 1)
> +                            for d, _, files in os.walk(source) for f in files])
> +            return [[f, file_sha256(os.path.join(GECKO, f))] for f in files]
> +        else:
> +            return file_sha256(source)
> +
> +    # Hash an ISO image step

??

::: taskcluster/taskgraph/qemu/imagehash.py:77
(Diff revision 1)
> +        if not isinstance(steps, list):
> +            return hash_source(steps)
> +        return [hash_step(step) for step in steps]
> +    image['cdromA'] = hash_steps(image['cdromA'])
> +    if 'cdromB' in image:
> +        image['cdromB'] = hash_steps(image['cdromB'])

It seems like this misses other parameters to the steps?

::: taskcluster/taskgraph/qemu/imageschema.py:121
(Diff revision 1)
> +    return validate_schema(Image, image, 'invalid image file: {}'.format(imagePath))
> +
> +
> +def load_and_validate_image(image_name):
> +    """Load and validate image meta-data from image.yml"""
> +    return deepcopy(_load_and_validate_image(image_name))

why two functions here?

::: taskcluster/taskgraph/qemu/run.py:54
(Diff revision 1)
> +        '-f', 'name=mach-qemu-tc-worker',
> +        '-f', 'event=start',
> +        '--format', '{{json .}}',
> +    ], stdout=subprocess.PIPE)
> +    proc.stdout.readline()
> +    proc.kill()

what's the purpose of this?

..ah, I see, it's waiting for the "start" event on that container.  A comment might help!  It's not clear this function is specifically intended for that container name, since name is an option to docker_run.
Attachment #8907093 - Flags: review?(dustin) → review+
Comment on attachment 8907096 [details]
Bug 1329282 - Documentation for in-tree QEMU images

https://reviewboard.mozilla.org/r/178814/#review184452

::: taskcluster/docs/qemu-images.rst:35
(Diff revision 1)
> +
> +Image Definition Format
> +=======================
> +An in-tree QEMU image is in defined in ``taskcluster/qemu/<image_name>/``, where
> +``<image_name>`` is the named it is referenced by in task configuration files.
> +The image folder must contain an ``image.yml` file satifying the schema defined

missed backtick

::: taskcluster/docs/qemu-images.rst:40
(Diff revision 1)
> +The image folder must contain an ``image.yml` file satifying the schema defined
> +and documented in ``taskcluster/taskgraph/qemu/imageschema.py``.
> +
> +In short the ``image.yml`` file defines
> + * Treeherder symbol the image building task,
> + * A boolean indicating if the imagte is private,

image

::: taskcluster/docs/qemu-images.rst:41
(Diff revision 1)
> +and documented in ``taskcluster/taskgraph/qemu/imageschema.py``.
> +
> +In short the ``image.yml`` file defines
> + * Treeherder symbol the image building task,
> + * A boolean indicating if the imagte is private,
> + * A human-readable markdown description of the image,

Could this be read from README.md in that directory, instead?  Just an idea :)

::: taskcluster/docs/qemu-images.rst:58
(Diff revision 1)
> +For Linux images it is often easiest to extract and repack the install media,
> +such that it includes custom scripts, tweaked grub configuration and
> +answer files for the installer.
> +For Windows images it is possible to the official install media as ``cdromA``,
> +and then construct ``cdromB`` with an ``Autounattend.xml``, custom install
> +scripts, and binaries.

It's not clear to me how this would be written in `image.yml`.. it has to be a sequence of steps ending with genisoimage, so it seems like there would have to be some extract and regen .. which wouldn't work with a Windows image, right?

::: taskcluster/docs/qemu-images.rst:79
(Diff revision 1)
> +pushing to try. This section outlines a local work-flow for developing QEMU
> +images using ``mach qemu`` commands.
> +
> +.. note::
> +  The various ``mach qemu`` commands requires a Linux machine with
> +  docker and KVM, all binaries like QEMU, etc. are stored in a docker image

..docker and KVM.  All binaries..
Attachment #8907096 - Flags: review?(dustin) → review+
Comment on attachment 8907094 [details]
Bug 1329282 - QEMU image for building QEMU images

https://reviewboard.mozilla.org/r/178810/#review184460

::: taskcluster/qemu/qemu-image-builder/custom-data/install.sh:22
(Diff revision 1)
> +echo ' - Installing docker'
> +apt-get install -y apt-transport-https ca-certificates
> +apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
> +echo 'deb https://apt.dockerproject.org/repo ubuntu-xenial main' > /etc/apt/sources.list.d/docker.list
> +apt-get update -y
> +apt-get install -y docker-engine

pin to a specific version?

::: taskcluster/qemu/qemu-image-builder/custom-data/install.sh:33
(Diff revision 1)
> +apt-get install -y python
> +
> +echo ' - Installing mercurial'
> +mkdir -p /setup /build
> +cp /tmp/custom-data/tooltool.py /setup/tooltool.py
> +cp /tmp/custom-data/tooltool.py /setup/tooltool.py

dupe

::: taskcluster/qemu/qemu-image-builder/custom-data/install.sh:40
(Diff revision 1)
> +cp /tmp/custom-data/robustcheckout.py /usr/local/mercurial/robustcheckout.py
> +. /tmp/custom-data/common.sh
> +. /tmp/custom-data/install-mercurial.sh
> +
> +echo ' - Configuring mercurial to use uncompressed bundles'
> +echo -e '[ui]\nclonebundleprefers = VERSION=packed1' > /home/worker/.hgrc

`cat <<'EOF'` might be easier to read

::: taskcluster/qemu/qemu-image-builder/custom-data/install.sh:44
(Diff revision 1)
> +echo ' - Configuring mercurial to use uncompressed bundles'
> +echo -e '[ui]\nclonebundleprefers = VERSION=packed1' > /home/worker/.hgrc
> +
> +echo ' - Installing test script'
> +cp '/tmp/custom-data/run-task' /usr/local/bin/run-task;
> +chmod +x /usr/local/bin/run-task;

trailing semicolons

::: taskcluster/qemu/qemu-image-builder/image.yml:41
(Diff revision 1)
> +  # Modify the grub boot menu to have a timeout of 1 second
> +  - sed: 's/timeout\s\+[0-9]\+/timeout 1/g'
> +    target: /isolinux/isolinux.cfg
> +
> +  # Modify boot options to specify preseed in kernel parameters
> +  - sed: '/label install/ilabel autoinstall\nmenu label ^Autoinstall Ubuntu Worker\nkernel /install/vmlinuz\nappend file=/cdrom/custom-data/worker.seed initrd=/install/initrd.gz auto=true priority=high preseed/file=/cdrom/custom-data/worker.seed --'

Could this be a multiline constant in the YAML?  The embedded \n make me sad..
Attachment #8907094 - Flags: review?(dustin) → review+
Comment on attachment 8907095 [details]
Bug 1329282 - Task kind for building QEMU images

https://reviewboard.mozilla.org/r/178812/#review184464

::: taskcluster/ci/qemu-image/kind.yml:15
(Diff revision 1)
> +  - taskgraph.transforms.task:transforms
> +
> +# taskgraph.loader.folders:loader will create a task for each subfolder in:
> +folder: taskcluster/qemu/
> +exclude:
> +  - qemu-image-builder  # excluding the qemu-image-builder image (which has to be manually built)

Perhaps just exclude any subdir containing `reference.yml`?

This could happen in the transforms, too: just don't yield anything if that file exists.  That would keep the folder loader clean.

Ah, I see you've done that in the transform.. so no need for exclude?

::: taskcluster/taskgraph/transforms/task.py:381
(Diff revision 1)
>              # name of the produced artifact (root of the names for
>              # type=directory)
>              Required('name'): basestring,
>          }],
>      }, {
> +        Required('implementation'): 'taskcluster-worker:qemu-engine',

We've been referring to these as just `qemu-engine`.  Hopefully eventually it's all tc-worker, so that `taskcluster-worker` will be redundant :)

See e.g., docker-engine, native-engine.
Attachment #8907095 - Flags: review?(dustin) → review+
Comment on attachment 8907097 [details]
Bug 1329282 - QEMU image for building docker images

https://reviewboard.mozilla.org/r/178816/#review184474

::: taskcluster/qemu/docker-image-builder/custom-data/install.sh:28
(Diff revision 1)
> +
> +echo ' - Grant worker user access to docker'
> +usermod -aG docker worker
> +
> +echo ' - Installing python'
> +apt-get install -y python

I think we should be pinning python and docker-engine versions here.

::: taskcluster/qemu/docker-image-builder/custom-data/install.sh:33
(Diff revision 1)
> +apt-get install -y python
> +
> +echo ' - Installing mercurial'
> +mkdir -p /setup /build
> +cp /tmp/custom-data/tooltool.py /setup/tooltool.py
> +cp /tmp/custom-data/tooltool.py /setup/tooltool.py

dupe
Attachment #8907097 - Flags: review?(dustin) → review+
Comment on attachment 8907093 [details]
Bug 1329282 - mach qemu commands

https://reviewboard.mozilla.org/r/178808/#review184440

> :glandium has done some work with this..

See python/mozbuild/mozbuild/mach_commands.py and look at def artifact_toolchain.
Comment on attachment 8907093 [details]
Bug 1329282 - mach qemu commands

https://reviewboard.mozilla.org/r/178808/#review184508

I'm not super keen on new top-level mach commands. Dealing with command bloat is something I've been trying to figure out a solution for. I don't yet have a good answer.

Unless you want to make these all `mach taskcluster-qemu-*`, I have no other suggestions.

I only glanced at the bulk of this code. While it does interest me greatly and I'm sure I'll read it someday, unless you want a 2nd pair of eyes, Dustin's review should be sufficient.

::: taskcluster/mach_commands.py:570
(Diff revision 1)
> +        # Check that we have docker
> +        try:
> +            which('docker')
> +        except Exception:
> +            print('|mach qemu| commands requires docker to be installed')
> +            sys.exit(1)

``return 1`` does the same thing.

::: taskcluster/taskgraph/qemu/httpcache.py:31
(Diff revision 1)
> +def fetch_file(url, targetFile, sha256=None, description=None):
> +    """ Download url to targetFile validating that content hash matches ''sha256''"""
> +    description = description or url
> +    for i in range(0, 8):
> +        try:
> +            req = urllib2.urlopen(url)
> +            size = int(req.info().getheader('Content-Length').strip())
> +            read = 0
> +            updated = time()
> +            with open(targetFile, 'wb') as f:
> +                while True:
> +                    c = req.read(CHUNK_SIZE)
> +                    if not c:
> +                        break
> +                    read += len(c)
> +                    f.write(c)
> +                    if time() - updated > 2:
> +                        updated = time()
> +                        print('downloading: {} at {} %'.format(
> +                            description, int(floor((float(read) / size)*100))))
> +            if read != size:
> +                print('retrying {} content-length mismatch'.format(url))
> +                continue
> +            if sha256 and not verify_file_sha256(targetFile, sha256):
> +                print('retrying {} sha256 mismatch'.format(url))
> +                continue
> +            break
> +        except urllib2.HTTPError as e:
> +            if 300 <= e.code < 400 or 400 <= e.code < 500:
> +                raise
> +        except urllib2.URLError as e:
> +            pass  # retry
> +        except IOError as e:
> +            pass  # retry
> +        delay = min(i*i*0.2+0.2, 30)  # sleep for at-most 30s
> +        print('retrying in {} s'.format(delay))
> +        sleep(delay)
> +    else:
> +        raise FailedCommandError('failed to fetch {} retries exchausted'.format(url))

I've seen this code written too many times. I'd love to see it extracted to a standalone module to facilitate reuse. Where, I'm not sure. Probably somewhere in testing/mozbase. But I don't see an obvious home and don't want to scope bloat this bug. Bleh.

::: taskcluster/taskgraph/qemu/imagecache.py:40
(Diff revision 1)
> +def _ensureFolder(folder):
> +    """Ensure that folder exists by creating it if it doesn't"""
> +    try:
> +        os.makedirs(folder)
> +    except OSError as e:
> +        import errno

Nit: errno imported above.

Deferred imports don't need to happen for most non-mach_commands.py files.

::: taskcluster/taskgraph/util/schema.py:29
(Diff revision 1)
>          return copy.deepcopy(schema(obj))
>      except voluptuous.MultipleInvalid as exc:
>          msg = [msg_prefix]
>          for error in exc.errors:
>              msg.append(str(error))
> -        raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
> +        raise FailedCommandError('\n'.join(msg) + '\n' + pprint.pformat(obj),

Raising FailedCommandError here is not appropriate, as it assumes a mach command context. FailedCommandError should only be emitted in mach_commands.py files or by mach itself.

If you need to catch a specific exception for schema validation errors, raise a custom error type and re-raise a FailedCommandError if necessary.
Attachment #8907093 - Flags: review?(gps)
Comment on attachment 8907094 [details]
Bug 1329282 - QEMU image for building QEMU images

https://reviewboard.mozilla.org/r/178810/#review184542

Would it be possible to change this to Debian Stretch? I'm not sure if Ubuntu gains us anything and I'd like to start standardizing around Debian because of their reproducible builds efforts. Ubuntu is based on Debian, so I don't think the switch will be much effort.
Comment on attachment 8907094 [details]
Bug 1329282 - QEMU image for building QEMU images

https://reviewboard.mozilla.org/r/178810/#review184544

::: taskcluster/qemu/qemu-image-builder/custom-data/install.sh:20
(Diff revision 1)
> +systemctl enable taskcluster-worker.service
> +
> +echo ' - Installing docker'
> +apt-get install -y apt-transport-https ca-certificates
> +apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
> +echo 'deb https://apt.dockerproject.org/repo ubuntu-xenial main' > /etc/apt/sources.list.d/docker.list

Given the rate of churn in the Docker project, I'm not super keen on depending on their package repository.

How do you feel about copying the relevant .deb packages to tooltool and using `dpkg -i` to install them?
Comment on attachment 8907093 [details]
Bug 1329282 - mach qemu commands

https://reviewboard.mozilla.org/r/178808/#review185078

::: taskcluster/taskgraph/qemu/buildimage.py:62
(Diff revision 1)
> +        # Download from tooltool
> +        if 'tooltool' in source:
> +            # TODO: Figure out how to call tooltool programmatically, it can't
> +            #       be right that it there is no way to just call:
> +            #         tooltool(digest, algorithm, size, targetFile, url=...)
> +            #       Or something like that would really nice.

Hmm, looking through `mozbuild.action.tooltool.fetch_file` I don't immeidately see how to use it, or how it cleans up correctly. As far as I can see it operates on cwd, which is just liable to make a mess of things..

We should probably make a single Artifact/tooltool/URL cache with some docs :)

I think I'll leave till I have to do Windows images which will require private tooltool files (install ISO).

::: taskcluster/taskgraph/qemu/imagehash.py:77
(Diff revision 1)
> +        if not isinstance(steps, list):
> +            return hash_source(steps)
> +        return [hash_step(step) for step in steps]
> +    image['cdromA'] = hash_steps(image['cdromA'])
> +    if 'cdromB' in image:
> +        image['cdromB'] = hash_steps(image['cdromB'])

I'm modifying the `image` dictionary in-place...

Some of the nested methods here could have a better name, but there's no pretty way to do this in python...

::: taskcluster/taskgraph/qemu/imageschema.py:121
(Diff revision 1)
> +    return validate_schema(Image, image, 'invalid image file: {}'.format(imagePath))
> +
> +
> +def load_and_validate_image(image_name):
> +    """Load and validate image meta-data from image.yml"""
> +    return deepcopy(_load_and_validate_image(image_name))

because I want to memorize, but also I always want to return a clone...

::: taskcluster/taskgraph/qemu/run.py:54
(Diff revision 1)
> +        '-f', 'name=mach-qemu-tc-worker',
> +        '-f', 'event=start',
> +        '--format', '{{json .}}',
> +    ], stdout=subprocess.PIPE)
> +    proc.stdout.readline()
> +    proc.kill()

Yes... a comment would help :)

::: taskcluster/taskgraph/util/schema.py:29
(Diff revision 1)
>          return copy.deepcopy(schema(obj))
>      except voluptuous.MultipleInvalid as exc:
>          msg = [msg_prefix]
>          for error in exc.errors:
>              msg.append(str(error))
> -        raise Exception('\n'.join(msg) + '\n' + pprint.pformat(obj))
> +        raise FailedCommandError('\n'.join(msg) + '\n' + pprint.pformat(obj),

Let's not do this... I realize we're adding mach.base as a dependency everywhere by doing this pattern...
But it's also really nice in that we don't have to catch every possible exception in mach_commands.py
Comment on attachment 8907093 [details]
Bug 1329282 - mach qemu commands

https://reviewboard.mozilla.org/r/178808/#review185118

::: taskcluster/taskgraph/qemu/imagehash.py:65
(Diff revision 2)
> +            return file_sha256(source)
> +
> +    # Hash an ISO image step
> +    def hash_step(step):
> +        if 'copy' in step:
> +            step['copy'] = hash_source(step['copy'])

ah, sorry, I read these as `step = hash_source(step['copy'])`, which lost the other step parameters.  My error.
Comment on attachment 8907094 [details]
Bug 1329282 - QEMU image for building QEMU images

https://reviewboard.mozilla.org/r/178810/#review184542

I took a quick look at it and couldn't find the `linux-image-virtual` package in debian.
Referenced in `d-i base-installer/kernel/override-image string linux-virtual` from the preseed.

I'm open to moving to debian, but tweaking install medias to work with these hacks is a bit of work.
And a few of the preseed options are Ubuntu specific at this point (not much).
Note: I assume ubuntu is getting reproducible builds too.

I suggest we do this later, once in-tree we can tweak this as much as we want.
Comment on attachment 8907094 [details]
Bug 1329282 - QEMU image for building QEMU images

https://reviewboard.mozilla.org/r/178810/#review184460

> pin to a specific version?

I guess ideally we should pin all package versions, but I guess `apt-get install -y docker-engine=1.11.0-0~xenial` is a good start...

> Could this be a multiline constant in the YAML?  The embedded \n make me sad..

I'm not sure there is anyway to make this nice... it's a really ugly sed expression.

I didn't do multiline, because of whitespace concerns...
Comment on attachment 8907094 [details]
Bug 1329282 - QEMU image for building QEMU images

https://reviewboard.mozilla.org/r/178810/#review184544

> Given the rate of churn in the Docker project, I'm not super keen on depending on their package repository.
> 
> How do you feel about copying the relevant .deb packages to tooltool and using `dpkg -i` to install them?

Let's just pin the version for now... none if this is likely to be docker version sensitive anyways.

Ideally, we should find a way to pin the versions for all the packages..
Comment on attachment 8907096 [details]
Bug 1329282 - Documentation for in-tree QEMU images

https://reviewboard.mozilla.org/r/178814/#review184452

> Could this be read from README.md in that directory, instead?  Just an idea :)

Hmm, yeah, but it's intended to be part of the task definition...
So maybe a full README.md file is too big.

That said I would like to encourage readme files :)

> It's not clear to me how this would be written in `image.yml`.. it has to be a sequence of steps ending with genisoimage, so it seems like there would have to be some extract and regen .. which wouldn't work with a Windows image, right?

huh, it suggest to use `cdromA` as the install media and `cdromB` for answer files and custom stuff.

So install media is undefined in that case...

Oh, you mean, how to use install media as `cdromA` unmodified... The schema does support `cdromA: {url: ..., sha256: ...}`, basically, each cdrom has to be a series of steps, or a source to fetch the ISO from.
Comment on attachment 8907096 [details]
Bug 1329282 - Documentation for in-tree QEMU images

https://reviewboard.mozilla.org/r/178814/#review185218

::: taskcluster/docs/qemu-images.rst:58
(Diff revision 3)
> +For Linux images it is often easiest to extract and repack the install media,
> +such that it includes custom scripts, tweaked grub configuration and
> +answer files for the installer.
> +For Windows images it is possible to the official install media as ``cdromA``,
> +and then construct ``cdromB`` with an ``Autounattend.xml``, custom install
> +scripts, and binaries.

Thanks for the explanation in review comments.  It would be good to have an example here, since I didn't catch the possibility from looking at the schema.
Comment on attachment 8907093 [details]
Bug 1329282 - mach qemu commands

I relinquish review to dustin.

So... this series has all r+, so from where I sit, it is ready to land :)
Attachment #8907093 - Flags: review?(gps)
See Also: → 1403749
Attachment #8907093 - Flags: review?(gps)
Attachment #8912977 - Flags: review?(dustin)
Attachment #8912978 - Flags: review?(dustin)
Comment on attachment 8912977 [details]
Bug 1329282 - Cover docker.py and docker_image.py in image-hash

https://reviewboard.mozilla.org/r/184336/#review189714

::: taskcluster/taskgraph/util/docker.py:142
(Diff revision 1)
> +        "context": ctx_hash.hexdigest(),  # hash of context tarball
> +        "code": hash_path(__file__),  # hash of file that computes hashes
> +        # hash of file that constructs the image-task
> +        "task": hash_path(os.path.abspath(os.path.join(
> +            os.path.dirname(__file__),
> +            os.path.join(os.path.pardir, 'transforms', 'docker_image.py'),

`os.path.join(GECKO, 'taskcluster', 'transforms', 'docker_image.py')` might be more succinct
Attachment #8912977 - Flags: review?(dustin) → review+
Comment on attachment 8912978 [details]
Bug 1329282 - Build docker images uses QEMU image

https://reviewboard.mozilla.org/r/184338/#review189716

That's really cool!
Attachment #8912978 - Flags: review?(dustin) → review+
triage update: this is still on my todo list :)
QA Contact: pmoore
Before Jonas left, we decided to WONTFIX this. Docker-in-docker (dind) is less painful these days.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Component: Worker → Workers
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: