Simon Sapin (:SimonSapin) Reporter
	Description • 6 years ago

# Background

The Servo repository contains a copy of web-platform-tests which has many files, and creating many files on Windows is slow. This made "git checkout" take about a minute, even though (for now) we’re not running those tests on Windows and don’t need them to build compile.

Using git’s "sparse checkouts" [1] fix this, but they involve creating a configuration file in a very specific format: Unix newlines, UTF-8 (or ASCII), no BOM. Creating this in a batch script turns out to be surprisingly difficult. Environment variables from the task payload’s `env` key are truncated at the first newline, because they are made available to the task by generating a `set NAME=VALUE` batch script. Escaping newlines in a batch variable requires an jumping through an number of hoops [2].

I ended up doing base64-encoding in the decision task, and (ab)using the `certutil` tool for base64-decoding [3].

generic-worker has the ability to "mount" arbitrary files into the task’s directory, but the file needs to be available at an HTTP URL or as a task artifact.

[1] https://git-scm.com/docs/git-read-tree#_sparse_checkout
[2] https://stackoverflow.com/a/5642300/1162888
[3] https://stackoverflow.com/a/16946681/1162888


# Proposal

For at least single-file mounts, add two options for the `content` key in addition to URL Content and Artifact Content:

* Text content: object of { "text": "<string>" }, the UTF-8 encoding of the string is written to the file
* Base64 content: object of { "base64": "<string>" } the base64 decoding of the string is written to the file

	Simon Sapin (:SimonSapin) Reporter
	Comment 1 • 6 years ago

By writing this I realize that my decision task could produce and artifact, which could be used as a mount content. Still, for very short files inline content would be nicer.

To avoid making task definitions too big there could be a size limit on inline mount content (a few kilobytes?) with a suggestion to use an artifact for larger files.

	Pete Moore [:pmoore][:pete]
	Comment 2 • 6 years ago

Max is taking a look at this in https://github.com/taskcluster/generic-worker/pull/129.

Many thanks Max!

	[github robot]
	Comment 3 • 6 years ago

Commits pushed to master at https://github.com/taskcluster/generic-worker

https://github.com/taskcluster/generic-worker/commit/501349a5494cb1b95f8b59950f260f9d6bd1b56a
added features requested in Bug 1495732

https://github.com/taskcluster/generic-worker/commit/c5e8e7078a0bbd745278719221037af7d00dc6eb
Merge pull request #129 from mraethel/master

Bug 1495732 - support inline content when mounting files/directories