the Chromium logo

The Chromium Projects

gsutil

gsutil is a command line tool for working with Google Storage (GS) buckets. We use it heavily in ChromiumOS to mirror our files around the world.

See the official gsutil Tool documentation for more details.

This document is a practical guide for our developers working with GS.

Installing gsutil

The easiest way to run gsutil on your workstation is to use the copy that is part of the CrOS checkout in chromite. Simply run ~/chromiumos/chromite/scripts/gsutil to get an up-to-date version. You can symlink that into your personal $PATH to simplify things.

Otherwise, feel free to follow the official Install gsutil documentation.

Configure authentication (.boto)

NB: If you previously used gsutil config to create your ~/.boto, those credentials should remain valid, so you can copy them around from existing installs. However, Google has turned down support for using gsutil to acquire new credentials.

If you don't have a ~/.boto file set up yet for your account, use gcloud from the Google Cloud SDK.

For Googlers, you can install it with sudo apt-get install google-cloud-sdk. For everyone else, please follow the Installation instructions.

Use the auth login subcommand to obtain credentials. It will print out directions for you to follow.

(outside) $ gcloud auth login

When it finishes, credentials are copied into a subdirectory that you will need to copy out of.

# NB: If this fails, replace "*" with your e-mail address.
(outside) $ cp ~/.config/gcloud/legacy_credentials/*/.boto ~/.boto

Note: The gcloud tool suggests to set PROJECT_ID, but for our purposes, there's no need to set it to anything.

Various GS buckets

There are a variety of GS buckets that show up in CrOS. We try to cover everything that CrOS developers might see.

As a general rule, buckets that start with gs://chromeos- are not browsable by the public, although some files might be readable for tools to access.

Buckets that start with gs://chromiumos- are fully readable by the public. Do not host any internal files on such buckets.

Note: We omit buckets that are already documented in Archive Mirrors.

Build/Release buckets

These buckets are used by our builders and release process.

cros flash and cros chrome-sdk will pull artifacts from these buckets.

Public artifacts

These buckets are automatically updated by builders for public consumption.

Testing buckets

These aren't used for anything other than testing. Code should not rely upon these for anything other than testing.

Android (ARC++) buckets

These are used by developers working on ARC++ (Android in CrOS).

VM/container (Crostini) buckets

These buckets are used for Containers/VM/Crostini.

ClusterFuzz buckets

See the fuzzing documentation for details on the buckets used by ClusterFuzz.

Old/dead buckets

These buckets are no longer used by current systems.

googlestorage_acl.txt file format

The googlestorage_acl.txt file is used to set permissions for files associated with a particular overlay/board/project. It is a series of arguments that are passed to the gsutil acl ch command. Comments start with a # character and extend to the end of the line.

# For information about this file, see
# https://www.chromium.org/chromium-os/developer-library/reference/tools/gsutil/#googlestorage_acltxt-file-format

# Give owners of the chromeos.int.bot@gmail.com project full control.
-g 00b4903a97fb6344be6306829e053825e18a04ab0cc5513e9585a2b8c9634c80:FULL_CONTROL

# Give viewers of the chromeos.int.bot@gmail.com project read access.
-g 00b4903a97ce95daf4ef249a9c21dddd83fdfb7126720b7c3440483b6229a03c:READ

# Give all Googlers read access.
-g google.com:READ

FAQ

How do you set up prebuilts for a new private overlay?

Add a googlestorage_acl.txt file in the root of the private overlay.

How do I create a new GS bucket?

Creating a new GS bucket is not normally something we do. You should double check to see if any of the existing buckets satisfy your use case, and if not, be prepared to provide extensive justification.

If you're confident you need a new bucket, please file a bug using the Infra>Client>ChromeOS component.