Skip to end of metadata
Go to start of metadata

SmartOS relies on images heavily. Images are templates that contain a disk or filesystem image and metadata which are using when creating new Zones or VM's.

Images are managed using the imgadm tool. With this tool you can:

  • View & Download images available on a public image server
  • Import remote images or Install from local images
  • List, Show or Print details about an image
  • Destroy images

Here we'll discuss how to find available images and start using them. Then we'll look at what images are and how to create our own.   Finally, we look at how to create your own private image server.

Basics

Viewing & Downloading Public Images

** note, this information has changed recently, and this paragraph could probably be edited for clarity and correctness **

The default image server is https://images.joyent.com. You can edit this list of sources using the "imgadm sources -a <URL>" command, as documented in the imgadm manpage. Using the command imgadm update, you'll cause the local cache (/var/db/imgadm/imgcache.json) to be updated with the available images on the servers found in your sources.list. Once your local cache is updated, you can list all available images for use using imgadm avail:

# cat /var/db/imgadm/sources.list
https://datasets.joyent.com/datasets

# imgadm update
updating local images database...
Get https://datasets.joyent.com/datasets...
done

# imgadm avail | head
UUID                                 OS      PUBLISHED  URN
6bf31ce2-f384-11e1-a338-e39c2fe4ab59 smartos 2012-08-31 sdc:sdc:mongodb:1.3.2
a0f8cf30-f2ea-11e1-8a51-5793736be67c smartos 2012-08-30 sdc:sdc:standard64:1.0.7
3390ca7c-f2e7-11e1-8818-c36e0b12e58b smartos 2012-08-30 sdc:sdc:standard:1.0.7
9604da58-f1ee-11e1-aba1-dbda3337ec46 smartos 2012-08-29 sdc:sdc:mongodb:1.3.1
15c77de2-f1ed-11e1-8688-47d8455c7932 smartos 2012-08-29 sdc:sdc:standard64:1.0.5
2baee822-f1eb-11e1-8383-f762d43a424c smartos 2012-08-29 sdc:sdc:standard:1.0.5
e8c41d40-f161-11e1-b839-a3631c115653 smartos 2012-08-28 sdc:sdc:base64:1.7.2
9012a9c2-f15d-11e1-a33a-afaec53ebde9 smartos 2012-08-28 sdc:sdc:base:1.7.2
45e0fa58-db5e-11e1-a1f0-df041127b335 smartos 2012-07-31 sdc:sdc:mongodb:1.3.0

To download one of these images, say "base64", we'll import it using the images UUID:

# imgadm import e8c41d40-f161-11e1-b839-a3631c115653
e8c41d40-f161-11e1-b839-a3631c115653 doesnt exist. continuing with install
e8c41d40-f161-11e1-b839-a3631c115653 successfully installed
image e8c41d40-f161-11e1-b839-a3631c115653 successfully imported

# imgadm list
UUID                                 OS      PUBLISHED  URN
e8c41d40-f161-11e1-b839-a3631c115653 smartos 2012-08-28 sdc:sdc:base64:1.7.2

To learn how to create a Zone or VM from these images, please refer to:

  • ...
  • ...

Advanced Topics

What exactly is an Image?

An image is the data and metadata required to create a new VM. The "data" is one or more compressed ZFS datasets which will be cloned to create a new VM. The "metadata" describes, as JSON, the data and outlines the specification for a machine that would utilize it.

Here is an example of the two files:

benr@magnolia:~/datasets$ ls -lh
total 41M
-rw-rw-r-- 1 benr benr 996 Sep 10 14:54 smartos-1.3.12.dsmanifest
-rw-rw-r-- 1 benr benr 41M Jun 10  2011 smartos-1.3.12.zfs.bz2

Image Manifests

The following is an example manifest taken from the public repository https://datasets.joyent.com/datasets/ (re-arranged and line breaks added for clarity).

You'll notice we have properties to identify the image (UUID, name, version, description, etc), the author (creator_name, creator_uuid, etc), when the image was created/updated/published, and then an array identifying the ZFS Dataset file or files, and finally an array outlining some requirements.

  {
    "uuid": "febaa412-6417-11e0-bc56-535d219f2590",
    "name": "smartos",
    "version": "1.3.12",
    "description": "Base template to build other templates on",

    "os": "smartos",
    "type": "zone-dataset",
    "platform_type": "smartos",
    "cloud_name": "sdc",
    "urn": "sdc:sdc:smartos:1.3.12",

    "creator_name": "sdc",
    "creator_uuid": "352971aa-31ba-496c-9ade-a379feaecd52",
    "vendor_uuid": "352971aa-31ba-496c-9ade-a379feaecd52",

    "created_at": "2011-04-11T08:45Z",
    "updated_at": "2011-04-11T08:45Z",
    "published_at": "2011-04-11T08:45Z",

    "files": [
      {
        "path": "smartos-1.3.12.zfs.bz2",
        "sha1": "246c9ae158dc8f204643afdd6bd4d3c4aa35e733",
        "size": 42016482,
        "url": "https://datasets.joyent.com/datasets/febaa412-6417-11e0-bc56-535d219f2590/smartos-1.3.12.zfs.bz2"
      }
    ],
    "requirements": {
      "networks": [
        {
          "name": "net0",
          "description": "public"
        }
      ]
    }
  }

When creating your own manifest, the following properties are required:

  • uuid: The UUID of the image (use an online UUID generator)
  • name: The name of the image (eg: "centos-6")
  • version: The version of the image (eg: "1.0.0")
  • description: A short description of the image
  • published_at: A timestamp for the date of publication on an image server (this does not need to be accurate); to output the current time in the proper format use the command: date +"%Y-%m-%dT%T.000Z"
  • creator_uuid: The UUID of the author of the image (use an online UUID generator if you don't have one)
  • creator_name: The name of the image author
  • urn: A special string for describing the image in the form "cloud_name:creator_name:name:version"; for the "cloud_name" I suggest "smartos" if you are unsure, the creator name is usually your organization. The string should not contain spaces. (eg: "smartos:cuddletech:plan9:1.0.0")
  • type: The type of image, either "zvol" for KVM or "zone-dataset" for Zones
  • os: The OS of this image (this is required, but not validated or used, so use whatever you like)
  • files: An array of one or more file objects, containing the following properties for each:
    • path: Local file path to the image data file (compressed zfs dump)
    • sha1: The SHA1 for the image data file; to obtain the SHA1 hash use: digest -a sha1 <file>
    • size: The file size of the image data file; to obtain use: ls -l <file>

The requirements section is recommended but not currently required, nor is it enforced.

Creating a Custom Zone Image

The process of creating a zone image looks like this:

  1. Create and customize a zone as you wish
  2. Purge the logs, etc. and run the sm-prepare-image to make the machine image-ready (remember to read the warning message!).
  3. Halt the zone: vmadm stop <UUID>
  4. Snapshot the Zone dataset: zfs snapshot zones/<UUID>@image
  5. Dump & Compress the dataset: zfs send zones/<UUID>@image | gzip > image_name.zfs.gz
  6. Create the manifest as described above

You can now import the image locally via imgadm or transfer it to an image server.

Dataset Compression
Datasets must be compressed. You may use either GZip or BZip2. BZip2 will offer a smaller file, but GZip compression is faster. Particularly for datasets larger than 10GB, GZip is highly recommended.

Creating a Custom KVM Image

The process of creating a KVM image looks like this:

  1. Create and customize a KVM instance as you wish
  2. Purge and ready the instance
  3. Halt the VM: vmadm stop <UUID>
  4. Snapshot the disk0 ZVol: zfs snapshot zones/<UUID>-disk0@image
  5. Dump & Compress the dataset: zfs send zones/<UUID>-disk0@image | gzip > image_name.zvol.gz

You can now import the image locally via imgadm or transfer it to an image server.

Importing Images Locally

Typically images are downloaded from an image server, however they can also be imported locally using: imgadm -m <manifest> -f <file>

The process looks like this:

# imgadm install -m smartos-1.3.12.dsmanifest -f smartos-1.3.12.zfs.bz2
febaa412-6417-11e0-bc56-535d219f2590 doesnt exist. continuing with install
febaa412-6417-11e0-bc56-535d219f2590 successfully installed
image febaa412-6417-11e0-bc56-535d219f2590 successfully imported

Serving Images

The default community image (formerly dataset) server is datasets.joyent.com. A wide variety of images are there for you to use and build new images from. But what if you want to distribute your own images for others to use? That's what we'll discuss here.

How Image Server Work

The functions of an image are very simple. When an image server is added to a clients sources.list and they preform an imgadm update, the tool will preform an HTTP GET operation against the source URL. This get will return an array of JSON objects which are dsmanifest files for each available image. Here is an example:

$ curl -ks https://datasets.joyent.com/datasets/
[
  {
    "name": "mongodb",
    "version": "1.3.2",
    "type": "zone-dataset",
    "description": "64-bit MongoDB 2.0 SmartMachine Database Appliance with Quickbackup and Replica Sets",
    "published_at": "2012-08-31T16:04:51.970Z",
    "os": "smartos",
    "uuid": "6bf31ce2-f384-11e1-a338-e39c2fe4ab59",
    "creator_uuid": "352971aa-31ba-496c-9ade-a379feaecd52",
    "vendor_uuid": "352971aa-31ba-496c-9ade-a379feaecd52",
    "creator_name": "sdc",
    "platform_type": "smartos",
    "cloud_name": "sdc",
    "urn": "sdc:sdc:mongodb:1.3.2",
    "created_at": "2012-08-31T16:04:51.970Z",
    "updated_at": "2012-08-31T16:04:51.970Z",
    "files": [
      {
        "path": "mongodb-1.3.2.zfs.bz2",
        "sha1": "dff4787bcc8cd115a2307d1e833a49d23a1ad9b0",
        "size": 115202324,
        "url": "https://datasets.joyent.com/datasets/6bf31ce2-f384-11e1-a338-e39c2fe4ab59/mongodb-1.3.2.zfs.bz2"
      }
    ],
    "requirements": {
      "networks": [
        {
          "name": "net0",
....

When the client downloads an image using imgadm import UUID, by default the client will download the image files (ZFS datasets) specified in the manifest in the form: <source_server_url>/<image_uuid>/<file_path>. So in the example above the file downloaded will be *https://datasets.joyent.com/datasets/6bf31ce2-f384-11e1-a338-e39c2fe4ab59/mongodb-1.3.2.zfs.bz2*. You'll notice that the file in the manifest includes a URL, if present it will be used, but it is not required.

The operation of an image server is therefore very simple and straight forward.

Creating a Poor Man's Image Server

As described above, the functions required of an "image" server are very basic. Therefore we can emulate a dataset servers basic functions in the following way:

  1. On your web server, create a directory for your SmartOS image server. We'll assume "images/" on http://mysite.com.
  2. Add each DS manifest into index.html. Do not include any HTML! We only use this filename because it is the default content sent when the directory is accessed. Remember that the file contains an array of manifest objects, therefore the format will be:
    [
     { manifest1... },
     { manifest2... }
    ]
    
  3. For each image, create a directory using the images UUID, ie: "images/6bf31ce2-f384-11e1-a338-e39c2fe4ab59".
  4. Copy the image file(s) into the UUID directory
  5. Now try it! "curl -ks http://mysite.com/images" should return your objects.
  6. Add it to your sources.list and imgadm update to use.

The solution described here is not elegant, nor optimal, but is a viable option for serving images from web accounts which do not have the option of using the more elegant smartos-image-server described next.

Using the smartos-image-server

A community image server project started by nshalman can be found on github: smartos-image-server

The server is implemented in Node.js and implements the basic functionality required to serve images. This is the recommended method of serving images.

Community Image Servers
Several community image servers are available. To find the existing community image servers, or to add your own, please go to the 3rd party datasets page.
Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Dec 06, 2012

    xin

    Can not import images locally

    Here comes the error info.

    [root@b4-b5-2f-63-ef-20 /zones/imgadmInstall/ubuntu/liveimage]# imgadm install -m test.dsmanifest -f ubuntulive.zvol.bz2
    ddab257c-09b9-4c82-9801-20386cc97338 doesnt exist. continuing with install
    stream.js:81
          throw er; // Unhandled stream error in pipe.
                ^
    Error: write EPIPE
        at errnoException (net.js:781:11)
        at Object.afterWrite (net.js:594:19)
    # imgadm install -m test.dsmanifest -f ubuntulive.zvol.bz2

    ddab257c-09b9-4c82-9801-20386cc97338 doesnt exist. continuing with install

    stream.js:81

          throw er; // Unhandled stream error in pipe.

                ^

    Error: write EPIPE

        at errnoException (net.js:781:11)

        at Object.afterWrite (net.js:594:19)

    The local image was created through the way this page referred.

    1. Create and customize a KVM instance as you wish           
    2. Purge and ready the instance
    3. Halt the VM: vmadm stop <UUID>
    4. Snapshot the disk0 ZVol: zfs snapshot zones/<UUID>-disk0@image
    5. Dump & Compress the dataset: zfs send zones/<UUID>-disk0@image | gzip > image_name.zvol.gz

     About the first step, I did try two ways, "Use image imported from the imgadm avail" and "Created from a iso file". Both of them are failed with the same error info.

    Could you please give a hand about this?

  2. Oct 22, 2013

    The Serving Images section seems to be out of date/doesn't work since the change from DSAPI to IMGAPI.