Skip to content

Managing Instances with vmadm

This page describes some common tasks, and some tips & trics for using vmadm(8) to manage instances.

Listing instances

Default fields and sorting:

[root@headnode (bh1-kvm1:0) ~]# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
1e6ea123-dca0-44e6-8c0c-34c7f543fe82  OS    128      running           zone55
20de2bfc-56de-4cd9-8e25-80b017615788  OS    128      running           billapi0
2508eead-dc8f-4df5-baf9-4ce6341f08ed  OS    128      running           zone56
29bdabe1-ff9e-4387-9316-39961d5dbe5c  OS    128      running           adminui0
384de5e9-c9d9-4382-9dd3-949ab410be45  OS    128      running           redis0
56a47549-bceb-48d7-b67e-fd390d083f09  OS    128      running           portal0
5c12a3ef-e60c-479a-a6be-ba93712a3893  OS    128      running           amon0
a7a4cb02-210a-4ccf-9744-99e3982148b0  OS    128      running           cloudapi0
2811071a-5edc-438a-9b7b-e3b81e8829c5  OS    256      running           ufds0
8d01d99b-2b18-41c6-8fec-7209e2a9eef3  OS    512      running           riak0
c3a03fac-dc2c-42ba-aff7-d6fd7a36967a  OS    1024     running           ca0
d104e56e-43df-4d0f-b5ce-43f39d0fef46  KVM   1024     running           ubuntu1
2b99a408-7dfb-445e-cfa1-bed4e14a2509  OS    32768    running           assets0
a4212873-e04a-ef84-b85c-c5aea32ce5f8  OS    32768    running           dhcpd0
ad170576-3ad3-e201-cd89-ad4260c3a3ce  OS    32768    running           rabbitmq0
c843e9b9-b937-c55f-c8dd-a4f6d1bb2ec0  OS    32768    running           mapi0

Use -o to change the output fields or -s to change the sort key.

[root@headnode (bh1-kvm1:0) ~]# vmadm list -o uuid,type,ram,quota,cpu_shares,zfs_io_priority
UUID                                  TYPE  RAM      QUOTA  CPU_SHARE  IO_PRIORITY
1e6ea123-dca0-44e6-8c0c-34c7f543fe82  OS    128      20     100        30
20de2bfc-56de-4cd9-8e25-80b017615788  OS    128      5      50         1
2508eead-dc8f-4df5-baf9-4ce6341f08ed  OS    128      20     100        30
29bdabe1-ff9e-4387-9316-39961d5dbe5c  OS    128      5      50         1
384de5e9-c9d9-4382-9dd3-949ab410be45  OS    128      5      50         1
56a47549-bceb-48d7-b67e-fd390d083f09  OS    128      5      50         1
5c12a3ef-e60c-479a-a6be-ba93712a3893  OS    128      5      50         1
a7a4cb02-210a-4ccf-9744-99e3982148b0  OS    128      5      50         1
2811071a-5edc-438a-9b7b-e3b81e8829c5  OS    256      5      100        1
8d01d99b-2b18-41c6-8fec-7209e2a9eef3  OS    512      10     50         4
c3a03fac-dc2c-42ba-aff7-d6fd7a36967a  OS    1024     15     100        8
d104e56e-43df-4d0f-b5ce-43f39d0fef46  KVM   1024     -      100        100
2b99a408-7dfb-445e-cfa1-bed4e14a2509  OS    32768    10     50         -
a4212873-e04a-ef84-b85c-c5aea32ce5f8  OS    32768    10     50         -
ad170576-3ad3-e201-cd89-ad4260c3a3ce  OS    32768    10     50         -
c843e9b9-b937-c55f-c8dd-a4f6d1bb2ec0  OS    32768    10     50         -
[root@headnode (bh1-kvm1:0) ~]#

Same as above, but sort by ram in DESCENDING order then by CPU shares in ascending order. Notice -s -ram to denote descending order.

[root@headnode (bh1-kvm1:0) ~]# vmadm list -o uuid,type,ram,quota,cpu_shares,zfs_io_priority -s -ram,cpu_shares
UUID                                  TYPE  RAM      QUOTA  CPU_SHARE  IO_PRIORITY
ad170576-3ad3-e201-cd89-ad4260c3a3ce  OS    32768    10     50         -
c843e9b9-b937-c55f-c8dd-a4f6d1bb2ec0  OS    32768    10     50         -
a4212873-e04a-ef84-b85c-c5aea32ce5f8  OS    32768    10     50         -
2b99a408-7dfb-445e-cfa1-bed4e14a2509  OS    32768    10     50         -
d104e56e-43df-4d0f-b5ce-43f39d0fef46  KVM   1024     -      100        100
c3a03fac-dc2c-42ba-aff7-d6fd7a36967a  OS    1024     15     100        8
8d01d99b-2b18-41c6-8fec-7209e2a9eef3  OS    512      10     50         4
2811071a-5edc-438a-9b7b-e3b81e8829c5  OS    256      5      100        1
5c12a3ef-e60c-479a-a6be-ba93712a3893  OS    128      5      50         1
384de5e9-c9d9-4382-9dd3-949ab410be45  OS    128      5      50         1
20de2bfc-56de-4cd9-8e25-80b017615788  OS    128      5      50         1
29bdabe1-ff9e-4387-9316-39961d5dbe5c  OS    128      5      50         1
a7a4cb02-210a-4ccf-9744-99e3982148b0  OS    128      5      50         1
56a47549-bceb-48d7-b67e-fd390d083f09  OS    128      5      50         1
1e6ea123-dca0-44e6-8c0c-34c7f543fe82  OS    128      20     100        30
2508eead-dc8f-4df5-baf9-4ce6341f08ed  OS    128      20     100        30

You can also filter by keys. In this example, TYPE=OS returns only joyent brand zones.

[root@headnode (bh1-kvm1:0) ~]# vmadm list -o uuid,type,ram,quota,cpu_shares,zfs_io_priority -s -ram,cpu_shares type=OS ram=~^1
UUID                                  TYPE  RAM      QUOTA  CPU_SHARE  I
O_PRIORITY
c3a03fac-dc2c-42ba-aff7-d6fd7a36967a  OS    1024     15     100        8
56a47549-bceb-48d7-b67e-fd390d083f09  OS    128      5      50         1
a7a4cb02-210a-4ccf-9744-99e3982148b0  OS    128      5      50         1
5c12a3ef-e60c-479a-a6be-ba93712a3893  OS    128      5      50         1
29bdabe1-ff9e-4387-9316-39961d5dbe5c  OS    128      5      50         1
20de2bfc-56de-4cd9-8e25-80b017615788  OS    128      5      50         1
384de5e9-c9d9-4382-9dd3-949ab410be45  OS    128      5      50         1
2508eead-dc8f-4df5-baf9-4ce6341f08ed  OS    128      20     100        30
1e6ea123-dca0-44e6-8c0c-34c7f543fe82  OS    128      20     100        30

Use -p to generate parsable output, without header and with fields separated by :. Notice nics.0.ip to obtain ip address of first NIC.

[root@headnode (bh1-kvm1:0) ~]# vmadm list -p -o uuid,nics.0.ip
ad170576-3ad3-e201-cd89-ad4260c3a3ce:10.0.0.1
c843e9b9-b937-c55f-c8dd-a4f6d1bb2ec0:10.0.0.2
a4212873-e04a-ef84-b85c-c5aea32ce5f8:10.0.0.3
2b99a408-7dfb-445e-cfa1-bed4e14a2509:10.0.0.4
d104e56e-43df-4d0f-b5ce-43f39d0fef46:10.0.0.5
c3a03fac-dc2c-42ba-aff7-d6fd7a36967a:10.0.0.6
8d01d99b-2b18-41c6-8fec-7209e2a9eef3:10.0.0.7
2811071a-5edc-438a-9b7b-e3b81e8829c5:10.0.0.8
5c12a3ef-e60c-479a-a6be-ba93712a3893:10.0.0.9
384de5e9-c9d9-4382-9dd3-949ab410be45:10.0.0.10
20de2bfc-56de-4cd9-8e25-80b017615788:10.0.0.11
29bdabe1-ff9e-4387-9316-39961d5dbe5c:10.0.0.12
a7a4cb02-210a-4ccf-9744-99e3982148b0:10.0.0.13
56a47549-bceb-48d7-b67e-fd390d083f09:10.0.0.14
1e6ea123-dca0-44e6-8c0c-34c7f543fe82:10.0.0.15
2508eead-dc8f-4df5-baf9-4ce6341f08ed:10.0.0.16

Creating new instances

See:

Getting an instances properties

[root@headnode (bh1-kvm1:0) ~]# vmadm get 54f1cc77-68f1-42ab-acac-5c4f64
f5d6e0
{
  "zonename": "54f1cc77-68f1-42ab-acac-5c4f64f5d6e0",
  "zonepath": "/zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0",
  "brand": "joyent",
  "limitpriv": "default,dtrace_proc,dtrace_user",
  "cpu_shares": 100,
  "zfs_io_priority": 30,
  "max_lwps": 2000,
  "max_physical_memory": 256,
  "max_locked_memory": 256,
  "max_swap": 256,
  "creation_timestamp": "2011-11-07T05:57:25.677Z",
  "billing_id": "47e6af92-daf0-11e0-ac11-473ca1173ab0",
  "owner_uuid": "00000000-0000-0000-0000-000000000000",
  "tmpfs": 256,
  "dns_domain": "local",
  "alias": "zone70",
  "nics": [
    {
      "nic_tag": "external",
      "mac": "b2:1e:ba:a5:6e:70",
      "physical": "net0",
      "index": 0,
      "ip": "10.2.121.70",
      "netmask": "255.255.0.0",
      "gateway": "10.2.121.1"
    }
  ],
  "zfs_storage_pool_name": "zones",
  "autoboot": true,
  "uuid": "54f1cc77-68f1-42ab-acac-5c4f64f5d6e0",
  "zoneid": "39",
  "state": "running",
  "quota": 20,
  "customer_metadata": {},
  "tags": {}
}

Lookup an instance by key

By default only the UUID will be returned. Note =~ uses pattern matching. This will look up all zones with an alias matching .*zone7.*.

[root@headnode (bh1-kvm1:0) ~]# vmadm lookup alias=~zone7
54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

You can also return the full json. This is like vmadm get <uuid>, but you don't need to know the UUID ahead of time.

[root@headnode (bh1-kvm1:0) ~]# vmadm lookup -j alias=~zone7
[
  {
    "zonename": "54f1cc77-68f1-42ab-acac-5c4f64f5d6e0",
    "zonepath": "/zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0",
    "brand": "joyent",
    "limitpriv": "default,dtrace_proc,dtrace_user",
    "cpu_shares": 100,
    "zfs_io_priority": 30,
    "max_lwps": 2000,
    "max_physical_memory": 256,
    "max_locked_memory": 256,
    "max_swap": 256,
    "creation_timestamp": "2011-11-07T05:57:25.677Z",
    "billing_id": "47e6af92-daf0-11e0-ac11-473ca1173ab0",
    "owner_uuid": "00000000-0000-0000-0000-000000000000",
    "tmpfs": 256,
    "dns_domain": "local",
    "alias": "zone70",
    "nics": [
      {
        "nic_tag": "external",
        "mac": "b2:1e:ba:a5:6e:70",
        "physical": "net0",
        "index": 0,
        "ip": "10.2.121.70",
        "netmask": "255.255.0.0",
        "gateway": "10.2.121.1"
      }
    ],
    "zfs_storage_pool_name": "zones",
    "autoboot": true,
    "uuid": "54f1cc77-68f1-42ab-acac-5c4f64f5d6e0",
    "zoneid": "39",
    "state": "running",
    "quota": 20,
    "customer_metadata": {},
    "tags": {},
    "ram": 256,
    "type": "OS"
  }
]
[root@headnode (bh1-kvm1:0) ~]#

You can also combine filters. This example looks up all 128M instances with an alias that starts with 'a' or 'b':

[root@headnode (bh1-kvm1:0) ~]# vmadm lookup ram=128 alias=~^[ab]
5c12a3ef-e60c-479a-a6be-ba93712a3893
29bdabe1-ff9e-4387-9316-39961d5dbe5c
20de2bfc-56de-4cd9-8e25-80b017615788

Modifying an instance

Many attributes can be updated live without needing a restart. See vmadm(8) details.

Here we can see that the quota set on the instance is 20GB. The quota value returned by vmadm is the same value returned by zfs.

[root@headnode (bh1-kvm1:0) ~]# vmadm get 54f1cc77-68f1-42ab-acac-5c4f64
f5d6e0 | json quota
20
[root@headnode (bh1-kvm1:0) ~]# zfs get quota zones/54f1cc77-68f1-42ab-a
cac-5c4f64f5d6e0
NAME                                        PROPERTY  VALUE  SOURCE
zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0  quota     20G    local

Updating the quota via vmadm will update the instance as well as the zfs dataset.

[root@headnode (bh1-kvm1:0) ~]# vmadm update 54f1cc77-68f1-42ab-acac-5c4
f64f5d6e0 quota=40
Successfully updated 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
[root@headnode (bh1-kvm1:0) ~]# vmadm get 54f1cc77-68f1-42ab-acac-5c4f64
f5d6e0 | json quota
40
[root@headnode (bh1-kvm1:0) ~]# zfs get quota zones/54f1cc77-68f1-42ab-a
cac-5c4f64f5d6e0
NAME                                        PROPERTY  VALUE  SOURCE
zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0  quota     40G    local
[root@headnode (bh1-kvm1:0) ~]#

You can also do an update using JSON input.

[root@headnode (bh1-kvm1:0) ~]# echo '{"cpu_shares": 100}' | vmadm update 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Successfully updated 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
[root@headnode (bh1-kvm1:0) ~]# vmadm list -o uuid,type,ram,quota,cpu_shares,state,alias alias=zone70
UUID                                  TYPE  RAM      QUOTA  CPU_SHARE  STATE             ALIAS
54f1cc77-68f1-42ab-acac-5c4f64f5d6e0  OS    256      20     100        running           zone70

Other modifications

Some fields (such as nics and disks) can't be updated directly. This is often the case with array fields. They need to have an operation specified.

Modifying NICs

This example uses add_nics and remove_nics.

Here, the zone has one nic currently.

[root@headnode (bh1-kvm1:0) ~]# vmadm get 54f1cc77-68f1-42ab-acac-5c4f64
f5d6e0  | json nics
[
  {
    "nic_tag": "external",
    "mac": "b2:1e:ba:a5:6e:70",
    "physical": "net0",
    "index": 0,
    "ip": "10.2.121.70",
    "netmask": "255.255.0.0",
    "gateway": "10.2.121.1"
  }
]

Then add a nic:

[root@headnode (bh1-kvm1:0) ~]# cat add_nic.json
{
    "add_nics": [
        {
            "physical": "net1",
            "index": 1,
            "nic_tag": "external",
            "mac": "b2:1e:ba:a5:6e:71",
            "ip": "10.2.121.71",
            "netmask": "255.255.0.0",
            "gateway": "10.2.121.1"
        }
    ]
}
[root@headnode (bh1-kvm1:0) ~]# cat add_nic.json | vmadm update 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Successfully updated 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

The zone now has two nics. Note, the pre-existing nic is now flagged as primary automaticaly. In order for the zone to make use of this new nic, it needs to be rebooted.

[root@headnode (bh1-kvm1:0) ~]# vmadm get 54f1cc77-68f1-42ab-acac-5c4f64
f5d6e0  | json nics
[
  {
    "nic_tag": "external",
    "mac": "b2:1e:ba:a5:6e:70",
    "physical": "net0",
    "index": 0,
    "ip": "10.2.121.70",
    "netmask": "255.255.0.0",
    "gateway": "10.2.121.1",
    "primary": true
  },
  {
    "nic_tag": "external",
    "mac": "b2:1e:ba:a5:6e:71",
    "physical": "net1",
    "index": 1,
    "ip": "10.2.121.71",
    "netmask": "255.255.0.0",
    "gateway": "10.2.121.1"
  }
]

This example uses update_nics to modify the IP of a nic. NICs are identified by the mac.

[root@headnode (bh1-kvm1:0) ~]# cat update_nic.json
{
    "update_nics": [
        {
            "mac": "b2:1e:ba:a5:6e:71",
            "ip": "10.2.121.72"
        }
    ]
}
[root@headnode (bh1-kvm1:0) ~]# cat update_nic.json | vmadm update 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Successfully updated 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

And finally, how to remove a nic. Again, the zone must be rebooted before it will stop being used by the zone.

[root@headnode (bh1-kvm1:0) ~]# echo '{"remove_nics": ["b2:1e:ba:a5:6e:71"]}' | vmadm update 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Successfully updated 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Modifying Disks

Add a disk.

[root@headnode (bh1-kvm1:0) ~]# cat add_disk.json
{
  "add_disks": [
    {
      "boot": false,
      "model": "virtio",
      "block_size": 8192,
      "size": 40960
   }
 ]
}
[root@headnode (bh1-kvm1:0) ~]# vmadm update 54f1cc77-68f1-42ab-acac-5c4
f64f5d6e0 -f add_disk.json
Successfully updated 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Results.

[root@headnode (bh1-kvm1:0) ~]# vmadm get 54f1cc77-68f1-42ab-acac-5c4f64
f5d6e0  | json disks
[
  {
    "path": "/dev/zvol/rdsk/zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0-disk0",
    "boot": true,
    "model": "virtio",
    "media": "disk",
    "size": 40960,
    "zfs_filesystem": "zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0-disk0",
    "zpool": "zones",
    "compression": "lz4",
    "block_size": 8192
  },
  {
    "path": "/dev/zvol/rdsk/zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0-disk1",
    "boot": false,
    "model": "virtio",
    "size": 40960,
    "zfs_filesystem": "zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0-disk1",
    "zpool": "zones",
    "compression": "lz4",
    "block_size": 8192
  }
]

Remove a disk. Note: This will permenantly destroy any data in the removed volume.

[root@headnode (bh1-kvm1:0) ~]# echo '{"remove_disks": ["/dev/zvol/rdsk/zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0-disk1"]}' | vmadm update 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Successfully updated 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Add a CDROM

You can do the same thing with CD-ROMs.

{
  "add_disks": [
        {
        "boot": false,
        "model": "ide",
          "media": "cdrom",
          "path": "/en_sql_server_2012_standard_edition_with_sp1_x64_dvd_1228198.iso"
        }
      ]
}

Managing Instances

Stop an instance

[root@headnode (bh1-kvm1:0) ~]# vmadm stop 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Succesfully completed stop for 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Start an instance

[root@headnode (bh1-kvm1:0) ~]# vmadm start 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Successfully started 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Reboot an instance

[root@headnode (bh1-kvm1:0) ~]# vmadm reboot 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Succesfully completed reboot for 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Delete an instance

Note: This will permenantly delete all data in that zone.

[root@headnode (bh1-kvm1:0) ~]# vmadm delete 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Successfully deleted 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Migrate an Instance to a Different Compute Node

A zone is composed of two components.

  • zone config
  • zfs datasets

Copying these to a different compute node will make a duplicate on the destination compute node. You can then delete the original to complete the migration.

There are two ways to migrate a zone.

  • manual migration
  • vmadm send/vmadm receive

Whichever method you choose, you will need to manually create the cores dataset before booting the zone on the desitination. See below.

Before starting, it's a good idea to make sure the origin image for the zone is installed on the desitation CN.

Manual Migration

First, export the zone config.

zonecfg -z <uuid> export > <uuid>.zcfg

On the destination CN, import it.

zonecfg -z <uuid> -f <uuid>.zcfg

The zone will now be in the configured state as reported by zoneadm.

Secondly, use zfs send to copy the datasets. Note that for KVM instances the disks are outside the zone root and must be sent individually. For OS, LX, or BHYVE instances you can send just the top level zone dataset.

You'll need to create a snapshot first.

zfs snapshot -r zones/<uuid>@migration

When you send, it's preferable to pre-load the image on the destination compute node and send an incremental from the zone's image. That is, the image the zone was created from (e.g., base-64).

zfs send -R -I zones/<image_uuid>@final zones/<uuid>@migration | ssh -c aes128-gcm@openssh.com <ip_address> 'zfs recv -d zones'

If for some reason you're unable to get the image on the destination, you can do a full send.

zfs send -R zones/<uuid>@migration | ssh -c aes128-gcm@openssh.com <ip_address> 'zfs recv -d zones'

See zfs(8) for information about additional send options. In particular, read up on resumable transfers. This helps if you have a large dataset and/or an unreliable network.

On the destination compute node, attach the zone. This will move the zone from configured to stopped. Note: You don't need to explictly copy or create the cores dataset because it gets created at attach time.

zoneadm -z <zone_uuid> attach

The zone should be ready to boot up.

Here is a demonstration of a manual migration.

Using vmadm send/receive

While vmadm send is easier to use, vmadm send and vmadm receive are experimental, unsupported, undocumented-in-the-manpage features. This is not a committed interface. The behavior may be modified or removed entirely without warning.

Use at your own risk.

vmadm send <uuid> |ssh -c aes128-gcm@openssh.com 'vmadm receive'

Re-create the cores dataset

As mentioned above, you'll need to manually recreate the cores dataset. See OS-6789 for more details.

zfs create -o quota=1000m -o compression=gzip -o mountpoint=/zones/:zone_uuid/cores zones/cores/:zone_uuid

Remove the Instance from the source compute node

# vmadm delete 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
Successfully deleted 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Here is a demonstration of using vmadm send.

Troubleshooting Migration Failure

If the migration fails, you may need to manually destroy the ZFS dataset for the zone on the destination machine before you can try sending the instance again. The below error is a symptom of this:

[root@headnode (bh1-kvm1:0) ~]# vmadm send 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0 |ssh 10.0.1.4 vmadm receive
vmunbundle exited with code 1
Successfully sent 54f1cc77-68f1-42ab-acac-5c4f64f5d6e0

Check to see if the ZFS dataset exists on the destination, then remove if it exists

[root@destinationNode ~]# zfs list
NAME                                           USED  AVAIL  REFER  MOUNTPOINT
zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0     651K  10.0G   651K  /zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0
[root@destinationNode ~]# zfs destroy zones/54f1cc77-68f1-42ab-acac-5c4f64f5d6e0