Skip to end of metadata
Go to start of metadata

Exposing Additional NICs in the Global Zone

Additional vnics can be added in the global zone. To create a vnic named storage0:

The -v 128 above creates the vnic on VLAN ID 128. For a vnic not on a VLAN, omit the -v option.

Now plumb and bring up the interface as you would any interface:

Or with a DHCP IP:

Persisting vnics across reboots

First determine the mac address of the physical nic:

Now add the config options for storage0 to /usbkey/config:

NOTE: the omission of the "0" in "storage_nic" below is NOT a typo. That line defines a nic_tag. The other lines define an instance (instance 0) of a vnic on the hardware referenced by that nic_tag.

You can omit the storage0_vlan_id option if you don't want storage0 on a VLAN. For a DHCP IP with no VLAN:

If you would like to make the changes in /usbkey/config active without a reboot, you will need to make them visible to "sysinfo" by running the following command:

Exposing Additional NICs in VMs

Additional physical NICs can be exposed to the VMs, via VNICs, running under your SmartOS hypervisor.

First determine the mac address of the physical nic:

In this example the mac address for the nic I want to expose is '8:0:27:19:64:7d' and is associated with link 'e1000g1'.

Next, modify /usbkey/config which holds the persistent network configuration for your hypervisor. Add the following line:

Finally in the JSON spec for your VM, reference this new tag under the 'nics' section:

This updated JSON spec will create a VM with multiple NICS, exposing the physical NIC tagged with external to the VM/Zone. See managing VMs with vmadm for more information on updating existing vms.

Adding VRRP nics to VMs

To provision a VM with a VRRP nic, you must set the vrrp_vrid (router ID) and vrrp_primary_ip attributes in that nic:


  • vrrp_primary_ip must be the IP address of one of the other nics in the system, since this address will be used to send router advertisements.
  • No MAC address is necessary for the VRRP nic, since its MAC address will be based on the router ID
  • For the time being, the NIC with the ip matching the vrrp_primary_ip needs to have allow_ip_spoofing (joyent-live Issue 136)
  • This does not work with kvm zones

Logging into the VM, you can see that net0 has the VRRP flag set. The interface isn't up yet - that will be handled by vrrpd, which handles bringing up and down VRRP vnics.

Now create the router with vrrpadm:

vrrpadm shows that the router is up, and is the master.

ifconfig shows that net0 is now UP:

You should now be able to ping this VM on both its primary and virtual IP.

Link Aggregations in the Global Zone

Link aggregations require SmartOS build 20121018 or later

First, determine the MAC addresses of the nics you want to include in the aggregation:

Now add a <aggregation name>_aggr config key to /usbkey/config. The value of that key is a comma-separated list of the MAC addresses of the nics in the aggregation:

Save and reboot. Once the system is back up, you should see the aggregations with sysinfo:

dladm should also show the aggregation:

VM nics provisioned on the admin and storage nic tags will now be over aggregation aggr0.

To manually create a link aggregation with the dladm create-aggr command in the Global Zone without rebooting, you will first need to disable the lldp/server service. By default lldpd will have a lock on all otherwise unused devices. After you've created the aggregation, you can re-enable it.

Modifying the MTU

MTU awareness in the SmartOS tools (nictagadm, vmadm, etc.) is in SmartOS build 201400626 or later

By default, all VMs that do not have an explicit MTU requested have their VNICs match the MTU of the physical nic of their nic tag. In addition, any VNIC that's created has a valid range of an MTU ranging from 1500 to the current MTU of the physical NIC. The best way to view this is by running the dladm command as follows:

The MTU of a physical device or VNIC only impacts the maximum MTU you can use on an IP interface. Even though the smallest MTU e1000g0 supports is 1500, the IP stack can use a lower MTU.

A nic tag now has the concept of a MTU. If unset, it defaults to the default value of the NIC, usually 1500. The nictagadm command can be used to update an existing nic tag with a MTU or set one when a new one is created. When the system next boots, it will set the MTU of the physical nic to the maximum of all of the MTUs specified by the nic tags for that NIC. See the following examples below:

Updating the MTU for VMs

vmadm has a new property 'nics.*.mtu'. If this property is set, it will always try to set the MTU to the specified value when the VM is booted. If that MTU cannot be honored, then it will cause the boot process to fail explicitly rather than come up with a different MTU. If an MTU is not specified, then it will always default to the current MTU of the physical device that corresponds to the nic tag.

Updating the MTU for aggrs and global zone vnics

You can also specify the MTU for an aggregation and a VNIC in the configuration file. Recall the storage0 and aggr0 example from earlier. If you wanted to set storage0's MTU to 9000, you would add line 'storage0_mtu=9000'. You also do something similar for an aggregation. That makes the whole stanza for our aggregation and vnic look like:

Why don't all my NICs appear in ifconfig?

All NICs are enabled, but they aren't plumbed up in the global zone.

They can be used in local zones without being plumbed in the global zone.

NICs that aren't plumbed in the global zone will not appear in ifconfig.

To show all physical NICs from the global zone:

You can also plumb additional NICs in the global zone with ifconfig if needed, (this will not persist across reboots - see above):

DHCP Support

Zone NICs will by default have dhcp hosting blocked on them and need to be opened up before a DHCP server will operate correctly.

You can confirm whether the dhcp protection bit is set with the following command, in this example net0 has the dhcp-nospoof protection bit set while net1 has it cleared as is required for serving DHCP on the net1 interface.

When creating a new VM the following property will disable the dhcp-nospoof bit.

To update this flag on an already configured zone you can use the vmadm update functionality as follows (in this case updating interface with mac 00:53:37:00:db:08).

NAT and other crazy tricks

NAT using Etherstubs

Gist about NAT

Gist about etherstubs

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Feb 06, 2013

    Can I use vrrp similar to that of in Linux, setting up a floating IP shared by two zones?

    1. Feb 06, 2013

      tofu, you'll have more luck getting answers on the smartos-discuss mailing list.