

LX Branded Zones¶

Background¶

Bryan's talk (both entertaining and informative) (slides)

Getting Started¶

this worked on 2017-10-30
log into the SmartOS machine as user root
imgadm avail | egrep lx; (I chose e74a9cd0-f2d0-11e6-8b69-b3acf2ef87f7 from https://docs.joyent.com/public-cloud/instances/infrastructure/images/debian)
imgadm import e74a9cd0-f2d0-11e6-8b69-b3acf2ef87f7 to download the snapshot and install in zones pool
imgadm show e74a9cd0-f2d0-11e6-8b69-b3acf2ef87f7 to figure out the kernel version

this command will create a running virtual machine, complete with disc/memory/network/cpu:

vmadm create <<EOL
    {
     "brand": "lx",
     "kernel_version": "3.16.0",
     "image_uuid": "e74a9cd0-f2d0-11e6-8b69-b3acf2ef87f7",
     "alias": "pg",
     "hostname": "pg",
     "max_physical_memory": 8192,
     "quota": 3000,
     "resolvers": ["192.168.180.1", "8.8.8.8"],
     "nics": [
      {
        "nic_tag": "v",
        "ip": "192.168.180.182",
        "netmask": "255.255.255.0",
        "gateway": "192.168.180.1",
        "vlan_id": 180
      }
     ]
    }
EOL

vmadm list gave me the ID for my running instance:

UUID                                  TYPE  RAM      STATE   ALIAS
9f7dfe6d-d6ec-e108-d3be-ce1638b5a7c2  LX    8192     running pg

I had difficulty logging into that machine (zlogin 9f7dfe6d-d6ec-e108-d3be-ce1638b5a7c2 should have worked, too), this is why I created this writeup in the first place. This is how you can get yourself into the newly created machine
```
cd /zones/9f7*/root/root/.ssh && cp ~/.ssh/id_pub.rsa authorized_keys
```
Log in
```
ssh root@192.168.180.182
```
final note: my SmartOS machine was on vlan 180, if you don't do vlans, leave out the vlan_id line and preceding comma, probably your interface is called admin; change paramaters as desired

Platform Version¶

For best results, run the latest SmartOS release available.

Import images¶

The lx zone images are on images.joyent.com so make sure that's one of your image sources. Then you can import images:

imgadm sources -a https://images.joyent.com
imgadm avail | grep lx-dataset   # See available images
imgadm import 05140a7e-279f-11e6-aedf-47d4f69d2887    # ubuntu-16.04 20160601

Create A Zone¶

Create a zone (replace the nic values with ones appropriate for your network)

If you're not sure what these values mean or what units they are in, check out the vmadm man page

{
  "alias": "lxtest",
  "brand": "lx",
  "kernel_version": "4.3.0",
  "max_physical_memory": 2048,
  "quota": 10,
  "image_uuid": "05140a7e-279f-11e6-aedf-47d4f69d2887",
  "resolvers": ["8.8.8.8","8.8.4.4"],
  "nics": [
    {
      "nic_tag": "external",
      "ips": ["dhcp"],
      "primary": true
    }
  ]
}

Cool Tech Demos¶

A selection of impressive demonstrations of the LX brand's capabilities. Please add your favorites here.

Show your friends¶

X11 forwarded Firefox and Thunderbird in lx32 zone
Plex Media Server usable on lx32 and lx64
dtrace a linux binary
Video of docker in SDC (coming soon to the JPC)
(your favorite demo here)

Networking¶

AF_INET and AF_INET6 mostly working (2014-12-19). Still have some sockopts missing.

ipadm is not working inside the lx brand, /native/sbin/ifconfig is usable.

Enabling IPv6 SLAAC auto configuration¶

Add addrconf to the ips array for the NIC to perform SLAAC and/or DHCPv6 as appropriate.

X11 forwarding¶

This is working fine on both 32-bit and 64-bit. After pmooney's AF_INET6 commit this is working fine out of the box!

Debugging¶

If debugging LX branded zones, first watch this video: Debugging LX branded zones on SmartOS

DTrace¶

Enabling `lx_debug` probes¶

As the video above mentioned, probes associated with lx_debug (that is, output associated with the LX_DEBUG environment variable) are turned on it LX_DTRACE is set to a non-zero value in the environment. However, this is not helpful if in a context that is hard to set environment variable (e.g., early in boot). The video above neglected to mention how to enable DTrace probes in this case. To do this, you must first change the disposition of the brand to enable these by default by modifying the binary. From the global zone, do the following:

cp /usr/lib/amd64/lx_brand.so.1 /var/tmp
cp /usr/lib/amd64/lx_brand.so.1 /var/tmp/lx_brand.so.1.$$
echo "lx_dtrace_lazyload?W 0" | mdb -w /var/tmp/lx_brand.so.1.$$
mount -O -F lofs /var/tmp/lx_brand.so.1.$$ /usr/lib/amd64/lx_brand.so.1

(If debugging 32-bit, the lx_brand.so.1 that should be copied and mounted over should be /usr/lib/lx_brand.so.1 not /usr/lib/amd64/lx_brand.so.1.)

At that point, lx``pid``:::debug probes should show up as processes start, which will allow you to see where things are failing:

dtrace -Zqn lx*:::debug'{printf("%s", copyinstr(arg0))}'

(String size and switch rate should be adjusted up if truncated strings or drops are seen.)

When debugging a new distro or otherwise debugging the death of processes before you can login, you can

hunting for unsupported syscalls¶

With a dtrace script, lxunsup.d

#!/usr/sbin/dtrace -s

#pragma D option quiet

BEGIN
{
    printf("%6s %20s %s\n", "PID", "NAME", "CALL")
}

sdt:lx_brand::brand-lx-unsupported
{
    printf("%6d %20s %S\n", pid, execname, stringof(arg0));
}

And some 1-liners.

dtrace -n 'sdt:lx_brand::brand-lx-unsupported {@num[execname,stringof(arg0)] = count()}'

dtrace -n 'lx-syscall::setsockopt:entry /arg1 == 0 && arg2 == 35/ { trace(arg4) }'

Harmless unsupported syscalls

prctl(PR_SET_DUMPABLE)
allow process to enable coredumps, should have no functional impact
prctl option 4
same as PR_SET_DUMPABLE
set_robust_list / get_robust_list
part of libc initialization, but if the feature is never used, then it doesn't really end up mattering.
futex 0x9
usespace mutex call, mostly harmless
socket flag MSG_CMSG_CLOEXEC
auto close fd's... dbus spamsthese a lot. Looks like they get closed anyway.
ioctl 0xc020660b (fiemap)
used to find holes in files by querying the inode extend info, apt tries to use this. have not noticed any impact.

Probing socket options¶

This is handy for figuring out why a networking app is not behaving properly.

D script lxsockopt.d

#!/usr/sbin/dtrace -s

#pragma D option quiet

/*
 * PROTO values can be found here:
 * https://en.wikipedia.org/wiki/List_of_IP_protocol_numbers
 * SOCKOPT values can be found here:
 * https://github.com/joyent/illumos-joyent/blob/master/usr/src/lib/brand/lx/lx_brand/common/socket.c
 */
BEGIN
{
    printf("%6s %20s %5s %s\n", "PID", "NAME", "PROTO", "SOCKOPT");
}

lx-syscall::setsockopt:entry
{
    printf("%6d %20s %5d %d\n", pid, execname, arg1, arg2);
}

Or, a 1-liner

dtrace -n 'lx-syscall::setsockopt:entry { @num[execname,arg2] = count()}'

Docker¶

Bryan's talk and demo of sdc-docker

Vagrant¶

Initial support for lx brand is complete:

Random Notes¶

Using /native binaries to work¶

SmartOS (in fact, the entire SmartOS userland) binaries are mounted in /native in all LX zones. They mostly work, but some native binaries execute other native binaries. (For example, arp calls netstat.) For programs written in C, this can be fixed, so please file a ticket if it doesn't work in a recent release.

Creating your own zone dataset¶

Joyent CI builds¶

If you run into a new bug, someone might ask you to try the latest CI build after it's been fixed. This is how.

The Joyent CI system constantly rebuilds the platform tarball when there are new commits. You can extract that image and put into place on e.g. an existing SmartOS USB stick to obtain the latest and greatest code from Joyent.

You can download that platform tarball like so:

MANTA_URL=https://us-east.manta.joyent.com
LATEST=$(curl ${MANTA_URL}/Joyent_Dev/public/builds/platform/master-latest)
curl -O ${MANTA_URL}${LATEST}/platform/platform-master-${LATEST##*-}.tgz

To update an existing SmartOS USB stick:

Extract that tarball onto your USB stick (you'll get a directory named "platform-<datestamp>")
Move aside the directory named "platform" to e.g. "platform-orig"
Rename "platform-<datestamp>" to "platform"
Boot from your USB stic