Category Archives: solaris

network load balancing with Solaris

I know that everybody forgot about this blog but I’ll try to liven it up.
Some days ago at Moscow OpenSolaris User Group meeting was 2 presentations about network. First one was about Crossbow and all around and second one was about network load balancing … with Linux and FreeBSD and nothing about Solaris.
So as nobody knew anything about network load balancing with Solaris i have decided to write about it.
There are some solutions for network balancing in FreeBSD and Linux and really nothing like this exists in Solaris 10.
Time stands still and Solaris 11 Express is already exists and OpenSolaris already doesn’t exists 🙂
It is in the OpenSolaris appeared support ILB ( Integrated Load Balancer ) and now it is included to Solaris 11 Express.
So let’s try to configure Solaris ILB.
First of all you need to check that ilb is installed and if no, install it.

root@solaris:~# pkg search ilbadm
INDEX      ACTION VALUE           PACKAGE
basename   file   usr/sbin/ilbadm pkg:/service/network/load-balancer/ilb@0.5.11-0.151.0.1
root@solaris:~# man pkg
root@solaris:~# pkg install pkg:/service/network/load-balancer/ilb@0.5.11-0.151.0.1
               Packages to install:     1
           Create boot environment:    No
               Services to restart:     1
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1       11/11      0.2/0.2

PHASE                                        ACTIONS
Install Phase                                  38/38 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 

PHASE                                          ITEMS
Reading Existing Index                           8/8 
Indexing Packages                                1/1
root@solaris:~#

Now ILB installed and we can configure load balancing.
First of all let’s start ilb daemon. But before this we ought to start ip forwarding. If you forget to do so ilbadm output would be really helpful.

root@solaris:~# ilbadm show-rule
ilbadm: socket() failed
root@solaris:~#

So let’s start ip forwarding and ilb daemon.

 root@solaris:~# svcadm enable svc:/network/ipv4-forwarding
root@solaris:~# svcadm enable svc:/network/loadbalancer/ilb:default

ILB supports 3 modes :
– DST
– Full NAT
– Half NAT

In the DSR ( Direct Server Return ) mode, ILB balances the incoming requests to the back-end servers, and letting the return traffic from the servers bypass the load balancer by being sent directly to the client.
NAT-based load balancing involves rewriting of IP header information,and handles both the request and the response traffic.
In Full NAT mode ILB involves rewriting of both source and destination IP fields, making it appear to the back-end servers that all connections are originating at the load balancer. Clients will also receive packets from preconfigured IP range.
In the half-NAT mode ILB rewrites only destination IP address.

I have used 3 servers to try ILB. Two of them i used as back-end servers. I have started nginx there. There IPs : 192.168.57.102,192.168.57.103. ILB was started at the third server with IP 192.168.57.101.

First of all I’ve configured server group, which will process http traffic :

root@solaris:~# ilbadm create-servergroup -s servers=192.168.57.102:80,192.168.57.103:80 websg
root@solaris:~# ilbadm show-servergroup
root@solaris:/tmp# ilbadm show-sg
SGNAME         SERVERID            MINPORT MAXPORT IP_ADDRESS
websg          _websg.0            80      80      192.168.57.102
websg          _websg.1            80      80      192.168.57.103

And then I’ve configured balance rule. First of all let’s try full NAT.

root@solaris:/tmp# ilbadm create-rule -e -i vip=192.168.56.101,port=80 -m lbalg=rr,type=NAT,proxy-src=192.168.56.101 -o servergroup=websg webrule
root@solaris:/tmp# ilbadm show-rule
RULENAME            STATUS LBALG       TYPE    PROTOCOL VIP         PORT
webrule             E      roundrobin  NAT     TCP  192.168.56.101  80

And that’s all!
I have configured that http traffic to 192.168.56.101 ( virtual IP ) will be load balanced across 2 servers using round robin algorithm. Clients will receive replies from IP 192.168.56.101.
Let’s change ILB mode from full NAT to Half NAT.

ilbadm  delete-rule -a
ilbadm create-rule -e -i vip=192.168.56.101,port=80 -m lbalg=rr,type=h -o servergroup=websg webrule
root@solaris:/tmp# ilbadm show-rule
RULENAME            STATUS LBALG       TYPE    PROTOCOL VIP         PORT
webrule             E      roundrobin  HALF-NAT TCP 192.168.56.101  80

This is really easy to configure. Of course there are a lot of options. For example you can select load balancing algorithm Moreover, ILB offers an optional server monitoring feature that can provide server health checks. Another great options are : Session persistence and Connection draining
More details about the ILB can be read on wiki
I’m not really an expert in load balancers but ILB looks much more useful and functional than FreeBSD or Linux analogs.

UFS internals

I wrote about fsdb usage for vxfs some time ago. But now rather often questions about ufs asked, in spite of file system age and rather good manual. So I decide to write about fsdb for ufs.

When you use fsdb you must be very accurate. One mistake and all your data lost.
Run fsdb with write permissions :

fsdb -o w /dev/rdsk/_your_file_system_

Fsdb syntax is rather unusual and similar to adb. All commands start with “:”.
For example 

:ls

command.
Also will work 

 :ls /

But you must remember that / is a root of file system, which you open with fsdb, and if you, for example, open var, it will be root of var file system, not a root file system.
:ls have only 2 options. -l will return list of files with inode numbers and -R will do recursive listing.
You can use :cd command to change directory.
Very useful command :base can change numeral system from hexadecimal ( by default) to octal :

:base=0t10

In fsdb use concept of “dot” . First of all you must select an object, to work with, and so give to “dor” value – address of this object. And all following commands will work regarding value of “dot”.
So if you decide to do something with inode 5, at first you must select it.
It can be done so :

5:inode

And after this you can display info about this inode : 

 ?i

Or you can unite this to commands to one :

5:inode?i

/dev/rdsk/c1t3d0s0 > 5:inode?i
i#: 5 md: d---rwxr-xr-x uid: 0 gid: 3
ln: 4 bs: 2 sz : c_flags : 0 200

db#0: 2fc
accessed: Wed Apr 29 16:24:50 2009
modified: Wed Feb 25 13:40:05 2009
created : Wed Feb 25 13:40:05 2009

I think that output is rather logical and can be easily understood. I’d like to look more attentively at the value of db field. DB is a direct block. Actually speaking in db you can find file data. I hope that everybody remember that inode in ufs consists of 12 direct blocks, 3 indirect blocks. IB, is a block, that consists as much as 2048 links to other blocks, and no data at all. It being known that only first one consists of links to db, and if it isn’t enough – second IB will be used. This IB also known as double indirect block. It consists of 2048 links to ib, which consists of links to db. As third ib is triple indirect block and I think you can understand what contains in it by yourself.
Going back to my output we see, that inode contain only one db ( zero ) in block 2fc.,
But we look aside.
Type of inode, file or directory can be easily understood from md (
mode ) field. If it contains a flag than it is directory, if not – file.
If it turn out that this is a directory, :ls will show it’s content.

As everybody remember directory in ufs nothing else but array, where correspondence of inode to file name listed.
After you select inode, which prove to be a directory, you can list and modify these fields.
Actually speaking :ls is showing them for you, but in another order. It can be done in the same order :

0:dir?d
1:dir?d
2:dir?d

If you are rather lazy to write next (3:dir?d) command you can just
press Enter and command :dir?d will be done for the next element.
If you so lazy even to press Enter 20 times you can display 20 elements from 0 block from 2 inode using this command sequence :

2:ino; 0:db:block,20?d

Or just 

308:fragment,20?d

If you decide that any field ( let it bee 5’th ) must link not to the 22
inode, but, for example, to the 66, you can change it yourself by this
command :

5:dir?d=42

because 42 – 66 in hex
Note, that file name will stay the same.

5:dir:nm="test"

will also change the name.
I think now you can do with directories everything you want. Lats go to files.
Everything pretty the same.

/dev/rdsk/c1t3d0s0 > :ls -l /etc/passwd
/etc:
i#: a317 passwd
/dev/rdsk/c1t3d0s0 > 0xa317:inode?i
i#: a317 md: ----rw-r--r-- uid: 0 gid: 3
ln: 1 bs: 2 sz : c_flags : 0 395

db#0: 6a8db
accessed: Wed Apr 29 16:20:06 2009
modified: Mon Apr 27 11:59:48 2009
created : Mon Apr 27 11:59:48 2009

/dev/rdsk/c1t3d0s0 > 0:db:block,100/c

And we have content of /etc/passwd on the screen. Now the question is how we can change it?
Easily!
It can be done by some ways.
To fill some pert of file with zero’s :

6a889:fragment,4=fill=0x0

Or just to write some date to any address :

1aa22400=0xffff

If you like to write text, it can be done with this command :

1aa36c00="root"

So, which way to remove inode at not mounted file system is the easiest one? Of cause by clri command 🙂

size of automaped dir

I founded this interesting behaviour then trying to collect Explorer by nfs.
If you want to know size of automounted filesystem you can see unexpected results:

root@serv # df -h /net/hostname/sharename
Filesystem size used avail capacity Mounted on
-hosts 0K 0K 0K 0% /net/hostname/sharename

but if you do:

root@serv # df -h /net/hostname/sharename/.
Filesystem size used avail capacity Mounted on
hostname:/sharename 128G 75G 53G 59% /net/hostname/sharename