Network speed problem/tuning

Dev Shed Forums Sponsor:

March 6th, 2012, 11:16 AM

zby

Contributing User

Join Date: Jul 2003

Location: Prague, Czech Republic

Posts: 867

Time spent in forums: 6 Days 23 h 41 m 23 sec
Reputation Power: 69

Network speed problem/tuning

Hi,
I have two servers and try to test one of its netcards speed.
Why the speed drops dramaticly once I start a test with more than 8 threads? And why at this point the speed of the threads is not well balanced as it is with less than 8 threads?

Servers: 2
Model: HP BL685C G7
CPU: AMD Opteron (6174) 2,2 GHz, 12 core, 512KB cache
CPUs: 4
RAM: 64GB
DISK: 73GB (single)
Controller: HP Smart Array P410i, firmware 5.14
RAID mode: 1(1+0)
NET: 8x Emulex OneConnect 10Gbps
OS: SUSE Enterprise 11 x86_64
Kernel: 2.6.32.54-0.3
Swap: 3GB

Partitions: /, /boot, /data, /home, /opt, /tmp, /usr, /var
Enabled netcard: 1 (eth0)
netcard module: hp-be2net (ServerEngines BladeEngine 10Gbps)
Netcard module ver: 2.104.225.7
Net. application: iperf

Server: iperf -s -w 2M
Client: iperf -c 10.200.140.157 -t 30 -i 2 -w 2M -P XY (see the attached zip with traces)

Sorry, cannot attach the /etc/sysctl.conf at the moment ...
Thank you.

Attached Files

iperf_test.zip (18.2 KB, 60 views)

March 6th, 2012, 08:16 PM

SimonJM

Contributing User

Join Date: Mar 2006

Posts: 2,111

Time spent in forums: 1 Month 1 Week 1 Day 7 h 43 m 38 sec
Reputation Power: 1485

Can't really comment on the network performance, but why (how, even!) are you running RAID with a single disk?

__________________
The moon on the one hand, the dawn on the other:
The moon is my sister, the dawn is my brother.
The moon on my left and the dawn on my right.
My brother, good morning: my sister, good night.
-- Hilaire Belloc

March 7th, 2012, 09:01 AM

zby

Contributing User

Join Date: Jul 2003

Location: Prague, Czech Republic

Posts: 867

Time spent in forums: 6 Days 23 h 41 m 23 sec
Reputation Power: 69

Hi and thank you for a reply.
The disk is a single one and was used to replace the RAID1 pair. That I should have clarified before. Sorry.
The servers have been handed over to me by our customer to perform a network throughput test.
So far I can access the servers only via ssh. Not physicaly.

What realy botheres me is the dramatic drop of speed with the number of ipref's threads above 8 (the server reports 8 cards - may that be a clue). From that point the speed remains almost the same no matter how many addidtional threads are there (I tested with 10, 20, 40, 60, 120).

I started off with sysctl.conf like this:

Code:

vm.swappiness = 10

net.core.rmem_default = 524287
net.core.rmem_max = 524287
net.core.wmem_default = 524287
net.core.wmem_max= 524287
net.core.netdev_max_backlog= 300000
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 10000000 10000000 10000000
net.ipv4.tcp_wmem = 10000000 10000000 10000000
net.ipv4.tcp_mem = 10000000 10000000 10000000

and ended up with this:

Code:

vm.swappiness = 5

net.core.rmem_default = 52582912
net.core.rmem_max = 52582912
net.core.wmem_default = 52582912
net.core.wmem_max= 52582912
net.core.netdev_max_backlog= 300000
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 10240 87380 52582912
net.ipv4.tcp_wmem = 10240 87380 52582912
net.ipv4.tcp_mem = 10240 87380 52582912
net.ipv4.tcp_no_metrics_save = 1

Still no change in the whole behavior. All that has changed is the amount of consumed memory. That, in the end, was at 80%.

Last edited by zby : March 7th, 2012 at 09:04 AM.

March 7th, 2012, 12:09 PM

SimonJM

Contributing User

Join Date: Mar 2006

Posts: 2,111

Time spent in forums: 1 Month 1 Week 1 Day 7 h 43 m 38 sec
Reputation Power: 1485

No way am I a network guru, let me state that up front, so be warned!
Initial thoughts after having had a look at a few of those logs you attached in your original post:
Are all 8 ports aggregated/trunked to present a single IP address, I am guessing that is so due to the lack of different IPs in the logs.
I am also guessing that the NIC supports a total of 10GB/s.
Generally, with ethernet, you'll get a maximum of 60-80% throughput, of rated levels. Any more than that and you will start having issues with packet collisions, etc. which force re-sends.
You seem to be getting a decent, sustained 8-9GB/s whilst thowing a lot of data across the network, until you start over-loading the card.
Is that really the sort of pattern you'd expect to see? Mass traffic involving large amounts of data?

March 7th, 2012, 03:40 PM

zby

Contributing User

Join Date: Jul 2003

Location: Prague, Czech Republic

Posts: 867

Time spent in forums: 6 Days 23 h 41 m 23 sec
Reputation Power: 69

The story behind the scene is not short as usual ... So I'll try to express all simply.

What I have heard is that their intention is to use the servers to host virtual ones OR to use it as powerful application servers. In the second case the JBoss app on one of the machines will use a connection pool to Oracle DB. As I am not the JBoss guru I think that more than 5 connections to one IP will be set. But most likely more than 10 I guess. The overall speed of a database server is another story, but not on the scene in my particular case.

The linux kernel reports 8 ethernet cards out of which only one is configured. The rest is disabled. How are the "ports" organized in the schasis, that I don't know. I'll ask.

Our customer wishes to know if the card can handle cirka 7-9GBits/s. That is I think possible, but it's necessary to explain the limits. Today I watched the traffic of "iperf" via "iptraf" and the max. speed I saw was a bit under 7GBits/s.

All of my tests were directed to the first ethernet of the second machine. So to just one IP adress.

Yes, the NIC supports 10GBits/s. Ethtool eth0 reports that speed.

Both of the serverst eth0 have their MTUs set to 1500. The default.

March 7th, 2012, 03:59 PM

SimonJM

Contributing User

Join Date: Mar 2006

Posts: 2,111

Time spent in forums: 1 Month 1 Week 1 Day 7 h 43 m 38 sec
Reputation Power: 1485

That speed does suggest around the max you'll get from a 10Gb/s NIC over ethernet.
Is that required rate (7-9 Gb/s) sustained or burst rate? In my view (remember, NOT a network guru!) you'll be hard pressed to get ethernet up to that sort of efficiency.
What is reported, and what is there, configured, linked and connected on the card may vary depending on the kernel and drivers. 8 rj45 ports on a single card does seem like quite a few (in my day a 4 port 10/100/1000 card was top of the line, so take that as it comes!)

Having done a little googling, it looks like the 8x part of the name is the speed of the PCI slot required, not the number of interfaces provided - images suggest that it is a dual-port card.

March 8th, 2012, 12:59 PM

zby

Contributing User

Join Date: Jul 2003

Location: Prague, Czech Republic

Posts: 867

Time spent in forums: 6 Days 23 h 41 m 23 sec
Reputation Power: 69

The server houses two network chips each capable of max. 10Gb/s.

I set MTU to 9000 today and the speed jumped up a bit, but not significantly. What was a benefit was a bit better "stability". The data-flow was a bit more even, but the limit of 6.8 Gb/s reported by "iptraf" a couple of days ago was not breached.

We do not want to talk about the rest of the equipment on the route to other resources. It's clear it must work at the same speed, otherwise ...

We may have a chance to enable the second chip to test. What I do not know is which "card" belong to what chip as I cannot access the chassis. Could the "lspci" be of a help?

Network speed problem/tuning

Developer Shed Advertisers and Affiliates