#1
  1. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jul 2003
    Location
    Prague, Czech Republic
    Posts
    868
    Rep Power
    70

    Network speed problem/tuning


    Hi,
    I have two servers and try to test one of its netcards speed.
    Why the speed drops dramaticly once I start a test with more than 8 threads? And why at this point the speed of the threads is not well balanced as it is with less than 8 threads?

    Servers: 2
    Model: HP BL685C G7
    CPU: AMD Opteron (6174) 2,2 GHz, 12 core, 512KB cache
    CPUs: 4
    RAM: 64GB
    DISK: 73GB (single)
    Controller: HP Smart Array P410i, firmware 5.14
    RAID mode: 1(1+0)
    NET: 8x Emulex OneConnect 10Gbps
    OS: SUSE Enterprise 11 x86_64
    Kernel: 2.6.32.54-0.3
    Swap: 3GB

    Partitions: /, /boot, /data, /home, /opt, /tmp, /usr, /var
    Enabled netcard: 1 (eth0)
    netcard module: hp-be2net (ServerEngines BladeEngine 10Gbps)
    Netcard module ver: 2.104.225.7
    Net. application: iperf

    Server: iperf -s -w 2M
    Client: iperf -c 10.200.140.157 -t 30 -i 2 -w 2M -P XY (see the attached zip with traces)

    Sorry, cannot attach the /etc/sysctl.conf at the moment ...
    Thank you.
    Attached Files
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,400
    Rep Power
    1688
    Can't really comment on the network performance, but why (how, even!) are you running RAID with a single disk?
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jul 2003
    Location
    Prague, Czech Republic
    Posts
    868
    Rep Power
    70
    Hi and thank you for a reply.
    The disk is a single one and was used to replace the RAID1 pair. That I should have clarified before. Sorry.
    The servers have been handed over to me by our customer to perform a network throughput test.
    So far I can access the servers only via ssh. Not physicaly.

    What realy botheres me is the dramatic drop of speed with the number of ipref's threads above 8 (the server reports 8 cards - may that be a clue). From that point the speed remains almost the same no matter how many addidtional threads are there (I tested with 10, 20, 40, 60, 120).

    I started off with sysctl.conf like this:
    Code:
    vm.swappiness = 10
    
    net.core.rmem_default = 524287
    net.core.rmem_max = 524287
    net.core.wmem_default = 524287
    net.core.wmem_max= 524287
    net.core.netdev_max_backlog= 300000
    net.ipv4.tcp_timestamps = 0
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_rmem = 10000000 10000000 10000000
    net.ipv4.tcp_wmem = 10000000 10000000 10000000
    net.ipv4.tcp_mem = 10000000 10000000 10000000
    and ended up with this:
    Code:
    vm.swappiness = 5
    
    net.core.rmem_default = 52582912
    net.core.rmem_max = 52582912
    net.core.wmem_default = 52582912
    net.core.wmem_max= 52582912
    net.core.netdev_max_backlog= 300000
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_rmem = 10240 87380 52582912
    net.ipv4.tcp_wmem = 10240 87380 52582912
    net.ipv4.tcp_mem = 10240 87380 52582912
    net.ipv4.tcp_no_metrics_save = 1
    Still no change in the whole behavior. All that has changed is the amount of consumed memory. That, in the end, was at 80%.
    Last edited by zby; March 7th, 2012 at 09:04 AM.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,400
    Rep Power
    1688
    No way am I a network guru, let me state that up front, so be warned!
    Initial thoughts after having had a look at a few of those logs you attached in your original post:
    Are all 8 ports aggregated/trunked to present a single IP address, I am guessing that is so due to the lack of different IPs in the logs.
    I am also guessing that the NIC supports a total of 10GB/s.
    Generally, with ethernet, you'll get a maximum of 60-80% throughput, of rated levels. Any more than that and you will start having issues with packet collisions, etc. which force re-sends.
    You seem to be getting a decent, sustained 8-9GB/s whilst thowing a lot of data across the network, until you start over-loading the card.
    Is that really the sort of pattern you'd expect to see? Mass traffic involving large amounts of data?
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jul 2003
    Location
    Prague, Czech Republic
    Posts
    868
    Rep Power
    70
    The story behind the scene is not short as usual ... So I'll try to express all simply.

    What I have heard is that their intention is to use the servers to host virtual ones OR to use it as powerful application servers. In the second case the JBoss app on one of the machines will use a connection pool to Oracle DB. As I am not the JBoss guru I think that more than 5 connections to one IP will be set. But most likely more than 10 I guess. The overall speed of a database server is another story, but not on the scene in my particular case.

    The linux kernel reports 8 ethernet cards out of which only one is configured. The rest is disabled. How are the "ports" organized in the schasis, that I don't know. I'll ask.

    Our customer wishes to know if the card can handle cirka 7-9GBits/s. That is I think possible, but it's necessary to explain the limits. Today I watched the traffic of "iperf" via "iptraf" and the max. speed I saw was a bit under 7GBits/s.

    All of my tests were directed to the first ethernet of the second machine. So to just one IP adress.

    Yes, the NIC supports 10GBits/s. Ethtool eth0 reports that speed.

    Both of the serverst eth0 have their MTUs set to 1500. The default.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,400
    Rep Power
    1688
    That speed does suggest around the max you'll get from a 10Gb/s NIC over ethernet.
    Is that required rate (7-9 Gb/s) sustained or burst rate? In my view (remember, NOT a network guru!) you'll be hard pressed to get ethernet up to that sort of efficiency.
    What is reported, and what is there, configured, linked and connected on the card may vary depending on the kernel and drivers. 8 rj45 ports on a single card does seem like quite a few (in my day a 4 port 10/100/1000 card was top of the line, so take that as it comes!)

    Having done a little googling, it looks like the 8x part of the name is the speed of the PCI slot required, not the number of interfaces provided - images suggest that it is a dual-port card.
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jul 2003
    Location
    Prague, Czech Republic
    Posts
    868
    Rep Power
    70
    The server houses two network chips each capable of max. 10Gb/s.

    I set MTU to 9000 today and the speed jumped up a bit, but not significantly. What was a benefit was a bit better "stability". The data-flow was a bit more even, but the limit of 6.8 Gb/s reported by "iptraf" a couple of days ago was not breached.

    We do not want to talk about the rest of the equipment on the route to other resources. It's clear it must work at the same speed, otherwise ...

    We may have a chance to enable the second chip to test. What I do not know is which "card" belong to what chip as I cannot access the chassis. Could the "lspci" be of a help?

IMN logo majestic logo threadwatch logo seochat tools logo