Start a conversation

How to benchmark an ExaNIC

<div>We use sockperf for testing because it is open-source, well understood and has similar properties to real applications.</div> <div></div> <div>For this example we use the ExaNIC X10. The ExaNIC X4 is our first generation NIC so it will be a bit slower than the X10 ( about 170ns slower). <br /><br /><span><b>1) Configure machine and the Linux kernel for high performance bench marking<br /></b><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">1.1) Download and install the latest software and firmware here: https://exablaze.com/support</span></span></div> <div><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">2.2) Update firmware on nic: eg. exanic-fwupdate -d exanic0 exanic_X10_20170821.fw<br />2.2) Configure kernel and hardware for high performance. See details here: </span></span><a href="https://exablaze.com/docs/exanic/user-guide/benchmarking/benchmarking/">https://exablaze.com/docs/exanic/user-guide/benchmarking/benchmarking/</a><span><br /><b><br />2) Check that the hardware and software is working correctly</b><br />2.1) Configure exanic to bypass only mode e.g.: exanic-config exanic0:0 bypass-only on<br />2.2) Configure exanic to local loopback mode e.g.: exanic-config exanic0:0 local-loopback on</span></div> <div><span>2.3) Run exanic-loopback test: cd /perf-test/; make exanic; </span><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">perf-test# taskset -c 2 ./exanic_loopback exanic0 0 0  64 1000000</span></span> <div><span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><b>min=706ns median=769ns max=5344ns first=2043ns cpu_ghz=3.492<br /></b><br /><span color="#cc0000" style="color: #cc0000;"><b>RESULT:<span> </span></b>Exanic local loopback application-to-wire-to-application time has a median of<span> </span><b><800ns (769ns).<span> </span></b>Therefore Exanic HW and host software is performing correctly. <br /></span><br /><b>3) Test IP configuration</b><br /><span color="#3c3e43" style="color: #3c3e43;">3.1) Set up a second machine with configuration options from 1). Connect port 0 together. </span><br /><span color="#3c3e43" style="color: #3c3e43;">3.2) Turn off bypass and local loopback e.g: </span></span><span>exanic-config exanic0:0 bypass-only off; </span><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">exanic-config exanic0:0 local-loopback off</span></span></div> <div><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">3.3) Set up IP's on both hosts e.g:<span> </span><br /><code>client# </code></span></span><code><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">ifconfig enp1s0 10.10.0.1</span></span></code></div> <div><code><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">server# </span></span><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">ifconfig enp1s0 10.10.0.2</span></span></code><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></span></div> <div><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">3.4) Test that the servers are properly connected</span></span></div> <div><code><span><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">client#</span></span>ping 10.10.0.2</code></div> <div><code>PING 10.10.0.2 (10.10.0.2) 56(84) bytes of data.</code></div> <div><code>64 bytes from 10.10.0.2: icmp_seq=1 ttl=64 time=0.030 ms</code></div> <div><code>64 bytes from 10.10.0.2: icmp_seq=2 ttl=64 time=0.010 ms</code></div> <div></div> <div> <div><code>server# ping 10.10.0.1</code></div> <div><code>PING 10.10.0.1 (10.10.0.1) 56(84) bytes of data.</code></div> <div><code>64 bytes from 10.10.0.1: icmp_seq=1 ttl=64 time=0.019 ms</code></div> <div><code>64 bytes from 10.10.0.1: icmp_seq=2 ttl=64 time=0.010 ms</code></div> <div><code>64 bytes from 10.10.0.1: icmp_seq=3 ttl=64 time=0.010 ms</code><br /><br /><b>RESULT: ExaNICs are connected and exchanging IP messages</b><br /><br /><b>4) Test unaccelerated (slow) sockperf: Run sockperf on client and server:</b></div> </div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span>server# </span>sockperf sr -i 10.10.0.2</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">client#  sockperf pp -i 10.10.0.2 -t5 -m 14</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: == version #3.1-16.gitc6a0d0e3ab53 == </span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> </span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">[ 0] IP = 10.10.0.2       PORT = 11111 # UDP</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: Warmup stage (sending a few dummy messages)...</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: Starting test...</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: Test end (interrupted by timer)</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: Test ended</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: [Total Run] RunTime=5.450 sec; SentMessages=449097; ReceivedMessages=449096</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ========= Printing statistics for Server No: 0</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: [Valid Duration] RunTime=5.000 sec; SentMessages=412007; ReceivedMessages=412007</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ====> avg-lat=  6.038 (std-dev=0.294)</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: Summary: Latency is 6.038 usec</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: Total 412007 observations; each percentile contains 4120.07 observations</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> <MAX> observation =   77.856</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> percentile 99.999 =   69.863</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> percentile 99.990 =   10.248</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> percentile 99.900 =    7.035</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> percentile 99.000 =    6.261</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> percentile 90.000 =    6.062</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> percentile 75.000 =    6.040</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> percentile 50.000 =    6.025</span></code></div> <div><code><span color="#3c3e43" face="Helvetica Neue, Arial, Helvetica, sans-serif" style="color: #3c3e43; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">sockperf: ---> percentile 25.000 =    6.009</span></code></div> <div><span face="Helvetica Neue, Arial, Helvetica, sans-serif" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><b><code><span color="#3c3e43" style="color: #3c3e43;">sockperf: ---> <MIN> observation =    4.473</span></code><br /></b><br /><span style="color: #000000;"><b><span color="#cc0000">Result: ExaNICs are exchanging UDP traffic through the (slow) kernel interface</span></b></span></span></div> <div><br /><br /></div> <div><span><b>5) Run accelerated  (fast) UDP sockperf on client and server:</b><br /></span> <div><code>server# exasock taskset -c 2 sockperf sr -i 10.10.0.2</code></div> </div> <div><code>client# exasock taskset -c 2  sockperf pp -i 10.10.0.2 -t5 -m 14</code></div> <div><code>sockperf: == version #3.1-16.gitc6a0d0e3ab53 == </code></div> <div><code>sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)</code></div> <div></div> <div><code>[ 0] IP = 10.10.0.2       PORT = 11111 # UDP</code></div> <div><code>sockperf: Warmup stage (sending a few dummy messages)...</code></div> <div><code>sockperf: Starting test...</code></div> <div><code>sockperf: Test end (interrupted by timer)</code></div> <div><code>sockperf: Test ended</code></div> <div><code>sockperf: [Total Run] RunTime=5.450 sec; SentMessages=2882949; ReceivedMessages=2882948</code></div> <div><code>sockperf: ========= Printing statistics for Server No: 0</code></div> <div><code>sockperf: [Valid Duration] RunTime=5.000 sec; SentMessages=2645088; ReceivedMessages=2645088</code></div> <div><code>sockperf: ====> avg-lat=  0.932 (std-dev=0.034)</code></div> <div><code>sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0</code></div> <div><code>sockperf: Summary: Latency is 0.932 usec</code></div> <div><code>sockperf: Total 2645088 observations; each percentile contains 26450.88 observations</code></div> <div><code>sockperf: ---> <MAX> observation =    3.266</code></div> <div><code>sockperf: ---> percentile 99.999 =    1.309</code></div> <div><code>sockperf: ---> percentile 99.990 =    1.233</code></div> <div><code>sockperf: ---> percentile 99.900 =    1.150</code></div> <div><code>sockperf: ---> percentile 99.000 =    1.060</code></div> <div><code>sockperf: ---> percentile 90.000 =    0.969</code></div> <div><code>sockperf: ---> percentile 75.000 =    0.945</code></div> <div><code><b>sockperf: ---> percentile 50.000 =    0.925</b></code></div> <div><code>sockperf: ---> percentile 25.000 =    0.912</code></div> <div><code><b>sockperf: ---> <MIN> observation =    0.862</b></code></div> <div></div> <div><code><span color="#cc0000" style="color: #cc0000;"><b>RESULT: Exasock does UDP 1/2RTT in 862ns. </b> </span></code></div> <div><br /><br /></div> <div><b>6)  Run accelerated TCP (fast) sosckperf on client and server</b></div> <div><code>server# exasock taskset -c 2 sockperf sr -i 10.10.0.2 --tcp</code></div> <div><code>client# # exasock taskset -c 2  sockperf pp -i 10.10.0.2 -t5 -m 14 --tcp</code></div> <div><code>sockperf: == version #3.1-16.gitc6a0d0e3ab53 == </code></div> <div><code>sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)</code></div> <div></div> <div><code>[ 0] IP = 10.10.0.2       PORT = 11111 # TCP</code></div> <div><code>sockperf: Warmup stage (sending a few dummy messages)...</code></div> <div><code>sockperf: Starting test...</code></div> <div><code>sockperf: Test end (interrupted by timer)</code></div> <div><code>sockperf: Test ended</code></div> <div><code>sockperf: [Total Run] RunTime=5.450 sec; SentMessages=2756552; ReceivedMessages=2756551</code></div> <div><code>sockperf: ========= Printing statistics for Server No: 0</code></div> <div><code>sockperf: [Valid Duration] RunTime=5.000 sec; SentMessages=2527996; ReceivedMessages=2527996</code></div> <div><code>sockperf: ====> avg-lat=  0.977 (std-dev=0.033)</code></div> <div><code>sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0</code></div> <div><code>sockperf: Summary: Latency is 0.977 usec</code></div> <div><code>sockperf: Total 2527996 observations; each percentile contains 25279.96 observations</code></div> <div><code>sockperf: ---> <MAX> observation =    4.008</code></div> <div><code>sockperf: ---> percentile 99.999 =    1.340</code></div> <div><code>sockperf: ---> percentile 99.990 =    1.263</code></div> <div><code>sockperf: ---> percentile 99.900 =    1.205</code></div> <div><code>sockperf: ---> percentile 99.000 =    1.094</code></div> <div><code>sockperf: ---> percentile 90.000 =    1.014</code></div> <div><code>sockperf: ---> percentile 75.000 =    0.991</code></div> <div><code>sockperf: ---> percentile 50.000 =    0.973</code></div> <div><code>sockperf: ---> percentile 25.000 =    0.957</code></div> <div><code>sockperf: ---> <MIN> observation =    0.905</code></div> <div></div> <div><b><span color="#cc0000" style="color: #cc0000;">RESULT: Exasock does TCP 1/2 RTT ~900ns. </span></b></div> <div></div> <div><span>You can find more details here </span><a href="https://exablaze.com/media/blog-exasock-acceleration">https://exablaze.com/media/blog-exasock-acceleration</a></div> <br /><br /></div>
Choose files or drag and drop files
Was this article helpful?
Yes
No
  1. Phil Manuel

  2. Posted
  3. Updated

Comments