Wireless Access

 View Only
  • 1.  High Datapath CPU Utilization

    Posted 22 days ago

    Hello,

    I have already been working with TAC on this. Thought we made some progress with BCMC Optimization enabled. However, here is the problem:

    At one location, we have two controllers in an HA pair, both 7210s. The APs are not evenly divided between the two but it doesn't seem to matter in this case. We have Apple devices and utilize local Apple Caching Servers.

    When these devices update (whether some or a lot of them), the resulting traffic causes the datapath CPU utilization to skyrocket. In the past, this has just meant slow or non-existent connections for the clients on the affect controller. Recently, this actually dropped connections from the controller entirely including web admin or even CLI. Worked with TAC, enabled BCMC Optimization on the VLAN Interfaces and things seemed better for a couple weeks.

    Today, the issue reappeared with the same symptoms. Top Traffic on the controller was to the caching server, and datapath CPU utilization was at 99% for cpu12, and 70+ for the other 3. The traffic is labeled as "amazon-aws" (it is ephemeral ports so that may be why it's categorized like this) with a destination of our local Apple Caching Server. The bandwidth is not the issue.

    Does anyone have any input or things to try?

    I appreciate any help!



  • 2.  RE: High Datapath CPU Utilization

    Posted 21 days ago

    BCMC Optimization only touches broadcast and multicast, and cache downloads are unicast, so I'd treat the quiet couple of weeks as luck rather than a fix. One core at 99% with the rest near 70 reads like per-flow load: long downloads to the caching server sit on a datapath CPU while the stateful firewall and DPI touch every packet (the amazon-aws label means DPI is classifying them). That's how bandwidth looks fine while the controller runs out of CPU.

    Next time it spikes, grab show datapath utilization and show datapath session table with the caching server's IP. The AOS 8 show datapath reference lists both: https://arubanetworking.hpe.com/techdocs/CLI-Bank/Content/aos8/sh-datapath.htm

    If those sessions line up with the hot core, test in a maintenance window with DPI off, and check whether the controller is routing clients to the caching server's VLAN when it could bridge them. What AOS build are you on?



    ------------------------------
    Dustin Burns
    @Worldcom Exchange, Inc.
    www.wei.com

    - - -
    The comments I post here are my own, and not my employers
    If my post was useful accept solution and/or give kudos
    Personal Blog: www.networkfieldnotes.com
    - - -
    ------------------------------



  • 3.  RE: High Datapath CPU Utilization

    Posted 21 days ago
    Edited by JP-Iw6358 21 days ago

    Hi Dustin,

    We are on 8.13.2.0 . We upgraded this summer from 8.11.x.x 

    I was looking at the datapath session table during this time (filtering by the IPs of the caching server) and there were really not many sessions. On the GUI though, it's pretty consistent that when we have issues, the top destination is the caching servers. 




  • 4.  RE: High Datapath CPU Utilization

    Posted 20 days ago

    That combination is the tell. Few sessions but the caching server on top means a small number of very large flows, and the datapath pins each flow to one core by hash. Three laptops pulling a macOS update from a local cache can hold one core at 99% while the session count looks like nothing. Bandwidth isn't the issue because it's per core, not per box.

    Two things would settle it. First, show datapath utilization during the spike, and read the Cpu Type next to cpu12. If it's a DPI core, that amazon-aws classification is the cost and no firewall dpi (reload required, one HA member at a time) is the test. If it's FP, it's raw per-packet work on that flow and DPI is still part of it. Second, a bandwidth contract on the caching server's traffic, per user, in the role. It caps each elephant, which spreads the update wave over time instead of over cores. Updates finish slower and the controller stays reachable, which for a cache is the right trade.

    Worth knowing 8.13.3.0 is out. I haven't matched a release-note entry to this yet, but if you open a case, ask TAC to check the 8.13.x resolved list for datapath CPU with DPI before they go hunting.



    ------------------------------
    Dustin Burns
    @Worldcom Exchange, Inc.
    www.wei.com

    - - -
    The comments I post here are my own, and not my employers
    If my post was useful accept solution and/or give kudos
    Personal Blog: www.networkfieldnotes.com
    - - -
    ------------------------------



  • 5.  RE: High Datapath CPU Utilization

    Posted 18 days ago

    It is always the FP cores. 

    I am not too familiar with setting the bandwidth contracts, I see where to configure it on a Role by service. I would create this "Per-Application Limits for This Role" for "amazon-aws" and set the limit down/up to a few Mbps each to test?





  • 6.  RE: High Datapath CPU Utilization

    Posted 18 days ago

    I'd change two things there. amazon-aws is a DPI guess that also catches real AWS traffic. App contracts are also shared across the whole role, so a few Mbps would be split between every Apple device in it at once. The AOS 8 Bandwidth Contract Configuration page says app contracts are per role by default: https://arubanetworking.hpe.com/techdocs/ArubaOS/AOS_8x_WebHelp/Content/arubaos-solutions/roles-policies/conf-band-cont.htm and the user-role CLI reference offers per-user only on the plain role contract, not on bw-contract app: https://arubanetworking.hpe.com/techdocs/CLI-Bank/Content/aos8/user-role.htm

    So target the cache itself. AppRF 2.0 lets a custom application carry a bandwidth contract. Match it on the caching server's IP, since the ports move. Size the limit as the total you're happy for the cache to push at once, well above a few Mbps. Try it on one role and watch cpu12 during the next update wave.

    App contracts need DPI on, so DPI stays in the path. If an FP core still pins with the cache capped, send TAC the datapath output.



    ------------------------------
    Dustin Burns
    @Worldcom Exchange, Inc.
    www.wei.com

    - - -
    The comments I post here are my own, and not my employers
    If my post was useful accept solution and/or give kudos
    Personal Blog: www.networkfieldnotes.com
    - - -
    ------------------------------



  • 7.  RE: High Datapath CPU Utilization

    Posted 9 days ago

    JP, did capping the cache with the custom app contract calm cpu12 on the next update wave? If an FP core still pins with the cap on, check whether the controller is routing clients to the cache's VLAN, since that's more per-packet work on the same flow. If it held, marking the reply that fixed it helps the next person.



    ------------------------------
    Dustin Burns
    @Worldcom Exchange, Inc.
    www.wei.com

    - - -
    The comments I post here are my own, and not my employers
    If my post was useful accept solution and/or give kudos
    Personal Blog: www.networkfieldnotes.com
    - - -
    ------------------------------