
================================================================================
README for App QoS

December 2020
================================================================================

CONTENTS
========

- Introduction
- Requirements and Installation
- Configuration
- Usage
- REST API
- Usage Scenario
- How To Generate self-signed SSL certificate
- Legal Disclaimer


INTRODUCTION
============

Application Quality of Service (App QoS) is a proof-of-concept software created to demonstrate the use of
Intel(R) RDT technologies (CAT, MBA) and Intel(R) SST technologies
Base Frequency (BF), Core Power (CP) to improve QoS for applications
via partitioning system resources.
App QoS allows to assign Apps to Pools with different resources.
In current version each of the Pools (group of cores) has fixed amount of
cache and memory bandwidth allocated. Thanks to Intel(R) SST BF and CP support,
App QoS allows the CPU to be deployed with an asymmetric core frequency
configuration. Amount of resources allocated and cores frequency
configuration could depend on Pool's priority.
Intel(R) RDT CAT and MBA configuration is done via "libpqos" library and
Intel(R) SST BF and CP configuration via external "Intel pwr" library.

App QoS provides simple, local, REST API management interface secured
with HTTPS and mTLS. REST API management interface allows user to
manage Apps and Pools. An App controlled by App QoS could be a process,
a container or a VNF.

App QoS is reference code written in Python. It is fully configurable and
can be easily modified to suit other use cases and allow remote management.


REQUIREMENTS AND INSTALLATION
=============================

App QoS requires minimum Linux kernel v4.20 (for Intel(R) RDT L3 CAT and MBA)
however, for support of Intel(R) SST BF and CP Linux kernel v5.2 is required.
App QoS is a part of Intel(R) RDT Software Package, it is located in "appqos"
directory.

App QoS requires Python v3.x and depends on the following Python modules
and external components:
 - pqos (libqos Python wrapper)
 - flask
 - flask_restful
 - jsonschema
 - Intel pwr library (github.com, CommsPowerManagement)

NOTE:
To install mentioned dependencies in virtualenv please use "make setup" command.

Intel(R) RDT CAT and MBA configuration is done via libpqos using MSR
or OS/resctrl interfaces (user configurable), for more information please see
"Software Compatibility" section of README for Intel(R) RDT Software Package.
For hardware requirements please see "Hardware Support" section of
mentioned README file.

For more information about "Intel pwr" library used to configure Intel(R) SST BF and CP
please see https://github.com/intel/CommsPowerManagement .


CONFIGURATION
=============

App QoS supports Pools of Apps, each with defined cores, cache and memory
bandwidth allocation. Each App describes a single app, with one or more PIDs.

Cache Allocation, Memory Bandwidth Allocation (via CAT and MBA), Base Frequency,
Core Power (via SST BF and CP) and affinity are configured by App QoS on per core
basis.

NOTE: make sure that cores are isolated via "isolcpu=" kernel param.

App QoS config is stored in JSON format in file "./appqos.conf" for which
JSON Schema file is available in "./schema" directory.

Configuration file is "Read-Only", no runtime changes are saved.

If there is no "Default" pool (with "id" equal to 0) defined in config file,
App QoS will dynamically create one on start-up.
All unassigned cores will be assigned to "Default" pool,
MBA will be configured to "no throttling" (100% (default) or ~2^32MBps (MBA CTRL enabled))
and CAT CBM to all cache ways.

EXAMPLE CONFIG

{
"rdt_iface": {"interface": "msr"},
"mba_ctrl": {"enabled": false},
"apps": [{"id": 1, "name": "App1", "pids": [1979, 1980]},
         {"id": 2, "cores": [4], "name": "App2", "pids": [1592, 1593, 1594]},
         {"id": 3, "cores": [5], "name": "App3", "pids": [1576, 1577, 1578]}],
"pools": [{"id": 1, "cores": [1, 2, 3], "name": "HP", "mba": 100, "cbm": "0xFF0", "apps": [1], "power_profile": 1},
          {"id": 2, "cores": [4, 5, 6], "name": "MP", "mba": 50, "cbm": "0xC", "apps": [2, 3]},
          {"id": 3, "cores": [7, 8, 9], "name": "LP", "mba": 10, "cbm": "0x3", "apps": []}],
"power_profiles": [{"id": 1, "min_freq": 2500, "max_freq": 2500, "epp": "performance", "name": "HP"}],
"sstbf": {"configured": false},
"power_profiles_expert_mode": false,
"power_profiles_verify": true
}

"rdt_iface" section, Intel(R) RDT configuration.
 - "interface" - requested libpqos interface ("msr" or "os")

"mba_ctrl" section, Intel(R) RDT configuration.
 - "enabled" - MBA CTRL requested state (requires "os" interface)

"apps" section, Apps being managed by App QoS.
 - "id" - App's ID
 - "name" - App's name (optional)
 - "cores" - cores being used by App (optional)
 - "pids" - list of App's PIDs

"pools" section, Pools of Apps.
 - "id" - Pool's ID
 - "name" - Pool's name
 - "apps" - list of Apps being part of the Pool
 - "cbm" - Intel RDT CAT CacheWayBitmask assigned to Pool
 - MBA related configuration:
    - "mba" - Intel RDT MBA rate [%] assigned to Pool (default, 1 - 100 [%])
   OR
    - "mba_bw" - Intel RDT MBA rate [MBps] assigned to Pool (1 - 2^32-1 [MBps])
      (requires MBA CTRL to be enabled, please see config's "mba_ctrl" section)
 - "cores" - cores being assigned to Pool
 - "power_profile" - Power Profile ID to be applied on pool's cores

"power_profiles" section, Power Profiles/SST-CP.
 - "id" - Profile's ID
 - "name" - Profiles's name
 - "min_freq" - min. frequency [MHz]
 - "max_freq" - max. frequency [MHz]
 - "epp" - Energy Performance Preference

"sstbf" section, Intel(R) SST-BF configuration.
 - "configured" - SST-BF requested state

NOTE:
"power profiles" are ignored when SST-BF is configured.

Global options:
 - "power_profiles_expert_mode" - make power profiles editable via REST API
   (Default: False)

 - "power_profiles_verify" - Admission Control feature for config file content,
   verifies Power Profiles and Pools configuration (Default: True)

USAGE
=====

App QoS is a Python script accepting the following, optional,
command line options:
 - -h, --help, show this help message and exit
 - -c PATH, --config PATH, Configuration file path
 - --port PORT, REST API port (default: 5000)
 - -V, --verbose, Verbose mode

NOTE: App QoS requires root privileges.

By default it will attempt to read from "appqos.conf" file from current folder.
Example command line to run App QoS in verbose mode and config file in
non-default location:

sudo ./appqos.py -c /tmp/appqos.conf -V

NOTE:
To run App QoS in virtualenv please use "make run" command.

For more information about configuration file please see CONFIGURATION paragraph,
for information about runtime configuration please see REST API paragraph.
For more info please see USAGE SCENARIO paragraph.


REST API
========

The REST API is a local (but can be easily modified to allow remote access),
secured interface that allows the user to control App QoS.
The REST API allows to:
 - add, remove or move App between Pools,
 - add, remove or modify Pools,
 - configure Intel(R) SST BF and CP technologies
 - configure Intel(R) RDT interface type and MBA mode

NOTE: None of configuration changes made via REST API are saved to configuration file.

App QoS REST API uses HTTPS and mutual TLS (mTLS) for authorization and authentication.
App QoS certificate and client certificate are signed using CA certificate - please see
HOW TO GENERATE APP QOS AND CLIENT CERTIFICATE paragraph for more info.

For curl, parameters for client cert, client key and cacert, need to be passed in
command-line:
$ curl --cert client_appqos.crt --key client_appqos.key --cacert ca.crt <REQUEST_URL>

REST API URIs
-------------

JSON Schema files for REST API commands and responses are available in "./schema" directory.

- GET /apps - get all/collection of apps

- POST /apps - create new app
 Example request:
  {"pool_id": 2,
  "name": "hello",
  "cores": [1,2],
  "pids": [1]}

 Result:
  {"id": 5}

- GET /apps/{id} - get app for given id
 Example response:
  {"id": 6,
  "cores": [2, 3, 11],
  "name": "App",
  "pids": [1748, 1749, 1750],
  "pool_id": 3
  }

- PUT /apps/{id} - update app (e.g.:move between pools) for given id
 Example request:
  {"pool_id": 2}

- DELETE /apps/{id} - delete app for given id


- GET /pools - get all/collection of pools

- POST /pools - create new pool

- GET /pools/{id} - get pool for given id

- PUT /pools/{id} - modify pool for given id

- DELETE /pools/{id} - delete empty pool for given id


- GET /power_profiles - get all/collection of power profiles

- POST /power_profiles - create new power profile

- GET /power_profiles/{id} - get power profile for given id

- PUT /power_profiles/{id} - modify power profile for given id

- DELETE /power_profiles/{id} - delete power profile for given id


- GET /stats - get stats


- GET /caps - get system capabilities
 Example response:
  {"capabilities": ["cat","mba","sstbf","power"]
  }


- GET /caps/sstbf - get Intel(R) SST-BF details
 Example response:
  {"configured": true,
   "hp_cores": [2,...,88],
   "std_cores": [0,...,95]
  }

- PUT /caps/sstbf - configure Intel(R) SST-BF
 Example request:
  {"configured": true}


- GET /caps/rdt_iface - get Intel(R) RDT interface type
 Example response:
  {"interface": "msr",
   "interface_supported": ["msr", "os"]
  }

- PUT /caps/rdt_iface - set Intel(R) RDT interface type
 Example request:
  {"interface": "os"}
 NOTE: Request will be processed only when there are no pools (except "default" #0) configured in the system.
 "Default" pool will be reset to default values (full cache and MEM BW access).


- GET /caps/mba - get Intel(R) RDT MBA details
 Example response:
  {"mba_enabled": true,
  "mba_bw_enabled": false
  }


- GET /caps/mba_ctrl - get Intel(R) RDT MBA CTRL state
 Example response:
  {"enabled": false,
  "supported": true
  }

- PUT /caps/mba_ctrl - set Intel(R) RDT MBA CTRL state
 Example request:
  {"enabled": true}
 NOTE: Request will be processed only when there are no pools (except "default" #0) configured in the system.
 "Default" pool will be reset to default values (full cache and MEM BW access).


USAGE SCENARIO
==============

App QoS, leveraging Intel(R) RDT (CAT, MBA) and SST (BF, CP) Technologies,
is to be used to prioritize and protect performance of high priority applications.

Please see below for complete step-by-step guide to configure App QoS to
leverage Intel(R) RDT (CAT, MBA) and SST-BF Technologies to prioritize and
protect performance of high priority application.

SETUP

Get mTLS certificates for App QoS CA, App QoS and REST client (for more
info please see HOW TO GENERATE APP QOS AND CLIENT CERTIFICATE paragraph).

Create appqos.conf file (for more info please see Configuration).

cat /tmp/appqos.conf
{
    "apps": [
    ],

    "sstbf": {
        "configured": true
    },

    "pools": [
    ],

    "power_profiles_expert_mode": true,

    "rdt_iface": {
        "interface": "msr"},
    "mba_ctrl": {
        "enabled": false
    }
}

Empty config file with essential configuration only:
 - RDT interface and MBA mode configuration
 - SST-BF configuration
 - option to enable Power Profile "Expert Mode" to allow us to
   add power profiles via REST API

App QoS command line parameters (for more info please see USAGE paragraph)

./appqos.py --help
usage: appqos.py [-h] [-c PATH] [--port PORT] [-V]

optional arguments:
  -h, --help            show this help message and exit
  -c PATH, --config PATH
                        Configuration file path
  --port PORT           REST API port
  -V, --verbose         Verbose mode

NOTE: REST API port (5000 by default) can be set via "--port"

START APP QOS

sudo make run

INFO  Interface MSR, MBA BW: unsupported.
INFO  Interface OS, MBA BW: supported.
INFO  Supported RDT interfaces: ['msr', 'os']
INFO  RDT initialized with 'msr' interface
INFO  Supported capabilities:
INFO  ['cat', 'mba', 'sstbf', 'power']
 * Serving Flask app "rest.rest_server" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
INFO   * Running on https://127.0.0.1:5000/ (Press CTRL+C to quit)
INFO  Configuring SST-BF
INFO  SST-BF enabled, configured.
INFO  SST-BF HP cores: [2, 10, 11, 12, 13, 14, 15, 16, 25, 34, 35, 36, 37, 38, 39, 40]
INFO  SST-BF STD cores: [0, 1, 3, 4, 5, 6, 7, 8, 9, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 41, 42, 43, 44, 45, 46, 47]
INFO  Power Profiles/EPP enabled, not configured, SST-BF is configured
INFO  Configuring RDT
INFO  RDT MBA CTRL disabled
INFO  Configuration changed, processing new config...

VERIFY INITIAL CONFIGURATION

List all Pools
curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq

  {
    "id": 0,
    "mba": 100,
    "cbm": 2047,
    "name": "Default",
    "cores": [
      0,
      1,
      2,
      3,
...
      43,
      44,
      45,
      46,
      47
    ]
  }
]

NOTE: "Default" pool automatically created (id=0).
Pool takes all cores (48), 100% of MBA and full CAT CBM (all cache ways).

List all Apps
curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq

{
    "message": "No apps in config file"
}

NOTE: No Apps defined.

List capabilities
curl https://localhost:5000/caps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq

{
    "capabilities": [
        "cat",
        "mba",
        "sstbf",
        "power"
    ]
}

NOTE: Detected capabilities listed,
Intel(R) RDT CAT, MBA and Intel(R) SST BF, CP technologies supported.

List RDT interface configuration details
curl https://localhost:5000/caps/rdt_iface -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"interface": "msr",
 "interface_supported": ["msr", "os"]
}
NOTE: MSR interface is used (as per config file). System supports both MSR and OS interfaces.

List MBA CTRL/MBA details
curl https://localhost:5000/caps/mba_ctrl -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"enabled": false,
 "supported": false
}
NOTE: MBA CTRL not supported for MSR interface.

curl https://localhost:5000/caps/mba -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"mba_enabled": true,
 "mba_bw_enabled": false}

NOTE: MBA CTRL is NOT enabled (as per config file) so "mba_bw" is not available,
user should use "mba" field to configure "pools" MBA in %.
If MBA CTRL would be enabled, user would use "mba_bw" to configure "pools" MBA in MBps.
Error 404 will be returned when MBA is not supported.

List Intel(R) SST-BF details
curl https://localhost:5000/caps/sstbf -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"configured":true
 "hp_cores": [2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40],
 "std_cores": [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47]}

NOTE: SST-BF enabled (as per config file), list of HP and STD cores provided.

Verify frequencies settings

sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))'

        Freq.   Freq.
Core#   Min     Max     EPP
0       2100    2100    balance_performance
1       2100    2100    balance_performance
2       2800    2800    balance_performance
3       2100    2100    balance_performance
...
45      2100    2100    balance_performance
46      2100    2100    balance_performance
47      2100    2100    balance_performance

NOTE: HP cores' frequencies are set to 2800MHz, STD cores' to 2100MHz


Unconfigure SST-BF

curl https://localhost:5000/caps/sstbf -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"configured": false}'

{
    "message": "SST-BF caps modified"
}

Verify frequencies settings

sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))'

        Freq.   Freq.
Core#   Min     Max     EPP
0       2300    2300    balance_performance
1       2300    2300    balance_performance
2       2300    2300    balance_performance
3       2300    2300    balance_performance
...
45      2300    2300    balance_performance
46      2300    2300    balance_performance
47      2300    2300    balance_performance

NOTE: HP and STD cores' frequencies are set to 2300MHz


CREATE APPS

Create multiple processes, list their PIDs and core affinity

killall sleep
for n in {1..5}; do bash -c "sleep 1d &"; done
for i in `pidof sleep`; do taskset -p -c $i; done

pid 13894's current affinity list: 0-47
pid 13892's current affinity list: 0-47
pid 13890's current affinity list: 0-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

Create App, specify cores

curl https://localhost:5000/apps -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "APP1", "cores": [0,1,2,3], "pids": [13894, 13892]}'

{
    "id": 1
}

Verify App's creation (list ALL apps)

curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

[
    {
        "name": " APP1",
        "cores": [
            0,
            1,
            2,
            3
        ],
        "pids": [
            13894,
            13892
        ],
        "id": 1,
        "pool_id": 0
    }
]

NOTE: App was assigned to Pool #0 as cores 0,1,2,3 are part of that Pool

Verify PIDs' core affinity

for i in `pidof sleep`; do taskset -p -c $i; done
pid 13894's current affinity list: 0-3
pid 13892's current affinity list: 0-3
pid 13890's current affinity list: 0-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

Create App, specify destination Pool ID

curl https://localhost:5000/apps -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "APP2", "pool_id": 0, "pids": [13890]}'

{
    "id": 2
}

Verify App's creation (get specific App)

curl https://localhost:5000/apps/2 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

{
    "name": "APP2",
    "pids": [
        13890
    ],
    "id": 2,
    "pool_id": 0
}

NOTE: App was assigned to Pool #0 as requested via "pool_id" param

Verify PIDs' core affinity

for i in `pidof sleep`; do taskset -p -c $i; done
pid 13894's current affinity list: 0-3
pid 13892's current affinity list: 0-3
pid 13890's current affinity list: 0-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

NOTE: App2 PID's core affinity is 0-47 as those are "Default" pool cores.

MODIFY "DEFAULT" POOL

List SST-BF's STD cores

curl https://localhost:5000/caps/sstbf -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq .std_cores -c
 [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47]

Modify "Default" pool to exclude HP cores, assign 50% of MBA and 4 LLC CWs (CBM: 0xF/15)

curl https://localhost:5000/pools/0 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"cores": [0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47], "mba": 50, "cbm": "0xf"}'

"POOL 0 updated"

Check new "Default" pool configuration

curl https://localhost:5000/pools/0 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"id":0,
"mba":50,
"cbm":15,
"name":"Default",
"cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47],
"apps":[1,2]}

Check how does "Default" pool modification has affected Apps in that pool

curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

[
    {
        "name": "APP1",
        "pids": [
            13894,
            13892
        ],
        "id": 1,
        "pool_id": 0
    },
    {
        "name": "APP2",
        "pids": [
            13890
        ],
        "id": 2,
        "pool_id": 0
    }
]

for i in `pidof sleep`; do taskset -p -c $i; done

pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13890's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

NOTE: As some of the cores that were specified for APP1 are not part of
"Default" pool anymore, core affinity for APP1 PIDs' was reset to
new "Default" pool cores.

CREATE HP POOL

Get list of HP cores

curl https://localhost:5000/caps/sstbf -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq .hp_cores -c
[2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40]

Create HP Pool, with 7 (isolated) LLC CWs allocated and 100% of MBA

curl https://localhost:5000/pools -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "HP", "cores": [2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40], "cbm": "0x7F0", "mba": 100}'

{
    "id": 7
}

NOTE: Pool with "id=7" was created.

Get All Pools to verify creation of new HP Pool(#7)

curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

[{"id":0,
"mba":50,
"cbm":15,
"name":"Default",
"cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47],
"apps":[1,2]},
{"name":"HP",
"cores":[2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40],
"cbm":2032,
"mba":100,
"id":7}]

DEFINE AND APPLY POWER PROFILES

Define HP power profile

curl https://localhost:5000/power_profiles -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "PP HP", "min_freq": 3000, "max_freq": 3000, "epp": "performance"}'

{
  "id": 0,
  "message": "New POWER PROFILE 0 added"
}

Define STD power profile

curl https://localhost:5000/power_profiles -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"name": "PP STD", "min_freq": 1500, "max_freq": 1500, "epp": "power"}'

{
  "id": 1,
  "message": "New POWER PROFILE 1 added"
}

Get All Power Profiles to verify creation of new Power Profiles

curl https://localhost:5000/power_profiles -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq

[
  {
    "name": "PP HP",
    "min_freq": 3000,
    "max_freq": 3000,
    "epp": "performance",
    "id": 0
  },
  {
    "name": "PP STD",
    "min_freq": 1500,
    "max_freq": 1500,
    "epp": "power",
    "id": 1
  }
]

Apply power profiles

curl https://localhost:5000/pools/7 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"power_profile": 0}'

{
  "message": "POOL 7 not updated, Power Profiles configuration would cause CPU to be oversubscribed."
}

NOTE:
App QoS supports Admission Control feature, which protects CPU from being oversubscribed,
as a result of that, order in which power profile are applied is important.
(Configuration change can be forced by setting "verify" flag to "false" for a request)

Lets apply the STD power profile first.

curl https://localhost:5000/pools/0 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"power_profile": 1}'

{
  "message": "POOL 0 updated"
}

curl https://localhost:5000/pools/7 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"power_profile": 0}'

{
  "message": "POOL 7 updated"
}

Verify power profiles configuration in HW

sudo python3 -c $'import pwr\nprint("\tFreq.\tFreq.\\nCore#\tMin\tMax\tEPP")\nfor c in pwr.get_cores(): c.refresh_stats(); print("{}\t{}\t{}\t{}".format(c.core_id, c.min_freq, c.max_freq, c.epp))'

        Freq.   Freq.
Core#   Min     Max     EPP
0       1500    1500    power
1       1500    1500    power
2       3000    3000    performance
3       1500    1500    power
...
45      1500    1500    power
46      1500    1500    power
47      1500    1500    power

NOTE:
Cores from HP pool have frequency fixed to 3000MHz and EPP set to "performance",
STD pool's cores have frequency fixed to 1500MHz and EPP set to "power".

MOVE APP#2

Move App#2 to HP Pool(#7)

curl https://localhost:5000/apps/2 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"pool_id": 7}'

"APP 2 moved to new pool"

Verify new configuration, list all apps

curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

[
    {
        "name": "APP1",
        "pids": [
            13894,
            13892
        ],
        "id": 1,
        "pool_id": 0
    },
    {
        "name": "APP2",
        "pids": [
            13890
        ],
        "id": 2,
        "pool_id": 7
    }
]

Verify new configuration, list pools

curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

[{"id":0,
"mba":50,
"cbm":15,
"name":"Default",
"cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47],
"apps":[1]},
{"name":"HP",
"cores":[2,10,11,12,13,14,15,16,25,34,35,36,37,38,39,40],
"cbm":2032,
"mba":100,
"id":7,
"apps":[2]}]

Verify new configuration, check core affinity

for i in `pidof sleep`; do taskset -p -c $i; done

pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13890's current affinity list: 2,10-16,25,34-40
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

MODIFY APP#2

Modify App core affinity and name

curl https://localhost:5000/apps/2 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"cores": [2,11,13], "name": "APP2 HP"}'

"APP 2 moved to new pool"

Verify new configuration, get App#2 details

curl https://localhost:5000/apps/2 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

{
    "name": "APP2 HP",
    "pids": [
        13890
    ],
    "id": 2,
    "cores": [
        2,
        11,
        13
    ],
    "pool_id": 7
}

Verify new configuration, check core affinity

for i in `pidof sleep`; do taskset -p -c $i; done

pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13890's current affinity list: 2,11,13
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

REMOVE APP#2

Remove App#2

curl https://localhost:5000/apps/2 -X DELETE --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

"APP 2 deleted"

Verify new configuration, get App#2 and all Apps details

curl https://localhost:5000/apps/2 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

{
    "message": "APP 2 not found in config"
}

NOTE: App#2 not found

curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

[
    {
        "name": "stress-ng",
        "pids": [
            13894,
            13892
        ],
        "id": 1,
        "pool_id": 0
    }
]

NOTE: No App#2 listed

Verify new configuration, check core affinity

for i in `pidof sleep`; do taskset -p -c $i; done
pid 13894's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13892's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13890's current affinity list: 0,1,3-9,17-24,26-33,41-47
pid 13888's current affinity list: 0-47
pid 13886's current affinity list: 0-47

NOTE: App#2's PID core affinity set to "Default" pool cores

REMOVE HP POOL

Remove HP Pool (#7)

curl https://localhost:5000/pools/7 -X DELETE --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

"POOL 7 deleted"

Verify new configuration, get Pool#7 and all  pools details

curl https://localhost:5000/pools/7 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

{
    "message": "POOL 7 not found in config"
}

NOTE: Pool#7 not found

curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

[{"id":0,
"mba":50,
"cbm":15,
"name":"Default",
"cores":[0,1,3,4,5,6,7,8,9,17,18,19,20,21,22,23,24,26,27,28,29,30,31,32,33,41,42,43,44,45,46,47],
"apps":[1]}]

NOTE: No Pool#7 listed

RESET CONFIGURATION

Perform configuration reset.

curl https://localhost:5000/reset -X POST --cert client_appqos.crt --key client_appqos.key --cacert ca.crt

"Reset performed. Configuration reloaded."

Verify configuration after reset

curl https://localhost:5000/apps -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -k

{
    "message": "No apps in config file"
}

NOTE: No Apps configured

curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

[{"id":0,
"mba":100,
"cbm":2047,
"name":"Default",
"cores":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47]}]

NOTE: "Default" pool reconfigured, takes all cores (48),
100% of MBA and full CAT CBM (all cache ways).

ENABLE RDT OS INTERFACE

Verify that RDT OS interface is supported.

curl https://localhost:5000/caps/rdt_iface -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"interface": "msr",
 "interface_supported": ["msr", "os"]
}

NOTE: RDT "os" interface is supported but NOT enabled.

Enable RDT OS interface.

curl https://localhost:5000/caps/rdt_iface -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"interface": "os"}'

{"message": "RDT Interface modified"}

NOTE: Can be done ONLY when there are no Pools configured (other than "default" Pool #0)
(e.g.: all non-default, configured Pools were removed).
"Default" pool will be reset to default values (full cache and MEM BW access).

List RDT configuration details

curl https://localhost:5000/caps/rdt_iface -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"interface": "os",
 "interface_supported": ["msr", "os"]
}

NOTE: RDT OS interface is now enabled.

ENABLE MBA CTRL

Verify MBA CTRL status.

curl https://localhost:5000/caps/mba_ctrl -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"enabled": false,
 "supported": true
}

NOTE: MBA CTRL is supported but NOT enabled.

Enable RDT MBA CTRL 

curl https://localhost:5000/caps/mba_ctrl -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"enabled": true}'

{"message": "MBA CTRL status changed."}

NOTE: Can be done ONLY when there are no Pools configured (other than "default" Pool #0)
(e.g.: all non-default, configured Pools were removed).
"Default" pool will be reset to default values (full cache and MEM BW access).

Verify MBA CTRL status

mount | grep resctrl

resctrl on /sys/fs/resctrl type resctrl (rw,relatime,mba_MBps)

NOTE: resctrl fs mounted with "mba_MBps" option, MBA CTRL is enabled.

curl https://localhost:5000/caps/mba_ctrl -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"enabled": true,
 "supported": true
}

curl https://localhost:5000/caps/mba -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq -c

{"mba_bw_enabled": true
 "mba_enabled": false}

NOTE: MBA CTRL is now enabled.
User can use "mba_bw" to configure pool's MBA allocation.

Verify MBA configuration for "Default" Pool

curl https://localhost:5000/pools -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq

[
  {
    "id": 0,
    "mba_bw": 4294967295,
    "cbm": 2047,
    "name": "Default",
    "cores": [
      0,
      1,
...
      45,
      46,
      47
    ]
  }
]

NOTE: MBA for Pool #0 set to maximum value 4294967295MBps (~2^32MBps).

Limit Pool #0 MBA to ~5GBps

curl https://localhost:5000/pools/0 -X PUT --cert client_appqos.crt --key client_appqos.key --cacert ca.crt -H "Content-Type: application/json" -d '{"mba_bw": 5000}'

{"message": "POOL 0 updated"}

Verify new Pool #0 MBA configuration.

curl https://localhost:5000/pools/0 -X GET --cert client_appqos.crt --key client_appqos.key --cacert ca.crt | jq

{
  "id": 0,
  "mba_bw": 5000,
  "cbm": 2047,
  "name": "Default",
  "cores": [
    0,
    1,
...
    45,
    46,
    47
  ]
}

NOTE: MBA for Pool #0 is set to 5000MBps

cat /sys/fs/resctrl/schemata

    L3:0=7ff;1=7ff
    MB:0=5000;1=5000

NOTE: Resctrl shows new MBA configuration for Pool #0.

APP QOS AND CLIENT CERTIFICATE
=============================
App QoS uses mutual TLS - both App QoS and client certificates need to
be signed using the same root ca certificate.

It is the user responsibility to deliver certificate and key files for App QoS.
App QoS expects certain names of CA certificate, App QoS certificate and associated
private key - files need to be located in 'appqos/ca' directory:
- ca.crt - contains the CA certificate, that will be used to authenticate clients and App QoS during mTLS
- appqos.crt - contains App QoS certificate, it needs to be signed using ca.crt
- appqos.key - contains private key for the App QoS certificate

App QoS REST Client certificate needs to be signed by the same CA as App QoS
certificate. In this instruction we assume that client certificate will be named
'client_appqos.crt' and client will have access to 'ca.crt' CA certificate.

HOW TO GENERATE TEST APP QOS AND CLIENT CERTIFICATE
==================================================

appqos/ca/ directory contains sample script called 'gen_test_certs.sh' - it
can be used for generating certifacates for the App QoS and the
client (e.g: curl) for testing purposes only

$ cd appqos/ca
$ ./gen_test_certs.sh

This script will create:
- ca certificate
- appqos certificate
- client certificate

Legal Disclaimer
================

THIS SOFTWARE IS PROVIDED BY INTEL"AS IS". NO LICENSE, EXPRESS OR
IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS
ARE GRANTED THROUGH USE. EXCEPT AS PROVIDED IN INTEL'S TERMS AND
CONDITIONS OF SALE, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL
DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR
USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO
FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT
OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
