Sunday, April 5, 2020

BMQ v5.6-r1 release

BMQ v5.6-r1 is released with the following changes

1. Fix compile warning on 32bit system. Thanks jwh7 for reporting.
2. Reverted code changes which are not essential part of scheduler code. Eg cpufreq and INITIAL_JIFFIES.
3. Extend BMQ for RT tasks. This is the improvement which make bit-map queues(0-99) for RT tasks(FIFO/RR policy) instead of single double-linked queue in previous releases. Thanks whoever comment this in v5.5-r2 release.

Your feedback will be welcome and helpful for BMQ scheduler development. Next release will be come 2 weeks later.

Enjoy BMQ for your linux kernel.

Full kernel tree repository can be found at https://gitlab.com/alfredchen/linux-bmq
And all-in-one patch can be found at gitlab.

Bug report at https://gitlab.com/alfredchen/bmq/issues

14 comments:

  1. Compiled on Ryzen@home, boots fine. Thanks!

    BUT I have noticed the following:
    bmq: cpu#0 llc_id = 0, llc_mask idx = 2
    bmq: cpu#1 llc_id = 0, llc_mask idx = 2
    bmq: cpu#2 llc_id = 0, llc_mask idx = 2
    bmq: cpu#3 llc_id = 0, llc_mask idx = 2
    bmq: cpu#4 llc_id = 0, llc_mask idx = 2
    bmq: cpu#5 llc_id = 0, llc_mask idx = 2
    bmq: cpu#6 llc_id = 0, llc_mask idx = 2
    bmq: cpu#7 llc_id = 0, llc_mask idx = 2
    bmq: cpu#8 llc_id = 0, llc_mask idx = 2
    bmq: cpu#9 llc_id = 0, llc_mask idx = 2
    bmq: cpu#10 llc_id = 0, llc_mask idx = 2
    bmq: cpu#11 llc_id = 0, llc_mask idx = 2
    bmq: cpu#12 llc_id = 0, llc_mask idx = 2
    bmq: cpu#13 llc_id = 0, llc_mask idx = 2
    bmq: cpu#14 llc_id = 0, llc_mask idx = 2
    bmq: cpu#15 llc_id = 0, llc_mask idx = 2

    Than can not be true. I should have a map with cpu+llcid from times when I added llc support for MuQSS scheduler if You need to know which CPU is related to which LLC.

    BR, Eduardo

    ReplyDelete
    Replies
    1. @Eduardo
      Yes, the llc_id(all zero) is not correct.
      Would you pls post a full output of "dmesg | grep -i bmq", the one I have is from older BMQ release.

      Delete
    2. Here it is:
      [ 0.103057] bmq: BMQ CPU Scheduler 5.6-r1 by Alfred Chen.
      [ 1.981194] bmq: cpu#0 affinity mask - smt 0x00000100
      [ 1.981194] bmq: cpu#0 affinity mask - coregroup 0x00000e0e
      [ 1.981194] bmq: cpu#0 affinity mask - core 0x0000f0f0
      [ 1.981194] bmq: cpu#0 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#1 affinity mask - smt 0x00000200
      [ 1.981194] bmq: cpu#1 affinity mask - coregroup 0x00000d0d
      [ 1.981194] bmq: cpu#1 affinity mask - core 0x0000f0f0
      [ 1.981194] bmq: cpu#1 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#2 affinity mask - smt 0x00000400
      [ 1.981194] bmq: cpu#2 affinity mask - coregroup 0x00000b0b
      [ 1.981194] bmq: cpu#2 affinity mask - core 0x0000f0f0
      [ 1.981194] bmq: cpu#2 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#3 affinity mask - smt 0x00000800
      [ 1.981194] bmq: cpu#3 affinity mask - coregroup 0x00000707
      [ 1.981194] bmq: cpu#3 affinity mask - core 0x0000f0f0
      [ 1.981194] bmq: cpu#3 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#4 affinity mask - smt 0x00001000
      [ 1.981194] bmq: cpu#4 affinity mask - coregroup 0x0000e0e0
      [ 1.981194] bmq: cpu#4 affinity mask - core 0x00000f0f
      [ 1.981194] bmq: cpu#4 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#5 affinity mask - smt 0x00002000
      [ 1.981194] bmq: cpu#5 affinity mask - coregroup 0x0000d0d0
      [ 1.981194] bmq: cpu#5 affinity mask - core 0x00000f0f
      [ 1.981194] bmq: cpu#5 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#6 affinity mask - smt 0x00004000
      [ 1.981194] bmq: cpu#6 affinity mask - coregroup 0x0000b0b0
      [ 1.981194] bmq: cpu#6 affinity mask - core 0x00000f0f
      [ 1.981194] bmq: cpu#6 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#7 affinity mask - smt 0x00008000
      [ 1.981194] bmq: cpu#7 affinity mask - coregroup 0x00007070
      [ 1.981194] bmq: cpu#7 affinity mask - core 0x00000f0f
      [ 1.981194] bmq: cpu#7 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#8 affinity mask - smt 0x00000001
      [ 1.981194] bmq: cpu#8 affinity mask - coregroup 0x00000e0e
      [ 1.981194] bmq: cpu#8 affinity mask - core 0x0000f0f0
      [ 1.981194] bmq: cpu#8 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#9 affinity mask - smt 0x00000002
      [ 1.981194] bmq: cpu#9 affinity mask - coregroup 0x00000d0d
      [ 1.981194] bmq: cpu#9 affinity mask - core 0x0000f0f0
      [ 1.981194] bmq: cpu#9 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#10 affinity mask - smt 0x00000004
      [ 1.981194] bmq: cpu#10 affinity mask - coregroup 0x00000b0b
      [ 1.981194] bmq: cpu#10 affinity mask - core 0x0000f0f0
      [ 1.981194] bmq: cpu#10 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#11 affinity mask - smt 0x00000008
      [ 1.981194] bmq: cpu#11 affinity mask - coregroup 0x00000707
      [ 1.981194] bmq: cpu#11 affinity mask - core 0x0000f0f0
      [ 1.981194] bmq: cpu#11 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#12 affinity mask - smt 0x00000010
      [ 1.981194] bmq: cpu#12 affinity mask - coregroup 0x0000e0e0
      [ 1.981194] bmq: cpu#12 affinity mask - core 0x00000f0f
      [ 1.981194] bmq: cpu#12 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#13 affinity mask - smt 0x00000020
      [ 1.981194] bmq: cpu#13 affinity mask - coregroup 0x0000d0d0
      [ 1.981194] bmq: cpu#13 affinity mask - core 0x00000f0f
      [ 1.981194] bmq: cpu#13 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#14 affinity mask - smt 0x00000040
      [ 1.981194] bmq: cpu#14 affinity mask - coregroup 0x0000b0b0
      [ 1.981194] bmq: cpu#14 affinity mask - core 0x00000f0f
      [ 1.981194] bmq: cpu#14 llc_id = 0, llc_mask idx = 2
      [ 1.981194] bmq: cpu#15 affinity mask - smt 0x00000080
      [ 1.981194] bmq: cpu#15 affinity mask - coregroup 0x00007070
      [ 1.981194] bmq: cpu#15 affinity mask - core 0x00000f0f
      [ 1.981194] bmq: cpu#15 llc_id = 0, llc_mask idx = 2

      BR, Eduardo

      Delete
    3. @Edurado
      Pls also checkout this quick fix at https://gitlab.com/alfredchen/linux-bmq/-/commit/d394787c8c388b4136358b313568a9ac6d9849bb
      This should fix the cpu topology setup in ryzen system, and this bug should be existed for some time. I still need to double check it for some rare cpu topology, may add anther commit to address this. But still, I want this quick fix be out for ryzen users.

      Delete
    4. Now it shows this:
      $ dmesg | grep -i "bmq: cpu"
      [ 1.752190] bmq: cpu#00 affinity mask: 0x00000100 - smt
      [ 1.752190] bmq: cpu#00 affinity mask: 0x00000e0e - coregroup
      [ 1.752190] bmq: cpu#00 affinity mask: 0x0000f0f0 - core
      [ 1.752190] bmq: cpu#00 llc_id = 0, llc_mask idx = 1
      [ 1.752190] bmq: cpu#01 affinity mask: 0x00000200 - smt
      [ 1.752190] bmq: cpu#01 affinity mask: 0x00000d0d - coregroup
      [ 1.752190] bmq: cpu#01 affinity mask: 0x0000f0f0 - core
      [ 1.752190] bmq: cpu#01 llc_id = 0, llc_mask idx = 1
      [ 1.752190] bmq: cpu#02 affinity mask: 0x00000400 - smt
      [ 1.752190] bmq: cpu#02 affinity mask: 0x00000b0b - coregroup
      [ 1.752190] bmq: cpu#02 affinity mask: 0x0000f0f0 - core
      [ 1.752190] bmq: cpu#02 llc_id = 0, llc_mask idx = 1
      [ 1.752190] bmq: cpu#03 affinity mask: 0x00000800 - smt
      [ 1.752190] bmq: cpu#03 affinity mask: 0x00000707 - coregroup
      [ 1.752190] bmq: cpu#03 affinity mask: 0x0000f0f0 - core
      [ 1.752190] bmq: cpu#03 llc_id = 0, llc_mask idx = 1
      [ 1.752190] bmq: cpu#04 affinity mask: 0x00001000 - smt
      [ 1.752190] bmq: cpu#04 affinity mask: 0x0000e0e0 - coregroup
      [ 1.752190] bmq: cpu#04 affinity mask: 0x00000f0f - core
      [ 1.752190] bmq: cpu#04 llc_id = 4, llc_mask idx = 1
      [ 1.752190] bmq: cpu#05 affinity mask: 0x00002000 - smt
      [ 1.752190] bmq: cpu#05 affinity mask: 0x0000d0d0 - coregroup
      [ 1.752190] bmq: cpu#05 affinity mask: 0x00000f0f - core
      [ 1.752190] bmq: cpu#05 llc_id = 4, llc_mask idx = 1
      [ 1.752190] bmq: cpu#06 affinity mask: 0x00004000 - smt
      [ 1.752190] bmq: cpu#06 affinity mask: 0x0000b0b0 - coregroup
      [ 1.752190] bmq: cpu#06 affinity mask: 0x00000f0f - core
      [ 1.752190] bmq: cpu#06 llc_id = 4, llc_mask idx = 1
      [ 1.752190] bmq: cpu#07 affinity mask: 0x00008000 - smt
      [ 1.752190] bmq: cpu#07 affinity mask: 0x00007070 - coregroup
      [ 1.752190] bmq: cpu#07 affinity mask: 0x00000f0f - core
      [ 1.752190] bmq: cpu#07 llc_id = 4, llc_mask idx = 1
      [ 1.752190] bmq: cpu#08 affinity mask: 0x00000001 - smt
      [ 1.752190] bmq: cpu#08 affinity mask: 0x00000e0e - coregroup
      [ 1.752190] bmq: cpu#08 affinity mask: 0x0000f0f0 - core
      [ 1.752190] bmq: cpu#08 llc_id = 0, llc_mask idx = 1
      [ 1.752190] bmq: cpu#09 affinity mask: 0x00000002 - smt
      [ 1.752190] bmq: cpu#09 affinity mask: 0x00000d0d - coregroup
      [ 1.752190] bmq: cpu#09 affinity mask: 0x0000f0f0 - core
      [ 1.752190] bmq: cpu#09 llc_id = 0, llc_mask idx = 1
      [ 1.752190] bmq: cpu#10 affinity mask: 0x00000004 - smt
      [ 1.752190] bmq: cpu#10 affinity mask: 0x00000b0b - coregroup
      [ 1.752190] bmq: cpu#10 affinity mask: 0x0000f0f0 - core
      [ 1.752190] bmq: cpu#10 llc_id = 0, llc_mask idx = 1
      [ 1.752190] bmq: cpu#11 affinity mask: 0x00000008 - smt
      [ 1.752190] bmq: cpu#11 affinity mask: 0x00000707 - coregroup
      [ 1.752190] bmq: cpu#11 affinity mask: 0x0000f0f0 - core
      [ 1.752190] bmq: cpu#11 llc_id = 0, llc_mask idx = 1
      [ 1.752190] bmq: cpu#12 affinity mask: 0x00000010 - smt
      [ 1.752190] bmq: cpu#12 affinity mask: 0x0000e0e0 - coregroup
      [ 1.752190] bmq: cpu#12 affinity mask: 0x00000f0f - core
      [ 1.752190] bmq: cpu#12 llc_id = 4, llc_mask idx = 1
      [ 1.752190] bmq: cpu#13 affinity mask: 0x00000020 - smt
      [ 1.752190] bmq: cpu#13 affinity mask: 0x0000d0d0 - coregroup
      [ 1.752190] bmq: cpu#13 affinity mask: 0x00000f0f - core
      [ 1.752190] bmq: cpu#13 llc_id = 4, llc_mask idx = 1
      [ 1.752190] bmq: cpu#14 affinity mask: 0x00000040 - smt
      [ 1.752190] bmq: cpu#14 affinity mask: 0x0000b0b0 - coregroup
      [ 1.752190] bmq: cpu#14 affinity mask: 0x00000f0f - core
      [ 1.752190] bmq: cpu#14 llc_id = 4, llc_mask idx = 1
      [ 1.752190] bmq: cpu#15 affinity mask: 0x00000080 - smt
      [ 1.752190] bmq: cpu#15 affinity mask: 0x00007070 - coregroup
      [ 1.752190] bmq: cpu#15 affinity mask: 0x00000f0f - core
      [ 1.752190] bmq: cpu#15 llc_id = 4, llc_mask idx = 1

      I have not checked the code, but llc_mask seems to be the same for all CPUs, is that correct?

      BR, Eduardo

      Delete
    5. In addition (I had to split posts, 4K chars max in one post) LLC seems to be correct except the id. I assume 0/1 or 0/4 does not matter much.
      My info below is based on cpu_llc_shared_map / per_cpu(cpu_llc_id, CPU), but for kernel 5.2:
      possible/present/online CPUs: 32/16/16
      CPU 0 llc 0
      CPU 1 llc 0
      CPU 2 llc 0
      CPU 3 llc 0
      CPU 4 llc 1
      CPU 5 llc 1
      CPU 6 llc 1
      CPU 7 llc 1
      CPU 8 llc 0
      CPU 9 llc 0
      CPU 10 llc 0
      CPU 11 llc 0
      CPU 12 llc 1
      CPU 13 llc 1
      CPU 14 llc 1
      CPU 15 llc 1

      BR, Eduardo

      Delete
    6. Looks good, llc id in bmq.c is the first cpu index from cpu_coregroup_mask(), it is happened to be 4 in you system.

      Delete
    7. Ok, gut :) Are llc_mask idx's correct in my pre-previous post above? They are strangely the same for all CPUs...

      BR, Eduardo

      Delete
    8. Yes, the llc_mask idx is correct.

      Delete
  2. Running on Arch Linux x86_64, the kernel failed to build without BMQ since v5.6-r0(still patched but not enable). There's the compile error(gcc 9.3.0):

    [ 193s] kernel/sched/core.c: In function 'is_cpu_allowed':
    [ 193s] kernel/sched/core.c:1467:6: error: implicit declaration of function 'is_per_cpu_kthread'; did you mean 'is_percpu_thread'? [-Werror=implicit-function-declaration]
    [ 193s] 1467 | if (is_per_cpu_kthread(p))
    [ 193s] | ^~~~~~~~~~~~~~~~~~
    [ 193s] | is_percpu_thread

    ReplyDelete
    Replies
    1. I have compiled BMQ for 5.6 in Ubuntu 16.04.x as well as for ArchLinux, no problems at all.
      Maybe there's a bug, when BMQ patchset is applied but not enabled, I have not tried to compile like that.
      If this is the issue, I would say it needs fixing, but there is a workaround - do not patch kernel with BMQ or if You use pf-kernel from gitlab along with other goodies, just revert specifically BMQ using Alfred's all-in-one patch :)

      BR, Eduardo

      Delete
    2. Please try this fix: https://gitlab.com/snippets/1963674

      Delete
    3. @Eduardo
      Thanks for reply and workaround. This can only happen when patchset is applied but not enabled.I have reenabled this patch and built without error. The latest release has some improvement for RT tasks which I would like to test.


      @Oleksandr
      Thanks for the fix, it works great.


      @Alfred
      Thank you for continuously improving BMQ.

      Thanks again :)

      Delete
    4. Thanks for reporting and @pf's quick fix patch. It's a cherry-pick mistake during sync-up.
      I have pushed my fix commit at https://gitlab.com/alfredchen/linux-bmq/-/commit/70ef06c844ba112020e548691bffb2b8da70602b

      Delete