gc-branch update with CK's BFS 0463

CK finally releases BFS 0463 against kernel 4.1 this week, so here comes the gc branch updates.

What's new:
1. Base on BFS 0463 and kernel v4.1.4
2. Fix/Sync against BFS 0463

• 3b14908 bfs: [Sync] 4.1 schedule_user().
• 9f9dc34 bfs: [Fix] 0463 remove unused register_task_migration_notifier().

3. New Sync patches which pick up sync changes from previous mainline changes (most from 3.17 to 3.18 and some fixes upon previous patch)

• 0145370 bfs: [Sync] TIF_POLLING_NRFLAG for wake_up_if_idle() and resched_curr().
• 775e28a bfs: [Sync] sched_init_numa().
• c6c5894 bfs: [Sync] task_sched_runtime().
• 4a48abf bfs: [Sync] sched_setscheduler() logic, v3

4. Give a meaningful version name for this patch sets "v4.1_0463_1"

• dc4fa45 bfs: -gc BFS enchancement patch set version.

Code has been forced push to bitbucket and github . For those just want to easier apply the patches, here is the one for all patch include all BFS related commits in my gc-branch: bfs_enhancement_v4.1_0463_1.patch

If you are using the gc-branch, I'll highly suggest you to upgrade to this gc release. An updated -vrq branch will be coming soon, no new commits is planned(have to delay to next week as much sync-up works this week), but will be some bug fixes for the existed ones.

BR Alfred Chen


Update:
Add one more commit to fix RCU stall issue.

4623b19

bfs: v4.1_0463_1 rcu stall fix.

UP booting issue in BFS

For those who have UP boot issue with BFS, I have found that this issue was introduced between 0458 and 0460 during kernel 3.17 to 3.18. It's caused by cpu becomes idle and enter C-states.

It's not yet found why BFS UP has this issue while mainline scheduler doesn't, maybe there is some piece of codes BFS is missing during 3.17 to 3.18 migration. We still need to look into it.

Here is a workaround if you have this kind of UP booting issue. Add "idle=halt" to kernel cmdline which will disable cpu C-states when cpu is idle. Plz try this and see if it works for you.

BR Alfred

4.1 vrq -- reworking

It takes much for me to work out the vrq on 4.1 branch. It's because the -vrq branch used to work well on three of core2 based machines but now it is not quit happy with the chromebook pixel. I'll said that the new platform expose the issues. It's not so bad that reproduce the issue is the 50% work done to solve the issue, :)

In order to find out more possible log when issues pop up. I have enabled some kernel hacking configs. And in order to see the crash kernel message even kernel hangs in earlier stage, I enabled earlyprintk and vt console output.

For the vrq commits, I am reworking them by moving the less potential troublesome issues ahead and test them if any stabilize problem existed. Now I am on stabilize issue when udev starting up which has one ten or less possibility to be reproduced. I meet it once but unlucky too slow to pick up the cell-phone to get a screenshot and the incoming kernel message just flow it away. 

I would update the vrq branch after a commit passed my stabilize test and you can check if it works on your hw to help with the development.

First of all, here are the kernel hacking configs I enabled for testing

CONFIG_SCHED_DEBUG
CONFIG_SCHEDSTATS
CONFIG_SCHED_STACK_END_CHECK
CONFIG_TIMER_STATS
CONFIG_PROVE_LOCKING
CONFIG_LOCK_STAT
CONFIG_DEBUG_LOCKDEP

CONFIG_VT_CONSOLE

and the kernel options/parameters is "earlyprintk=vga,keep"
add the below line into /etc/sysctl.conf
kernel.printk = 7 7 1 7

Current -gc branch should works fine with above setup, plz have a try and I'll push -vrq branch soon.

BR Alfred


Edit:
-vrq branch for 4.1 has been pushed to bitbucket and github. Which now have four more vrq commits add upon -gc branch and it should consider stable as -gc branch itself, if you have any issue plz try to enable above debug methods and report back. 

Time to have fun with kernel 4.1

Just pushed my BFS0462 port and -gc bfs enhancement patches for kernel 4.1. There is no new features in -gc branch but some bug fixes and sync up changes with mainline kernel.

Nothing remarkable items, I have put all of them in commits if I remember correctly.

Pls check it from bitbucket or github

PS, recently I got a google chromebook pixel(2013), I could have some test with SMT after I set up the system on it.

BR Alfred Chen

Edit: We found an issue that UP is broken in BFS since kernel 3.18, investigation is going on but I put it in low priority than the kernel 4.1 -vrq branch release.

Edit(Jul 17): Update -gc branch to rebase kernel v4.1.2 and fix compile error when enable some kernel hack config.

9654667 bfs: [Fix] Fix undeclared sched_domains_mutex. 

b6e4eaf bfs: [Fix] Fix wrong rcu_dereference_check() usage. 

I have done a force update on the linux-4.1.y-gc branch, so if you have fetched it before, please delete the remote branch in your git and re-fetched it again.

About hotplug affinity enhancement

This enhancement comes from investigating the issue from Brandon BerHent who back-port the -gc branch to 3.10 for android system and build customized kernel for nexus 6. It's very cool thing and I got to said "Hello Moto", which recall the memory of my first cell-phone.

The android system, unlike pc platform, seems use a lot cpu hotplug mechanism for power-saving functionality. When I look at the cpu hotplug code, I notice the below behaviors.

p5qe ~ # schedtool -a 0x02 1388
p5qe ~ # schedtool 1388
PID  1388: PRIO   0, POLICY N: SCHED_NORMAL  , NICE   0, AFFINITY 0x2
p5qe ~ # cat /sys/devices/system/cpu/cpu1/online
1
p5qe ~ # echo 0 > /sys/devices/system/cpu/cpu1/online
p5qe ~ # schedtool 1388
PID  1388: PRIO   0, POLICY N: SCHED_NORMAL  , NICE   0, AFFINITY 0x1
p5qe ~ # cat /sys/devices/system/cpu/cpu1/online
0
p5qe ~ # echo 1 > /sys/devices/system/cpu/cpu1/online
p5qe ~ # schedtool 1388
PID  1388: PRIO   0, POLICY N: SCHED_NORMAL  , NICE   0, AFFINITY 0x3

As you can see, after cpu 1 offline then online, task's affinity changes from 0x2 to 0x3, which include the new online cpu 1, but not the original design what the task to run on. And the most interesting thing is, it is not only the behaviors of BFS, it's same for mainline CFS.
Normally, for pc platform, it's not a big problem, as there is not much cpu hotplug events unless suspend/resume. But if just a small enhancement that can maintenance task's original affinity intend, why not? Below is the behaviors with the enhancement.

p5qe ~ # schedtool 1375
PID  1375: PRIO   0, POLICY N: SCHED_NORMAL  , NICE   0, AFFINITY 0xf
p5qe ~ # schedtool -a 0x2 1375
p5qe ~ # schedtool 1375
PID  1375: PRIO   0, POLICY N: SCHED_NORMAL  , NICE   0, AFFINITY 0x2
p5qe ~ # echo 0 > /sys/devices/system/cpu/cpu1/online
p5qe ~ # schedtool 1375
PID  1375: PRIO   0, POLICY N: SCHED_NORMAL  , NICE   0, AFFINITY 0x1
p5qe ~ # dmesg | tail
[    9.771522] zram3: detected capacity change from 0 to 268435456
[    9.783513] Adding 262140k swap on /dev/zram0.  Priority:10 extents:1 across:262140k SSFS
[    9.785789] Adding 262140k swap on /dev/zram1.  Priority:10 extents:1 across:262140k SSFS
[    9.788066] Adding 262140k swap on /dev/zram2.  Priority:10 extents:1 across:262140k SSFS
[    9.790311] Adding 262140k swap on /dev/zram3.  Priority:10 extents:1 across:262140k SSFS
[   12.103469] sky2 0000:02:00.0 eth1: Link is up at 1000 Mbps, full duplex, flow control both
[   25.360122] random: nonblocking pool is initialized
[  105.757001] Renew affinity for 198 processes to cpu 1
[  105.757001] kvm: disabling virtualization on CPU1
[  105.757140] smpboot: CPU 1 is now offline
p5qe ~ # echo 1 > /sys/devices/system/cpu/cpu1/online
p5qe ~ # schedtool 1375
PID  1375: PRIO   0, POLICY N: SCHED_NORMAL  , NICE   0, AFFINITY 0x2
p5qe ~ # dmesg | tail
[    9.790311] Adding 262140k swap on /dev/zram3.  Priority:10 extents:1 across:262140k SSFS
[   12.103469] sky2 0000:02:00.0 eth1: Link is up at 1000 Mbps, full duplex, flow control both
[   25.360122] random: nonblocking pool is initialized
[  105.757001] Renew affinity for 198 processes to cpu 1
[  105.757001] kvm: disabling virtualization on CPU1
[  105.757140] smpboot: CPU 1 is now offline
[  137.348718] x86: Booting SMP configuration:
[  137.348722] smpboot: Booting Node 0 Processor 1 APIC 0x1
[  137.359727] kvm: enabling virtualization on CPU1
[  137.363338] Renew affinity for 203 processes to cpu 1

 This enhancement changes the default behaviors of the kernel/system, I have tested it for a while with different use cases, all looks good. So I mark this changes version 1, if you have any comments/concert, please let me know. I'll look into it.

Here is the commit of this enhancement.

BR Alfred

Edit: Just push a minor fix when CONFIG_HOTPLUG_CPU is not enabled.

v4.0.2-gc updates

Here comes the sync update for 4.0 gc branch

1. Add ck's sync patch upon 0462 for v4.0.2
2. Fix return type error for SMT_NICE, thanks pf pointing it out.
3. bfs: hotplug affinity enhancement, v1
    This is a little long story and basically it is not for pc, I'll open another topic for it later.

I also enable the github repo, so the code are on GitHub and BitBucket, feel free to pick whatever you like. Having fun!

Sanity Test for -gc&-vrq branch for linux 4.0

Here are the sanity test results of BFS, -gc branch and -vrq branch. No regression found on -gc branch, still doing better than origin BFS at 50% and 100% workload.

For vrq branch, there is not huge improvement against the gc branch, 50% and 300% workload performance are almost the same, there is even little regression at 100% workload,  the only good news is there are improvement at 150% workload.

The reasons why vrq doesn't make good performance that I expected are
1. Introduced some additional rq lock sessions when implement the new lock strategy.
2. The grq lock conflict doesn't seem to be a major problem for system with few cores, at least like my test hw platform(4 cores).

I wished I had a chance to reach some 30+ cores monsters to prove that all codes in vrq worthy it. But before that, I'll continue the unfinished features in vrq like the cache_count, see how much performance can gain from these opened doors.

BFS0462:
>>>>>50% workload
>>>>>round 1
real    5m21.850s
user    9m55.977s
sys     0m41.537s
>>>>>round 2
real    5m21.653s
user    9m55.750s
sys     0m41.411s
>>>>>round 3
real    5m21.973s
user    9m56.570s
sys     0m41.192s
>>>>>100% workload
>>>>>round 1
real    2m52.203s
user    10m8.151s
sys     0m43.575s
>>>>>round 2
real    2m52.050s
user    10m8.423s
sys     0m43.515s
>>>>>round 3
real    2m50.865s
user    10m8.283s
sys     0m43.700s
>>>>>150% workload
>>>>>round 1
real    2m56.355s
user    10m29.334s
sys     0m44.955s
>>>>>round 2
real    2m56.189s
user    10m29.469s
sys     0m44.782s
>>>>>round 3
real    2m56.264s
user    10m29.485s
sys     0m44.845s
>>>>>300% workload
>>>>>round 1
real    3m0.412s
user    10m42.805s
sys     0m46.352s
>>>>>round 2
real    3m1.408s
user    10m42.618s
sys     0m46.341s
>>>>>round 3
real    3m0.287s
user    10m43.304s
sys     0m46.244s

linux-4.0.y-gc:
>>>>>50% workload
>>>>>round 1
real    5m18.823s
user    9m50.911s
sys     0m41.302s
>>>>>round 2
real    5m19.032s
user    9m51.597s
sys     0m40.984s
>>>>>round 3
real    5m18.960s
user    9m51.490s
sys     0m41.046s
>>>>>100% workload
>>>>>round 1
real    2m51.085s
user    10m8.806s
sys     0m43.699s
>>>>>round 2
real    2m50.870s
user    10m8.108s
sys     0m44.142s
>>>>>round 3
real    2m50.839s
user    10m8.290s
sys     0m43.979s
>>>>>150% workload
>>>>>round 1
real    2m56.285s
user    10m30.045s
sys     0m44.629s
>>>>>round 2
real    2m56.286s
user    10m30.054s
sys     0m44.866s
>>>>>round 3
real    2m56.333s
user    10m30.379s
sys     0m44.425s
>>>>>300% workload
>>>>>round 1
real    3m0.427s
user    10m43.455s
sys     0m46.739s
>>>>>round 2
real    3m0.222s
user    10m43.341s
sys     0m46.519s
>>>>>round 3
real    3m0.244s
user    10m43.029s
sys     0m46.608s

linux-4.0.y-vrq:
>>>>>round 1
real    5m18.905s
user    9m51.214s
sys     0m40.890s
>>>>>round 2
real    5m18.994s
user    9m51.203s
sys     0m41.029s
>>>>>round 3
real    5m18.818s
user    9m51.266s
sys     0m40.819s
>>>>>100% workload
>>>>>round 1
real    2m51.414s
user    10m7.739s
sys     0m43.785s
>>>>>round 2
real    2m51.146s
user    10m7.449s
sys     0m43.848s
>>>>>round 3
real    2m51.103s
user    10m7.721s
sys     0m43.499s
>>>>>150% workload
>>>>>round 1
real    2m54.407s
user    10m21.732s
sys     0m44.407s
>>>>>round 2
real    2m54.436s
user    10m21.212s
sys     0m44.824s
>>>>>round 3
real    2m55.156s
user    10m21.279s
sys     0m44.796s
>>>>>300% workload
>>>>>round 1
real    3m0.549s
user    10m43.723s
sys     0m46.342s
>>>>>round 2
real    3m0.475s
user    10m44.249s
sys     0m45.982s
>>>>>round 3
real    3m0.393s
user    10m44.088s
sys     0m46.114s