Monday, April 13, 2015

BFS fun for Raspberry Pi 2

Recently I brought a Raspberry Pi 2 as there are 4 cores than the single one on the first generation and double ram size. I would like to check whether it can be a thin-client replacement for me at this moment and for sure, Pi is very extendable and a fun toy to play with, I already have several idea want to be made from Pi.

After try with the NOOB to make sure the hardware works, I installed funtoo into Pi, system runs fine, X works with FB driver, FF 31 can be compiled and works, but it runs slow than I expected. I may try Xorg rpi driver to see if it acts better. But before that, I tried to custom the pi kernel for my needs and of course, get BFS run on it.

Pi now using 3.18.x kernel, the default pi kernel conig is a little be "fat", I am not going to customize the kernel config heavily right now, as it's the first time to touch this arch. Changed two kernel config to my needed, rebuild the kernel and it works on Pi. BFS 0460 can be clearly applied and be built, the kernel is running for 8H now. So far, BFS on Pi2 goes smoothly, going to apply -gc branch onto it, to be continued...


Posted by at
Labels: , , ,

17 comments:

  1. UnknownApril 15, 2015 at 5:57 PM

    Hi Alfred, I'm running a BFS back port with your -GC branch back ported to 3.10 android/msm on the nexus 6. Everything runs incredible, fast, stable. Except I can't use the cpumask full LLC selection or it causes a panic at resched_closest_idle. Do you think this is something that only works on x86 or depends on some newer upstream cpufreq stuff?

    ReplyDelete
    Replies
    1. Alfred ChenApril 16, 2015 at 7:09 AM

      I am very glade that it runs on android.
      1. For x86 platform, I haven't back port my patch to 3.10, so I don't have any idea it still works back there, as there maybe some sync-up changes form release to release need to be reverted, in order to make it works.
      2. For arm platform, lucky that I have a pi now, the -gc branch works are most ported to the pi kernel, and it runs for 2 days(mostly just idle), so for no crash or dump I can see from the dmesg. Pi 2 is running on 3.18.11.
      3. For the panic at resched_closest_idle(), should you pls send me the dmesg crash log so I can look into it? And btw, pls check your kernel config whether CONFIG_SCHED_SMT and CONFIG_SCHED_MC enabled?

      PS, nexus 6 android use 64bit kernel or 32bits!?

      BR Alfred

      Delete
    2. UnknownApril 16, 2015 at 9:13 AM

      http://pastebin.com/aJwVraqD
      That is the pastebin of the crash. Nexus 6 is 32 bit SCHED_MC is set but SCHED_SMT is not set.

      Delete
    3. UnknownApril 16, 2015 at 9:53 AM

      And it is only with the cpumask LLC selection patch everything else from -GC runs great.

      Delete
    4. Alfred ChenApril 16, 2015 at 6:20 PM

      The most important things first. The crash, from the log I can't tell why it crashed, it happens 5000+ seconds after system has been up and resched_closest_idle() should been called many times before it happens. As I can tell, the call flow of this crash just exist in arm platform( do_work_pending ), a few question/helps you can take as below
      1. Can this kind of crash reproduce?
      2. If yes, does it happens all after the similar
      ***
      [ 5597.713075] Unable to handle kernel paging request at virtual address 66ef9faf
      [ 5597.713099] pgd = d2ea8000
      [ 5597.713107] [66ef9faf] *pgd=0c61c831, *pte=00000000, *ppte=00000000
      ***
      3. If you want to have a try, remove all resched_closest_idle() calls from swap_stick(), there should be just 2 reference there and only those 2 ref in the whole bfs.c. I don't know what will happen in your side(Pi runs well with this change as expected), may be there will be another Oops in the rest of schedule() call after resched_closest_idle(), that can prove is not cause by resched_closest_idle().

      Delete
    5. Alfred ChenApril 16, 2015 at 6:27 PM

      For 32bits system, I don't know which the priodl patch you have picked, as it's not atomic for 32bit system to read 64bits, so I have add grq.priodl_lock to protect the r/w, just make sure you pick the right version.
      Keep SCHED_MC enabled, I fixed a compile error when SMP enabled, but none of SCHED_MC nor SCHED_SMT enabled. It's fixed when applying to pi kernel, so it's not publish yet. Just keep SCHED_MC that should be ok.

      BR Alfred

      Delete
    6. UnknownApril 16, 2015 at 10:45 PM

      Thanks Alfred, I don't expect much support since it's all back to 3.10 I just wasn't sure if it was something else potentially in the cpumask/LLC selection patch or not
      1. I can't reproduce it under any specific circumstances, all I can say is it always happens at resched_closest_idle and it will happen at no specific time, could be 5000 seconds, 10000 seconds. I haven't gone more than 24 hours though without getting the crash
      2. When it does happen, it is all similar spot, with PC at resched_closest_idle

      I have the newest priodl patch with the lock for 32-bit, and everything works great. With the LLC cpumask selection stuff reverted I get uptime for days.and everything runs great.

      The bfs.c for 3.10 with -gc and cpumask LLC selection is here:
      https://raw.githubusercontent.com/bbedward/ZenKernel_Shamu/89a03712a210990d6d8568d2b50c88259e8848fd/kernel/sched/bfs.c (Throughout my BFS backport I excluded SMT nice because it isn't relevant on ARM)

      I appreciate your work very much, I'm happy you have had the time to work on improving BFS - especially with all these mobile devices I think it's a great alternative CPU scheduler. Everybody running it says they notice very specific interactivity-related issues on stock android on the nexus 6, which are reduced significantly or eliminated with BFS.

      Delete
    7. Alfred ChenApril 17, 2015 at 1:54 AM

      Based on your answer 1 and 2, I suggest you to remove resched_closest_idle() call reference as I previous post, it does no harm.

      Another thing I can think of is the "big.LETTLE" system arm cpu is using, 4 cores for powersaveing, 4 cores for performance, I haven't check if it is used in nexus 6's cpu, and haven't check the code how it works, but *guess* it might affect cpumask/llc patch in a unexpected way. Please check CONFIG_MCPM in your kernel config.

      Another thing I can do is check the cpumask api I used to see if it has some potential issue back to 3.10 on arm platform. I forgot which release, it said that it is a mass in kernel arm world.

      Thanks for your testing.

      BR Alfred

      Delete
    8. UnknownApril 17, 2015 at 2:53 PM

      Thanks Alfred, I will remove those and see how far I go

      Delete
    9. UnknownApril 17, 2015 at 3:04 PM

      And N6 is not a big.LITTLE CPU, so MCPM is not enabled.

      Delete
    10. UnknownApril 18, 2015 at 4:49 PM

      No issues 24+hr without reached_closest_idle calls. This makes closest_CPU_mask method unused as well.

      On the default without the LLC selection patch I have no issues with the calls in swap_sticky.

      Do you think this has any negative impact removing them? Such as frequency scaling, etc?

      Delete
    11. Alfred ChenApril 19, 2015 at 1:08 AM

      Just keep it running and see how far it can go, I'd not like to make a conclusion yet.

      There is no negative impact removing them, in my latest -vrq, I have replaced the sticky task implementation. And the reached_closest_idle() has been removed/replaced earlier, the reason is that when a task became sticky, it will sit in grq and wait for an unscaled cpu or its original cpu to pick it up, so reschedule an idle cpu is very unlikely to pick that sticky task, b/c the idle cpu are most likely scaled.

      The cpumask apis I used are clean in 3.10 for arm platform. Another thing I can think of is the grq.cpu_idle_map out of sync when cpu hotplug happens, so would you please send me a full dmesg log (*NOT* the one just after system boot up, a few hours dmesg after system up will do), so I can check the LOCALITY and if cpu hotplugs happen there.

      BR Alfred

      Delete
    12. Alfred ChenApril 19, 2015 at 7:33 PM

      Just learn a little bit about android kernel, does the "hotplug" governor is used in your kernel?

      Delete
    13. UnknownApril 20, 2015 at 6:33 AM

      Hi, the phone does not hotplug CPUs while it is active but only after suspend (probably in the msm PM driver). This seemed to happen while phone was active. I will get the full log soon.

      Delete
    14. UnknownApril 20, 2015 at 10:26 AM

      http://pastebin.com/SR4p9p48

      That is a longer version of the trace with several lines before the crash. There are several hotplug events before the crash all during suspend/resume probably from the power management driver.

      Delete
    15. Alfred ChenApril 21, 2015 at 1:08 AM

      From the log, there are hotplug events from time to time.
      The first issue is, the bindzero design of the BFS cpu hotplug code has an issue that the online cpu will set to process's affinity even original it is not. As the result, which cause all process's affinity set to all cpus. I found this recently and backlog it for a fix solution, I'll post the detail later.
      Second one is there may be a chance the cpu bit in grq.cpu_idle_map is set, while that cpu already offline, I am planning a enhancement for this, hopefully can fix the issue.

      BTW, how long your kernel which remove eached_closest_idle() has been up so far?

      Delete
    16. UnknownApril 21, 2015 at 9:02 AM

      Thanks for your answer,
      I've been up for about 3 days without reached_closest idle calls.

      Delete

Subscribe to: Post Comments (Atom)