commit 6c493795f76964bb10f45020b067ce3186d21825 Author: Alexandre Frade Date: Fri Mar 26 00:56:46 2021 +0000 Linux 5.11.10-rt11-xanmod1 Signed-off-by: Alexandre Frade commit a9cd6ce009a10adfb1b7fba9fb4b4dd0315bde9b Author: Alexandre Frade Date: Sun Oct 13 03:10:39 2019 -0300 kconfig: set PREEMPT_RT and RCU_BOOST without delay by default Signed-off-by: Alexandre Frade commit 1d16ba4ffb3049430884dae41c1427c9efc77242 Author: Alexandre Frade Date: Sat Oct 17 15:44:32 2020 +0000 sched/core: Set nr_migrate to increases number of tasks to iterate in a single balance run Signed-off-by: Alexandre Frade commit e759857253cb180880a1216d8d3a4329276f8fd1 Merge: 6e89a237de76 b40b73f9011a Author: Alexandre Frade Date: Fri Mar 26 00:45:54 2021 +0000 Merge tag 'v5.11.4-rt11' into 5.11-rt v5.11.4-rt11 commit 6e89a237de76f0487df8f4655a25dcb81ffa05ea Author: Alexandre Frade Date: Fri Mar 26 00:44:59 2021 +0000 Revert "kconfig: set PREEMPT and RCU_BOOST without delay by default" This reverts commit 9e669018a5bb61e5f948001ade696ed7493f3cd4. commit 30208336dfcd469d5c9cd1debbe69f47ab8d2013 Author: Alexandre Frade Date: Fri Mar 26 00:42:40 2021 +0000 Revert "sched/core: nr_migrate = 256 increases number of tasks to iterate in a single balance run." This reverts commit 1a270fff183c22a440555ef243e51c79c713e70d. commit b40b73f9011adbe48a7500bc8df9f0a610247fb2 Author: Sebastian Andrzej Siewior Date: Tue Mar 9 18:03:30 2021 +0100 v5.11.4-rt11 Signed-off-by: Sebastian Andrzej Siewior commit f2c6e3011625ed9d0c9a5ef342524a32e7970531 Author: Sebastian Andrzej Siewior Date: Tue Mar 9 17:10:29 2021 +0100 softirq: Update the softirq/tasklet patches This is an all-in-one commit updating the current softirq patches to the version posted to the list as of: 2021-03-09 09:42 Thomas Gleixner [patch 00/14] tasklets: Replace the spin wait loops and make it RT safe 2021-03-09 09:55 Thomas Gleixner [patch V3 0/6] softirq: Add RT specific softirq accounting and a fix which was posted to the list. Signed-off-by: Sebastian Andrzej Siewior commit ebd50b8b2dfba55c06bff4ab726b60a8eb2e0aa4 Author: Sebastian Andrzej Siewior Date: Tue Mar 9 17:29:45 2021 +0100 v5.11.4-rt10 Signed-off-by: Sebastian Andrzej Siewior commit ea82ae15e1ee6f7501412617cea1ccea2cc261b1 Merge: 6970cfbedc75 79a7af2679aa Author: Sebastian Andrzej Siewior Date: Tue Mar 9 17:29:21 2021 +0100 Merge tag 'v5.11.4' into linux-5.11.y-rt This is the 5.11.4 stable release commit 6970cfbedc758d182994e44a319a8acf0931bb50 Author: Sebastian Andrzej Siewior Date: Tue Mar 2 20:41:34 2021 +0100 v5.11.2-rt9 Signed-off-by: Sebastian Andrzej Siewior commit 15af525df7a25e1ba218684e3f21f1c7cf2ced9c Author: Sebastian Andrzej Siewior Date: Tue Mar 2 20:37:32 2021 +0100 slub: Update the SLUB and page_alloc patch Upate the SLUB and page_alloc patches with the version in the queue. Notable changes: - There used to be a per-CPU list for pages which should have been given back to the page-alloctor but the caller was atomic. The atomic sections from within SLUB are gone, the per-CPU list is gone, too. - The SLUB_CPU_PARTIAL switch can now be enabled. I looked at the resulting latency numbers and enabling leads to higher latency. Therefore it is off by default on RT. - We used to split the free process of the pcp-pages into two stages and so slightly decouple the IRQ-off section from the zone-lock section. I don't see the need to keep doing it since the local-lock removes all the IRQ-off regions on RT and for !RT it shouldn't make much difference. This split is gone now. - The alloc/free tracker sysfs file uses one PAGE size for collecting the results. If it runs out of space it reallocates more memory with disabled interrupts. The reallocation is not forbidden on PREEMPT_RT. Signed-off-by: Sebastian Andrzej Siewior commit de40f609b52a2a16c02dbc3622bcd30d14c7dde9 Author: Sebastian Andrzej Siewior Date: Tue Mar 2 20:35:51 2021 +0100 signal: Sync the sigqueue cache patch This updates the sigqueue cache patch to the version in the queue. Signed-off-by: Sebastian Andrzej Siewior commit 18ac3d36e34d141dd7a7c2ffa80ec3c7901906fe Author: Sebastian Andrzej Siewior Date: Tue Mar 2 20:34:50 2021 +0100 hrtimer: Move cpu_chill() definition. Move the cpu_chill() definition to hrtimer.h to avoid a warning about a non-static function without a prototype. Signed-off-by: Sebastian Andrzej Siewior commit b54e1db50dddfe9058f47ede96aba482dd8b5bb4 Author: Sebastian Andrzej Siewior Date: Tue Mar 2 20:39:21 2021 +0100 v5.11.2-rt8 Signed-off-by: Sebastian Andrzej Siewior commit dc995aed609ed372d66e04bd0df4ad92267612e1 Merge: fd08052568e1 27e543cca13f Author: Sebastian Andrzej Siewior Date: Tue Mar 2 17:53:39 2021 +0100 Merge tag 'v5.11.2' into linux-5.11.y-rt This is the 5.11.2 stable release Signed-off-by: Sebastian Andrzej Siewior commit fd08052568e1bb4bde3e1c5ea5dfede7ce1acfc0 Author: Sebastian Andrzej Siewior Date: Fri Feb 19 18:36:18 2021 +0100 v5.11-rt7 Signed-off-by: Sebastian Andrzej Siewior commit 8e4525b77908a55e8838f6fe1df7c89a9242fab1 Author: Sebastian Andrzej Siewior Date: Fri Feb 19 18:25:19 2021 +0100 printk: Update the printk code This is an all-on-one patch replacing the current RT related printk patches with an update as provided by John Ogness. Signed-off-by: Sebastian Andrzej Siewior commit 8f3a6b6a12e00ef7c281ed9411ac9b13bf58f627 Author: Sebastian Andrzej Siewior Date: Fri Feb 19 17:51:07 2021 +0100 powerpc/mm: Move the linear_mapping_mutex to the ifdef where it is used The mutex linear_mapping_mutex is defined at the of the file while its only two user are within the CONFIG_MEMORY_HOTPLUG block. A compile without CONFIG_MEMORY_HOTPLUG set fails on PREEMPT_RT because its mutex implementation is smart enough to realize that it is unused. Move the definition of linear_mapping_mutex to ifdef block where it is used. Fixes: 1f73ad3e8d755 ("powerpc/mm: print warning in arch_remove_linear_mapping()") Signed-off-by: Sebastian Andrzej Siewior commit 16624ad2a853197335fe2fae0f23037dee3b02e9 Author: Sebastian Andrzej Siewior Date: Thu Feb 18 19:25:08 2021 +0100 v5.11-rt6 Signed-off-by: Sebastian Andrzej Siewior commit 778b6851e19aa9ed42472e8ee363e11c02172a5d Author: Sebastian Andrzej Siewior Date: Thu Feb 18 17:55:56 2021 +0100 trace: Add the flags for need_resched_lazy() During the rework of the tracing flags, the evaluation of need_resched_lazy() got lost. Add the evaluation of need_resched_lazy() to recorded tracing flags. Signed-off-by: Sebastian Andrzej Siewior commit 61c24671c859432b2083e389088794ef56f3abee Author: Sebastian Andrzej Siewior Date: Thu Feb 18 19:24:20 2021 +0100 tracing: Merge irqflags + preemt counter, add RT bits This is an all-in-one patch updating the tracing patches to what Steven merged into this tracing tree and posted as part of his for-next series in https://lkml.kernel.org/r/20210203160517.982448432@goodmis.org Signed-off-by: Sebastian Andrzej Siewior commit ded7bdd0e99a7567c2a89969cb576d012b6fe235 Author: Sebastian Andrzej Siewior Date: Thu Feb 18 19:22:52 2021 +0100 zswap: Incremental update of "mm/zswap: add a flag to indicate if zpool can do sleep map" This updates the patch to v2 as posted by Barry Song. Link: https://lkml.kernel.org/r/20210213075831.10852-2-song.bao.hua@hisilicon.com Signed-off-by: Sebastian Andrzej Siewior commit d0768975d5065305a5890c83adef48d68b383533 Author: Tian Tao Date: Thu Jan 21 17:57:10 2021 +0800 mm/zswap: fix variable 'entry' is uninitialized when used the entry has not been initialized when it is used, so allocate PAGE_SIZE Signed-off-by: Tian Tao Reported-by: kernel test robot Signed-off-by: Sebastian Andrzej Siewior commit 4bacfca0f1d7d92edcd579ace44a90edc73da41d Author: Sebastian Andrzej Siewior Date: Thu Feb 18 18:31:26 2021 +0100 kcov: Remove kcov include from sched.h and move it to its users. The recent addition of in_serving_softirq() to kconv.h results in compile failure on PREEMPT_RT because it requires task_struct::softirq_disable_cnt. This is not available if kconv.h is included from sched.h. It is not needed to include kconv.h from sched.h. All but the net/ user already include the kconv header file. Move the include of the kconv.h header from sched.h it its users. Additionally include sched.h from kconv.h to ensure that everything task_struct related is available. Signed-off-by: Sebastian Andrzej Siewior commit e21923700ff120a15519da639f6fe8c690c52231 Author: Sebastian Andrzej Siewior Date: Tue Feb 16 17:56:46 2021 +0100 v5.11-rt5 Signed-off-by: Sebastian Andrzej Siewior commit 25454b5d365652826e394f50ea81a58b41ac10b4 Author: Sebastian Andrzej Siewior Date: Mon Feb 15 18:44:12 2021 +0100 smp: Wake ksoftirqd on PREEMPT_RT instead do_softirq(). The softirq implementation on PREEMPT_RT does not provide do_softirq(). The other user of do_softirq() is replaced with a local_bh_disable() + enable() around the possible raise-softirq invocation. This can not be done here because migration_cpu_stop() is invoked with disabled preemption. Wake the softirq thread on PREEMPT_RT if there are any pending softirqs. Signed-off-by: Sebastian Andrzej Siewior commit 798775cee2466ce233b82df82ee70da592e025c2 Author: Sebastian Andrzej Siewior Date: Sat Jan 23 21:10:25 2021 +0100 smp: Process pending softirqs in flush_smp_call_function_from_idle() send_call_function_single_ipi() may wake an idle CPU without sending an IPI. The woken up CPU will process the SMP-functions in flush_smp_call_function_from_idle(). Any raised softirq from within the SMP-function call will not be processed. Should the CPU have no tasks assigned, then it will go back to idle with pending softirqs and the NOHZ will rightfully complain. Process pending softirqs on return from flush_smp_call_function_queue(). Fixes: b2a02fc43a1f4 ("smp: Optimize send_call_function_single_ipi()") Reported-by: Jens Axboe Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210123201027.3262800-2-bigeasy@linutronix.de Signed-off-by: Sebastian Andrzej Siewior commit cb9e907467865131cf5d27d46eabf395932dee59 Author: Christian Eggers Date: Wed Feb 10 12:36:14 2021 +0100 softirq: Provide tasklet_unlock*() for RT+!SMP builds The tasklet_unlock*() functions should be provided by the C file as defined in the header file. Signed-off-by: Christian Eggers Signed-off-by: Sebastian Andrzej Siewior commit d93ec6b9e30cf740fd4a3ab224c695e1d974c49a Author: Sebastian Andrzej Siewior Date: Tue Feb 2 18:01:04 2021 +0100 chelsio: cxgb: Disable the card on error in threaded interrupt t1_fatal_err() is invoked from the interrupt handler. The bad part is that it invokes (via t1_sge_stop()) del_timer_sync() and tasklet_kill(). Both functions must not be called from an interrupt because it is possible that it will wait for the completion of the timer/tasklet it just interrupted. In case of a fatal error, use t1_interrupts_disable() to disable all interrupt sources and then wake the interrupt thread with F_PL_INTR_SGE_ERR as pending flag. The threaded-interrupt will stop the card via t1_sge_stop() and not re-enable the interrupts again. Signed-off-by: Sebastian Andrzej Siewior commit f7b9825f0b493c17bc202d54ff009d328ddde5aa Author: Sebastian Andrzej Siewior Date: Tue Feb 2 18:01:03 2021 +0100 chelsio: cxgb: Replace the workqueue with threaded interrupt The external interrupt (F_PL_INTR_EXT) needs to be handled in a process context and this is accomplished by utilizing a workqueue. The process context can also be provided by a threaded interrupt instead of a workqueue. The threaded interrupt can be used later for other interrupt related processing which require non-atomic context without using yet another workqueue. free_irq() also ensures that the thread is done which is currently missing (the worker could continue after the module has been removed). Save pending flags in pending_thread_intr. Use the same mechanism to disable F_PL_INTR_EXT as interrupt source like it is used before the worker is scheduled. Enable the interrupt again once t1_elmer0_ext_intr_handler() is done. Signed-off-by: Sebastian Andrzej Siewior commit ddc60d21536014c06cf2315775164067de3b0309 Author: Sebastian Andrzej Siewior Date: Tue Feb 9 18:58:01 2021 +0100 powerpc: Fix preempt-lazy on 64bit Schedule must not be invoked on a lazy-preempt request if preemption is disabled. This regression was introduced in v5.9-rc2-rt1. Check the preemption counter before looking at the lazy-preemption counter and both have to be 0. Signed-off-by: Sebastian Andrzej Siewior commit c447c09e6e7c79ae3ce4af93e8fb82a8cf23aefa Author: Sebastian Andrzej Siewior Date: Tue Feb 16 16:14:27 2021 +0100 v5.11-rt4 Signed-off-by: Sebastian Andrzej Siewior commit 8087f8f47c4f82262687602a1b71dac07ceb0255 Merge: 0d095070fc88 f40ddce88593 Author: Sebastian Andrzej Siewior Date: Tue Feb 16 16:14:03 2021 +0100 Merge tag 'v5.11' into linux-5.11.y-rt Linux 5.11 commit 0d095070fc8864d4a4e3470398c77e69f018cf6b Author: Sebastian Andrzej Siewior Date: Fri Jan 29 19:25:58 2021 +0100 v5.11-rc5-rt3 Signed-off-by: Sebastian Andrzej Siewior commit ae326455df7dd7c0c8e79358da05105d2eba72f0 Author: Sebastian Andrzej Siewior Date: Fri Jan 29 19:24:31 2021 +0100 softirq: Update the WIP This is an all-in-one update of the softirq work. The tasklet_disable() interface can now sleep and driver which were identified as sleeping in atomic conext (on !RT) were converted to the legacy interface. Signed-off-by: Sebastian Andrzej Siewior commit 2292a1fce139856c9a56ffaac81763f636cdc26b Author: Sebastian Andrzej Siewior Date: Fri Jan 29 18:29:31 2021 +0100 v5.11-rc5-rt2 Signed-off-by: Sebastian Andrzej Siewior commit 47d84119ada4657044eda3cbd2b30d412d67958d Merge: 4a7805e223e0 6ee1d745b7c9 Author: Sebastian Andrzej Siewior Date: Fri Jan 29 18:29:10 2021 +0100 Merge tag 'v5.11-rc5' into linux-5.11.y-rt Linux 5.11-rc5 commit 4a7805e223e04660df86d5ca0d16f1acdd3c91eb Author: Thomas Gleixner Date: Fri Jul 8 20:25:16 2011 +0200 Add localversion for -RT release Signed-off-by: Thomas Gleixner commit e4bddd4bd366faad84e133da287f39be98c06f5a Author: Clark Williams Date: Sat Jul 30 21:55:53 2011 -0500 sysfs: Add /sys/kernel/realtime entry Add a /sys/kernel entry to indicate that the kernel is a realtime kernel. Clark says that he needs this for udev rules, udev needs to evaluate if its a PREEMPT_RT kernel a few thousand times and parsing uname output is too slow or so. Are there better solutions? Should it exist and return 0 on !-rt? Signed-off-by: Clark Williams Signed-off-by: Peter Zijlstra commit ec20fd947d2b5b1b5b13e95372d7691135390c21 Author: Ingo Molnar Date: Fri Jul 3 08:29:57 2009 -0500 genirq: Disable irqpoll on -rt Creates long latencies for no value Signed-off-by: Ingo Molnar Signed-off-by: Thomas Gleixner commit 8fca0392f42f73d2061642588b9e15e28e30b050 Author: Matt Fleming Date: Tue Apr 7 10:54:13 2020 +0100 signal: Prevent double-free of user struct The way user struct reference counting works changed significantly with, fda31c50292a ("signal: avoid double atomic counter increments for user accounting") Now user structs are only freed once the last pending signal is dequeued. Make sigqueue_free_current() follow this new convention to avoid freeing the user struct multiple times and triggering this warning: refcount_t: underflow; use-after-free. WARNING: CPU: 0 PID: 6794 at lib/refcount.c:288 refcount_dec_not_one+0x45/0x50 Call Trace: refcount_dec_and_lock_irqsave+0x16/0x60 free_uid+0x31/0xa0 __dequeue_signal+0x17c/0x190 dequeue_signal+0x5a/0x1b0 do_sigtimedwait+0x208/0x250 __x64_sys_rt_sigtimedwait+0x6f/0xd0 do_syscall_64+0x72/0x200 entry_SYSCALL_64_after_hwframe+0x49/0xbe Signed-off-by: Matt Fleming Reported-by: Daniel Wagner Signed-off-by: Sebastian Andrzej Siewior commit 00e87dacee91feca7bc5f96ba2456a390425b406 Author: Thomas Gleixner Date: Fri Jul 3 08:44:56 2009 -0500 signals: Allow rt tasks to cache one sigqueue struct To avoid allocation allow rt tasks to cache one sigqueue struct in task struct. Signed-off-by: Thomas Gleixner commit efbc1d830ff553af99ec6f26f24dfd5fa974c29a Author: Haris Okanovic Date: Tue Aug 15 15:13:08 2017 -0500 tpm_tis: fix stall after iowrite*()s ioread8() operations to TPM MMIO addresses can stall the cpu when immediately following a sequence of iowrite*()'s to the same region. For example, cyclitest measures ~400us latency spikes when a non-RT usermode application communicates with an SPI-based TPM chip (Intel Atom E3940 system, PREEMPT_RT kernel). The spikes are caused by a stalling ioread8() operation following a sequence of 30+ iowrite8()s to the same address. I believe this happens because the write sequence is buffered (in cpu or somewhere along the bus), and gets flushed on the first LOAD instruction (ioread*()) that follows. The enclosed change appears to fix this issue: read the TPM chip's access register (status code) after every iowrite*() operation to amortize the cost of flushing data to chip across multiple instructions. Signed-off-by: Haris Okanovic Signed-off-by: Sebastian Andrzej Siewior commit 480df0a46fe89b1694417c0fdfa4ede990932606 Author: Mike Galbraith Date: Thu Mar 31 04:08:28 2016 +0200 drivers/block/zram: Replace bit spinlocks with rtmutex for -rt They're nondeterministic, and lead to ___might_sleep() splats in -rt. OTOH, they're a lot less wasteful than an rtmutex per page. Signed-off-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior commit 6866f593b45cfeba985a3b0e0c81f000cc2981e0 Author: Sebastian Andrzej Siewior Date: Fri Oct 11 13:14:41 2019 +0200 POWERPC: Allow to enable RT Allow to select RT. Signed-off-by: Sebastian Andrzej Siewior commit a1b5278d022f10c31a6ab2a49f3462e024791b94 Author: Sebastian Andrzej Siewior Date: Fri Jan 8 19:48:21 2021 +0100 powerpc: Avoid recursive header includes - The include of bug.h leads to an include of printk.h which gets back to spinlock.h and complains then about missing xchg(). Remove bug.h and add bits.h which is needed for BITS_PER_BYTE. - Avoid the "please don't include this file directly" error from rwlock-rt. Allow an include from/with rtmutex.h. Signed-off-by: Sebastian Andrzej Siewior commit eb21f033b5ddf19272f6b77db81ba5c34ec8f163 Author: Sebastian Andrzej Siewior Date: Tue Mar 26 18:31:29 2019 +0100 powerpc/stackprotector: work around stack-guard init from atomic This is invoked from the secondary CPU in atomic context. On x86 we use tsc instead. On Power we XOR it against mftb() so lets use stack address as the initial value. Cc: stable-rt@vger.kernel.org Signed-off-by: Sebastian Andrzej Siewior commit 86284322c966c029064ba7124f82a016e50384cf Author: Bogdan Purcareata Date: Fri Apr 24 15:53:13 2015 +0000 powerpc/kvm: Disable in-kernel MPIC emulation for PREEMPT_RT While converting the openpic emulation code to use a raw_spinlock_t enables guests to run on RT, there's still a performance issue. For interrupts sent in directed delivery mode with a multiple CPU mask, the emulated openpic will loop through all of the VCPUs, and for each VCPUs, it call IRQ_check, which will loop through all the pending interrupts for that VCPU. This is done while holding the raw_lock, meaning that in all this time the interrupts and preemption are disabled on the host Linux. A malicious user app can max both these number and cause a DoS. This temporary fix is sent for two reasons. First is so that users who want to use the in-kernel MPIC emulation are aware of the potential latencies, thus making sure that the hardware MPIC and their usage scenario does not involve interrupts sent in directed delivery mode, and the number of possible pending interrupts is kept small. Secondly, this should incentivize the development of a proper openpic emulation that would be better suited for RT. Acked-by: Scott Wood Signed-off-by: Bogdan Purcareata Signed-off-by: Sebastian Andrzej Siewior commit 343e792f4417de8fe1f001b05d1a4607449b7ef9 Author: Sebastian Andrzej Siewior Date: Tue Mar 26 18:31:54 2019 +0100 powerpc/pseries/iommu: Use a locallock instead local_irq_save() The locallock protects the per-CPU variable tce_page. The function attempts to allocate memory while tce_page is protected (by disabling interrupts). Use local_irq_save() instead of local_irq_disable(). Cc: stable-rt@vger.kernel.org Signed-off-by: Sebastian Andrzej Siewior commit 780658dac52f648c343eab4fcde017949e2c268c Author: Sebastian Andrzej Siewior Date: Fri Jul 26 11:30:49 2019 +0200 powerpc: traps: Use PREEMPT_RT Add PREEMPT_RT to the backtrace if enabled. Signed-off-by: Sebastian Andrzej Siewior commit cbcb2ae9aac22882bdecc64ff9958fbffde01061 Author: Sebastian Andrzej Siewior Date: Fri Oct 11 13:14:35 2019 +0200 ARM64: Allow to enable RT Allow to select RT. Signed-off-by: Sebastian Andrzej Siewior commit 79d6413650a13d0676a4a5a79de7f6db4e3ecbb7 Author: Sebastian Andrzej Siewior Date: Fri Oct 11 13:14:29 2019 +0200 ARM: Allow to enable RT Allow to select RT. Signed-off-by: Sebastian Andrzej Siewior commit f7e4aa4977f2f85f7efd0e536caa9a5ff5297fa5 Author: Sebastian Andrzej Siewior Date: Thu Nov 7 17:49:20 2019 +0100 x86: Enable RT also on 32bit Signed-off-by: Sebastian Andrzej Siewior commit a652cb6203f6e0bb3d6a9e013b4fd40a5d70182b Author: Sebastian Andrzej Siewior Date: Wed Jul 25 14:02:38 2018 +0200 arm64: fpsimd: Delay freeing memory in fpsimd_flush_thread() fpsimd_flush_thread() invokes kfree() via sve_free() within a preempt disabled section which is not working on -RT. Delay freeing of memory until preemption is enabled again. Signed-off-by: Sebastian Andrzej Siewior commit ef63a31e731610a8d8990fc0b406129a7051e15c Author: Josh Cartwright Date: Thu Feb 11 11:54:01 2016 -0600 KVM: arm/arm64: downgrade preempt_disable()d region to migrate_disable() kvm_arch_vcpu_ioctl_run() disables the use of preemption when updating the vgic and timer states to prevent the calling task from migrating to another CPU. It does so to prevent the task from writing to the incorrect per-CPU GIC distributor registers. On -rt kernels, it's possible to maintain the same guarantee with the use of migrate_{disable,enable}(), with the added benefit that the migrate-disabled region is preemptible. Update kvm_arch_vcpu_ioctl_run() to do so. Cc: Christoffer Dall Reported-by: Manish Jaggi Signed-off-by: Josh Cartwright Signed-off-by: Sebastian Andrzej Siewior commit dc987cef674a5fcfcb4b18b9d250606fc11dd43d Author: Josh Cartwright Date: Thu Feb 11 11:54:00 2016 -0600 genirq: update irq_set_irqchip_state documentation On -rt kernels, the use of migrate_disable()/migrate_enable() is sufficient to guarantee a task isn't moved to another CPU. Update the irq_set_irqchip_state() documentation to reflect this. Signed-off-by: Josh Cartwright Signed-off-by: Sebastian Andrzej Siewior commit df91fcc6a5badaed32ef0ccc5db1f05e77c658ae Author: Yadi.hu Date: Wed Dec 10 10:32:09 2014 +0800 ARM: enable irq in translation/section permission fault handlers Probably happens on all ARM, with CONFIG_PREEMPT_RT CONFIG_DEBUG_ATOMIC_SLEEP This simple program.... int main() { *((char*)0xc0001000) = 0; }; [ 512.742724] BUG: sleeping function called from invalid context at kernel/rtmutex.c:658 [ 512.743000] in_atomic(): 0, irqs_disabled(): 128, pid: 994, name: a [ 512.743217] INFO: lockdep is turned off. [ 512.743360] irq event stamp: 0 [ 512.743482] hardirqs last enabled at (0): [< (null)>] (null) [ 512.743714] hardirqs last disabled at (0): [] copy_process+0x3b0/0x11c0 [ 512.744013] softirqs last enabled at (0): [] copy_process+0x3b0/0x11c0 [ 512.744303] softirqs last disabled at (0): [< (null)>] (null) [ 512.744631] [] (unwind_backtrace+0x0/0x104) [ 512.745001] [] (dump_stack+0x20/0x24) [ 512.745355] [] (__might_sleep+0x1dc/0x1e0) [ 512.745717] [] (rt_spin_lock+0x34/0x6c) [ 512.746073] [] (do_force_sig_info+0x34/0xf0) [ 512.746457] [] (force_sig_info+0x18/0x1c) [ 512.746829] [] (__do_user_fault+0x9c/0xd8) [ 512.747185] [] (do_bad_area+0x7c/0x94) [ 512.747536] [] (do_sect_fault+0x40/0x48) [ 512.747898] [] (do_DataAbort+0x40/0xa0) [ 512.748181] Exception stack(0xecaa1fb0 to 0xecaa1ff8) Oxc0000000 belongs to kernel address space, user task can not be allowed to access it. For above condition, correct result is that test case should receive a “segment fault” and exits but not stacks. the root cause is commit 02fe2845d6a8 ("avoid enabling interrupts in prefetch/data abort handlers"),it deletes irq enable block in Data abort assemble code and move them into page/breakpiont/alignment fault handlers instead. But author does not enable irq in translation/section permission fault handlers. ARM disables irq when it enters exception/ interrupt mode, if kernel doesn't enable irq, it would be still disabled during translation/section permission fault. We see the above splat because do_force_sig_info is still called with IRQs off, and that code eventually does a: spin_lock_irqsave(&t->sighand->siglock, flags); As this is architecture independent code, and we've not seen any other need for other arch to have the siglock converted to raw lock, we can conclude that we should enable irq for ARM translation/section permission exception. Signed-off-by: Yadi.hu Signed-off-by: Sebastian Andrzej Siewior commit e47ed9d6ca701b098239f3b5200b1463b4c21663 Author: Thomas Gleixner Date: Tue Jan 8 21:36:51 2013 +0100 tty/serial/pl011: Make the locking work on RT The lock is a sleeping lock and local_irq_save() is not the optimsation we are looking for. Redo it to make it work on -RT and non-RT. Signed-off-by: Thomas Gleixner commit 9f218ccdbb087185a8f03eda26eaea3c370adf1e Author: Thomas Gleixner Date: Thu Jul 28 13:32:57 2011 +0200 tty/serial/omap: Make the locking RT aware The lock is a sleeping lock and local_irq_save() is not the optimsation we are looking for. Redo it to make it work on -RT and non-RT. Signed-off-by: Thomas Gleixner commit c4a2635c4051a0aeeff01dc34820591f7936bafc Author: Sebastian Andrzej Siewior Date: Thu Jan 23 14:45:59 2014 +0100 leds: trigger: disable CPU trigger on -RT as it triggers: |CPU: 0 PID: 0 Comm: swapper Not tainted 3.12.8-rt10 #141 |[] (unwind_backtrace+0x0/0xf8) from [] (show_stack+0x1c/0x20) |[] (show_stack+0x1c/0x20) from [] (dump_stack+0x20/0x2c) |[] (dump_stack+0x20/0x2c) from [] (__might_sleep+0x13c/0x170) |[] (__might_sleep+0x13c/0x170) from [] (__rt_spin_lock+0x28/0x38) |[] (__rt_spin_lock+0x28/0x38) from [] (rt_read_lock+0x68/0x7c) |[] (rt_read_lock+0x68/0x7c) from [] (led_trigger_event+0x2c/0x5c) |[] (led_trigger_event+0x2c/0x5c) from [] (ledtrig_cpu+0x54/0x5c) |[] (ledtrig_cpu+0x54/0x5c) from [] (arch_cpu_idle_exit+0x18/0x1c) |[] (arch_cpu_idle_exit+0x18/0x1c) from [] (cpu_startup_entry+0xa8/0x234) |[] (cpu_startup_entry+0xa8/0x234) from [] (rest_init+0xb8/0xe0) |[] (rest_init+0xb8/0xe0) from [] (start_kernel+0x2c4/0x380) Signed-off-by: Sebastian Andrzej Siewior commit 7c66becf10fae3c1c0def9400c5798ea5fc35f0f Author: Thomas Gleixner Date: Wed Jul 8 17:14:48 2015 +0200 jump-label: disable if stop_machine() is used Some architectures are using stop_machine() while switching the opcode which leads to latency spikes. The architectures which use stop_machine() atm: - ARM stop machine - s390 stop machine The architecures which use other sorcery: - MIPS - X86 - powerpc - sparc - arm64 Signed-off-by: Thomas Gleixner [bigeasy: only ARM for now] Signed-off-by: Sebastian Andrzej Siewior commit c31d3bf2ec73cdf7b25c213e6964d7e8525072cd Author: Anders Roxell Date: Thu May 14 17:52:17 2015 +0200 arch/arm64: Add lazy preempt support arm64 is missing support for PREEMPT_RT. The main feature which is lacking is support for lazy preemption. The arch-specific entry code, thread information structure definitions, and associated data tables have to be extended to provide this support. Then the Kconfig file has to be extended to indicate the support is available, and also to indicate that support for full RT preemption is now available. Signed-off-by: Anders Roxell commit 7ea2f45d08395c8f830794da4e10107de5c822b0 Author: Thomas Gleixner Date: Thu Nov 1 10:14:11 2012 +0100 powerpc: Add support for lazy preemption Implement the powerpc pieces for lazy preempt. Signed-off-by: Thomas Gleixner commit c971bf08b7dcb08bd449690ce7702b5718465fab Author: Thomas Gleixner Date: Wed Oct 31 12:04:11 2012 +0100 arm: Add support for lazy preemption Implement the arm pieces for lazy preempt. Signed-off-by: Thomas Gleixner commit cceacc2b9634a2efa0a1fb58ea2970e14f255185 Author: Thomas Gleixner Date: Thu Nov 1 11:03:47 2012 +0100 x86: Support for lazy preemption Implement the x86 pieces for lazy preempt. Signed-off-by: Thomas Gleixner commit 503c6a6802f5709b818499fb9a7b07d6e79214e0 Author: Sebastian Andrzej Siewior Date: Tue Jun 30 11:45:14 2020 +0200 x86/entry: Use should_resched() in idtentry_exit_cond_resched() The TIF_NEED_RESCHED bit is inlined on x86 into the preemption counter. By using should_resched(0) instead of need_resched() the same check can be performed which uses the same variable as 'preempt_count()` which was issued before. Use should_resched(0) instead need_resched(). Signed-off-by: Sebastian Andrzej Siewior commit 9f3d3e7525ca323480360ea48650840339fbb17b Author: Thomas Gleixner Date: Fri Oct 26 18:50:54 2012 +0100 sched: Add support for lazy preemption It has become an obsession to mitigate the determinism vs. throughput loss of RT. Looking at the mainline semantics of preemption points gives a hint why RT sucks throughput wise for ordinary SCHED_OTHER tasks. One major issue is the wakeup of tasks which are right away preempting the waking task while the waking task holds a lock on which the woken task will block right after having preempted the wakee. In mainline this is prevented due to the implicit preemption disable of spin/rw_lock held regions. On RT this is not possible due to the fully preemptible nature of sleeping spinlocks. Though for a SCHED_OTHER task preempting another SCHED_OTHER task this is really not a correctness issue. RT folks are concerned about SCHED_FIFO/RR tasks preemption and not about the purely fairness driven SCHED_OTHER preemption latencies. So I introduced a lazy preemption mechanism which only applies to SCHED_OTHER tasks preempting another SCHED_OTHER task. Aside of the existing preempt_count each tasks sports now a preempt_lazy_count which is manipulated on lock acquiry and release. This is slightly incorrect as for lazyness reasons I coupled this on migrate_disable/enable so some other mechanisms get the same treatment (e.g. get_cpu_light). Now on the scheduler side instead of setting NEED_RESCHED this sets NEED_RESCHED_LAZY in case of a SCHED_OTHER/SCHED_OTHER preemption and therefor allows to exit the waking task the lock held region before the woken task preempts. That also works better for cross CPU wakeups as the other side can stay in the adaptive spinning loop. For RT class preemption there is no change. This simply sets NEED_RESCHED and forgoes the lazy preemption counter. Initial test do not expose any observable latency increasement, but history shows that I've been proven wrong before :) The lazy preemption mode is per default on, but with CONFIG_SCHED_DEBUG enabled it can be disabled via: # echo NO_PREEMPT_LAZY >/sys/kernel/debug/sched_features and reenabled via # echo PREEMPT_LAZY >/sys/kernel/debug/sched_features The test results so far are very machine and workload dependent, but there is a clear trend that it enhances the non RT workload performance. Signed-off-by: Thomas Gleixner commit 9ba18a014cc38ae1c62d5efd2caa75afbe513d43 Author: Thomas Gleixner Date: Fri Jul 3 08:44:34 2009 -0500 mm/scatterlist: Do not disable irqs on RT For -RT it is enough to keep pagefault disabled (which is currently handled by kmap_atomic()). Signed-off-by: Thomas Gleixner commit 38066bdf4017ce46e44dc85a14157233f69e0688 Author: Sebastian Andrzej Siewior Date: Wed Aug 7 18:15:38 2019 +0200 x86: Allow to enable RT Allow to select RT. Signed-off-by: Sebastian Andrzej Siewior commit 8625844736a47337903af1d8eabb596259cfcf42 Author: Mike Galbraith Date: Sun Jan 8 09:32:25 2017 +0100 cpuset: Convert callback_lock to raw_spinlock_t The two commits below add up to a cpuset might_sleep() splat for RT: 8447a0fee974 cpuset: convert callback_mutex to a spinlock 344736f29b35 cpuset: simplify cpuset_node_allowed API BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:995 in_atomic(): 0, irqs_disabled(): 1, pid: 11718, name: cset CPU: 135 PID: 11718 Comm: cset Tainted: G E 4.10.0-rt1-rt #4 Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0056.R01.1409242327 09/24/2014 Call Trace: ? dump_stack+0x5c/0x81 ? ___might_sleep+0xf4/0x170 ? rt_spin_lock+0x1c/0x50 ? __cpuset_node_allowed+0x66/0xc0 ? ___slab_alloc+0x390/0x570 ? anon_vma_fork+0x8f/0x140 ? copy_page_range+0x6cf/0xb00 ? anon_vma_fork+0x8f/0x140 ? __slab_alloc.isra.74+0x5a/0x81 ? anon_vma_fork+0x8f/0x140 ? kmem_cache_alloc+0x1b5/0x1f0 ? anon_vma_fork+0x8f/0x140 ? copy_process.part.35+0x1670/0x1ee0 ? _do_fork+0xdd/0x3f0 ? _do_fork+0xdd/0x3f0 ? do_syscall_64+0x61/0x170 ? entry_SYSCALL64_slow_path+0x25/0x25 The later ensured that a NUMA box WILL take callback_lock in atomic context by removing the allocator and reclaim path __GFP_HARDWALL usage which prevented such contexts from taking callback_mutex. One option would be to reinstate __GFP_HARDWALL protections for RT, however, as the 8447a0fee974 changelog states: The callback_mutex is only used to synchronize reads/updates of cpusets' flags and cpu/node masks. These operations should always proceed fast so there's no reason why we can't use a spinlock instead of the mutex. Cc: stable-rt@vger.kernel.org Signed-off-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior commit 4f6ff38ae01ab4318871f69907b16e766579302d Author: Sebastian Andrzej Siewior Date: Tue Jul 7 12:25:11 2020 +0200 drm/i915/gt: Only disable interrupts for the timeline lock on !force-threaded According to commit d67739268cf0e ("drm/i915/gt: Mark up the nested engine-pm timeline lock as irqsafe") the intrrupts are disabled the code may be called from an interrupt handler and from preemptible context. With `force_irqthreads' set the timeline mutex is never observed in IRQ context so it is not neede to disable interrupts. Disable only interrupts if not in `force_irqthreads' mode. Signed-off-by: Sebastian Andrzej Siewior commit eee65101691bc4eb4bb09fc4c1efed365edbdf54 Author: Sebastian Andrzej Siewior Date: Wed Dec 19 10:47:02 2018 +0100 drm/i915: skip DRM_I915_LOW_LEVEL_TRACEPOINTS with NOTRACE The order of the header files is important. If this header file is included after tracepoint.h was included then the NOTRACE here becomes a nop. Currently this happens for two .c files which use the tracepoitns behind DRM_I915_LOW_LEVEL_TRACEPOINTS. Signed-off-by: Sebastian Andrzej Siewior commit 2fb51314b88277ee59b0fc8cf7756b598516f4d7 Author: Sebastian Andrzej Siewior Date: Thu Dec 6 09:52:20 2018 +0100 drm/i915: disable tracing on -RT Luca Abeni reported this: | BUG: scheduling while atomic: kworker/u8:2/15203/0x00000003 | CPU: 1 PID: 15203 Comm: kworker/u8:2 Not tainted 4.19.1-rt3 #10 | Call Trace: | rt_spin_lock+0x3f/0x50 | gen6_read32+0x45/0x1d0 [i915] | g4x_get_vblank_counter+0x36/0x40 [i915] | trace_event_raw_event_i915_pipe_update_start+0x7d/0xf0 [i915] The tracing events use trace_i915_pipe_update_start() among other events use functions acquire spin locks. A few trace points use intel_get_crtc_scanline(), others use ->get_vblank_counter() wich also might acquire a sleeping lock. Based on this I don't see any other way than disable trace points on RT. Cc: stable-rt@vger.kernel.org Reported-by: Luca Abeni Signed-off-by: Sebastian Andrzej Siewior commit 377adf5d1d94b38540eb9eae61c6f1d3b82214a8 Author: Mike Galbraith Date: Sat Feb 27 09:01:42 2016 +0100 drm/i915: Don't disable interrupts on PREEMPT_RT during atomic updates Commit 8d7849db3eab7 ("drm/i915: Make sprite updates atomic") started disabling interrupts across atomic updates. This breaks on PREEMPT_RT because within this section the code attempt to acquire spinlock_t locks which are sleeping locks on PREEMPT_RT. According to the comment the interrupts are disabled to avoid random delays and not required for protection or synchronisation. Don't disable interrupts on PREEMPT_RT during atomic updates. [bigeasy: drop local locks, commit message] Signed-off-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior commit 5a89f39d3c16f145006f2fa6e08935fc452b37f0 Author: Mike Galbraith Date: Sat Feb 27 08:09:11 2016 +0100 drm,radeon,i915: Use preempt_disable/enable_rt() where recommended DRM folks identified the spots, so use them. Signed-off-by: Mike Galbraith Cc: Sebastian Andrzej Siewior Cc: linux-rt-users Signed-off-by: Thomas Gleixner commit aa42f95e383c5358ff3c2a61a41f9caccd0b50da Author: Sebastian Andrzej Siewior Date: Tue Oct 17 16:36:18 2017 +0200 lockdep: disable self-test The self-test wasn't always 100% accurate for RT. We disabled a few tests which failed because they had a different semantic for RT. Some still reported false positives. Now the selftest locks up the system during boot and it needs to be investigated… Signed-off-by: Sebastian Andrzej Siewior commit 203e01cd8d6ac352f6f4030f96a206f6f6866898 Author: Josh Cartwright Date: Wed Jan 28 13:08:45 2015 -0600 lockdep: selftest: fix warnings due to missing PREEMPT_RT conditionals "lockdep: Selftest: Only do hardirq context test for raw spinlock" disabled the execution of certain tests with PREEMPT_RT, but did not prevent the tests from still being defined. This leads to warnings like: ./linux/lib/locking-selftest.c:574:1: warning: 'irqsafe1_hard_rlock_12' defined but not used [-Wunused-function] ./linux/lib/locking-selftest.c:574:1: warning: 'irqsafe1_hard_rlock_21' defined but not used [-Wunused-function] ./linux/lib/locking-selftest.c:577:1: warning: 'irqsafe1_hard_wlock_12' defined but not used [-Wunused-function] ./linux/lib/locking-selftest.c:577:1: warning: 'irqsafe1_hard_wlock_21' defined but not used [-Wunused-function] ./linux/lib/locking-selftest.c:580:1: warning: 'irqsafe1_soft_spin_12' defined but not used [-Wunused-function] ... Fixed by wrapping the test definitions in #ifndef CONFIG_PREEMPT_RT conditionals. Signed-off-by: Josh Cartwright Signed-off-by: Xander Huff Acked-by: Gratian Crisan Signed-off-by: Sebastian Andrzej Siewior commit c94f7af8eb22ce500072a25c9442e09a6e5fd669 Author: Yong Zhang Date: Mon Apr 16 15:01:56 2012 +0800 lockdep: selftest: Only do hardirq context test for raw spinlock On -rt there is no softirq context any more and rwlock is sleepable, disable softirq context test and rwlock+irq test. Signed-off-by: Yong Zhang Cc: Yong Zhang Link: http://lkml.kernel.org/r/1334559716-18447-3-git-send-email-yong.zhang0@gmail.com Signed-off-by: Thomas Gleixner commit e5ae80d73b84d9f7736b0861353083be59b2cda0 Author: Thomas Gleixner Date: Sun Jul 17 18:51:23 2011 +0200 lockdep: Make it RT aware teach lockdep that we don't really do softirqs on -RT. Signed-off-by: Thomas Gleixner commit 0117ff1af5a0af3b26d77e8e1a47262731a558c5 Author: Priyanka Jain Date: Thu May 17 09:35:11 2012 +0530 net: Remove preemption disabling in netif_rx() 1)enqueue_to_backlog() (called from netif_rx) should be bind to a particluar CPU. This can be achieved by disabling migration. No need to disable preemption 2)Fixes crash "BUG: scheduling while atomic: ksoftirqd" in case of RT. If preemption is disabled, enqueue_to_backog() is called in atomic context. And if backlog exceeds its count, kfree_skb() is called. But in RT, kfree_skb() might gets scheduled out, so it expects non atomic context. -Replace preempt_enable(), preempt_disable() with migrate_enable(), migrate_disable() respectively -Replace get_cpu(), put_cpu() with get_cpu_light(), put_cpu_light() respectively Signed-off-by: Priyanka Jain Acked-by: Rajan Srivastava Cc: Link: http://lkml.kernel.org/r/1337227511-2271-1-git-send-email-Priyanka.Jain@freescale.com Signed-off-by: Thomas Gleixner [bigeasy: Remove assumption about migrate_disable() from the description.] Signed-off-by: Sebastian Andrzej Siewior commit ba2c7627966101ec9ee0abf1dd125d5856fcb426 Author: Thomas Gleixner Date: Tue Aug 21 20:38:50 2012 +0200 random: Make it work on rt Delegate the random insertion to the forced threaded interrupt handler. Store the return IP of the hard interrupt handler in the irq descriptor and feed it into the random generator as a source of entropy. Signed-off-by: Thomas Gleixner commit e33831f3026802c9df51a037d342e861bdf2e0e0 Author: Thomas Gleixner Date: Thu Dec 16 14:25:18 2010 +0100 x86: stackprotector: Avoid random pool on rt CPU bringup calls into the random pool to initialize the stack canary. During boot that works nicely even on RT as the might sleep checks are disabled. During CPU hotplug the might sleep checks trigger. Making the locks in random raw is a major PITA, so avoid the call on RT is the only sensible solution. This is basically the same randomness which we get during boot where the random pool has no entropy and we rely on the TSC randomnness. Reported-by: Carsten Emde Signed-off-by: Thomas Gleixner commit 3fe18846774771b8eeff8ad1338fe1f66cdf06bc Author: Thomas Gleixner Date: Tue Jul 14 14:26:34 2015 +0200 panic: skip get_random_bytes for RT_FULL in init_oops_id Disable on -RT. If this is invoked from irq-context we will have problems to acquire the sleeping lock. Signed-off-by: Thomas Gleixner commit 1440bc0889c97da7c8dcb503205a02931f1b0950 Author: Sebastian Andrzej Siewior Date: Thu Jul 26 18:52:00 2018 +0200 crypto: cryptd - add a lock instead preempt_disable/local_bh_disable cryptd has a per-CPU lock which protected with local_bh_disable() and preempt_disable(). Add an explicit spin_lock to make the locking context more obvious and visible to lockdep. Since it is a per-CPU lock, there should be no lock contention on the actual spinlock. There is a small race-window where we could be migrated to another CPU after the cpu_queue has been obtain. This is not a problem because the actual ressource is protected by the spinlock. Signed-off-by: Sebastian Andrzej Siewior commit bb6ff58f59c74afc00d7a91968798ba6e61284c4 Author: Sebastian Andrzej Siewior Date: Thu Nov 30 13:40:10 2017 +0100 crypto: limit more FPU-enabled sections Those crypto drivers use SSE/AVX/… for their crypto work and in order to do so in kernel they need to enable the "FPU" in kernel mode which disables preemption. There are two problems with the way they are used: - the while loop which processes X bytes may create latency spikes and should be avoided or limited. - the cipher-walk-next part may allocate/free memory and may use kmap_atomic(). The whole kernel_fpu_begin()/end() processing isn't probably that cheap. It most likely makes sense to process as much of those as possible in one go. The new *_fpu_sched_rt() schedules only if a RT task is pending. Probably we should measure the performance those ciphers in pure SW mode and with this optimisations to see if it makes sense to keep them for RT. This kernel_fpu_resched() makes the code more preemptible which might hurt performance. Cc: stable-rt@vger.kernel.org Signed-off-by: Sebastian Andrzej Siewior commit 665b658ab75b93cc4bc70a88a4067b64bba5dfab Author: Sebastian Andrzej Siewior Date: Fri Feb 21 17:24:04 2014 +0100 crypto: Reduce preempt disabled regions, more algos Don Estabrook reported | kernel: WARNING: CPU: 2 PID: 858 at kernel/sched/core.c:2428 migrate_disable+0xed/0x100() | kernel: WARNING: CPU: 2 PID: 858 at kernel/sched/core.c:2462 migrate_enable+0x17b/0x200() | kernel: WARNING: CPU: 3 PID: 865 at kernel/sched/core.c:2428 migrate_disable+0xed/0x100() and his backtrace showed some crypto functions which looked fine. The problem is the following sequence: glue_xts_crypt_128bit() { blkcipher_walk_virt(); /* normal migrate_disable() */ glue_fpu_begin(); /* get atomic */ while (nbytes) { __glue_xts_crypt_128bit(); blkcipher_walk_done(); /* with nbytes = 0, migrate_enable() * while we are atomic */ }; glue_fpu_end() /* no longer atomic */ } and this is why the counter get out of sync and the warning is printed. The other problem is that we are non-preemptible between glue_fpu_begin() and glue_fpu_end() and the latency grows. To fix this, I shorten the FPU off region and ensure blkcipher_walk_done() is called with preemption enabled. This might hurt the performance because we now enable/disable the FPU state more often but we gain lower latency and the bug is gone. Reported-by: Don Estabrook Signed-off-by: Sebastian Andrzej Siewior commit 2eac8ddbaceed454a68fc981105e9537917a3b9a Author: Peter Zijlstra Date: Mon Nov 14 18:19:27 2011 +0100 x86: crypto: Reduce preempt disabled regions Restrict the preempt disabled regions to the actual floating point operations and enable preemption for the administrative actions. This is necessary on RT to avoid that kfree and other operations are called with preemption disabled. Reported-and-tested-by: Carsten Emde Signed-off-by: Peter Zijlstra Signed-off-by: Thomas Gleixner commit 58dfeab9b347fe4a684774a05f0918e6d242ced0 Author: Sebastian Andrzej Siewior Date: Tue Jun 23 15:32:51 2015 +0200 irqwork: push most work into softirq context Initially we defered all irqwork into softirq because we didn't want the latency spikes if perf or another user was busy and delayed the RT task. The NOHZ trigger (nohz_full_kick_work) was the first user that did not work as expected if it did not run in the original irqwork context so we had to bring it back somehow for it. push_irq_work_func is the second one that requires this. This patch adds the IRQ_WORK_HARD_IRQ which makes sure the callback runs in raw-irq context. Everything else is defered into softirq context. Without -RT we have the orignal behavior. This patch incorporates tglx orignal work which revoked a little bringing back the arch_irq_work_raise() if possible and a few fixes from Steven Rostedt and Mike Galbraith, [bigeasy: melt tglx's irq_work_tick_soft() which splits irq_work_tick() into a hard and soft variant] Signed-off-by: Sebastian Andrzej Siewior commit 9e663d359a80dd1c7fdef28aac455e2ecac520ce Author: Sebastian Andrzej Siewior Date: Wed Mar 30 13:36:29 2016 +0200 net: dev: always take qdisc's busylock in __dev_xmit_skb() The root-lock is dropped before dev_hard_start_xmit() is invoked and after setting the __QDISC___STATE_RUNNING bit. If this task is now pushed away by a task with a higher priority then the task with the higher priority won't be able to submit packets to the NIC directly instead they will be enqueued into the Qdisc. The NIC will remain idle until the task(s) with higher priority leave the CPU and the task with lower priority gets back and finishes the job. If we take always the busylock we ensure that the RT task can boost the low-prio task and submit the packet. Signed-off-by: Sebastian Andrzej Siewior commit f902d33bf925a3ca8893715a65f25af7d6ae83a7 Author: Sebastian Andrzej Siewior Date: Wed Sep 16 16:15:39 2020 +0200 net: Dequeue in dev_cpu_dead() without the lock Upstream uses skb_dequeue() to acquire lock of `input_pkt_queue'. The reason is to synchronize against a remote CPU which still thinks that the CPU is online enqueues packets to this CPU. There are no guarantees that the packet is enqueued before the callback is run, it just hope. RT however complains about an not initialized lock because it uses another lock for `input_pkt_queue' due to the IRQ-off nature of the context. Use the unlocked dequeue version for `input_pkt_queue'. Signed-off-by: Sebastian Andrzej Siewior commit 097ea81e5ae9dedf434755037836e1b5904ce8e8 Author: Thomas Gleixner Date: Tue Jul 12 15:38:34 2011 +0200 net: Use skbufhead with raw lock Use the rps lock as rawlock so we can keep irq-off regions. It looks low latency. However we can't kfree() from this context therefore we defer this to the softirq and use the tofree_queue list for it (similar to process_queue). Signed-off-by: Thomas Gleixner commit dcffe98632cf1598f801e5d7f342801b7cae35e1 Author: Thomas Gleixner Date: Sun Jul 17 21:41:35 2011 +0200 debugobjects: Make RT aware Avoid filling the pool / allocating memory with irqs off(). Signed-off-by: Thomas Gleixner commit be544a497867b5a4fe9e26d2da4e17af20be0dbb Author: Thomas Gleixner Date: Wed Mar 7 21:00:34 2012 +0100 fs: namespace: Use cpu_chill() in trylock loops Retry loops on RT might loop forever when the modifying side was preempted. Use cpu_chill() instead of cpu_relax() to let the system make progress. Signed-off-by: Thomas Gleixner commit 27edaa184748eb9acce2454b363d73b4f23bf02e Author: Thomas Gleixner Date: Wed Mar 7 20:51:03 2012 +0100 rt: Introduce cpu_chill() Retry loops on RT might loop forever when the modifying side was preempted. Add cpu_chill() to replace cpu_relax(). cpu_chill() defaults to cpu_relax() for non RT. On RT it puts the looping task to sleep for a tick so the preempted task can make progress. Steven Rostedt changed it to use a hrtimer instead of msleep(): | |Ulrich Obergfell pointed out that cpu_chill() calls msleep() which is woken |up by the ksoftirqd running the TIMER softirq. But as the cpu_chill() is |called from softirq context, it may block the ksoftirqd() from running, in |which case, it may never wake up the msleep() causing the deadlock. + bigeasy later changed to schedule_hrtimeout() |If a task calls cpu_chill() and gets woken up by a regular or spurious |wakeup and has a signal pending, then it exits the sleep loop in |do_nanosleep() and sets up the restart block. If restart->nanosleep.type is |not TI_NONE then this results in accessing a stale user pointer from a |previously interrupted syscall and a copy to user based on the stale |pointer or a BUG() when 'type' is not supported in nanosleep_copyout(). + bigeasy: add PF_NOFREEZE: | [....] Waiting for /dev to be fully populated... | ===================================== | [ BUG: udevd/229 still has locks held! ] | 3.12.11-rt17 #23 Not tainted | ------------------------------------- | 1 lock held by udevd/229: | #0: (&type->i_mutex_dir_key#2){+.+.+.}, at: lookup_slow+0x28/0x98 | | stack backtrace: | CPU: 0 PID: 229 Comm: udevd Not tainted 3.12.11-rt17 #23 | (unwind_backtrace+0x0/0xf8) from (show_stack+0x10/0x14) | (show_stack+0x10/0x14) from (dump_stack+0x74/0xbc) | (dump_stack+0x74/0xbc) from (do_nanosleep+0x120/0x160) | (do_nanosleep+0x120/0x160) from (hrtimer_nanosleep+0x90/0x110) | (hrtimer_nanosleep+0x90/0x110) from (cpu_chill+0x30/0x38) | (cpu_chill+0x30/0x38) from (dentry_kill+0x158/0x1ec) | (dentry_kill+0x158/0x1ec) from (dput+0x74/0x15c) | (dput+0x74/0x15c) from (lookup_real+0x4c/0x50) | (lookup_real+0x4c/0x50) from (__lookup_hash+0x34/0x44) | (__lookup_hash+0x34/0x44) from (lookup_slow+0x38/0x98) | (lookup_slow+0x38/0x98) from (path_lookupat+0x208/0x7fc) | (path_lookupat+0x208/0x7fc) from (filename_lookup+0x20/0x60) | (filename_lookup+0x20/0x60) from (user_path_at_empty+0x50/0x7c) | (user_path_at_empty+0x50/0x7c) from (user_path_at+0x14/0x1c) | (user_path_at+0x14/0x1c) from (vfs_fstatat+0x48/0x94) | (vfs_fstatat+0x48/0x94) from (SyS_stat64+0x14/0x30) | (SyS_stat64+0x14/0x30) from (ret_fast_syscall+0x0/0x48) Signed-off-by: Thomas Gleixner Signed-off-by: Steven Rostedt Signed-off-by: Sebastian Andrzej Siewior commit 03aa693f27a9f6a56f67744628b9c2ffda74e863 Author: Mike Galbraith Date: Wed Feb 18 16:05:28 2015 +0100 sunrpc: Make svc_xprt_do_enqueue() use get_cpu_light() |BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:915 |in_atomic(): 1, irqs_disabled(): 0, pid: 3194, name: rpc.nfsd |Preemption disabled at:[] svc_xprt_received+0x4b/0xc0 [sunrpc] |CPU: 6 PID: 3194 Comm: rpc.nfsd Not tainted 3.18.7-rt1 #9 |Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.404 11/06/2014 | ffff880409630000 ffff8800d9a33c78 ffffffff815bdeb5 0000000000000002 | 0000000000000000 ffff8800d9a33c98 ffffffff81073c86 ffff880408dd6008 | ffff880408dd6000 ffff8800d9a33cb8 ffffffff815c3d84 ffff88040b3ac000 |Call Trace: | [] dump_stack+0x4f/0x9e | [] __might_sleep+0xe6/0x150 | [] rt_spin_lock+0x24/0x50 | [] svc_xprt_do_enqueue+0x80/0x230 [sunrpc] | [] svc_xprt_received+0x4b/0xc0 [sunrpc] | [] svc_add_new_perm_xprt+0x6d/0x80 [sunrpc] | [] svc_addsock+0x143/0x200 [sunrpc] | [] write_ports+0x28c/0x340 [nfsd] | [] nfsctl_transaction_write+0x4c/0x80 [nfsd] | [] vfs_write+0xb3/0x1d0 | [] SyS_write+0x49/0xb0 | [] system_call_fastpath+0x16/0x1b Signed-off-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior commit 5adf645c9b1ff1dc33150c977ddb17056fc28263 Author: Thomas Gleixner Date: Sat Nov 12 14:00:48 2011 +0100 scsi/fcoe: Make RT aware. Do not disable preemption while taking sleeping locks. All user look safe for migrate_diable() only. Signed-off-by: Thomas Gleixner commit 4369dfe6d2e91dc2aff9578f8afac50c1e4f382c Author: Thomas Gleixner Date: Tue Apr 6 16:51:31 2010 +0200 md: raid5: Make raid5_percpu handling RT aware __raid_run_ops() disables preemption with get_cpu() around the access to the raid5_percpu variables. That causes scheduling while atomic spews on RT. Serialize the access to the percpu data with a lock and keep the code preemptible. Reported-by: Udo van den Heuvel Signed-off-by: Thomas Gleixner Tested-by: Udo van den Heuvel commit e6b5f604e85f72ba5a56c1af8fc75d5d5e106039 Author: Sebastian Andrzej Siewior Date: Tue Jul 14 14:26:34 2015 +0200 block/mq: do not invoke preempt_disable() preempt_disable() and get_cpu() don't play well together with the sleeping locks it tries to allocate later. It seems to be enough to replace it with get_cpu_light() and migrate_disable(). Signed-off-by: Sebastian Andrzej Siewior commit e1e26ea67d180f96f14b9d3b24048ad571021a57 Author: Thomas Gleixner Date: Tue Jul 12 11:39:36 2011 +0200 mm/vmalloc: Another preempt disable region which sucks Avoid the preempt disable version of get_cpu_var(). The inner-lock should provide enough serialisation. Signed-off-by: Thomas Gleixner commit 788826a028cbb5a7959792e7f2eef1ae9d9394d4 Author: Scott Wood Date: Wed Sep 11 17:57:29 2019 +0100 rcutorture: Avoid problematic critical section nesting on RT rcutorture was generating some nesting scenarios that are not reasonable. Constrain the state selection to avoid them. Example #1: 1. preempt_disable() 2. local_bh_disable() 3. preempt_enable() 4. local_bh_enable() On PREEMPT_RT, BH disabling takes a local lock only when called in non-atomic context. Thus, atomic context must be retained until after BH is re-enabled. Likewise, if BH is initially disabled in non-atomic context, it cannot be re-enabled in atomic context. Example #2: 1. rcu_read_lock() 2. local_irq_disable() 3. rcu_read_unlock() 4. local_irq_enable() If the thread is preempted between steps 1 and 2, rcu_read_unlock_special.b.blocked will be set, but it won't be acted on in step 3 because IRQs are disabled. Thus, reporting of the quiescent state will be delayed beyond the local_irq_enable(). For now, these scenarios will continue to be tested on non-PREEMPT_RT kernels, until debug checks are added to ensure that they are not happening elsewhere. Signed-off-by: Scott Wood Signed-off-by: Sebastian Andrzej Siewior commit 738720342a089eef583c21a98ead6d20ab75f682 Author: Sebastian Andrzej Siewior Date: Tue Nov 19 09:25:04 2019 +0100 locking: Make spinlock_t and rwlock_t a RCU section on RT On !RT a locked spinlock_t and rwlock_t disables preemption which implies a RCU read section. There is code that relies on that behaviour. Add an explicit RCU read section on RT while a sleeping lock (a lock which would disables preemption on !RT) acquired. Signed-off-by: Sebastian Andrzej Siewior commit cd9059653eacaacc9210ed1cca91a95f92e84639 Author: Sebastian Andrzej Siewior Date: Fri Aug 4 17:40:42 2017 +0200 locking: don't check for __LINUX_SPINLOCK_TYPES_H on -RT archs Upstream uses arch_spinlock_t within spinlock_t and requests that spinlock_types.h header file is included first. On -RT we have the rt_mutex with its raw_lock wait_lock which needs architectures' spinlock_types.h header file for its definition. However we need rt_mutex first because it is used to build the spinlock_t so that check does not work for us. Therefore I am dropping that check. Signed-off-by: Sebastian Andrzej Siewior commit cd11b57676f44ebf99554f3b77fedb5fe38e254a Author: Thomas Gleixner Date: Sun Jul 17 21:56:42 2011 +0200 trace: Add migrate-disabled counter to tracing output Signed-off-by: Thomas Gleixner commit 92544551c0ccdd707377eac54c36c6ad8747cab0 Author: Sebastian Andrzej Siewior Date: Sat May 27 19:02:06 2017 +0200 kernel/sched: add {put|get}_cpu_light() Signed-off-by: Sebastian Andrzej Siewior commit a0e4cd37c1dee1673a8f8507a6f46c6c537bc586 Author: Oleg Nesterov Date: Tue Nov 3 12:39:01 2020 +0100 ptrace: fix ptrace_unfreeze_traced() race with rt-lock The patch "ptrace: fix ptrace vs tasklist_lock race" changed ptrace_freeze_traced() to take task->saved_state into account, but ptrace_unfreeze_traced() has the same problem and needs a similar fix: it should check/update both ->state and ->saved_state. Reported-by: Luis Claudio R. Goncalves Fixes: "ptrace: fix ptrace vs tasklist_lock race" Signed-off-by: Oleg Nesterov Signed-off-by: Sebastian Andrzej Siewior Cc: stable-rt@vger.kernel.org commit 4bb9e153afa4c838985120d2d32888882e88422c Author: Sebastian Andrzej Siewior Date: Thu Aug 29 18:21:04 2013 +0200 ptrace: fix ptrace vs tasklist_lock race As explained by Alexander Fyodorov : |read_lock(&tasklist_lock) in ptrace_stop() is converted to mutex on RT kernel, |and it can remove __TASK_TRACED from task->state (by moving it to |task->saved_state). If parent does wait() on child followed by a sys_ptrace |call, the following race can happen: | |- child sets __TASK_TRACED in ptrace_stop() |- parent does wait() which eventually calls wait_task_stopped() and returns | child's pid |- child blocks on read_lock(&tasklist_lock) in ptrace_stop() and moves | __TASK_TRACED flag to saved_state |- parent calls sys_ptrace, which calls ptrace_check_attach() and wait_task_inactive() The patch is based on his initial patch where an additional check is added in case the __TASK_TRACED moved to ->saved_state. The pi_lock is taken in case the caller is interrupted between looking into ->state and ->saved_state. Signed-off-by: Sebastian Andrzej Siewior commit 6790854a965f065ac472ead33e7e093be72b94b4 Author: Grygorii Strashko Date: Tue Jul 21 19:43:56 2015 +0300 pid.h: include atomic.h This patch fixes build error: CC kernel/pid_namespace.o In file included from kernel/pid_namespace.c:11:0: include/linux/pid.h: In function 'get_pid': include/linux/pid.h:78:3: error: implicit declaration of function 'atomic_inc' [-Werror=implicit-function-declaration] atomic_inc(&pid->count); ^ which happens when CONFIG_PROVE_LOCKING=n CONFIG_DEBUG_SPINLOCK=n CONFIG_DEBUG_MUTEXES=n CONFIG_DEBUG_LOCK_ALLOC=n CONFIG_PID_NS=y Vanilla gets this via spinlock.h. Signed-off-by: Grygorii Strashko Signed-off-by: Sebastian Andrzej Siewior commit 1ebaeed9d71afb7a82103403915140857d8f5518 Author: Sebastian Andrzej Siewior Date: Fri Jun 16 19:03:16 2017 +0200 net/core: use local_bh_disable() in netif_rx_ni() In 2004 netif_rx_ni() gained a preempt_disable() section around netif_rx() and its do_softirq() + testing for it. The do_softirq() part is required because netif_rx() raises the softirq but does not invoke it. The preempt_disable() is required to remain on the same CPU which added the skb to the per-CPU list. All this can be avoided be putting this into a local_bh_disable()ed section. The local_bh_enable() part will invoke do_softirq() if required. Signed-off-by: Sebastian Andrzej Siewior commit 030180457ba6bf9b95b3c952d14034b6a78a7c3c Author: Thomas Gleixner Date: Mon Jul 18 13:59:17 2011 +0200 softirq: Disable softirq stacks for RT Disable extra stacks for softirqs. We want to preempt softirqs and having them on special IRQ-stack does not make this easier. Signed-off-by: Thomas Gleixner commit 6bc6f595fd4a7892585f82db58b1abf30b0f2cd4 Author: Thomas Gleixner Date: Sun Nov 13 17:17:09 2011 +0100 softirq: Check preemption after reenabling interrupts raise_softirq_irqoff() disables interrupts and wakes the softirq daemon, but after reenabling interrupts there is no preemption check, so the execution of the softirq thread might be delayed arbitrarily. In principle we could add that check to local_irq_enable/restore, but that's overkill as the rasie_softirq_irqoff() sections are the only ones which show this behaviour. Reported-by: Carsten Emde Signed-off-by: Thomas Gleixner commit c2b51928262ffbe2e0d816a122325684a18340a4 Author: Thomas Gleixner Date: Tue Sep 13 16:42:35 2011 +0200 sched: Disable TTWU_QUEUE on RT The queued remote wakeup mechanism can introduce rather large latencies if the number of migrated tasks is high. Disable it for RT. Signed-off-by: Thomas Gleixner commit 61e8303c99773b6e9ba0540f6b2fedec060c46d3 Author: Thomas Gleixner Date: Tue Jun 7 09:19:06 2011 +0200 sched: Do not account rcu_preempt_depth on RT in might_sleep() RT changes the rcu_preempt_depth semantics, so we cannot check for it in might_sleep(). Signed-off-by: Thomas Gleixner commit abecdf70d7811574c5f53402e2ff844fd37ad8d7 Author: Sebastian Andrzej Siewior Date: Mon Nov 21 19:31:08 2016 +0100 kernel/sched: move stack + kprobe clean up to __put_task_struct() There is no need to free the stack before the task struct (except for reasons mentioned in commit 68f24b08ee89 ("sched/core: Free the stack early if CONFIG_THREAD_INFO_IN_TASK")). This also comes handy on -RT because we can't free memory in preempt disabled region. vfree_atomic() delays the memory cleanup to a worker. Since we move everything to the RCU callback, we can also free it immediately. Cc: stable-rt@vger.kernel.org #for kprobe_flush_task() Signed-off-by: Sebastian Andrzej Siewior commit 27c7b9a8e102a9b3de9b7f63d276b8179d958956 Author: Thomas Gleixner Date: Mon Jun 6 12:20:33 2011 +0200 sched: Move mmdrop to RCU on RT Takes sleeping locks and calls into the memory allocator, so nothing we want to do in task switch and oder atomic contexts. Signed-off-by: Thomas Gleixner commit 6409ad6a78bfe637fcb78f92b42be48f17c280ae Author: Thomas Gleixner Date: Mon Jun 6 12:12:51 2011 +0200 sched: Limit the number of task migrations per batch Put an upper limit on the number of tasks which are migrated per batch to avoid large latencies. Signed-off-by: Thomas Gleixner commit 4d00b6dbe8b45ed40c877ae59e1062025cbc8d36 Author: Sebastian Andrzej Siewior Date: Mon Oct 28 12:19:57 2013 +0100 wait.h: include atomic.h | CC init/main.o |In file included from include/linux/mmzone.h:9:0, | from include/linux/gfp.h:4, | from include/linux/kmod.h:22, | from include/linux/module.h:13, | from init/main.c:15: |include/linux/wait.h: In function ‘wait_on_atomic_t’: |include/linux/wait.h:982:2: error: implicit declaration of function ‘atomic_read’ [-Werror=implicit-function-declaration] | if (atomic_read(val) == 0) | ^ This pops up on ARM. Non-RT gets its atomic.h include from spinlock.h Signed-off-by: Sebastian Andrzej Siewior commit 324fb6f5d6b09203b1c564ecf3b95163f578a3e0 Author: Thomas Gleixner Date: Sun Nov 6 12:26:18 2011 +0100 x86: kvm Require const tsc for RT Non constant TSC is a nightmare on bare metal already, but with virtualization it becomes a complete disaster because the workarounds are horrible latency wise. That's also a preliminary for running RT in a guest on top of a RT host. Signed-off-by: Thomas Gleixner commit 2bd09583ce8b4b06db535bbc963f5eb64fb4044a Author: Mike Galbraith Date: Tue Mar 22 11:16:09 2016 +0100 mm/zsmalloc: copy with get_cpu_var() and locking get_cpu_var() disables preemption and triggers a might_sleep() splat later. This is replaced with get_locked_var(). This bitspinlocks are replaced with a proper mutex which requires a slightly larger struct to allocate. Signed-off-by: Mike Galbraith [bigeasy: replace the bitspin_lock() with a mutex, get_locked_var(). Mike then fixed the size magic] Signed-off-by: Sebastian Andrzej Siewior commit a1de774403f14072918523f9413e3b1d27d2b033 Author: Sebastian Andrzej Siewior Date: Wed Jan 28 17:14:16 2015 +0100 mm/memcontrol: Replace local_irq_disable with local locks There are a few local_irq_disable() which then take sleeping locks. This patch converts them local locks. [bigeasy: Move unlock after memcg_check_events() in mem_cgroup_swapout(), pointed out by Matt Fleming ] Signed-off-by: Sebastian Andrzej Siewior commit b2325c1decb2b116270545a9ad3c4db3748701b4 Author: Yang Shi Date: Wed Oct 30 11:48:33 2013 -0700 mm/memcontrol: Don't call schedule_work_on in preemption disabled context The following trace is triggered when running ltp oom test cases: BUG: sleeping function called from invalid context at kernel/rtmutex.c:659 in_atomic(): 1, irqs_disabled(): 0, pid: 17188, name: oom03 Preemption disabled at:[] mem_cgroup_reclaim+0x90/0xe0 CPU: 2 PID: 17188 Comm: oom03 Not tainted 3.10.10-rt3 #2 Hardware name: Intel Corporation Calpella platform/MATXM-CORE-411-B, BIOS 4.6.3 08/18/2010 ffff88007684d730 ffff880070df9b58 ffffffff8169918d ffff880070df9b70 ffffffff8106db31 ffff88007688b4a0 ffff880070df9b88 ffffffff8169d9c0 ffff88007688b4a0 ffff880070df9bc8 ffffffff81059da1 0000000170df9bb0 Call Trace: [] dump_stack+0x19/0x1b [] __might_sleep+0xf1/0x170 [] rt_spin_lock+0x20/0x50 [] queue_work_on+0x61/0x100 [] drain_all_stock+0xe1/0x1c0 [] mem_cgroup_reclaim+0x90/0xe0 [] __mem_cgroup_try_charge+0x41a/0xc40 [] ? release_pages+0x1b1/0x1f0 [] ? sched_exec+0x40/0xb0 [] mem_cgroup_charge_common+0x37/0x70 [] mem_cgroup_newpage_charge+0x26/0x30 [] handle_pte_fault+0x618/0x840 [] ? unpin_current_cpu+0x16/0x70 [] ? migrate_enable+0xd4/0x200 [] handle_mm_fault+0x145/0x1e0 [] __do_page_fault+0x1a1/0x4c0 [] ? preempt_schedule_irq+0x4b/0x70 [] ? retint_kernel+0x37/0x40 [] do_page_fault+0xe/0x10 [] page_fault+0x22/0x30 So, to prevent schedule_work_on from being called in preempt disabled context, replace the pair of get/put_cpu() to get/put_cpu_light(). Signed-off-by: Yang Shi Signed-off-by: Sebastian Andrzej Siewior commit d46a529b2dabb6412021705822df914bd8325104 Author: Sebastian Andrzej Siewior Date: Tue Aug 18 10:30:00 2020 +0200 mm: memcontrol: Provide a local_lock for per-CPU memcg_stock The interrupts are disabled to ensure CPU-local access to the per-CPU variable `memcg_stock'. As the code inside the interrupt disabled section acquires regular spinlocks, which are converted to 'sleeping' spinlocks on a PREEMPT_RT kernel, this conflicts with the RT semantics. Convert it to a local_lock which allows RT kernels to substitute them with a real per CPU lock. On non RT kernels this maps to local_irq_save() as before, but provides also lockdep coverage of the critical region. No functional change. Signed-off-by: Sebastian Andrzej Siewior commit 419ff92393c9a0c7c86a3fd042c8ed36b98e28a6 Author: Sebastian Andrzej Siewior Date: Wed Apr 15 19:00:47 2015 +0200 slub: Disable SLUB_CPU_PARTIAL |BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:915 |in_atomic(): 1, irqs_disabled(): 0, pid: 87, name: rcuop/7 |1 lock held by rcuop/7/87: | #0: (rcu_callback){......}, at: [] rcu_nocb_kthread+0x1ca/0x5d0 |Preemption disabled at:[] put_cpu_partial+0x29/0x220 | |CPU: 0 PID: 87 Comm: rcuop/7 Tainted: G W 4.0.0-rt0+ #477 |Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 | 000000000007a9fc ffff88013987baf8 ffffffff817441c7 0000000000000007 | 0000000000000000 ffff88013987bb18 ffffffff810eee51 0000000000000000 | ffff88013fc10200 ffff88013987bb48 ffffffff8174a1c4 000000000007a9fc |Call Trace: | [] dump_stack+0x4f/0x90 | [] ___might_sleep+0x121/0x1b0 | [] rt_spin_lock+0x24/0x60 | [] __free_pages_ok+0xaa/0x540 | [] __free_pages+0x1d/0x30 | [] __free_slab+0xc5/0x1e0 | [] free_delayed+0x56/0x70 | [] put_cpu_partial+0x14d/0x220 | [] __slab_free+0x158/0x2c0 | [] kmem_cache_free+0x221/0x2d0 | [] file_free_rcu+0x2c/0x40 | [] rcu_nocb_kthread+0x243/0x5d0 | [] kthread+0xfc/0x120 | [] ret_from_fork+0x58/0x90 Signed-off-by: Sebastian Andrzej Siewior commit 1062410ed3b1a1c03b266e6886b0616a4d703e3d Author: Thomas Gleixner Date: Wed Jan 9 12:08:15 2013 +0100 slub: Enable irqs for __GFP_WAIT SYSTEM_RUNNING might be too late for enabling interrupts. Allocations with GFP_WAIT can happen before that. So use this as an indicator. [bigeasy: Add warning on RT for allocations in atomic context. Don't enable interrupts on allocations during SYSTEM_SUSPEND. This is done during suspend by ACPI, noticed by Liwei Song ] Signed-off-by: Thomas Gleixner commit 4a7cb90a82456508ef37e1bd8027c1fca597de44 Author: Sebastian Andrzej Siewior Date: Thu Jul 16 18:47:50 2020 +0200 mm/slub: Make object_map_lock a raw_spinlock_t The variable object_map is protected by object_map_lock. The lock is always acquired in debug code and within already atomic context Make object_map_lock a raw_spinlock_t. Signed-off-by: Sebastian Andrzej Siewior commit b9640bf0f2b86c95e98e011d0fc51ec946884cf2 Author: Ingo Molnar Date: Fri Jul 3 08:29:37 2009 -0500 mm: page_alloc: rt-friendly per-cpu pages rt-friendly per-cpu pages: convert the irqs-off per-cpu locking method into a preemptible, explicit-per-cpu-locks method. Contains fixes from: Peter Zijlstra Thomas Gleixner Signed-off-by: Ingo Molnar Signed-off-by: Thomas Gleixner commit d6975d9470cc15c6991dbbf853e7009b59151364 Author: Sebastian Andrzej Siewior Date: Thu Jul 2 14:27:23 2020 +0200 mm/page_alloc: Use migrate_disable() in drain_local_pages_wq() drain_local_pages_wq() disables preemption to avoid CPU migration during CPU hotplug. Using migrate_disable() makes the function preemptible on PREEMPT_RT but still avoids CPU migrations during CPU-hotplug. On !PREEMPT_RT it behaves like preempt_disable(). Use migrate_disable() in drain_local_pages_wq(). Signed-off-by: Sebastian Andrzej Siewior commit df7a8bcc628ebd04809bbb0454d6c6337a08034c Author: Thomas Gleixner Date: Thu Jun 21 17:29:19 2018 +0200 mm/SLUB: delay giving back empty slubs to IRQ enabled regions __free_slab() is invoked with disabled interrupts which increases the irq-off time while __free_pages() is doing the work. Allow __free_slab() to be invoked with enabled interrupts and move everything from interrupts-off invocations to a temporary per-CPU list so it can be processed later. Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit c8bfae23cea74cd255cf847bb436590ddf4cef2b Author: Thomas Gleixner Date: Mon May 28 15:24:22 2018 +0200 mm/SLxB: change list_lock to raw_spinlock_t The list_lock is used with used with IRQs off on RT. Make it a raw_spinlock_t otherwise the interrupts won't be disabled on -RT. The locking rules remain the same on !RT. This patch changes it for SLAB and SLUB since both share the same header file for struct kmem_cache_node defintion. Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit 2d2d3b076058d305a3c359d37c5bfd6fb44f47d4 Author: Peter Zijlstra Date: Mon May 28 15:24:21 2018 +0200 Split IRQ-off and zone->lock while freeing pages from PCP list #2 Split the IRQ-off section while accessing the PCP list from zone->lock while freeing pages. Introcude isolate_pcp_pages() which separates the pages from the PCP list onto a temporary list and then free the temporary list via free_pcppages_bulk(). Signed-off-by: Peter Zijlstra Signed-off-by: Sebastian Andrzej Siewior commit f857915c4f7a2e952a39d7edc13f592bc3716608 Author: Peter Zijlstra Date: Mon May 28 15:24:20 2018 +0200 Split IRQ-off and zone->lock while freeing pages from PCP list #1 Split the IRQ-off section while accessing the PCP list from zone->lock while freeing pages. Introcude isolate_pcp_pages() which separates the pages from the PCP list onto a temporary list and then free the temporary list via free_pcppages_bulk(). Signed-off-by: Peter Zijlstra Signed-off-by: Sebastian Andrzej Siewior commit 9609300fe906266f7940531b69d6f5329a27e3b1 Author: Oleg Nesterov Date: Tue Jul 14 14:26:34 2015 +0200 signal/x86: Delay calling signals in atomic On x86_64 we must disable preemption before we enable interrupts for stack faults, int3 and debugging, because the current task is using a per CPU debug stack defined by the IST. If we schedule out, another task can come in and use the same stack and cause the stack to be corrupted and crash the kernel on return. When CONFIG_PREEMPT_RT is enabled, spin_locks become mutexes, and one of these is the spin lock used in signal handling. Some of the debug code (int3) causes do_trap() to send a signal. This function calls a spin lock that has been converted to a mutex and has the possibility to sleep. If this happens, the above issues with the corrupted stack is possible. Instead of calling the signal right away, for PREEMPT_RT and x86_64, the signal information is stored on the stacks task_struct and TIF_NOTIFY_RESUME is set. Then on exit of the trap, the signal resume code will send the signal when preemption is enabled. [ rostedt: Switched from #ifdef CONFIG_PREEMPT_RT to ARCH_RT_DELAYS_SIGNAL_SEND and added comments to the code. ] Signed-off-by: Oleg Nesterov Signed-off-by: Steven Rostedt Signed-off-by: Thomas Gleixner [bigeasy: also needed on 32bit as per Yang Shi ] Signed-off-by: Sebastian Andrzej Siewior commit 5120e45e97216393cf6c4d886f17d306ab5cdba3 Author: Thomas Gleixner Date: Mon Jun 20 09:03:47 2011 +0200 rt: Add local irq locks Introduce locallock. For !RT this maps to preempt_disable()/ local_irq_disable() so there is not much that changes. For RT this will map to a spinlock. This makes preemption possible and locked "ressource" gets the lockdep anotation it wouldn't have otherwise. The locks are recursive for owner == current. Also, all locks user migrate_disable() which ensures that the task is not migrated to another CPU while the lock is held and the owner is preempted. Signed-off-by: Thomas Gleixner commit 320c233d91ab91da863d258ab5d74faa72dc2fc8 Author: Sebastian Andrzej Siewior Date: Thu Jul 26 15:06:10 2018 +0200 efi: Allow efi=runtime In case the command line option "efi=noruntime" is default at built-time, the user could overwrite its state by `efi=runtime' and allow it again. Acked-by: Ard Biesheuvel Signed-off-by: Sebastian Andrzej Siewior commit c1a861a238f6649992c33a1442341b1cc0e8bd3c Author: Sebastian Andrzej Siewior Date: Thu Jul 26 15:03:16 2018 +0200 efi: Disable runtime services on RT Based on meassurements the EFI functions get_variable / get_next_variable take up to 2us which looks okay. The functions get_time, set_time take around 10ms. Those 10ms are too much. Even one ms would be too much. Ard mentioned that SetVariable might even trigger larger latencies if the firware will erase flash blocks on NOR. The time-functions are used by efi-rtc and can be triggered during runtimed (either via explicit read/write or ntp sync). The variable write could be used by pstore. These functions can be disabled without much of a loss. The poweroff / reboot hooks may be provided by PSCI. Disable EFI's runtime wrappers. This was observed on "EFI v2.60 by SoftIron Overdrive 1000". Acked-by: Ard Biesheuvel Signed-off-by: Sebastian Andrzej Siewior commit bc754517db7f2d002cfd59b5659a91670154ffe7 Author: Sebastian Andrzej Siewior Date: Sat May 27 19:02:06 2017 +0200 net/core: disable NET_RX_BUSY_POLL on RT napi_busy_loop() disables preemption and performs a NAPI poll. We can't acquire sleeping locks with disabled preemption so we would have to work around this and add explicit locking for synchronisation against ksoftirqd. Without explicit synchronisation a low priority process would "own" the NAPI state (by setting NAPIF_STATE_SCHED) and could be scheduled out (no preempt_disable() and BH is preemptible on RT). In case a network packages arrives then the interrupt handler would set NAPIF_STATE_MISSED and the system would wait until the task owning the NAPI would be scheduled in again. Should a task with RT priority busy poll then it would consume the CPU instead allowing tasks with lower priority to run. The NET_RX_BUSY_POLL is disabled by default (the system wide sysctls for poll/read are set to zero) so disable NET_RX_BUSY_POLL on RT to avoid wrong locking context on RT. Should this feature be considered useful on RT systems then it could be enabled again with proper locking and synchronisation. Signed-off-by: Sebastian Andrzej Siewior commit e899f864a0441f7ef1a4af223fb86c4341f38f6c Author: Thomas Gleixner Date: Mon Jul 18 17:03:52 2011 +0200 sched: Disable CONFIG_RT_GROUP_SCHED on RT Carsten reported problems when running: taskset 01 chrt -f 1 sleep 1 from within rc.local on a F15 machine. The task stays running and never gets on the run queue because some of the run queues have rt_throttled=1 which does not go away. Works nice from a ssh login shell. Disabling CONFIG_RT_GROUP_SCHED solves that as well. Signed-off-by: Thomas Gleixner commit c2d824f3f5b84aacc64741b3c79d6a10771bc5d1 Author: Ingo Molnar Date: Fri Jul 3 08:44:03 2009 -0500 mm: Allow only SLUB on RT Memory allocation disables interrupts as part of the allocation and freeing process. For -RT it is important that this section remain short and don't depend on the size of the request or an internal state of the memory allocator. At the beginning the SLAB memory allocator was adopted for RT's needs and it required substantial changes. Later, with the addition of the SLUB memory allocator we adopted this one as well and the changes were smaller. More important, due to the design of the SLUB allocator it performs better and its worst case latency was smaller. In the end only SLUB remained supported. Disable SLAB and SLOB on -RT. Only SLUB is adopted to -RT needs. Signed-off-by: Ingo Molnar Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit 7645a26c5d701a7c5ecc91c9b845246ae3eb2d2c Author: Thomas Gleixner Date: Sun Jul 24 12:11:43 2011 +0200 kconfig: Disable config options which are not RT compatible Disable stuff which is known to have issues on RT Signed-off-by: Thomas Gleixner commit 6194d1ae51d699e4031a6423e81dc48063eb4760 Author: Sebastian Andrzej Siewior Date: Tue Sep 8 16:57:11 2020 +0200 net: Properly annotate the try-lock for the seqlock In patch ("net/Qdisc: use a seqlock instead seqcount") the seqcount has been replaced with a seqlock to allow to reader to boost the preempted writer. The try_write_seqlock() acquired the lock with a try-lock but the seqcount annotation was "lock". Opencode write_seqcount_t_begin() and use the try-lock annotation for lockdep. Reported-by: Mike Galbraith Cc: stable-rt@vger.kernel.org Signed-off-by: Sebastian Andrzej Siewior commit 2017a35927ac22a12517d1da90de70922ca5e85c Author: Sebastian Andrzej Siewior Date: Wed Sep 14 17:36:35 2016 +0200 net/Qdisc: use a seqlock instead seqcount The seqcount disables preemption on -RT while it is held which can't remove. Also we don't want the reader to spin for ages if the writer is scheduled out. The seqlock on the other hand will serialize / sleep on the lock while writer is active. Signed-off-by: Sebastian Andrzej Siewior commit fd688c5e5073afd33ff8ef41833960db341e613d Author: Sebastian Andrzej Siewior Date: Fri Oct 20 11:29:53 2017 +0200 fs/dcache: disable preemption on i_dir_seq's write side i_dir_seq is an opencoded seqcounter. Based on the code it looks like we could have two writers in parallel despite the fact that the d_lock is held. The problem is that during the write process on RT the preemption is still enabled and if this process is interrupted by a reader with RT priority then we lock up. To avoid that lock up I am disabling the preemption during the update. The rename of i_dir_seq is here to ensure to catch new write sides in future. Cc: stable-rt@vger.kernel.org Reported-by: Oleg.Karfich@wago.com Signed-off-by: Sebastian Andrzej Siewior commit 9e45aa1294ab2c53c00afcabf2f2dcf6455f3463 Author: Sebastian Andrzej Siewior Date: Wed Sep 14 14:35:49 2016 +0200 fs/dcache: use swait_queue instead of waitqueue __d_lookup_done() invokes wake_up_all() while holding a hlist_bl_lock() which disables preemption. As a workaround convert it to swait. Signed-off-by: Sebastian Andrzej Siewior commit 47c248fab0e760bf53839b67796d403805f1d73e Author: Sebastian Andrzej Siewior Date: Mon Aug 17 12:28:10 2020 +0200 u64_stats: Disable preemption on 32bit-UP/SMP with RT during updates On RT the seqcount_t is required even on UP because the softirq can be preempted. The IRQ handler is threaded so it is also preemptible. Disable preemption on 32bit-RT during value updates. There is no need to disable interrupts on RT because the handler is run threaded. Therefore disabling preemption is enough to guarantee that the update is not interruped. Signed-off-by: Sebastian Andrzej Siewior commit be8c22fe3866908b8d9085c1994526f0e9513aee Author: Ahmed S. Darwish Date: Wed Jun 10 12:53:22 2020 +0200 xfrm: Use sequence counter with associated spinlock A sequence counter write side critical section must be protected by some form of locking to serialize writers. A plain seqcount_t does not contain the information of which lock must be held when entering a write side critical section. Use the new seqcount_spinlock_t data type, which allows to associate a spinlock with the sequence counter. This enables lockdep to verify that the spinlock used for writer serialization is held when the write side critical section is entered. If lockdep is disabled this lock association is compiled out and has neither storage size nor runtime overhead. Upstream-status: The xfrm locking used for seqcoun writer serialization appears to be broken. If that's the case, a proper fix will need to be submitted upstream. (e.g. make the seqcount per network namespace?) Signed-off-by: Ahmed S. Darwish Signed-off-by: Sebastian Andrzej Siewior commit 9fddc665637dd0699a3ed381c0a676d25f59bedb Author: Sebastian Andrzej Siewior Date: Wed Oct 28 18:15:32 2020 +0100 mm/memcontrol: Disable preemption in __mod_memcg_lruvec_state() The callers expect disabled preemption/interrupts while invoking __mod_memcg_lruvec_state(). This works mainline because a lock of somekind is acquired. Use preempt_disable_rt() where per-CPU variables are accessed and a stable pointer is expected. This is also done in __mod_zone_page_state() for the same reason. Cc: stable-rt@vger.kernel.org Signed-off-by: Sebastian Andrzej Siewior commit d7d2f661f60914679c6b16e9a34754a7508a65bf Author: Ingo Molnar Date: Fri Jul 3 08:30:13 2009 -0500 mm/vmstat: Protect per cpu variables with preempt disable on RT Disable preemption on -RT for the vmstat code. On vanila the code runs in IRQ-off regions while on -RT it is not. "preempt_disable" ensures that the same ressources is not updated in parallel due to preemption. Signed-off-by: Ingo Molnar Signed-off-by: Thomas Gleixner commit b20dee0efe4cef97d4ad770aabf725cc7bb731b6 Author: Thomas Gleixner Date: Fri Jul 24 12:38:56 2009 +0200 preempt: Provide preempt_*_(no)rt variants RT needs a few preempt_disable/enable points which are not necessary otherwise. Implement variants to avoid #ifdeffery. Signed-off-by: Thomas Gleixner commit 4f1e8fd08bc05c3d6e9b76b1ec1144c7c7d12474 Author: Thomas Gleixner Date: Wed Sep 21 19:57:12 2011 +0200 signal: Revert ptrace preempt magic Upstream commit '53da1d9456fe7f8 fix ptrace slowness' is nothing more than a bandaid around the ptrace design trainwreck. It's not a correctness issue, it's merily a cosmetic bandaid. Signed-off-by: Thomas Gleixner commit 73c60121b24bbbf23fd35e54b959559c9aa18742 Author: Sebastian Andrzej Siewior Date: Tue Oct 6 13:07:17 2020 +0200 locking/rtmutex: Use custom scheduling function for spin-schedule() PREEMPT_RT builds the rwsem, mutex, spinlock and rwlock typed locks on top of a rtmutex lock. While blocked task->pi_blocked_on is set (tsk_is_pi_blocked()) and task needs to schedule away while waiting. The schedule process must distinguish between blocking on a regular sleeping lock (rwsem and mutex) and a RT-only sleeping lock (spinlock and rwlock): - rwsem and mutex must flush block requests (blk_schedule_flush_plug()) even if blocked on a lock. This can not deadlock because this also happens for non-RT. There should be a warning if the scheduling point is within a RCU read section. - spinlock and rwlock must not flush block requests. This will deadlock if the callback attempts to acquire a lock which is already acquired. Similarly to being preempted, there should be no warning if the scheduling point is within a RCU read section. Add preempt_schedule_lock() which is invoked if scheduling is required while blocking on a PREEMPT_RT-only sleeping lock. Remove tsk_is_pi_blocked() from the scheduler path which is no longer needed with the additional scheduler entry point. Signed-off-by: Sebastian Andrzej Siewior commit a7dfd744668b66d9103b73c1de1aae99b2208529 Author: Sebastian Andrzej Siewior Date: Thu Oct 12 17:34:38 2017 +0200 locking/rtmutex: add ww_mutex addon for mutex-rt Signed-off-by: Sebastian Andrzej Siewior commit 2d015b4f63c6655dc7bd2811b015a8fabe5832e3 Author: Thomas Gleixner Date: Thu Oct 12 17:31:14 2017 +0200 locking/rtmutex: wire up RT's locking Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit d4f0faa08ff743f4cdfd1a626a1e7125361e16a9 Author: Thomas Gleixner Date: Thu Oct 12 17:18:06 2017 +0200 locking/rtmutex: add rwlock implementation based on rtmutex The implementation is bias-based, similar to the rwsem implementation. Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit f97150e6ac1e20b1e5b4e31ea60e71cfcc2fbde2 Author: Thomas Gleixner Date: Thu Oct 12 17:28:34 2017 +0200 locking/rtmutex: add rwsem implementation based on rtmutex The RT specific R/W semaphore implementation restricts the number of readers to one because a writer cannot block on multiple readers and inherit its priority or budget. The single reader restricting is painful in various ways: - Performance bottleneck for multi-threaded applications in the page fault path (mmap sem) - Progress blocker for drivers which are carefully crafted to avoid the potential reader/writer deadlock in mainline. The analysis of the writer code paths shows, that properly written RT tasks should not take them. Syscalls like mmap(), file access which take mmap sem write locked have unbound latencies which are completely unrelated to mmap sem. Other R/W sem users like graphics drivers are not suitable for RT tasks either. So there is little risk to hurt RT tasks when the RT rwsem implementation is changed in the following way: - Allow concurrent readers - Make writers block until the last reader left the critical section. This blocking is not subject to priority/budget inheritance. - Readers blocked on a writer inherit their priority/budget in the normal way. There is a drawback with this scheme. R/W semaphores become writer unfair though the applications which have triggered writer starvation (mostly on mmap_sem) in the past are not really the typical workloads running on a RT system. So while it's unlikely to hit writer starvation, it's possible. If there are unexpected workloads on RT systems triggering it, we need to rethink the approach. Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit 295920971afb7b0d8a4c3091bbbcd069b5b49e81 Author: Thomas Gleixner Date: Thu Oct 12 17:17:03 2017 +0200 locking/rtmutex: add mutex implementation based on rtmutex Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit 71dea561b641b30f47a506cfa72af316556d860c Author: Sebastian Andrzej Siewior Date: Wed Dec 2 11:34:07 2015 +0100 locking/rtmutex: Allow rt_mutex_trylock() on PREEMPT_RT Non PREEMPT_RT kernel can deadlock on rt_mutex_trylock() in softirq context. On PREEMPT_RT the softirq context is handled in thread context. This avoids the deadlock in the slow path and PI-boosting will be done on the correct thread. Signed-off-by: Sebastian Andrzej Siewior commit 0c1c01e407d3871198aab32f10aa5c9334b1227f Author: Thomas Gleixner Date: Thu Oct 12 17:11:19 2017 +0200 locking/rtmutex: add sleeping lock implementation Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit 90e5599c5c846034ab486a674a45ecdf8b52e786 Author: Thomas Gleixner Date: Sat Jun 25 09:21:04 2011 +0200 sched: Add saved_state for tasks blocked on sleeping locks Spinlocks are state preserving in !RT. RT changes the state when a task gets blocked on a lock. So we need to remember the state before the lock contention. If a regular wakeup (not a RTmutex related wakeup) happens, the saved_state is updated to running. When the lock sleep is done, the saved state is restored. Signed-off-by: Thomas Gleixner commit 1f4c72b6851c98bd687a3fd7549ef3a02667e312 Author: Thomas Gleixner Date: Thu Oct 12 16:36:39 2017 +0200 locking/rtmutex: export lockdep-less version of rt_mutex's lock, trylock and unlock Required for lock implementation ontop of rtmutex. Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit c5a2c9070cb3b00c299d7fe4f2ef392afdfb9ccd Author: Thomas Gleixner Date: Thu Oct 12 16:14:22 2017 +0200 locking/rtmutex: Provide rt_mutex_slowlock_locked() This is the inner-part of rt_mutex_slowlock(), required for rwsem-rt. Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit 73b59137cff5ade8f00a4f4b089b5fe5e37b9bb0 Author: Sebastian Andrzej Siewior Date: Fri Aug 14 17:08:41 2020 +0200 locking: split out the rbtree definition rtmutex.h needs the definition for rb_root_cached. By including kernel.h we will get to spinlock.h which requires rtmutex.h again. Split out the required struct definition and move it into its own header file which can be included by rtmutex.h Signed-off-by: Sebastian Andrzej Siewior commit 3183769632ab2cb060c3b8cd977d8f14dc9fb889 Author: Sebastian Andrzej Siewior Date: Fri Aug 14 16:55:25 2020 +0200 lockdep: Reduce header files in debug_locks.h The inclusion of printk.h leads to circular dependency if spinlock_t is based on rt_mutex. Include only atomic.h (xchg()) and cache.h (__read_mostly). Signed-off-by: Sebastian Andrzej Siewior commit cebee2979fb157cc46f921abc4b15640bd9fa4b8 Author: Thomas Gleixner Date: Wed Jun 29 20:06:39 2011 +0200 locking/rtmutex: Avoid include hell Include only the required raw types. This avoids pulling in the complete spinlock header which in turn requires rtmutex.h at some point. Signed-off-by: Thomas Gleixner commit 066a36b17cb9e688eaac7154844e59b0d0c27a06 Author: Thomas Gleixner Date: Wed Jun 29 19:34:01 2011 +0200 locking/spinlock: Split the lock types header Split raw_spinlock into its own file and the remaining spinlock_t into its own non-RT header. The non-RT header will be replaced later by sleeping spinlocks. Signed-off-by: Thomas Gleixner commit 14bdb0063f08089368324f060e42f58c965d8883 Author: Thomas Gleixner Date: Sat Apr 1 12:50:59 2017 +0200 locking/rtmutex: Make lock_killable work Locking an rt mutex killable does not work because signal handling is restricted to TASK_INTERRUPTIBLE. Use signal_pending_state() unconditionally. Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior commit fd9365803c12c209ec6ef03a1d31f3fd1d028aab Author: Steven Rostedt Date: Tue Jul 14 14:26:34 2015 +0200 futex: Fix bug on when a requeued RT task times out Requeue with timeout causes a bug with PREEMPT_RT. The bug comes from a timed out condition. TASK 1 TASK 2 ------ ------ futex_wait_requeue_pi() futex_wait_queue_me() double_lock_hb(); raw_spin_lock(pi_lock); if (current->pi_blocked_on) { } else { current->pi_blocked_on = PI_WAKE_INPROGRESS; run_spin_unlock(pi_lock); spin_lock(hb->lock); <-- blocked! plist_for_each_entry_safe(this) { rt_mutex_start_proxy_lock(); task_blocks_on_rt_mutex(); BUG_ON(task->pi_blocked_on)!!!! The BUG_ON() actually has a check for PI_WAKE_INPROGRESS, but the problem is that, after TASK 1 sets PI_WAKE_INPROGRESS, it then tries to grab the hb->lock, which it fails to do so. As the hb->lock is a mutex, it will block and set the "pi_blocked_on" to the hb->lock. When TASK 2 goes to requeue it, the check for PI_WAKE_INPROGESS fails because the task1's pi_blocked_on is no longer set to that, but instead, set to the hb->lock. The fix: When calling rt_mutex_start_proxy_lock() a check is made to see if the proxy tasks pi_blocked_on is set. If so, exit out early. Otherwise set it to a new flag PI_REQUEUE_INPROGRESS, which notifies the proxy task that it is being requeued, and will handle things appropriately. Signed-off-by: Steven Rostedt Signed-off-by: Thomas Gleixner commit 25f427f1414dc6d461e35aae2cbf6a300298dcca Author: Thomas Gleixner Date: Fri Jun 10 11:04:15 2011 +0200 locking/rtmutex: Handle the various new futex race conditions RT opens a few new interesting race conditions in the rtmutex/futex combo due to futex hash bucket lock being a 'sleeping' spinlock and therefor not disabling preemption. Signed-off-by: Thomas Gleixner commit c7f0a6073b53ff73802d714708462278a8bb1f86 Author: Sebastian Andrzej Siewior Date: Wed Oct 7 12:11:33 2020 +0200 locking/rtmutex: Remove rt_mutex_timed_lock() rt_mutex_timed_lock() has no callers since commit c051b21f71d1f ("rtmutex: Confine deadlock logic to futex") Remove rt_mutex_timed_lock(). Signed-off-by: Sebastian Andrzej Siewior commit c918e396414e23541efc9b5c06868754c17c8eb7 Author: Sebastian Andrzej Siewior Date: Tue Sep 29 16:32:49 2020 +0200 locking/rtmutex: Move rt_mutex_init() outside of CONFIG_DEBUG_RT_MUTEXES rt_mutex_init() only initializes lockdep if CONFIG_DEBUG_RT_MUTEXES is enabled. The static initializer (DEFINE_RT_MUTEX) does not have such a restriction. Move rt_mutex_init() outside of CONFIG_DEBUG_RT_MUTEXES. Move the remaining functions in this CONFIG_DEBUG_RT_MUTEXES block to the upper block. Signed-off-by: Sebastian Andrzej Siewior commit f75de460e364c3474701f45f005ca931675ca248 Author: Sebastian Andrzej Siewior Date: Tue Sep 29 16:05:11 2020 +0200 locking/rtmutex: Remove output from deadlock detector. In commit f5694788ad8da ("rt_mutex: Add lockdep annotations") rtmutex gained lockdep annotation for rt_mutex_lock() and and related functions. lockdep will see the locking order and may complain about a deadlock before rtmutex' own mechanism gets a chance to detect it. The rtmutex deadlock detector will only complain locks with the RT_MUTEX_MIN_CHAINWALK and a waiter must be pending. That means it works only for in-kernel locks because the futex interface always uses RT_MUTEX_FULL_CHAINWALK. The requirement for an active waiter limits the detector to actual deadlocks and makes it possible to report potential deadlocks like lockdep does. It looks like lockdep is better suited for reporting deadlocks. Remove rtmutex' debug print on deadlock detection. Signed-off-by: Sebastian Andrzej Siewior commit 69824ec3521068bd7924033aae066fccd815fba9 Author: Sebastian Andrzej Siewior Date: Tue Sep 29 15:21:17 2020 +0200 locking/rtmutex: Remove cruft Most of this is around since the very beginning. I'm not sure if this was used while the rtmutex-deadlock-tester was around but today it seems to only waste memory: - save_state: No users - name: Assigned and printed if a dead lock was detected. I'm keeping it but want to point out that lockdep has the same information. - file + line: Printed if ::name was NULL. This is only used for in-kernel locks so it ::name shouldn't be NULL and then ::file and ::line isn't used. - magic: Assigned to NULL by rt_mutex_destroy(). Remove members of rt_mutex which are not used. Signed-off-by: Sebastian Andrzej Siewior commit 2fed448c1463c651254f83f7a951790b358b792c Author: Sebastian Andrzej Siewior Date: Fri Dec 4 18:00:31 2020 +0100 irqtime: Use irq_count() instead of preempt_count() preempt_count() does not contain the softirq bits on a PREEMPT_RT kernel. irq_count() does. Signed-off-by: Sebastian Andrzej Siewior commit 020f575b95e064c4987d9d592863aca7ff92eca8 Author: Thomas Gleixner Date: Fri Nov 13 15:02:26 2020 +0100 tasklets: Prevent kill/unlock_wait deadlock on RT tasklet_kill() and tasklet_unlock_wait() spin and wait for the TASKLET_STATE_SCHED resp. TASKLET_STATE_RUN bit in the tasklet state to be cleared. This works on !RT nicely because the corresponding execution can only happen on a different CPU. On RT softirq processing is preemptible, therefore a task preempting the softirq processing thread can spin forever. Prevent this by invoking local_bh_disable()/enable() inside the loop. In case that the softirq processing thread was preempted by the current task, current will block on the local lock which yields the CPU to the preempted softirq processing thread. If the tasklet is processed on a different CPU then the local_bh_disable()/enable() pair is just a waste of processor cycles. Signed-off-by: Thomas Gleixner commit 322a5c2aabe4f4d03959c7de945cc3d66d9c483a Author: Thomas Gleixner Date: Fri Nov 13 15:02:25 2020 +0100 tasklets: Use static inlines for stub implementations Inlines exist for a reason. Signed-off-by: Thomas Gleixner commit f3be5de734dbcd8e4aa395b77db3d1ddc86e940e Author: Thomas Gleixner Date: Fri Nov 13 15:02:24 2020 +0100 softirq: Replace barrier() with cpu_relax() in tasklet_unlock_wait() A barrier() in a tight loop which waits for something to happen on a remote CPU is a pointless exercise. Replace it with cpu_relax() which allows HT siblings to make progress. Signed-off-by: Thomas Gleixner commit b28a33e0c90e80259f02d300e44b251b4a7ba514 Author: Thomas Gleixner Date: Fri Nov 13 15:02:23 2020 +0100 rcu: Prevent false positive softirq warning on RT Soft interrupt disabled sections can legitimately be preempted or schedule out when blocking on a lock on RT enabled kernels so the RCU preempt check warning has to be disabled for RT kernels. Signed-off-by: Thomas Gleixner commit a0e89fc79984625d4bafffea9ad0edad1866625a Author: Thomas Gleixner Date: Fri Nov 13 15:02:22 2020 +0100 tick/sched: Prevent false positive softirq pending warnings on RT On RT a task which has soft interrupts disabled can block on a lock and schedule out to idle while soft interrupts are pending. This triggers the warning in the NOHZ idle code which complains about going idle with pending soft interrupts. But as the task is blocked soft interrupt processing is temporarily blocked as well which means that such a warning is a false positive. To prevent that check the per CPU state which indicates that a scheduled out task has soft interrupts disabled. Signed-off-by: Thomas Gleixner commit 4c96501488178c7a31e442d5b9c4738e538e01a6 Author: Thomas Gleixner Date: Fri Nov 13 15:02:21 2020 +0100 softirq: Make softirq control and processing RT aware Provide a local lock based serialization for soft interrupts on RT which allows the local_bh_disabled() sections and servicing soft interrupts to be preemptible. Provide the necessary inline helpers which allow to reuse the bulk of the softirq processing code. Signed-off-by: Thomas Gleixner commit af68064a6b93924112c6be13191716ca46ee12c7 Author: Thomas Gleixner Date: Fri Nov 13 15:02:20 2020 +0100 softirq: Move various protections into inline helpers To allow reuse of the bulk of softirq processing code for RT and to avoid #ifdeffery all over the place, split protections for various code sections out into inline helpers so the RT variant can just replace them in one go. Signed-off-by: Thomas Gleixner commit 29585b79c2d4c4294c602d40566ed310abea6f5a Author: Thomas Gleixner Date: Fri Nov 13 15:02:19 2020 +0100 softirq: Add RT specific softirq accounting RT requires the softirq processing and local bottomhalf disabled regions to be preemptible. Using the normal preempt count based serialization is therefore not possible because this implicitely disables preemption. RT kernels use a per CPU local lock to serialize bottomhalfs. As local_bh_disable() can nest the lock can only be acquired on the outermost invocation of local_bh_disable() and released when the nest count becomes zero. Tasks which hold the local lock can be preempted so its required to keep track of the nest count per task. Add a RT only counter to task struct and adjust the relevant macros in preempt.h. Signed-off-by: Thomas Gleixner commit 6095a7c7b05e863b31ea6e3ba0d626dffb9348a8 Author: Sebastian Andrzej Siewior Date: Mon Oct 12 17:33:54 2020 +0200 tcp: Remove superfluous BH-disable around listening_hash Commit 9652dc2eb9e40 ("tcp: relax listening_hash operations") removed the need to disable bottom half while acquiring listening_hash.lock. There are still two callers left which disable bottom half before the lock is acquired. Drop local_bh_disable() around __inet_hash() which acquires listening_hash->lock, invoke inet_ehash_nolisten() with disabled BH. inet_unhash() conditionally acquires listening_hash->lock. Reported-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior Link: https://lore.kernel.org/linux-rt-users/12d6f9879a97cd56c09fb53dee343cbb14f7f1f7.camel@gmx.de/ Link: https://lkml.kernel.org/r/X9CheYjuXWc75Spa@hirez.programming.kicks-ass.net commit a851f74e7b7c91e37a9f9bfa67ff504f8c7c56f0 Author: Thomas Gleixner Date: Tue Sep 8 07:32:20 2020 +0200 net: Move lockdep where it belongs Signed-off-by: Thomas Gleixner commit a98966b04667e74d4fde05d869c979e21241665b Author: Sebastian Andrzej Siewior Date: Fri Aug 14 18:53:34 2020 +0200 shmem: Use raw_spinlock_t for ->stat_lock Each CPU has SHMEM_INO_BATCH inodes available in `->ino_batch' which is per-CPU. Access here is serialized by disabling preemption. If the pool is empty, it gets reloaded from `->next_ino'. Access here is serialized by ->stat_lock which is a spinlock_t and can not be acquired with disabled preemption. One way around it would make per-CPU ino_batch struct containing the inode number a local_lock_t. Another sollution is to promote ->stat_lock to a raw_spinlock_t. The critical sections are short. The mpol_put() should be moved outside of the critical section to avoid invoking the destrutor with disabled preemption. Signed-off-by: Sebastian Andrzej Siewior commit 2a8112ffbbd44068750c98234d6126378b82c381 Author: Sebastian Andrzej Siewior Date: Mon Feb 11 10:40:46 2019 +0100 mm: workingset: replace IRQ-off check with a lockdep assert. Commit 68d48e6a2df57 ("mm: workingset: add vmstat counter for shadow nodes") introduced an IRQ-off check to ensure that a lock is held which also disabled interrupts. This does not work the same way on -RT because none of the locks, that are held, disable interrupts. Replace this check with a lockdep assert which ensures that the lock is held. Cc: Peter Zijlstra Signed-off-by: Sebastian Andrzej Siewior commit 9b885ad9e87cfa37ab95fb5306aca8da3e9de2bc Author: Sebastian Andrzej Siewior Date: Tue Jul 3 18:19:48 2018 +0200 cgroup: use irqsave in cgroup_rstat_flush_locked() All callers of cgroup_rstat_flush_locked() acquire cgroup_rstat_lock either with spin_lock_irq() or spin_lock_irqsave(). cgroup_rstat_flush_locked() itself acquires cgroup_rstat_cpu_lock which is a raw_spin_lock. This lock is also acquired in cgroup_rstat_updated() in IRQ context and therefore requires _irqsave() locking suffix in cgroup_rstat_flush_locked(). Since there is no difference between spin_lock_t and raw_spin_lock_t on !RT lockdep does not complain here. On RT lockdep complains because the interrupts were not disabled here and a deadlock is possible. Acquire the raw_spin_lock_t with disabled interrupts. Signed-off-by: Sebastian Andrzej Siewior commit 87618762d616c714ac8ffad0436e25917bbdfd61 Author: John Ogness Date: Mon Nov 30 01:42:10 2020 +0106 printk: add pr_flush() Provide a function to allow waiting for console printers to catch up to the latest logged message. Use pr_flush() to give console printers a chance to finish in critical situations if no atomic console is available. For now pr_flush() is only used in the most common error paths: panic(), print_oops_end_marker(), report_bug(), kmsg_dump(). Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 2cbe1991b798735372a957a10a621373667baa96 Author: John Ogness Date: Mon Nov 30 01:42:09 2020 +0106 printk: add console handover If earlyprintk is used, a boot console will print directly to the console immediately. The boot console will unregister itself as soon as a non-boot console registers. However, the non-boot console does not begin printing until its kthread has started. Since this happens much later, there is a long pause in the console output. If the ringbuffer is small, messages could even be dropped during the pause. Add a new CON_HANDOVER console flag to be used internally by printk in order to track which non-boot console took over from a boot console. If handover consoles have implemented write_atomic(), they are allowed to print directly to the console until their kthread can take over. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 5cdae3829ce3a7a50e562bff377e42fbe382a4e1 Author: John Ogness Date: Mon Nov 30 01:42:08 2020 +0106 printk: remove deferred printing Since printing occurs either atomically or from the printing kthread, there is no need for any deferring or tracking possible recursion paths. Remove all printk context tracking. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 42afcbce7b62cd3e1bb00468ff150281d68dd8ee Author: John Ogness Date: Mon Nov 30 01:42:07 2020 +0106 printk: move console printing to kthreads Create a kthread for each console to perform console printing. Now all console printing is fully asynchronous except for the boot console and when the kernel enters sync mode (and there are atomic consoles available). The console_lock() and console_unlock() functions now only do what their name says... locking and unlocking of the console. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 8d32aa06ff4991a7be6892017be468ed2b749f61 Author: John Ogness Date: Mon Nov 30 01:42:06 2020 +0106 printk: introduce kernel sync mode When the kernel performs an OOPS, enter into "sync mode": - only atomic consoles (write_atomic() callback) will print - printing occurs within vprintk_store() instead of console_unlock() Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 3033f17272c9b7b78eb5035efb361c2a3e3a145a Author: John Ogness Date: Mon Nov 30 01:42:05 2020 +0106 printk: change @console_seq to atomic64_t In preparation for atomic printing, change @console_seq to atomic so that it can be accessed without requiring @console_sem. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 4a38acb647ccff72bdea78511c186fe906051305 Author: John Ogness Date: Mon Nov 30 01:42:04 2020 +0106 printk: combine boot_delay_msec() into printk_delay() boot_delay_msec() is always called immediately before printk_delay() so just combine the two. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit d2b734e62848ec5bd9435a06f61dc17a07ee0459 Author: John Ogness Date: Mon Nov 30 01:42:03 2020 +0106 printk: relocate printk_delay() and vprintk_default() Move printk_delay() and vprintk_default() "as is" further up so that they can be used by new functions in an upcoming commit. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 51b3098d1203a2faf85abccff74644944099e9c5 Author: John Ogness Date: Mon Nov 30 01:42:02 2020 +0106 serial: 8250: implement write_atomic Implement a non-sleeping NMI-safe write_atomic() console function in order to support emergency console printing. Since interrupts need to be disabled during transmit, all usage of the IER register is wrapped with access functions that use the console_atomic_lock() function to synchronize register access while tracking the state of the interrupts. This is necessary because write_atomic() can be called from an NMI context that has preempted write_atomic(). Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit f77703ebd357afea727d084aa5034555c6320a84 Author: John Ogness Date: Mon Nov 30 01:42:01 2020 +0106 console: add write_atomic interface Add a write_atomic() callback to the console. This is an optional function for console drivers. The function must be atomic (including NMI safe) for writing to the console. Console drivers must still implement the write() callback. The write_atomic() callback will only be used in special situations, such as when the kernel panics. Creating an NMI safe write_atomic() that must synchronize with write() requires a careful implementation of the console driver. To aid with the implementation, a set of console_atomic_*() functions are provided: void console_atomic_lock(unsigned int *flags); void console_atomic_unlock(unsigned int flags); These functions synchronize using a processor-reentrant spinlock (called a cpulock). Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 77154700caaf319e037966e524156ad31a40cd4c Author: John Ogness Date: Mon Nov 30 01:41:56 2020 +0106 printk: refactor kmsg_dump_get_buffer() kmsg_dump_get_buffer() requires nearly the same logic as syslog_print_all(), but uses different variable names and does not make use of the ringbuffer loop macros. Modify kmsg_dump_get_buffer() so that the implementation is as similar to syslog_print_all() as possible. At some point it would be nice to have this code factored into a helper function. But until then, the code should at least look similar enough so that it is obvious there is logic duplication implemented. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 54fc9bd615de9e3d990cc27c0502f6fd8425d44c Author: John Ogness Date: Mon Nov 30 01:42:00 2020 +0106 printk: remove safe buffers With @logbuf_lock removed, the high level printk functions for storing messages are lockless. Messages can be stored from any context, so there is no need for the NMI and safe buffers anymore. Remove the NMI and safe buffers. In NMI or safe contexts, store the message immediately but still use irq_work to defer the console printing. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 5dacccbe3ab20311f3bc994ef34e771358c5360b Author: John Ogness Date: Mon Nov 30 01:41:59 2020 +0106 printk: remove logbuf_lock Since the ringbuffer is lockless, there is no need for it to be protected by @logbuf_lock. Remove @logbuf_lock. This means that printk_nmi_direct and printk_safe_flush_on_panic() no longer need to acquire any lock to run. @console_seq, @exclusive_console_stop_seq, @console_dropped are protected by @console_lock. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 4d16e1d97a8a4a4bc14912cf650963e84d110b6b Author: John Ogness Date: Fri Dec 18 11:40:08 2020 +0000 printk: invoke dumper using a copy of the iterator In preparation for removal of @logbuf_lock, invoke a copy of the dumper to iterate over the records. This will allow kmsg_dump() to be called simultaneously on multiple CPUs. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 9c1ccfe232944ef4628d387f4ef8add7e5453fb5 Author: John Ogness Date: Fri Dec 11 00:55:25 2020 +0106 printk: track/limit recursion Limit printk() recursion to 1 level. This is enough to print a stacktrace for the printk call, should a WARN or BUG occur. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit e213428972d035584c7b045ff660593bfa340a80 Author: John Ogness Date: Thu Dec 10 12:48:01 2020 +0106 define CONSOLE_LOG_MAX in printk.h CONSOLE_EXT_LOG_MAX for extended console messages is already defined in printk.h. Define CONSOLE_LOG_MAX there as well so that future changes can make use of the constant for non-extended console messages. Use CONSOLE_LOG_MAX instead of LOG_LINE_MAX + PREFIX_MAX. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 0650844387769a80c4402b07308f5b00bb239747 Author: John Ogness Date: Thu Dec 10 16:58:02 2020 +0106 printk: add syslog_lock The global variables @syslog_seq, @syslog_partial, @syslog_time and write access to @clear_seq are protected by @logbuf_lock. Once @logbuf_lock is removed, these variables with need their own synchronization method. Introduce @syslog_lock for this purpose. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit a360b3aae0e4e62d61c8af8efd626b36805e33cd Author: John Ogness Date: Thu Dec 10 15:33:40 2020 +0106 printk: use atomic64_t for devkmsg_user.seq @user->seq is indirectly protected by @logbuf_lock. Once @logbuf_lock is removed, @user->seq will be no longer safe from an atomicity point of view. In preparation for the removal of @logbuf_lock, change it to atomic64_t to provide this safety. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 5c50994cd088b3387fc9f2d12fa0d8e48fe3e1de Author: John Ogness Date: Mon Nov 30 01:41:58 2020 +0106 printk: use seqcount_latch for clear_seq kmsg_dump_rewind_nolock() locklessly reads @clear_seq. However, this is not done atomically. Since @clear_seq is 64-bit, this cannot be an atomic operation for all platforms. Therefore, use a seqcount_latch to allow readers to always read a consistent value. Signed-off-by: John Ogness Signed-off-by: Sebastian Andrzej Siewior commit 32e31fb58796d84d81ed33694f8d1008046fec9b Author: Sebastian Andrzej Siewior Date: Mon Feb 11 11:33:11 2019 +0100 tpm: remove tpm_dev_wq_lock Added in commit 9e1b74a63f776 ("tpm: add support for nonblocking operation") but never actually used it. Cc: Philip Tricca Cc: Tadeusz Struk Cc: Jarkko Sakkinen Signed-off-by: Sebastian Andrzej Siewior commit e46ddbfcf2b73cb9a2898677268e2b14336a366d Author: Sebastian Andrzej Siewior Date: Tue Jan 12 21:01:38 2021 +0100 tracing: Remove NULL check from current in tracing_generic_entry_update(). I can't imagine when or why `current' would return a NULL pointer. This check was added in commit 72829bc3d63cd ("ftrace: move enums to ftrace.h and make helper function global") but it doesn't give me hint why it was needed. Assume `current' never returns a NULL pointer and remove the check. Signed-off-by: Sebastian Andrzej Siewior commit 34e9ddfcda69d3fdc175ede69c72f1af9fff16c7 Author: Sebastian Andrzej Siewior Date: Tue Jan 12 20:37:54 2021 +0100 tracing: Use in_serving_softirq() to deduct softirq status. PREEMPT_RT does not report "serving softirq" because the tracing core looks at the preemption counter while PREEMPT_RT does not update it while processing softirqs in order to remain preemptible. The information is stored somewhere else. The in_serving_softirq() macro and the SOFTIRQ_OFFSET define are still working but not on the preempt-counter. Use in_serving_softirq() macro which works on PREEMPT_RT. On !PREEMPT_RT the compiler (gcc-10 / clang-11) is smart enough to optimize the in_serving_softirq() related read of the preemption counter away. The only difference I noticed by using in_serving_softirq() on !PREEMPT_RT is that gcc-10 implemented tracing_gen_ctx_flags() as reading FLAG, jmp _tracing_gen_ctx_flags(). Without in_serving_softirq() it inlined _tracing_gen_ctx_flags() into tracing_gen_ctx_flags(). Signed-off-by: Sebastian Andrzej Siewior commit 56cdf87576b0bc5981062b26f5bcb96b15854408 Author: Sebastian Andrzej Siewior Date: Tue Jan 12 15:58:53 2021 +0100 tracing: Merge irqflags + preempt counter. The state of the interrupts (irqflags) and the preemption counter are both passed down to tracing_generic_entry_update(). Only one bit of irqflags is actually required: The on/off state. The complete 32bit of the preemption counter isn't needed. Just whether of the upper bits (softirq, hardirq and NMI) are set and the preemption depth is needed. The irqflags and the preemption counter could be evaluated early and the information stored in an integer `trace_ctx'. tracing_generic_entry_update() would use the upper bits as the TRACE_FLAG_* and the lower 8bit as the disabled-preemption depth (considering that one must be substracted from the counter in one special cases). The actual preemption value is not used except for the tracing record. The `irqflags' variable is mostly used only for the tracing record. An exception here is for instance wakeup_tracer_call() or probe_wakeup_sched_switch() which explicilty disable interrupts and use that `irqflags' to save (and restore) the IRQ state and to record the state. Struct trace_event_buffer has also the `pc' and flags' members which can be replaced with `trace_ctx' since their actual value is not used outside of trace recording. This will reduce tracing_generic_entry_update() to simply assign values to struct trace_entry. The evaluation of the TRACE_FLAG_* bits is moved to _tracing_gen_ctx_flags() which replaces preempt_count() and local_save_flags() invocations. As an example, ftrace_syscall_enter() may invoke: - trace_buffer_lock_reserve() -> … -> tracing_generic_entry_update() - event_trigger_unlock_commit() -> ftrace_trace_stack() -> … -> tracing_generic_entry_update() -> ftrace_trace_userstack() -> … -> tracing_generic_entry_update() In this case the TRACE_FLAG_* bits were evaluated three times. By using the `trace_ctx' they are evaluated once and assigned three times. A build with all tracers enabled on x86-64 with and without the patch: text data bss dec hex filename 21970669 17084168 7639260 46694097 2c87ed1 vmlinux.old 21970293 17084168 7639260 46693721 2c87d59 vmlinux.new text shrank by 379 bytes, data remained constant. Signed-off-by: Sebastian Andrzej Siewior commit 6f414a1e5953b049a894d9c4981dc212611c2743 Author: Sebastian Andrzej Siewior Date: Tue Dec 15 15:16:49 2020 +0100 doc: Use CONFIG_PREEMPTION CONFIG_PREEMPTION is selected by CONFIG_PREEMPT and by CONFIG_PREEMPT_RT. Both PREEMPT and PREEMPT_RT require the same functionality which today depends on CONFIG_PREEMPT. Update the documents and mention CONFIG_PREEMPTION. Spell out CONFIG_PREEMPT_RT (instead PREEMPT_RT) since it is an option now. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Paul E. McKenney Signed-off-by: Sebastian Andrzej Siewior commit 21127334842d5c573839714e9471b832719197c4 Author: Sebastian Andrzej Siewior Date: Tue Dec 15 15:16:48 2020 +0100 doc: Update RCU's requirements page about the PREEMPT_RT wiki. The PREEMPT_RT wiki moved from kernel.org to the Linux Foundation wiki. The kernel.org wiki is read only. This commit therefore updates the URL of the active PREEMPT_RT wiki. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Paul E. McKenney Signed-off-by: Sebastian Andrzej Siewior commit 286d8b7cad2e65729875bb0e6b869a4248668f6c Author: Julia Cartwright Date: Tue Dec 15 15:16:47 2020 +0100 rcu: Enable rcu_normal_after_boot unconditionally for RT Expedited RCU grace periods send IPIs to all non-idle CPUs, and thus can disrupt time-critical code in real-time applications. However, there is a portion of boot-time processing (presumably before any real-time applications have started) where expedited RCU grace periods are the only option. And so it is that experience with the -rt patchset indicates that PREEMPT_RT systems should always set the rcupdate.rcu_normal_after_boot kernel boot parameter. This commit therefore makes the post-boot application environment safe for real-time applications by making PREEMPT_RT systems disable the rcupdate.rcu_normal_after_boot kernel boot parameter and acting as if this parameter had been set. This means that post-boot calls to synchronize_rcu_expedited() will be treated as if they were instead calls to synchronize_rcu(), thus preventing the IPIs, and thus avoiding disrupting real-time applications. Suggested-by: Luiz Capitulino Acked-by: Paul E. McKenney Signed-off-by: Julia Cartwright Signed-off-by: Sebastian Andrzej Siewior [ paulmck: Update kernel-parameters.txt accordingly. ] Signed-off-by: Paul E. McKenney Signed-off-by: Sebastian Andrzej Siewior commit 38239b11a38ba4fa74ea09547994509cfacd0707 Author: Scott Wood Date: Tue Dec 15 15:16:46 2020 +0100 rcu: Unconditionally use rcuc threads on PREEMPT_RT PREEMPT_RT systems have long used the rcutree.use_softirq kernel boot parameter to avoid use of RCU_SOFTIRQ handlers, which can disrupt real-time applications by invoking callbacks during return from interrupts that arrived while executing time-critical code. This kernel boot parameter instead runs RCU core processing in an 'rcuc' kthread, thus allowing the scheduler to do its job of avoiding disrupting time-critical code. This commit therefore disables the rcutree.use_softirq kernel boot parameter on PREEMPT_RT systems, thus forcing such systems to do RCU core processing in 'rcuc' kthreads. This approach has long been in use by users of the -rt patchset, and there have been no complaints. There is therefore no way for the system administrator to override this choice, at least without modifying and rebuilding the kernel. Signed-off-by: Scott Wood [bigeasy: Reword commit message] Signed-off-by: Sebastian Andrzej Siewior [ paulmck: Update kernel-parameters.txt accordingly. ] Signed-off-by: Paul E. McKenney Signed-off-by: Sebastian Andrzej Siewior commit 79a8f542c49a4254827aceeefe4a99a83cb166c9 Author: Sebastian Andrzej Siewior Date: Tue Dec 15 15:16:45 2020 +0100 rcu: Make RCU_BOOST default on CONFIG_PREEMPT_RT On PREEMPT_RT kernels, RCU callbacks are deferred to the `rcuc' kthread. This can stall RCU grace periods due to lengthy preemption not only of RCU readers but also of 'rcuc' kthreads, either of which prevent grace periods from completing, which can in turn result in OOM. Because PREEMPT_RT kernels have more kthreads that can block grace periods, it is more important for such kernels to enable RCU_BOOST. This commit therefore makes RCU_BOOST the default on PREEMPT_RT. RCU_BOOST can still be manually disabled if need be. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Paul E. McKenney Signed-off-by: Sebastian Andrzej Siewior commit 95d3feffb2eb1b41a5d65c27df0827814b213e78 Author: Valentin Schneider Date: Sun Nov 22 20:19:04 2020 +0000 notifier: Make atomic_notifiers use raw_spinlock Booting a recent PREEMPT_RT kernel (v5.10-rc3-rt7-rebase) on my arm64 Juno leads to the idle task blocking on an RT sleeping spinlock down some notifier path: [ 1.809101] BUG: scheduling while atomic: swapper/5/0/0x00000002 [ 1.809116] Modules linked in: [ 1.809123] Preemption disabled at: [ 1.809125] secondary_start_kernel (arch/arm64/kernel/smp.c:227) [ 1.809146] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G W 5.10.0-rc3-rt7 #168 [ 1.809153] Hardware name: ARM Juno development board (r0) (DT) [ 1.809158] Call trace: [ 1.809160] dump_backtrace (arch/arm64/kernel/stacktrace.c:100 (discriminator 1)) [ 1.809170] show_stack (arch/arm64/kernel/stacktrace.c:198) [ 1.809178] dump_stack (lib/dump_stack.c:122) [ 1.809188] __schedule_bug (kernel/sched/core.c:4886) [ 1.809197] __schedule (./arch/arm64/include/asm/preempt.h:18 kernel/sched/core.c:4913 kernel/sched/core.c:5040) [ 1.809204] preempt_schedule_lock (kernel/sched/core.c:5365 (discriminator 1)) [ 1.809210] rt_spin_lock_slowlock_locked (kernel/locking/rtmutex.c:1072) [ 1.809217] rt_spin_lock_slowlock (kernel/locking/rtmutex.c:1110) [ 1.809224] rt_spin_lock (./include/linux/rcupdate.h:647 kernel/locking/rtmutex.c:1139) [ 1.809231] atomic_notifier_call_chain_robust (kernel/notifier.c:71 kernel/notifier.c:118 kernel/notifier.c:186) [ 1.809240] cpu_pm_enter (kernel/cpu_pm.c:39 kernel/cpu_pm.c:93) [ 1.809249] psci_enter_idle_state (drivers/cpuidle/cpuidle-psci.c:52 drivers/cpuidle/cpuidle-psci.c:129) [ 1.809258] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:238) [ 1.809267] cpuidle_enter (drivers/cpuidle/cpuidle.c:353) [ 1.809275] do_idle (kernel/sched/idle.c:132 kernel/sched/idle.c:213 kernel/sched/idle.c:273) [ 1.809282] cpu_startup_entry (kernel/sched/idle.c:368 (discriminator 1)) [ 1.809288] secondary_start_kernel (arch/arm64/kernel/smp.c:273) Two points worth noting: 1) That this is conceptually the same issue as pointed out in: 313c8c16ee62 ("PM / CPU: replace raw_notifier with atomic_notifier") 2) Only the _robust() variant of atomic_notifier callchains suffer from this AFAICT only the cpu_pm_notifier_chain really needs to be changed, but singling it out would mean introducing a new (truly) non-blocking API. At the same time, callers that are fine with any blocking within the call chain should use blocking notifiers, so patching up all atomic_notifier's doesn't seem *too* crazy to me. Fixes: 70d932985757 ("notifier: Fix broken error handling pattern") Signed-off-by: Valentin Schneider Reviewed-by: Daniel Bristot de Oliveira Link: https://lkml.kernel.org/r/20201122201904.30940-1-valentin.schneider@arm.com Signed-off-by: Sebastian Andrzej Siewior commit 86b948f5fd8397bf3a811792be384e8cdf728da0 Author: Thomas Gleixner Date: Mon Nov 9 23:32:39 2020 +0100 genirq: Move prio assignment into the newly created thread With enabled threaded interrupts the nouveau driver reported the following: | Chain exists of: | &mm->mmap_lock#2 --> &device->mutex --> &cpuset_rwsem | | Possible unsafe locking scenario: | | CPU0 CPU1 | ---- ---- | lock(&cpuset_rwsem); | lock(&device->mutex); | lock(&cpuset_rwsem); | lock(&mm->mmap_lock#2); The device->mutex is nvkm_device::mutex. Unblocking the lockchain at `cpuset_rwsem' is probably the easiest thing to do. Move the priority assignment to the start of the newly created thread. Fixes: 710da3c8ea7df ("sched/core: Prevent race condition between cpuset and __sched_setscheduler()") Reported-by: Mike Galbraith Signed-off-by: Thomas Gleixner [bigeasy: Patch description] Signed-off-by: Sebastian Andrzej Siewior Link: https://lkml.kernel.org/r/a23a826af7c108ea5651e73b8fbae5e653f16e86.camel@gmx.de commit 6c0bc3174621eba33f30bdf91f9d0e79228f4141 Author: Sebastian Andrzej Siewior Date: Mon Nov 9 21:30:41 2020 +0100 kthread: Move prio/affinite change into the newly created thread With enabled threaded interrupts the nouveau driver reported the following: | Chain exists of: | &mm->mmap_lock#2 --> &device->mutex --> &cpuset_rwsem | | Possible unsafe locking scenario: | | CPU0 CPU1 | ---- ---- | lock(&cpuset_rwsem); | lock(&device->mutex); | lock(&cpuset_rwsem); | lock(&mm->mmap_lock#2); The device->mutex is nvkm_device::mutex. Unblocking the lockchain at `cpuset_rwsem' is probably the easiest thing to do. Move the priority reset to the start of the newly created thread. Fixes: 710da3c8ea7df ("sched/core: Prevent race condition between cpuset and __sched_setscheduler()") Reported-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior Link: https://lkml.kernel.org/r/a23a826af7c108ea5651e73b8fbae5e653f16e86.camel@gmx.de commit 10920effa84a4ace7b6ac5a164fb416b1c4b2e08 Author: Sebastian Andrzej Siewior Date: Wed Oct 28 11:08:21 2020 +0100 blk-mq: Use llist_head for blk_cpu_done With llist_head it is possible to avoid the locking (the irq-off region) when items are added. This makes it possible to add items on a remote CPU without additional locking. llist_add() returns true if the list was previously empty. This can be used to invoke the SMP function call / raise sofirq only if the first item was added (otherwise it is already pending). This simplifies the code a little and reduces the IRQ-off regions. blk_mq_raise_softirq() needs a preempt-disable section to ensure the request is enqueued on the same CPU as the softirq is raised. Some callers (USB-storage) invoke this path in preemptible context. Signed-off-by: Sebastian Andrzej Siewior commit 13556b8162fa2215835e2059a4d8ceffe5349098 Author: Sebastian Andrzej Siewior Date: Wed Oct 28 11:07:09 2020 +0100 blk-mq: Always complete remote completions requests in softirq Controllers with multiple queues have their IRQ-handelers pinned to a CPU. The core shouldn't need to complete the request on a remote CPU. Remove this case and always raise the softirq to complete the request. Signed-off-by: Sebastian Andrzej Siewior commit acf39aa031b6f5e70aab610c87943b3e9a14a5fb Author: Tian Tao Date: Tue Jan 19 13:54:43 2021 +0800 mm: set the sleep_mapped to true for zbud and z3fold zpool driver add a flag to indicates whether the zpool driver can enter an atomic context after mapping, this patch set it true for z3fold and zbud. Signed-off-by: Tian Tao Reviewed-by: Vitaly Wool Acked-by: Sebastian Andrzej Siewior Reported-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior commit 20ccc38b8098cf4cae148b18ebf69de9c75ff08c Author: Tian Tao Date: Tue Jan 19 13:54:42 2021 +0800 mm/zswap: add the flag can_sleep_mapped add a flag to zpool, named is "can_sleep_mapped", and have it set true for zbud/z3fold, not set this flag for zsmalloc, so its default value is false. Then zswap could go the current path if the flag is true; and if it's false, copy data from src to a temporary buffer, then unmap the handle, take the mutex, process the buffer instead of src to avoid sleeping function called from atomic context. Signed-off-by: Tian Tao Reviewed-by: Vitaly Wool Acked-by: Sebastian Andrzej Siewior Reported-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior commit 2f2f465cc2f36dd5d5f2c5a4c40a3b559605d0d1 Author: Thomas Gleixner Date: Sun Dec 6 22:40:07 2020 +0100 timers: Move clearing of base::timer_running under base::lock syzbot reported KCSAN data races vs. timer_base::timer_running being set to NULL without holding base::lock in expire_timers(). This looks innocent and most reads are clearly not problematic but for a non-RT kernel it's completely irrelevant whether the store happens before or after taking the lock. For an RT kernel moving the store under the lock requires an extra unlock/lock pair in the case that there is a waiter for the timer. But that's not the end of the world and definitely not worth the trouble of adding boatloads of comments and annotations to the code. Famous last words... Reported-by: syzbot+aa7c2385d46c5eba0b89@syzkaller.appspotmail.com Reported-by: syzbot+abea4558531bae1ba9fe@syzkaller.appspotmail.com Link: https://lkml.kernel.org/r/87lfea7gw8.fsf@nanos.tec.linutronix.de Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior Cc: stable-rt@vger.kernel.org commit 99d6f6e50570d543cecf47f06279a42e48a7fc6e Author: Sebastian Andrzej Siewior Date: Fri Oct 30 13:59:06 2020 +0100 highmem: Don't disable preemption on RT in kmap_atomic() Disabling preemption makes it impossible to acquire sleeping locks within kmap_atomic() section. For PREEMPT_RT it is sufficient to disable migration. Signed-off-by: Sebastian Andrzej Siewior