diff options
| author | Frederic Weisbecker <frederic@kernel.org> | 2021-11-23 01:37:03 +0100 |
|---|---|---|
| committer | Paul E. McKenney <paulmck@kernel.org> | 2021-12-09 11:34:07 -0800 |
| commit | 2ebc45c44c4f3cc4c757430b2409ece4f976892e (patch) | |
| tree | 0c9e06441054f59d678ec2ca7e81e475bfc1c954 /kernel/rcu/tree.h | |
| parent | 0598a4d4429c0a952ac0e99e5280354cf4ccc01c (diff) | |
rcu/nocb: Remove rcu_node structure from nocb list when de-offloaded
The nocb_gp_wait() function iterates over all CPUs in its group,
including even those CPUs that have been de-offloaded. This is of
course suboptimal, especially if none of the CPUs within the group are
currently offloaded. This will become even more of a problem once a
nocb kthread is created for all possible CPUs.
Therefore use a standard double linked list to link all the offloaded
rcu_data structures and safely add or delete these structure as we
offload or de-offload them, respectively.
Reviewed-by: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Uladzislau Rezki <urezki@gmail.com>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Joel Fernandes <joel@joelfernandes.org>
Tested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Diffstat (limited to 'kernel/rcu/tree.h')
| -rw-r--r-- | kernel/rcu/tree.h | 7 |
1 files changed, 5 insertions, 2 deletions
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 4f6c67b3ccd5..5884380f4039 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -227,8 +227,11 @@ struct rcu_data { struct swait_queue_head nocb_gp_wq; /* For nocb kthreads to sleep on. */ bool nocb_cb_sleep; /* Is the nocb CB thread asleep? */ struct task_struct *nocb_cb_kthread; - struct rcu_data *nocb_next_cb_rdp; - /* Next rcu_data in wakeup chain. */ + struct list_head nocb_head_rdp; /* + * Head of rcu_data list in wakeup chain, + * if rdp_gp. + */ + struct list_head nocb_entry_rdp; /* rcu_data node in wakeup chain. */ /* The following fields are used by CB kthread, hence new cacheline. */ struct rcu_data *nocb_gp_rdp ____cacheline_internodealigned_in_smp; |