brintos / linux-shallow public Read only

1// SPDX-License-Identifier: GPL-2.0-or-later2/*3 * Budget Fair Queueing (BFQ) I/O scheduler.4 *5 * Based on ideas and code from CFQ:6 * Copyright (C) 2003 Jens Axboe <axboe@kernel.dk>7 *8 * Copyright (C) 2008 Fabio Checconi <fabio@gandalf.sssup.it>9 *		      Paolo Valente <paolo.valente@unimore.it>10 *11 * Copyright (C) 2010 Paolo Valente <paolo.valente@unimore.it>12 *                    Arianna Avanzini <avanzini@google.com>13 *14 * Copyright (C) 2017 Paolo Valente <paolo.valente@linaro.org>15 *16 * BFQ is a proportional-share I/O scheduler, with some extra17 * low-latency capabilities. BFQ also supports full hierarchical18 * scheduling through cgroups. Next paragraphs provide an introduction19 * on BFQ inner workings. Details on BFQ benefits, usage and20 * limitations can be found in Documentation/block/bfq-iosched.rst.21 *22 * BFQ is a proportional-share storage-I/O scheduling algorithm based23 * on the slice-by-slice service scheme of CFQ. But BFQ assigns24 * budgets, measured in number of sectors, to processes instead of25 * time slices. The device is not granted to the in-service process26 * for a given time slice, but until it has exhausted its assigned27 * budget. This change from the time to the service domain enables BFQ28 * to distribute the device throughput among processes as desired,29 * without any distortion due to throughput fluctuations, or to device30 * internal queueing. BFQ uses an ad hoc internal scheduler, called31 * B-WF2Q+, to schedule processes according to their budgets. More32 * precisely, BFQ schedules queues associated with processes. Each33 * process/queue is assigned a user-configurable weight, and B-WF2Q+34 * guarantees that each queue receives a fraction of the throughput35 * proportional to its weight. Thanks to the accurate policy of36 * B-WF2Q+, BFQ can afford to assign high budgets to I/O-bound37 * processes issuing sequential requests (to boost the throughput),38 * and yet guarantee a low latency to interactive and soft real-time39 * applications.40 *41 * In particular, to provide these low-latency guarantees, BFQ42 * explicitly privileges the I/O of two classes of time-sensitive43 * applications: interactive and soft real-time. In more detail, BFQ44 * behaves this way if the low_latency parameter is set (default45 * configuration). This feature enables BFQ to provide applications in46 * these classes with a very low latency.47 *48 * To implement this feature, BFQ constantly tries to detect whether49 * the I/O requests in a bfq_queue come from an interactive or a soft50 * real-time application. For brevity, in these cases, the queue is51 * said to be interactive or soft real-time. In both cases, BFQ52 * privileges the service of the queue, over that of non-interactive53 * and non-soft-real-time queues. This privileging is performed,54 * mainly, by raising the weight of the queue. So, for brevity, we55 * call just weight-raising periods the time periods during which a56 * queue is privileged, because deemed interactive or soft real-time.57 *58 * The detection of soft real-time queues/applications is described in59 * detail in the comments on the function60 * bfq_bfqq_softrt_next_start. On the other hand, the detection of an61 * interactive queue works as follows: a queue is deemed interactive62 * if it is constantly non empty only for a limited time interval,63 * after which it does become empty. The queue may be deemed64 * interactive again (for a limited time), if it restarts being65 * constantly non empty, provided that this happens only after the66 * queue has remained empty for a given minimum idle time.67 *68 * By default, BFQ computes automatically the above maximum time69 * interval, i.e., the time interval after which a constantly70 * non-empty queue stops being deemed interactive. Since a queue is71 * weight-raised while it is deemed interactive, this maximum time72 * interval happens to coincide with the (maximum) duration of the73 * weight-raising for interactive queues.74 *75 * Finally, BFQ also features additional heuristics for76 * preserving both a low latency and a high throughput on NCQ-capable,77 * rotational or flash-based devices, and to get the job done quickly78 * for applications consisting in many I/O-bound processes.79 *80 * NOTE: if the main or only goal, with a given device, is to achieve81 * the maximum-possible throughput at all times, then do switch off82 * all low-latency heuristics for that device, by setting low_latency83 * to 0.84 *85 * BFQ is described in [1], where also a reference to the initial,86 * more theoretical paper on BFQ can be found. The interested reader87 * can find in the latter paper full details on the main algorithm, as88 * well as formulas of the guarantees and formal proofs of all the89 * properties.  With respect to the version of BFQ presented in these90 * papers, this implementation adds a few more heuristics, such as the91 * ones that guarantee a low latency to interactive and soft real-time92 * applications, and a hierarchical extension based on H-WF2Q+.93 *94 * B-WF2Q+ is based on WF2Q+, which is described in [2], together with95 * H-WF2Q+, while the augmented tree used here to implement B-WF2Q+96 * with O(log N) complexity derives from the one introduced with EEVDF97 * in [3].98 *99 * [1] P. Valente, A. Avanzini, "Evolution of the BFQ Storage I/O100 *     Scheduler", Proceedings of the First Workshop on Mobile System101 *     Technologies (MST-2015), May 2015.102 *     http://algogroup.unimore.it/people/paolo/disk_sched/mst-2015.pdf103 *104 * [2] Jon C.R. Bennett and H. Zhang, "Hierarchical Packet Fair Queueing105 *     Algorithms", IEEE/ACM Transactions on Networking, 5(5):675-689,106 *     Oct 1997.107 *108 * http://www.cs.cmu.edu/~hzhang/papers/TON-97-Oct.ps.gz109 *110 * [3] I. Stoica and H. Abdel-Wahab, "Earliest Eligible Virtual Deadline111 *     First: A Flexible and Accurate Mechanism for Proportional Share112 *     Resource Allocation", technical report.113 *114 * http://www.cs.berkeley.edu/~istoica/papers/eevdf-tr-95.pdf115 */116#include <linux/module.h>117#include <linux/slab.h>118#include <linux/blkdev.h>119#include <linux/cgroup.h>120#include <linux/ktime.h>121#include <linux/rbtree.h>122#include <linux/ioprio.h>123#include <linux/sbitmap.h>124#include <linux/delay.h>125#include <linux/backing-dev.h>126 127#include <trace/events/block.h>128 129#include "elevator.h"130#include "blk.h"131#include "blk-mq.h"132#include "blk-mq-sched.h"133#include "bfq-iosched.h"134#include "blk-wbt.h"135 136#define BFQ_BFQQ_FNS(name)						\137void bfq_mark_bfqq_##name(struct bfq_queue *bfqq)			\138{									\139	__set_bit(BFQQF_##name, &(bfqq)->flags);			\140}									\141void bfq_clear_bfqq_##name(struct bfq_queue *bfqq)			\142{									\143	__clear_bit(BFQQF_##name, &(bfqq)->flags);		\144}									\145int bfq_bfqq_##name(const struct bfq_queue *bfqq)			\146{									\147	return test_bit(BFQQF_##name, &(bfqq)->flags);		\148}149 150BFQ_BFQQ_FNS(just_created);151BFQ_BFQQ_FNS(busy);152BFQ_BFQQ_FNS(wait_request);153BFQ_BFQQ_FNS(non_blocking_wait_rq);154BFQ_BFQQ_FNS(fifo_expire);155BFQ_BFQQ_FNS(has_short_ttime);156BFQ_BFQQ_FNS(sync);157BFQ_BFQQ_FNS(IO_bound);158BFQ_BFQQ_FNS(in_large_burst);159BFQ_BFQQ_FNS(coop);160BFQ_BFQQ_FNS(split_coop);161BFQ_BFQQ_FNS(softrt_update);162#undef BFQ_BFQQ_FNS						\163 164/* Expiration time of async (0) and sync (1) requests, in ns. */165static const u64 bfq_fifo_expire[2] = { NSEC_PER_SEC / 4, NSEC_PER_SEC / 8 };166 167/* Maximum backwards seek (magic number lifted from CFQ), in KiB. */168static const int bfq_back_max = 16 * 1024;169 170/* Penalty of a backwards seek, in number of sectors. */171static const int bfq_back_penalty = 2;172 173/* Idling period duration, in ns. */174static u64 bfq_slice_idle = NSEC_PER_SEC / 125;175 176/* Minimum number of assigned budgets for which stats are safe to compute. */177static const int bfq_stats_min_budgets = 194;178 179/* Default maximum budget values, in sectors and number of requests. */180static const int bfq_default_max_budget = 16 * 1024;181 182/*183 * When a sync request is dispatched, the queue that contains that184 * request, and all the ancestor entities of that queue, are charged185 * with the number of sectors of the request. In contrast, if the186 * request is async, then the queue and its ancestor entities are187 * charged with the number of sectors of the request, multiplied by188 * the factor below. This throttles the bandwidth for async I/O,189 * w.r.t. to sync I/O, and it is done to counter the tendency of async190 * writes to steal I/O throughput to reads.191 *192 * The current value of this parameter is the result of a tuning with193 * several hardware and software configurations. We tried to find the194 * lowest value for which writes do not cause noticeable problems to195 * reads. In fact, the lower this parameter, the stabler I/O control,196 * in the following respect.  The lower this parameter is, the less197 * the bandwidth enjoyed by a group decreases198 * - when the group does writes, w.r.t. to when it does reads;199 * - when other groups do reads, w.r.t. to when they do writes.200 */201static const int bfq_async_charge_factor = 3;202 203/* Default timeout values, in jiffies, approximating CFQ defaults. */204const int bfq_timeout = HZ / 8;205 206/*207 * Time limit for merging (see comments in bfq_setup_cooperator). Set208 * to the slowest value that, in our tests, proved to be effective in209 * removing false positives, while not causing true positives to miss210 * queue merging.211 *212 * As can be deduced from the low time limit below, queue merging, if213 * successful, happens at the very beginning of the I/O of the involved214 * cooperating processes, as a consequence of the arrival of the very215 * first requests from each cooperator.  After that, there is very216 * little chance to find cooperators.217 */218static const unsigned long bfq_merge_time_limit = HZ/10;219 220static struct kmem_cache *bfq_pool;221 222/* Below this threshold (in ns), we consider thinktime immediate. */223#define BFQ_MIN_TT		(2 * NSEC_PER_MSEC)224 225/* hw_tag detection: parallel requests threshold and min samples needed. */226#define BFQ_HW_QUEUE_THRESHOLD	3227#define BFQ_HW_QUEUE_SAMPLES	32228 229#define BFQQ_SEEK_THR		(sector_t)(8 * 100)230#define BFQQ_SECT_THR_NONROT	(sector_t)(2 * 32)231#define BFQ_RQ_SEEKY(bfqd, last_pos, rq) \232	(get_sdist(last_pos, rq) >			\233	 BFQQ_SEEK_THR &&				\234	 (!blk_queue_nonrot(bfqd->queue) ||		\235	  blk_rq_sectors(rq) < BFQQ_SECT_THR_NONROT))236#define BFQQ_CLOSE_THR		(sector_t)(8 * 1024)237#define BFQQ_SEEKY(bfqq)	(hweight32(bfqq->seek_history) > 19)238/*239 * Sync random I/O is likely to be confused with soft real-time I/O,240 * because it is characterized by limited throughput and apparently241 * isochronous arrival pattern. To avoid false positives, queues242 * containing only random (seeky) I/O are prevented from being tagged243 * as soft real-time.244 */245#define BFQQ_TOTALLY_SEEKY(bfqq)	(bfqq->seek_history == -1)246 247/* Min number of samples required to perform peak-rate update */248#define BFQ_RATE_MIN_SAMPLES	32249/* Min observation time interval required to perform a peak-rate update (ns) */250#define BFQ_RATE_MIN_INTERVAL	(300*NSEC_PER_MSEC)251/* Target observation time interval for a peak-rate update (ns) */252#define BFQ_RATE_REF_INTERVAL	NSEC_PER_SEC253 254/*255 * Shift used for peak-rate fixed precision calculations.256 * With257 * - the current shift: 16 positions258 * - the current type used to store rate: u32259 * - the current unit of measure for rate: [sectors/usec], or, more precisely,260 *   [(sectors/usec) / 2^BFQ_RATE_SHIFT] to take into account the shift,261 * the range of rates that can be stored is262 * [1 / 2^BFQ_RATE_SHIFT, 2^(32 - BFQ_RATE_SHIFT)] sectors/usec =263 * [1 / 2^16, 2^16] sectors/usec = [15e-6, 65536] sectors/usec =264 * [15, 65G] sectors/sec265 * Which, assuming a sector size of 512B, corresponds to a range of266 * [7.5K, 33T] B/sec267 */268#define BFQ_RATE_SHIFT		16269 270/*271 * When configured for computing the duration of the weight-raising272 * for interactive queues automatically (see the comments at the273 * beginning of this file), BFQ does it using the following formula:274 * duration = (ref_rate / r) * ref_wr_duration,275 * where r is the peak rate of the device, and ref_rate and276 * ref_wr_duration are two reference parameters.  In particular,277 * ref_rate is the peak rate of the reference storage device (see278 * below), and ref_wr_duration is about the maximum time needed, with279 * BFQ and while reading two files in parallel, to load typical large280 * applications on the reference device (see the comments on281 * max_service_from_wr below, for more details on how ref_wr_duration282 * is obtained).  In practice, the slower/faster the device at hand283 * is, the more/less it takes to load applications with respect to the284 * reference device.  Accordingly, the longer/shorter BFQ grants285 * weight raising to interactive applications.286 *287 * BFQ uses two different reference pairs (ref_rate, ref_wr_duration),288 * depending on whether the device is rotational or non-rotational.289 *290 * In the following definitions, ref_rate[0] and ref_wr_duration[0]291 * are the reference values for a rotational device, whereas292 * ref_rate[1] and ref_wr_duration[1] are the reference values for a293 * non-rotational device. The reference rates are not the actual peak294 * rates of the devices used as a reference, but slightly lower295 * values. The reason for using slightly lower values is that the296 * peak-rate estimator tends to yield slightly lower values than the297 * actual peak rate (it can yield the actual peak rate only if there298 * is only one process doing I/O, and the process does sequential299 * I/O).300 *301 * The reference peak rates are measured in sectors/usec, left-shifted302 * by BFQ_RATE_SHIFT.303 */304static int ref_rate[2] = {14000, 33000};305/*306 * To improve readability, a conversion function is used to initialize307 * the following array, which entails that the array can be308 * initialized only in a function.309 */310static int ref_wr_duration[2];311 312/*313 * BFQ uses the above-detailed, time-based weight-raising mechanism to314 * privilege interactive tasks. This mechanism is vulnerable to the315 * following false positives: I/O-bound applications that will go on316 * doing I/O for much longer than the duration of weight317 * raising. These applications have basically no benefit from being318 * weight-raised at the beginning of their I/O. On the opposite end,319 * while being weight-raised, these applications320 * a) unjustly steal throughput to applications that may actually need321 * low latency;322 * b) make BFQ uselessly perform device idling; device idling results323 * in loss of device throughput with most flash-based storage, and may324 * increase latencies when used purposelessly.325 *326 * BFQ tries to reduce these problems, by adopting the following327 * countermeasure. To introduce this countermeasure, we need first to328 * finish explaining how the duration of weight-raising for329 * interactive tasks is computed.330 *331 * For a bfq_queue deemed as interactive, the duration of weight332 * raising is dynamically adjusted, as a function of the estimated333 * peak rate of the device, so as to be equal to the time needed to334 * execute the 'largest' interactive task we benchmarked so far. By335 * largest task, we mean the task for which each involved process has336 * to do more I/O than for any of the other tasks we benchmarked. This337 * reference interactive task is the start-up of LibreOffice Writer,338 * and in this task each process/bfq_queue needs to have at most ~110K339 * sectors transferred.340 *341 * This last piece of information enables BFQ to reduce the actual342 * duration of weight-raising for at least one class of I/O-bound343 * applications: those doing sequential or quasi-sequential I/O. An344 * example is file copy. In fact, once started, the main I/O-bound345 * processes of these applications usually consume the above 110K346 * sectors in much less time than the processes of an application that347 * is starting, because these I/O-bound processes will greedily devote348 * almost all their CPU cycles only to their target,349 * throughput-friendly I/O operations. This is even more true if BFQ350 * happens to be underestimating the device peak rate, and thus351 * overestimating the duration of weight raising. But, according to352 * our measurements, once transferred 110K sectors, these processes353 * have no right to be weight-raised any longer.354 *355 * Basing on the last consideration, BFQ ends weight-raising for a356 * bfq_queue if the latter happens to have received an amount of357 * service at least equal to the following constant. The constant is358 * set to slightly more than 110K, to have a minimum safety margin.359 *360 * This early ending of weight-raising reduces the amount of time361 * during which interactive false positives cause the two problems362 * described at the beginning of these comments.363 */364static const unsigned long max_service_from_wr = 120000;365 366/*367 * Maximum time between the creation of two queues, for stable merge368 * to be activated (in ms)369 */370static const unsigned long bfq_activation_stable_merging = 600;371/*372 * Minimum time to be waited before evaluating delayed stable merge (in ms)373 */374static const unsigned long bfq_late_stable_merging = 600;375 376#define RQ_BIC(rq)		((struct bfq_io_cq *)((rq)->elv.priv[0]))377#define RQ_BFQQ(rq)		((rq)->elv.priv[1])378 379struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync,380			      unsigned int actuator_idx)381{382	if (is_sync)383		return bic->bfqq[1][actuator_idx];384 385	return bic->bfqq[0][actuator_idx];386}387 388static void bfq_put_stable_ref(struct bfq_queue *bfqq);389 390void bic_set_bfqq(struct bfq_io_cq *bic,391		  struct bfq_queue *bfqq,392		  bool is_sync,393		  unsigned int actuator_idx)394{395	struct bfq_queue *old_bfqq = bic->bfqq[is_sync][actuator_idx];396 397	/*398	 * If bfqq != NULL, then a non-stable queue merge between399	 * bic->bfqq and bfqq is happening here. This causes troubles400	 * in the following case: bic->bfqq has also been scheduled401	 * for a possible stable merge with bic->stable_merge_bfqq,402	 * and bic->stable_merge_bfqq == bfqq happens to403	 * hold. Troubles occur because bfqq may then undergo a split,404	 * thereby becoming eligible for a stable merge. Yet, if405	 * bic->stable_merge_bfqq points exactly to bfqq, then bfqq406	 * would be stably merged with itself. To avoid this anomaly,407	 * we cancel the stable merge if408	 * bic->stable_merge_bfqq == bfqq.409	 */410	struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[actuator_idx];411 412	/* Clear bic pointer if bfqq is detached from this bic */413	if (old_bfqq && old_bfqq->bic == bic)414		old_bfqq->bic = NULL;415 416	if (is_sync)417		bic->bfqq[1][actuator_idx] = bfqq;418	else419		bic->bfqq[0][actuator_idx] = bfqq;420 421	if (bfqq && bfqq_data->stable_merge_bfqq == bfqq) {422		/*423		 * Actually, these same instructions are executed also424		 * in bfq_setup_cooperator, in case of abort or actual425		 * execution of a stable merge. We could avoid426		 * repeating these instructions there too, but if we427		 * did so, we would nest even more complexity in this428		 * function.429		 */430		bfq_put_stable_ref(bfqq_data->stable_merge_bfqq);431 432		bfqq_data->stable_merge_bfqq = NULL;433	}434}435 436struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic)437{438	return bic->icq.q->elevator->elevator_data;439}440 441/**442 * icq_to_bic - convert iocontext queue structure to bfq_io_cq.443 * @icq: the iocontext queue.444 */445static struct bfq_io_cq *icq_to_bic(struct io_cq *icq)446{447	/* bic->icq is the first member, %NULL will convert to %NULL */448	return container_of(icq, struct bfq_io_cq, icq);449}450 451/**452 * bfq_bic_lookup - search into @ioc a bic associated to @bfqd.453 * @q: the request queue.454 */455static struct bfq_io_cq *bfq_bic_lookup(struct request_queue *q)456{457	struct bfq_io_cq *icq;458	unsigned long flags;459 460	if (!current->io_context)461		return NULL;462 463	spin_lock_irqsave(&q->queue_lock, flags);464	icq = icq_to_bic(ioc_lookup_icq(q));465	spin_unlock_irqrestore(&q->queue_lock, flags);466 467	return icq;468}469 470/*471 * Scheduler run of queue, if there are requests pending and no one in the472 * driver that will restart queueing.473 */474void bfq_schedule_dispatch(struct bfq_data *bfqd)475{476	lockdep_assert_held(&bfqd->lock);477 478	if (bfqd->queued != 0) {479		bfq_log(bfqd, "schedule dispatch");480		blk_mq_run_hw_queues(bfqd->queue, true);481	}482}483 484#define bfq_class_idle(bfqq)	((bfqq)->ioprio_class == IOPRIO_CLASS_IDLE)485 486#define bfq_sample_valid(samples)	((samples) > 80)487 488/*489 * Lifted from AS - choose which of rq1 and rq2 that is best served now.490 * We choose the request that is closer to the head right now.  Distance491 * behind the head is penalized and only allowed to a certain extent.492 */493static struct request *bfq_choose_req(struct bfq_data *bfqd,494				      struct request *rq1,495				      struct request *rq2,496				      sector_t last)497{498	sector_t s1, s2, d1 = 0, d2 = 0;499	unsigned long back_max;500#define BFQ_RQ1_WRAP	0x01 /* request 1 wraps */501#define BFQ_RQ2_WRAP	0x02 /* request 2 wraps */502	unsigned int wrap = 0; /* bit mask: requests behind the disk head? */503 504	if (!rq1 || rq1 == rq2)505		return rq2;506	if (!rq2)507		return rq1;508 509	if (rq_is_sync(rq1) && !rq_is_sync(rq2))510		return rq1;511	else if (rq_is_sync(rq2) && !rq_is_sync(rq1))512		return rq2;513	if ((rq1->cmd_flags & REQ_META) && !(rq2->cmd_flags & REQ_META))514		return rq1;515	else if ((rq2->cmd_flags & REQ_META) && !(rq1->cmd_flags & REQ_META))516		return rq2;517 518	s1 = blk_rq_pos(rq1);519	s2 = blk_rq_pos(rq2);520 521	/*522	 * By definition, 1KiB is 2 sectors.523	 */524	back_max = bfqd->bfq_back_max * 2;525 526	/*527	 * Strict one way elevator _except_ in the case where we allow528	 * short backward seeks which are biased as twice the cost of a529	 * similar forward seek.530	 */531	if (s1 >= last)532		d1 = s1 - last;533	else if (s1 + back_max >= last)534		d1 = (last - s1) * bfqd->bfq_back_penalty;535	else536		wrap |= BFQ_RQ1_WRAP;537 538	if (s2 >= last)539		d2 = s2 - last;540	else if (s2 + back_max >= last)541		d2 = (last - s2) * bfqd->bfq_back_penalty;542	else543		wrap |= BFQ_RQ2_WRAP;544 545	/* Found required data */546 547	/*548	 * By doing switch() on the bit mask "wrap" we avoid having to549	 * check two variables for all permutations: --> faster!550	 */551	switch (wrap) {552	case 0: /* common case for CFQ: rq1 and rq2 not wrapped */553		if (d1 < d2)554			return rq1;555		else if (d2 < d1)556			return rq2;557 558		if (s1 >= s2)559			return rq1;560		else561			return rq2;562 563	case BFQ_RQ2_WRAP:564		return rq1;565	case BFQ_RQ1_WRAP:566		return rq2;567	case BFQ_RQ1_WRAP|BFQ_RQ2_WRAP: /* both rqs wrapped */568	default:569		/*570		 * Since both rqs are wrapped,571		 * start with the one that's further behind head572		 * (--> only *one* back seek required),573		 * since back seek takes more time than forward.574		 */575		if (s1 <= s2)576			return rq1;577		else578			return rq2;579	}580}581 582#define BFQ_LIMIT_INLINE_DEPTH 16583 584#ifdef CONFIG_BFQ_GROUP_IOSCHED585static bool bfqq_request_over_limit(struct bfq_queue *bfqq, int limit)586{587	struct bfq_data *bfqd = bfqq->bfqd;588	struct bfq_entity *entity = &bfqq->entity;589	struct bfq_entity *inline_entities[BFQ_LIMIT_INLINE_DEPTH];590	struct bfq_entity **entities = inline_entities;591	int depth, level, alloc_depth = BFQ_LIMIT_INLINE_DEPTH;592	int class_idx = bfqq->ioprio_class - 1;593	struct bfq_sched_data *sched_data;594	unsigned long wsum;595	bool ret = false;596 597	if (!entity->on_st_or_in_serv)598		return false;599 600retry:601	spin_lock_irq(&bfqd->lock);602	/* +1 for bfqq entity, root cgroup not included */603	depth = bfqg_to_blkg(bfqq_group(bfqq))->blkcg->css.cgroup->level + 1;604	if (depth > alloc_depth) {605		spin_unlock_irq(&bfqd->lock);606		if (entities != inline_entities)607			kfree(entities);608		entities = kmalloc_array(depth, sizeof(*entities), GFP_NOIO);609		if (!entities)610			return false;611		alloc_depth = depth;612		goto retry;613	}614 615	sched_data = entity->sched_data;616	/* Gather our ancestors as we need to traverse them in reverse order */617	level = 0;618	for_each_entity(entity) {619		/*620		 * If at some level entity is not even active, allow request621		 * queueing so that BFQ knows there's work to do and activate622		 * entities.623		 */624		if (!entity->on_st_or_in_serv)625			goto out;626		/* Uh, more parents than cgroup subsystem thinks? */627		if (WARN_ON_ONCE(level >= depth))628			break;629		entities[level++] = entity;630	}631	WARN_ON_ONCE(level != depth);632	for (level--; level >= 0; level--) {633		entity = entities[level];634		if (level > 0) {635			wsum = bfq_entity_service_tree(entity)->wsum;636		} else {637			int i;638			/*639			 * For bfqq itself we take into account service trees640			 * of all higher priority classes and multiply their641			 * weights so that low prio queue from higher class642			 * gets more requests than high prio queue from lower643			 * class.644			 */645			wsum = 0;646			for (i = 0; i <= class_idx; i++) {647				wsum = wsum * IOPRIO_BE_NR +648					sched_data->service_tree[i].wsum;649			}650		}651		if (!wsum)652			continue;653		limit = DIV_ROUND_CLOSEST(limit * entity->weight, wsum);654		if (entity->allocated >= limit) {655			bfq_log_bfqq(bfqq->bfqd, bfqq,656				"too many requests: allocated %d limit %d level %d",657				entity->allocated, limit, level);658			ret = true;659			break;660		}661	}662out:663	spin_unlock_irq(&bfqd->lock);664	if (entities != inline_entities)665		kfree(entities);666	return ret;667}668#else669static bool bfqq_request_over_limit(struct bfq_queue *bfqq, int limit)670{671	return false;672}673#endif674 675/*676 * Async I/O can easily starve sync I/O (both sync reads and sync677 * writes), by consuming all tags. Similarly, storms of sync writes,678 * such as those that sync(2) may trigger, can starve sync reads.679 * Limit depths of async I/O and sync writes so as to counter both680 * problems.681 *682 * Also if a bfq queue or its parent cgroup consume more tags than would be683 * appropriate for their weight, we trim the available tag depth to 1. This684 * avoids a situation where one cgroup can starve another cgroup from tags and685 * thus block service differentiation among cgroups. Note that because the686 * queue / cgroup already has many requests allocated and queued, this does not687 * significantly affect service guarantees coming from the BFQ scheduling688 * algorithm.689 */690static void bfq_limit_depth(blk_opf_t opf, struct blk_mq_alloc_data *data)691{692	struct bfq_data *bfqd = data->q->elevator->elevator_data;693	struct bfq_io_cq *bic = bfq_bic_lookup(data->q);694	int depth;695	unsigned limit = data->q->nr_requests;696	unsigned int act_idx;697 698	/* Sync reads have full depth available */699	if (op_is_sync(opf) && !op_is_write(opf)) {700		depth = 0;701	} else {702		depth = bfqd->word_depths[!!bfqd->wr_busy_queues][op_is_sync(opf)];703		limit = (limit * depth) >> bfqd->full_depth_shift;704	}705 706	for (act_idx = 0; bic && act_idx < bfqd->num_actuators; act_idx++) {707		struct bfq_queue *bfqq =708			bic_to_bfqq(bic, op_is_sync(opf), act_idx);709 710		/*711		 * Does queue (or any parent entity) exceed number of712		 * requests that should be available to it? Heavily713		 * limit depth so that it cannot consume more714		 * available requests and thus starve other entities.715		 */716		if (bfqq && bfqq_request_over_limit(bfqq, limit)) {717			depth = 1;718			break;719		}720	}721	bfq_log(bfqd, "[%s] wr_busy %d sync %d depth %u",722		__func__, bfqd->wr_busy_queues, op_is_sync(opf), depth);723	if (depth)724		data->shallow_depth = depth;725}726 727static struct bfq_queue *728bfq_rq_pos_tree_lookup(struct bfq_data *bfqd, struct rb_root *root,729		     sector_t sector, struct rb_node **ret_parent,730		     struct rb_node ***rb_link)731{732	struct rb_node **p, *parent;733	struct bfq_queue *bfqq = NULL;734 735	parent = NULL;736	p = &root->rb_node;737	while (*p) {738		struct rb_node **n;739 740		parent = *p;741		bfqq = rb_entry(parent, struct bfq_queue, pos_node);742 743		/*744		 * Sort strictly based on sector. Smallest to the left,745		 * largest to the right.746		 */747		if (sector > blk_rq_pos(bfqq->next_rq))748			n = &(*p)->rb_right;749		else if (sector < blk_rq_pos(bfqq->next_rq))750			n = &(*p)->rb_left;751		else752			break;753		p = n;754		bfqq = NULL;755	}756 757	*ret_parent = parent;758	if (rb_link)759		*rb_link = p;760 761	bfq_log(bfqd, "rq_pos_tree_lookup %llu: returning %d",762		(unsigned long long)sector,763		bfqq ? bfqq->pid : 0);764 765	return bfqq;766}767 768static bool bfq_too_late_for_merging(struct bfq_queue *bfqq)769{770	return bfqq->service_from_backlogged > 0 &&771		time_is_before_jiffies(bfqq->first_IO_time +772				       bfq_merge_time_limit);773}774 775/*776 * The following function is not marked as __cold because it is777 * actually cold, but for the same performance goal described in the778 * comments on the likely() at the beginning of779 * bfq_setup_cooperator(). Unexpectedly, to reach an even lower780 * execution time for the case where this function is not invoked, we781 * had to add an unlikely() in each involved if().782 */783void __cold784bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq)785{786	struct rb_node **p, *parent;787	struct bfq_queue *__bfqq;788 789	if (bfqq->pos_root) {790		rb_erase(&bfqq->pos_node, bfqq->pos_root);791		bfqq->pos_root = NULL;792	}793 794	/* oom_bfqq does not participate in queue merging */795	if (bfqq == &bfqd->oom_bfqq)796		return;797 798	/*799	 * bfqq cannot be merged any longer (see comments in800	 * bfq_setup_cooperator): no point in adding bfqq into the801	 * position tree.802	 */803	if (bfq_too_late_for_merging(bfqq))804		return;805 806	if (bfq_class_idle(bfqq))807		return;808	if (!bfqq->next_rq)809		return;810 811	bfqq->pos_root = &bfqq_group(bfqq)->rq_pos_tree;812	__bfqq = bfq_rq_pos_tree_lookup(bfqd, bfqq->pos_root,813			blk_rq_pos(bfqq->next_rq), &parent, &p);814	if (!__bfqq) {815		rb_link_node(&bfqq->pos_node, parent, p);816		rb_insert_color(&bfqq->pos_node, bfqq->pos_root);817	} else818		bfqq->pos_root = NULL;819}820 821/*822 * The following function returns false either if every active queue823 * must receive the same share of the throughput (symmetric scenario),824 * or, as a special case, if bfqq must receive a share of the825 * throughput lower than or equal to the share that every other active826 * queue must receive.  If bfqq does sync I/O, then these are the only827 * two cases where bfqq happens to be guaranteed its share of the828 * throughput even if I/O dispatching is not plugged when bfqq remains829 * temporarily empty (for more details, see the comments in the830 * function bfq_better_to_idle()). For this reason, the return value831 * of this function is used to check whether I/O-dispatch plugging can832 * be avoided.833 *834 * The above first case (symmetric scenario) occurs when:835 * 1) all active queues have the same weight,836 * 2) all active queues belong to the same I/O-priority class,837 * 3) all active groups at the same level in the groups tree have the same838 *    weight,839 * 4) all active groups at the same level in the groups tree have the same840 *    number of children.841 *842 * Unfortunately, keeping the necessary state for evaluating exactly843 * the last two symmetry sub-conditions above would be quite complex844 * and time consuming. Therefore this function evaluates, instead,845 * only the following stronger three sub-conditions, for which it is846 * much easier to maintain the needed state:847 * 1) all active queues have the same weight,848 * 2) all active queues belong to the same I/O-priority class,849 * 3) there is at most one active group.850 * In particular, the last condition is always true if hierarchical851 * support or the cgroups interface are not enabled, thus no state852 * needs to be maintained in this case.853 */854static bool bfq_asymmetric_scenario(struct bfq_data *bfqd,855				   struct bfq_queue *bfqq)856{857	bool smallest_weight = bfqq &&858		bfqq->weight_counter &&859		bfqq->weight_counter ==860		container_of(861			rb_first_cached(&bfqd->queue_weights_tree),862			struct bfq_weight_counter,863			weights_node);864 865	/*866	 * For queue weights to differ, queue_weights_tree must contain867	 * at least two nodes.868	 */869	bool varied_queue_weights = !smallest_weight &&870		!RB_EMPTY_ROOT(&bfqd->queue_weights_tree.rb_root) &&871		(bfqd->queue_weights_tree.rb_root.rb_node->rb_left ||872		 bfqd->queue_weights_tree.rb_root.rb_node->rb_right);873 874	bool multiple_classes_busy =875		(bfqd->busy_queues[0] && bfqd->busy_queues[1]) ||876		(bfqd->busy_queues[0] && bfqd->busy_queues[2]) ||877		(bfqd->busy_queues[1] && bfqd->busy_queues[2]);878 879	return varied_queue_weights || multiple_classes_busy880#ifdef CONFIG_BFQ_GROUP_IOSCHED881	       || bfqd->num_groups_with_pending_reqs > 1882#endif883		;884}885 886/*887 * If the weight-counter tree passed as input contains no counter for888 * the weight of the input queue, then add that counter; otherwise just889 * increment the existing counter.890 *891 * Note that weight-counter trees contain few nodes in mostly symmetric892 * scenarios. For example, if all queues have the same weight, then the893 * weight-counter tree for the queues may contain at most one node.894 * This holds even if low_latency is on, because weight-raised queues895 * are not inserted in the tree.896 * In most scenarios, the rate at which nodes are created/destroyed897 * should be low too.898 */899void bfq_weights_tree_add(struct bfq_queue *bfqq)900{901	struct rb_root_cached *root = &bfqq->bfqd->queue_weights_tree;902	struct bfq_entity *entity = &bfqq->entity;903	struct rb_node **new = &(root->rb_root.rb_node), *parent = NULL;904	bool leftmost = true;905 906	/*907	 * Do not insert if the queue is already associated with a908	 * counter, which happens if:909	 *   1) a request arrival has caused the queue to become both910	 *      non-weight-raised, and hence change its weight, and911	 *      backlogged; in this respect, each of the two events912	 *      causes an invocation of this function,913	 *   2) this is the invocation of this function caused by the914	 *      second event. This second invocation is actually useless,915	 *      and we handle this fact by exiting immediately. More916	 *      efficient or clearer solutions might possibly be adopted.917	 */918	if (bfqq->weight_counter)919		return;920 921	while (*new) {922		struct bfq_weight_counter *__counter = container_of(*new,923						struct bfq_weight_counter,924						weights_node);925		parent = *new;926 927		if (entity->weight == __counter->weight) {928			bfqq->weight_counter = __counter;929			goto inc_counter;930		}931		if (entity->weight < __counter->weight)932			new = &((*new)->rb_left);933		else {934			new = &((*new)->rb_right);935			leftmost = false;936		}937	}938 939	bfqq->weight_counter = kzalloc(sizeof(struct bfq_weight_counter),940				       GFP_ATOMIC);941 942	/*943	 * In the unlucky event of an allocation failure, we just944	 * exit. This will cause the weight of queue to not be945	 * considered in bfq_asymmetric_scenario, which, in its turn,946	 * causes the scenario to be deemed wrongly symmetric in case947	 * bfqq's weight would have been the only weight making the948	 * scenario asymmetric.  On the bright side, no unbalance will949	 * however occur when bfqq becomes inactive again (the950	 * invocation of this function is triggered by an activation951	 * of queue).  In fact, bfq_weights_tree_remove does nothing952	 * if !bfqq->weight_counter.953	 */954	if (unlikely(!bfqq->weight_counter))955		return;956 957	bfqq->weight_counter->weight = entity->weight;958	rb_link_node(&bfqq->weight_counter->weights_node, parent, new);959	rb_insert_color_cached(&bfqq->weight_counter->weights_node, root,960				leftmost);961 962inc_counter:963	bfqq->weight_counter->num_active++;964	bfqq->ref++;965}966 967/*968 * Decrement the weight counter associated with the queue, and, if the969 * counter reaches 0, remove the counter from the tree.970 * See the comments to the function bfq_weights_tree_add() for considerations971 * about overhead.972 */973void bfq_weights_tree_remove(struct bfq_queue *bfqq)974{975	struct rb_root_cached *root;976 977	if (!bfqq->weight_counter)978		return;979 980	root = &bfqq->bfqd->queue_weights_tree;981	bfqq->weight_counter->num_active--;982	if (bfqq->weight_counter->num_active > 0)983		goto reset_entity_pointer;984 985	rb_erase_cached(&bfqq->weight_counter->weights_node, root);986	kfree(bfqq->weight_counter);987 988reset_entity_pointer:989	bfqq->weight_counter = NULL;990	bfq_put_queue(bfqq);991}992 993/*994 * Return expired entry, or NULL to just start from scratch in rbtree.995 */996static struct request *bfq_check_fifo(struct bfq_queue *bfqq,997				      struct request *last)998{999	struct request *rq;1000 1001	if (bfq_bfqq_fifo_expire(bfqq))1002		return NULL;1003 1004	bfq_mark_bfqq_fifo_expire(bfqq);1005 1006	rq = rq_entry_fifo(bfqq->fifo.next);1007 1008	if (rq == last || blk_time_get_ns() < rq->fifo_time)1009		return NULL;1010 1011	bfq_log_bfqq(bfqq->bfqd, bfqq, "check_fifo: returned %p", rq);1012	return rq;1013}1014 1015static struct request *bfq_find_next_rq(struct bfq_data *bfqd,1016					struct bfq_queue *bfqq,1017					struct request *last)1018{1019	struct rb_node *rbnext = rb_next(&last->rb_node);1020	struct rb_node *rbprev = rb_prev(&last->rb_node);1021	struct request *next, *prev = NULL;1022 1023	/* Follow expired path, else get first next available. */1024	next = bfq_check_fifo(bfqq, last);1025	if (next)1026		return next;1027 1028	if (rbprev)1029		prev = rb_entry_rq(rbprev);1030 1031	if (rbnext)1032		next = rb_entry_rq(rbnext);1033	else {1034		rbnext = rb_first(&bfqq->sort_list);1035		if (rbnext && rbnext != &last->rb_node)1036			next = rb_entry_rq(rbnext);1037	}1038 1039	return bfq_choose_req(bfqd, next, prev, blk_rq_pos(last));1040}1041 1042/* see the definition of bfq_async_charge_factor for details */1043static unsigned long bfq_serv_to_charge(struct request *rq,1044					struct bfq_queue *bfqq)1045{1046	if (bfq_bfqq_sync(bfqq) || bfqq->wr_coeff > 1 ||1047	    bfq_asymmetric_scenario(bfqq->bfqd, bfqq))1048		return blk_rq_sectors(rq);1049 1050	return blk_rq_sectors(rq) * bfq_async_charge_factor;1051}1052 1053/**1054 * bfq_updated_next_req - update the queue after a new next_rq selection.1055 * @bfqd: the device data the queue belongs to.1056 * @bfqq: the queue to update.1057 *1058 * If the first request of a queue changes we make sure that the queue1059 * has enough budget to serve at least its first request (if the1060 * request has grown).  We do this because if the queue has not enough1061 * budget for its first request, it has to go through two dispatch1062 * rounds to actually get it dispatched.1063 */1064static void bfq_updated_next_req(struct bfq_data *bfqd,1065				 struct bfq_queue *bfqq)1066{1067	struct bfq_entity *entity = &bfqq->entity;1068	struct request *next_rq = bfqq->next_rq;1069	unsigned long new_budget;1070 1071	if (!next_rq)1072		return;1073 1074	if (bfqq == bfqd->in_service_queue)1075		/*1076		 * In order not to break guarantees, budgets cannot be1077		 * changed after an entity has been selected.1078		 */1079		return;1080 1081	new_budget = max_t(unsigned long,1082			   max_t(unsigned long, bfqq->max_budget,1083				 bfq_serv_to_charge(next_rq, bfqq)),1084			   entity->service);1085	if (entity->budget != new_budget) {1086		entity->budget = new_budget;1087		bfq_log_bfqq(bfqd, bfqq, "updated next rq: new budget %lu",1088					 new_budget);1089		bfq_requeue_bfqq(bfqd, bfqq, false);1090	}1091}1092 1093static unsigned int bfq_wr_duration(struct bfq_data *bfqd)1094{1095	u64 dur;1096 1097	dur = bfqd->rate_dur_prod;1098	do_div(dur, bfqd->peak_rate);1099 1100	/*1101	 * Limit duration between 3 and 25 seconds. The upper limit1102	 * has been conservatively set after the following worst case:1103	 * on a QEMU/KVM virtual machine1104	 * - running in a slow PC1105	 * - with a virtual disk stacked on a slow low-end 5400rpm HDD1106	 * - serving a heavy I/O workload, such as the sequential reading1107	 *   of several files1108	 * mplayer took 23 seconds to start, if constantly weight-raised.1109	 *1110	 * As for higher values than that accommodating the above bad1111	 * scenario, tests show that higher values would often yield1112	 * the opposite of the desired result, i.e., would worsen1113	 * responsiveness by allowing non-interactive applications to1114	 * preserve weight raising for too long.1115	 *1116	 * On the other end, lower values than 3 seconds make it1117	 * difficult for most interactive tasks to complete their jobs1118	 * before weight-raising finishes.1119	 */1120	return clamp_val(dur, msecs_to_jiffies(3000), msecs_to_jiffies(25000));1121}1122 1123/* switch back from soft real-time to interactive weight raising */1124static void switch_back_to_interactive_wr(struct bfq_queue *bfqq,1125					  struct bfq_data *bfqd)1126{1127	bfqq->wr_coeff = bfqd->bfq_wr_coeff;1128	bfqq->wr_cur_max_time = bfq_wr_duration(bfqd);1129	bfqq->last_wr_start_finish = bfqq->wr_start_at_switch_to_srt;1130}1131 1132static void1133bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd,1134		      struct bfq_io_cq *bic, bool bfq_already_existing)1135{1136	unsigned int old_wr_coeff = 1;1137	bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq);1138	unsigned int a_idx = bfqq->actuator_idx;1139	struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx];1140 1141	if (bfqq_data->saved_has_short_ttime)1142		bfq_mark_bfqq_has_short_ttime(bfqq);1143	else1144		bfq_clear_bfqq_has_short_ttime(bfqq);1145 1146	if (bfqq_data->saved_IO_bound)1147		bfq_mark_bfqq_IO_bound(bfqq);1148	else1149		bfq_clear_bfqq_IO_bound(bfqq);1150 1151	bfqq->last_serv_time_ns = bfqq_data->saved_last_serv_time_ns;1152	bfqq->inject_limit = bfqq_data->saved_inject_limit;1153	bfqq->decrease_time_jif = bfqq_data->saved_decrease_time_jif;1154 1155	bfqq->entity.new_weight = bfqq_data->saved_weight;1156	bfqq->ttime = bfqq_data->saved_ttime;1157	bfqq->io_start_time = bfqq_data->saved_io_start_time;1158	bfqq->tot_idle_time = bfqq_data->saved_tot_idle_time;1159	/*1160	 * Restore weight coefficient only if low_latency is on1161	 */1162	if (bfqd->low_latency) {1163		old_wr_coeff = bfqq->wr_coeff;1164		bfqq->wr_coeff = bfqq_data->saved_wr_coeff;1165	}1166	bfqq->service_from_wr = bfqq_data->saved_service_from_wr;1167	bfqq->wr_start_at_switch_to_srt =1168		bfqq_data->saved_wr_start_at_switch_to_srt;1169	bfqq->last_wr_start_finish = bfqq_data->saved_last_wr_start_finish;1170	bfqq->wr_cur_max_time = bfqq_data->saved_wr_cur_max_time;1171 1172	if (bfqq->wr_coeff > 1 && (bfq_bfqq_in_large_burst(bfqq) ||1173	    time_is_before_jiffies(bfqq->last_wr_start_finish +1174				   bfqq->wr_cur_max_time))) {1175		if (bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time &&1176		    !bfq_bfqq_in_large_burst(bfqq) &&1177		    time_is_after_eq_jiffies(bfqq->wr_start_at_switch_to_srt +1178					     bfq_wr_duration(bfqd))) {1179			switch_back_to_interactive_wr(bfqq, bfqd);1180		} else {1181			bfqq->wr_coeff = 1;1182			bfq_log_bfqq(bfqq->bfqd, bfqq,1183				     "resume state: switching off wr");1184		}1185	}1186 1187	/* make sure weight will be updated, however we got here */1188	bfqq->entity.prio_changed = 1;1189 1190	if (likely(!busy))1191		return;1192 1193	if (old_wr_coeff == 1 && bfqq->wr_coeff > 1)1194		bfqd->wr_busy_queues++;1195	else if (old_wr_coeff > 1 && bfqq->wr_coeff == 1)1196		bfqd->wr_busy_queues--;1197}1198 1199static int bfqq_process_refs(struct bfq_queue *bfqq)1200{1201	return bfqq->ref - bfqq->entity.allocated -1202		bfqq->entity.on_st_or_in_serv -1203		(bfqq->weight_counter != NULL) - bfqq->stable_ref;1204}1205 1206/* Empty burst list and add just bfqq (see comments on bfq_handle_burst) */1207static void bfq_reset_burst_list(struct bfq_data *bfqd, struct bfq_queue *bfqq)1208{1209	struct bfq_queue *item;1210	struct hlist_node *n;1211 1212	hlist_for_each_entry_safe(item, n, &bfqd->burst_list, burst_list_node)1213		hlist_del_init(&item->burst_list_node);1214 1215	/*1216	 * Start the creation of a new burst list only if there is no1217	 * active queue. See comments on the conditional invocation of1218	 * bfq_handle_burst().1219	 */1220	if (bfq_tot_busy_queues(bfqd) == 0) {1221		hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list);1222		bfqd->burst_size = 1;1223	} else1224		bfqd->burst_size = 0;1225 1226	bfqd->burst_parent_entity = bfqq->entity.parent;1227}1228 1229/* Add bfqq to the list of queues in current burst (see bfq_handle_burst) */1230static void bfq_add_to_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq)1231{1232	/* Increment burst size to take into account also bfqq */1233	bfqd->burst_size++;1234 1235	if (bfqd->burst_size == bfqd->bfq_large_burst_thresh) {1236		struct bfq_queue *pos, *bfqq_item;1237		struct hlist_node *n;1238 1239		/*1240		 * Enough queues have been activated shortly after each1241		 * other to consider this burst as large.1242		 */1243		bfqd->large_burst = true;1244 1245		/*1246		 * We can now mark all queues in the burst list as1247		 * belonging to a large burst.1248		 */1249		hlist_for_each_entry(bfqq_item, &bfqd->burst_list,1250				     burst_list_node)1251			bfq_mark_bfqq_in_large_burst(bfqq_item);1252		bfq_mark_bfqq_in_large_burst(bfqq);1253 1254		/*1255		 * From now on, and until the current burst finishes, any1256		 * new queue being activated shortly after the last queue1257		 * was inserted in the burst can be immediately marked as1258		 * belonging to a large burst. So the burst list is not1259		 * needed any more. Remove it.1260		 */1261		hlist_for_each_entry_safe(pos, n, &bfqd->burst_list,1262					  burst_list_node)1263			hlist_del_init(&pos->burst_list_node);1264	} else /*1265		* Burst not yet large: add bfqq to the burst list. Do1266		* not increment the ref counter for bfqq, because bfqq1267		* is removed from the burst list before freeing bfqq1268		* in put_queue.1269		*/1270		hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list);1271}1272 1273/*1274 * If many queues belonging to the same group happen to be created1275 * shortly after each other, then the processes associated with these1276 * queues have typically a common goal. In particular, bursts of queue1277 * creations are usually caused by services or applications that spawn1278 * many parallel threads/processes. Examples are systemd during boot,1279 * or git grep. To help these processes get their job done as soon as1280 * possible, it is usually better to not grant either weight-raising1281 * or device idling to their queues, unless these queues must be1282 * protected from the I/O flowing through other active queues.1283 *1284 * In this comment we describe, firstly, the reasons why this fact1285 * holds, and, secondly, the next function, which implements the main1286 * steps needed to properly mark these queues so that they can then be1287 * treated in a different way.1288 *1289 * The above services or applications benefit mostly from a high1290 * throughput: the quicker the requests of the activated queues are1291 * cumulatively served, the sooner the target job of these queues gets1292 * completed. As a consequence, weight-raising any of these queues,1293 * which also implies idling the device for it, is almost always1294 * counterproductive, unless there are other active queues to isolate1295 * these new queues from. If there no other active queues, then1296 * weight-raising these new queues just lowers throughput in most1297 * cases.1298 *1299 * On the other hand, a burst of queue creations may be caused also by1300 * the start of an application that does not consist of a lot of1301 * parallel I/O-bound threads. In fact, with a complex application,1302 * several short processes may need to be executed to start-up the1303 * application. In this respect, to start an application as quickly as1304 * possible, the best thing to do is in any case to privilege the I/O1305 * related to the application with respect to all other1306 * I/O. Therefore, the best strategy to start as quickly as possible1307 * an application that causes a burst of queue creations is to1308 * weight-raise all the queues created during the burst. This is the1309 * exact opposite of the best strategy for the other type of bursts.1310 *1311 * In the end, to take the best action for each of the two cases, the1312 * two types of bursts need to be distinguished. Fortunately, this1313 * seems relatively easy, by looking at the sizes of the bursts. In1314 * particular, we found a threshold such that only bursts with a1315 * larger size than that threshold are apparently caused by1316 * services or commands such as systemd or git grep. For brevity,1317 * hereafter we call just 'large' these bursts. BFQ *does not*1318 * weight-raise queues whose creation occurs in a large burst. In1319 * addition, for each of these queues BFQ performs or does not perform1320 * idling depending on which choice boosts the throughput more. The1321 * exact choice depends on the device and request pattern at1322 * hand.1323 *1324 * Unfortunately, false positives may occur while an interactive task1325 * is starting (e.g., an application is being started). The1326 * consequence is that the queues associated with the task do not1327 * enjoy weight raising as expected. Fortunately these false positives1328 * are very rare. They typically occur if some service happens to1329 * start doing I/O exactly when the interactive task starts.1330 *1331 * Turning back to the next function, it is invoked only if there are1332 * no active queues (apart from active queues that would belong to the1333 * same, possible burst bfqq would belong to), and it implements all1334 * the steps needed to detect the occurrence of a large burst and to1335 * properly mark all the queues belonging to it (so that they can then1336 * be treated in a different way). This goal is achieved by1337 * maintaining a "burst list" that holds, temporarily, the queues that1338 * belong to the burst in progress. The list is then used to mark1339 * these queues as belonging to a large burst if the burst does become1340 * large. The main steps are the following.1341 *1342 * . when the very first queue is created, the queue is inserted into the1343 *   list (as it could be the first queue in a possible burst)1344 *1345 * . if the current burst has not yet become large, and a queue Q that does1346 *   not yet belong to the burst is activated shortly after the last time1347 *   at which a new queue entered the burst list, then the function appends1348 *   Q to the burst list1349 *1350 * . if, as a consequence of the previous step, the burst size reaches1351 *   the large-burst threshold, then1352 *1353 *     . all the queues in the burst list are marked as belonging to a1354 *       large burst1355 *1356 *     . the burst list is deleted; in fact, the burst list already served1357 *       its purpose (keeping temporarily track of the queues in a burst,1358 *       so as to be able to mark them as belonging to a large burst in the1359 *       previous sub-step), and now is not needed any more1360 *1361 *     . the device enters a large-burst mode1362 *1363 * . if a queue Q that does not belong to the burst is created while1364 *   the device is in large-burst mode and shortly after the last time1365 *   at which a queue either entered the burst list or was marked as1366 *   belonging to the current large burst, then Q is immediately marked1367 *   as belonging to a large burst.1368 *1369 * . if a queue Q that does not belong to the burst is created a while1370 *   later, i.e., not shortly after, than the last time at which a queue1371 *   either entered the burst list or was marked as belonging to the1372 *   current large burst, then the current burst is deemed as finished and:1373 *1374 *        . the large-burst mode is reset if set1375 *1376 *        . the burst list is emptied1377 *1378 *        . Q is inserted in the burst list, as Q may be the first queue1379 *          in a possible new burst (then the burst list contains just Q1380 *          after this step).1381 */1382static void bfq_handle_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq)1383{1384	/*1385	 * If bfqq is already in the burst list or is part of a large1386	 * burst, or finally has just been split, then there is1387	 * nothing else to do.1388	 */1389	if (!hlist_unhashed(&bfqq->burst_list_node) ||1390	    bfq_bfqq_in_large_burst(bfqq) ||1391	    time_is_after_eq_jiffies(bfqq->split_time +1392				     msecs_to_jiffies(10)))1393		return;1394 1395	/*1396	 * If bfqq's creation happens late enough, or bfqq belongs to1397	 * a different group than the burst group, then the current1398	 * burst is finished, and related data structures must be1399	 * reset.1400	 *1401	 * In this respect, consider the special case where bfqq is1402	 * the very first queue created after BFQ is selected for this1403	 * device. In this case, last_ins_in_burst and1404	 * burst_parent_entity are not yet significant when we get1405	 * here. But it is easy to verify that, whether or not the1406	 * following condition is true, bfqq will end up being1407	 * inserted into the burst list. In particular the list will1408	 * happen to contain only bfqq. And this is exactly what has1409	 * to happen, as bfqq may be the first queue of the first1410	 * burst.1411	 */1412	if (time_is_before_jiffies(bfqd->last_ins_in_burst +1413	    bfqd->bfq_burst_interval) ||1414	    bfqq->entity.parent != bfqd->burst_parent_entity) {1415		bfqd->large_burst = false;1416		bfq_reset_burst_list(bfqd, bfqq);1417		goto end;1418	}1419 1420	/*1421	 * If we get here, then bfqq is being activated shortly after the1422	 * last queue. So, if the current burst is also large, we can mark1423	 * bfqq as belonging to this large burst immediately.1424	 */1425	if (bfqd->large_burst) {1426		bfq_mark_bfqq_in_large_burst(bfqq);1427		goto end;1428	}1429 1430	/*1431	 * If we get here, then a large-burst state has not yet been1432	 * reached, but bfqq is being activated shortly after the last1433	 * queue. Then we add bfqq to the burst.1434	 */1435	bfq_add_to_burst(bfqd, bfqq);1436end:1437	/*1438	 * At this point, bfqq either has been added to the current1439	 * burst or has caused the current burst to terminate and a1440	 * possible new burst to start. In particular, in the second1441	 * case, bfqq has become the first queue in the possible new1442	 * burst.  In both cases last_ins_in_burst needs to be moved1443	 * forward.1444	 */1445	bfqd->last_ins_in_burst = jiffies;1446}1447 1448static int bfq_bfqq_budget_left(struct bfq_queue *bfqq)1449{1450	struct bfq_entity *entity = &bfqq->entity;1451 1452	return entity->budget - entity->service;1453}1454 1455/*1456 * If enough samples have been computed, return the current max budget1457 * stored in bfqd, which is dynamically updated according to the1458 * estimated disk peak rate; otherwise return the default max budget1459 */1460static int bfq_max_budget(struct bfq_data *bfqd)1461{1462	if (bfqd->budgets_assigned < bfq_stats_min_budgets)1463		return bfq_default_max_budget;1464	else1465		return bfqd->bfq_max_budget;1466}1467 1468/*1469 * Return min budget, which is a fraction of the current or default1470 * max budget (trying with 1/32)1471 */1472static int bfq_min_budget(struct bfq_data *bfqd)1473{1474	if (bfqd->budgets_assigned < bfq_stats_min_budgets)1475		return bfq_default_max_budget / 32;1476	else1477		return bfqd->bfq_max_budget / 32;1478}1479 1480/*1481 * The next function, invoked after the input queue bfqq switches from1482 * idle to busy, updates the budget of bfqq. The function also tells1483 * whether the in-service queue should be expired, by returning1484 * true. The purpose of expiring the in-service queue is to give bfqq1485 * the chance to possibly preempt the in-service queue, and the reason1486 * for preempting the in-service queue is to achieve one of the two1487 * goals below.1488 *1489 * 1. Guarantee to bfqq its reserved bandwidth even if bfqq has1490 * expired because it has remained idle. In particular, bfqq may have1491 * expired for one of the following two reasons:1492 *1493 * - BFQQE_NO_MORE_REQUESTS bfqq did not enjoy any device idling1494 *   and did not make it to issue a new request before its last1495 *   request was served;1496 *1497 * - BFQQE_TOO_IDLE bfqq did enjoy device idling, but did not issue1498 *   a new request before the expiration of the idling-time.1499 *1500 * Even if bfqq has expired for one of the above reasons, the process1501 * associated with the queue may be however issuing requests greedily,1502 * and thus be sensitive to the bandwidth it receives (bfqq may have1503 * remained idle for other reasons: CPU high load, bfqq not enjoying1504 * idling, I/O throttling somewhere in the path from the process to1505 * the I/O scheduler, ...). But if, after every expiration for one of1506 * the above two reasons, bfqq has to wait for the service of at least1507 * one full budget of another queue before being served again, then1508 * bfqq is likely to get a much lower bandwidth or resource time than1509 * its reserved ones. To address this issue, two countermeasures need1510 * to be taken.1511 *1512 * First, the budget and the timestamps of bfqq need to be updated in1513 * a special way on bfqq reactivation: they need to be updated as if1514 * bfqq did not remain idle and did not expire. In fact, if they are1515 * computed as if bfqq expired and remained idle until reactivation,1516 * then the process associated with bfqq is treated as if, instead of1517 * being greedy, it stopped issuing requests when bfqq remained idle,1518 * and restarts issuing requests only on this reactivation. In other1519 * words, the scheduler does not help the process recover the "service1520 * hole" between bfqq expiration and reactivation. As a consequence,1521 * the process receives a lower bandwidth than its reserved one. In1522 * contrast, to recover this hole, the budget must be updated as if1523 * bfqq was not expired at all before this reactivation, i.e., it must1524 * be set to the value of the remaining budget when bfqq was1525 * expired. Along the same line, timestamps need to be assigned the1526 * value they had the last time bfqq was selected for service, i.e.,1527 * before last expiration. Thus timestamps need to be back-shifted1528 * with respect to their normal computation (see [1] for more details1529 * on this tricky aspect).1530 *1531 * Secondly, to allow the process to recover the hole, the in-service1532 * queue must be expired too, to give bfqq the chance to preempt it1533 * immediately. In fact, if bfqq has to wait for a full budget of the1534 * in-service queue to be completed, then it may become impossible to1535 * let the process recover the hole, even if the back-shifted1536 * timestamps of bfqq are lower than those of the in-service queue. If1537 * this happens for most or all of the holes, then the process may not1538 * receive its reserved bandwidth. In this respect, it is worth noting1539 * that, being the service of outstanding requests unpreemptible, a1540 * little fraction of the holes may however be unrecoverable, thereby1541 * causing a little loss of bandwidth.1542 *1543 * The last important point is detecting whether bfqq does need this1544 * bandwidth recovery. In this respect, the next function deems the1545 * process associated with bfqq greedy, and thus allows it to recover1546 * the hole, if: 1) the process is waiting for the arrival of a new1547 * request (which implies that bfqq expired for one of the above two1548 * reasons), and 2) such a request has arrived soon. The first1549 * condition is controlled through the flag non_blocking_wait_rq,1550 * while the second through the flag arrived_in_time. If both1551 * conditions hold, then the function computes the budget in the1552 * above-described special way, and signals that the in-service queue1553 * should be expired. Timestamp back-shifting is done later in1554 * __bfq_activate_entity.1555 *1556 * 2. Reduce latency. Even if timestamps are not backshifted to let1557 * the process associated with bfqq recover a service hole, bfqq may1558 * however happen to have, after being (re)activated, a lower finish1559 * timestamp than the in-service queue.	 That is, the next budget of1560 * bfqq may have to be completed before the one of the in-service1561 * queue. If this is the case, then preempting the in-service queue1562 * allows this goal to be achieved, apart from the unpreemptible,1563 * outstanding requests mentioned above.1564 *1565 * Unfortunately, regardless of which of the above two goals one wants1566 * to achieve, service trees need first to be updated to know whether1567 * the in-service queue must be preempted. To have service trees1568 * correctly updated, the in-service queue must be expired and1569 * rescheduled, and bfqq must be scheduled too. This is one of the1570 * most costly operations (in future versions, the scheduling1571 * mechanism may be re-designed in such a way to make it possible to1572 * know whether preemption is needed without needing to update service1573 * trees). In addition, queue preemptions almost always cause random1574 * I/O, which may in turn cause loss of throughput. Finally, there may1575 * even be no in-service queue when the next function is invoked (so,1576 * no queue to compare timestamps with). Because of these facts, the1577 * next function adopts the following simple scheme to avoid costly1578 * operations, too frequent preemptions and too many dependencies on1579 * the state of the scheduler: it requests the expiration of the1580 * in-service queue (unconditionally) only for queues that need to1581 * recover a hole. Then it delegates to other parts of the code the1582 * responsibility of handling the above case 2.1583 */1584static bool bfq_bfqq_update_budg_for_activation(struct bfq_data *bfqd,1585						struct bfq_queue *bfqq,1586						bool arrived_in_time)1587{1588	struct bfq_entity *entity = &bfqq->entity;1589 1590	/*1591	 * In the next compound condition, we check also whether there1592	 * is some budget left, because otherwise there is no point in1593	 * trying to go on serving bfqq with this same budget: bfqq1594	 * would be expired immediately after being selected for1595	 * service. This would only cause useless overhead.1596	 */1597	if (bfq_bfqq_non_blocking_wait_rq(bfqq) && arrived_in_time &&1598	    bfq_bfqq_budget_left(bfqq) > 0) {1599		/*1600		 * We do not clear the flag non_blocking_wait_rq here, as1601		 * the latter is used in bfq_activate_bfqq to signal1602		 * that timestamps need to be back-shifted (and is1603		 * cleared right after).1604		 */1605 1606		/*1607		 * In next assignment we rely on that either1608		 * entity->service or entity->budget are not updated1609		 * on expiration if bfqq is empty (see1610		 * __bfq_bfqq_recalc_budget). Thus both quantities1611		 * remain unchanged after such an expiration, and the1612		 * following statement therefore assigns to1613		 * entity->budget the remaining budget on such an1614		 * expiration.1615		 */1616		entity->budget = min_t(unsigned long,1617				       bfq_bfqq_budget_left(bfqq),1618				       bfqq->max_budget);1619 1620		/*1621		 * At this point, we have used entity->service to get1622		 * the budget left (needed for updating1623		 * entity->budget). Thus we finally can, and have to,1624		 * reset entity->service. The latter must be reset1625		 * because bfqq would otherwise be charged again for1626		 * the service it has received during its previous1627		 * service slot(s).1628		 */1629		entity->service = 0;1630 1631		return true;1632	}1633 1634	/*1635	 * We can finally complete expiration, by setting service to 0.1636	 */1637	entity->service = 0;1638	entity->budget = max_t(unsigned long, bfqq->max_budget,1639			       bfq_serv_to_charge(bfqq->next_rq, bfqq));1640	bfq_clear_bfqq_non_blocking_wait_rq(bfqq);1641	return false;1642}1643 1644/*1645 * Return the farthest past time instant according to jiffies1646 * macros.1647 */1648static unsigned long bfq_smallest_from_now(void)1649{1650	return jiffies - MAX_JIFFY_OFFSET;1651}1652 1653static void bfq_update_bfqq_wr_on_rq_arrival(struct bfq_data *bfqd,1654					     struct bfq_queue *bfqq,1655					     unsigned int old_wr_coeff,1656					     bool wr_or_deserves_wr,1657					     bool interactive,1658					     bool in_burst,1659					     bool soft_rt)1660{1661	if (old_wr_coeff == 1 && wr_or_deserves_wr) {1662		/* start a weight-raising period */1663		if (interactive) {1664			bfqq->service_from_wr = 0;1665			bfqq->wr_coeff = bfqd->bfq_wr_coeff;1666			bfqq->wr_cur_max_time = bfq_wr_duration(bfqd);1667		} else {1668			/*1669			 * No interactive weight raising in progress1670			 * here: assign minus infinity to1671			 * wr_start_at_switch_to_srt, to make sure1672			 * that, at the end of the soft-real-time1673			 * weight raising periods that is starting1674			 * now, no interactive weight-raising period1675			 * may be wrongly considered as still in1676			 * progress (and thus actually started by1677			 * mistake).1678			 */1679			bfqq->wr_start_at_switch_to_srt =1680				bfq_smallest_from_now();1681			bfqq->wr_coeff = bfqd->bfq_wr_coeff *1682				BFQ_SOFTRT_WEIGHT_FACTOR;1683			bfqq->wr_cur_max_time =1684				bfqd->bfq_wr_rt_max_time;1685		}1686 1687		/*1688		 * If needed, further reduce budget to make sure it is1689		 * close to bfqq's backlog, so as to reduce the1690		 * scheduling-error component due to a too large1691		 * budget. Do not care about throughput consequences,1692		 * but only about latency. Finally, do not assign a1693		 * too small budget either, to avoid increasing1694		 * latency by causing too frequent expirations.1695		 */1696		bfqq->entity.budget = min_t(unsigned long,1697					    bfqq->entity.budget,1698					    2 * bfq_min_budget(bfqd));1699	} else if (old_wr_coeff > 1) {1700		if (interactive) { /* update wr coeff and duration */1701			bfqq->wr_coeff = bfqd->bfq_wr_coeff;1702			bfqq->wr_cur_max_time = bfq_wr_duration(bfqd);1703		} else if (in_burst)1704			bfqq->wr_coeff = 1;1705		else if (soft_rt) {1706			/*1707			 * The application is now or still meeting the1708			 * requirements for being deemed soft rt.  We1709			 * can then correctly and safely (re)charge1710			 * the weight-raising duration for the1711			 * application with the weight-raising1712			 * duration for soft rt applications.1713			 *1714			 * In particular, doing this recharge now, i.e.,1715			 * before the weight-raising period for the1716			 * application finishes, reduces the probability1717			 * of the following negative scenario:1718			 * 1) the weight of a soft rt application is1719			 *    raised at startup (as for any newly1720			 *    created application),1721			 * 2) since the application is not interactive,1722			 *    at a certain time weight-raising is1723			 *    stopped for the application,1724			 * 3) at that time the application happens to1725			 *    still have pending requests, and hence1726			 *    is destined to not have a chance to be1727			 *    deemed soft rt before these requests are1728			 *    completed (see the comments to the1729			 *    function bfq_bfqq_softrt_next_start()1730			 *    for details on soft rt detection),1731			 * 4) these pending requests experience a high1732			 *    latency because the application is not1733			 *    weight-raised while they are pending.1734			 */1735			if (bfqq->wr_cur_max_time !=1736				bfqd->bfq_wr_rt_max_time) {1737				bfqq->wr_start_at_switch_to_srt =1738					bfqq->last_wr_start_finish;1739 1740				bfqq->wr_cur_max_time =1741					bfqd->bfq_wr_rt_max_time;1742				bfqq->wr_coeff = bfqd->bfq_wr_coeff *1743					BFQ_SOFTRT_WEIGHT_FACTOR;1744			}1745			bfqq->last_wr_start_finish = jiffies;1746		}1747	}1748}1749 1750static bool bfq_bfqq_idle_for_long_time(struct bfq_data *bfqd,1751					struct bfq_queue *bfqq)1752{1753	return bfqq->dispatched == 0 &&1754		time_is_before_jiffies(1755			bfqq->budget_timeout +1756			bfqd->bfq_wr_min_idle_time);1757}1758 1759 1760/*1761 * Return true if bfqq is in a higher priority class, or has a higher1762 * weight than the in-service queue.1763 */1764static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq,1765					    struct bfq_queue *in_serv_bfqq)1766{1767	int bfqq_weight, in_serv_weight;1768 1769	if (bfqq->ioprio_class < in_serv_bfqq->ioprio_class)1770		return true;1771 1772	if (in_serv_bfqq->entity.parent == bfqq->entity.parent) {1773		bfqq_weight = bfqq->entity.weight;1774		in_serv_weight = in_serv_bfqq->entity.weight;1775	} else {1776		if (bfqq->entity.parent)1777			bfqq_weight = bfqq->entity.parent->weight;1778		else1779			bfqq_weight = bfqq->entity.weight;1780		if (in_serv_bfqq->entity.parent)1781			in_serv_weight = in_serv_bfqq->entity.parent->weight;1782		else1783			in_serv_weight = in_serv_bfqq->entity.weight;1784	}1785 1786	return bfqq_weight > in_serv_weight;1787}1788 1789/*1790 * Get the index of the actuator that will serve bio.1791 */1792static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio)1793{1794	unsigned int i;1795	sector_t end;1796 1797	/* no search needed if one or zero ranges present */1798	if (bfqd->num_actuators == 1)1799		return 0;1800 1801	/* bio_end_sector(bio) gives the sector after the last one */1802	end = bio_end_sector(bio) - 1;1803 1804	for (i = 0; i < bfqd->num_actuators; i++) {1805		if (end >= bfqd->sector[i] &&1806		    end < bfqd->sector[i] + bfqd->nr_sectors[i])1807			return i;1808	}1809 1810	WARN_ONCE(true,1811		  "bfq_actuator_index: bio sector out of ranges: end=%llu\n",1812		  end);1813	return 0;1814}1815 1816static bool bfq_better_to_idle(struct bfq_queue *bfqq);1817 1818static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd,1819					     struct bfq_queue *bfqq,1820					     int old_wr_coeff,1821					     struct request *rq,1822					     bool *interactive)1823{1824	bool soft_rt, in_burst,	wr_or_deserves_wr,1825		bfqq_wants_to_preempt,1826		idle_for_long_time = bfq_bfqq_idle_for_long_time(bfqd, bfqq),1827		/*1828		 * See the comments on1829		 * bfq_bfqq_update_budg_for_activation for1830		 * details on the usage of the next variable.1831		 */1832		arrived_in_time =  blk_time_get_ns() <=1833			bfqq->ttime.last_end_request +1834			bfqd->bfq_slice_idle * 3;1835	unsigned int act_idx = bfq_actuator_index(bfqd, rq->bio);1836	bool bfqq_non_merged_or_stably_merged =1837		bfqq->bic || RQ_BIC(rq)->bfqq_data[act_idx].stably_merged;1838 1839	/*1840	 * bfqq deserves to be weight-raised if:1841	 * - it is sync,1842	 * - it does not belong to a large burst,1843	 * - it has been idle for enough time or is soft real-time,1844	 * - is linked to a bfq_io_cq (it is not shared in any sense),1845	 * - has a default weight (otherwise we assume the user wanted1846	 *   to control its weight explicitly)1847	 */1848	in_burst = bfq_bfqq_in_large_burst(bfqq);1849	soft_rt = bfqd->bfq_wr_max_softrt_rate > 0 &&1850		!BFQQ_TOTALLY_SEEKY(bfqq) &&1851		!in_burst &&1852		time_is_before_jiffies(bfqq->soft_rt_next_start) &&1853		bfqq->dispatched == 0 &&1854		bfqq->entity.new_weight == 40;1855	*interactive = !in_burst && idle_for_long_time &&1856		bfqq->entity.new_weight == 40;1857	/*1858	 * Merged bfq_queues are kept out of weight-raising1859	 * (low-latency) mechanisms. The reason is that these queues1860	 * are usually created for non-interactive and1861	 * non-soft-real-time tasks. Yet this is not the case for1862	 * stably-merged queues. These queues are merged just because1863	 * they are created shortly after each other. So they may1864	 * easily serve the I/O of an interactive or soft-real time1865	 * application, if the application happens to spawn multiple1866	 * processes. So let also stably-merged queued enjoy weight1867	 * raising.1868	 */1869	wr_or_deserves_wr = bfqd->low_latency &&1870		(bfqq->wr_coeff > 1 ||1871		 (bfq_bfqq_sync(bfqq) && bfqq_non_merged_or_stably_merged &&1872		  (*interactive || soft_rt)));1873 1874	/*1875	 * Using the last flag, update budget and check whether bfqq1876	 * may want to preempt the in-service queue.1877	 */1878	bfqq_wants_to_preempt =1879		bfq_bfqq_update_budg_for_activation(bfqd, bfqq,1880						    arrived_in_time);1881 1882	/*1883	 * If bfqq happened to be activated in a burst, but has been1884	 * idle for much more than an interactive queue, then we1885	 * assume that, in the overall I/O initiated in the burst, the1886	 * I/O associated with bfqq is finished. So bfqq does not need1887	 * to be treated as a queue belonging to a burst1888	 * anymore. Accordingly, we reset bfqq's in_large_burst flag1889	 * if set, and remove bfqq from the burst list if it's1890	 * there. We do not decrement burst_size, because the fact1891	 * that bfqq does not need to belong to the burst list any1892	 * more does not invalidate the fact that bfqq was created in1893	 * a burst.1894	 */1895	if (likely(!bfq_bfqq_just_created(bfqq)) &&1896	    idle_for_long_time &&1897	    time_is_before_jiffies(1898		    bfqq->budget_timeout +1899		    msecs_to_jiffies(10000))) {1900		hlist_del_init(&bfqq->burst_list_node);1901		bfq_clear_bfqq_in_large_burst(bfqq);1902	}1903 1904	bfq_clear_bfqq_just_created(bfqq);1905 1906	if (bfqd->low_latency) {1907		if (unlikely(time_is_after_jiffies(bfqq->split_time)))1908			/* wraparound */1909			bfqq->split_time =1910				jiffies - bfqd->bfq_wr_min_idle_time - 1;1911 1912		if (time_is_before_jiffies(bfqq->split_time +1913					   bfqd->bfq_wr_min_idle_time)) {1914			bfq_update_bfqq_wr_on_rq_arrival(bfqd, bfqq,1915							 old_wr_coeff,1916							 wr_or_deserves_wr,1917							 *interactive,1918							 in_burst,1919							 soft_rt);1920 1921			if (old_wr_coeff != bfqq->wr_coeff)1922				bfqq->entity.prio_changed = 1;1923		}1924	}1925 1926	bfqq->last_idle_bklogged = jiffies;1927	bfqq->service_from_backlogged = 0;1928	bfq_clear_bfqq_softrt_update(bfqq);1929 1930	bfq_add_bfqq_busy(bfqq);1931 1932	/*1933	 * Expire in-service queue if preemption may be needed for1934	 * guarantees or throughput. As for guarantees, we care1935	 * explicitly about two cases. The first is that bfqq has to1936	 * recover a service hole, as explained in the comments on1937	 * bfq_bfqq_update_budg_for_activation(), i.e., that1938	 * bfqq_wants_to_preempt is true. However, if bfqq does not1939	 * carry time-critical I/O, then bfqq's bandwidth is less1940	 * important than that of queues that carry time-critical I/O.1941	 * So, as a further constraint, we consider this case only if1942	 * bfqq is at least as weight-raised, i.e., at least as time1943	 * critical, as the in-service queue.1944	 *1945	 * The second case is that bfqq is in a higher priority class,1946	 * or has a higher weight than the in-service queue. If this1947	 * condition does not hold, we don't care because, even if1948	 * bfqq does not start to be served immediately, the resulting1949	 * delay for bfqq's I/O is however lower or much lower than1950	 * the ideal completion time to be guaranteed to bfqq's I/O.1951	 *1952	 * In both cases, preemption is needed only if, according to1953	 * the timestamps of both bfqq and of the in-service queue,1954	 * bfqq actually is the next queue to serve. So, to reduce1955	 * useless preemptions, the return value of1956	 * next_queue_may_preempt() is considered in the next compound1957	 * condition too. Yet next_queue_may_preempt() just checks a1958	 * simple, necessary condition for bfqq to be the next queue1959	 * to serve. In fact, to evaluate a sufficient condition, the1960	 * timestamps of the in-service queue would need to be1961	 * updated, and this operation is quite costly (see the1962	 * comments on bfq_bfqq_update_budg_for_activation()).1963	 *1964	 * As for throughput, we ask bfq_better_to_idle() whether we1965	 * still need to plug I/O dispatching. If bfq_better_to_idle()1966	 * says no, then plugging is not needed any longer, either to1967	 * boost throughput or to perserve service guarantees. Then1968	 * the best option is to stop plugging I/O, as not doing so1969	 * would certainly lower throughput. We may end up in this1970	 * case if: (1) upon a dispatch attempt, we detected that it1971	 * was better to plug I/O dispatch, and to wait for a new1972	 * request to arrive for the currently in-service queue, but1973	 * (2) this switch of bfqq to busy changes the scenario.1974	 */1975	if (bfqd->in_service_queue &&1976	    ((bfqq_wants_to_preempt &&1977	      bfqq->wr_coeff >= bfqd->in_service_queue->wr_coeff) ||1978	     bfq_bfqq_higher_class_or_weight(bfqq, bfqd->in_service_queue) ||1979	     !bfq_better_to_idle(bfqd->in_service_queue)) &&1980	    next_queue_may_preempt(bfqd))1981		bfq_bfqq_expire(bfqd, bfqd->in_service_queue,1982				false, BFQQE_PREEMPTED);1983}1984 1985static void bfq_reset_inject_limit(struct bfq_data *bfqd,1986				   struct bfq_queue *bfqq)1987{1988	/* invalidate baseline total service time */1989	bfqq->last_serv_time_ns = 0;1990 1991	/*1992	 * Reset pointer in case we are waiting for1993	 * some request completion.1994	 */1995	bfqd->waited_rq = NULL;1996 1997	/*1998	 * If bfqq has a short think time, then start by setting the1999	 * inject limit to 0 prudentially, because the service time of2000	 * an injected I/O request may be higher than the think time2001	 * of bfqq, and therefore, if one request was injected when2002	 * bfqq remains empty, this injected request might delay the2003	 * service of the next I/O request for bfqq significantly. In2004	 * case bfqq can actually tolerate some injection, then the2005	 * adaptive update will however raise the limit soon. This2006	 * lucky circumstance holds exactly because bfqq has a short2007	 * think time, and thus, after remaining empty, is likely to2008	 * get new I/O enqueued---and then completed---before being2009	 * expired. This is the very pattern that gives the2010	 * limit-update algorithm the chance to measure the effect of2011	 * injection on request service times, and then to update the2012	 * limit accordingly.2013	 *2014	 * However, in the following special case, the inject limit is2015	 * left to 1 even if the think time is short: bfqq's I/O is2016	 * synchronized with that of some other queue, i.e., bfqq may2017	 * receive new I/O only after the I/O of the other queue is2018	 * completed. Keeping the inject limit to 1 allows the2019	 * blocking I/O to be served while bfqq is in service. And2020	 * this is very convenient both for bfqq and for overall2021	 * throughput, as explained in detail in the comments in2022	 * bfq_update_has_short_ttime().2023	 *2024	 * On the opposite end, if bfqq has a long think time, then2025	 * start directly by 1, because:2026	 * a) on the bright side, keeping at most one request in2027	 * service in the drive is unlikely to cause any harm to the2028	 * latency of bfqq's requests, as the service time of a single2029	 * request is likely to be lower than the think time of bfqq;2030	 * b) on the downside, after becoming empty, bfqq is likely to2031	 * expire before getting its next request. With this request2032	 * arrival pattern, it is very hard to sample total service2033	 * times and update the inject limit accordingly (see comments2034	 * on bfq_update_inject_limit()). So the limit is likely to be2035	 * never, or at least seldom, updated.  As a consequence, by2036	 * setting the limit to 1, we avoid that no injection ever2037	 * occurs with bfqq. On the downside, this proactive step2038	 * further reduces chances to actually compute the baseline2039	 * total service time. Thus it reduces chances to execute the2040	 * limit-update algorithm and possibly raise the limit to more2041	 * than 1.2042	 */2043	if (bfq_bfqq_has_short_ttime(bfqq))2044		bfqq->inject_limit = 0;2045	else2046		bfqq->inject_limit = 1;2047 2048	bfqq->decrease_time_jif = jiffies;2049}2050 2051static void bfq_update_io_intensity(struct bfq_queue *bfqq, u64 now_ns)2052{2053	u64 tot_io_time = now_ns - bfqq->io_start_time;2054 2055	if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfqq->dispatched == 0)2056		bfqq->tot_idle_time +=2057			now_ns - bfqq->ttime.last_end_request;2058 2059	if (unlikely(bfq_bfqq_just_created(bfqq)))2060		return;2061 2062	/*2063	 * Must be busy for at least about 80% of the time to be2064	 * considered I/O bound.2065	 */2066	if (bfqq->tot_idle_time * 5 > tot_io_time)2067		bfq_clear_bfqq_IO_bound(bfqq);2068	else2069		bfq_mark_bfqq_IO_bound(bfqq);2070 2071	/*2072	 * Keep an observation window of at most 200 ms in the past2073	 * from now.2074	 */2075	if (tot_io_time > 200 * NSEC_PER_MSEC) {2076		bfqq->io_start_time = now_ns - (tot_io_time>>1);2077		bfqq->tot_idle_time >>= 1;2078	}2079}2080 2081/*2082 * Detect whether bfqq's I/O seems synchronized with that of some2083 * other queue, i.e., whether bfqq, after remaining empty, happens to2084 * receive new I/O only right after some I/O request of the other2085 * queue has been completed. We call waker queue the other queue, and2086 * we assume, for simplicity, that bfqq may have at most one waker2087 * queue.2088 *2089 * A remarkable throughput boost can be reached by unconditionally2090 * injecting the I/O of the waker queue, every time a new2091 * bfq_dispatch_request happens to be invoked while I/O is being2092 * plugged for bfqq.  In addition to boosting throughput, this2093 * unblocks bfqq's I/O, thereby improving bandwidth and latency for2094 * bfqq. Note that these same results may be achieved with the general2095 * injection mechanism, but less effectively. For details on this2096 * aspect, see the comments on the choice of the queue for injection2097 * in bfq_select_queue().2098 *2099 * Turning back to the detection of a waker queue, a queue Q is deemed as a2100 * waker queue for bfqq if, for three consecutive times, bfqq happens to become2101 * non empty right after a request of Q has been completed within given2102 * timeout. In this respect, even if bfqq is empty, we do not check for a waker2103 * if it still has some in-flight I/O. In fact, in this case bfqq is actually2104 * still being served by the drive, and may receive new I/O on the completion2105 * of some of the in-flight requests. In particular, on the first time, Q is2106 * tentatively set as a candidate waker queue, while on the third consecutive2107 * time that Q is detected, the field waker_bfqq is set to Q, to confirm that Q2108 * is a waker queue for bfqq. These detection steps are performed only if bfqq2109 * has a long think time, so as to make it more likely that bfqq's I/O is2110 * actually being blocked by a synchronization. This last filter, plus the2111 * above three-times requirement and time limit for detection, make false2112 * positives less likely.2113 *2114 * NOTE2115 *2116 * The sooner a waker queue is detected, the sooner throughput can be2117 * boosted by injecting I/O from the waker queue. Fortunately,2118 * detection is likely to be actually fast, for the following2119 * reasons. While blocked by synchronization, bfqq has a long think2120 * time. This implies that bfqq's inject limit is at least equal to 12121 * (see the comments in bfq_update_inject_limit()). So, thanks to2122 * injection, the waker queue is likely to be served during the very2123 * first I/O-plugging time interval for bfqq. This triggers the first2124 * step of the detection mechanism. Thanks again to injection, the2125 * candidate waker queue is then likely to be confirmed no later than2126 * during the next I/O-plugging interval for bfqq.2127 *2128 * ISSUE2129 *2130 * On queue merging all waker information is lost.2131 */2132static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq,2133			    u64 now_ns)2134{2135	char waker_name[MAX_BFQQ_NAME_LENGTH];2136 2137	if (!bfqd->last_completed_rq_bfqq ||2138	    bfqd->last_completed_rq_bfqq == bfqq ||2139	    bfq_bfqq_has_short_ttime(bfqq) ||2140	    now_ns - bfqd->last_completion >= 4 * NSEC_PER_MSEC ||2141	    bfqd->last_completed_rq_bfqq == &bfqd->oom_bfqq ||2142	    bfqq == &bfqd->oom_bfqq)2143		return;2144 2145	/*2146	 * We reset waker detection logic also if too much time has passed2147 	 * since the first detection. If wakeups are rare, pointless idling2148	 * doesn't hurt throughput that much. The condition below makes sure2149	 * we do not uselessly idle blocking waker in more than 1/64 cases.2150	 */2151	if (bfqd->last_completed_rq_bfqq !=2152	    bfqq->tentative_waker_bfqq ||2153	    now_ns > bfqq->waker_detection_started +2154					128 * (u64)bfqd->bfq_slice_idle) {2155		/*2156		 * First synchronization detected with a2157		 * candidate waker queue, or with a different2158		 * candidate waker queue from the current one.2159		 */2160		bfqq->tentative_waker_bfqq =2161			bfqd->last_completed_rq_bfqq;2162		bfqq->num_waker_detections = 1;2163		bfqq->waker_detection_started = now_ns;2164		bfq_bfqq_name(bfqq->tentative_waker_bfqq, waker_name,2165			      MAX_BFQQ_NAME_LENGTH);2166		bfq_log_bfqq(bfqd, bfqq, "set tentative waker %s", waker_name);2167	} else /* Same tentative waker queue detected again */2168		bfqq->num_waker_detections++;2169 2170	if (bfqq->num_waker_detections == 3) {2171		bfqq->waker_bfqq = bfqd->last_completed_rq_bfqq;2172		bfqq->tentative_waker_bfqq = NULL;2173		bfq_bfqq_name(bfqq->waker_bfqq, waker_name,2174			      MAX_BFQQ_NAME_LENGTH);2175		bfq_log_bfqq(bfqd, bfqq, "set waker %s", waker_name);2176 2177		/*2178		 * If the waker queue disappears, then2179		 * bfqq->waker_bfqq must be reset. To2180		 * this goal, we maintain in each2181		 * waker queue a list, woken_list, of2182		 * all the queues that reference the2183		 * waker queue through their2184		 * waker_bfqq pointer. When the waker2185		 * queue exits, the waker_bfqq pointer2186		 * of all the queues in the woken_list2187		 * is reset.2188		 *2189		 * In addition, if bfqq is already in2190		 * the woken_list of a waker queue,2191		 * then, before being inserted into2192		 * the woken_list of a new waker2193		 * queue, bfqq must be removed from2194		 * the woken_list of the old waker2195		 * queue.2196		 */2197		if (!hlist_unhashed(&bfqq->woken_list_node))2198			hlist_del_init(&bfqq->woken_list_node);2199		hlist_add_head(&bfqq->woken_list_node,2200			       &bfqd->last_completed_rq_bfqq->woken_list);2201	}2202}2203 2204static void bfq_add_request(struct request *rq)2205{2206	struct bfq_queue *bfqq = RQ_BFQQ(rq);2207	struct bfq_data *bfqd = bfqq->bfqd;2208	struct request *next_rq, *prev;2209	unsigned int old_wr_coeff = bfqq->wr_coeff;2210	bool interactive = false;2211	u64 now_ns = blk_time_get_ns();2212 2213	bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq));2214	bfqq->queued[rq_is_sync(rq)]++;2215	/*2216	 * Updating of 'bfqd->queued' is protected by 'bfqd->lock', however, it2217	 * may be read without holding the lock in bfq_has_work().2218	 */2219	WRITE_ONCE(bfqd->queued, bfqd->queued + 1);2220 2221	if (bfq_bfqq_sync(bfqq) && RQ_BIC(rq)->requests <= 1) {2222		bfq_check_waker(bfqd, bfqq, now_ns);2223 2224		/*2225		 * Periodically reset inject limit, to make sure that2226		 * the latter eventually drops in case workload2227		 * changes, see step (3) in the comments on2228		 * bfq_update_inject_limit().2229		 */2230		if (time_is_before_eq_jiffies(bfqq->decrease_time_jif +2231					     msecs_to_jiffies(1000)))2232			bfq_reset_inject_limit(bfqd, bfqq);2233 2234		/*2235		 * The following conditions must hold to setup a new2236		 * sampling of total service time, and then a new2237		 * update of the inject limit:2238		 * - bfqq is in service, because the total service2239		 *   time is evaluated only for the I/O requests of2240		 *   the queues in service;2241		 * - this is the right occasion to compute or to2242		 *   lower the baseline total service time, because2243		 *   there are actually no requests in the drive,2244		 *   or2245		 *   the baseline total service time is available, and2246		 *   this is the right occasion to compute the other2247		 *   quantity needed to update the inject limit, i.e.,2248		 *   the total service time caused by the amount of2249		 *   injection allowed by the current value of the2250		 *   limit. It is the right occasion because injection2251		 *   has actually been performed during the service2252		 *   hole, and there are still in-flight requests,2253		 *   which are very likely to be exactly the injected2254		 *   requests, or part of them;2255		 * - the minimum interval for sampling the total2256		 *   service time and updating the inject limit has2257		 *   elapsed.2258		 */2259		if (bfqq == bfqd->in_service_queue &&2260		    (bfqd->tot_rq_in_driver == 0 ||2261		     (bfqq->last_serv_time_ns > 0 &&2262		      bfqd->rqs_injected && bfqd->tot_rq_in_driver > 0)) &&2263		    time_is_before_eq_jiffies(bfqq->decrease_time_jif +2264					      msecs_to_jiffies(10))) {2265			bfqd->last_empty_occupied_ns = blk_time_get_ns();2266			/*2267			 * Start the state machine for measuring the2268			 * total service time of rq: setting2269			 * wait_dispatch will cause bfqd->waited_rq to2270			 * be set when rq will be dispatched.2271			 */2272			bfqd->wait_dispatch = true;2273			/*2274			 * If there is no I/O in service in the drive,2275			 * then possible injection occurred before the2276			 * arrival of rq will not affect the total2277			 * service time of rq. So the injection limit2278			 * must not be updated as a function of such2279			 * total service time, unless new injection2280			 * occurs before rq is completed. To have the2281			 * injection limit updated only in the latter2282			 * case, reset rqs_injected here (rqs_injected2283			 * will be set in case injection is performed2284			 * on bfqq before rq is completed).2285			 */2286			if (bfqd->tot_rq_in_driver == 0)2287				bfqd->rqs_injected = false;2288		}2289	}2290 2291	if (bfq_bfqq_sync(bfqq))2292		bfq_update_io_intensity(bfqq, now_ns);2293 2294	elv_rb_add(&bfqq->sort_list, rq);2295 2296	/*2297	 * Check if this request is a better next-serve candidate.2298	 */2299	prev = bfqq->next_rq;2300	next_rq = bfq_choose_req(bfqd, bfqq->next_rq, rq, bfqd->last_position);2301	bfqq->next_rq = next_rq;2302 2303	/*2304	 * Adjust priority tree position, if next_rq changes.2305	 * See comments on bfq_pos_tree_add_move() for the unlikely().2306	 */2307	if (unlikely(!bfqd->nonrot_with_queueing && prev != bfqq->next_rq))2308		bfq_pos_tree_add_move(bfqd, bfqq);2309 2310	if (!bfq_bfqq_busy(bfqq)) /* switching to busy ... */2311		bfq_bfqq_handle_idle_busy_switch(bfqd, bfqq, old_wr_coeff,2312						 rq, &interactive);2313	else {2314		if (bfqd->low_latency && old_wr_coeff == 1 && !rq_is_sync(rq) &&2315		    time_is_before_jiffies(2316				bfqq->last_wr_start_finish +2317				bfqd->bfq_wr_min_inter_arr_async)) {2318			bfqq->wr_coeff = bfqd->bfq_wr_coeff;2319			bfqq->wr_cur_max_time = bfq_wr_duration(bfqd);2320 2321			bfqd->wr_busy_queues++;2322			bfqq->entity.prio_changed = 1;2323		}2324		if (prev != bfqq->next_rq)2325			bfq_updated_next_req(bfqd, bfqq);2326	}2327 2328	/*2329	 * Assign jiffies to last_wr_start_finish in the following2330	 * cases:2331	 *2332	 * . if bfqq is not going to be weight-raised, because, for2333	 *   non weight-raised queues, last_wr_start_finish stores the2334	 *   arrival time of the last request; as of now, this piece2335	 *   of information is used only for deciding whether to2336	 *   weight-raise async queues2337	 *2338	 * . if bfqq is not weight-raised, because, if bfqq is now2339	 *   switching to weight-raised, then last_wr_start_finish2340	 *   stores the time when weight-raising starts2341	 *2342	 * . if bfqq is interactive, because, regardless of whether2343	 *   bfqq is currently weight-raised, the weight-raising2344	 *   period must start or restart (this case is considered2345	 *   separately because it is not detected by the above2346	 *   conditions, if bfqq is already weight-raised)2347	 *2348	 * last_wr_start_finish has to be updated also if bfqq is soft2349	 * real-time, because the weight-raising period is constantly2350	 * restarted on idle-to-busy transitions for these queues, but2351	 * this is already done in bfq_bfqq_handle_idle_busy_switch if2352	 * needed.2353	 */2354	if (bfqd->low_latency &&2355		(old_wr_coeff == 1 || bfqq->wr_coeff == 1 || interactive))2356		bfqq->last_wr_start_finish = jiffies;2357}2358 2359static struct request *bfq_find_rq_fmerge(struct bfq_data *bfqd,2360					  struct bio *bio,2361					  struct request_queue *q)2362{2363	struct bfq_queue *bfqq = bfqd->bio_bfqq;2364 2365 2366	if (bfqq)2367		return elv_rb_find(&bfqq->sort_list, bio_end_sector(bio));2368 2369	return NULL;2370}2371 2372static sector_t get_sdist(sector_t last_pos, struct request *rq)2373{2374	if (last_pos)2375		return abs(blk_rq_pos(rq) - last_pos);2376 2377	return 0;2378}2379 2380static void bfq_remove_request(struct request_queue *q,2381			       struct request *rq)2382{2383	struct bfq_queue *bfqq = RQ_BFQQ(rq);2384	struct bfq_data *bfqd = bfqq->bfqd;2385	const int sync = rq_is_sync(rq);2386 2387	if (bfqq->next_rq == rq) {2388		bfqq->next_rq = bfq_find_next_rq(bfqd, bfqq, rq);2389		bfq_updated_next_req(bfqd, bfqq);2390	}2391 2392	if (rq->queuelist.prev != &rq->queuelist)2393		list_del_init(&rq->queuelist);2394	bfqq->queued[sync]--;2395	/*2396	 * Updating of 'bfqd->queued' is protected by 'bfqd->lock', however, it2397	 * may be read without holding the lock in bfq_has_work().2398	 */2399	WRITE_ONCE(bfqd->queued, bfqd->queued - 1);2400	elv_rb_del(&bfqq->sort_list, rq);2401 2402	elv_rqhash_del(q, rq);2403	if (q->last_merge == rq)2404		q->last_merge = NULL;2405 2406	if (RB_EMPTY_ROOT(&bfqq->sort_list)) {2407		bfqq->next_rq = NULL;2408 2409		if (bfq_bfqq_busy(bfqq) && bfqq != bfqd->in_service_queue) {2410			bfq_del_bfqq_busy(bfqq, false);2411			/*2412			 * bfqq emptied. In normal operation, when2413			 * bfqq is empty, bfqq->entity.service and2414			 * bfqq->entity.budget must contain,2415			 * respectively, the service received and the2416			 * budget used last time bfqq emptied. These2417			 * facts do not hold in this case, as at least2418			 * this last removal occurred while bfqq is2419			 * not in service. To avoid inconsistencies,2420			 * reset both bfqq->entity.service and2421			 * bfqq->entity.budget, if bfqq has still a2422			 * process that may issue I/O requests to it.2423			 */2424			bfqq->entity.budget = bfqq->entity.service = 0;2425		}2426 2427		/*2428		 * Remove queue from request-position tree as it is empty.2429		 */2430		if (bfqq->pos_root) {2431			rb_erase(&bfqq->pos_node, bfqq->pos_root);2432			bfqq->pos_root = NULL;2433		}2434	} else {2435		/* see comments on bfq_pos_tree_add_move() for the unlikely() */2436		if (unlikely(!bfqd->nonrot_with_queueing))2437			bfq_pos_tree_add_move(bfqd, bfqq);2438	}2439 2440	if (rq->cmd_flags & REQ_META)2441		bfqq->meta_pending--;2442 2443}2444 2445static bool bfq_bio_merge(struct request_queue *q, struct bio *bio,2446		unsigned int nr_segs)2447{2448	struct bfq_data *bfqd = q->elevator->elevator_data;2449	struct request *free = NULL;2450	/*2451	 * bfq_bic_lookup grabs the queue_lock: invoke it now and2452	 * store its return value for later use, to avoid nesting2453	 * queue_lock inside the bfqd->lock. We assume that the bic2454	 * returned by bfq_bic_lookup does not go away before2455	 * bfqd->lock is taken.2456	 */2457	struct bfq_io_cq *bic = bfq_bic_lookup(q);2458	bool ret;2459 2460	spin_lock_irq(&bfqd->lock);2461 2462	if (bic) {2463		/*2464		 * Make sure cgroup info is uptodate for current process before2465		 * considering the merge.2466		 */2467		bfq_bic_update_cgroup(bic, bio);2468 2469		bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf),2470					     bfq_actuator_index(bfqd, bio));2471	} else {2472		bfqd->bio_bfqq = NULL;2473	}2474	bfqd->bio_bic = bic;2475 2476	ret = blk_mq_sched_try_merge(q, bio, nr_segs, &free);2477 2478	spin_unlock_irq(&bfqd->lock);2479	if (free)2480		blk_mq_free_request(free);2481 2482	return ret;2483}2484 2485static int bfq_request_merge(struct request_queue *q, struct request **req,2486			     struct bio *bio)2487{2488	struct bfq_data *bfqd = q->elevator->elevator_data;2489	struct request *__rq;2490 2491	__rq = bfq_find_rq_fmerge(bfqd, bio, q);2492	if (__rq && elv_bio_merge_ok(__rq, bio)) {2493		*req = __rq;2494 2495		if (blk_discard_mergable(__rq))2496			return ELEVATOR_DISCARD_MERGE;2497		return ELEVATOR_FRONT_MERGE;2498	}2499 2500	return ELEVATOR_NO_MERGE;2501}2502 2503static void bfq_request_merged(struct request_queue *q, struct request *req,2504			       enum elv_merge type)2505{2506	if (type == ELEVATOR_FRONT_MERGE &&2507	    rb_prev(&req->rb_node) &&2508	    blk_rq_pos(req) <2509	    blk_rq_pos(container_of(rb_prev(&req->rb_node),2510				    struct request, rb_node))) {2511		struct bfq_queue *bfqq = RQ_BFQQ(req);2512		struct bfq_data *bfqd;2513		struct request *prev, *next_rq;2514 2515		if (!bfqq)2516			return;2517 2518		bfqd = bfqq->bfqd;2519 2520		/* Reposition request in its sort_list */2521		elv_rb_del(&bfqq->sort_list, req);2522		elv_rb_add(&bfqq->sort_list, req);2523 2524		/* Choose next request to be served for bfqq */2525		prev = bfqq->next_rq;2526		next_rq = bfq_choose_req(bfqd, bfqq->next_rq, req,2527					 bfqd->last_position);2528		bfqq->next_rq = next_rq;2529		/*2530		 * If next_rq changes, update both the queue's budget to2531		 * fit the new request and the queue's position in its2532		 * rq_pos_tree.2533		 */2534		if (prev != bfqq->next_rq) {2535			bfq_updated_next_req(bfqd, bfqq);2536			/*2537			 * See comments on bfq_pos_tree_add_move() for2538			 * the unlikely().2539			 */2540			if (unlikely(!bfqd->nonrot_with_queueing))2541				bfq_pos_tree_add_move(bfqd, bfqq);2542		}2543	}2544}2545 2546/*2547 * This function is called to notify the scheduler that the requests2548 * rq and 'next' have been merged, with 'next' going away.  BFQ2549 * exploits this hook to address the following issue: if 'next' has a2550 * fifo_time lower that rq, then the fifo_time of rq must be set to2551 * the value of 'next', to not forget the greater age of 'next'.2552 *2553 * NOTE: in this function we assume that rq is in a bfq_queue, basing2554 * on that rq is picked from the hash table q->elevator->hash, which,2555 * in its turn, is filled only with I/O requests present in2556 * bfq_queues, while BFQ is in use for the request queue q. In fact,2557 * the function that fills this hash table (elv_rqhash_add) is called2558 * only by bfq_insert_request.2559 */2560static void bfq_requests_merged(struct request_queue *q, struct request *rq,2561				struct request *next)2562{2563	struct bfq_queue *bfqq = RQ_BFQQ(rq),2564		*next_bfqq = RQ_BFQQ(next);2565 2566	if (!bfqq)2567		goto remove;2568 2569	/*2570	 * If next and rq belong to the same bfq_queue and next is older2571	 * than rq, then reposition rq in the fifo (by substituting next2572	 * with rq). Otherwise, if next and rq belong to different2573	 * bfq_queues, never reposition rq: in fact, we would have to2574	 * reposition it with respect to next's position in its own fifo,2575	 * which would most certainly be too expensive with respect to2576	 * the benefits.2577	 */2578	if (bfqq == next_bfqq &&2579	    !list_empty(&rq->queuelist) && !list_empty(&next->queuelist) &&2580	    next->fifo_time < rq->fifo_time) {2581		list_del_init(&rq->queuelist);2582		list_replace_init(&next->queuelist, &rq->queuelist);2583		rq->fifo_time = next->fifo_time;2584	}2585 2586	if (bfqq->next_rq == next)2587		bfqq->next_rq = rq;2588 2589	bfqg_stats_update_io_merged(bfqq_group(bfqq), next->cmd_flags);2590remove:2591	/* Merged request may be in the IO scheduler. Remove it. */2592	if (!RB_EMPTY_NODE(&next->rb_node)) {2593		bfq_remove_request(next->q, next);2594		if (next_bfqq)2595			bfqg_stats_update_io_remove(bfqq_group(next_bfqq),2596						    next->cmd_flags);2597	}2598}2599 2600/* Must be called with bfqq != NULL */2601static void bfq_bfqq_end_wr(struct bfq_queue *bfqq)2602{2603	/*2604	 * If bfqq has been enjoying interactive weight-raising, then2605	 * reset soft_rt_next_start. We do it for the following2606	 * reason. bfqq may have been conveying the I/O needed to load2607	 * a soft real-time application. Such an application actually2608	 * exhibits a soft real-time I/O pattern after it finishes2609	 * loading, and finally starts doing its job. But, if bfqq has2610	 * been receiving a lot of bandwidth so far (likely to happen2611	 * on a fast device), then soft_rt_next_start now contains a2612	 * high value that. So, without this reset, bfqq would be2613	 * prevented from being possibly considered as soft_rt for a2614	 * very long time.2615	 */2616 2617	if (bfqq->wr_cur_max_time !=2618	    bfqq->bfqd->bfq_wr_rt_max_time)2619		bfqq->soft_rt_next_start = jiffies;2620 2621	if (bfq_bfqq_busy(bfqq))2622		bfqq->bfqd->wr_busy_queues--;2623	bfqq->wr_coeff = 1;2624	bfqq->wr_cur_max_time = 0;2625	bfqq->last_wr_start_finish = jiffies;2626	/*2627	 * Trigger a weight change on the next invocation of2628	 * __bfq_entity_update_weight_prio.2629	 */2630	bfqq->entity.prio_changed = 1;2631}2632 2633void bfq_end_wr_async_queues(struct bfq_data *bfqd,2634			     struct bfq_group *bfqg)2635{2636	int i, j, k;2637 2638	for (k = 0; k < bfqd->num_actuators; k++) {2639		for (i = 0; i < 2; i++)2640			for (j = 0; j < IOPRIO_NR_LEVELS; j++)2641				if (bfqg->async_bfqq[i][j][k])2642					bfq_bfqq_end_wr(bfqg->async_bfqq[i][j][k]);2643		if (bfqg->async_idle_bfqq[k])2644			bfq_bfqq_end_wr(bfqg->async_idle_bfqq[k]);2645	}2646}2647 2648static void bfq_end_wr(struct bfq_data *bfqd)2649{2650	struct bfq_queue *bfqq;2651	int i;2652 2653	spin_lock_irq(&bfqd->lock);2654 2655	for (i = 0; i < bfqd->num_actuators; i++) {2656		list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list)2657			bfq_bfqq_end_wr(bfqq);2658	}2659	list_for_each_entry(bfqq, &bfqd->idle_list, bfqq_list)2660		bfq_bfqq_end_wr(bfqq);2661	bfq_end_wr_async(bfqd);2662 2663	spin_unlock_irq(&bfqd->lock);2664}2665 2666static sector_t bfq_io_struct_pos(void *io_struct, bool request)2667{2668	if (request)2669		return blk_rq_pos(io_struct);2670	else2671		return ((struct bio *)io_struct)->bi_iter.bi_sector;2672}2673 2674static int bfq_rq_close_to_sector(void *io_struct, bool request,2675				  sector_t sector)2676{2677	return abs(bfq_io_struct_pos(io_struct, request) - sector) <=2678	       BFQQ_CLOSE_THR;2679}2680 2681static struct bfq_queue *bfqq_find_close(struct bfq_data *bfqd,2682					 struct bfq_queue *bfqq,2683					 sector_t sector)2684{2685	struct rb_root *root = &bfqq_group(bfqq)->rq_pos_tree;2686	struct rb_node *parent, *node;2687	struct bfq_queue *__bfqq;2688 2689	if (RB_EMPTY_ROOT(root))2690		return NULL;2691 2692	/*2693	 * First, if we find a request starting at the end of the last2694	 * request, choose it.2695	 */2696	__bfqq = bfq_rq_pos_tree_lookup(bfqd, root, sector, &parent, NULL);2697	if (__bfqq)2698		return __bfqq;2699 2700	/*2701	 * If the exact sector wasn't found, the parent of the NULL leaf2702	 * will contain the closest sector (rq_pos_tree sorted by2703	 * next_request position).2704	 */2705	__bfqq = rb_entry(parent, struct bfq_queue, pos_node);2706	if (bfq_rq_close_to_sector(__bfqq->next_rq, true, sector))2707		return __bfqq;2708 2709	if (blk_rq_pos(__bfqq->next_rq) < sector)2710		node = rb_next(&__bfqq->pos_node);2711	else2712		node = rb_prev(&__bfqq->pos_node);2713	if (!node)2714		return NULL;2715 2716	__bfqq = rb_entry(node, struct bfq_queue, pos_node);2717	if (bfq_rq_close_to_sector(__bfqq->next_rq, true, sector))2718		return __bfqq;2719 2720	return NULL;2721}2722 2723static struct bfq_queue *bfq_find_close_cooperator(struct bfq_data *bfqd,2724						   struct bfq_queue *cur_bfqq,2725						   sector_t sector)2726{2727	struct bfq_queue *bfqq;2728 2729	/*2730	 * We shall notice if some of the queues are cooperating,2731	 * e.g., working closely on the same area of the device. In2732	 * that case, we can group them together and: 1) don't waste2733	 * time idling, and 2) serve the union of their requests in2734	 * the best possible order for throughput.2735	 */2736	bfqq = bfqq_find_close(bfqd, cur_bfqq, sector);2737	if (!bfqq || bfqq == cur_bfqq)2738		return NULL;2739 2740	return bfqq;2741}2742 2743static struct bfq_queue *2744bfq_setup_merge(struct bfq_queue *bfqq, struct bfq_queue *new_bfqq)2745{2746	int process_refs, new_process_refs;2747	struct bfq_queue *__bfqq;2748 2749	/*2750	 * If there are no process references on the new_bfqq, then it is2751	 * unsafe to follow the ->new_bfqq chain as other bfqq's in the chain2752	 * may have dropped their last reference (not just their last process2753	 * reference).2754	 */2755	if (!bfqq_process_refs(new_bfqq))2756		return NULL;2757 2758	/* Avoid a circular list and skip interim queue merges. */2759	while ((__bfqq = new_bfqq->new_bfqq)) {2760		if (__bfqq == bfqq)2761			return NULL;2762		new_bfqq = __bfqq;2763	}2764 2765	process_refs = bfqq_process_refs(bfqq);2766	new_process_refs = bfqq_process_refs(new_bfqq);2767	/*2768	 * If the process for the bfqq has gone away, there is no2769	 * sense in merging the queues.2770	 */2771	if (process_refs == 0 || new_process_refs == 0)2772		return NULL;2773 2774	/*2775	 * Make sure merged queues belong to the same parent. Parents could2776	 * have changed since the time we decided the two queues are suitable2777	 * for merging.2778	 */2779	if (new_bfqq->entity.parent != bfqq->entity.parent)2780		return NULL;2781 2782	bfq_log_bfqq(bfqq->bfqd, bfqq, "scheduling merge with queue %d",2783		new_bfqq->pid);2784 2785	/*2786	 * Merging is just a redirection: the requests of the process2787	 * owning one of the two queues are redirected to the other queue.2788	 * The latter queue, in its turn, is set as shared if this is the2789	 * first time that the requests of some process are redirected to2790	 * it.2791	 *2792	 * We redirect bfqq to new_bfqq and not the opposite, because2793	 * we are in the context of the process owning bfqq, thus we2794	 * have the io_cq of this process. So we can immediately2795	 * configure this io_cq to redirect the requests of the2796	 * process to new_bfqq. In contrast, the io_cq of new_bfqq is2797	 * not available any more (new_bfqq->bic == NULL).2798	 *2799	 * Anyway, even in case new_bfqq coincides with the in-service2800	 * queue, redirecting requests the in-service queue is the2801	 * best option, as we feed the in-service queue with new2802	 * requests close to the last request served and, by doing so,2803	 * are likely to increase the throughput.2804	 */2805	bfqq->new_bfqq = new_bfqq;2806	/*2807	 * The above assignment schedules the following redirections:2808	 * each time some I/O for bfqq arrives, the process that2809	 * generated that I/O is disassociated from bfqq and2810	 * associated with new_bfqq. Here we increases new_bfqq->ref2811	 * in advance, adding the number of processes that are2812	 * expected to be associated with new_bfqq as they happen to2813	 * issue I/O.2814	 */2815	new_bfqq->ref += process_refs;2816	return new_bfqq;2817}2818 2819static bool bfq_may_be_close_cooperator(struct bfq_queue *bfqq,2820					struct bfq_queue *new_bfqq)2821{2822	if (bfq_too_late_for_merging(new_bfqq))2823		return false;2824 2825	if (bfq_class_idle(bfqq) || bfq_class_idle(new_bfqq) ||2826	    (bfqq->ioprio_class != new_bfqq->ioprio_class))2827		return false;2828 2829	/*2830	 * If either of the queues has already been detected as seeky,2831	 * then merging it with the other queue is unlikely to lead to2832	 * sequential I/O.2833	 */2834	if (BFQQ_SEEKY(bfqq) || BFQQ_SEEKY(new_bfqq))2835		return false;2836 2837	/*2838	 * Interleaved I/O is known to be done by (some) applications2839	 * only for reads, so it does not make sense to merge async2840	 * queues.2841	 */2842	if (!bfq_bfqq_sync(bfqq) || !bfq_bfqq_sync(new_bfqq))2843		return false;2844 2845	return true;2846}2847 2848static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd,2849					     struct bfq_queue *bfqq);2850 2851static struct bfq_queue *2852bfq_setup_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq,2853		       struct bfq_queue *stable_merge_bfqq,2854		       struct bfq_iocq_bfqq_data *bfqq_data)2855{2856	int proc_ref = min(bfqq_process_refs(bfqq),2857			   bfqq_process_refs(stable_merge_bfqq));2858	struct bfq_queue *new_bfqq = NULL;2859 2860	bfqq_data->stable_merge_bfqq = NULL;2861	if (idling_boosts_thr_without_issues(bfqd, bfqq) || proc_ref == 0)2862		goto out;2863 2864	/* next function will take at least one ref */2865	new_bfqq = bfq_setup_merge(bfqq, stable_merge_bfqq);2866 2867	if (new_bfqq) {2868		bfqq_data->stably_merged = true;2869		if (new_bfqq->bic) {2870			unsigned int new_a_idx = new_bfqq->actuator_idx;2871			struct bfq_iocq_bfqq_data *new_bfqq_data =2872				&new_bfqq->bic->bfqq_data[new_a_idx];2873 2874			new_bfqq_data->stably_merged = true;2875		}2876	}2877 2878out:2879	/* deschedule stable merge, because done or aborted here */2880	bfq_put_stable_ref(stable_merge_bfqq);2881 2882	return new_bfqq;2883}2884 2885/*2886 * Attempt to schedule a merge of bfqq with the currently in-service2887 * queue or with a close queue among the scheduled queues.  Return2888 * NULL if no merge was scheduled, a pointer to the shared bfq_queue2889 * structure otherwise.2890 *2891 * The OOM queue is not allowed to participate to cooperation: in fact, since2892 * the requests temporarily redirected to the OOM queue could be redirected2893 * again to dedicated queues at any time, the state needed to correctly2894 * handle merging with the OOM queue would be quite complex and expensive2895 * to maintain. Besides, in such a critical condition as an out of memory,2896 * the benefits of queue merging may be little relevant, or even negligible.2897 *2898 * WARNING: queue merging may impair fairness among non-weight raised2899 * queues, for at least two reasons: 1) the original weight of a2900 * merged queue may change during the merged state, 2) even being the2901 * weight the same, a merged queue may be bloated with many more2902 * requests than the ones produced by its originally-associated2903 * process.2904 */2905static struct bfq_queue *2906bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq,2907		     void *io_struct, bool request, struct bfq_io_cq *bic)2908{2909	struct bfq_queue *in_service_bfqq, *new_bfqq;2910	unsigned int a_idx = bfqq->actuator_idx;2911	struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx];2912 2913	/* if a merge has already been setup, then proceed with that first */2914	new_bfqq = bfqq->new_bfqq;2915	if (new_bfqq) {2916		while (new_bfqq->new_bfqq)2917			new_bfqq = new_bfqq->new_bfqq;2918		return new_bfqq;2919	}2920 2921	/*2922	 * Check delayed stable merge for rotational or non-queueing2923	 * devs. For this branch to be executed, bfqq must not be2924	 * currently merged with some other queue (i.e., bfqq->bic2925	 * must be non null). If we considered also merged queues,2926	 * then we should also check whether bfqq has already been2927	 * merged with bic->stable_merge_bfqq. But this would be2928	 * costly and complicated.2929	 */2930	if (unlikely(!bfqd->nonrot_with_queueing)) {2931		/*2932		 * Make sure also that bfqq is sync, because2933		 * bic->stable_merge_bfqq may point to some queue (for2934		 * stable merging) also if bic is associated with a2935		 * sync queue, but this bfqq is async2936		 */2937		if (bfq_bfqq_sync(bfqq) && bfqq_data->stable_merge_bfqq &&2938		    !bfq_bfqq_just_created(bfqq) &&2939		    time_is_before_jiffies(bfqq->split_time +2940					  msecs_to_jiffies(bfq_late_stable_merging)) &&2941		    time_is_before_jiffies(bfqq->creation_time +2942					   msecs_to_jiffies(bfq_late_stable_merging))) {2943			struct bfq_queue *stable_merge_bfqq =2944				bfqq_data->stable_merge_bfqq;2945 2946			return bfq_setup_stable_merge(bfqd, bfqq,2947						      stable_merge_bfqq,2948						      bfqq_data);2949		}2950	}2951 2952	/*2953	 * Do not perform queue merging if the device is non2954	 * rotational and performs internal queueing. In fact, such a2955	 * device reaches a high speed through internal parallelism2956	 * and pipelining. This means that, to reach a high2957	 * throughput, it must have many requests enqueued at the same2958	 * time. But, in this configuration, the internal scheduling2959	 * algorithm of the device does exactly the job of queue2960	 * merging: it reorders requests so as to obtain as much as2961	 * possible a sequential I/O pattern. As a consequence, with2962	 * the workload generated by processes doing interleaved I/O,2963	 * the throughput reached by the device is likely to be the2964	 * same, with and without queue merging.2965	 *2966	 * Disabling merging also provides a remarkable benefit in2967	 * terms of throughput. Merging tends to make many workloads2968	 * artificially more uneven, because of shared queues2969	 * remaining non empty for incomparably more time than2970	 * non-merged queues. This may accentuate workload2971	 * asymmetries. For example, if one of the queues in a set of2972	 * merged queues has a higher weight than a normal queue, then2973	 * the shared queue may inherit such a high weight and, by2974	 * staying almost always active, may force BFQ to perform I/O2975	 * plugging most of the time. This evidently makes it harder2976	 * for BFQ to let the device reach a high throughput.2977	 *2978	 * Finally, the likely() macro below is not used because one2979	 * of the two branches is more likely than the other, but to2980	 * have the code path after the following if() executed as2981	 * fast as possible for the case of a non rotational device2982	 * with queueing. We want it because this is the fastest kind2983	 * of device. On the opposite end, the likely() may lengthen2984	 * the execution time of BFQ for the case of slower devices2985	 * (rotational or at least without queueing). But in this case2986	 * the execution time of BFQ matters very little, if not at2987	 * all.2988	 */2989	if (likely(bfqd->nonrot_with_queueing))2990		return NULL;2991 2992	/*2993	 * Prevent bfqq from being merged if it has been created too2994	 * long ago. The idea is that true cooperating processes, and2995	 * thus their associated bfq_queues, are supposed to be2996	 * created shortly after each other. This is the case, e.g.,2997	 * for KVM/QEMU and dump I/O threads. Basing on this2998	 * assumption, the following filtering greatly reduces the2999	 * probability that two non-cooperating processes, which just3000	 * happen to do close I/O for some short time interval, have3001	 * their queues merged by mistake.3002	 */3003	if (bfq_too_late_for_merging(bfqq))3004		return NULL;3005 3006	if (!io_struct || unlikely(bfqq == &bfqd->oom_bfqq))3007		return NULL;3008 3009	/* If there is only one backlogged queue, don't search. */3010	if (bfq_tot_busy_queues(bfqd) == 1)3011		return NULL;3012 3013	in_service_bfqq = bfqd->in_service_queue;3014 3015	if (in_service_bfqq && in_service_bfqq != bfqq &&3016	    likely(in_service_bfqq != &bfqd->oom_bfqq) &&3017	    bfq_rq_close_to_sector(io_struct, request,3018				   bfqd->in_serv_last_pos) &&3019	    bfqq->entity.parent == in_service_bfqq->entity.parent &&3020	    bfq_may_be_close_cooperator(bfqq, in_service_bfqq)) {3021		new_bfqq = bfq_setup_merge(bfqq, in_service_bfqq);3022		if (new_bfqq)3023			return new_bfqq;3024	}3025	/*3026	 * Check whether there is a cooperator among currently scheduled3027	 * queues. The only thing we need is that the bio/request is not3028	 * NULL, as we need it to establish whether a cooperator exists.3029	 */3030	new_bfqq = bfq_find_close_cooperator(bfqd, bfqq,3031			bfq_io_struct_pos(io_struct, request));3032 3033	if (new_bfqq && likely(new_bfqq != &bfqd->oom_bfqq) &&3034	    bfq_may_be_close_cooperator(bfqq, new_bfqq))3035		return bfq_setup_merge(bfqq, new_bfqq);3036 3037	return NULL;3038}3039 3040static void bfq_bfqq_save_state(struct bfq_queue *bfqq)3041{3042	struct bfq_io_cq *bic = bfqq->bic;3043	unsigned int a_idx = bfqq->actuator_idx;3044	struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx];3045 3046	/*3047	 * If !bfqq->bic, the queue is already shared or its requests3048	 * have already been redirected to a shared queue; both idle window3049	 * and weight raising state have already been saved. Do nothing.3050	 */3051	if (!bic)3052		return;3053 3054	bfqq_data->saved_last_serv_time_ns = bfqq->last_serv_time_ns;3055	bfqq_data->saved_inject_limit =	bfqq->inject_limit;3056	bfqq_data->saved_decrease_time_jif = bfqq->decrease_time_jif;3057 3058	bfqq_data->saved_weight = bfqq->entity.orig_weight;3059	bfqq_data->saved_ttime = bfqq->ttime;3060	bfqq_data->saved_has_short_ttime =3061		bfq_bfqq_has_short_ttime(bfqq);3062	bfqq_data->saved_IO_bound = bfq_bfqq_IO_bound(bfqq);3063	bfqq_data->saved_io_start_time = bfqq->io_start_time;3064	bfqq_data->saved_tot_idle_time = bfqq->tot_idle_time;3065	bfqq_data->saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq);3066	bfqq_data->was_in_burst_list =3067		!hlist_unhashed(&bfqq->burst_list_node);3068 3069	if (unlikely(bfq_bfqq_just_created(bfqq) &&3070		     !bfq_bfqq_in_large_burst(bfqq) &&3071		     bfqq->bfqd->low_latency)) {3072		/*3073		 * bfqq being merged right after being created: bfqq3074		 * would have deserved interactive weight raising, but3075		 * did not make it to be set in a weight-raised state,3076		 * because of this early merge.	Store directly the3077		 * weight-raising state that would have been assigned3078		 * to bfqq, so that to avoid that bfqq unjustly fails3079		 * to enjoy weight raising if split soon.3080		 */3081		bfqq_data->saved_wr_coeff = bfqq->bfqd->bfq_wr_coeff;3082		bfqq_data->saved_wr_start_at_switch_to_srt =3083			bfq_smallest_from_now();3084		bfqq_data->saved_wr_cur_max_time =3085			bfq_wr_duration(bfqq->bfqd);3086		bfqq_data->saved_last_wr_start_finish = jiffies;3087	} else {3088		bfqq_data->saved_wr_coeff = bfqq->wr_coeff;3089		bfqq_data->saved_wr_start_at_switch_to_srt =3090			bfqq->wr_start_at_switch_to_srt;3091		bfqq_data->saved_service_from_wr =3092			bfqq->service_from_wr;3093		bfqq_data->saved_last_wr_start_finish =3094			bfqq->last_wr_start_finish;3095		bfqq_data->saved_wr_cur_max_time = bfqq->wr_cur_max_time;3096	}3097}3098 3099 3100void bfq_reassign_last_bfqq(struct bfq_queue *cur_bfqq,3101			    struct bfq_queue *new_bfqq)3102{3103	if (cur_bfqq->entity.parent &&3104	    cur_bfqq->entity.parent->last_bfqq_created == cur_bfqq)3105		cur_bfqq->entity.parent->last_bfqq_created = new_bfqq;3106	else if (cur_bfqq->bfqd && cur_bfqq->bfqd->last_bfqq_created == cur_bfqq)3107		cur_bfqq->bfqd->last_bfqq_created = new_bfqq;3108}3109 3110void bfq_release_process_ref(struct bfq_data *bfqd, struct bfq_queue *bfqq)3111{3112	/*3113	 * To prevent bfqq's service guarantees from being violated,3114	 * bfqq may be left busy, i.e., queued for service, even if3115	 * empty (see comments in __bfq_bfqq_expire() for3116	 * details). But, if no process will send requests to bfqq any3117	 * longer, then there is no point in keeping bfqq queued for3118	 * service. In addition, keeping bfqq queued for service, but3119	 * with no process ref any longer, may have caused bfqq to be3120	 * freed when dequeued from service. But this is assumed to3121	 * never happen.3122	 */3123	if (bfq_bfqq_busy(bfqq) && RB_EMPTY_ROOT(&bfqq->sort_list) &&3124	    bfqq != bfqd->in_service_queue)3125		bfq_del_bfqq_busy(bfqq, false);3126 3127	bfq_reassign_last_bfqq(bfqq, NULL);3128 3129	bfq_put_queue(bfqq);3130}3131 3132static struct bfq_queue *bfq_merge_bfqqs(struct bfq_data *bfqd,3133					 struct bfq_io_cq *bic,3134					 struct bfq_queue *bfqq)3135{3136	struct bfq_queue *new_bfqq = bfqq->new_bfqq;3137 3138	bfq_log_bfqq(bfqd, bfqq, "merging with queue %lu",3139		(unsigned long)new_bfqq->pid);3140	/* Save weight raising and idle window of the merged queues */3141	bfq_bfqq_save_state(bfqq);3142	bfq_bfqq_save_state(new_bfqq);3143	if (bfq_bfqq_IO_bound(bfqq))3144		bfq_mark_bfqq_IO_bound(new_bfqq);3145	bfq_clear_bfqq_IO_bound(bfqq);3146 3147	/*3148	 * The processes associated with bfqq are cooperators of the3149	 * processes associated with new_bfqq. So, if bfqq has a3150	 * waker, then assume that all these processes will be happy3151	 * to let bfqq's waker freely inject I/O when they have no3152	 * I/O.3153	 */3154	if (bfqq->waker_bfqq && !new_bfqq->waker_bfqq &&3155	    bfqq->waker_bfqq != new_bfqq) {3156		new_bfqq->waker_bfqq = bfqq->waker_bfqq;3157		new_bfqq->tentative_waker_bfqq = NULL;3158 3159		/*3160		 * If the waker queue disappears, then3161		 * new_bfqq->waker_bfqq must be reset. So insert3162		 * new_bfqq into the woken_list of the waker. See3163		 * bfq_check_waker for details.3164		 */3165		hlist_add_head(&new_bfqq->woken_list_node,3166			       &new_bfqq->waker_bfqq->woken_list);3167 3168	}3169 3170	/*3171	 * If bfqq is weight-raised, then let new_bfqq inherit3172	 * weight-raising. To reduce false positives, neglect the case3173	 * where bfqq has just been created, but has not yet made it3174	 * to be weight-raised (which may happen because EQM may merge3175	 * bfqq even before bfq_add_request is executed for the first3176	 * time for bfqq). Handling this case would however be very3177	 * easy, thanks to the flag just_created.3178	 */3179	if (new_bfqq->wr_coeff == 1 && bfqq->wr_coeff > 1) {3180		new_bfqq->wr_coeff = bfqq->wr_coeff;3181		new_bfqq->wr_cur_max_time = bfqq->wr_cur_max_time;3182		new_bfqq->last_wr_start_finish = bfqq->last_wr_start_finish;3183		new_bfqq->wr_start_at_switch_to_srt =3184			bfqq->wr_start_at_switch_to_srt;3185		if (bfq_bfqq_busy(new_bfqq))3186			bfqd->wr_busy_queues++;3187		new_bfqq->entity.prio_changed = 1;3188	}3189 3190	if (bfqq->wr_coeff > 1) { /* bfqq has given its wr to new_bfqq */3191		bfqq->wr_coeff = 1;3192		bfqq->entity.prio_changed = 1;3193		if (bfq_bfqq_busy(bfqq))3194			bfqd->wr_busy_queues--;3195	}3196 3197	bfq_log_bfqq(bfqd, new_bfqq, "merge_bfqqs: wr_busy %d",3198		     bfqd->wr_busy_queues);3199 3200	/*3201	 * Merge queues (that is, let bic redirect its requests to new_bfqq)3202	 */3203	bic_set_bfqq(bic, new_bfqq, true, bfqq->actuator_idx);3204	bfq_mark_bfqq_coop(new_bfqq);3205	/*3206	 * new_bfqq now belongs to at least two bics (it is a shared queue):3207	 * set new_bfqq->bic to NULL. bfqq either:3208	 * - does not belong to any bic any more, and hence bfqq->bic must3209	 *   be set to NULL, or3210	 * - is a queue whose owning bics have already been redirected to a3211	 *   different queue, hence the queue is destined to not belong to3212	 *   any bic soon and bfqq->bic is already NULL (therefore the next3213	 *   assignment causes no harm).3214	 */3215	new_bfqq->bic = NULL;3216	/*3217	 * If the queue is shared, the pid is the pid of one of the associated3218	 * processes. Which pid depends on the exact sequence of merge events3219	 * the queue underwent. So printing such a pid is useless and confusing3220	 * because it reports a random pid between those of the associated3221	 * processes.3222	 * We mark such a queue with a pid -1, and then print SHARED instead of3223	 * a pid in logging messages.3224	 */3225	new_bfqq->pid = -1;3226	bfqq->bic = NULL;3227 3228	bfq_reassign_last_bfqq(bfqq, new_bfqq);3229 3230	bfq_release_process_ref(bfqd, bfqq);3231 3232	return new_bfqq;3233}3234 3235static bool bfq_allow_bio_merge(struct request_queue *q, struct request *rq,3236				struct bio *bio)3237{3238	struct bfq_data *bfqd = q->elevator->elevator_data;3239	bool is_sync = op_is_sync(bio->bi_opf);3240	struct bfq_queue *bfqq = bfqd->bio_bfqq, *new_bfqq;3241 3242	/*3243	 * Disallow merge of a sync bio into an async request.3244	 */3245	if (is_sync && !rq_is_sync(rq))3246		return false;3247 3248	/*3249	 * Lookup the bfqq that this bio will be queued with. Allow3250	 * merge only if rq is queued there.3251	 */3252	if (!bfqq)3253		return false;3254 3255	/*3256	 * We take advantage of this function to perform an early merge3257	 * of the queues of possible cooperating processes.3258	 */3259	new_bfqq = bfq_setup_cooperator(bfqd, bfqq, bio, false, bfqd->bio_bic);3260	if (new_bfqq) {3261		/*3262		 * bic still points to bfqq, then it has not yet been3263		 * redirected to some other bfq_queue, and a queue3264		 * merge between bfqq and new_bfqq can be safely3265		 * fulfilled, i.e., bic can be redirected to new_bfqq3266		 * and bfqq can be put.3267		 */3268		while (bfqq != new_bfqq)3269			bfqq = bfq_merge_bfqqs(bfqd, bfqd->bio_bic, bfqq);3270 3271		/*3272		 * Change also bqfd->bio_bfqq, as3273		 * bfqd->bio_bic now points to new_bfqq, and3274		 * this function may be invoked again (and then may3275		 * use again bqfd->bio_bfqq).3276		 */3277		bfqd->bio_bfqq = bfqq;3278	}3279 3280	return bfqq == RQ_BFQQ(rq);3281}3282 3283/*3284 * Set the maximum time for the in-service queue to consume its3285 * budget. This prevents seeky processes from lowering the throughput.3286 * In practice, a time-slice service scheme is used with seeky3287 * processes.3288 */3289static void bfq_set_budget_timeout(struct bfq_data *bfqd,3290				   struct bfq_queue *bfqq)3291{3292	unsigned int timeout_coeff;3293 3294	if (bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time)3295		timeout_coeff = 1;3296	else3297		timeout_coeff = bfqq->entity.weight / bfqq->entity.orig_weight;3298 3299	bfqd->last_budget_start = blk_time_get();3300 3301	bfqq->budget_timeout = jiffies +3302		bfqd->bfq_timeout * timeout_coeff;3303}3304 3305static void __bfq_set_in_service_queue(struct bfq_data *bfqd,3306				       struct bfq_queue *bfqq)3307{3308	if (bfqq) {3309		bfq_clear_bfqq_fifo_expire(bfqq);3310 3311		bfqd->budgets_assigned = (bfqd->budgets_assigned * 7 + 256) / 8;3312 3313		if (time_is_before_jiffies(bfqq->last_wr_start_finish) &&3314		    bfqq->wr_coeff > 1 &&3315		    bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time &&3316		    time_is_before_jiffies(bfqq->budget_timeout)) {3317			/*3318			 * For soft real-time queues, move the start3319			 * of the weight-raising period forward by the3320			 * time the queue has not received any3321			 * service. Otherwise, a relatively long3322			 * service delay is likely to cause the3323			 * weight-raising period of the queue to end,3324			 * because of the short duration of the3325			 * weight-raising period of a soft real-time3326			 * queue.  It is worth noting that this move3327			 * is not so dangerous for the other queues,3328			 * because soft real-time queues are not3329			 * greedy.3330			 *3331			 * To not add a further variable, we use the3332			 * overloaded field budget_timeout to3333			 * determine for how long the queue has not3334			 * received service, i.e., how much time has3335			 * elapsed since the queue expired. However,3336			 * this is a little imprecise, because3337			 * budget_timeout is set to jiffies if bfqq3338			 * not only expires, but also remains with no3339			 * request.3340			 */3341			if (time_after(bfqq->budget_timeout,3342				       bfqq->last_wr_start_finish))3343				bfqq->last_wr_start_finish +=3344					jiffies - bfqq->budget_timeout;3345			else3346				bfqq->last_wr_start_finish = jiffies;3347		}3348 3349		bfq_set_budget_timeout(bfqd, bfqq);3350		bfq_log_bfqq(bfqd, bfqq,3351			     "set_in_service_queue, cur-budget = %d",3352			     bfqq->entity.budget);3353	}3354 3355	bfqd->in_service_queue = bfqq;3356	bfqd->in_serv_last_pos = 0;3357}3358 3359/*3360 * Get and set a new queue for service.3361 */3362static struct bfq_queue *bfq_set_in_service_queue(struct bfq_data *bfqd)3363{3364	struct bfq_queue *bfqq = bfq_get_next_queue(bfqd);3365 3366	__bfq_set_in_service_queue(bfqd, bfqq);3367	return bfqq;3368}3369 3370static void bfq_arm_slice_timer(struct bfq_data *bfqd)3371{3372	struct bfq_queue *bfqq = bfqd->in_service_queue;3373	u32 sl;3374 3375	bfq_mark_bfqq_wait_request(bfqq);3376 3377	/*3378	 * We don't want to idle for seeks, but we do want to allow3379	 * fair distribution of slice time for a process doing back-to-back3380	 * seeks. So allow a little bit of time for him to submit a new rq.3381	 */3382	sl = bfqd->bfq_slice_idle;3383	/*3384	 * Unless the queue is being weight-raised or the scenario is3385	 * asymmetric, grant only minimum idle time if the queue3386	 * is seeky. A long idling is preserved for a weight-raised3387	 * queue, or, more in general, in an asymmetric scenario,3388	 * because a long idling is needed for guaranteeing to a queue3389	 * its reserved share of the throughput (in particular, it is3390	 * needed if the queue has a higher weight than some other3391	 * queue).3392	 */3393	if (BFQQ_SEEKY(bfqq) && bfqq->wr_coeff == 1 &&3394	    !bfq_asymmetric_scenario(bfqd, bfqq))3395		sl = min_t(u64, sl, BFQ_MIN_TT);3396	else if (bfqq->wr_coeff > 1)3397		sl = max_t(u32, sl, 20ULL * NSEC_PER_MSEC);3398 3399	bfqd->last_idling_start = blk_time_get();3400	bfqd->last_idling_start_jiffies = jiffies;3401 3402	hrtimer_start(&bfqd->idle_slice_timer, ns_to_ktime(sl),3403		      HRTIMER_MODE_REL);3404	bfqg_stats_set_start_idle_time(bfqq_group(bfqq));3405}3406 3407/*3408 * In autotuning mode, max_budget is dynamically recomputed as the3409 * amount of sectors transferred in timeout at the estimated peak3410 * rate. This enables BFQ to utilize a full timeslice with a full3411 * budget, even if the in-service queue is served at peak rate. And3412 * this maximises throughput with sequential workloads.3413 */3414static unsigned long bfq_calc_max_budget(struct bfq_data *bfqd)3415{3416	return (u64)bfqd->peak_rate * USEC_PER_MSEC *3417		jiffies_to_msecs(bfqd->bfq_timeout)>>BFQ_RATE_SHIFT;3418}3419 3420/*3421 * Update parameters related to throughput and responsiveness, as a3422 * function of the estimated peak rate. See comments on3423 * bfq_calc_max_budget(), and on the ref_wr_duration array.3424 */3425static void update_thr_responsiveness_params(struct bfq_data *bfqd)3426{3427	if (bfqd->bfq_user_max_budget == 0) {3428		bfqd->bfq_max_budget =3429			bfq_calc_max_budget(bfqd);3430		bfq_log(bfqd, "new max_budget = %d", bfqd->bfq_max_budget);3431	}3432}3433 3434static void bfq_reset_rate_computation(struct bfq_data *bfqd,3435				       struct request *rq)3436{3437	if (rq != NULL) { /* new rq dispatch now, reset accordingly */3438		bfqd->last_dispatch = bfqd->first_dispatch = blk_time_get_ns();3439		bfqd->peak_rate_samples = 1;3440		bfqd->sequential_samples = 0;3441		bfqd->tot_sectors_dispatched = bfqd->last_rq_max_size =3442			blk_rq_sectors(rq);3443	} else /* no new rq dispatched, just reset the number of samples */3444		bfqd->peak_rate_samples = 0; /* full re-init on next disp. */3445 3446	bfq_log(bfqd,3447		"reset_rate_computation at end, sample %u/%u tot_sects %llu",3448		bfqd->peak_rate_samples, bfqd->sequential_samples,3449		bfqd->tot_sectors_dispatched);3450}3451 3452static void bfq_update_rate_reset(struct bfq_data *bfqd, struct request *rq)3453{3454	u32 rate, weight, divisor;3455 3456	/*3457	 * For the convergence property to hold (see comments on3458	 * bfq_update_peak_rate()) and for the assessment to be3459	 * reliable, a minimum number of samples must be present, and3460	 * a minimum amount of time must have elapsed. If not so, do3461	 * not compute new rate. Just reset parameters, to get ready3462	 * for a new evaluation attempt.3463	 */3464	if (bfqd->peak_rate_samples < BFQ_RATE_MIN_SAMPLES ||3465	    bfqd->delta_from_first < BFQ_RATE_MIN_INTERVAL)3466		goto reset_computation;3467 3468	/*3469	 * If a new request completion has occurred after last3470	 * dispatch, then, to approximate the rate at which requests3471	 * have been served by the device, it is more precise to3472	 * extend the observation interval to the last completion.3473	 */3474	bfqd->delta_from_first =3475		max_t(u64, bfqd->delta_from_first,3476		      bfqd->last_completion - bfqd->first_dispatch);3477 3478	/*3479	 * Rate computed in sects/usec, and not sects/nsec, for3480	 * precision issues.3481	 */3482	rate = div64_ul(bfqd->tot_sectors_dispatched<<BFQ_RATE_SHIFT,3483			div_u64(bfqd->delta_from_first, NSEC_PER_USEC));3484 3485	/*3486	 * Peak rate not updated if:3487	 * - the percentage of sequential dispatches is below 3/4 of the3488	 *   total, and rate is below the current estimated peak rate3489	 * - rate is unreasonably high (> 20M sectors/sec)3490	 */3491	if ((bfqd->sequential_samples < (3 * bfqd->peak_rate_samples)>>2 &&3492	     rate <= bfqd->peak_rate) ||3493		rate > 20<<BFQ_RATE_SHIFT)3494		goto reset_computation;3495 3496	/*3497	 * We have to update the peak rate, at last! To this purpose,3498	 * we use a low-pass filter. We compute the smoothing constant3499	 * of the filter as a function of the 'weight' of the new3500	 * measured rate.3501	 *3502	 * As can be seen in next formulas, we define this weight as a3503	 * quantity proportional to how sequential the workload is,3504	 * and to how long the observation time interval is.3505	 *3506	 * The weight runs from 0 to 8. The maximum value of the3507	 * weight, 8, yields the minimum value for the smoothing3508	 * constant. At this minimum value for the smoothing constant,3509	 * the measured rate contributes for half of the next value of3510	 * the estimated peak rate.3511	 *3512	 * So, the first step is to compute the weight as a function3513	 * of how sequential the workload is. Note that the weight3514	 * cannot reach 9, because bfqd->sequential_samples cannot3515	 * become equal to bfqd->peak_rate_samples, which, in its3516	 * turn, holds true because bfqd->sequential_samples is not3517	 * incremented for the first sample.3518	 */3519	weight = (9 * bfqd->sequential_samples) / bfqd->peak_rate_samples;3520 3521	/*3522	 * Second step: further refine the weight as a function of the3523	 * duration of the observation interval.3524	 */3525	weight = min_t(u32, 8,3526		       div_u64(weight * bfqd->delta_from_first,3527			       BFQ_RATE_REF_INTERVAL));3528 3529	/*3530	 * Divisor ranging from 10, for minimum weight, to 2, for3531	 * maximum weight.3532	 */3533	divisor = 10 - weight;3534 3535	/*3536	 * Finally, update peak rate:3537	 *3538	 * peak_rate = peak_rate * (divisor-1) / divisor  +  rate / divisor3539	 */3540	bfqd->peak_rate *= divisor-1;3541	bfqd->peak_rate /= divisor;3542	rate /= divisor; /* smoothing constant alpha = 1/divisor */3543 3544	bfqd->peak_rate += rate;3545 3546	/*3547	 * For a very slow device, bfqd->peak_rate can reach 0 (see3548	 * the minimum representable values reported in the comments3549	 * on BFQ_RATE_SHIFT). Push to 1 if this happens, to avoid3550	 * divisions by zero where bfqd->peak_rate is used as a3551	 * divisor.3552	 */3553	bfqd->peak_rate = max_t(u32, 1, bfqd->peak_rate);3554 3555	update_thr_responsiveness_params(bfqd);3556 3557reset_computation:3558	bfq_reset_rate_computation(bfqd, rq);3559}3560 3561/*3562 * Update the read/write peak rate (the main quantity used for3563 * auto-tuning, see update_thr_responsiveness_params()).3564 *3565 * It is not trivial to estimate the peak rate (correctly): because of3566 * the presence of sw and hw queues between the scheduler and the3567 * device components that finally serve I/O requests, it is hard to3568 * say exactly when a given dispatched request is served inside the3569 * device, and for how long. As a consequence, it is hard to know3570 * precisely at what rate a given set of requests is actually served3571 * by the device.3572 *3573 * On the opposite end, the dispatch time of any request is trivially3574 * available, and, from this piece of information, the "dispatch rate"3575 * of requests can be immediately computed. So, the idea in the next3576 * function is to use what is known, namely request dispatch times3577 * (plus, when useful, request completion times), to estimate what is3578 * unknown, namely in-device request service rate.3579 *3580 * The main issue is that, because of the above facts, the rate at3581 * which a certain set of requests is dispatched over a certain time3582 * interval can vary greatly with respect to the rate at which the3583 * same requests are then served. But, since the size of any3584 * intermediate queue is limited, and the service scheme is lossless3585 * (no request is silently dropped), the following obvious convergence3586 * property holds: the number of requests dispatched MUST become3587 * closer and closer to the number of requests completed as the3588 * observation interval grows. This is the key property used in3589 * the next function to estimate the peak service rate as a function3590 * of the observed dispatch rate. The function assumes to be invoked3591 * on every request dispatch.3592 */3593static void bfq_update_peak_rate(struct bfq_data *bfqd, struct request *rq)3594{3595	u64 now_ns = blk_time_get_ns();3596 3597	if (bfqd->peak_rate_samples == 0) { /* first dispatch */3598		bfq_log(bfqd, "update_peak_rate: goto reset, samples %d",3599			bfqd->peak_rate_samples);3600		bfq_reset_rate_computation(bfqd, rq);3601		goto update_last_values; /* will add one sample */3602	}3603 3604	/*3605	 * Device idle for very long: the observation interval lasting3606	 * up to this dispatch cannot be a valid observation interval3607	 * for computing a new peak rate (similarly to the late-3608	 * completion event in bfq_completed_request()). Go to3609	 * update_rate_and_reset to have the following three steps3610	 * taken:3611	 * - close the observation interval at the last (previous)3612	 *   request dispatch or completion3613	 * - compute rate, if possible, for that observation interval3614	 * - start a new observation interval with this dispatch3615	 */3616	if (now_ns - bfqd->last_dispatch > 100*NSEC_PER_MSEC &&3617	    bfqd->tot_rq_in_driver == 0)3618		goto update_rate_and_reset;3619 3620	/* Update sampling information */3621	bfqd->peak_rate_samples++;3622 3623	if ((bfqd->tot_rq_in_driver > 0 ||3624		now_ns - bfqd->last_completion < BFQ_MIN_TT)3625	    && !BFQ_RQ_SEEKY(bfqd, bfqd->last_position, rq))3626		bfqd->sequential_samples++;3627 3628	bfqd->tot_sectors_dispatched += blk_rq_sectors(rq);3629 3630	/* Reset max observed rq size every 32 dispatches */3631	if (likely(bfqd->peak_rate_samples % 32))3632		bfqd->last_rq_max_size =3633			max_t(u32, blk_rq_sectors(rq), bfqd->last_rq_max_size);3634	else3635		bfqd->last_rq_max_size = blk_rq_sectors(rq);3636 3637	bfqd->delta_from_first = now_ns - bfqd->first_dispatch;3638 3639	/* Target observation interval not yet reached, go on sampling */3640	if (bfqd->delta_from_first < BFQ_RATE_REF_INTERVAL)3641		goto update_last_values;3642 3643update_rate_and_reset:3644	bfq_update_rate_reset(bfqd, rq);3645update_last_values:3646	bfqd->last_position = blk_rq_pos(rq) + blk_rq_sectors(rq);3647	if (RQ_BFQQ(rq) == bfqd->in_service_queue)3648		bfqd->in_serv_last_pos = bfqd->last_position;3649	bfqd->last_dispatch = now_ns;3650}3651 3652/*3653 * Remove request from internal lists.3654 */3655static void bfq_dispatch_remove(struct request_queue *q, struct request *rq)3656{3657	struct bfq_queue *bfqq = RQ_BFQQ(rq);3658 3659	/*3660	 * For consistency, the next instruction should have been3661	 * executed after removing the request from the queue and3662	 * dispatching it.  We execute instead this instruction before3663	 * bfq_remove_request() (and hence introduce a temporary3664	 * inconsistency), for efficiency.  In fact, should this3665	 * dispatch occur for a non in-service bfqq, this anticipated3666	 * increment prevents two counters related to bfqq->dispatched3667	 * from risking to be, first, uselessly decremented, and then3668	 * incremented again when the (new) value of bfqq->dispatched3669	 * happens to be taken into account.3670	 */3671	bfqq->dispatched++;3672	bfq_update_peak_rate(q->elevator->elevator_data, rq);3673 3674	bfq_remove_request(q, rq);3675}3676 3677/*3678 * There is a case where idling does not have to be performed for3679 * throughput concerns, but to preserve the throughput share of3680 * the process associated with bfqq.3681 *3682 * To introduce this case, we can note that allowing the drive3683 * to enqueue more than one request at a time, and hence3684 * delegating de facto final scheduling decisions to the3685 * drive's internal scheduler, entails loss of control on the3686 * actual request service order. In particular, the critical3687 * situation is when requests from different processes happen3688 * to be present, at the same time, in the internal queue(s)3689 * of the drive. In such a situation, the drive, by deciding3690 * the service order of the internally-queued requests, does3691 * determine also the actual throughput distribution among3692 * these processes. But the drive typically has no notion or3693 * concern about per-process throughput distribution, and3694 * makes its decisions only on a per-request basis. Therefore,3695 * the service distribution enforced by the drive's internal3696 * scheduler is likely to coincide with the desired throughput3697 * distribution only in a completely symmetric, or favorably3698 * skewed scenario where:3699 * (i-a) each of these processes must get the same throughput as3700 *	 the others,3701 * (i-b) in case (i-a) does not hold, it holds that the process3702 *       associated with bfqq must receive a lower or equal3703 *	 throughput than any of the other processes;3704 * (ii)  the I/O of each process has the same properties, in3705 *       terms of locality (sequential or random), direction3706 *       (reads or writes), request sizes, greediness3707 *       (from I/O-bound to sporadic), and so on;3708 3709 * In fact, in such a scenario, the drive tends to treat the requests3710 * of each process in about the same way as the requests of the3711 * others, and thus to provide each of these processes with about the3712 * same throughput.  This is exactly the desired throughput3713 * distribution if (i-a) holds, or, if (i-b) holds instead, this is an3714 * even more convenient distribution for (the process associated with)3715 * bfqq.3716 *3717 * In contrast, in any asymmetric or unfavorable scenario, device3718 * idling (I/O-dispatch plugging) is certainly needed to guarantee3719 * that bfqq receives its assigned fraction of the device throughput3720 * (see [1] for details).3721 *3722 * The problem is that idling may significantly reduce throughput with3723 * certain combinations of types of I/O and devices. An important3724 * example is sync random I/O on flash storage with command3725 * queueing. So, unless bfqq falls in cases where idling also boosts3726 * throughput, it is important to check conditions (i-a), i(-b) and3727 * (ii) accurately, so as to avoid idling when not strictly needed for3728 * service guarantees.3729 *3730 * Unfortunately, it is extremely difficult to thoroughly check3731 * condition (ii). And, in case there are active groups, it becomes3732 * very difficult to check conditions (i-a) and (i-b) too.  In fact,3733 * if there are active groups, then, for conditions (i-a) or (i-b) to3734 * become false 'indirectly', it is enough that an active group3735 * contains more active processes or sub-groups than some other active3736 * group. More precisely, for conditions (i-a) or (i-b) to become3737 * false because of such a group, it is not even necessary that the3738 * group is (still) active: it is sufficient that, even if the group3739 * has become inactive, some of its descendant processes still have3740 * some request already dispatched but still waiting for3741 * completion. In fact, requests have still to be guaranteed their3742 * share of the throughput even after being dispatched. In this3743 * respect, it is easy to show that, if a group frequently becomes3744 * inactive while still having in-flight requests, and if, when this3745 * happens, the group is not considered in the calculation of whether3746 * the scenario is asymmetric, then the group may fail to be3747 * guaranteed its fair share of the throughput (basically because3748 * idling may not be performed for the descendant processes of the3749 * group, but it had to be).  We address this issue with the following3750 * bi-modal behavior, implemented in the function3751 * bfq_asymmetric_scenario().3752 *3753 * If there are groups with requests waiting for completion3754 * (as commented above, some of these groups may even be3755 * already inactive), then the scenario is tagged as3756 * asymmetric, conservatively, without checking any of the3757 * conditions (i-a), (i-b) or (ii). So the device is idled for bfqq.3758 * This behavior matches also the fact that groups are created3759 * exactly if controlling I/O is a primary concern (to3760 * preserve bandwidth and latency guarantees).3761 *3762 * On the opposite end, if there are no groups with requests waiting3763 * for completion, then only conditions (i-a) and (i-b) are actually3764 * controlled, i.e., provided that conditions (i-a) or (i-b) holds,3765 * idling is not performed, regardless of whether condition (ii)3766 * holds.  In other words, only if conditions (i-a) and (i-b) do not3767 * hold, then idling is allowed, and the device tends to be prevented3768 * from queueing many requests, possibly of several processes. Since3769 * there are no groups with requests waiting for completion, then, to3770 * control conditions (i-a) and (i-b) it is enough to check just3771 * whether all the queues with requests waiting for completion also3772 * have the same weight.3773 *3774 * Not checking condition (ii) evidently exposes bfqq to the3775 * risk of getting less throughput than its fair share.3776 * However, for queues with the same weight, a further3777 * mechanism, preemption, mitigates or even eliminates this3778 * problem. And it does so without consequences on overall3779 * throughput. This mechanism and its benefits are explained3780 * in the next three paragraphs.3781 *3782 * Even if a queue, say Q, is expired when it remains idle, Q3783 * can still preempt the new in-service queue if the next3784 * request of Q arrives soon (see the comments on3785 * bfq_bfqq_update_budg_for_activation). If all queues and3786 * groups have the same weight, this form of preemption,3787 * combined with the hole-recovery heuristic described in the3788 * comments on function bfq_bfqq_update_budg_for_activation,3789 * are enough to preserve a correct bandwidth distribution in3790 * the mid term, even without idling. In fact, even if not3791 * idling allows the internal queues of the device to contain3792 * many requests, and thus to reorder requests, we can rather3793 * safely assume that the internal scheduler still preserves a3794 * minimum of mid-term fairness.3795 *3796 * More precisely, this preemption-based, idleless approach3797 * provides fairness in terms of IOPS, and not sectors per3798 * second. This can be seen with a simple example. Suppose3799 * that there are two queues with the same weight, but that3800 * the first queue receives requests of 8 sectors, while the3801 * second queue receives requests of 1024 sectors. In3802 * addition, suppose that each of the two queues contains at3803 * most one request at a time, which implies that each queue3804 * always remains idle after it is served. Finally, after3805 * remaining idle, each queue receives very quickly a new3806 * request. It follows that the two queues are served3807 * alternatively, preempting each other if needed. This3808 * implies that, although both queues have the same weight,3809 * the queue with large requests receives a service that is3810 * 1024/8 times as high as the service received by the other3811 * queue.3812 *3813 * The motivation for using preemption instead of idling (for3814 * queues with the same weight) is that, by not idling,3815 * service guarantees are preserved (completely or at least in3816 * part) without minimally sacrificing throughput. And, if3817 * there is no active group, then the primary expectation for3818 * this device is probably a high throughput.3819 *3820 * We are now left only with explaining the two sub-conditions in the3821 * additional compound condition that is checked below for deciding3822 * whether the scenario is asymmetric. To explain the first3823 * sub-condition, we need to add that the function3824 * bfq_asymmetric_scenario checks the weights of only3825 * non-weight-raised queues, for efficiency reasons (see comments on3826 * bfq_weights_tree_add()). Then the fact that bfqq is weight-raised3827 * is checked explicitly here. More precisely, the compound condition3828 * below takes into account also the fact that, even if bfqq is being3829 * weight-raised, the scenario is still symmetric if all queues with3830 * requests waiting for completion happen to be3831 * weight-raised. Actually, we should be even more precise here, and3832 * differentiate between interactive weight raising and soft real-time3833 * weight raising.3834 *3835 * The second sub-condition checked in the compound condition is3836 * whether there is a fair amount of already in-flight I/O not3837 * belonging to bfqq. If so, I/O dispatching is to be plugged, for the3838 * following reason. The drive may decide to serve in-flight3839 * non-bfqq's I/O requests before bfqq's ones, thereby delaying the3840 * arrival of new I/O requests for bfqq (recall that bfqq is sync). If3841 * I/O-dispatching is not plugged, then, while bfqq remains empty, a3842 * basically uncontrolled amount of I/O from other queues may be3843 * dispatched too, possibly causing the service of bfqq's I/O to be3844 * delayed even longer in the drive. This problem gets more and more3845 * serious as the speed and the queue depth of the drive grow,3846 * because, as these two quantities grow, the probability to find no3847 * queue busy but many requests in flight grows too. By contrast,3848 * plugging I/O dispatching minimizes the delay induced by already3849 * in-flight I/O, and enables bfqq to recover the bandwidth it may3850 * lose because of this delay.3851 *3852 * As a side note, it is worth considering that the above3853 * device-idling countermeasures may however fail in the following3854 * unlucky scenario: if I/O-dispatch plugging is (correctly) disabled3855 * in a time period during which all symmetry sub-conditions hold, and3856 * therefore the device is allowed to enqueue many requests, but at3857 * some later point in time some sub-condition stops to hold, then it3858 * may become impossible to make requests be served in the desired3859 * order until all the requests already queued in the device have been3860 * served. The last sub-condition commented above somewhat mitigates3861 * this problem for weight-raised queues.3862 *3863 * However, as an additional mitigation for this problem, we preserve3864 * plugging for a special symmetric case that may suddenly turn into3865 * asymmetric: the case where only bfqq is busy. In this case, not3866 * expiring bfqq does not cause any harm to any other queues in terms3867 * of service guarantees. In contrast, it avoids the following unlucky3868 * sequence of events: (1) bfqq is expired, (2) a new queue with a3869 * lower weight than bfqq becomes busy (or more queues), (3) the new3870 * queue is served until a new request arrives for bfqq, (4) when bfqq3871 * is finally served, there are so many requests of the new queue in3872 * the drive that the pending requests for bfqq take a lot of time to3873 * be served. In particular, event (2) may case even already3874 * dispatched requests of bfqq to be delayed, inside the drive. So, to3875 * avoid this series of events, the scenario is preventively declared3876 * as asymmetric also if bfqq is the only busy queues3877 */3878static bool idling_needed_for_service_guarantees(struct bfq_data *bfqd,3879						 struct bfq_queue *bfqq)3880{3881	int tot_busy_queues = bfq_tot_busy_queues(bfqd);3882 3883	/* No point in idling for bfqq if it won't get requests any longer */3884	if (unlikely(!bfqq_process_refs(bfqq)))3885		return false;3886 3887	return (bfqq->wr_coeff > 1 &&3888		(bfqd->wr_busy_queues < tot_busy_queues ||3889		 bfqd->tot_rq_in_driver >= bfqq->dispatched + 4)) ||3890		bfq_asymmetric_scenario(bfqd, bfqq) ||3891		tot_busy_queues == 1;3892}3893 3894static bool __bfq_bfqq_expire(struct bfq_data *bfqd, struct bfq_queue *bfqq,3895			      enum bfqq_expiration reason)3896{3897	/*3898	 * If this bfqq is shared between multiple processes, check3899	 * to make sure that those processes are still issuing I/Os3900	 * within the mean seek distance. If not, it may be time to3901	 * break the queues apart again.3902	 */3903	if (bfq_bfqq_coop(bfqq) && BFQQ_SEEKY(bfqq))3904		bfq_mark_bfqq_split_coop(bfqq);3905 3906	/*3907	 * Consider queues with a higher finish virtual time than3908	 * bfqq. If idling_needed_for_service_guarantees(bfqq) returns3909	 * true, then bfqq's bandwidth would be violated if an3910	 * uncontrolled amount of I/O from these queues were3911	 * dispatched while bfqq is waiting for its new I/O to3912	 * arrive. This is exactly what may happen if this is a forced3913	 * expiration caused by a preemption attempt, and if bfqq is3914	 * not re-scheduled. To prevent this from happening, re-queue3915	 * bfqq if it needs I/O-dispatch plugging, even if it is3916	 * empty. By doing so, bfqq is granted to be served before the3917	 * above queues (provided that bfqq is of course eligible).3918	 */3919	if (RB_EMPTY_ROOT(&bfqq->sort_list) &&3920	    !(reason == BFQQE_PREEMPTED &&3921	      idling_needed_for_service_guarantees(bfqd, bfqq))) {3922		if (bfqq->dispatched == 0)3923			/*3924			 * Overloading budget_timeout field to store3925			 * the time at which the queue remains with no3926			 * backlog and no outstanding request; used by3927			 * the weight-raising mechanism.3928			 */3929			bfqq->budget_timeout = jiffies;3930 3931		bfq_del_bfqq_busy(bfqq, true);3932	} else {3933		bfq_requeue_bfqq(bfqd, bfqq, true);3934		/*3935		 * Resort priority tree of potential close cooperators.3936		 * See comments on bfq_pos_tree_add_move() for the unlikely().3937		 */3938		if (unlikely(!bfqd->nonrot_with_queueing &&3939			     !RB_EMPTY_ROOT(&bfqq->sort_list)))3940			bfq_pos_tree_add_move(bfqd, bfqq);3941	}3942 3943	/*3944	 * All in-service entities must have been properly deactivated3945	 * or requeued before executing the next function, which3946	 * resets all in-service entities as no more in service. This3947	 * may cause bfqq to be freed. If this happens, the next3948	 * function returns true.3949	 */3950	return __bfq_bfqd_reset_in_service(bfqd);3951}3952 3953/**3954 * __bfq_bfqq_recalc_budget - try to adapt the budget to the @bfqq behavior.3955 * @bfqd: device data.3956 * @bfqq: queue to update.3957 * @reason: reason for expiration.3958 *3959 * Handle the feedback on @bfqq budget at queue expiration.3960 * See the body for detailed comments.3961 */3962static void __bfq_bfqq_recalc_budget(struct bfq_data *bfqd,3963				     struct bfq_queue *bfqq,3964				     enum bfqq_expiration reason)3965{3966	struct request *next_rq;3967	int budget, min_budget;3968 3969	min_budget = bfq_min_budget(bfqd);3970 3971	if (bfqq->wr_coeff == 1)3972		budget = bfqq->max_budget;3973	else /*3974	      * Use a constant, low budget for weight-raised queues,3975	      * to help achieve a low latency. Keep it slightly higher3976	      * than the minimum possible budget, to cause a little3977	      * bit fewer expirations.3978	      */3979		budget = 2 * min_budget;3980 3981	bfq_log_bfqq(bfqd, bfqq, "recalc_budg: last budg %d, budg left %d",3982		bfqq->entity.budget, bfq_bfqq_budget_left(bfqq));3983	bfq_log_bfqq(bfqd, bfqq, "recalc_budg: last max_budg %d, min budg %d",3984		budget, bfq_min_budget(bfqd));3985	bfq_log_bfqq(bfqd, bfqq, "recalc_budg: sync %d, seeky %d",3986		bfq_bfqq_sync(bfqq), BFQQ_SEEKY(bfqd->in_service_queue));3987 3988	if (bfq_bfqq_sync(bfqq) && bfqq->wr_coeff == 1) {3989		switch (reason) {3990		/*3991		 * Caveat: in all the following cases we trade latency3992		 * for throughput.3993		 */3994		case BFQQE_TOO_IDLE:3995			/*3996			 * This is the only case where we may reduce3997			 * the budget: if there is no request of the3998			 * process still waiting for completion, then3999			 * we assume (tentatively) that the timer has4000			 * expired because the batch of requests of4001			 * the process could have been served with a4002			 * smaller budget.  Hence, betting that4003			 * process will behave in the same way when it4004			 * becomes backlogged again, we reduce its4005			 * next budget.  As long as we guess right,4006			 * this budget cut reduces the latency4007			 * experienced by the process.4008			 *4009			 * However, if there are still outstanding4010			 * requests, then the process may have not yet4011			 * issued its next request just because it is4012			 * still waiting for the completion of some of4013			 * the still outstanding ones.  So in this4014			 * subcase we do not reduce its budget, on the4015			 * contrary we increase it to possibly boost4016			 * the throughput, as discussed in the4017			 * comments to the BUDGET_TIMEOUT case.4018			 */4019			if (bfqq->dispatched > 0) /* still outstanding reqs */4020				budget = min(budget * 2, bfqd->bfq_max_budget);4021			else {4022				if (budget > 5 * min_budget)4023					budget -= 4 * min_budget;4024				else4025					budget = min_budget;4026			}4027			break;4028		case BFQQE_BUDGET_TIMEOUT:4029			/*4030			 * We double the budget here because it gives4031			 * the chance to boost the throughput if this4032			 * is not a seeky process (and has bumped into4033			 * this timeout because of, e.g., ZBR).4034			 */4035			budget = min(budget * 2, bfqd->bfq_max_budget);4036			break;4037		case BFQQE_BUDGET_EXHAUSTED:4038			/*4039			 * The process still has backlog, and did not4040			 * let either the budget timeout or the disk4041			 * idling timeout expire. Hence it is not4042			 * seeky, has a short thinktime and may be4043			 * happy with a higher budget too. So4044			 * definitely increase the budget of this good4045			 * candidate to boost the disk throughput.4046			 */4047			budget = min(budget * 4, bfqd->bfq_max_budget);4048			break;4049		case BFQQE_NO_MORE_REQUESTS:4050			/*4051			 * For queues that expire for this reason, it4052			 * is particularly important to keep the4053			 * budget close to the actual service they4054			 * need. Doing so reduces the timestamp4055			 * misalignment problem described in the4056			 * comments in the body of4057			 * __bfq_activate_entity. In fact, suppose4058			 * that a queue systematically expires for4059			 * BFQQE_NO_MORE_REQUESTS and presents a4060			 * new request in time to enjoy timestamp4061			 * back-shifting. The larger the budget of the4062			 * queue is with respect to the service the4063			 * queue actually requests in each service4064			 * slot, the more times the queue can be4065			 * reactivated with the same virtual finish4066			 * time. It follows that, even if this finish4067			 * time is pushed to the system virtual time4068			 * to reduce the consequent timestamp4069			 * misalignment, the queue unjustly enjoys for4070			 * many re-activations a lower finish time4071			 * than all newly activated queues.4072			 *4073			 * The service needed by bfqq is measured4074			 * quite precisely by bfqq->entity.service.4075			 * Since bfqq does not enjoy device idling,4076			 * bfqq->entity.service is equal to the number4077			 * of sectors that the process associated with4078			 * bfqq requested to read/write before waiting4079			 * for request completions, or blocking for4080			 * other reasons.4081			 */4082			budget = max_t(int, bfqq->entity.service, min_budget);4083			break;4084		default:4085			return;4086		}4087	} else if (!bfq_bfqq_sync(bfqq)) {4088		/*4089		 * Async queues get always the maximum possible4090		 * budget, as for them we do not care about latency4091		 * (in addition, their ability to dispatch is limited4092		 * by the charging factor).4093		 */4094		budget = bfqd->bfq_max_budget;4095	}4096 4097	bfqq->max_budget = budget;4098 4099	if (bfqd->budgets_assigned >= bfq_stats_min_budgets &&4100	    !bfqd->bfq_user_max_budget)4101		bfqq->max_budget = min(bfqq->max_budget, bfqd->bfq_max_budget);4102 4103	/*4104	 * If there is still backlog, then assign a new budget, making4105	 * sure that it is large enough for the next request.  Since4106	 * the finish time of bfqq must be kept in sync with the4107	 * budget, be sure to call __bfq_bfqq_expire() *after* this4108	 * update.4109	 *4110	 * If there is no backlog, then no need to update the budget;4111	 * it will be updated on the arrival of a new request.4112	 */4113	next_rq = bfqq->next_rq;4114	if (next_rq)4115		bfqq->entity.budget = max_t(unsigned long, bfqq->max_budget,4116					    bfq_serv_to_charge(next_rq, bfqq));4117 4118	bfq_log_bfqq(bfqd, bfqq, "head sect: %u, new budget %d",4119			next_rq ? blk_rq_sectors(next_rq) : 0,4120			bfqq->entity.budget);4121}4122 4123/*4124 * Return true if the process associated with bfqq is "slow". The slow4125 * flag is used, in addition to the budget timeout, to reduce the4126 * amount of service provided to seeky processes, and thus reduce4127 * their chances to lower the throughput. More details in the comments4128 * on the function bfq_bfqq_expire().4129 *4130 * An important observation is in order: as discussed in the comments4131 * on the function bfq_update_peak_rate(), with devices with internal4132 * queues, it is hard if ever possible to know when and for how long4133 * an I/O request is processed by the device (apart from the trivial4134 * I/O pattern where a new request is dispatched only after the4135 * previous one has been completed). This makes it hard to evaluate4136 * the real rate at which the I/O requests of each bfq_queue are4137 * served.  In fact, for an I/O scheduler like BFQ, serving a4138 * bfq_queue means just dispatching its requests during its service4139 * slot (i.e., until the budget of the queue is exhausted, or the4140 * queue remains idle, or, finally, a timeout fires). But, during the4141 * service slot of a bfq_queue, around 100 ms at most, the device may4142 * be even still processing requests of bfq_queues served in previous4143 * service slots. On the opposite end, the requests of the in-service4144 * bfq_queue may be completed after the service slot of the queue4145 * finishes.4146 *4147 * Anyway, unless more sophisticated solutions are used4148 * (where possible), the sum of the sizes of the requests dispatched4149 * during the service slot of a bfq_queue is probably the only4150 * approximation available for the service received by the bfq_queue4151 * during its service slot. And this sum is the quantity used in this4152 * function to evaluate the I/O speed of a process.4153 */4154static bool bfq_bfqq_is_slow(struct bfq_data *bfqd, struct bfq_queue *bfqq,4155				 bool compensate, unsigned long *delta_ms)4156{4157	ktime_t delta_ktime;4158	u32 delta_usecs;4159	bool slow = BFQQ_SEEKY(bfqq); /* if delta too short, use seekyness */4160 4161	if (!bfq_bfqq_sync(bfqq))4162		return false;4163 4164	if (compensate)4165		delta_ktime = bfqd->last_idling_start;4166	else4167		delta_ktime = blk_time_get();4168	delta_ktime = ktime_sub(delta_ktime, bfqd->last_budget_start);4169	delta_usecs = ktime_to_us(delta_ktime);4170 4171	/* don't use too short time intervals */4172	if (delta_usecs < 1000) {4173		if (blk_queue_nonrot(bfqd->queue))4174			 /*4175			  * give same worst-case guarantees as idling4176			  * for seeky4177			  */4178			*delta_ms = BFQ_MIN_TT / NSEC_PER_MSEC;4179		else /* charge at least one seek */4180			*delta_ms = bfq_slice_idle / NSEC_PER_MSEC;4181 4182		return slow;4183	}4184 4185	*delta_ms = delta_usecs / USEC_PER_MSEC;4186 4187	/*4188	 * Use only long (> 20ms) intervals to filter out excessive4189	 * spikes in service rate estimation.4190	 */4191	if (delta_usecs > 20000) {4192		/*4193		 * Caveat for rotational devices: processes doing I/O4194		 * in the slower disk zones tend to be slow(er) even4195		 * if not seeky. In this respect, the estimated peak4196		 * rate is likely to be an average over the disk4197		 * surface. Accordingly, to not be too harsh with4198		 * unlucky processes, a process is deemed slow only if4199		 * its rate has been lower than half of the estimated4200		 * peak rate.4201		 */4202		slow = bfqq->entity.service < bfqd->bfq_max_budget / 2;4203	}4204 4205	bfq_log_bfqq(bfqd, bfqq, "bfq_bfqq_is_slow: slow %d", slow);4206 4207	return slow;4208}4209 4210/*4211 * To be deemed as soft real-time, an application must meet two4212 * requirements. First, the application must not require an average4213 * bandwidth higher than the approximate bandwidth required to playback or4214 * record a compressed high-definition video.4215 * The next function is invoked on the completion of the last request of a4216 * batch, to compute the next-start time instant, soft_rt_next_start, such4217 * that, if the next request of the application does not arrive before4218 * soft_rt_next_start, then the above requirement on the bandwidth is met.4219 *4220 * The second requirement is that the request pattern of the application is4221 * isochronous, i.e., that, after issuing a request or a batch of requests,4222 * the application stops issuing new requests until all its pending requests4223 * have been completed. After that, the application may issue a new batch,4224 * and so on.4225 * For this reason the next function is invoked to compute4226 * soft_rt_next_start only for applications that meet this requirement,4227 * whereas soft_rt_next_start is set to infinity for applications that do4228 * not.4229 *4230 * Unfortunately, even a greedy (i.e., I/O-bound) application may4231 * happen to meet, occasionally or systematically, both the above4232 * bandwidth and isochrony requirements. This may happen at least in4233 * the following circumstances. First, if the CPU load is high. The4234 * application may stop issuing requests while the CPUs are busy4235 * serving other processes, then restart, then stop again for a while,4236 * and so on. The other circumstances are related to the storage4237 * device: the storage device is highly loaded or reaches a low-enough4238 * throughput with the I/O of the application (e.g., because the I/O4239 * is random and/or the device is slow). In all these cases, the4240 * I/O of the application may be simply slowed down enough to meet4241 * the bandwidth and isochrony requirements. To reduce the probability4242 * that greedy applications are deemed as soft real-time in these4243 * corner cases, a further rule is used in the computation of4244 * soft_rt_next_start: the return value of this function is forced to4245 * be higher than the maximum between the following two quantities.4246 *4247 * (a) Current time plus: (1) the maximum time for which the arrival4248 *     of a request is waited for when a sync queue becomes idle,4249 *     namely bfqd->bfq_slice_idle, and (2) a few extra jiffies. We4250 *     postpone for a moment the reason for adding a few extra4251 *     jiffies; we get back to it after next item (b).  Lower-bounding4252 *     the return value of this function with the current time plus4253 *     bfqd->bfq_slice_idle tends to filter out greedy applications,4254 *     because the latter issue their next request as soon as possible4255 *     after the last one has been completed. In contrast, a soft4256 *     real-time application spends some time processing data, after a4257 *     batch of its requests has been completed.4258 *4259 * (b) Current value of bfqq->soft_rt_next_start. As pointed out4260 *     above, greedy applications may happen to meet both the4261 *     bandwidth and isochrony requirements under heavy CPU or4262 *     storage-device load. In more detail, in these scenarios, these4263 *     applications happen, only for limited time periods, to do I/O4264 *     slowly enough to meet all the requirements described so far,4265 *     including the filtering in above item (a). These slow-speed4266 *     time intervals are usually interspersed between other time4267 *     intervals during which these applications do I/O at a very high4268 *     speed. Fortunately, exactly because of the high speed of the4269 *     I/O in the high-speed intervals, the values returned by this4270 *     function happen to be so high, near the end of any such4271 *     high-speed interval, to be likely to fall *after* the end of4272 *     the low-speed time interval that follows. These high values are4273 *     stored in bfqq->soft_rt_next_start after each invocation of4274 *     this function. As a consequence, if the last value of4275 *     bfqq->soft_rt_next_start is constantly used to lower-bound the4276 *     next value that this function may return, then, from the very4277 *     beginning of a low-speed interval, bfqq->soft_rt_next_start is4278 *     likely to be constantly kept so high that any I/O request4279 *     issued during the low-speed interval is considered as arriving4280 *     to soon for the application to be deemed as soft4281 *     real-time. Then, in the high-speed interval that follows, the4282 *     application will not be deemed as soft real-time, just because4283 *     it will do I/O at a high speed. And so on.4284 *4285 * Getting back to the filtering in item (a), in the following two4286 * cases this filtering might be easily passed by a greedy4287 * application, if the reference quantity was just4288 * bfqd->bfq_slice_idle:4289 * 1) HZ is so low that the duration of a jiffy is comparable to or4290 *    higher than bfqd->bfq_slice_idle. This happens, e.g., on slow4291 *    devices with HZ=100. The time granularity may be so coarse4292 *    that the approximation, in jiffies, of bfqd->bfq_slice_idle4293 *    is rather lower than the exact value.4294 * 2) jiffies, instead of increasing at a constant rate, may stop increasing4295 *    for a while, then suddenly 'jump' by several units to recover the lost4296 *    increments. This seems to happen, e.g., inside virtual machines.4297 * To address this issue, in the filtering in (a) we do not use as a4298 * reference time interval just bfqd->bfq_slice_idle, but4299 * bfqd->bfq_slice_idle plus a few jiffies. In particular, we add the4300 * minimum number of jiffies for which the filter seems to be quite4301 * precise also in embedded systems and KVM/QEMU virtual machines.4302 */4303static unsigned long bfq_bfqq_softrt_next_start(struct bfq_data *bfqd,4304						struct bfq_queue *bfqq)4305{4306	return max3(bfqq->soft_rt_next_start,4307		    bfqq->last_idle_bklogged +4308		    HZ * bfqq->service_from_backlogged /4309		    bfqd->bfq_wr_max_softrt_rate,4310		    jiffies + nsecs_to_jiffies(bfqq->bfqd->bfq_slice_idle) + 4);4311}4312 4313/**4314 * bfq_bfqq_expire - expire a queue.4315 * @bfqd: device owning the queue.4316 * @bfqq: the queue to expire.4317 * @compensate: if true, compensate for the time spent idling.4318 * @reason: the reason causing the expiration.4319 *4320 * If the process associated with bfqq does slow I/O (e.g., because it4321 * issues random requests), we charge bfqq with the time it has been4322 * in service instead of the service it has received (see4323 * bfq_bfqq_charge_time for details on how this goal is achieved). As4324 * a consequence, bfqq will typically get higher timestamps upon4325 * reactivation, and hence it will be rescheduled as if it had4326 * received more service than what it has actually received. In the4327 * end, bfqq receives less service in proportion to how slowly its4328 * associated process consumes its budgets (and hence how seriously it4329 * tends to lower the throughput). In addition, this time-charging4330 * strategy guarantees time fairness among slow processes. In4331 * contrast, if the process associated with bfqq is not slow, we4332 * charge bfqq exactly with the service it has received.4333 *4334 * Charging time to the first type of queues and the exact service to4335 * the other has the effect of using the WF2Q+ policy to schedule the4336 * former on a timeslice basis, without violating service domain4337 * guarantees among the latter.4338 */4339void bfq_bfqq_expire(struct bfq_data *bfqd,4340		     struct bfq_queue *bfqq,4341		     bool compensate,4342		     enum bfqq_expiration reason)4343{4344	bool slow;4345	unsigned long delta = 0;4346	struct bfq_entity *entity = &bfqq->entity;4347 4348	/*4349	 * Check whether the process is slow (see bfq_bfqq_is_slow).4350	 */4351	slow = bfq_bfqq_is_slow(bfqd, bfqq, compensate, &delta);4352 4353	/*4354	 * As above explained, charge slow (typically seeky) and4355	 * timed-out queues with the time and not the service4356	 * received, to favor sequential workloads.4357	 *4358	 * Processes doing I/O in the slower disk zones will tend to4359	 * be slow(er) even if not seeky. Therefore, since the4360	 * estimated peak rate is actually an average over the disk4361	 * surface, these processes may timeout just for bad luck. To4362	 * avoid punishing them, do not charge time to processes that4363	 * succeeded in consuming at least 2/3 of their budget. This4364	 * allows BFQ to preserve enough elasticity to still perform4365	 * bandwidth, and not time, distribution with little unlucky4366	 * or quasi-sequential processes.4367	 */4368	if (bfqq->wr_coeff == 1 &&4369	    (slow ||4370	     (reason == BFQQE_BUDGET_TIMEOUT &&4371	      bfq_bfqq_budget_left(bfqq) >=  entity->budget / 3)))4372		bfq_bfqq_charge_time(bfqd, bfqq, delta);4373 4374	if (bfqd->low_latency && bfqq->wr_coeff == 1)4375		bfqq->last_wr_start_finish = jiffies;4376 4377	if (bfqd->low_latency && bfqd->bfq_wr_max_softrt_rate > 0 &&4378	    RB_EMPTY_ROOT(&bfqq->sort_list)) {4379		/*4380		 * If we get here, and there are no outstanding4381		 * requests, then the request pattern is isochronous4382		 * (see the comments on the function4383		 * bfq_bfqq_softrt_next_start()). Therefore we can4384		 * compute soft_rt_next_start.4385		 *4386		 * If, instead, the queue still has outstanding4387		 * requests, then we have to wait for the completion4388		 * of all the outstanding requests to discover whether4389		 * the request pattern is actually isochronous.4390		 */4391		if (bfqq->dispatched == 0)4392			bfqq->soft_rt_next_start =4393				bfq_bfqq_softrt_next_start(bfqd, bfqq);4394		else if (bfqq->dispatched > 0) {4395			/*4396			 * Schedule an update of soft_rt_next_start to when4397			 * the task may be discovered to be isochronous.4398			 */4399			bfq_mark_bfqq_softrt_update(bfqq);4400		}4401	}4402 4403	bfq_log_bfqq(bfqd, bfqq,4404		"expire (%d, slow %d, num_disp %d, short_ttime %d)", reason,4405		slow, bfqq->dispatched, bfq_bfqq_has_short_ttime(bfqq));4406 4407	/*4408	 * bfqq expired, so no total service time needs to be computed4409	 * any longer: reset state machine for measuring total service4410	 * times.4411	 */4412	bfqd->rqs_injected = bfqd->wait_dispatch = false;4413	bfqd->waited_rq = NULL;4414 4415	/*4416	 * Increase, decrease or leave budget unchanged according to4417	 * reason.4418	 */4419	__bfq_bfqq_recalc_budget(bfqd, bfqq, reason);4420	if (__bfq_bfqq_expire(bfqd, bfqq, reason))4421		/* bfqq is gone, no more actions on it */4422		return;4423 4424	/* mark bfqq as waiting a request only if a bic still points to it */4425	if (!bfq_bfqq_busy(bfqq) &&4426	    reason != BFQQE_BUDGET_TIMEOUT &&4427	    reason != BFQQE_BUDGET_EXHAUSTED) {4428		bfq_mark_bfqq_non_blocking_wait_rq(bfqq);4429		/*4430		 * Not setting service to 0, because, if the next rq4431		 * arrives in time, the queue will go on receiving4432		 * service with this same budget (as if it never expired)4433		 */4434	} else4435		entity->service = 0;4436 4437	/*4438	 * Reset the received-service counter for every parent entity.4439	 * Differently from what happens with bfqq->entity.service,4440	 * the resetting of this counter never needs to be postponed4441	 * for parent entities. In fact, in case bfqq may have a4442	 * chance to go on being served using the last, partially4443	 * consumed budget, bfqq->entity.service needs to be kept,4444	 * because if bfqq then actually goes on being served using4445	 * the same budget, the last value of bfqq->entity.service is4446	 * needed to properly decrement bfqq->entity.budget by the4447	 * portion already consumed. In contrast, it is not necessary4448	 * to keep entity->service for parent entities too, because4449	 * the bubble up of the new value of bfqq->entity.budget will4450	 * make sure that the budgets of parent entities are correct,4451	 * even in case bfqq and thus parent entities go on receiving4452	 * service with the same budget.4453	 */4454	entity = entity->parent;4455	for_each_entity(entity)4456		entity->service = 0;4457}4458 4459/*4460 * Budget timeout is not implemented through a dedicated timer, but4461 * just checked on request arrivals and completions, as well as on4462 * idle timer expirations.4463 */4464static bool bfq_bfqq_budget_timeout(struct bfq_queue *bfqq)4465{4466	return time_is_before_eq_jiffies(bfqq->budget_timeout);4467}4468 4469/*4470 * If we expire a queue that is actively waiting (i.e., with the4471 * device idled) for the arrival of a new request, then we may incur4472 * the timestamp misalignment problem described in the body of the4473 * function __bfq_activate_entity. Hence we return true only if this4474 * condition does not hold, or if the queue is slow enough to deserve4475 * only to be kicked off for preserving a high throughput.4476 */4477static bool bfq_may_expire_for_budg_timeout(struct bfq_queue *bfqq)4478{4479	bfq_log_bfqq(bfqq->bfqd, bfqq,4480		"may_budget_timeout: wait_request %d left %d timeout %d",4481		bfq_bfqq_wait_request(bfqq),4482			bfq_bfqq_budget_left(bfqq) >=  bfqq->entity.budget / 3,4483		bfq_bfqq_budget_timeout(bfqq));4484 4485	return (!bfq_bfqq_wait_request(bfqq) ||4486		bfq_bfqq_budget_left(bfqq) >=  bfqq->entity.budget / 3)4487		&&4488		bfq_bfqq_budget_timeout(bfqq);4489}4490 4491static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd,4492					     struct bfq_queue *bfqq)4493{4494	bool rot_without_queueing =4495		!blk_queue_nonrot(bfqd->queue) && !bfqd->hw_tag,4496		bfqq_sequential_and_IO_bound,4497		idling_boosts_thr;4498 4499	/* No point in idling for bfqq if it won't get requests any longer */4500	if (unlikely(!bfqq_process_refs(bfqq)))4501		return false;4502 4503	bfqq_sequential_and_IO_bound = !BFQQ_SEEKY(bfqq) &&4504		bfq_bfqq_IO_bound(bfqq) && bfq_bfqq_has_short_ttime(bfqq);4505 4506	/*4507	 * The next variable takes into account the cases where idling4508	 * boosts the throughput.4509	 *4510	 * The value of the variable is computed considering, first, that4511	 * idling is virtually always beneficial for the throughput if:4512	 * (a) the device is not NCQ-capable and rotational, or4513	 * (b) regardless of the presence of NCQ, the device is rotational and4514	 *     the request pattern for bfqq is I/O-bound and sequential, or4515	 * (c) regardless of whether it is rotational, the device is4516	 *     not NCQ-capable and the request pattern for bfqq is4517	 *     I/O-bound and sequential.4518	 *4519	 * Secondly, and in contrast to the above item (b), idling an4520	 * NCQ-capable flash-based device would not boost the4521	 * throughput even with sequential I/O; rather it would lower4522	 * the throughput in proportion to how fast the device4523	 * is. Accordingly, the next variable is true if any of the4524	 * above conditions (a), (b) or (c) is true, and, in4525	 * particular, happens to be false if bfqd is an NCQ-capable4526	 * flash-based device.4527	 */4528	idling_boosts_thr = rot_without_queueing ||4529		((!blk_queue_nonrot(bfqd->queue) || !bfqd->hw_tag) &&4530		 bfqq_sequential_and_IO_bound);4531 4532	/*4533	 * The return value of this function is equal to that of4534	 * idling_boosts_thr, unless a special case holds. In this4535	 * special case, described below, idling may cause problems to4536	 * weight-raised queues.4537	 *4538	 * When the request pool is saturated (e.g., in the presence4539	 * of write hogs), if the processes associated with4540	 * non-weight-raised queues ask for requests at a lower rate,4541	 * then processes associated with weight-raised queues have a4542	 * higher probability to get a request from the pool4543	 * immediately (or at least soon) when they need one. Thus4544	 * they have a higher probability to actually get a fraction4545	 * of the device throughput proportional to their high4546	 * weight. This is especially true with NCQ-capable drives,4547	 * which enqueue several requests in advance, and further4548	 * reorder internally-queued requests.4549	 *4550	 * For this reason, we force to false the return value if4551	 * there are weight-raised busy queues. In this case, and if4552	 * bfqq is not weight-raised, this guarantees that the device4553	 * is not idled for bfqq (if, instead, bfqq is weight-raised,4554	 * then idling will be guaranteed by another variable, see4555	 * below). Combined with the timestamping rules of BFQ (see4556	 * [1] for details), this behavior causes bfqq, and hence any4557	 * sync non-weight-raised queue, to get a lower number of4558	 * requests served, and thus to ask for a lower number of4559	 * requests from the request pool, before the busy4560	 * weight-raised queues get served again. This often mitigates4561	 * starvation problems in the presence of heavy write4562	 * workloads and NCQ, thereby guaranteeing a higher4563	 * application and system responsiveness in these hostile4564	 * scenarios.4565	 */4566	return idling_boosts_thr &&4567		bfqd->wr_busy_queues == 0;4568}4569 4570/*4571 * For a queue that becomes empty, device idling is allowed only if4572 * this function returns true for that queue. As a consequence, since4573 * device idling plays a critical role for both throughput boosting4574 * and service guarantees, the return value of this function plays a4575 * critical role as well.4576 *4577 * In a nutshell, this function returns true only if idling is4578 * beneficial for throughput or, even if detrimental for throughput,4579 * idling is however necessary to preserve service guarantees (low4580 * latency, desired throughput distribution, ...). In particular, on4581 * NCQ-capable devices, this function tries to return false, so as to4582 * help keep the drives' internal queues full, whenever this helps the4583 * device boost the throughput without causing any service-guarantee4584 * issue.4585 *4586 * Most of the issues taken into account to get the return value of4587 * this function are not trivial. We discuss these issues in the two4588 * functions providing the main pieces of information needed by this4589 * function.4590 */4591static bool bfq_better_to_idle(struct bfq_queue *bfqq)4592{4593	struct bfq_data *bfqd = bfqq->bfqd;4594	bool idling_boosts_thr_with_no_issue, idling_needed_for_service_guar;4595 4596	/* No point in idling for bfqq if it won't get requests any longer */4597	if (unlikely(!bfqq_process_refs(bfqq)))4598		return false;4599 4600	if (unlikely(bfqd->strict_guarantees))4601		return true;4602 4603	/*4604	 * Idling is performed only if slice_idle > 0. In addition, we4605	 * do not idle if4606	 * (a) bfqq is async4607	 * (b) bfqq is in the idle io prio class: in this case we do4608	 * not idle because we want to minimize the bandwidth that4609	 * queues in this class can steal to higher-priority queues4610	 */4611	if (bfqd->bfq_slice_idle == 0 || !bfq_bfqq_sync(bfqq) ||4612	   bfq_class_idle(bfqq))4613		return false;4614 4615	idling_boosts_thr_with_no_issue =4616		idling_boosts_thr_without_issues(bfqd, bfqq);4617 4618	idling_needed_for_service_guar =4619		idling_needed_for_service_guarantees(bfqd, bfqq);4620 4621	/*4622	 * We have now the two components we need to compute the4623	 * return value of the function, which is true only if idling4624	 * either boosts the throughput (without issues), or is4625	 * necessary to preserve service guarantees.4626	 */4627	return idling_boosts_thr_with_no_issue ||4628		idling_needed_for_service_guar;4629}4630 4631/*4632 * If the in-service queue is empty but the function bfq_better_to_idle4633 * returns true, then:4634 * 1) the queue must remain in service and cannot be expired, and4635 * 2) the device must be idled to wait for the possible arrival of a new4636 *    request for the queue.4637 * See the comments on the function bfq_better_to_idle for the reasons4638 * why performing device idling is the best choice to boost the throughput4639 * and preserve service guarantees when bfq_better_to_idle itself4640 * returns true.4641 */4642static bool bfq_bfqq_must_idle(struct bfq_queue *bfqq)4643{4644	return RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_better_to_idle(bfqq);4645}4646 4647/*4648 * This function chooses the queue from which to pick the next extra4649 * I/O request to inject, if it finds a compatible queue. See the4650 * comments on bfq_update_inject_limit() for details on the injection4651 * mechanism, and for the definitions of the quantities mentioned4652 * below.4653 */4654static struct bfq_queue *4655bfq_choose_bfqq_for_injection(struct bfq_data *bfqd)4656{4657	struct bfq_queue *bfqq, *in_serv_bfqq = bfqd->in_service_queue;4658	unsigned int limit = in_serv_bfqq->inject_limit;4659	int i;4660 4661	/*4662	 * If4663	 * - bfqq is not weight-raised and therefore does not carry4664	 *   time-critical I/O,4665	 * or4666	 * - regardless of whether bfqq is weight-raised, bfqq has4667	 *   however a long think time, during which it can absorb the4668	 *   effect of an appropriate number of extra I/O requests4669	 *   from other queues (see bfq_update_inject_limit for4670	 *   details on the computation of this number);4671	 * then injection can be performed without restrictions.4672	 */4673	bool in_serv_always_inject = in_serv_bfqq->wr_coeff == 1 ||4674		!bfq_bfqq_has_short_ttime(in_serv_bfqq);4675 4676	/*4677	 * If4678	 * - the baseline total service time could not be sampled yet,4679	 *   so the inject limit happens to be still 0, and4680	 * - a lot of time has elapsed since the plugging of I/O4681	 *   dispatching started, so drive speed is being wasted4682	 *   significantly;4683	 * then temporarily raise inject limit to one request.4684	 */4685	if (limit == 0 && in_serv_bfqq->last_serv_time_ns == 0 &&4686	    bfq_bfqq_wait_request(in_serv_bfqq) &&4687	    time_is_before_eq_jiffies(bfqd->last_idling_start_jiffies +4688				      bfqd->bfq_slice_idle)4689		)4690		limit = 1;4691 4692	if (bfqd->tot_rq_in_driver >= limit)4693		return NULL;4694 4695	/*4696	 * Linear search of the source queue for injection; but, with4697	 * a high probability, very few steps are needed to find a4698	 * candidate queue, i.e., a queue with enough budget left for4699	 * its next request. In fact:4700	 * - BFQ dynamically updates the budget of every queue so as4701	 *   to accommodate the expected backlog of the queue;4702	 * - if a queue gets all its requests dispatched as injected4703	 *   service, then the queue is removed from the active list4704	 *   (and re-added only if it gets new requests, but then it4705	 *   is assigned again enough budget for its new backlog).4706	 */4707	for (i = 0; i < bfqd->num_actuators; i++) {4708		list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list)4709			if (!RB_EMPTY_ROOT(&bfqq->sort_list) &&4710				(in_serv_always_inject || bfqq->wr_coeff > 1) &&4711				bfq_serv_to_charge(bfqq->next_rq, bfqq) <=4712				bfq_bfqq_budget_left(bfqq)) {4713			/*4714			 * Allow for only one large in-flight request4715			 * on non-rotational devices, for the4716			 * following reason. On non-rotationl drives,4717			 * large requests take much longer than4718			 * smaller requests to be served. In addition,4719			 * the drive prefers to serve large requests4720			 * w.r.t. to small ones, if it can choose. So,4721			 * having more than one large requests queued4722			 * in the drive may easily make the next first4723			 * request of the in-service queue wait for so4724			 * long to break bfqq's service guarantees. On4725			 * the bright side, large requests let the4726			 * drive reach a very high throughput, even if4727			 * there is only one in-flight large request4728			 * at a time.4729			 */4730			if (blk_queue_nonrot(bfqd->queue) &&4731			    blk_rq_sectors(bfqq->next_rq) >=4732			    BFQQ_SECT_THR_NONROT &&4733			    bfqd->tot_rq_in_driver >= 1)4734				continue;4735			else {4736				bfqd->rqs_injected = true;4737				return bfqq;4738			}4739		}4740	}4741 4742	return NULL;4743}4744 4745static struct bfq_queue *4746bfq_find_active_bfqq_for_actuator(struct bfq_data *bfqd, int idx)4747{4748	struct bfq_queue *bfqq;4749 4750	if (bfqd->in_service_queue &&4751	    bfqd->in_service_queue->actuator_idx == idx)4752		return bfqd->in_service_queue;4753 4754	list_for_each_entry(bfqq, &bfqd->active_list[idx], bfqq_list) {4755		if (!RB_EMPTY_ROOT(&bfqq->sort_list) &&4756			bfq_serv_to_charge(bfqq->next_rq, bfqq) <=4757				bfq_bfqq_budget_left(bfqq)) {4758			return bfqq;4759		}4760	}4761 4762	return NULL;4763}4764 4765/*4766 * Perform a linear scan of each actuator, until an actuator is found4767 * for which the following three conditions hold: the load of the4768 * actuator is below the threshold (see comments on4769 * actuator_load_threshold for details) and lower than that of the4770 * next actuator (comments on this extra condition below), and there4771 * is a queue that contains I/O for that actuator. On success, return4772 * that queue.4773 *4774 * Performing a plain linear scan entails a prioritization among4775 * actuators. The extra condition above breaks this prioritization and4776 * tends to distribute injection uniformly across actuators.4777 */4778static struct bfq_queue *4779bfq_find_bfqq_for_underused_actuator(struct bfq_data *bfqd)4780{4781	int i;4782 4783	for (i = 0 ; i < bfqd->num_actuators; i++) {4784		if (bfqd->rq_in_driver[i] < bfqd->actuator_load_threshold &&4785		    (i == bfqd->num_actuators - 1 ||4786		     bfqd->rq_in_driver[i] < bfqd->rq_in_driver[i+1])) {4787			struct bfq_queue *bfqq =4788				bfq_find_active_bfqq_for_actuator(bfqd, i);4789 4790			if (bfqq)4791				return bfqq;4792		}4793	}4794 4795	return NULL;4796}4797 4798 4799/*4800 * Select a queue for service.  If we have a current queue in service,4801 * check whether to continue servicing it, or retrieve and set a new one.4802 */4803static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd)4804{4805	struct bfq_queue *bfqq, *inject_bfqq;4806	struct request *next_rq;4807	enum bfqq_expiration reason = BFQQE_BUDGET_TIMEOUT;4808 4809	bfqq = bfqd->in_service_queue;4810	if (!bfqq)4811		goto new_queue;4812 4813	bfq_log_bfqq(bfqd, bfqq, "select_queue: already in-service queue");4814 4815	/*4816	 * Do not expire bfqq for budget timeout if bfqq may be about4817	 * to enjoy device idling. The reason why, in this case, we4818	 * prevent bfqq from expiring is the same as in the comments4819	 * on the case where bfq_bfqq_must_idle() returns true, in4820	 * bfq_completed_request().4821	 */4822	if (bfq_may_expire_for_budg_timeout(bfqq) &&4823	    !bfq_bfqq_must_idle(bfqq))4824		goto expire;4825 4826check_queue:4827	/*4828	 *  If some actuator is underutilized, but the in-service4829	 *  queue does not contain I/O for that actuator, then try to4830	 *  inject I/O for that actuator.4831	 */4832	inject_bfqq = bfq_find_bfqq_for_underused_actuator(bfqd);4833	if (inject_bfqq && inject_bfqq != bfqq)4834		return inject_bfqq;4835 4836	/*4837	 * This loop is rarely executed more than once. Even when it4838	 * happens, it is much more convenient to re-execute this loop4839	 * than to return NULL and trigger a new dispatch to get a4840	 * request served.4841	 */4842	next_rq = bfqq->next_rq;4843	/*4844	 * If bfqq has requests queued and it has enough budget left to4845	 * serve them, keep the queue, otherwise expire it.4846	 */4847	if (next_rq) {4848		if (bfq_serv_to_charge(next_rq, bfqq) >4849			bfq_bfqq_budget_left(bfqq)) {4850			/*4851			 * Expire the queue for budget exhaustion,4852			 * which makes sure that the next budget is4853			 * enough to serve the next request, even if4854			 * it comes from the fifo expired path.4855			 */4856			reason = BFQQE_BUDGET_EXHAUSTED;4857			goto expire;4858		} else {4859			/*4860			 * The idle timer may be pending because we may4861			 * not disable disk idling even when a new request4862			 * arrives.4863			 */4864			if (bfq_bfqq_wait_request(bfqq)) {4865				/*4866				 * If we get here: 1) at least a new request4867				 * has arrived but we have not disabled the4868				 * timer because the request was too small,4869				 * 2) then the block layer has unplugged4870				 * the device, causing the dispatch to be4871				 * invoked.4872				 *4873				 * Since the device is unplugged, now the4874				 * requests are probably large enough to4875				 * provide a reasonable throughput.4876				 * So we disable idling.4877				 */4878				bfq_clear_bfqq_wait_request(bfqq);4879				hrtimer_try_to_cancel(&bfqd->idle_slice_timer);4880			}4881			goto keep_queue;4882		}4883	}4884 4885	/*4886	 * No requests pending. However, if the in-service queue is idling4887	 * for a new request, or has requests waiting for a completion and4888	 * may idle after their completion, then keep it anyway.4889	 *4890	 * Yet, inject service from other queues if it boosts4891	 * throughput and is possible.4892	 */4893	if (bfq_bfqq_wait_request(bfqq) ||4894	    (bfqq->dispatched != 0 && bfq_better_to_idle(bfqq))) {4895		unsigned int act_idx = bfqq->actuator_idx;4896		struct bfq_queue *async_bfqq = NULL;4897		struct bfq_queue *blocked_bfqq =4898			!hlist_empty(&bfqq->woken_list) ?4899			container_of(bfqq->woken_list.first,4900				     struct bfq_queue,4901				     woken_list_node)4902			: NULL;4903 4904		if (bfqq->bic && bfqq->bic->bfqq[0][act_idx] &&4905		    bfq_bfqq_busy(bfqq->bic->bfqq[0][act_idx]) &&4906		    bfqq->bic->bfqq[0][act_idx]->next_rq)4907			async_bfqq = bfqq->bic->bfqq[0][act_idx];4908		/*4909		 * The next four mutually-exclusive ifs decide4910		 * whether to try injection, and choose the queue to4911		 * pick an I/O request from.4912		 *4913		 * The first if checks whether the process associated4914		 * with bfqq has also async I/O pending. If so, it4915		 * injects such I/O unconditionally. Injecting async4916		 * I/O from the same process can cause no harm to the4917		 * process. On the contrary, it can only increase4918		 * bandwidth and reduce latency for the process.4919		 *4920		 * The second if checks whether there happens to be a4921		 * non-empty waker queue for bfqq, i.e., a queue whose4922		 * I/O needs to be completed for bfqq to receive new4923		 * I/O. This happens, e.g., if bfqq is associated with4924		 * a process that does some sync. A sync generates4925		 * extra blocking I/O, which must be completed before4926		 * the process associated with bfqq can go on with its4927		 * I/O. If the I/O of the waker queue is not served,4928		 * then bfqq remains empty, and no I/O is dispatched,4929		 * until the idle timeout fires for bfqq. This is4930		 * likely to result in lower bandwidth and higher4931		 * latencies for bfqq, and in a severe loss of total4932		 * throughput. The best action to take is therefore to4933		 * serve the waker queue as soon as possible. So do it4934		 * (without relying on the third alternative below for4935		 * eventually serving waker_bfqq's I/O; see the last4936		 * paragraph for further details). This systematic4937		 * injection of I/O from the waker queue does not4938		 * cause any delay to bfqq's I/O. On the contrary,4939		 * next bfqq's I/O is brought forward dramatically,4940		 * for it is not blocked for milliseconds.4941		 *4942		 * The third if checks whether there is a queue woken4943		 * by bfqq, and currently with pending I/O. Such a4944		 * woken queue does not steal bandwidth from bfqq,4945		 * because it remains soon without I/O if bfqq is not4946		 * served. So there is virtually no risk of loss of4947		 * bandwidth for bfqq if this woken queue has I/O4948		 * dispatched while bfqq is waiting for new I/O.4949		 *4950		 * The fourth if checks whether bfqq is a queue for4951		 * which it is better to avoid injection. It is so if4952		 * bfqq delivers more throughput when served without4953		 * any further I/O from other queues in the middle, or4954		 * if the service times of bfqq's I/O requests both4955		 * count more than overall throughput, and may be4956		 * easily increased by injection (this happens if bfqq4957		 * has a short think time). If none of these4958		 * conditions holds, then a candidate queue for4959		 * injection is looked for through4960		 * bfq_choose_bfqq_for_injection(). Note that the4961		 * latter may return NULL (for example if the inject4962		 * limit for bfqq is currently 0).4963		 *4964		 * NOTE: motivation for the second alternative4965		 *4966		 * Thanks to the way the inject limit is updated in4967		 * bfq_update_has_short_ttime(), it is rather likely4968		 * that, if I/O is being plugged for bfqq and the4969		 * waker queue has pending I/O requests that are4970		 * blocking bfqq's I/O, then the fourth alternative4971		 * above lets the waker queue get served before the4972		 * I/O-plugging timeout fires. So one may deem the4973		 * second alternative superfluous. It is not, because4974		 * the fourth alternative may be way less effective in4975		 * case of a synchronization. For two main4976		 * reasons. First, throughput may be low because the4977		 * inject limit may be too low to guarantee the same4978		 * amount of injected I/O, from the waker queue or4979		 * other queues, that the second alternative4980		 * guarantees (the second alternative unconditionally4981		 * injects a pending I/O request of the waker queue4982		 * for each bfq_dispatch_request()). Second, with the4983		 * fourth alternative, the duration of the plugging,4984		 * i.e., the time before bfqq finally receives new I/O,4985		 * may not be minimized, because the waker queue may4986		 * happen to be served only after other queues.4987		 */4988		if (async_bfqq &&4989		    icq_to_bic(async_bfqq->next_rq->elv.icq) == bfqq->bic &&4990		    bfq_serv_to_charge(async_bfqq->next_rq, async_bfqq) <=4991		    bfq_bfqq_budget_left(async_bfqq))4992			bfqq = async_bfqq;4993		else if (bfqq->waker_bfqq &&4994			   bfq_bfqq_busy(bfqq->waker_bfqq) &&4995			   bfqq->waker_bfqq->next_rq &&4996			   bfq_serv_to_charge(bfqq->waker_bfqq->next_rq,4997					      bfqq->waker_bfqq) <=4998			   bfq_bfqq_budget_left(bfqq->waker_bfqq)4999			)5000			bfqq = bfqq->waker_bfqq;5001		else if (blocked_bfqq &&5002			   bfq_bfqq_busy(blocked_bfqq) &&5003			   blocked_bfqq->next_rq &&5004			   bfq_serv_to_charge(blocked_bfqq->next_rq,5005					      blocked_bfqq) <=5006			   bfq_bfqq_budget_left(blocked_bfqq)5007			)5008			bfqq = blocked_bfqq;5009		else if (!idling_boosts_thr_without_issues(bfqd, bfqq) &&5010			 (bfqq->wr_coeff == 1 || bfqd->wr_busy_queues > 1 ||5011			  !bfq_bfqq_has_short_ttime(bfqq)))5012			bfqq = bfq_choose_bfqq_for_injection(bfqd);5013		else5014			bfqq = NULL;5015 5016		goto keep_queue;5017	}5018 5019	reason = BFQQE_NO_MORE_REQUESTS;5020expire:5021	bfq_bfqq_expire(bfqd, bfqq, false, reason);5022new_queue:5023	bfqq = bfq_set_in_service_queue(bfqd);5024	if (bfqq) {5025		bfq_log_bfqq(bfqd, bfqq, "select_queue: checking new queue");5026		goto check_queue;5027	}5028keep_queue:5029	if (bfqq)5030		bfq_log_bfqq(bfqd, bfqq, "select_queue: returned this queue");5031	else5032		bfq_log(bfqd, "select_queue: no queue returned");5033 5034	return bfqq;5035}5036 5037static void bfq_update_wr_data(struct bfq_data *bfqd, struct bfq_queue *bfqq)5038{5039	struct bfq_entity *entity = &bfqq->entity;5040 5041	if (bfqq->wr_coeff > 1) { /* queue is being weight-raised */5042		bfq_log_bfqq(bfqd, bfqq,5043			"raising period dur %u/%u msec, old coeff %u, w %d(%d)",5044			jiffies_to_msecs(jiffies - bfqq->last_wr_start_finish),5045			jiffies_to_msecs(bfqq->wr_cur_max_time),5046			bfqq->wr_coeff,5047			bfqq->entity.weight, bfqq->entity.orig_weight);5048 5049		if (entity->prio_changed)5050			bfq_log_bfqq(bfqd, bfqq, "WARN: pending prio change");5051 5052		/*5053		 * If the queue was activated in a burst, or too much5054		 * time has elapsed from the beginning of this5055		 * weight-raising period, then end weight raising.5056		 */5057		if (bfq_bfqq_in_large_burst(bfqq))5058			bfq_bfqq_end_wr(bfqq);5059		else if (time_is_before_jiffies(bfqq->last_wr_start_finish +5060						bfqq->wr_cur_max_time)) {5061			if (bfqq->wr_cur_max_time != bfqd->bfq_wr_rt_max_time ||5062			time_is_before_jiffies(bfqq->wr_start_at_switch_to_srt +5063					       bfq_wr_duration(bfqd))) {5064				/*5065				 * Either in interactive weight5066				 * raising, or in soft_rt weight5067				 * raising with the5068				 * interactive-weight-raising period5069				 * elapsed (so no switch back to5070				 * interactive weight raising).5071				 */5072				bfq_bfqq_end_wr(bfqq);5073			} else { /*5074				  * soft_rt finishing while still in5075				  * interactive period, switch back to5076				  * interactive weight raising5077				  */5078				switch_back_to_interactive_wr(bfqq, bfqd);5079				bfqq->entity.prio_changed = 1;5080			}5081		}5082		if (bfqq->wr_coeff > 1 &&5083		    bfqq->wr_cur_max_time != bfqd->bfq_wr_rt_max_time &&5084		    bfqq->service_from_wr > max_service_from_wr) {5085			/* see comments on max_service_from_wr */5086			bfq_bfqq_end_wr(bfqq);5087		}5088	}5089	/*5090	 * To improve latency (for this or other queues), immediately5091	 * update weight both if it must be raised and if it must be5092	 * lowered. Since, entity may be on some active tree here, and5093	 * might have a pending change of its ioprio class, invoke5094	 * next function with the last parameter unset (see the5095	 * comments on the function).5096	 */5097	if ((entity->weight > entity->orig_weight) != (bfqq->wr_coeff > 1))5098		__bfq_entity_update_weight_prio(bfq_entity_service_tree(entity),5099						entity, false);5100}5101 5102/*5103 * Dispatch next request from bfqq.5104 */5105static struct request *bfq_dispatch_rq_from_bfqq(struct bfq_data *bfqd,5106						 struct bfq_queue *bfqq)5107{5108	struct request *rq = bfqq->next_rq;5109	unsigned long service_to_charge;5110 5111	service_to_charge = bfq_serv_to_charge(rq, bfqq);5112 5113	bfq_bfqq_served(bfqq, service_to_charge);5114 5115	if (bfqq == bfqd->in_service_queue && bfqd->wait_dispatch) {5116		bfqd->wait_dispatch = false;5117		bfqd->waited_rq = rq;5118	}5119 5120	bfq_dispatch_remove(bfqd->queue, rq);5121 5122	if (bfqq != bfqd->in_service_queue)5123		return rq;5124 5125	/*5126	 * If weight raising has to terminate for bfqq, then next5127	 * function causes an immediate update of bfqq's weight,5128	 * without waiting for next activation. As a consequence, on5129	 * expiration, bfqq will be timestamped as if has never been5130	 * weight-raised during this service slot, even if it has5131	 * received part or even most of the service as a5132	 * weight-raised queue. This inflates bfqq's timestamps, which5133	 * is beneficial, as bfqq is then more willing to leave the5134	 * device immediately to possible other weight-raised queues.5135	 */5136	bfq_update_wr_data(bfqd, bfqq);5137 5138	/*5139	 * Expire bfqq, pretending that its budget expired, if bfqq5140	 * belongs to CLASS_IDLE and other queues are waiting for5141	 * service.5142	 */5143	if (bfq_tot_busy_queues(bfqd) > 1 && bfq_class_idle(bfqq))5144		bfq_bfqq_expire(bfqd, bfqq, false, BFQQE_BUDGET_EXHAUSTED);5145 5146	return rq;5147}5148 5149static bool bfq_has_work(struct blk_mq_hw_ctx *hctx)5150{5151	struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;5152 5153	/*5154	 * Avoiding lock: a race on bfqd->queued should cause at5155	 * most a call to dispatch for nothing5156	 */5157	return !list_empty_careful(&bfqd->dispatch) ||5158		READ_ONCE(bfqd->queued);5159}5160 5161static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx)5162{5163	struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;5164	struct request *rq = NULL;5165	struct bfq_queue *bfqq = NULL;5166 5167	if (!list_empty(&bfqd->dispatch)) {5168		rq = list_first_entry(&bfqd->dispatch, struct request,5169				      queuelist);5170		list_del_init(&rq->queuelist);5171 5172		bfqq = RQ_BFQQ(rq);5173 5174		if (bfqq) {5175			/*5176			 * Increment counters here, because this5177			 * dispatch does not follow the standard5178			 * dispatch flow (where counters are5179			 * incremented)5180			 */5181			bfqq->dispatched++;5182 5183			goto inc_in_driver_start_rq;5184		}5185 5186		/*5187		 * We exploit the bfq_finish_requeue_request hook to5188		 * decrement tot_rq_in_driver, but5189		 * bfq_finish_requeue_request will not be invoked on5190		 * this request. So, to avoid unbalance, just start5191		 * this request, without incrementing tot_rq_in_driver. As5192		 * a negative consequence, tot_rq_in_driver is deceptively5193		 * lower than it should be while this request is in5194		 * service. This may cause bfq_schedule_dispatch to be5195		 * invoked uselessly.5196		 *5197		 * As for implementing an exact solution, the5198		 * bfq_finish_requeue_request hook, if defined, is5199		 * probably invoked also on this request. So, by5200		 * exploiting this hook, we could 1) increment5201		 * tot_rq_in_driver here, and 2) decrement it in5202		 * bfq_finish_requeue_request. Such a solution would5203		 * let the value of the counter be always accurate,5204		 * but it would entail using an extra interface5205		 * function. This cost seems higher than the benefit,5206		 * being the frequency of non-elevator-private5207		 * requests very low.5208		 */5209		goto start_rq;5210	}5211 5212	bfq_log(bfqd, "dispatch requests: %d busy queues",5213		bfq_tot_busy_queues(bfqd));5214 5215	if (bfq_tot_busy_queues(bfqd) == 0)5216		goto exit;5217 5218	/*5219	 * Force device to serve one request at a time if5220	 * strict_guarantees is true. Forcing this service scheme is5221	 * currently the ONLY way to guarantee that the request5222	 * service order enforced by the scheduler is respected by a5223	 * queueing device. Otherwise the device is free even to make5224	 * some unlucky request wait for as long as the device5225	 * wishes.5226	 *5227	 * Of course, serving one request at a time may cause loss of5228	 * throughput.5229	 */5230	if (bfqd->strict_guarantees && bfqd->tot_rq_in_driver > 0)5231		goto exit;5232 5233	bfqq = bfq_select_queue(bfqd);5234	if (!bfqq)5235		goto exit;5236 5237	rq = bfq_dispatch_rq_from_bfqq(bfqd, bfqq);5238 5239	if (rq) {5240inc_in_driver_start_rq:5241		bfqd->rq_in_driver[bfqq->actuator_idx]++;5242		bfqd->tot_rq_in_driver++;5243start_rq:5244		rq->rq_flags |= RQF_STARTED;5245	}5246exit:5247	return rq;5248}5249 5250#ifdef CONFIG_BFQ_CGROUP_DEBUG5251static void bfq_update_dispatch_stats(struct request_queue *q,5252				      struct request *rq,5253				      struct bfq_queue *in_serv_queue,5254				      bool idle_timer_disabled)5255{5256	struct bfq_queue *bfqq = rq ? RQ_BFQQ(rq) : NULL;5257 5258	if (!idle_timer_disabled && !bfqq)5259		return;5260 5261	/*5262	 * rq and bfqq are guaranteed to exist until this function5263	 * ends, for the following reasons. First, rq can be5264	 * dispatched to the device, and then can be completed and5265	 * freed, only after this function ends. Second, rq cannot be5266	 * merged (and thus freed because of a merge) any longer,5267	 * because it has already started. Thus rq cannot be freed5268	 * before this function ends, and, since rq has a reference to5269	 * bfqq, the same guarantee holds for bfqq too.5270	 *5271	 * In addition, the following queue lock guarantees that5272	 * bfqq_group(bfqq) exists as well.5273	 */5274	spin_lock_irq(&q->queue_lock);5275	if (idle_timer_disabled)5276		/*5277		 * Since the idle timer has been disabled,5278		 * in_serv_queue contained some request when5279		 * __bfq_dispatch_request was invoked above, which5280		 * implies that rq was picked exactly from5281		 * in_serv_queue. Thus in_serv_queue == bfqq, and is5282		 * therefore guaranteed to exist because of the above5283		 * arguments.5284		 */5285		bfqg_stats_update_idle_time(bfqq_group(in_serv_queue));5286	if (bfqq) {5287		struct bfq_group *bfqg = bfqq_group(bfqq);5288 5289		bfqg_stats_update_avg_queue_size(bfqg);5290		bfqg_stats_set_start_empty_time(bfqg);5291		bfqg_stats_update_io_remove(bfqg, rq->cmd_flags);5292	}5293	spin_unlock_irq(&q->queue_lock);5294}5295#else5296static inline void bfq_update_dispatch_stats(struct request_queue *q,5297					     struct request *rq,5298					     struct bfq_queue *in_serv_queue,5299					     bool idle_timer_disabled) {}5300#endif /* CONFIG_BFQ_CGROUP_DEBUG */5301 5302static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx)5303{5304	struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;5305	struct request *rq;5306	struct bfq_queue *in_serv_queue;5307	bool waiting_rq, idle_timer_disabled = false;5308 5309	spin_lock_irq(&bfqd->lock);5310 5311	in_serv_queue = bfqd->in_service_queue;5312	waiting_rq = in_serv_queue && bfq_bfqq_wait_request(in_serv_queue);5313 5314	rq = __bfq_dispatch_request(hctx);5315	if (in_serv_queue == bfqd->in_service_queue) {5316		idle_timer_disabled =5317			waiting_rq && !bfq_bfqq_wait_request(in_serv_queue);5318	}5319 5320	spin_unlock_irq(&bfqd->lock);5321	bfq_update_dispatch_stats(hctx->queue, rq,5322			idle_timer_disabled ? in_serv_queue : NULL,5323				idle_timer_disabled);5324 5325	return rq;5326}5327 5328/*5329 * Task holds one reference to the queue, dropped when task exits.  Each rq5330 * in-flight on this queue also holds a reference, dropped when rq is freed.5331 *5332 * Scheduler lock must be held here. Recall not to use bfqq after calling5333 * this function on it.5334 */5335void bfq_put_queue(struct bfq_queue *bfqq)5336{5337	struct bfq_queue *item;5338	struct hlist_node *n;5339	struct bfq_group *bfqg = bfqq_group(bfqq);5340 5341	bfq_log_bfqq(bfqq->bfqd, bfqq, "put_queue: %p %d", bfqq, bfqq->ref);5342 5343	bfqq->ref--;5344	if (bfqq->ref)5345		return;5346 5347	if (!hlist_unhashed(&bfqq->burst_list_node)) {5348		hlist_del_init(&bfqq->burst_list_node);5349		/*5350		 * Decrement also burst size after the removal, if the5351		 * process associated with bfqq is exiting, and thus5352		 * does not contribute to the burst any longer. This5353		 * decrement helps filter out false positives of large5354		 * bursts, when some short-lived process (often due to5355		 * the execution of commands by some service) happens5356		 * to start and exit while a complex application is5357		 * starting, and thus spawning several processes that5358		 * do I/O (and that *must not* be treated as a large5359		 * burst, see comments on bfq_handle_burst).5360		 *5361		 * In particular, the decrement is performed only if:5362		 * 1) bfqq is not a merged queue, because, if it is,5363		 * then this free of bfqq is not triggered by the exit5364		 * of the process bfqq is associated with, but exactly5365		 * by the fact that bfqq has just been merged.5366		 * 2) burst_size is greater than 0, to handle5367		 * unbalanced decrements. Unbalanced decrements may5368		 * happen in te following case: bfqq is inserted into5369		 * the current burst list--without incrementing5370		 * bust_size--because of a split, but the current5371		 * burst list is not the burst list bfqq belonged to5372		 * (see comments on the case of a split in5373		 * bfq_set_request).5374		 */5375		if (bfqq->bic && bfqq->bfqd->burst_size > 0)5376			bfqq->bfqd->burst_size--;5377	}5378 5379	/*5380	 * bfqq does not exist any longer, so it cannot be woken by5381	 * any other queue, and cannot wake any other queue. Then bfqq5382	 * must be removed from the woken list of its possible waker5383	 * queue, and all queues in the woken list of bfqq must stop5384	 * having a waker queue. Strictly speaking, these updates5385	 * should be performed when bfqq remains with no I/O source5386	 * attached to it, which happens before bfqq gets freed. In5387	 * particular, this happens when the last process associated5388	 * with bfqq exits or gets associated with a different5389	 * queue. However, both events lead to bfqq being freed soon,5390	 * and dangling references would come out only after bfqq gets5391	 * freed. So these updates are done here, as a simple and safe5392	 * way to handle all cases.5393	 */5394	/* remove bfqq from woken list */5395	if (!hlist_unhashed(&bfqq->woken_list_node))5396		hlist_del_init(&bfqq->woken_list_node);5397 5398	/* reset waker for all queues in woken list */5399	hlist_for_each_entry_safe(item, n, &bfqq->woken_list,5400				  woken_list_node) {5401		item->waker_bfqq = NULL;5402		hlist_del_init(&item->woken_list_node);5403	}5404 5405	if (bfqq->bfqd->last_completed_rq_bfqq == bfqq)5406		bfqq->bfqd->last_completed_rq_bfqq = NULL;5407 5408	WARN_ON_ONCE(!list_empty(&bfqq->fifo));5409	WARN_ON_ONCE(!RB_EMPTY_ROOT(&bfqq->sort_list));5410	WARN_ON_ONCE(bfqq->dispatched);5411 5412	kmem_cache_free(bfq_pool, bfqq);5413	bfqg_and_blkg_put(bfqg);5414}5415 5416static void bfq_put_stable_ref(struct bfq_queue *bfqq)5417{5418	bfqq->stable_ref--;5419	bfq_put_queue(bfqq);5420}5421 5422void bfq_put_cooperator(struct bfq_queue *bfqq)5423{5424	struct bfq_queue *__bfqq, *next;5425 5426	/*5427	 * If this queue was scheduled to merge with another queue, be5428	 * sure to drop the reference taken on that queue (and others in5429	 * the merge chain). See bfq_setup_merge and bfq_merge_bfqqs.5430	 */5431	__bfqq = bfqq->new_bfqq;5432	while (__bfqq) {5433		next = __bfqq->new_bfqq;5434		bfq_put_queue(__bfqq);5435		__bfqq = next;5436	}5437 5438	bfq_release_process_ref(bfqq->bfqd, bfqq);5439}5440 5441static void bfq_exit_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq)5442{5443	if (bfqq == bfqd->in_service_queue) {5444		__bfq_bfqq_expire(bfqd, bfqq, BFQQE_BUDGET_TIMEOUT);5445		bfq_schedule_dispatch(bfqd);5446	}5447 5448	bfq_log_bfqq(bfqd, bfqq, "exit_bfqq: %p, %d", bfqq, bfqq->ref);5449 5450	bfq_put_cooperator(bfqq);5451}5452 5453static void bfq_exit_icq_bfqq(struct bfq_io_cq *bic, bool is_sync,5454			      unsigned int actuator_idx)5455{5456	struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, actuator_idx);5457	struct bfq_data *bfqd;5458 5459	if (bfqq)5460		bfqd = bfqq->bfqd; /* NULL if scheduler already exited */5461 5462	if (bfqq && bfqd) {5463		bic_set_bfqq(bic, NULL, is_sync, actuator_idx);5464		bfq_exit_bfqq(bfqd, bfqq);5465	}5466}5467 5468static void _bfq_exit_icq(struct bfq_io_cq *bic, unsigned int num_actuators)5469{5470	struct bfq_iocq_bfqq_data *bfqq_data = bic->bfqq_data;5471	unsigned int act_idx;5472 5473	for (act_idx = 0; act_idx < num_actuators; act_idx++) {5474		if (bfqq_data[act_idx].stable_merge_bfqq)5475			bfq_put_stable_ref(bfqq_data[act_idx].stable_merge_bfqq);5476 5477		bfq_exit_icq_bfqq(bic, true, act_idx);5478		bfq_exit_icq_bfqq(bic, false, act_idx);5479	}5480}5481 5482static void bfq_exit_icq(struct io_cq *icq)5483{5484	struct bfq_io_cq *bic = icq_to_bic(icq);5485	struct bfq_data *bfqd = bic_to_bfqd(bic);5486	unsigned long flags;5487 5488	/*5489	 * If bfqd and thus bfqd->num_actuators is not available any5490	 * longer, then cycle over all possible per-actuator bfqqs in5491	 * next loop. We rely on bic being zeroed on creation, and5492	 * therefore on its unused per-actuator fields being NULL.5493	 *5494	 * bfqd is NULL if scheduler already exited, and in that case5495	 * this is the last time these queues are accessed.5496	 */5497	if (bfqd) {5498		spin_lock_irqsave(&bfqd->lock, flags);5499		_bfq_exit_icq(bic, bfqd->num_actuators);5500		spin_unlock_irqrestore(&bfqd->lock, flags);5501	} else {5502		_bfq_exit_icq(bic, BFQ_MAX_ACTUATORS);5503	}5504}5505 5506/*5507 * Update the entity prio values; note that the new values will not5508 * be used until the next (re)activation.5509 */5510static void5511bfq_set_next_ioprio_data(struct bfq_queue *bfqq, struct bfq_io_cq *bic)5512{5513	struct task_struct *tsk = current;5514	int ioprio_class;5515	struct bfq_data *bfqd = bfqq->bfqd;5516 5517	if (!bfqd)5518		return;5519 5520	ioprio_class = IOPRIO_PRIO_CLASS(bic->ioprio);5521	switch (ioprio_class) {5522	default:5523		pr_err("bdi %s: bfq: bad prio class %d\n",5524			bdi_dev_name(bfqq->bfqd->queue->disk->bdi),5525			ioprio_class);5526		fallthrough;5527	case IOPRIO_CLASS_NONE:5528		/*5529		 * No prio set, inherit CPU scheduling settings.5530		 */5531		bfqq->new_ioprio = task_nice_ioprio(tsk);5532		bfqq->new_ioprio_class = task_nice_ioclass(tsk);5533		break;5534	case IOPRIO_CLASS_RT:5535		bfqq->new_ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio);5536		bfqq->new_ioprio_class = IOPRIO_CLASS_RT;5537		break;5538	case IOPRIO_CLASS_BE:5539		bfqq->new_ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio);5540		bfqq->new_ioprio_class = IOPRIO_CLASS_BE;5541		break;5542	case IOPRIO_CLASS_IDLE:5543		bfqq->new_ioprio_class = IOPRIO_CLASS_IDLE;5544		bfqq->new_ioprio = IOPRIO_NR_LEVELS - 1;5545		break;5546	}5547 5548	if (bfqq->new_ioprio >= IOPRIO_NR_LEVELS) {5549		pr_crit("bfq_set_next_ioprio_data: new_ioprio %d\n",5550			bfqq->new_ioprio);5551		bfqq->new_ioprio = IOPRIO_NR_LEVELS - 1;5552	}5553 5554	bfqq->entity.new_weight = bfq_ioprio_to_weight(bfqq->new_ioprio);5555	bfq_log_bfqq(bfqd, bfqq, "new_ioprio %d new_weight %d",5556		     bfqq->new_ioprio, bfqq->entity.new_weight);5557	bfqq->entity.prio_changed = 1;5558}5559 5560static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd,5561				       struct bio *bio, bool is_sync,5562				       struct bfq_io_cq *bic,5563				       bool respawn);5564 5565static void bfq_check_ioprio_change(struct bfq_io_cq *bic, struct bio *bio)5566{5567	struct bfq_data *bfqd = bic_to_bfqd(bic);5568	struct bfq_queue *bfqq;5569	int ioprio = bic->icq.ioc->ioprio;5570 5571	/*5572	 * This condition may trigger on a newly created bic, be sure to5573	 * drop the lock before returning.5574	 */5575	if (unlikely(!bfqd) || likely(bic->ioprio == ioprio))5576		return;5577 5578	bic->ioprio = ioprio;5579 5580	bfqq = bic_to_bfqq(bic, false, bfq_actuator_index(bfqd, bio));5581	if (bfqq) {5582		struct bfq_queue *old_bfqq = bfqq;5583 5584		bfqq = bfq_get_queue(bfqd, bio, false, bic, true);5585		bic_set_bfqq(bic, bfqq, false, bfq_actuator_index(bfqd, bio));5586		bfq_release_process_ref(bfqd, old_bfqq);5587	}5588 5589	bfqq = bic_to_bfqq(bic, true, bfq_actuator_index(bfqd, bio));5590	if (bfqq)5591		bfq_set_next_ioprio_data(bfqq, bic);5592}5593 5594static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq,5595			  struct bfq_io_cq *bic, pid_t pid, int is_sync,5596			  unsigned int act_idx)5597{5598	u64 now_ns = blk_time_get_ns();5599 5600	bfqq->actuator_idx = act_idx;5601	RB_CLEAR_NODE(&bfqq->entity.rb_node);5602	INIT_LIST_HEAD(&bfqq->fifo);5603	INIT_HLIST_NODE(&bfqq->burst_list_node);5604	INIT_HLIST_NODE(&bfqq->woken_list_node);5605	INIT_HLIST_HEAD(&bfqq->woken_list);5606 5607	bfqq->ref = 0;5608	bfqq->bfqd = bfqd;5609 5610	if (bic)5611		bfq_set_next_ioprio_data(bfqq, bic);5612 5613	if (is_sync) {5614		/*5615		 * No need to mark as has_short_ttime if in5616		 * idle_class, because no device idling is performed5617		 * for queues in idle class5618		 */5619		if (!bfq_class_idle(bfqq))5620			/* tentatively mark as has_short_ttime */5621			bfq_mark_bfqq_has_short_ttime(bfqq);5622		bfq_mark_bfqq_sync(bfqq);5623		bfq_mark_bfqq_just_created(bfqq);5624	} else5625		bfq_clear_bfqq_sync(bfqq);5626 5627	/* set end request to minus infinity from now */5628	bfqq->ttime.last_end_request = now_ns + 1;5629 5630	bfqq->creation_time = jiffies;5631 5632	bfqq->io_start_time = now_ns;5633 5634	bfq_mark_bfqq_IO_bound(bfqq);5635 5636	bfqq->pid = pid;5637 5638	/* Tentative initial value to trade off between thr and lat */5639	bfqq->max_budget = (2 * bfq_max_budget(bfqd)) / 3;5640	bfqq->budget_timeout = bfq_smallest_from_now();5641 5642	bfqq->wr_coeff = 1;5643	bfqq->last_wr_start_finish = jiffies;5644	bfqq->wr_start_at_switch_to_srt = bfq_smallest_from_now();5645	bfqq->split_time = bfq_smallest_from_now();5646 5647	/*5648	 * To not forget the possibly high bandwidth consumed by a5649	 * process/queue in the recent past,5650	 * bfq_bfqq_softrt_next_start() returns a value at least equal5651	 * to the current value of bfqq->soft_rt_next_start (see5652	 * comments on bfq_bfqq_softrt_next_start).  Set5653	 * soft_rt_next_start to now, to mean that bfqq has consumed5654	 * no bandwidth so far.5655	 */5656	bfqq->soft_rt_next_start = jiffies;5657 5658	/* first request is almost certainly seeky */5659	bfqq->seek_history = 1;5660 5661	bfqq->decrease_time_jif = jiffies;5662}5663 5664static struct bfq_queue **bfq_async_queue_prio(struct bfq_data *bfqd,5665					       struct bfq_group *bfqg,5666					       int ioprio_class, int ioprio, int act_idx)5667{5668	switch (ioprio_class) {5669	case IOPRIO_CLASS_RT:5670		return &bfqg->async_bfqq[0][ioprio][act_idx];5671	case IOPRIO_CLASS_NONE:5672		ioprio = IOPRIO_BE_NORM;5673		fallthrough;5674	case IOPRIO_CLASS_BE:5675		return &bfqg->async_bfqq[1][ioprio][act_idx];5676	case IOPRIO_CLASS_IDLE:5677		return &bfqg->async_idle_bfqq[act_idx];5678	default:5679		return NULL;5680	}5681}5682 5683static struct bfq_queue *5684bfq_do_early_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq,5685			  struct bfq_io_cq *bic,5686			  struct bfq_queue *last_bfqq_created)5687{5688	unsigned int a_idx = last_bfqq_created->actuator_idx;5689	struct bfq_queue *new_bfqq =5690		bfq_setup_merge(bfqq, last_bfqq_created);5691 5692	if (!new_bfqq)5693		return bfqq;5694 5695	if (new_bfqq->bic)5696		new_bfqq->bic->bfqq_data[a_idx].stably_merged = true;5697	bic->bfqq_data[a_idx].stably_merged = true;5698 5699	/*5700	 * Reusing merge functions. This implies that5701	 * bfqq->bic must be set too, for5702	 * bfq_merge_bfqqs to correctly save bfqq's5703	 * state before killing it.5704	 */5705	bfqq->bic = bic;5706	return bfq_merge_bfqqs(bfqd, bic, bfqq);5707}5708 5709/*5710 * Many throughput-sensitive workloads are made of several parallel5711 * I/O flows, with all flows generated by the same application, or5712 * more generically by the same task (e.g., system boot). The most5713 * counterproductive action with these workloads is plugging I/O5714 * dispatch when one of the bfq_queues associated with these flows5715 * remains temporarily empty.5716 *5717 * To avoid this plugging, BFQ has been using a burst-handling5718 * mechanism for years now. This mechanism has proven effective for5719 * throughput, and not detrimental for service guarantees. The5720 * following function pushes this mechanism a little bit further,5721 * basing on the following two facts.5722 *5723 * First, all the I/O flows of a the same application or task5724 * contribute to the execution/completion of that common application5725 * or task. So the performance figures that matter are total5726 * throughput of the flows and task-wide I/O latency.  In particular,5727 * these flows do not need to be protected from each other, in terms5728 * of individual bandwidth or latency.5729 *5730 * Second, the above fact holds regardless of the number of flows.5731 *5732 * Putting these two facts together, this commits merges stably the5733 * bfq_queues associated with these I/O flows, i.e., with the5734 * processes that generate these IO/ flows, regardless of how many the5735 * involved processes are.5736 *5737 * To decide whether a set of bfq_queues is actually associated with5738 * the I/O flows of a common application or task, and to merge these5739 * queues stably, this function operates as follows: given a bfq_queue,5740 * say Q2, currently being created, and the last bfq_queue, say Q1,5741 * created before Q2, Q2 is merged stably with Q1 if5742 * - very little time has elapsed since when Q1 was created5743 * - Q2 has the same ioprio as Q15744 * - Q2 belongs to the same group as Q15745 *5746 * Merging bfq_queues also reduces scheduling overhead. A fio test5747 * with ten random readers on /dev/nullb shows a throughput boost of5748 * 40%, with a quadcore. Since BFQ's execution time amounts to ~50% of5749 * the total per-request processing time, the above throughput boost5750 * implies that BFQ's overhead is reduced by more than 50%.5751 *5752 * This new mechanism most certainly obsoletes the current5753 * burst-handling heuristics. We keep those heuristics for the moment.5754 */5755static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd,5756						      struct bfq_queue *bfqq,5757						      struct bfq_io_cq *bic)5758{5759	struct bfq_queue **source_bfqq = bfqq->entity.parent ?5760		&bfqq->entity.parent->last_bfqq_created :5761		&bfqd->last_bfqq_created;5762 5763	struct bfq_queue *last_bfqq_created = *source_bfqq;5764 5765	/*5766	 * If last_bfqq_created has not been set yet, then init it. If5767	 * it has been set already, but too long ago, then move it5768	 * forward to bfqq. Finally, move also if bfqq belongs to a5769	 * different group than last_bfqq_created, or if bfqq has a5770	 * different ioprio, ioprio_class or actuator_idx. If none of5771	 * these conditions holds true, then try an early stable merge5772	 * or schedule a delayed stable merge. As for the condition on5773	 * actuator_idx, the reason is that, if queues associated with5774	 * different actuators are merged, then control is lost on5775	 * each actuator. Therefore some actuator may be5776	 * underutilized, and throughput may decrease.5777	 *5778	 * A delayed merge is scheduled (instead of performing an5779	 * early merge), in case bfqq might soon prove to be more5780	 * throughput-beneficial if not merged. Currently this is5781	 * possible only if bfqd is rotational with no queueing. For5782	 * such a drive, not merging bfqq is better for throughput if5783	 * bfqq happens to contain sequential I/O. So, we wait a5784	 * little bit for enough I/O to flow through bfqq. After that,5785	 * if such an I/O is sequential, then the merge is5786	 * canceled. Otherwise the merge is finally performed.5787	 */5788	if (!last_bfqq_created ||5789	    time_before(last_bfqq_created->creation_time +5790			msecs_to_jiffies(bfq_activation_stable_merging),5791			bfqq->creation_time) ||5792		bfqq->entity.parent != last_bfqq_created->entity.parent ||5793		bfqq->ioprio != last_bfqq_created->ioprio ||5794		bfqq->ioprio_class != last_bfqq_created->ioprio_class ||5795		bfqq->actuator_idx != last_bfqq_created->actuator_idx)5796		*source_bfqq = bfqq;5797	else if (time_after_eq(last_bfqq_created->creation_time +5798				 bfqd->bfq_burst_interval,5799				 bfqq->creation_time)) {5800		if (likely(bfqd->nonrot_with_queueing))5801			/*5802			 * With this type of drive, leaving5803			 * bfqq alone may provide no5804			 * throughput benefits compared with5805			 * merging bfqq. So merge bfqq now.5806			 */5807			bfqq = bfq_do_early_stable_merge(bfqd, bfqq,5808							 bic,5809							 last_bfqq_created);5810		else { /* schedule tentative stable merge */5811			/*5812			 * get reference on last_bfqq_created,5813			 * to prevent it from being freed,5814			 * until we decide whether to merge5815			 */5816			last_bfqq_created->ref++;5817			/*5818			 * need to keep track of stable refs, to5819			 * compute process refs correctly5820			 */5821			last_bfqq_created->stable_ref++;5822			/*5823			 * Record the bfqq to merge to.5824			 */5825			bic->bfqq_data[last_bfqq_created->actuator_idx].stable_merge_bfqq =5826				last_bfqq_created;5827		}5828	}5829 5830	return bfqq;5831}5832 5833 5834static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd,5835				       struct bio *bio, bool is_sync,5836				       struct bfq_io_cq *bic,5837				       bool respawn)5838{5839	const int ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio);5840	const int ioprio_class = IOPRIO_PRIO_CLASS(bic->ioprio);5841	struct bfq_queue **async_bfqq = NULL;5842	struct bfq_queue *bfqq;5843	struct bfq_group *bfqg;5844 5845	bfqg = bfq_bio_bfqg(bfqd, bio);5846	if (!is_sync) {5847		async_bfqq = bfq_async_queue_prio(bfqd, bfqg, ioprio_class,5848						  ioprio,5849						  bfq_actuator_index(bfqd, bio));5850		bfqq = *async_bfqq;5851		if (bfqq)5852			goto out;5853	}5854 5855	bfqq = kmem_cache_alloc_node(bfq_pool,5856				     GFP_NOWAIT | __GFP_ZERO | __GFP_NOWARN,5857				     bfqd->queue->node);5858 5859	if (bfqq) {5860		bfq_init_bfqq(bfqd, bfqq, bic, current->pid,5861			      is_sync, bfq_actuator_index(bfqd, bio));5862		bfq_init_entity(&bfqq->entity, bfqg);5863		bfq_log_bfqq(bfqd, bfqq, "allocated");5864	} else {5865		bfqq = &bfqd->oom_bfqq;5866		bfq_log_bfqq(bfqd, bfqq, "using oom bfqq");5867		goto out;5868	}5869 5870	/*5871	 * Pin the queue now that it's allocated, scheduler exit will5872	 * prune it.5873	 */5874	if (async_bfqq) {5875		bfqq->ref++; /*5876			      * Extra group reference, w.r.t. sync5877			      * queue. This extra reference is removed5878			      * only if bfqq->bfqg disappears, to5879			      * guarantee that this queue is not freed5880			      * until its group goes away.5881			      */5882		bfq_log_bfqq(bfqd, bfqq, "get_queue, bfqq not in async: %p, %d",5883			     bfqq, bfqq->ref);5884		*async_bfqq = bfqq;5885	}5886 5887out:5888	bfqq->ref++; /* get a process reference to this queue */5889 5890	if (bfqq != &bfqd->oom_bfqq && is_sync && !respawn)5891		bfqq = bfq_do_or_sched_stable_merge(bfqd, bfqq, bic);5892	return bfqq;5893}5894 5895static void bfq_update_io_thinktime(struct bfq_data *bfqd,5896				    struct bfq_queue *bfqq)5897{5898	struct bfq_ttime *ttime = &bfqq->ttime;5899	u64 elapsed;5900 5901	/*5902	 * We are really interested in how long it takes for the queue to5903	 * become busy when there is no outstanding IO for this queue. So5904	 * ignore cases when the bfq queue has already IO queued.5905	 */5906	if (bfqq->dispatched || bfq_bfqq_busy(bfqq))5907		return;5908	elapsed = blk_time_get_ns() - bfqq->ttime.last_end_request;5909	elapsed = min_t(u64, elapsed, 2ULL * bfqd->bfq_slice_idle);5910 5911	ttime->ttime_samples = (7*ttime->ttime_samples + 256) / 8;5912	ttime->ttime_total = div_u64(7*ttime->ttime_total + 256*elapsed,  8);5913	ttime->ttime_mean = div64_ul(ttime->ttime_total + 128,5914				     ttime->ttime_samples);5915}5916 5917static void5918bfq_update_io_seektime(struct bfq_data *bfqd, struct bfq_queue *bfqq,5919		       struct request *rq)5920{5921	bfqq->seek_history <<= 1;5922	bfqq->seek_history |= BFQ_RQ_SEEKY(bfqd, bfqq->last_request_pos, rq);5923 5924	if (bfqq->wr_coeff > 1 &&5925	    bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time &&5926	    BFQQ_TOTALLY_SEEKY(bfqq)) {5927		if (time_is_before_jiffies(bfqq->wr_start_at_switch_to_srt +5928					   bfq_wr_duration(bfqd))) {5929			/*5930			 * In soft_rt weight raising with the5931			 * interactive-weight-raising period5932			 * elapsed (so no switch back to5933			 * interactive weight raising).5934			 */5935			bfq_bfqq_end_wr(bfqq);5936		} else { /*5937			  * stopping soft_rt weight raising5938			  * while still in interactive period,5939			  * switch back to interactive weight5940			  * raising5941			  */5942			switch_back_to_interactive_wr(bfqq, bfqd);5943			bfqq->entity.prio_changed = 1;5944		}5945	}5946}5947 5948static void bfq_update_has_short_ttime(struct bfq_data *bfqd,5949				       struct bfq_queue *bfqq,5950				       struct bfq_io_cq *bic)5951{5952	bool has_short_ttime = true, state_changed;5953 5954	/*5955	 * No need to update has_short_ttime if bfqq is async or in5956	 * idle io prio class, or if bfq_slice_idle is zero, because5957	 * no device idling is performed for bfqq in this case.5958	 */5959	if (!bfq_bfqq_sync(bfqq) || bfq_class_idle(bfqq) ||5960	    bfqd->bfq_slice_idle == 0)5961		return;5962 5963	/* Idle window just restored, statistics are meaningless. */5964	if (time_is_after_eq_jiffies(bfqq->split_time +5965				     bfqd->bfq_wr_min_idle_time))5966		return;5967 5968	/* Think time is infinite if no process is linked to5969	 * bfqq. Otherwise check average think time to decide whether5970	 * to mark as has_short_ttime. To this goal, compare average5971	 * think time with half the I/O-plugging timeout.5972	 */5973	if (atomic_read(&bic->icq.ioc->active_ref) == 0 ||5974	    (bfq_sample_valid(bfqq->ttime.ttime_samples) &&5975	     bfqq->ttime.ttime_mean > bfqd->bfq_slice_idle>>1))5976		has_short_ttime = false;5977 5978	state_changed = has_short_ttime != bfq_bfqq_has_short_ttime(bfqq);5979 5980	if (has_short_ttime)5981		bfq_mark_bfqq_has_short_ttime(bfqq);5982	else5983		bfq_clear_bfqq_has_short_ttime(bfqq);5984 5985	/*5986	 * Until the base value for the total service time gets5987	 * finally computed for bfqq, the inject limit does depend on5988	 * the think-time state (short|long). In particular, the limit5989	 * is 0 or 1 if the think time is deemed, respectively, as5990	 * short or long (details in the comments in5991	 * bfq_update_inject_limit()). Accordingly, the next5992	 * instructions reset the inject limit if the think-time state5993	 * has changed and the above base value is still to be5994	 * computed.5995	 *5996	 * However, the reset is performed only if more than 100 ms5997	 * have elapsed since the last update of the inject limit, or5998	 * (inclusive) if the change is from short to long think5999	 * time. The reason for this waiting is as follows.6000	 *6001	 * bfqq may have a long think time because of a6002	 * synchronization with some other queue, i.e., because the6003	 * I/O of some other queue may need to be completed for bfqq6004	 * to receive new I/O. Details in the comments on the choice6005	 * of the queue for injection in bfq_select_queue().6006	 *6007	 * As stressed in those comments, if such a synchronization is6008	 * actually in place, then, without injection on bfqq, the6009	 * blocking I/O cannot happen to served while bfqq is in6010	 * service. As a consequence, if bfqq is granted6011	 * I/O-dispatch-plugging, then bfqq remains empty, and no I/O6012	 * is dispatched, until the idle timeout fires. This is likely6013	 * to result in lower bandwidth and higher latencies for bfqq,6014	 * and in a severe loss of total throughput.6015	 *6016	 * On the opposite end, a non-zero inject limit may allow the6017	 * I/O that blocks bfqq to be executed soon, and therefore6018	 * bfqq to receive new I/O soon.6019	 *6020	 * But, if the blocking gets actually eliminated, then the6021	 * next think-time sample for bfqq may be very low. This in6022	 * turn may cause bfqq's think time to be deemed6023	 * short. Without the 100 ms barrier, this new state change6024	 * would cause the body of the next if to be executed6025	 * immediately. But this would set to 0 the inject6026	 * limit. Without injection, the blocking I/O would cause the6027	 * think time of bfqq to become long again, and therefore the6028	 * inject limit to be raised again, and so on. The only effect6029	 * of such a steady oscillation between the two think-time6030	 * states would be to prevent effective injection on bfqq.6031	 *6032	 * In contrast, if the inject limit is not reset during such a6033	 * long time interval as 100 ms, then the number of short6034	 * think time samples can grow significantly before the reset6035	 * is performed. As a consequence, the think time state can6036	 * become stable before the reset. Therefore there will be no6037	 * state change when the 100 ms elapse, and no reset of the6038	 * inject limit. The inject limit remains steadily equal to 16039	 * both during and after the 100 ms. So injection can be6040	 * performed at all times, and throughput gets boosted.6041	 *6042	 * An inject limit equal to 1 is however in conflict, in6043	 * general, with the fact that the think time of bfqq is6044	 * short, because injection may be likely to delay bfqq's I/O6045	 * (as explained in the comments in6046	 * bfq_update_inject_limit()). But this does not happen in6047	 * this special case, because bfqq's low think time is due to6048	 * an effective handling of a synchronization, through6049	 * injection. In this special case, bfqq's I/O does not get6050	 * delayed by injection; on the contrary, bfqq's I/O is6051	 * brought forward, because it is not blocked for6052	 * milliseconds.6053	 *6054	 * In addition, serving the blocking I/O much sooner, and much6055	 * more frequently than once per I/O-plugging timeout, makes6056	 * it much quicker to detect a waker queue (the concept of6057	 * waker queue is defined in the comments in6058	 * bfq_add_request()). This makes it possible to start sooner6059	 * to boost throughput more effectively, by injecting the I/O6060	 * of the waker queue unconditionally on every6061	 * bfq_dispatch_request().6062	 *6063	 * One last, important benefit of not resetting the inject6064	 * limit before 100 ms is that, during this time interval, the6065	 * base value for the total service time is likely to get6066	 * finally computed for bfqq, freeing the inject limit from6067	 * its relation with the think time.6068	 */6069	if (state_changed && bfqq->last_serv_time_ns == 0 &&6070	    (time_is_before_eq_jiffies(bfqq->decrease_time_jif +6071				      msecs_to_jiffies(100)) ||6072	     !has_short_ttime))6073		bfq_reset_inject_limit(bfqd, bfqq);6074}6075 6076/*6077 * Called when a new fs request (rq) is added to bfqq.  Check if there's6078 * something we should do about it.6079 */6080static void bfq_rq_enqueued(struct bfq_data *bfqd, struct bfq_queue *bfqq,6081			    struct request *rq)6082{6083	if (rq->cmd_flags & REQ_META)6084		bfqq->meta_pending++;6085 6086	bfqq->last_request_pos = blk_rq_pos(rq) + blk_rq_sectors(rq);6087 6088	if (bfqq == bfqd->in_service_queue && bfq_bfqq_wait_request(bfqq)) {6089		bool small_req = bfqq->queued[rq_is_sync(rq)] == 1 &&6090				 blk_rq_sectors(rq) < 32;6091		bool budget_timeout = bfq_bfqq_budget_timeout(bfqq);6092 6093		/*6094		 * There is just this request queued: if6095		 * - the request is small, and6096		 * - we are idling to boost throughput, and6097		 * - the queue is not to be expired,6098		 * then just exit.6099		 *6100		 * In this way, if the device is being idled to wait6101		 * for a new request from the in-service queue, we6102		 * avoid unplugging the device and committing the6103		 * device to serve just a small request. In contrast6104		 * we wait for the block layer to decide when to6105		 * unplug the device: hopefully, new requests will be6106		 * merged to this one quickly, then the device will be6107		 * unplugged and larger requests will be dispatched.6108		 */6109		if (small_req && idling_boosts_thr_without_issues(bfqd, bfqq) &&6110		    !budget_timeout)6111			return;6112 6113		/*6114		 * A large enough request arrived, or idling is being6115		 * performed to preserve service guarantees, or6116		 * finally the queue is to be expired: in all these6117		 * cases disk idling is to be stopped, so clear6118		 * wait_request flag and reset timer.6119		 */6120		bfq_clear_bfqq_wait_request(bfqq);6121		hrtimer_try_to_cancel(&bfqd->idle_slice_timer);6122 6123		/*6124		 * The queue is not empty, because a new request just6125		 * arrived. Hence we can safely expire the queue, in6126		 * case of budget timeout, without risking that the6127		 * timestamps of the queue are not updated correctly.6128		 * See [1] for more details.6129		 */6130		if (budget_timeout)6131			bfq_bfqq_expire(bfqd, bfqq, false,6132					BFQQE_BUDGET_TIMEOUT);6133	}6134}6135 6136static void bfqq_request_allocated(struct bfq_queue *bfqq)6137{6138	struct bfq_entity *entity = &bfqq->entity;6139 6140	for_each_entity(entity)6141		entity->allocated++;6142}6143 6144static void bfqq_request_freed(struct bfq_queue *bfqq)6145{6146	struct bfq_entity *entity = &bfqq->entity;6147 6148	for_each_entity(entity)6149		entity->allocated--;6150}6151 6152/* returns true if it causes the idle timer to be disabled */6153static bool __bfq_insert_request(struct bfq_data *bfqd, struct request *rq)6154{6155	struct bfq_queue *bfqq = RQ_BFQQ(rq),6156		*new_bfqq = bfq_setup_cooperator(bfqd, bfqq, rq, true,6157						 RQ_BIC(rq));6158	bool waiting, idle_timer_disabled = false;6159 6160	if (new_bfqq) {6161		struct bfq_queue *old_bfqq = bfqq;6162		/*6163		 * Release the request's reference to the old bfqq6164		 * and make sure one is taken to the shared queue.6165		 */6166		bfqq_request_allocated(new_bfqq);6167		bfqq_request_freed(bfqq);6168		new_bfqq->ref++;6169		/*6170		 * If the bic associated with the process6171		 * issuing this request still points to bfqq6172		 * (and thus has not been already redirected6173		 * to new_bfqq or even some other bfq_queue),6174		 * then complete the merge and redirect it to6175		 * new_bfqq.6176		 */6177		if (bic_to_bfqq(RQ_BIC(rq), true,6178				bfq_actuator_index(bfqd, rq->bio)) == bfqq) {6179			while (bfqq != new_bfqq)6180				bfqq = bfq_merge_bfqqs(bfqd, RQ_BIC(rq), bfqq);6181		}6182 6183		bfq_clear_bfqq_just_created(old_bfqq);6184		/*6185		 * rq is about to be enqueued into new_bfqq,6186		 * release rq reference on bfqq6187		 */6188		bfq_put_queue(old_bfqq);6189		rq->elv.priv[1] = new_bfqq;6190	}6191 6192	bfq_update_io_thinktime(bfqd, bfqq);6193	bfq_update_has_short_ttime(bfqd, bfqq, RQ_BIC(rq));6194	bfq_update_io_seektime(bfqd, bfqq, rq);6195 6196	waiting = bfqq && bfq_bfqq_wait_request(bfqq);6197	bfq_add_request(rq);6198	idle_timer_disabled = waiting && !bfq_bfqq_wait_request(bfqq);6199 6200	rq->fifo_time = blk_time_get_ns() + bfqd->bfq_fifo_expire[rq_is_sync(rq)];6201	list_add_tail(&rq->queuelist, &bfqq->fifo);6202 6203	bfq_rq_enqueued(bfqd, bfqq, rq);6204 6205	return idle_timer_disabled;6206}6207 6208#ifdef CONFIG_BFQ_CGROUP_DEBUG6209static void bfq_update_insert_stats(struct request_queue *q,6210				    struct bfq_queue *bfqq,6211				    bool idle_timer_disabled,6212				    blk_opf_t cmd_flags)6213{6214	if (!bfqq)6215		return;6216 6217	/*6218	 * bfqq still exists, because it can disappear only after6219	 * either it is merged with another queue, or the process it6220	 * is associated with exits. But both actions must be taken by6221	 * the same process currently executing this flow of6222	 * instructions.6223	 *6224	 * In addition, the following queue lock guarantees that6225	 * bfqq_group(bfqq) exists as well.6226	 */6227	spin_lock_irq(&q->queue_lock);6228	bfqg_stats_update_io_add(bfqq_group(bfqq), bfqq, cmd_flags);6229	if (idle_timer_disabled)6230		bfqg_stats_update_idle_time(bfqq_group(bfqq));6231	spin_unlock_irq(&q->queue_lock);6232}6233#else6234static inline void bfq_update_insert_stats(struct request_queue *q,6235					   struct bfq_queue *bfqq,6236					   bool idle_timer_disabled,6237					   blk_opf_t cmd_flags) {}6238#endif /* CONFIG_BFQ_CGROUP_DEBUG */6239 6240static struct bfq_queue *bfq_init_rq(struct request *rq);6241 6242static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,6243			       blk_insert_t flags)6244{6245	struct request_queue *q = hctx->queue;6246	struct bfq_data *bfqd = q->elevator->elevator_data;6247	struct bfq_queue *bfqq;6248	bool idle_timer_disabled = false;6249	blk_opf_t cmd_flags;6250	LIST_HEAD(free);6251 6252#ifdef CONFIG_BFQ_GROUP_IOSCHED6253	if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio)6254		bfqg_stats_update_legacy_io(q, rq);6255#endif6256	spin_lock_irq(&bfqd->lock);6257	bfqq = bfq_init_rq(rq);6258	if (blk_mq_sched_try_insert_merge(q, rq, &free)) {6259		spin_unlock_irq(&bfqd->lock);6260		blk_mq_free_requests(&free);6261		return;6262	}6263 6264	trace_block_rq_insert(rq);6265 6266	if (flags & BLK_MQ_INSERT_AT_HEAD) {6267		list_add(&rq->queuelist, &bfqd->dispatch);6268	} else if (!bfqq) {6269		list_add_tail(&rq->queuelist, &bfqd->dispatch);6270	} else {6271		idle_timer_disabled = __bfq_insert_request(bfqd, rq);6272		/*6273		 * Update bfqq, because, if a queue merge has occurred6274		 * in __bfq_insert_request, then rq has been6275		 * redirected into a new queue.6276		 */6277		bfqq = RQ_BFQQ(rq);6278 6279		if (rq_mergeable(rq)) {6280			elv_rqhash_add(q, rq);6281			if (!q->last_merge)6282				q->last_merge = rq;6283		}6284	}6285 6286	/*6287	 * Cache cmd_flags before releasing scheduler lock, because rq6288	 * may disappear afterwards (for example, because of a request6289	 * merge).6290	 */6291	cmd_flags = rq->cmd_flags;6292	spin_unlock_irq(&bfqd->lock);6293 6294	bfq_update_insert_stats(q, bfqq, idle_timer_disabled,6295				cmd_flags);6296}6297 6298static void bfq_insert_requests(struct blk_mq_hw_ctx *hctx,6299				struct list_head *list,6300				blk_insert_t flags)6301{6302	while (!list_empty(list)) {6303		struct request *rq;6304 6305		rq = list_first_entry(list, struct request, queuelist);6306		list_del_init(&rq->queuelist);6307		bfq_insert_request(hctx, rq, flags);6308	}6309}6310 6311static void bfq_update_hw_tag(struct bfq_data *bfqd)6312{6313	struct bfq_queue *bfqq = bfqd->in_service_queue;6314 6315	bfqd->max_rq_in_driver = max_t(int, bfqd->max_rq_in_driver,6316				       bfqd->tot_rq_in_driver);6317 6318	if (bfqd->hw_tag == 1)6319		return;6320 6321	/*6322	 * This sample is valid if the number of outstanding requests6323	 * is large enough to allow a queueing behavior.  Note that the6324	 * sum is not exact, as it's not taking into account deactivated6325	 * requests.6326	 */6327	if (bfqd->tot_rq_in_driver + bfqd->queued <= BFQ_HW_QUEUE_THRESHOLD)6328		return;6329 6330	/*6331	 * If active queue hasn't enough requests and can idle, bfq might not6332	 * dispatch sufficient requests to hardware. Don't zero hw_tag in this6333	 * case6334	 */6335	if (bfqq && bfq_bfqq_has_short_ttime(bfqq) &&6336	    bfqq->dispatched + bfqq->queued[0] + bfqq->queued[1] <6337	    BFQ_HW_QUEUE_THRESHOLD &&6338	    bfqd->tot_rq_in_driver < BFQ_HW_QUEUE_THRESHOLD)6339		return;6340 6341	if (bfqd->hw_tag_samples++ < BFQ_HW_QUEUE_SAMPLES)6342		return;6343 6344	bfqd->hw_tag = bfqd->max_rq_in_driver > BFQ_HW_QUEUE_THRESHOLD;6345	bfqd->max_rq_in_driver = 0;6346	bfqd->hw_tag_samples = 0;6347 6348	bfqd->nonrot_with_queueing =6349		blk_queue_nonrot(bfqd->queue) && bfqd->hw_tag;6350}6351 6352static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)6353{6354	u64 now_ns;6355	u32 delta_us;6356 6357	bfq_update_hw_tag(bfqd);6358 6359	bfqd->rq_in_driver[bfqq->actuator_idx]--;6360	bfqd->tot_rq_in_driver--;6361	bfqq->dispatched--;6362 6363	if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {6364		/*6365		 * Set budget_timeout (which we overload to store the6366		 * time at which the queue remains with no backlog and6367		 * no outstanding request; used by the weight-raising6368		 * mechanism).6369		 */6370		bfqq->budget_timeout = jiffies;6371 6372		bfq_del_bfqq_in_groups_with_pending_reqs(bfqq);6373		bfq_weights_tree_remove(bfqq);6374	}6375 6376	now_ns = blk_time_get_ns();6377 6378	bfqq->ttime.last_end_request = now_ns;6379 6380	/*6381	 * Using us instead of ns, to get a reasonable precision in6382	 * computing rate in next check.6383	 */6384	delta_us = div_u64(now_ns - bfqd->last_completion, NSEC_PER_USEC);6385 6386	/*6387	 * If the request took rather long to complete, and, according6388	 * to the maximum request size recorded, this completion latency6389	 * implies that the request was certainly served at a very low6390	 * rate (less than 1M sectors/sec), then the whole observation6391	 * interval that lasts up to this time instant cannot be a6392	 * valid time interval for computing a new peak rate.  Invoke6393	 * bfq_update_rate_reset to have the following three steps6394	 * taken:6395	 * - close the observation interval at the last (previous)6396	 *   request dispatch or completion6397	 * - compute rate, if possible, for that observation interval6398	 * - reset to zero samples, which will trigger a proper6399	 *   re-initialization of the observation interval on next6400	 *   dispatch6401	 */6402	if (delta_us > BFQ_MIN_TT/NSEC_PER_USEC &&6403	   (bfqd->last_rq_max_size<<BFQ_RATE_SHIFT)/delta_us <6404			1UL<<(BFQ_RATE_SHIFT - 10))6405		bfq_update_rate_reset(bfqd, NULL);6406	bfqd->last_completion = now_ns;6407	/*6408	 * Shared queues are likely to receive I/O at a high6409	 * rate. This may deceptively let them be considered as wakers6410	 * of other queues. But a false waker will unjustly steal6411	 * bandwidth to its supposedly woken queue. So considering6412	 * also shared queues in the waking mechanism may cause more6413	 * control troubles than throughput benefits. Then reset6414	 * last_completed_rq_bfqq if bfqq is a shared queue.6415	 */6416	if (!bfq_bfqq_coop(bfqq))6417		bfqd->last_completed_rq_bfqq = bfqq;6418	else6419		bfqd->last_completed_rq_bfqq = NULL;6420 6421	/*6422	 * If we are waiting to discover whether the request pattern6423	 * of the task associated with the queue is actually6424	 * isochronous, and both requisites for this condition to hold6425	 * are now satisfied, then compute soft_rt_next_start (see the6426	 * comments on the function bfq_bfqq_softrt_next_start()). We6427	 * do not compute soft_rt_next_start if bfqq is in interactive6428	 * weight raising (see the comments in bfq_bfqq_expire() for6429	 * an explanation). We schedule this delayed update when bfqq6430	 * expires, if it still has in-flight requests.6431	 */6432	if (bfq_bfqq_softrt_update(bfqq) && bfqq->dispatched == 0 &&6433	    RB_EMPTY_ROOT(&bfqq->sort_list) &&6434	    bfqq->wr_coeff != bfqd->bfq_wr_coeff)6435		bfqq->soft_rt_next_start =6436			bfq_bfqq_softrt_next_start(bfqd, bfqq);6437 6438	/*6439	 * If this is the in-service queue, check if it needs to be expired,6440	 * or if we want to idle in case it has no pending requests.6441	 */6442	if (bfqd->in_service_queue == bfqq) {6443		if (bfq_bfqq_must_idle(bfqq)) {6444			if (bfqq->dispatched == 0)6445				bfq_arm_slice_timer(bfqd);6446			/*6447			 * If we get here, we do not expire bfqq, even6448			 * if bfqq was in budget timeout or had no6449			 * more requests (as controlled in the next6450			 * conditional instructions). The reason for6451			 * not expiring bfqq is as follows.6452			 *6453			 * Here bfqq->dispatched > 0 holds, but6454			 * bfq_bfqq_must_idle() returned true. This6455			 * implies that, even if no request arrives6456			 * for bfqq before bfqq->dispatched reaches 0,6457			 * bfqq will, however, not be expired on the6458			 * completion event that causes bfqq->dispatch6459			 * to reach zero. In contrast, on this event,6460			 * bfqq will start enjoying device idling6461			 * (I/O-dispatch plugging).6462			 *6463			 * But, if we expired bfqq here, bfqq would6464			 * not have the chance to enjoy device idling6465			 * when bfqq->dispatched finally reaches6466			 * zero. This would expose bfqq to violation6467			 * of its reserved service guarantees.6468			 */6469			return;6470		} else if (bfq_may_expire_for_budg_timeout(bfqq))6471			bfq_bfqq_expire(bfqd, bfqq, false,6472					BFQQE_BUDGET_TIMEOUT);6473		else if (RB_EMPTY_ROOT(&bfqq->sort_list) &&6474			 (bfqq->dispatched == 0 ||6475			  !bfq_better_to_idle(bfqq)))6476			bfq_bfqq_expire(bfqd, bfqq, false,6477					BFQQE_NO_MORE_REQUESTS);6478	}6479 6480	if (!bfqd->tot_rq_in_driver)6481		bfq_schedule_dispatch(bfqd);6482}6483 6484/*6485 * The processes associated with bfqq may happen to generate their6486 * cumulative I/O at a lower rate than the rate at which the device6487 * could serve the same I/O. This is rather probable, e.g., if only6488 * one process is associated with bfqq and the device is an SSD. It6489 * results in bfqq becoming often empty while in service. In this6490 * respect, if BFQ is allowed to switch to another queue when bfqq6491 * remains empty, then the device goes on being fed with I/O requests,6492 * and the throughput is not affected. In contrast, if BFQ is not6493 * allowed to switch to another queue---because bfqq is sync and6494 * I/O-dispatch needs to be plugged while bfqq is temporarily6495 * empty---then, during the service of bfqq, there will be frequent6496 * "service holes", i.e., time intervals during which bfqq gets empty6497 * and the device can only consume the I/O already queued in its6498 * hardware queues. During service holes, the device may even get to6499 * remaining idle. In the end, during the service of bfqq, the device6500 * is driven at a lower speed than the one it can reach with the kind6501 * of I/O flowing through bfqq.6502 *6503 * To counter this loss of throughput, BFQ implements a "request6504 * injection mechanism", which tries to fill the above service holes6505 * with I/O requests taken from other queues. The hard part in this6506 * mechanism is finding the right amount of I/O to inject, so as to6507 * both boost throughput and not break bfqq's bandwidth and latency6508 * guarantees. In this respect, the mechanism maintains a per-queue6509 * inject limit, computed as below. While bfqq is empty, the injection6510 * mechanism dispatches extra I/O requests only until the total number6511 * of I/O requests in flight---i.e., already dispatched but not yet6512 * completed---remains lower than this limit.6513 *6514 * A first definition comes in handy to introduce the algorithm by6515 * which the inject limit is computed.  We define as first request for6516 * bfqq, an I/O request for bfqq that arrives while bfqq is in6517 * service, and causes bfqq to switch from empty to non-empty. The6518 * algorithm updates the limit as a function of the effect of6519 * injection on the service times of only the first requests of6520 * bfqq. The reason for this restriction is that these are the6521 * requests whose service time is affected most, because they are the6522 * first to arrive after injection possibly occurred.6523 *6524 * To evaluate the effect of injection, the algorithm measures the6525 * "total service time" of first requests. We define as total service6526 * time of an I/O request, the time that elapses since when the6527 * request is enqueued into bfqq, to when it is completed. This6528 * quantity allows the whole effect of injection to be measured. It is6529 * easy to see why. Suppose that some requests of other queues are6530 * actually injected while bfqq is empty, and that a new request R6531 * then arrives for bfqq. If the device does start to serve all or6532 * part of the injected requests during the service hole, then,6533 * because of this extra service, it may delay the next invocation of6534 * the dispatch hook of BFQ. Then, even after R gets eventually6535 * dispatched, the device may delay the actual service of R if it is6536 * still busy serving the extra requests, or if it decides to serve,6537 * before R, some extra request still present in its queues. As a6538 * conclusion, the cumulative extra delay caused by injection can be6539 * easily evaluated by just comparing the total service time of first6540 * requests with and without injection.6541 *6542 * The limit-update algorithm works as follows. On the arrival of a6543 * first request of bfqq, the algorithm measures the total time of the6544 * request only if one of the three cases below holds, and, for each6545 * case, it updates the limit as described below:6546 *6547 * (1) If there is no in-flight request. This gives a baseline for the6548 *     total service time of the requests of bfqq. If the baseline has6549 *     not been computed yet, then, after computing it, the limit is6550 *     set to 1, to start boosting throughput, and to prepare the6551 *     ground for the next case. If the baseline has already been6552 *     computed, then it is updated, in case it results to be lower6553 *     than the previous value.6554 *6555 * (2) If the limit is higher than 0 and there are in-flight6556 *     requests. By comparing the total service time in this case with6557 *     the above baseline, it is possible to know at which extent the6558 *     current value of the limit is inflating the total service6559 *     time. If the inflation is below a certain threshold, then bfqq6560 *     is assumed to be suffering from no perceivable loss of its6561 *     service guarantees, and the limit is even tentatively6562 *     increased. If the inflation is above the threshold, then the6563 *     limit is decreased. Due to the lack of any hysteresis, this6564 *     logic makes the limit oscillate even in steady workload6565 *     conditions. Yet we opted for it, because it is fast in reaching6566 *     the best value for the limit, as a function of the current I/O6567 *     workload. To reduce oscillations, this step is disabled for a6568 *     short time interval after the limit happens to be decreased.6569 *6570 * (3) Periodically, after resetting the limit, to make sure that the6571 *     limit eventually drops in case the workload changes. This is6572 *     needed because, after the limit has gone safely up for a6573 *     certain workload, it is impossible to guess whether the6574 *     baseline total service time may have changed, without measuring6575 *     it again without injection. A more effective version of this6576 *     step might be to just sample the baseline, by interrupting6577 *     injection only once, and then to reset/lower the limit only if6578 *     the total service time with the current limit does happen to be6579 *     too large.6580 *6581 * More details on each step are provided in the comments on the6582 * pieces of code that implement these steps: the branch handling the6583 * transition from empty to non empty in bfq_add_request(), the branch6584 * handling injection in bfq_select_queue(), and the function6585 * bfq_choose_bfqq_for_injection(). These comments also explain some6586 * exceptions, made by the injection mechanism in some special cases.6587 */6588static void bfq_update_inject_limit(struct bfq_data *bfqd,6589				    struct bfq_queue *bfqq)6590{6591	u64 tot_time_ns = blk_time_get_ns() - bfqd->last_empty_occupied_ns;6592	unsigned int old_limit = bfqq->inject_limit;6593 6594	if (bfqq->last_serv_time_ns > 0 && bfqd->rqs_injected) {6595		u64 threshold = (bfqq->last_serv_time_ns * 3)>>1;6596 6597		if (tot_time_ns >= threshold && old_limit > 0) {6598			bfqq->inject_limit--;6599			bfqq->decrease_time_jif = jiffies;6600		} else if (tot_time_ns < threshold &&6601			   old_limit <= bfqd->max_rq_in_driver)6602			bfqq->inject_limit++;6603	}6604 6605	/*6606	 * Either we still have to compute the base value for the6607	 * total service time, and there seem to be the right6608	 * conditions to do it, or we can lower the last base value6609	 * computed.6610	 *6611	 * NOTE: (bfqd->tot_rq_in_driver == 1) means that there is no I/O6612	 * request in flight, because this function is in the code6613	 * path that handles the completion of a request of bfqq, and,6614	 * in particular, this function is executed before6615	 * bfqd->tot_rq_in_driver is decremented in such a code path.6616	 */6617	if ((bfqq->last_serv_time_ns == 0 && bfqd->tot_rq_in_driver == 1) ||6618	    tot_time_ns < bfqq->last_serv_time_ns) {6619		if (bfqq->last_serv_time_ns == 0) {6620			/*6621			 * Now we certainly have a base value: make sure we6622			 * start trying injection.6623			 */6624			bfqq->inject_limit = max_t(unsigned int, 1, old_limit);6625		}6626		bfqq->last_serv_time_ns = tot_time_ns;6627	} else if (!bfqd->rqs_injected && bfqd->tot_rq_in_driver == 1)6628		/*6629		 * No I/O injected and no request still in service in6630		 * the drive: these are the exact conditions for6631		 * computing the base value of the total service time6632		 * for bfqq. So let's update this value, because it is6633		 * rather variable. For example, it varies if the size6634		 * or the spatial locality of the I/O requests in bfqq6635		 * change.6636		 */6637		bfqq->last_serv_time_ns = tot_time_ns;6638 6639 6640	/* update complete, not waiting for any request completion any longer */6641	bfqd->waited_rq = NULL;6642	bfqd->rqs_injected = false;6643}6644 6645/*6646 * Handle either a requeue or a finish for rq. The things to do are6647 * the same in both cases: all references to rq are to be dropped. In6648 * particular, rq is considered completed from the point of view of6649 * the scheduler.6650 */6651static void bfq_finish_requeue_request(struct request *rq)6652{6653	struct bfq_queue *bfqq = RQ_BFQQ(rq);6654	struct bfq_data *bfqd;6655	unsigned long flags;6656 6657	/*6658	 * rq either is not associated with any icq, or is an already6659	 * requeued request that has not (yet) been re-inserted into6660	 * a bfq_queue.6661	 */6662	if (!rq->elv.icq || !bfqq)6663		return;6664 6665	bfqd = bfqq->bfqd;6666 6667	if (rq->rq_flags & RQF_STARTED)6668		bfqg_stats_update_completion(bfqq_group(bfqq),6669					     rq->start_time_ns,6670					     rq->io_start_time_ns,6671					     rq->cmd_flags);6672 6673	spin_lock_irqsave(&bfqd->lock, flags);6674	if (likely(rq->rq_flags & RQF_STARTED)) {6675		if (rq == bfqd->waited_rq)6676			bfq_update_inject_limit(bfqd, bfqq);6677 6678		bfq_completed_request(bfqq, bfqd);6679	}6680	bfqq_request_freed(bfqq);6681	bfq_put_queue(bfqq);6682	RQ_BIC(rq)->requests--;6683	spin_unlock_irqrestore(&bfqd->lock, flags);6684 6685	/*6686	 * Reset private fields. In case of a requeue, this allows6687	 * this function to correctly do nothing if it is spuriously6688	 * invoked again on this same request (see the check at the6689	 * beginning of the function). Probably, a better general6690	 * design would be to prevent blk-mq from invoking the requeue6691	 * or finish hooks of an elevator, for a request that is not6692	 * referred by that elevator.6693	 *6694	 * Resetting the following fields would break the6695	 * request-insertion logic if rq is re-inserted into a bfq6696	 * internal queue, without a re-preparation. Here we assume6697	 * that re-insertions of requeued requests, without6698	 * re-preparation, can happen only for pass_through or at_head6699	 * requests (which are not re-inserted into bfq internal6700	 * queues).6701	 */6702	rq->elv.priv[0] = NULL;6703	rq->elv.priv[1] = NULL;6704}6705 6706static void bfq_finish_request(struct request *rq)6707{6708	bfq_finish_requeue_request(rq);6709 6710	if (rq->elv.icq) {6711		put_io_context(rq->elv.icq->ioc);6712		rq->elv.icq = NULL;6713	}6714}6715 6716/*6717 * Removes the association between the current task and bfqq, assuming6718 * that bic points to the bfq iocontext of the task.6719 * Returns NULL if a new bfqq should be allocated, or the old bfqq if this6720 * was the last process referring to that bfqq.6721 */6722static struct bfq_queue *6723bfq_split_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq)6724{6725	bfq_log_bfqq(bfqq->bfqd, bfqq, "splitting queue");6726 6727	if (bfqq_process_refs(bfqq) == 1 && !bfqq->new_bfqq) {6728		bfqq->pid = current->pid;6729		bfq_clear_bfqq_coop(bfqq);6730		bfq_clear_bfqq_split_coop(bfqq);6731		return bfqq;6732	}6733 6734	bic_set_bfqq(bic, NULL, true, bfqq->actuator_idx);6735 6736	bfq_put_cooperator(bfqq);6737	return NULL;6738}6739 6740static struct bfq_queue *6741__bfq_get_bfqq_handle_split(struct bfq_data *bfqd, struct bfq_io_cq *bic,6742			    struct bio *bio, bool split, bool is_sync,6743			    bool *new_queue)6744{6745	unsigned int act_idx = bfq_actuator_index(bfqd, bio);6746	struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx);6747	struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[act_idx];6748 6749	if (likely(bfqq && bfqq != &bfqd->oom_bfqq))6750		return bfqq;6751 6752	if (new_queue)6753		*new_queue = true;6754 6755	if (bfqq)6756		bfq_put_queue(bfqq);6757	bfqq = bfq_get_queue(bfqd, bio, is_sync, bic, split);6758 6759	bic_set_bfqq(bic, bfqq, is_sync, act_idx);6760	if (split && is_sync) {6761		if ((bfqq_data->was_in_burst_list && bfqd->large_burst) ||6762		    bfqq_data->saved_in_large_burst)6763			bfq_mark_bfqq_in_large_burst(bfqq);6764		else {6765			bfq_clear_bfqq_in_large_burst(bfqq);6766			if (bfqq_data->was_in_burst_list)6767				/*6768				 * If bfqq was in the current6769				 * burst list before being6770				 * merged, then we have to add6771				 * it back. And we do not need6772				 * to increase burst_size, as6773				 * we did not decrement6774				 * burst_size when we removed6775				 * bfqq from the burst list as6776				 * a consequence of a merge6777				 * (see comments in6778				 * bfq_put_queue). In this6779				 * respect, it would be rather6780				 * costly to know whether the6781				 * current burst list is still6782				 * the same burst list from6783				 * which bfqq was removed on6784				 * the merge. To avoid this6785				 * cost, if bfqq was in a6786				 * burst list, then we add6787				 * bfqq to the current burst6788				 * list without any further6789				 * check. This can cause6790				 * inappropriate insertions,6791				 * but rarely enough to not6792				 * harm the detection of large6793				 * bursts significantly.6794				 */6795				hlist_add_head(&bfqq->burst_list_node,6796					       &bfqd->burst_list);6797		}6798		bfqq->split_time = jiffies;6799	}6800 6801	return bfqq;6802}6803 6804/*6805 * Only reset private fields. The actual request preparation will be6806 * performed by bfq_init_rq, when rq is either inserted or merged. See6807 * comments on bfq_init_rq for the reason behind this delayed6808 * preparation.6809 */6810static void bfq_prepare_request(struct request *rq)6811{6812	rq->elv.icq = ioc_find_get_icq(rq->q);6813 6814	/*6815	 * Regardless of whether we have an icq attached, we have to6816	 * clear the scheduler pointers, as they might point to6817	 * previously allocated bic/bfqq structs.6818	 */6819	rq->elv.priv[0] = rq->elv.priv[1] = NULL;6820}6821 6822static struct bfq_queue *bfq_waker_bfqq(struct bfq_queue *bfqq)6823{6824	struct bfq_queue *new_bfqq = bfqq->new_bfqq;6825	struct bfq_queue *waker_bfqq = bfqq->waker_bfqq;6826 6827	if (!waker_bfqq)6828		return NULL;6829 6830	while (new_bfqq) {6831		if (new_bfqq == waker_bfqq) {6832			/*6833			 * If waker_bfqq is in the merge chain, and current6834			 * is the only procress.6835			 */6836			if (bfqq_process_refs(waker_bfqq) == 1)6837				return NULL;6838			break;6839		}6840 6841		new_bfqq = new_bfqq->new_bfqq;6842	}6843 6844	return waker_bfqq;6845}6846 6847static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd,6848						   struct bfq_io_cq *bic,6849						   struct bio *bio,6850						   unsigned int idx,6851						   bool is_sync)6852{6853	struct bfq_queue *waker_bfqq;6854	struct bfq_queue *bfqq;6855	bool new_queue = false;6856 6857	bfqq = __bfq_get_bfqq_handle_split(bfqd, bic, bio, false, is_sync,6858					   &new_queue);6859	if (unlikely(new_queue))6860		return bfqq;6861 6862	/* If the queue was seeky for too long, break it apart. */6863	if (!bfq_bfqq_coop(bfqq) || !bfq_bfqq_split_coop(bfqq) ||6864	    bic->bfqq_data[idx].stably_merged)6865		return bfqq;6866 6867	waker_bfqq = bfq_waker_bfqq(bfqq);6868 6869	/* Update bic before losing reference to bfqq */6870	if (bfq_bfqq_in_large_burst(bfqq))6871		bic->bfqq_data[idx].saved_in_large_burst = true;6872 6873	bfqq = bfq_split_bfqq(bic, bfqq);6874	if (bfqq) {6875		bfq_bfqq_resume_state(bfqq, bfqd, bic, true);6876		return bfqq;6877	}6878 6879	bfqq = __bfq_get_bfqq_handle_split(bfqd, bic, bio, true, is_sync, NULL);6880	if (unlikely(bfqq == &bfqd->oom_bfqq))6881		return bfqq;6882 6883	bfq_bfqq_resume_state(bfqq, bfqd, bic, false);6884	bfqq->waker_bfqq = waker_bfqq;6885	bfqq->tentative_waker_bfqq = NULL;6886 6887	/*6888	 * If the waker queue disappears, then new_bfqq->waker_bfqq must be6889	 * reset. So insert new_bfqq into the6890	 * woken_list of the waker. See6891	 * bfq_check_waker for details.6892	 */6893	if (waker_bfqq)6894		hlist_add_head(&bfqq->woken_list_node,6895			       &bfqq->waker_bfqq->woken_list);6896 6897	return bfqq;6898}6899 6900/*6901 * If needed, init rq, allocate bfq data structures associated with6902 * rq, and increment reference counters in the destination bfq_queue6903 * for rq. Return the destination bfq_queue for rq, or NULL is rq is6904 * not associated with any bfq_queue.6905 *6906 * This function is invoked by the functions that perform rq insertion6907 * or merging. One may have expected the above preparation operations6908 * to be performed in bfq_prepare_request, and not delayed to when rq6909 * is inserted or merged. The rationale behind this delayed6910 * preparation is that, after the prepare_request hook is invoked for6911 * rq, rq may still be transformed into a request with no icq, i.e., a6912 * request not associated with any queue. No bfq hook is invoked to6913 * signal this transformation. As a consequence, should these6914 * preparation operations be performed when the prepare_request hook6915 * is invoked, and should rq be transformed one moment later, bfq6916 * would end up in an inconsistent state, because it would have6917 * incremented some queue counters for an rq destined to6918 * transformation, without any chance to correctly lower these6919 * counters back. In contrast, no transformation can still happen for6920 * rq after rq has been inserted or merged. So, it is safe to execute6921 * these preparation operations when rq is finally inserted or merged.6922 */6923static struct bfq_queue *bfq_init_rq(struct request *rq)6924{6925	struct request_queue *q = rq->q;6926	struct bio *bio = rq->bio;6927	struct bfq_data *bfqd = q->elevator->elevator_data;6928	struct bfq_io_cq *bic;6929	const int is_sync = rq_is_sync(rq);6930	struct bfq_queue *bfqq;6931	unsigned int a_idx = bfq_actuator_index(bfqd, bio);6932 6933	if (unlikely(!rq->elv.icq))6934		return NULL;6935 6936	/*6937	 * Assuming that RQ_BFQQ(rq) is set only if everything is set6938	 * for this rq. This holds true, because this function is6939	 * invoked only for insertion or merging, and, after such6940	 * events, a request cannot be manipulated any longer before6941	 * being removed from bfq.6942	 */6943	if (RQ_BFQQ(rq))6944		return RQ_BFQQ(rq);6945 6946	bic = icq_to_bic(rq->elv.icq);6947	bfq_check_ioprio_change(bic, bio);6948	bfq_bic_update_cgroup(bic, bio);6949	bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio, a_idx, is_sync);6950 6951	bfqq_request_allocated(bfqq);6952	bfqq->ref++;6953	bic->requests++;6954	bfq_log_bfqq(bfqd, bfqq, "get_request %p: bfqq %p, %d",6955		     rq, bfqq, bfqq->ref);6956 6957	rq->elv.priv[0] = bic;6958	rq->elv.priv[1] = bfqq;6959 6960	/*6961	 * If a bfq_queue has only one process reference, it is owned6962	 * by only this bic: we can then set bfqq->bic = bic. in6963	 * addition, if the queue has also just been split, we have to6964	 * resume its state.6965	 */6966	if (likely(bfqq != &bfqd->oom_bfqq) && !bfqq->new_bfqq &&6967	    bfqq_process_refs(bfqq) == 1)6968		bfqq->bic = bic;6969 6970	/*6971	 * Consider bfqq as possibly belonging to a burst of newly6972	 * created queues only if:6973	 * 1) A burst is actually happening (bfqd->burst_size > 0)6974	 * or6975	 * 2) There is no other active queue. In fact, if, in6976	 *    contrast, there are active queues not belonging to the6977	 *    possible burst bfqq may belong to, then there is no gain6978	 *    in considering bfqq as belonging to a burst, and6979	 *    therefore in not weight-raising bfqq. See comments on6980	 *    bfq_handle_burst().6981	 *6982	 * This filtering also helps eliminating false positives,6983	 * occurring when bfqq does not belong to an actual large6984	 * burst, but some background task (e.g., a service) happens6985	 * to trigger the creation of new queues very close to when6986	 * bfqq and its possible companion queues are created. See6987	 * comments on bfq_handle_burst() for further details also on6988	 * this issue.6989	 */6990	if (unlikely(bfq_bfqq_just_created(bfqq) &&6991		     (bfqd->burst_size > 0 ||6992		      bfq_tot_busy_queues(bfqd) == 0)))6993		bfq_handle_burst(bfqd, bfqq);6994 6995	return bfqq;6996}6997 6998static void6999bfq_idle_slice_timer_body(struct bfq_data *bfqd, struct bfq_queue *bfqq)7000{7001	enum bfqq_expiration reason;7002	unsigned long flags;7003 7004	spin_lock_irqsave(&bfqd->lock, flags);7005 7006	/*7007	 * Considering that bfqq may be in race, we should firstly check7008	 * whether bfqq is in service before doing something on it. If7009	 * the bfqq in race is not in service, it has already been expired7010	 * through __bfq_bfqq_expire func and its wait_request flags has7011	 * been cleared in __bfq_bfqd_reset_in_service func.7012	 */7013	if (bfqq != bfqd->in_service_queue) {7014		spin_unlock_irqrestore(&bfqd->lock, flags);7015		return;7016	}7017 7018	bfq_clear_bfqq_wait_request(bfqq);7019 7020	if (bfq_bfqq_budget_timeout(bfqq))7021		/*7022		 * Also here the queue can be safely expired7023		 * for budget timeout without wasting7024		 * guarantees7025		 */7026		reason = BFQQE_BUDGET_TIMEOUT;7027	else if (bfqq->queued[0] == 0 && bfqq->queued[1] == 0)7028		/*7029		 * The queue may not be empty upon timer expiration,7030		 * because we may not disable the timer when the7031		 * first request of the in-service queue arrives7032		 * during disk idling.7033		 */7034		reason = BFQQE_TOO_IDLE;7035	else7036		goto schedule_dispatch;7037 7038	bfq_bfqq_expire(bfqd, bfqq, true, reason);7039 7040schedule_dispatch:7041	bfq_schedule_dispatch(bfqd);7042	spin_unlock_irqrestore(&bfqd->lock, flags);7043}7044 7045/*7046 * Handler of the expiration of the timer running if the in-service queue7047 * is idling inside its time slice.7048 */7049static enum hrtimer_restart bfq_idle_slice_timer(struct hrtimer *timer)7050{7051	struct bfq_data *bfqd = container_of(timer, struct bfq_data,7052					     idle_slice_timer);7053	struct bfq_queue *bfqq = bfqd->in_service_queue;7054 7055	/*7056	 * Theoretical race here: the in-service queue can be NULL or7057	 * different from the queue that was idling if a new request7058	 * arrives for the current queue and there is a full dispatch7059	 * cycle that changes the in-service queue.  This can hardly7060	 * happen, but in the worst case we just expire a queue too7061	 * early.7062	 */7063	if (bfqq)7064		bfq_idle_slice_timer_body(bfqd, bfqq);7065 7066	return HRTIMER_NORESTART;7067}7068 7069static void __bfq_put_async_bfqq(struct bfq_data *bfqd,7070				 struct bfq_queue **bfqq_ptr)7071{7072	struct bfq_queue *bfqq = *bfqq_ptr;7073 7074	bfq_log(bfqd, "put_async_bfqq: %p", bfqq);7075	if (bfqq) {7076		bfq_bfqq_move(bfqd, bfqq, bfqd->root_group);7077 7078		bfq_log_bfqq(bfqd, bfqq, "put_async_bfqq: putting %p, %d",7079			     bfqq, bfqq->ref);7080		bfq_put_queue(bfqq);7081		*bfqq_ptr = NULL;7082	}7083}7084 7085/*7086 * Release all the bfqg references to its async queues.  If we are7087 * deallocating the group these queues may still contain requests, so7088 * we reparent them to the root cgroup (i.e., the only one that will7089 * exist for sure until all the requests on a device are gone).7090 */7091void bfq_put_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg)7092{7093	int i, j, k;7094 7095	for (k = 0; k < bfqd->num_actuators; k++) {7096		for (i = 0; i < 2; i++)7097			for (j = 0; j < IOPRIO_NR_LEVELS; j++)7098				__bfq_put_async_bfqq(bfqd, &bfqg->async_bfqq[i][j][k]);7099 7100		__bfq_put_async_bfqq(bfqd, &bfqg->async_idle_bfqq[k]);7101	}7102}7103 7104/*7105 * See the comments on bfq_limit_depth for the purpose of7106 * the depths set in the function. Return minimum shallow depth we'll use.7107 */7108static void bfq_update_depths(struct bfq_data *bfqd, struct sbitmap_queue *bt)7109{7110	unsigned int depth = 1U << bt->sb.shift;7111 7112	bfqd->full_depth_shift = bt->sb.shift;7113	/*7114	 * In-word depths if no bfq_queue is being weight-raised:7115	 * leaving 25% of tags only for sync reads.7116	 *7117	 * In next formulas, right-shift the value7118	 * (1U<<bt->sb.shift), instead of computing directly7119	 * (1U<<(bt->sb.shift - something)), to be robust against7120	 * any possible value of bt->sb.shift, without having to7121	 * limit 'something'.7122	 */7123	/* no more than 50% of tags for async I/O */7124	bfqd->word_depths[0][0] = max(depth >> 1, 1U);7125	/*7126	 * no more than 75% of tags for sync writes (25% extra tags7127	 * w.r.t. async I/O, to prevent async I/O from starving sync7128	 * writes)7129	 */7130	bfqd->word_depths[0][1] = max((depth * 3) >> 2, 1U);7131 7132	/*7133	 * In-word depths in case some bfq_queue is being weight-7134	 * raised: leaving ~63% of tags for sync reads. This is the7135	 * highest percentage for which, in our tests, application7136	 * start-up times didn't suffer from any regression due to tag7137	 * shortage.7138	 */7139	/* no more than ~18% of tags for async I/O */7140	bfqd->word_depths[1][0] = max((depth * 3) >> 4, 1U);7141	/* no more than ~37% of tags for sync writes (~20% extra tags) */7142	bfqd->word_depths[1][1] = max((depth * 6) >> 4, 1U);7143}7144 7145static void bfq_depth_updated(struct blk_mq_hw_ctx *hctx)7146{7147	struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;7148	struct blk_mq_tags *tags = hctx->sched_tags;7149 7150	bfq_update_depths(bfqd, &tags->bitmap_tags);7151	sbitmap_queue_min_shallow_depth(&tags->bitmap_tags, 1);7152}7153 7154static int bfq_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int index)7155{7156	bfq_depth_updated(hctx);7157	return 0;7158}7159 7160static void bfq_exit_queue(struct elevator_queue *e)7161{7162	struct bfq_data *bfqd = e->elevator_data;7163	struct bfq_queue *bfqq, *n;7164	unsigned int actuator;7165 7166	hrtimer_cancel(&bfqd->idle_slice_timer);7167 7168	spin_lock_irq(&bfqd->lock);7169	list_for_each_entry_safe(bfqq, n, &bfqd->idle_list, bfqq_list)7170		bfq_deactivate_bfqq(bfqd, bfqq, false, false);7171	spin_unlock_irq(&bfqd->lock);7172 7173	for (actuator = 0; actuator < bfqd->num_actuators; actuator++)7174		WARN_ON_ONCE(bfqd->rq_in_driver[actuator]);7175	WARN_ON_ONCE(bfqd->tot_rq_in_driver);7176 7177	hrtimer_cancel(&bfqd->idle_slice_timer);7178 7179	/* release oom-queue reference to root group */7180	bfqg_and_blkg_put(bfqd->root_group);7181 7182#ifdef CONFIG_BFQ_GROUP_IOSCHED7183	blkcg_deactivate_policy(bfqd->queue->disk, &blkcg_policy_bfq);7184#else7185	spin_lock_irq(&bfqd->lock);7186	bfq_put_async_queues(bfqd, bfqd->root_group);7187	kfree(bfqd->root_group);7188	spin_unlock_irq(&bfqd->lock);7189#endif7190 7191	blk_stat_disable_accounting(bfqd->queue);7192	clear_bit(ELEVATOR_FLAG_DISABLE_WBT, &e->flags);7193	wbt_enable_default(bfqd->queue->disk);7194 7195	kfree(bfqd);7196}7197 7198static void bfq_init_root_group(struct bfq_group *root_group,7199				struct bfq_data *bfqd)7200{7201	int i;7202 7203#ifdef CONFIG_BFQ_GROUP_IOSCHED7204	root_group->entity.parent = NULL;7205	root_group->my_entity = NULL;7206	root_group->bfqd = bfqd;7207#endif7208	root_group->rq_pos_tree = RB_ROOT;7209	for (i = 0; i < BFQ_IOPRIO_CLASSES; i++)7210		root_group->sched_data.service_tree[i] = BFQ_SERVICE_TREE_INIT;7211	root_group->sched_data.bfq_class_idle_last_service = jiffies;7212}7213 7214static int bfq_init_queue(struct request_queue *q, struct elevator_type *e)7215{7216	struct bfq_data *bfqd;7217	struct elevator_queue *eq;7218	unsigned int i;7219	struct blk_independent_access_ranges *ia_ranges = q->disk->ia_ranges;7220 7221	eq = elevator_alloc(q, e);7222	if (!eq)7223		return -ENOMEM;7224 7225	bfqd = kzalloc_node(sizeof(*bfqd), GFP_KERNEL, q->node);7226	if (!bfqd) {7227		kobject_put(&eq->kobj);7228		return -ENOMEM;7229	}7230	eq->elevator_data = bfqd;7231 7232	spin_lock_irq(&q->queue_lock);7233	q->elevator = eq;7234	spin_unlock_irq(&q->queue_lock);7235 7236	/*7237	 * Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues.7238	 * Grab a permanent reference to it, so that the normal code flow7239	 * will not attempt to free it.7240	 * Set zero as actuator index: we will pretend that7241	 * all I/O requests are for the same actuator.7242	 */7243	bfq_init_bfqq(bfqd, &bfqd->oom_bfqq, NULL, 1, 0, 0);7244	bfqd->oom_bfqq.ref++;7245	bfqd->oom_bfqq.new_ioprio = BFQ_DEFAULT_QUEUE_IOPRIO;7246	bfqd->oom_bfqq.new_ioprio_class = IOPRIO_CLASS_BE;7247	bfqd->oom_bfqq.entity.new_weight =7248		bfq_ioprio_to_weight(bfqd->oom_bfqq.new_ioprio);7249 7250	/* oom_bfqq does not participate to bursts */7251	bfq_clear_bfqq_just_created(&bfqd->oom_bfqq);7252 7253	/*7254	 * Trigger weight initialization, according to ioprio, at the7255	 * oom_bfqq's first activation. The oom_bfqq's ioprio and ioprio7256	 * class won't be changed any more.7257	 */7258	bfqd->oom_bfqq.entity.prio_changed = 1;7259 7260	bfqd->queue = q;7261 7262	bfqd->num_actuators = 1;7263	/*7264	 * If the disk supports multiple actuators, copy independent7265	 * access ranges from the request queue structure.7266	 */7267	spin_lock_irq(&q->queue_lock);7268	if (ia_ranges) {7269		/*7270		 * Check if the disk ia_ranges size exceeds the current bfq7271		 * actuator limit.7272		 */7273		if (ia_ranges->nr_ia_ranges > BFQ_MAX_ACTUATORS) {7274			pr_crit("nr_ia_ranges higher than act limit: iars=%d, max=%d.\n",7275				ia_ranges->nr_ia_ranges, BFQ_MAX_ACTUATORS);7276			pr_crit("Falling back to single actuator mode.\n");7277		} else {7278			bfqd->num_actuators = ia_ranges->nr_ia_ranges;7279 7280			for (i = 0; i < bfqd->num_actuators; i++) {7281				bfqd->sector[i] = ia_ranges->ia_range[i].sector;7282				bfqd->nr_sectors[i] =7283					ia_ranges->ia_range[i].nr_sectors;7284			}7285		}7286	}7287 7288	/* Otherwise use single-actuator dev info */7289	if (bfqd->num_actuators == 1) {7290		bfqd->sector[0] = 0;7291		bfqd->nr_sectors[0] = get_capacity(q->disk);7292	}7293	spin_unlock_irq(&q->queue_lock);7294 7295	INIT_LIST_HEAD(&bfqd->dispatch);7296 7297	hrtimer_init(&bfqd->idle_slice_timer, CLOCK_MONOTONIC,7298		     HRTIMER_MODE_REL);7299	bfqd->idle_slice_timer.function = bfq_idle_slice_timer;7300 7301	bfqd->queue_weights_tree = RB_ROOT_CACHED;7302#ifdef CONFIG_BFQ_GROUP_IOSCHED7303	bfqd->num_groups_with_pending_reqs = 0;7304#endif7305 7306	INIT_LIST_HEAD(&bfqd->active_list[0]);7307	INIT_LIST_HEAD(&bfqd->active_list[1]);7308	INIT_LIST_HEAD(&bfqd->idle_list);7309	INIT_HLIST_HEAD(&bfqd->burst_list);7310 7311	bfqd->hw_tag = -1;7312	bfqd->nonrot_with_queueing = blk_queue_nonrot(bfqd->queue);7313 7314	bfqd->bfq_max_budget = bfq_default_max_budget;7315 7316	bfqd->bfq_fifo_expire[0] = bfq_fifo_expire[0];7317	bfqd->bfq_fifo_expire[1] = bfq_fifo_expire[1];7318	bfqd->bfq_back_max = bfq_back_max;7319	bfqd->bfq_back_penalty = bfq_back_penalty;7320	bfqd->bfq_slice_idle = bfq_slice_idle;7321	bfqd->bfq_timeout = bfq_timeout;7322 7323	bfqd->bfq_large_burst_thresh = 8;7324	bfqd->bfq_burst_interval = msecs_to_jiffies(180);7325 7326	bfqd->low_latency = true;7327 7328	/*7329	 * Trade-off between responsiveness and fairness.7330	 */7331	bfqd->bfq_wr_coeff = 30;7332	bfqd->bfq_wr_rt_max_time = msecs_to_jiffies(300);7333	bfqd->bfq_wr_min_idle_time = msecs_to_jiffies(2000);7334	bfqd->bfq_wr_min_inter_arr_async = msecs_to_jiffies(500);7335	bfqd->bfq_wr_max_softrt_rate = 7000; /*7336					      * Approximate rate required7337					      * to playback or record a7338					      * high-definition compressed7339					      * video.7340					      */7341	bfqd->wr_busy_queues = 0;7342 7343	/*7344	 * Begin by assuming, optimistically, that the device peak7345	 * rate is equal to 2/3 of the highest reference rate.7346	 */7347	bfqd->rate_dur_prod = ref_rate[blk_queue_nonrot(bfqd->queue)] *7348		ref_wr_duration[blk_queue_nonrot(bfqd->queue)];7349	bfqd->peak_rate = ref_rate[blk_queue_nonrot(bfqd->queue)] * 2 / 3;7350 7351	/* see comments on the definition of next field inside bfq_data */7352	bfqd->actuator_load_threshold = 4;7353 7354	spin_lock_init(&bfqd->lock);7355 7356	/*7357	 * The invocation of the next bfq_create_group_hierarchy7358	 * function is the head of a chain of function calls7359	 * (bfq_create_group_hierarchy->blkcg_activate_policy->7360	 * blk_mq_freeze_queue) that may lead to the invocation of the7361	 * has_work hook function. For this reason,7362	 * bfq_create_group_hierarchy is invoked only after all7363	 * scheduler data has been initialized, apart from the fields7364	 * that can be initialized only after invoking7365	 * bfq_create_group_hierarchy. This, in particular, enables7366	 * has_work to correctly return false. Of course, to avoid7367	 * other inconsistencies, the blk-mq stack must then refrain7368	 * from invoking further scheduler hooks before this init7369	 * function is finished.7370	 */7371	bfqd->root_group = bfq_create_group_hierarchy(bfqd, q->node);7372	if (!bfqd->root_group)7373		goto out_free;7374	bfq_init_root_group(bfqd->root_group, bfqd);7375	bfq_init_entity(&bfqd->oom_bfqq.entity, bfqd->root_group);7376 7377	/* We dispatch from request queue wide instead of hw queue */7378	blk_queue_flag_set(QUEUE_FLAG_SQ_SCHED, q);7379 7380	set_bit(ELEVATOR_FLAG_DISABLE_WBT, &eq->flags);7381	wbt_disable_default(q->disk);7382	blk_stat_enable_accounting(q);7383 7384	return 0;7385 7386out_free:7387	kfree(bfqd);7388	kobject_put(&eq->kobj);7389	return -ENOMEM;7390}7391 7392static void bfq_slab_kill(void)7393{7394	kmem_cache_destroy(bfq_pool);7395}7396 7397static int __init bfq_slab_setup(void)7398{7399	bfq_pool = KMEM_CACHE(bfq_queue, 0);7400	if (!bfq_pool)7401		return -ENOMEM;7402	return 0;7403}7404 7405static ssize_t bfq_var_show(unsigned int var, char *page)7406{7407	return sprintf(page, "%u\n", var);7408}7409 7410static int bfq_var_store(unsigned long *var, const char *page)7411{7412	unsigned long new_val;7413	int ret = kstrtoul(page, 10, &new_val);7414 7415	if (ret)7416		return ret;7417	*var = new_val;7418	return 0;7419}7420 7421#define SHOW_FUNCTION(__FUNC, __VAR, __CONV)				\7422static ssize_t __FUNC(struct elevator_queue *e, char *page)		\7423{									\7424	struct bfq_data *bfqd = e->elevator_data;			\7425	u64 __data = __VAR;						\7426	if (__CONV == 1)						\7427		__data = jiffies_to_msecs(__data);			\7428	else if (__CONV == 2)						\7429		__data = div_u64(__data, NSEC_PER_MSEC);		\7430	return bfq_var_show(__data, (page));				\7431}7432SHOW_FUNCTION(bfq_fifo_expire_sync_show, bfqd->bfq_fifo_expire[1], 2);7433SHOW_FUNCTION(bfq_fifo_expire_async_show, bfqd->bfq_fifo_expire[0], 2);7434SHOW_FUNCTION(bfq_back_seek_max_show, bfqd->bfq_back_max, 0);7435SHOW_FUNCTION(bfq_back_seek_penalty_show, bfqd->bfq_back_penalty, 0);7436SHOW_FUNCTION(bfq_slice_idle_show, bfqd->bfq_slice_idle, 2);7437SHOW_FUNCTION(bfq_max_budget_show, bfqd->bfq_user_max_budget, 0);7438SHOW_FUNCTION(bfq_timeout_sync_show, bfqd->bfq_timeout, 1);7439SHOW_FUNCTION(bfq_strict_guarantees_show, bfqd->strict_guarantees, 0);7440SHOW_FUNCTION(bfq_low_latency_show, bfqd->low_latency, 0);7441#undef SHOW_FUNCTION7442 7443#define USEC_SHOW_FUNCTION(__FUNC, __VAR)				\7444static ssize_t __FUNC(struct elevator_queue *e, char *page)		\7445{									\7446	struct bfq_data *bfqd = e->elevator_data;			\7447	u64 __data = __VAR;						\7448	__data = div_u64(__data, NSEC_PER_USEC);			\7449	return bfq_var_show(__data, (page));				\7450}7451USEC_SHOW_FUNCTION(bfq_slice_idle_us_show, bfqd->bfq_slice_idle);7452#undef USEC_SHOW_FUNCTION7453 7454#define STORE_FUNCTION(__FUNC, __PTR, MIN, MAX, __CONV)			\7455static ssize_t								\7456__FUNC(struct elevator_queue *e, const char *page, size_t count)	\7457{									\7458	struct bfq_data *bfqd = e->elevator_data;			\7459	unsigned long __data, __min = (MIN), __max = (MAX);		\7460	int ret;							\7461									\7462	ret = bfq_var_store(&__data, (page));				\7463	if (ret)							\7464		return ret;						\7465	if (__data < __min)						\7466		__data = __min;						\7467	else if (__data > __max)					\7468		__data = __max;						\7469	if (__CONV == 1)						\7470		*(__PTR) = msecs_to_jiffies(__data);			\7471	else if (__CONV == 2)						\7472		*(__PTR) = (u64)__data * NSEC_PER_MSEC;			\7473	else								\7474		*(__PTR) = __data;					\7475	return count;							\7476}7477STORE_FUNCTION(bfq_fifo_expire_sync_store, &bfqd->bfq_fifo_expire[1], 1,7478		INT_MAX, 2);7479STORE_FUNCTION(bfq_fifo_expire_async_store, &bfqd->bfq_fifo_expire[0], 1,7480		INT_MAX, 2);7481STORE_FUNCTION(bfq_back_seek_max_store, &bfqd->bfq_back_max, 0, INT_MAX, 0);7482STORE_FUNCTION(bfq_back_seek_penalty_store, &bfqd->bfq_back_penalty, 1,7483		INT_MAX, 0);7484STORE_FUNCTION(bfq_slice_idle_store, &bfqd->bfq_slice_idle, 0, INT_MAX, 2);7485#undef STORE_FUNCTION7486 7487#define USEC_STORE_FUNCTION(__FUNC, __PTR, MIN, MAX)			\7488static ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count)\7489{									\7490	struct bfq_data *bfqd = e->elevator_data;			\7491	unsigned long __data, __min = (MIN), __max = (MAX);		\7492	int ret;							\7493									\7494	ret = bfq_var_store(&__data, (page));				\7495	if (ret)							\7496		return ret;						\7497	if (__data < __min)						\7498		__data = __min;						\7499	else if (__data > __max)					\7500		__data = __max;						\7501	*(__PTR) = (u64)__data * NSEC_PER_USEC;				\7502	return count;							\7503}7504USEC_STORE_FUNCTION(bfq_slice_idle_us_store, &bfqd->bfq_slice_idle, 0,7505		    UINT_MAX);7506#undef USEC_STORE_FUNCTION7507 7508static ssize_t bfq_max_budget_store(struct elevator_queue *e,7509				    const char *page, size_t count)7510{7511	struct bfq_data *bfqd = e->elevator_data;7512	unsigned long __data;7513	int ret;7514 7515	ret = bfq_var_store(&__data, (page));7516	if (ret)7517		return ret;7518 7519	if (__data == 0)7520		bfqd->bfq_max_budget = bfq_calc_max_budget(bfqd);7521	else {7522		if (__data > INT_MAX)7523			__data = INT_MAX;7524		bfqd->bfq_max_budget = __data;7525	}7526 7527	bfqd->bfq_user_max_budget = __data;7528 7529	return count;7530}7531 7532/*7533 * Leaving this name to preserve name compatibility with cfq7534 * parameters, but this timeout is used for both sync and async.7535 */7536static ssize_t bfq_timeout_sync_store(struct elevator_queue *e,7537				      const char *page, size_t count)7538{7539	struct bfq_data *bfqd = e->elevator_data;7540	unsigned long __data;7541	int ret;7542 7543	ret = bfq_var_store(&__data, (page));7544	if (ret)7545		return ret;7546 7547	if (__data < 1)7548		__data = 1;7549	else if (__data > INT_MAX)7550		__data = INT_MAX;7551 7552	bfqd->bfq_timeout = msecs_to_jiffies(__data);7553	if (bfqd->bfq_user_max_budget == 0)7554		bfqd->bfq_max_budget = bfq_calc_max_budget(bfqd);7555 7556	return count;7557}7558 7559static ssize_t bfq_strict_guarantees_store(struct elevator_queue *e,7560				     const char *page, size_t count)7561{7562	struct bfq_data *bfqd = e->elevator_data;7563	unsigned long __data;7564	int ret;7565 7566	ret = bfq_var_store(&__data, (page));7567	if (ret)7568		return ret;7569 7570	if (__data > 1)7571		__data = 1;7572	if (!bfqd->strict_guarantees && __data == 17573	    && bfqd->bfq_slice_idle < 8 * NSEC_PER_MSEC)7574		bfqd->bfq_slice_idle = 8 * NSEC_PER_MSEC;7575 7576	bfqd->strict_guarantees = __data;7577 7578	return count;7579}7580 7581static ssize_t bfq_low_latency_store(struct elevator_queue *e,7582				     const char *page, size_t count)7583{7584	struct bfq_data *bfqd = e->elevator_data;7585	unsigned long __data;7586	int ret;7587 7588	ret = bfq_var_store(&__data, (page));7589	if (ret)7590		return ret;7591 7592	if (__data > 1)7593		__data = 1;7594	if (__data == 0 && bfqd->low_latency != 0)7595		bfq_end_wr(bfqd);7596	bfqd->low_latency = __data;7597 7598	return count;7599}7600 7601#define BFQ_ATTR(name) \7602	__ATTR(name, 0644, bfq_##name##_show, bfq_##name##_store)7603 7604static struct elv_fs_entry bfq_attrs[] = {7605	BFQ_ATTR(fifo_expire_sync),7606	BFQ_ATTR(fifo_expire_async),7607	BFQ_ATTR(back_seek_max),7608	BFQ_ATTR(back_seek_penalty),7609	BFQ_ATTR(slice_idle),7610	BFQ_ATTR(slice_idle_us),7611	BFQ_ATTR(max_budget),7612	BFQ_ATTR(timeout_sync),7613	BFQ_ATTR(strict_guarantees),7614	BFQ_ATTR(low_latency),7615	__ATTR_NULL7616};7617 7618static struct elevator_type iosched_bfq_mq = {7619	.ops = {7620		.limit_depth		= bfq_limit_depth,7621		.prepare_request	= bfq_prepare_request,7622		.requeue_request        = bfq_finish_requeue_request,7623		.finish_request		= bfq_finish_request,7624		.exit_icq		= bfq_exit_icq,7625		.insert_requests	= bfq_insert_requests,7626		.dispatch_request	= bfq_dispatch_request,7627		.next_request		= elv_rb_latter_request,7628		.former_request		= elv_rb_former_request,7629		.allow_merge		= bfq_allow_bio_merge,7630		.bio_merge		= bfq_bio_merge,7631		.request_merge		= bfq_request_merge,7632		.requests_merged	= bfq_requests_merged,7633		.request_merged		= bfq_request_merged,7634		.has_work		= bfq_has_work,7635		.depth_updated		= bfq_depth_updated,7636		.init_hctx		= bfq_init_hctx,7637		.init_sched		= bfq_init_queue,7638		.exit_sched		= bfq_exit_queue,7639	},7640 7641	.icq_size =		sizeof(struct bfq_io_cq),7642	.icq_align =		__alignof__(struct bfq_io_cq),7643	.elevator_attrs =	bfq_attrs,7644	.elevator_name =	"bfq",7645	.elevator_owner =	THIS_MODULE,7646};7647MODULE_ALIAS("bfq-iosched");7648 7649static int __init bfq_init(void)7650{7651	int ret;7652 7653#ifdef CONFIG_BFQ_GROUP_IOSCHED7654	ret = blkcg_policy_register(&blkcg_policy_bfq);7655	if (ret)7656		return ret;7657#endif7658 7659	ret = -ENOMEM;7660	if (bfq_slab_setup())7661		goto err_pol_unreg;7662 7663	/*7664	 * Times to load large popular applications for the typical7665	 * systems installed on the reference devices (see the7666	 * comments before the definition of the next7667	 * array). Actually, we use slightly lower values, as the7668	 * estimated peak rate tends to be smaller than the actual7669	 * peak rate.  The reason for this last fact is that estimates7670	 * are computed over much shorter time intervals than the long7671	 * intervals typically used for benchmarking. Why? First, to7672	 * adapt more quickly to variations. Second, because an I/O7673	 * scheduler cannot rely on a peak-rate-evaluation workload to7674	 * be run for a long time.7675	 */7676	ref_wr_duration[0] = msecs_to_jiffies(7000); /* actually 8 sec */7677	ref_wr_duration[1] = msecs_to_jiffies(2500); /* actually 3 sec */7678 7679	ret = elv_register(&iosched_bfq_mq);7680	if (ret)7681		goto slab_kill;7682 7683	return 0;7684 7685slab_kill:7686	bfq_slab_kill();7687err_pol_unreg:7688#ifdef CONFIG_BFQ_GROUP_IOSCHED7689	blkcg_policy_unregister(&blkcg_policy_bfq);7690#endif7691	return ret;7692}7693 7694static void __exit bfq_exit(void)7695{7696	elv_unregister(&iosched_bfq_mq);7697#ifdef CONFIG_BFQ_GROUP_IOSCHED7698	blkcg_policy_unregister(&blkcg_policy_bfq);7699#endif7700	bfq_slab_kill();7701}7702 7703module_init(bfq_init);7704module_exit(bfq_exit);7705 7706MODULE_AUTHOR("Paolo Valente");7707MODULE_LICENSE("GPL");7708MODULE_DESCRIPTION("MQ Budget Fair Queueing I/O Scheduler");7709