Waiting without freezing

In lesson 9.2, Model::reply sends the conversation with reqwest::blocking. Blocking means what it sounds like: the program stops on that line until the whole answer has come back. A model writes its answer a few words at a time, and a long one can take many seconds, but Chatterbox sees none of it until the last word arrives. It cannot print the first words, and it cannot notice you pressing a key, because it is not doing anything. It is waiting.

Real chat apps show the answer as it is written. For that, a program has to wait for several things at once: the next piece of the answer, and you. Rust's way to do that is async code, code that can pause at a wait and let other work run meanwhile. This lesson teaches it on small programs; Chatterbox stays at v1.0 until the next lesson.

Put async in front of fn and the function changes in one surprising way: calling it does not run it. It hands back a future, a value that describes the work, like a recipe nobody has cooked yet. The work happens when you write .await after the future. .await means "do this now, and if it has to wait, pause here until it is done".

You have met this laziness before. In lesson 6.5, .map(...) on an iterator did nothing until .collect() asked for the results. A future is the same: nothing happens until something asks. (The #[tokio::main] line is explained just below.) Guess the first line this prints:

Something has to run futures: keep track of which ones are waiting, and wake each one up when what it waits for is ready. That something is called a runtime. The Rust language gives you async and .await but no runtime, so you pick one from a crate. The most used one is tokio.

The line #[tokio::main] above async fn main() is an attribute, like #[test] in lesson 7.3. It rewrites main so that it starts a tokio runtime and hands it your async main to run. Without it, nobody does the cooking, and the compiler says so. Here is the mistake almost everyone makes first: .await inside a plain main.

error[E0728]: `await` is only allowed inside `async` functions and blocks
 --> src/main.rs:6:29
  |
5 | fn main() {
  | --------- this is not `async`
6 |     let text = greet("Sam").await;
  |                             ^^^^^ only allowed inside `async` functions and blocks

Now the point of all this. tokio::time::sleep(Duration::from_millis(1000)) makes a future that is ready after 1000 milliseconds, one second. Duration is the standard library's type for a length of time. Instant::now() notes the current moment, so that start.elapsed() can tell how long has passed since, and .as_secs() gives that in whole seconds. Awaiting two sleeps one after the other takes two seconds, as you would expect:

$ cargo run
first answer is ready
second answer is ready
took 2 s

tokio::join! is different. Like println!, it is a macro (the ! says so). It takes several futures, runs them all at once, waits until every one is done, and hands back their results together as a tuple (lesson 2.6). The same two one-second waits, now joined:

No second copy of the program ran. One runtime drove both futures, switching between them at their waits. That is why async is cheap: a future that is waiting costs almost nothing, so one program can have many waits going at once.

A future can only step aside at an .await. Code that waits without one never steps aside. std::thread::sleep is the blocking kind of sleep: it stops the whole thread (the worker that runs your code, step by step), and with it every future that thread is driving. Swap it into the joined program and the join stops helping:

$ cargo run
first answer is ready
second answer is ready
took 2 s

This is Chatterbox's problem in miniature. reqwest::blocking waits the way std::thread::sleep does. reqwest also has an async side, where .send() gives back a future you .await, and the rest of this chapter switches to it.

So the plan for Chatterbox sounds simple: make reply an async fn in the Replier trait. Rust does allow async functions in traits. But Chatterbox keeps its replier as Box<dyn Replier> (lesson 8.4), so that /model can swap the echo, the pretend model and the real one while it runs. Here is a cut-down version of that. Will it still compile?

error[E0038]: the trait `Replier` is not dyn compatible
  --> src/main.rs:14:26
   |
14 |     let replier: Box<dyn Replier> = Box::new(Echo);
   |                          ^^^^^^^ `Replier` is not dyn compatible
   |
note: for a trait to be dyn compatible it needs to allow building a vtable
      for more information, visit <https://doc.rust-lang.org/reference/items/traits.html#dyn-compatibility>
  --> src/main.rs:2:14
   |
 1 | trait Replier {
   |       ------- this trait is not dyn compatible...
 2 |     async fn reply(&self, message: &str) -> String;
   |              ^^^^^ ...because method `reply` is `async`
   = help: consider moving `reply` to another trait
   = help: only type `Echo` implements `Replier`; consider using it directly instead.

That is the problem the next lesson solves. It keeps reply a plain function that returns at once, and what it returns is a channel: a pipe the answer arrives through later, piece by piece. Every replier returns the same kind of channel, so Box<dyn Replier> works again, and each word can be printed the moment it comes out of the pipe.

Sources The Rust Book: Async and Await · Tokio tutorial · tokio::join! on docs.rs · Rust error index: E0728 · The Rust Reference: dyn compatibility