🏡
  • ↔
  • →
  1. Why make software? | Sean Voisen
  2. Building a new snapshot fuzzer & fuzzing IDA
  3. How to Build a Small Solar Power System | LOW←TECH MAGAZINE
  4. Szymon Kaliski — Q2 2024
  5. Terminal colours are tricky

generated: October 16, 2024 at 10:18:01

  1. October 15, 2024
    1. 🔗 astral-sh/uv 0.4.22 release

      Release Notes

      Enhancements

      • Respect [tool.uv.sources] in build requirements (#7172)

      Preview features

      • Add a dedicated uv publish error message for missing usernames (#8045)
      • Support interactive input in uv publish (#8158)
      • Use raw filenames in uv publish (#8204)

      Performance

      • Reuse the result of which git (#8224)

      Bug fixes

      • Avoid environment check optimization for uv pip install --exact (#8219)
      • Do not use free-threaded interpreters without a free-threaded request (#8191)
      • Don't recommend --prerelease=allow during build requirement resolution errors (#8192)
      • Prefer optimized builds for free-threaded Python downloads (#8196)
      • Retain old python-build-standalone releases (#8216)
      • Run uv build builds in the source distribution bucket (#8220)

      Install uv 0.4.22

      Install prebuilt binaries via shell script

      curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.4.22/uv-installer.sh | sh

      Install prebuilt binaries via powershell script

      powershell -ExecutionPolicy ByPass -c "irm https://github.com/astral-sh/uv/releases/download/0.4.22/uv-installer.ps1 | iex"

      Download uv 0.4.22

      File | Platform | Checksum
      ---|---|---
      uv-aarch64-apple-darwin.tar.gz | Apple Silicon macOS | checksum
      uv-x86_64-apple-darwin.tar.gz | Intel macOS | checksum
      uv-i686-pc-windows-msvc.zip | x86 Windows | checksum
      uv-x86_64-pc-windows-msvc.zip | x64 Windows | checksum
      uv-aarch64-unknown-linux-gnu.tar.gz | ARM64 Linux | checksum
      uv-i686-unknown-linux-gnu.tar.gz | x86 Linux | checksum
      uv-powerpc64-unknown-linux-gnu.tar.gz | PPC64 Linux | checksum
      uv-powerpc64le-unknown-linux-gnu.tar.gz | PPC64LE Linux | checksum
      uv-s390x-unknown-linux-gnu.tar.gz | S390x Linux | checksum
      uv-x86_64-unknown-linux-gnu.tar.gz | x64 Linux | checksum
      uv-armv7-unknown-linux-gnueabihf.tar.gz | ARMv7 Linux | checksum
      uv-aarch64-unknown-linux-musl.tar.gz | ARM64 MUSL Linux | checksum
      uv-i686-unknown-linux-musl.tar.gz | x86 MUSL Linux | checksum
      uv-x86_64-unknown-linux-musl.tar.gz | x64 MUSL Linux | checksum
      uv-arm-unknown-linux-musleabihf.tar.gz | ARMv6 MUSL Linux (Hardfloat) | checksum
      uv-armv7-unknown-linux-musleabihf.tar.gz | ARMv7 MUSL Linux | checksum

    2. 🔗 @trailofbits@infosec.exchange In this week's [@riskybusiness](https://infosec.exchange/@riskybusiness) mastodon

      In this week's @riskybusiness episode, @dguido talks about all things post-quantum cryptography:
      • Quantum computer timeline
      • PQC implementation challenges
      • Hybrid encryption solutions
      • Device lifespan considerations
      https://risky.biz/RBNEWSSI62/

      A few of our fav PQC resources:
      https://blog.trailofbits.com/2024/08/15/we-wrote-the-code-and-the-code- won/
      https://blog.trailofbits.com/2024/07/01/quantum-is-unimportant-to-post- quantum/
      https://blog.trailofbits.com/2024/08/21/yolo-is-not-a-valid-hash- construction/
      https://blog.trailofbits.com/category/cryptography/

    3. 🔗 Andrew Healey's Blog Compiling Lisp to Bytecode and Running It rss

      Extending my Lisp compiler and adding a fast virtual machine.

  2. October 14, 2024
    1. 🔗 astral-sh/uv 0.4.21 release

      Release Notes

      Enhancements

      • Add support for managed installations of free-threaded Python (#8100)
      • Add note about uvx to uv tool run short help (#7695)
      • Enable HTTP/2 requests (#8049)
      • Support uv tree --no-dev (#8109)
      • Support PEP 723 metadata with uv run - (#8111)
      • Support pip install --exact (#8044)
      • Support uv export --no-header (#8096)
      • ADd Python 3.13 images to Docker publish (#8105)
      • Support remote (https://) scripts in uv run (#6375)
      • Allow comma value-delimited arguments in uv run --with (#7909)

      Configuration

      • Support wildcards in UV_INSECURE_HOST (#8052)

      Performance

      • Use shared index when fetching metadata in lock satisfaction routine (#8147)

      Bug fixes

      • Add prerelease compatibility check to uv python CLI (#8020)
      • Avoid deleting a project environment directory if we cannot tell if a pyvenv.cfg file exists (#8012)
      • Avoid excluding valid wheels for exact requires-python bounds (#8140)
      • Bump netrc crate to latest commit (#8021)
      • Fix uv python pin 3.13t failure when parsing version for project requires check (#8056)
      • Fix handling of != intersections in requires-python (#7897)
      • Remove the newly created tool environment if sync failed (#8038)
      • Respect dynamic extras in uv lock and uv sync (#8091)
      • Treat resolver failures as fatal in lockfile validation (#8083)
      • Use git config --get for author information for improved backwards compatibility (#8101)
      • Use comma-separated values for UV_FIND_LINKS (#8061)
      • Use shared resolver state between add and lock to avoid double Git update (#8146)
      • Make --relocatable entrypoints robust to symlinking (#8079)
      • Improve compatibility with VSCode PS1 prompt (#8006)
      • Fix "Stream did not contain valid UTF-8" failures in Windows (#8120)
      • Use --with-requirements in uvx error hint (#8112)

      Documentation

      • Include uvx installation in Docker examples (#8179)
      • Make the instructions for the Windows standalone installer consistent across README and documentation (#8125)
      • Update pip compatibility guide to note transitive URL dependency support (#8081)
      • Document --reinstall with --exclude-newer to ensure downgrades (#6721)

      Install uv 0.4.21

      Install prebuilt binaries via shell script

      curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.4.21/uv-installer.sh | sh

      Install prebuilt binaries via powershell script

      powershell -ExecutionPolicy ByPass -c "irm https://github.com/astral-sh/uv/releases/download/0.4.21/uv-installer.ps1 | iex"

      Download uv 0.4.21

      File | Platform | Checksum
      ---|---|---
      uv-aarch64-apple-darwin.tar.gz | Apple Silicon macOS | checksum
      uv-x86_64-apple-darwin.tar.gz | Intel macOS | checksum
      uv-i686-pc-windows-msvc.zip | x86 Windows | checksum
      uv-x86_64-pc-windows-msvc.zip | x64 Windows | checksum
      uv-aarch64-unknown-linux-gnu.tar.gz | ARM64 Linux | checksum
      uv-i686-unknown-linux-gnu.tar.gz | x86 Linux | checksum
      uv-powerpc64-unknown-linux-gnu.tar.gz | PPC64 Linux | checksum
      uv-powerpc64le-unknown-linux-gnu.tar.gz | PPC64LE Linux | checksum
      uv-s390x-unknown-linux-gnu.tar.gz | S390x Linux | checksum
      uv-x86_64-unknown-linux-gnu.tar.gz | x64 Linux | checksum
      uv-armv7-unknown-linux-gnueabihf.tar.gz | ARMv7 Linux | checksum
      uv-aarch64-unknown-linux-musl.tar.gz | ARM64 MUSL Linux | checksum
      uv-i686-unknown-linux-musl.tar.gz | x86 MUSL Linux | checksum
      uv-x86_64-unknown-linux-musl.tar.gz | x64 MUSL Linux | checksum
      uv-arm-unknown-linux-musleabihf.tar.gz | ARMv6 MUSL Linux (Hardfloat) | checksum
      uv-armv7-unknown-linux-musleabihf.tar.gz | ARMv7 MUSL Linux | checksum

    2. 🔗 sacha chua :: living an awesome life Change Org Mode TODO keyword color based on the state and the current Modus theme rss

      I use modus-theme-toggle to switch between modus-vivendi-tinted and modus-operandi-tinted depending on whether I want a dark background or a light one. I also customize my org-todo-keyword-faces to visually distinguish TODO, DONE, WAITING, and SOMEDAY. This is how to colour them based on the current Modus theme. 

      (defun my-org-todo-set-keyword-faces ()
  (setq org-todo-keyword-faces
        `(("TODO" . (:foreground ,(modus-themes-get-color-value 'blue-warmer) :weight bold))
          ("DONE" . (:foreground ,(modus-themes-get-color-value 'green-warmer) :weight bold))
          ("WAITING" . (:foreground ,(modus-themes-get-color-value 'red-warmer) :weight bold))
          ("SOMEDAY" . (:foreground ,(modus-themes-get-color-value 'fg-dim) :weight bold))))
  (when (derived-mode-p 'org-mode)
    (font-lock-fontify-buffer)))
(with-eval-after-load 'modus-themes
  (add-hook 'modus-themes-after-load-theme-hook #'my-org-todo-set-keyword-faces))
      2024-10-14-17-21-48.svg
      Figure 1: Light background
      2024-10-14-17-22-31.svg
      Figure 2: Dark background
      This is part of my Emacs configuration.
    3. 🔗 @binaryninja@infosec.exchange Have you missed our live streams? A hurricane delayed us last week, but don't mastodon

      Have you missed our live streams? A hurricane delayed us last week, but don't panic, feature stream 4.2 Frogstar is here! 5pm ET today. 4.2 is a massive release with so many new features.

      Dynamic Shared Cache Preview, new Debugger super power, MSP430, VXWorks support, and several surprises including some of the top voted features of all time! Join us in 2 hours for all the surprises!

      https://www.youtube.com/@vector35/live

    4. 🔗 Cercle (YouTube) Black Coffee: Can you feel the groove? #blackcoffee #cercle rss

      Looking back on Black Coffee's legendary DJ set at Salle Wagram for Cercle. Six years ago and it sounds just as fresh today.

      blackcoffee #cercle #electronicmusic #afrohouse #dj #deephouse

    5. 🔗 sacha chua :: living an awesome life 2024-10-14 Emacs news rss

      Links from reddit.com/r/emacs, r/orgmode, r/spacemacs, r/planetemacs, Mastodon #emacs, Hacker News, lobste.rs, programming.dev, lemmy.world, lemmy.ml, communick.news, planet.emacslife.com, YouTube, the Emacs NEWS file, Emacs Calendar, and emacs-devel. Thanks to Andrés Ramírez for emacs-devel links. Do you have an Emacs-related link or announcement? Please e-mail me at sacha@sachachua.com. Thank you!

    6. 🔗 jank blog jank development update - Moving to LLVM IR rss

      Hi everyone! It's been a few months since the last update and I'm excited to outline what's been going on and what's upcoming for jank, the native Clojure dialect. Many thanks to Clojurists Together and my Github sponsors for the support. Let's get into it!

    7. 🔗 Baby Steps The `Overwrite` trait and `Pin` rss

      In July, boats presented a compelling vision in their post pinned places. With the Overwrite trait that I introduced in my previous post, however, I think we can get somewhere even more compelling, albeit at the cost of a tricky transition. As I will argue in this post, the Overwrite trait effectively becomes a better version of the existing Unpin trait, one that effects not only pinned references but also regular &mut references. Through this it's able to make Pin fit much more seamlessly with the rest of Rust.

      Just show me the dang code

      Before I dive into the details, let's start by reviewing a few examples to show you what we are aiming at (you can also skip to the TL;DR, in the FAQ).

      I'm assuming a few changes here:

      • Adding an Overwrite trait and changing most types to be !Overwrite by default.
        • The Option<T> (and maybe others) would opt-in to Overwrite, permitting x.take().
      • Integrating pin into the borrow checker, extending auto-ref to also "auto-pin" and produce a Pin<&mut T>. The borrow checker only permits you to pin values that you own. Once a place has been pinned, you are not permitted to move out from it anymore (unless the value is overwritten).

      The first change is "mildly" backwards incompatible. I'm not going to worry about that in this post, but I'll cover the ways I think we can make the transition in a follow up post.

      Example 1: Converting a generator into an iterator

      We would really like to add a generator syntax that lets you write an iterator more conveniently.1 For example, given some slice strings: &[String], we should be able to define a generator that iterates over the string lengths like so:

      fn do_computation() -> usize {
    let hashes = gen {
        let strings: Vec<String> = compute_input_strings();
        for string in &strings {
            yield compute_hash(&string);
        }
    };

    // ...
}

      But there is a catch here! To permit the borrow of strings, which is owned by the generator, the generator will have to be pinned.2 That means that generators cannot directly implement Iterator, because generators need a Pin<&mut Self> signature for their next methods. It is possible, however, to implement Iterator for Pin<&mut G> where G is a generator.3

      In today's Rust, that means that using a generator as an iterator would require explicit pinning:

      fn do_computation() -> usize {
    let hashes = gen {....};
    let hashes = pin!(hashes); // <-- explicit pin
    if let Some(h) = hashes.next() {
        // process first hash
    };
    // ...
}

      With pinned places, this feels more builtin, but it still requires users to actively think about pinning for even the most basic use case:

      fn do_computation() -> usize {
    let hashes = gen {....};
    let pinned mut hashes = hashes;
    if let Some(h) = hashes.next() {
        // process first hash
    };
    // ...
}

      Under this proposal, users would simply be able to ignore pinning altogether:

      fn do_computation() -> usize {
    let mut hashes = gen {....};
    if let Some(h) = hashes.next() {
        // process first hash
    };
    // ...
}

      Pinning is still happening: once a user has called next, they would not be able to move hashes after that point. If they tried to do so, the borrow checker (which now understands pinning natively) would give an error like:

      error[E0596]: cannot borrow `hashes` as mutable, as it is not declared as mutable
 --> src/lib.rs:4:22
  |
4 |     if let Some(h) = hashes.next() {
  |                      ------ value in `hashes` was pinned here
  |     ...
7 |     move_somewhere_else(hashes);
  |                         ^^^^^^ cannot move a pinned value
help: if you want to move `hashes`, consider using `Box::pin` to allocate a pinned box
  |
3 |     let mut hashes = Box::pin(gen { .... });
  |                      +++++++++            +

      As noted, it is possible to move hashes after pinning, but only if you pin it into a heap-allocated box. So we can advise users how to do that.

      Example 2: Implementing the MaybeDone future

      The pinned places post included an example future called MaybeDone. I'm going to implement that same future in the system I describe here. There are some comments in the example comparing it to the version from the pinned places post.

      enum MaybeDone<F: Future> {
    //         ---------
    //         I'm assuming we are in Rust.Next, and so the default
    //         bounds for `F` do not include `Overwrite`.
    //         In other words, `F: ?Overwrite` is the default
    //         (just as it is with every other trait besides `Sized`).

    Polling(F),
    //      -
    //      We don't need to declare `pinned F`.

    Done(Option<F::Output>),
}

impl<F: Future> MaybeDone<F> {
    fn maybe_poll(self: Pin<&mut Self>, cx: &mut Context<'_>) {
        //        --------------------
        //        I'm not bothering with the `&pinned mut self`
        //        sugar here, though certainly we could still
        //        add it.
        if let MaybeDone::Polling(fut) = self {
            //                    ---
            //       Just as in the original example,
            //       we are able to project from `Pin<&mut Self>`
            //       to a `Pin<&mut F>`.
            //
            //       The key is that we can safely project
            //       from an owner of type `Pin<&mut Self>`
            //       to its field of type `Pin<&mut F>`
            //       so long as the owner type `Self: !Overwrite`
            //       (which is the default for structs in Rust.Next).
            if let Poll::Ready(res) = fut.poll(cx) {
                *self = MaybeDone::Done(Some(res));
            }
        }
    }

    fn is_done(&self) -> bool {
        matches!(self, &MaybeDone::Done(_))
    }

    fn take_output(&mut self) -> Option<F::Output> {
        //         ---------
        //   In pinned places, this method had to be
        //   `&pinned mut self`, but under this design,
        //   it can be a regular `&mut self`.
        //   
        //   That's because `Pin<&mut Self>` becomes
        //   a subtype of `&mut Self`.
        if let MaybeDone::Done(res) = self {
            res.take()
        } else {
            None
        }
    }
}

      Example 3: Implementing the Join combinator

      Let's complete the journey by implementing a Join future:

      struct Join<F1: Future, F2: Future> {
    // These fields do not have to be declared `pinned`:
    fut1: MaybeDone<F1>,
    fut2: MaybeDone<F2>,
}

impl<F1, F2> Future for Join<F1, F2>
where
    F1: Future,
    F2: Future,
{
    type Output = (F1::Output, F2::Output);

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        //  --------------------
        // Again, I've dropped the sugar here.

        // This looks just the same as in the
        // "Pinned Places" example. This again
        // leans on the ability to project
        // from a `Pin<&mut Self>` owner so long as
        // `Self: !Overwrite` (the default for structs
        // in Rust.Next).
        self.fut1.maybe_poll(cx);
        self.fut2.maybe_poll(cx);

        if self.fut1.is_done() && self.fut2.is_done() {
            // This code looks the same as it did with pinned places,
            // but there is an important difference. `take_output`
            // is now an `&mut self` method, not a `Pin<&mut Self>`
            // method. This demonstrates that we can also get
            // a regular `&mut` reference to our fields.
            let res1 = self.fut1.take_output().unwrap();
            let res2 = self.fut2.take_output().unwrap();
            Poll::Ready((res1, res2))
        } else {
            Poll::Pending
        }
    }
}

      How I think about pin

      OK, now that I've lured you in with code examples, let me drive you away by diving into the details of Pin. I'm going to cover the way that I think about Pin. It is similar to but different from how Pin is presented in the pinned places post - in particular, I prefer to think about places that pin their values and not pinned places. In any case, Pin is surprisingly subtle, and I recommend that if you want to go deeper, you read boat's history of Pin post and/or the stdlib documentation for Pin.

      The Pin<P> type is a modifier on the pointer P

      The Pin<P> type is unusual in Rust. It looks similar to a "smart pointer" type, like Arc<T>, but it functions differently. Pin<P> is not a pointer, it is a modifier on another pointer, so

      • a Pin<&T> represents a pinned reference,
      • a Pin<&mut T> represents a pinned mutable reference,
      • a Pin<Box<T>> represents a pinned box,

      and so forth.

      You can think of a Pin<P> type as being a pointer of type P that refers to a place (Rust jargon for a location in memory that stores a value) whose valuev has been pinned. A pinned value v can never be moved to another place in memory. Moreover, v must be dropped before its place can be reassigned to another value.

      Pinning is part of the "lifecycle" of a place

      The way I think about, every place in memory has a lifecycle:

      flowchart TD
Uninitialized 
Initialized
Pinned

Uninitialized --
    p = v where v: T
--> Initialized

Initialized -- 
    move out, drop, or forget
--> Uninitialized

Initialized --
    pin value v in p
    (only possible when T is !Unpin)
--> Pinned

Pinned --
    drop value
--> Uninitialized

Pinned --
    move out or forget
--> UB

Uninitialized --
    free the place
--> Freed

UB[💥 Undefined behavior 💥]

      When first allocated, a place p is uninitialized - that is, p has no value at all.

      An uninitialized place can be freed. This corresponds to e.g. popping a stack frame or invoking free.

      p may at some point become initialized by an assignment like p = v. At that point, there are three ways to transition back to uninitialized:

      • The value v could be moved somewhere else, e.g. by moving it somewhere else, like let p2 = p. At that point, p goes back to being uninitialized.
      • The value v can be forgotten , with std::mem::forget(p). At this point, no destructor runs, but p goes back to being considered uninitialized.
      • The value v can be dropped , which occurs when the place p goes out of scope. At this point, the destructor runs, and p goes back to being considered uninitialized.

      Alternatively, the value v can be pinned in place :

      • At this point, v cannot be moved again, and the only way for p to be reused is for v to be dropped.

      Once a value is pinned, moving or forgetting the value is not allowed. These actions are "undefined behavior", and safe Rust must not permit them to occur.

      A digression on forgetting vs other ways to leak

      As most folks know, Rust does not guarantee that destructors run. If you have a value v whose destructor never runs, we say that value is leaked. There are however two ways to leak a value, and they are quite different in their impact:

      • Option A: Forgetting. Using std::mem::forget, you can forget the value v. The place p that was storing that value will go from initialized to uninitialized , at which point the place p can be freed.
        • Forgetting a value is undefined behavior if that value has been pinned, however!
      • Option B: Leak the place. When you leak a place, it just stays in the initialized or pinned state forever, so its value is never dropped. This can happen, for example, with a ref-count cycle.
        • This is safe even if the value is pinned!

      In retrospect, I wish that Option A did not exist - I wish that we had not added std::mem::forget. We did so as part of working through the impact of ref-count cycles. It seemed equivalent at the time ("the dtor doesn't run anyway, why not make it easy to do") but I think this diagram shows why it adding forget made things permanently more complicated for relatively little gain.4 Oh well! Can't win 'em all.

      Values of types implementing Unpin cannot be pinned

      There is one subtle aspect here: not all values can be pinned. If a type T implements Unpin, then values of type T cannot be pinned. When you have a pinned reference to them, they can still squirm out from under you via swap or other techniques. Another way to say the same thing is to say that values can only be pinned if their type is!Unpin ("does not implement Unpin").

      Types that are !Unpin can be called address sensitive , meaning that once they pinned, there can be pointers to the internals of that value that will be invalidated if the address changes. Types that implement Unpin would therefore be address insensitive. Traditionally, all Rust types have been address insensitive, and therefore Unpin is an auto trait, implemented by most types by default.

      Pin<&mut T> is really a "maybe pinned" reference

      Looking at the state machine as I describe it here, we can see that possessing a Pin<&mut T> isn't really a pinned mutable reference, in the sense that it doesn't always refer to a place that is pinning its value. If T: Unpin, then it's just a regular reference. But if T: !Unpin, then a pinned reference guarantees that the value it refers to is pinned in place.

      This fits with the name Unpin, which I believe was meant to convey that idea that, even if you have a pinned reference to a value of type T: Unpin, that value can become unpinned. I've heard the metaphor of "if T: Unpin, you can left out the pin, swap in a different value, and put the pin back".

      Pin picked a peck of pickled pain

      Everyone agrees that Pin is confusing and a pain to use. But what makes it such a pain?

      If you are attempting to author a Pin-based API, there are two primary problems:

      1. Pin<&mut Self> methods can't make use of regular &mut self methods.
      2. Pin<&mut Self> methods can't access fields by default. Crates like pin-project-lite make this easier but still require learning obscure concepts like structural pinning.

      If you attempting to consume a Pin-based API, the primary annoyance is that getting a pinned reference is hard. You can't just call Pin<&mut Self> methods normally, you have to remember to use Box::pin or pin! first. (We saw this in Example 1 from this post.)

      My proposal in a nutshell

      This post is focused on a proposal with two parts:

      1. Making Pin-based APIs easier to author by replacing the Unpin trait with Overwrite.
      2. Making Pin-based APIs easier to call by integrating pinning into the borrow checker.

      I'm going to walk through those in turn.

      Making Pin-based APIs easier to author

      Overwrite as the better Unpin

      The first part of my proposalis a change I call s/Unpin/Overwrite/. The idea is to introduce Overwrite and then change the "place lifecycle" to reference Overwrite instead of Unpin:

      flowchart TD
Uninitialized 
Initialized
Pinned

Uninitialized --
    p = v where v: T
--> Initialized

Initialized -- 
    move out, drop, or forget
--> Uninitialized

Initialized --
    pin value v in p
    (only possible when  
T is 👉** _!Overwrite_** 👈)
--> Pinned

Pinned --
    drop value
--> Uninitialized

Pinned --
    move out or forget
--> UB

Uninitialized --
    free the place
--> Freed

UB[💥 Undefined behavior 💥]

      For s/Unpin/Overwrite/ to work well, we have to make all !Unpin types also be !Overwrite. This is not, strictly speaking, backwards compatible, since today !Unpin types (like all types) can be overwritten and swapped. I think eventually we want every type to be !Overwrite by default, but I don't think we can change that default in a general way without an edition. But for !Unpin types in particular I suspect we can get away with it, because !Unpin types are pretty rare, and the simplification we get from doing so is pretty large. (And, as I argued in the previous post, there is no loss of expressiveness; code today that overwrites or swaps !Unpin values can be locally rewritten.)

      Why swaps are bad without s/Unpin/Overwrite/

      Today, Pin<&mut T> cannot be converted into an &mut T reference unless T: Unpin.5 This because it would allow safe Rust code to create Undefined Behavior by swapping the referent of the &mut T reference and hence moving the pinned value. By requiring that T: Unpin, the DerefMut impl is effectively limiting itself to references that are not, in fact, in the "pinned" state, but just in the "initialized" state.

      As a result, Pin<&mut T> and &mut T methods don't interoperate today

      This leads directly to our first two pain points. To start, from a Pin<&mut Self> method, you can only invoke &self methods (via the Deref impl) or other Pin<&mut Self> methods. This schism separates out the "regular" methods of a type from its pinned methods; it also means that methods doing field assignments don't compile:

      fn increment_field(self: Pin<&mut Self>) {
    self.field = self.field + 1;
}

      This errors because compiling a field assignment requires a DerefMut impl and Pin<&mut Self> doesn't have one.

      With s/Unpin/Overwrite/, Pin<&mut Self> is a subtype of &mut self

      s/Unpin/Overwrite/ allows us to implement DerefMut for all pinned types. This is because, unlike Unpin, Overwrite affects how &mut works, and hence &mut T would preserve the pinned state for the place it references. Consider the two possibilities for the value of type T referred to by the &mut T:

      • If T: Overwrite, then the value is not pinnable, and so the place cannot be in the pinned state.
      • If T: !Overwrite, the value could be pinned, but we also cannot overwrite or swap it, and so pinning is preserved.

      This implies that Pin<&mut T> is in fact a generalized version of &mut T. Every &'a mut T keeps the value pinned for the duration of its lifetime 'a, but a Pin<&mut T> ensures the value stays pinned for the lifetime of the underlying storage.

      If we have a DerefMut impl, then Pin<&mut Self> methods can freely call &mut self methods. Big win!

      Today you must categorize fields as "structurally pinned" or not

      The other pain point today with Pin is that we have no native support for "pin projection"6. That is, you cannot safely go from a Pin<&mut Self> reference to a Pin<&mut F> method that referring to some field self.f without relying on unsafe code.

      The most common practice today is to use a custom crate like pin-project- lite. Even then, you also have to make a choice for each field between whether you want to be able to get a Pin<&mut F> reference or a normal &mut F reference. Fields for which you can get a pinned reference are called structurally pinned and the criteria for which one you should use is rather subtle. Ultimately this choice is required because Pin<&mut F> and &mut F don't play nicely together.

      Pin projection is safe from any !Overwrite type

      With s/Unpin/Overwrite/, we can scrap the idea of structural pinning. Instead, if we have a field owner self: Pin<&mut Self>, pinned projection is allowed so long as Self: !Overwrite. That is, if Self: !Overwrite, then I can always get a Pin<&mut F> reference to some field self.f of type F. How is that possible?

      Actually, the full explanation relies on borrow checker extensions I haven't introduced yet. But let's see how far we get without them, so that we can see the gap that the borrow checker has to close.

      Assume we are creating a Pin<&'a mut F> reference r to some field self.f, where self: Pin<&mut Self>:

      • We are creating a Pin<&'a mut F> reference to the value in self.f:
        • If F: Overwrite, then the value is not pinnable, so this is equivalent to an ordinary &mut F and we have nothing to prove.
        • Else, if F: !Overwrite, then we have to show that the value in self.f will not move for the remainder of its lifetime.
        • Pin projection from `*selfis only valid ifSelf: !Overwriteandself: Pin<&'b mut Self>, so we know that the value in *self is pinned for the remainder of its lifetime by induction.
        • We have to show then that the value v_f in self.f will never be moved until the end of its lifetime.

      There are three ways to move a value out of self.f:

      • You can assign a new value to self.f, like self.f = ....
        • This will run the destructor, ending the lifetime of the value v_f.
      • You can create a mutable reference r = &mut self.f and then…
        • assign a new value to *r: but that will be an error because F: !Overwrite.
        • swap the value in *r with another: but that will be an error because F: !Overwrite.

      QED. =)

      Making Pin-based APIs easier to call

      Today, getting a Pin<&mut> requires using the pin! macro, going through Box::pin, or some similar explicit action. This adds "syntactic salt" to calling a Pin<&mut Self> some other abstraction rooted in unsafe (e.g., Box::pin). There is no built-in way to safely create a pinned reference. This is fine but introduces ergonomic hurdles

      We want to make calling a Pin<&mut Self> method as easy as calling an &mut self method. To do this, we need to extra the compiler's notion of "auto-ref" to include the option of "auto-pin-ref":

      // Instead of this:
let future: Pin<&mut impl Future> = pin!(async { ... });
future.poll(cx);

// We would do this:
let mut future: impl Future = async { ... };
future.poll(cx); // <-- Wowee!

      Just as a typical method call like vec.len() expands to Vec::len(&vec), the compiler would be expanding future.poll(cx) to something like so:

      Future::poll(&pinned mut future, cx)
//           ^^^^^^^^^^^ but what, what's this?

      This expansion though includes a new piece of syntax that doesn't exist today, the &pinned mut operation. (I'm lifting this syntax from boats' pinned places proposal.)

      Whereas &mut var results in an &mut T reference (assuming var: T), &pinned mut var borrow would result in a Pin<&mut T>. It would also make the borrow checker consider the value in future to be pinned. That means that it is illegal to move out from var. The pinned state continues indefinitely until var goes out of scope or is overwritten by an assignment like var = ... (which drops the heretofore pinned value). This is a fairly straightforward extension to the borrow checker's existing logic.

      New syntax not strictly required

      It's worth noting that we don't actually need the &pinned mut syntax (which means we don't need the pinned keyword). We could make it so that the only way to get the compiler to do a pinned borrow is via auto-ref. We could even add a silly trait to make it explicit, like so:

      trait Pinned {
    fn pinned(self: Pin<&mut Self>) -> Pin<&mut Self>;
}

impl<T: ?Sized> Pinned for T {
    fn pinned(self: Pin<&mut T>) -> Pin<&mut T> {
        self
    }
}

      Now you can write var.pinned(), which the compiler would desugar to Pinned::pinned(&rustc#pinned mut var). Here I am using rustc#pinned to denote an "internal keyword" that users can't type.7

      Frequently asked questions

      So…there's a lot here. What's the key takeaways?

      The shortest version of this post I can manage is8

      • Pinning fits smoothly into Rust if we make two changes:
        • Limit the ability to swap types by default, making Pin<&mut T> a subtype of &mut T and enabling uniform pin projection.
        • Integrate pinning in the auto-ref rules and the borrow checker.

      Why do you only mention swaps? Doesn't Overwrite affect other things?

      Indeed the Overwrite trait as I defined it is overkill for pinning. The more precise, we might imagine two special traits that affect how and when we can drop or move values:

      trait DropWhileBorrowed: Sized { }
trait Swap: DropWhileBorrowed { }
      • Given a reference r: &mut T, overwriting its referent *r with a new value would require T: DropWhileBorrowed;
      • Swapping two values of type T requires that T: Swap.
        • This is true regardless of whether they are borrowed or not.

      Today, every type is Swap. What I argued in the previous post is that we should make the default be that user-defined types implement neither of these two traits (over an edition, etc etc). Instead, you could opt-in to both of them at once by implementing Overwrite.

      But we could get all the pin benefits by making a weaker change. Instead of having types opt out from both traits by default, they could only opt out of Swap, but continue to implement DropWhileBorrowed. This is enough to make pinning work smoothly. To see why, recall the pinning state diagram: dropping the value in *r (permitted by DropWhileBorrowed) will exit the "pinned" state and return to the "uninitialized" state. This is valid. Swapping, in contrast, is UB.

      Two subtle observations here worth calling out:

      1. Both DropWhileBorrowed and Swap have Sized as a supertrait. Today in Rust you can't drop a &mut dyn SomeTrait value and replace it with another, for example. I think it's a bit unclear whether unsafe could do this if it knows the dynamic type of value behind the dyn. But under this model, it would only be valid for unsafe code do that drop if (a) it knew the dynamic type and (b) the dynamic type implemented DropWhileBorrowed. Same applies to Swap.
      2. The Swap trait applies longer than just the duration of a borrow. This is because, once you pin a value to create a Pin<&mut T> reference, the state of being pinned persists even after that reference has ended. I say a bit more about this in another FAQ below.

      EDIT: An earlier draft of this post named the trait Swap. This was wrong, as described in the FAQ on subtle reasoning.

      Why then did you propose opting out from both overwrites and swaps?

      Opting out of overwrites (i.e., making the default be neither DropWhileBorrowed nor Swap) gives us the additional benefit of truly immutable fields. This will make cross-function borrows less of an issue, as I described in my previous post, and make some other things (e.g., variance) less relevant. Moreover, I don't think overwriting an entire reference like *r is that common, versus accessing individual fields. And in the cases where people do do it, it is easy to make a dummy struct with a single field, and then overwrite r.value instead of *r. To me, therefore, distinguishing between DropWhileBorrowed and Swap doesn't obviously carry its weight.

      Can you come up with a more semantic name for Overwrite?

      All the trait names I've given so far (Overwrite, DropWhileBorrowed, Swap) answer the question of "what operation does this trait allow". That's pretty common for traits (e.g., Clone or, for that matter, Unpin) but it is sometimes useful to think instead about "what kinds of types should implement this trait" (or not implement it, as the case may be).

      My current favorite "semantic style name" is Mobile, which corresponds to implementing Swap. A mobile type is one that, while borrowed, can move to a new place. This name doesn't convey that it's also ok to drop the value, but that follows, since if you can swap the value to a new place, you can presumably drop that new place.

      I don't have a "semantic" name for DropWhileBorrowed. As I said, I'm hard pressed to characterize the type that would want to implement DropWhileBorrowed but not Swap.

      What do DropWhileBorrowed and Swap have in common?

      These traits pertain to whether an owner who lends out a local variable (i.e., executes r = &mut lv) can rely on that local variable lv to store the same value after the borrow completes. Under this model, the answer depends on the type T of the local variable:

      • If T: DropWhileBorrowed (or T: Swap, which implies DropWhileBorrowed), the answer is "no", the local variable may point at some other value, because it is possible to do *r = /* new value */.
      • But if T: !DropWhileBorrowed, then the owner can be sure that lv still stores the same value (though lv's fields may have changed).

      Let's use an analogy. Suppose I own a house and I lease it out to someone else to use. I expect that they will make changes on the inside, such as hanging up a new picture. But I don't expect them to tear down the house and build a new one on the same lot. I also don't expect them to drive up a flatbed truck, load my house onto it, and move it somewhere else (while proving me with a new one in return). In Rust today, a reference r: &mut T reference allows all of these things:

      • Mutating a field like r.count += 1 corresponds to hanging up a picture. The values inside r change, but r still refers to the same conceptual value.
      • Overwriting *r = t with a new value t is like tearing down the house and building a new one. The original value that was in r no longer exists.
      • Swapping *r with some other reference *r2 is like moving my house somewhere else and putting a new house in its place.

      EDIT: Wording refined based on feedback.

      What does it mean to be the "same value"?

      One question I received was what it meant for two structs to have the "same value"? Imagine a struct with all public fields - can we make any sense of it having an identity? The way I think of it, every struct has a "ghost" private field $identity (one that doesn't exist at runtime) that contains its identity. Every StructName { } expression has an implicit $identity: new_value() that assigns the identity a distinct value from every other struct that has been created thus far. If two struct values have the same $identity, then they are the same value.

      Admittedly, if a struct has all public fields, then it doesn't really matter whether it's identity is the same, except perhaps to philosophers. But most structs don't.

      An example that can help clarify this is what I call the "scope pattern". Imagine I have a Scope type that has some private fields and which can be "installed" in some way and later "deinstalled" (perhaps it modifies thread- local values):

      pub struct Scope {...}

impl Scope {
    fn new() -> Self { /* install scope */ }
}

impl Drop for Scope {
    fn drop(&mut self) {
        /* deinstall scope */
    }
}

      And the only way for users to get their hands on a "scope" is to use with_scope, which ensures it is installed and deinstalled properly:

      pub fn with_scope(op: impl FnOnce(&mut Scope)) {
    let mut scope = Scope::new();
    op(&mut scope);
}

      It may appear that this code enforces a "stack discipline", where nested scopes will be installed and deinstalled in a stack-like fashion. But in fact, thanks to std::mem::swap, this is not guaranteed:

      with_scope(|s1| {
    with_scope(|s2| {
        std::mem::swap(s1, s2);
    })
})

      This could easily cause logic bugs or, in unsafe is involved, something worse. This is why lending out scopes requires some extra step to be safe, such as using a &-reference or adding a "fresh" lifetime paramteer of some kind to ensure that each scope has a unique type. In principle you could also use a type like &mut dyn ScopeTrait, because the compiler disallows overwriting or swapping dyn Trait values: but I think it's ambiguous today whether unsafe code could validly do such a swap.

      EDIT: Question added based on feedback.

      There's a lot of subtle reasoning in this post. Are you sure this is

      correct?

      I am pretty sure! But not 100%. I'm definitely scared that people will point out some obvious flaw in my reasoning. But of course, if there's a flaw I want to know. To help people analyze, let me recap the two subtle arguments that I made in this post and recap the reasoning.

      Lemma. Given some local variable lv: T where T: !Overwrite mutably borrowed by a reference r: &'a mut T, the value in lv cannot be dropped, moved, or forgotten for the lifetime 'a.

      During 'a, the variable lv cannot be accessed directly (per the borrow checker's usual rules). Therefore, any drops/moves/forgets must take place to *r:

      • Because T: !Overwrite, it is not possible to overwrite or swap *r with a new value; it is only legal to mutate individual fields. Therefore the value cannot be dropped or moved.
      • Forgetting a value (via std::mem::forget) requires ownership and is not accesible while lv is borrowed.

      Theorem A. If we replace T: Unpin and T: Overwrite, then Pin<&mut T> is a safe subtype of &mut T.

      The argument proceeds by cases:

      • If T: Overwrite, then Pin<&mut T> does not refer to a pinned value, and hence it is semantically equivalent to &mut T.
      • If T: !Overwrite, then Pin<&mut T> does refer to a pinned value, so we must show that the pinning guarantee cannot be disturbed by the &mut T. By our lemma, the &mut T cannot move or forget the pinned value, which is the only way to disturb the pinning guarantee.

      Theorem B. Given some field owner o: O where O: !Overwrite with a field f: F, it is safe to pin-project from Pin<&mut O> to a Pin<&mut F> reference referring to o.f.

      The argument proceeds by cases:

      • If F: Overwrite, then Pin<&mut F> is equivalent to &mut F. We showed in Theorem A that Pin<&mut O> could be upcast to &mut O and it is possible to create an &mut F from &mut O, so this must be safe.
      • If F: !Overwrite, then Pin<&mut F> refers to a pinned value found in o.f. The lemma tells us that the value in o.f will not be disturbed for the duration of the borrow.

      EDIT: It was pointed out to me that this last theorem isn't quite proving what it needs to prove. It shows that o.f will not be disturbed for the duration of the borrow, but to meet the pin rules, we need to ensure that the value is not swapped even after the borrow ends. We can do this by committing to never permit swaps of values unless T: Overwrite, regardless of whether they are borrowed. I meant to clarify this in the post but forgot about it, and then I made a mistake and talked about Swap - but Swap is the right name.

      What part of this post are you most proud of?

      Geez, I'm so glad you asked! Such a thoughtful question. To be honest, the part of this post that I am happiest with is the state diagram for places, which I've found very useful in helping me to understand Pin:

      flowchart TD
Uninitialized 
Initialized
Pinned

Uninitialized --
    `p = v` where `v: T`
--> Initialized

Initialized -- 
    move out, drop, or forget
--> Uninitialized

Initialized --
    pin value `v` in `p`
    (only possible when `T` is `!Unpin`)
--> Pinned

Pinned --
    drop value
--> Uninitialized

Pinned --
    move out or forget
--> UB

Uninitialized --
    free the place
--> Freed

UB[💥 Undefined behavior 💥]

      Obviously this question was just an excuse to reproduce it again. Some of the key insights that it helped me to crystallize:

      • A value that is Unpin cannot be pinned:
        • And hence Pin<&mut Self> really means "reference to a maybe-pinned value" (a value that is pinned if it can be).
      • Forgetting a value is very different from leaking the place that value is stored:
        • In both cases, the value's Drop never runs, but only one of them can lead to a "freed place".

      In thinking through the stuff I wrote in this post, I've found it very useful to go back to this diagram and trace through it with my finger.

      Is this backwards compatible?

      Maybe? The question does not have a simple answer. I will address in a future blog post in this series. Let me say a few points here though:

      First, the s/Unpin/Overwrite/ proposal is not backwards compatible as I described. It would mean for example that all futures returned by async fn are no longer Overwrite. It is quite possible we simply can't get away with it.

      That's not fatal, but it makes things more annoying. It would mean there exist types that are !Unpin but which can be overwritten. This in turn means that Pin<&mut Self> is not a subtype of &mut Self for all types. Pinned mutable references would be a subtype for almost all types, but not those that are !Unpin && Overwrite.

      Second, a naive, conservative transition would definitely be rough. My current thinking is that, in older editions, we add T: Overwrite bounds by default on type parameters T and, when you have a T: SomeTrait bound, we would expand that to include a Overwrite bound on associated types in SomeTrait, like T: SomeTrait<AssocType: Overwrite>. When you move to a newer edition I think we would just not add those bounds. This is kind of a mess, though, because if you call code from an older edition, you are still going to need those bounds to be present.

      That all sounds painful enough that I think we might have to do something smarter, where we don't always add Overwrite bounds, but instead use some kind of inference in older editions to avoid it most of the time.

      Conclusion

      My takeaway from authoring this post is that something like Overwrite has the potential to turn Pin from wizard level Rust into mere "advanced Rust", somewhat akin to knowing the borrow checker really well. If we had no backwards compatibility constraints to work with, it seems clear that this would be a better design than Unpin as it is today.

      Of course, we do have backwards compatibility constraints, so the real question is how we can make the transition. I don't know the answer yet! I'm planning on thinking more deeply about it (and talking to folks) once this post is out. My hope was first to make the case for the value of Overwrite (and to be sure my reasoning is sound) before I invest too much into thinking how we can make the transition.

      Assuming we can make the transition, I'm wondering two things. First, is Overwrite the right name? Second, should we take the time to re-evaluate the default bounds on generic types in a more complete way? For example, to truly have a nice async story, and for myraid other reasons, I think we need must move types. How does that fit in?

      1. The precise design of generators is of course an ongoing topic of some controversy. I am not trying to flesh out a true design here or take a position. Mostly I want to show that we can create ergonomic bridges between "must pin" types like generators and "non pin" interfaces like Iterator in an ergonomic way without explicit mentioning of pinning. ↩︎

      2. Boats has argued that, since no existing iterator can support borrows over a yield point, generators might not need to do so either. I don't agree. I think supporting borrows over yield points is necessary for ergonomics just as it was in futures. ↩︎

      3. Actually for Pin<impl DerefMut<Target: Generator>>. ↩︎

      4. I will say, I use std::mem::forget quite regularly, but mostly to make up for a shortcoming in Drop. I would like it if Drop had a separate method, fn drop_on_unwind(&mut self), and we invoked that method when unwinding. Most of the time, it would be the same as regular drop, but in some cases it's useful to have cleanup logic that only runs in the case of unwinding. ↩︎

      5. In contrast, a Pin<&mut T> reference can be safely converted into an &T reference, as evidenced by Pin's Deref impl. This is because, even if T: !Unpin, a &T reference cannot do anything that is invalid for a pinned value. You can't swap the underlying value or read from it. ↩︎

      6. Projection is the wonky PL term for "accessing a field". It's never made much sense to me, but I don't have a better term to use, so I'm sticking with it. ↩︎

      7. We have a syntax k#foo for explicitly referred to a keyword foo. It is meant to be used only for keywords that will be added in future Rust editions. However, I sometimes think it'd be neat to internal-ish keywords (like k#pinned) that are used in desugaring but rarely need to be typed explicitly; you would still be able to write k#pinned if for whatever reason you wanted to. And of course we could later opt to stabilize it as pinned (no prefix required) in a future edition. ↩︎

      8. I tried asking ChatGPT to summarize the post but, when I pasted in my post, it replied, "The message you submitted was too long, please reload the conversation and submit something shorter." Dang ChatGPT, that's rude! Gemini at least gave it the old college try. Score one for Google. Plus, it called my post "thought-provoking!" Aww, I'm blushing! ↩︎

    8. 🔗 matklad A Missing IDE Feature rss

      A Missing IDE Feature Oct 14, 2024

      Slightly unusual genre -- with this article, I want to try to enact a change in the world. I believe that there is a "missing" IDE feature which is:

      • very easy to implement (these days),
      • is a large force multiplier for experienced users,
      • is conspicuously missing from almost every editor.

      The target audience here is anyone who can land a PR in Zed, VS Code, Helix, Neovim, Emacs, Kakoune, or any other editor or any language server. The blog post would be a success if one of you feels sufficiently inspired to do the thing!

      [The Feature ](https://matklad.github.io/2024/10/14/missing-ide-

      feature.html#The-Feature)

      Suppose you are casually reading the source code of rust-analyzer, and are curious about handling of method bodies. There's a Body struct in the code base, and you want to understand how it is used.

      Would you rather look at this?

      Or this?

      (The screenshots are from IntelliJ/RustRover, because of course it gets this right)

      The second option is clearly superior -- it conveys significantly more useful information in the same amount of pixels. Function names, argument lists and return types are so much more valuable than a body of any particular function. Especially if the function is a page-full of boilerplate code!

      And this is the feature I am asking for -- make the code look like the second image. Or, specifically, Fold Method Bodies by Default.

      There are two components here. First , only method bodies are folded. This is a syntactic check -- we are not folding the second level. For code like

      fn f() { ... }

impl S {
    fn g(&self) { ... }
}

      Both f and g are folded, but impl S is not. Similarly, function parameters and function body are actually on the same level of folding hierarchy, but it is imperative that parameters are not folded. This is the part that was hard ten years ago but is easy today. "what is function body" is a non-trivial question, which requires proper parsing of the code. These days, either an LSP server or Tree-sitter can answer this question quickly and reliably.

      The second component of the feature is that folded is a default state. It is not a "fold method bodies" action. It is a setting that ensures that, whenever you visit a new file, bodies are folded by default. To make this work, the editor should be smart to seamlessly unfold specific function when appropriate. For example, if you "go to definition" to a function, that function should get unfolded, while the surrounding code should remail folded.

      Now that I have explained how the feature works, I will not try to motivate it. I think it is pretty obvious how awesome this actually is. Code is read more often than written, and this is one of the best multipliers for readability. Most of the code is in method bodies, but most important code is in function signatures. Folding bodies auto-magically hide the 80% of boring code, leaving the most important 20%. It was in 2018 when I last the IDE (IntelliJ) which has this implemented properly, and I've been missing this function ever since!

      You might also be wondering whether it is the same feature as the Outline, that special UI which shows a graphical, hierarchical table of contents of the file. It is true that outline and fold-bodies-by-default attack the same issue. But I'd argue that folding solves it better. This is an instance of a common pattern. In a smart editor, it is often possible to implement any given feature either by "lowering" it to plain text, or by creating a dedicated GUI. And the lowering approach almost always wins, because it gets to re-use all existing functionality for free. For example, the folding approach trivially gives you an ability to move a bunch of functions from one impl block to the other by selecting them with Shift `+` Down, cutting with Ctrl `+` X and pasting with Ctrl `+` V.

      [Call to Action ](https://matklad.github.io/2024/10/14/missing-ide-

      feature.html#Call-to-Action)

      So, if you are a committer to one of the editors, please consider adding a "fold function bodies by default" mode. It probably should be off by default, as it can easily scare new users away, but it should be there for power users to enable, and it should be prominently documented, so that people can learn that they want it. After the checkbox is in place, see if you can implement the actual logic! If your editor uses Tree-sitter, this should be relatively easy -- its syntax tree contains all the information you need. Just make sure that:

      • bodies are folded when the new file is opened,
      • the editor unfolds them when appropriate (generally, when navigated to a function from elsewhere).

      If your editor is not based on Tree-sitter, you'll have a harder time. In theory, the information should be readily available from the language server, but LSP currently doesn't expose it. Here's the problem:

      https://microsoft.github.io/language-server- protocol/specifications/lsp/3.17/specification/#foldingRangeKind

      There's no body kind there! Adding it should be trivially technically, but it always is a pain to get something into the protocol if you are not VS Code.

      [My Role ](https://matklad.github.io/2024/10/14/missing-ide-

      feature.html#My-Role)

      What is my job here, besides sitting there and writing blog posts? I actually think that writing this down is quite valuable!

      I suppose the feature is still commonly missing due to a two-sided market failure -- the feature doesn't exist, so prospective users don't realize that it is possible, and don't ask editor's authors to implement it. Without users asking, editor authors themselves don't realize this feature could exist, and don't rush implementing it. This is exacerbated by the fact that it was a hard feature to implement ten years ago, when we didn't have Tree-sitter/LSP, so there are poor workarounds in place -- actions to fold a certain level. These workarounds the prevent the proper feature from gaining momentum.

      So here I hope to maybe tip the equilibrium's scale a bit, and start a feedback loop where more people realize that they want this feature, such that it is implemented in some of the more experimental editors, which hopefully would expose the feature to more users, popularizing it until it gets implemented everywhere!

      Still, just talking isn 't everything I did here! Six years ago, I implemented the language-server side of this in rust-analyzer:

      https://github.com/rust-lang/rust- analyzer/commit/23b040962ff299feeef1f967bc2d5ba92b01c2bc

      This currently isn 't exposed to LSP, because it doesn't allow flagging a folding range as method body. To fix that I opened this VS Code issue (LSP generally avoids doing something before VS Code):

      https://github.com/microsoft/vscode/issues/128912

      And since then I have been quietly waiting for some editor (not necessary VS Code) to pick this up. This hasn't happened yet, hence this article!

      Thanks for reading!

  3. October 13, 2024
    1. 🔗 Mr. Money Mustache Six Dumb Misconceptions About The Economy (that the Politicians Want You To Believe) rss

      -

      Well, it looks like we’re here in another US election year already.

      As Advanced Mustachians, we already know that the ongoing battle of Harris vs. Trump should not be consuming much of our time. Sure, we do our research and cast our votes but after that we move right on to focus on other things within our own circle of control.

      But out of all the things the politicians like to bicker about, there’s one area where MMM does need to set the record straight, and that area is of course money. Your money, the economy in general, and the overall wealth of the nation.

      Politicians are already not known for being the sharpest tools in the shed when it comes to technical stuff like science, technology, or economics. But this year the discourse has become particularly dumb, as our candidates try to manipulate undecided voters in swing states with ideas that are based on irrational emotions rather than sound economic sense.

      For one particularly funny example, you may have noticed that the competing party (Trump in this case) is attacking the incumbents (Biden/Harris) over the “bad economy.” When in fact the US economy is stronger than it has ever been , with the lowest unemployment we’ve ever seen as well.

      It’s hard to imagine a better situation than we have right now, and in fact the recent bout of higher inflation is a sign that things have been going too well, and we needed to step on the brakes with the help of higher interest rates.

      But somehow the people still seem to believe that we have a “bad” economy. Take a look at this Gallup poll showing that while most people (85%) are doing really well right now, they assume that it’s just their own good fortune - only 17% believe the economy is doing well.

      This is mathematically impossible, because if most people are doing well, that’s the definition of a good economy! And suspiciously enough, this widespread wrongness correlates quite nicely with the rise of social media misinformation.

      -

      So the politicians and the news have been doing the opposite of what they should be doing in an ideal situation (sharing accurate information). And sure, we can always just ignore their speeches and go on with our lives. But when it comes to economics, knowledge is power (and money). The more accurately we understand how things really work, the wealthier we will all become.

      So with all that in mind, I hereby present you with my list of the…

      Top Dumb Things Politicians Want You To Believe About The Economy

      1: The President Controls the Economy

      If there’s a recession, the opposition party likes to blame it on the current president. If the economy is booming, the current president likes to give himself ( or possibly soon herself) credit for all of that success. But really, the US economy is way too big - and thankfully way too _free - _for the president to control or really even influence all that strongly.

      In reality, our economy is a gigantic machine which converts labor and materials into things like iPhones, hospitals and pumpkin pies. And although we’re the biggest economy at 26% of the planet, we are still heavily influenced by that much bigger 74% of economic activity that the other 7.6 billion people on Earth are busy producing everywhere else.

      When we have our inevitable little boom and bust cycles, they are mostly caused by the normal cycle of irrational exuberance (and greed) like the 2007 housing boom, followed by brief periods of extreme fear and pessimism like the 2008-2012 financial and housing crash.

      The government does play a role too, by setting tax rates and other rules. But the effects of these policies are usually so delayed and unpredictable, that you can’t draw a straight line between today’s president and today’s economy. In other words, the government does its best to adjust the rudder on our giant ship, but in the short term our economy lurches around on the waves and storms of the ocean.

      2: The President Controls Interest Rates

      This one is especially funny to me, as our candidates feign sympathy for the hard life of middle class Americans, who now face higher borrowing costs on their credit cards and car loans and mortgages. They claim they will fight to bring the interest rates down. Trump even goes as far as bullying our Federal Reserve board members (who can only do their jobs if we allow them to function as independent experts) and suggesting that he would take over the whole department, if elected.

      The real story is that while monetary policy would be a terrible tool to leave in the hands of a sitting president (see Argentina), it does function as an excellent set of gas and brake pedals for the economy if used properly. When things slow down and unemployment gets too high, a cut to the interest rates will produce a boost in everything from new jobs to stock prices. But if things get too hot, you get rapid inflation which can mess up the system.

      3: Inflation has Made Life Harder for Americans (and the President Can Magically Reverse it)

      This line of reasoning is even dumber than the last one. For a couple of years after the Covid era, we had rapid inflation. It was caused by a rare combination of a goods shortage caused by things like factory closures and remote work, plentiful demand from government stimulus spending and low interest rates. These factors have since ironed themselves out, and inflation is back down to an ultra-low 2.4%.

      Steve Ballmer explains the inflation vs wages debate in his useful new video series called USA Facts (see note below)

      But most significantly, wages have still risen faster than inflation so we are all better off than before! Since 2019, overall prices are up 19% and our wages are up 21%. So even after all that inflation, we are still doing just fine. But the candidates are still bickering over inflation as if it’s an actual problem, and even worse promising to “bring prices back down”. And they've managed to convince the electorate that "higher wages and prices" is the same thing as "a bad economy". Which is just plain wrong.

      Bonus dumbness: politicians also occasionally blame “greedy corporations” for increasing prices to hoard profits. While price increases are totally acceptable in a market system (as a business owner you are free to set prices wherever you like), in reality it doesn’t usually happen because our markets are too competitive. For example, a recent deep analysis from NPR showed that no, grocery stores haven’t made any windfall profit at all off of this recent bout of Covid-fueled inflation.

      4: The President Controls Housing Prices

      One important thing that has changed over the past ten years is that US house prices and rents have both risen much faster than general inflation and even wages. On the positive side, interest rates have also risen which tends to make houses feel more expensive and is supposed to help bring house prices down. But it hasn’t happened yet which means we have the double whammy of higher prices and higher interest costs for mortgage borrowers.

      The dumb part is that our candidates are proposing things that would make the problem even worse, like subsidies for first-time homebuyers or schemes to reduce the interest rates. When really the solution is to increase the supply of housing , which I personally think will happen if we stop putting up roadblocks for homebuilders (myself included) to build housing.

      Things like faster and cheaper permits, less onerous and expensive building codes, eliminating suburban-style zoning and setback and car parking rules, and changing laws so that NIMBYs no longer get any say over what other people do with their own land could all help reduce the cost of building a house by about 50%, quickly and permanently.

      5: The President Controls Gas Prices, and They Are Currently “High” and We Want Them Lower

      Ahh, gasoline! The most ridiculous of things to worry about and the fuel for many of MMM’s rants since 2011.

      First of all, on an inflation-adjusted basis, gasoline is still about the same price as it was in 1950: in the $3-4 range per gallon, in today’s dollars.

      Secondly, it is so cheap that even with our huge inefficient American vehicles, the average household is still only spending 2.5% of their disposable income on the stuff! (The funny part is that they spend many times more on the rest of the car ownership experience while thinking gas is the part that is expensive)

      Third, gasoline has been obsolete for almost a decade now. You can get a used electric car for less than the price of a comparable used gas car, or if you’re a fancypants money waster like me, new EVs are also cheaper than their gas counterparts. You get a faster, nicer car that almost never needs maintenance OR gasoline, and save money.

      So why are we even still talking about this antique fuel of a previous era? Why aren’t the candidates also arguing over the price of Kodak film or typewriters or fax machines?

      6: The Economy is Something We Should Even Worry About

      The funniest part about all this economic talk is that we’re focusing on the wrong thing. While hard work and business and advancing the frontiers of human knowledge are all fun things, the reality is that we passed the point of having “Enough” decades ago. When the American middle class complains about how hard we have it these days, it’s like a bunch of overfed people at a buffet wishing they could just have one more flavor of donuts stacked onto the table.

      Yes, we have income and wealth inequality so that the rich tend to get richer more quickly. And yes, we should keep that in check with a somewhat progressive tax system because a more equal society tends to be a more peaceful and happy one.

      But have you noticed that as the rich people get richer, they don’t get any happier? It’s because after you pass the point of “Enough”, adding more money doesn’t really help much.

      And “Enough” is much more defined by your mindset (and your collection of life skills) than your paycheck. So if the politicians really cared about improving our happiness and wellbeing, they’d be preaching the Principles of Mustachianism rather than pandering to the specific requests of coal miners or billionaires.

      But alas, winning an election is a very different thing than proposing stuff that is actually best for the country. And for that reason, we cast our votes for the best party and then tune back out until the next election.

      Happy voting!

      In the Comments: Has the election season been getting you down, pumping you up, or just giving you a thorough dose of “Meh”?

      Further Reading/Watching:

      While researching economic stats for this article, I came across a quirky but informative series of videos called USA Facts by none other than Microsoft co-founder Steve Ballmer. It seems that he had the same frustration as me: Americans are fighting over a bunch of opinions and misinformation without even bothering to look up the actual facts. So he made a well-produced series of videos that just share the facts without the baggage of political hype on top of them. I wish our politicians could do the same thing!

      Bonus Podcast based on this article!
      Thanks to the magic of AI, you can direct the wizardry within Google to generate a custom-made podcast on almost anything on the Internet. A reader just emailed me this take on this episode - remarkably human-like and even entertaining!
      https://notebooklm.google.com/notebook/0e1d0af8-8888-466c-abe4-8b1da8986773/audio

    2. 🔗 oxigraph/oxigraph v0.4.1 release
      • New: HTTP server: return SPARQL service description on GET requests.
      • JS: add default_graph and named_graphs query options to set the default and named graphs.
      • SPARQL: fixes REGEX function evaluation when the regex is not a literal constant
      • SPARQL: small fixes related to SERVICE optimizations
      • Parsers: minor fixes for bugs found when fuzzing the parsers
    3. 🔗 Register Spill Joy & Curiosity #10 rss

      Last week we took the kids to a small amusement park. We got there early and when we walked in, employees of the park were still getting the park ready for the day: sweeping, feeding animals, raking leaves, putting new bags in trash cans. And I kept thinking: "this is exactly like in Rollercoaster Tycoon!"

      So this week, let me remind you: "Sawyer wrote 99% of the code for RollerCoaster Tycoon in x86 assembly language for the Microsoft Macro Assembler, with the remaining one percent written in C."

      • Patrick and Mariano are organizing a meetup in Munich called Hombrew Systems Club and it sounds wonderful. I'm going to give a short talk on writing Tucan, the optimizing compiler in Rust that I hacked on for a few years. Filling 10 minutes won't be an issue, but I'm really worried about creating slides that match the "~~Vibes~~"

      • Derek Sivers was on the Andrew Wilkinson podcast. Derek's books have meant a lot to me over the years and listening to him talk like this for an hour created some very enjoyable moments of reflection. Super interesting bit from the episode: the Derek Sivers as I know him -- author of books, TED talker, blogger, /now-page haver -- only started to exist when he was 38.

      • Where did the \n in the Rust compiler come from? Mind-bending, to-the-point, fun blog post. "perfect blog post" indeed.

      • As a response to my post here, Glad I did it in Go, a reader sent me an email to share a document he authored with me: Argument against Python in Education. I don't enjoy a lot of discussions around language design because most of them are very abstract and very academic -- this document though, it's refreshing and interesting because it's grounded in practice, in reality. Really enjoyed that, thanks for sharing Patricio!

      • And as a response to my post this week, Use data that looks like data, Sam shared this blog post: Greppability is an underrated code metric. Again: this is as real, as rubber-hits-the-road, as practical as it gets. I love it. I've used greppability as a metric in the past and it makes me happy that I now have something to link to.

      • I've thought a lot about the PATH environment variable in the past few months (due to fixing or improving a lot of environment-handling things in Zed -- I wrote a long doc about it here) and then came across this blog post this week: Modern PATH environment variable. "as of the year 2024 that's it", they write, and present a really short PATH -- I'm telling you: if everyone would have a well-managed PATH like this, 50% of the world's bugs would disappear (not really, but still.)

      • Deno 2 was announced this week and, again, it's all about practice and tooling. I guess that's a given, since Deno is a runtime and tooling and not a new language, but it still made me think back to what a friend said last year: "Ruby missed the boat because Matz thinks developer happiness is purely about the language and they missed that it's more and more about the tooling."

      • A Haskell programmer wrote about their "negative views on Rust". I agree with some of it (on the Good and the Bad side), but what stuck with me was use of "tamagotchi": "This aspect of Rust puts me off. I don't need another tamagotchi."

      • This whole discussion around the Deppenapostroph in German was _fascinating -- _so many flashbacks to early 2000s internet and people being annoying and annoyed by wrong punctuation on forums and IRC. Plenking was a capital offense just like Deppenapostroph and while I haven't thought of that term in ten years, I still get shivers when someone puts a space in front of puncutation mark.

      • Keep thinking about this Patrick Collison tweet on distraction. I only have half-formed thoughts here, but Zed's Slack is the quietest Slack I've ever had in a workplace and over the past year I've been thinking a lot about that and which effects it has.

      • Another good tweet: "capital structure ripples through management and ultimately ripples through employee experience." Nice lens to look through at past experiences working at startups -- in Germany, in US, inside large corporation.

      If you also wrote a hugely successful game purely in assembly, you should subscribe:

generated: October 16, 2024 at 10:19:09