Aaron Schlesinger's Personal Site

What is Hyperlight (part II)?

Unknown — Wed, 14 Feb 2024 21:58:32 +0000

In part 1</a> of this series, I detailed the motivations and background of the Hyperlight project. In doing so, I also mentioned some of the challenges we’ve set out to solve and, at a high level, how we've solved them. In this post, I’m going to talk about these challenges in much greater detail.

Hyperlight is systems-level software, which means it interacts with some of the lowest-level details of how software runs on a computer. The concepts and challenges with which it's concerned are very nuanced and detailed, so I think it's especially critical to talk about them directly, right alongside the high-level description from part 1.

Thus, you'll find this post contains much more technical content than the last, and I try to take time to explain some of the more complex terms and concepts throughout.

Let’s get started!

Virtual CPUs: the workhorse of Hyperlight</h2>
In part 1, I mentioned Hyperlight relies on hypervisor isolation to ensure we provide the security guarantees to which we’ve committed. I also explained that our use of Hypervisor isolation -- also called virtual machines or VMs -- provides a set of virtualized hardware we use to run native code inside a VM. For the purposes of this post today, it's helpful to think of a single VM as an individual "slice" of a physical computer running in the cloud 1</a>.
With this "slice" metaphor in mind, we can visualize a physical machine as a pie with a specified number of slices, each of which can run arbitrary code, which we call a "guest" in Hyperlight parlance, that has been compiled to the underlying hardware architecture of the physical machine 2</a>.
Just as a CPU generally executes code on your laptop or desktop computer, a VM's virtual CPU (vCPU) executes code inside a Hyperlight `Sandbox</code>, which is the term we use to describe a "slice." `
`Sandbox</code>es abstract away the underlying VM and vCPU, and currently support KVM or Hyper-V across Windows and Linux.`

`“Low level” code execution basics</h3>`On nearly all systems, the CPU/vCPU is the most important part of the hardware/software stack involved in making software run 3</a>. There are, however, at least several more components involved in making most running code work, including virtual hardware devices, the virtual memory hierarchy, and many more. Since there are so many components involved, an operating system (OS) kernel is usually required to manage them all and provide a manageable abstraction for programs to compile (and link) against. This abstraction is commonly presented as a group of system calls</a> ("syscalls") that allow programs to safely interact with the OS kernel, which in turn interacts with the hardware. Through syscalls, a program can interact with a filesystem, process model, OS threads, and more. Most general-purpose software requires all of these features and more. Low level code execution for a specific use case</h2> Recall, however, from part 1 that we're not primarily aiming to power general-purpose software with Hyperlight. Instead, we're trying to run smaller pieces of software intended to execute repeatedly in response to an event. Typically, this model is called "functions-as-a-service," abbreviated to "FaaS". The FaaS model presents requirements that are much different from those of general-purpose software. In particular, this model requires a set of features much, much smaller than the those provided by an OS kernel. In many cases, we can get away with providing none of them. Thus, we default to avoiding these features altogether inside Hyperlight Sandbox</code>es. Thus, a Sandbox</code> provides a runtime environment that by default has only a vCPU, its virtual registers, and a single linear block of memory. What you get when you run without an OS</h3> While it might seem like no program could ever execute in this stripped-down environment (and maybe that this is a dumb idea!), some can. And, for our specific use case, this idea turns out to work very well. First, the main benefits to this approach are twofold, as follows: No boot-up process: because we've removed the OS entirely, there's no kernel to boot up and we can start running a user's code with significantly reduced latency</li> Reduced attack surface: because there’s no OS, the attack surface inside the VM – the amount of running software that can be compromised by an attacker – is exactly as big as the running binary, and usually much smaller than that of a running OS</li> </ul> Borrowing from existing approaches</h3> Second, this approach is not really novel; it's been used several times throughout history, and its most recent incarnation is called unikernels</a>. These systems, like Hyperlight, require you to compile and link against a library, in your language of choice, that provides the abstractions you need to run your code. Thus, instead of calling OS-provided syscalls, you call a function provided by a library, which is then directly responsible for interacting with the underlying system to do what you need. Since modern compilers and linkers are very good at optimizing code, the result of this approach is that unikernel applications ship with only the abstractions they need and nothing more. There are drawbacks of this approach (no shared implementation, harder to build software, etc...), but the benefits approximately match those in the above list. Unikernels: an unpopular but useful (for us) technology</h2> If you've heard of unikernels, you're probably in the minority. They're not very popular at all, and for good reason; they are a poor choice for running much of the software we use in our everyday lives. As I write this, for example, I'm running at least 9 different applications on my laptop, each of which is relying on a wide range of syscalls. The OS, which implements all of those syscalls, is in turn utilizing all the different features of the hardware inside my laptop. This is a very common scenario, and it thus makes sense to have one common, underlying operating system to manage that hardware in one place and provide a single, managed abstraction to all the applications that run atop it. In the cloud, though, we have a very different situation. Instead of running 10 or 20 different applications, we're running millions or more, each with wildly different requirements. Some need to run for a long time, some need to run for a very short time, some need to use a lot of memory, some need to use a lot of CPU, and so on. In this kind of environment, it pays to specialize as much as possible. As I mentioned above, Hyperlight targets applications that need to run for a very short time and need to use a very small set of resources. For specifically this use case, unikernel-like technologies like Hyperlight are a great fit. How you actually execute code without an operating system</h2> We now know some important details behind Hyperlight and we get why it's important. Let's now look at how we accomplish Hyperlight's core goal of running arbitrary customer code. First, it would be a bad product experience to require our customers compile their code against some unikernel library we provide. It's also not feasible to rewrite their code to use the unikernel library and then compile it to a unikernel binary. The solution to this problem is (as with many problems in computer science) to add another layer of abstraction. Back in part 1, I talked about two different kinds of VMs, and how we're running one VM inside another VM. The inner VM is the abstraction in question. We build a relatively small number of guest binaries, each of which runs a different VM. We then load the appropriate guest binary (based on the code the customer furnishes) into a Sandbox</code>, load their code into the right place in memory, and finally execute the Sandbox</code>. We currently have a guest binary called hyperlight-wasm</code> that can execute the Wasm</a> instruction set, and we've prototyped others. Wasm, and a different kind of VM</h3> The hyperlight-wasm</code> guest is an important one because it can accommodate any programming language that can be compiled to Wasm. That list of languages is large and growing, which means Hyperlight can easily support a wide array of languages and their associated ecosystems with very little extra work from us. If a user gives us their code in some language, we need to compile it to Wasm and load the hyperlight-wasm</code> binary into a Sandbox</code>'s memory. The binary then does roughly the following: Loads the compiled Wasm instructions into memory </li> Launch a Wasm VM (we currently use WAMR</a> due to its small footprint and controlled feature set)</li> Execute the Wasm instructions inside the VM</li> Return the result to the host (the code that started the Sandbox</code>)</li> </ul> Essentially, our hyperlight-wasm</code> binary is a very, very lightweight approximate equivalent to an operating system that only executes Wasm instructions. Also, the Wasm instruction set supports only a simple processing unit and a linear memory space -- a very close match to the capabilities of a default Sandbox</code>. Wasm does, however, have facilities like WASI</a> that go beyond the defaults and utilize some of the familiar OS-level abstractions. If we encounter those use cases, we can initialize and provide those abstractions in a lazy or "on-demand" fashion and preserve the fast startup times I described above. Results and conclusion</h2> The results of this approach are exciting. In most cases, we can start a sandbox and begin executing a user’s Wasm module in less than 1 millisecond (1/1000 of a second). This latency is competitive with some other hosted FaaS systems, which generally don’t currently provide hardware-level isolation, and an order-of-magnitude improvement over other FaaS systems, which generally do so. I believe the key innovation Hyperlight provides, especially when used in such a FaaS scenario, is the combination of the hardware-level isolation from the latter system and the low latencies of the former system. Because we've achieved this combination, I'm very excited about the future of Hyperlight for the following two reasons: The use-cases: while FaaS is the first and primary use case, I think there are other use cases for which a system like Hyperlight could be a great fit</li> The technology: our use of Rust is expanding within the project (I talk about our use of Rust below in an addendum), we're developing new guests to run different VMs, and we're expanding the security features of the system</li> </ol> Overall, I think Hyperlight represents a new way to run very specific kinds of software in the cloud, and I'm excited to see how it grows. Addendum: Rust powers everything</h2> Because Hyperlight is such foundational technology due to its position in the software/hardware stack, the tools we use to build it are critically important for its security, speed, reliability, and overall success. I'm going to move from discussing how we designed Hyperlight to discussing some of the tools we used to actually build it. I believe the former is as important as the latter in this case. Recall from part 1 that Wasm provides mathematically-provable safety guarantees</a>, but these proofs are only valid if the software that implements the math is bug-free. The same requirement applies to Hyperlight itself. It's hard (and in many cases, impossible) to guarantee bug-free software, so we aim to eliminate several classes (or, types) of bugs and have taken aim at several types of memory safety bugs like buffer overflows</a> and use-after-free</a> issues. To accomplish these goals, we must implement Hyperlight 4</a> in a robust, secure and efficient manner. The Rust programming language</a>, or "Rust," has proven to be an invaluable tool to help us do that job. Why Rust for Hyperlight?</h3> Rust is a relatively new language, especially in the group of systems languages that let the programmer access low-level functionality on the machine. The language differs from most of those other systems languages, however, by providing some very strong memory safety guarantees. To build Hyperlight, we need almost exactly those features. Recall from part 1 that memory safety is an area of focus for the software technology industry at-large, and Rust is being adopted and supported in large part because it does a good job of providing memory safety guarantees. In some cases, it even can provide formal mathematical guarantees, and we've in fact been able to reap some of those benefits in our codebase. Bugs are still possible, but we believe Rust gives us the best chance to build safe, reliable and generally high-quality software 5</a>. There is much more to say about specifically how we exploit the power of Rust, but you can get some ideas from the "Calling Rust from C# and back again" series (part 1</a> and part 2</a>) I wrote on this blog previously. 1</a> Note this metaphor is not perfect because the physical machine is not always divided equally or even logically, and in many cases may be oversubscribed on vCPUs or memory. 2</a> In many cases, this architecture is x86</a>, but ARM</a> is becoming more widely used, and it’s very possible RISC-V</a> will become relevant in the cloud as well 3</a> These days, GPUs and other specialized computing devices are becoming more and more common, especially in modern AI/ML and computer graphics applications. 4</a> And not just this system as specified here and in part 1. We also need to write robust testing and monitoring/alerting code, security features, memory management features, and everything else you’d need to run software like this in an enormous distributed system in the cloud. 5</a> The benefits of Rust in this domain go beyond memory safety. A good representative example is Clippy</a>, which is a static analysis tool included by default with the toolchain that can either directly catch before compile time or prevent code that is likely to lead to future bugs.

What is Hyperlight? Unknown — Mon, 08 Jan 2024 21:58:32 +0000 After I wrote those posts about C#-to-Rust and Rust-to-C# communication (part 1</a> and part 2</a>), a lot of people reached out with questions about what Hyperlight is in the first place. There's not a lot of information out there about the project besides what I've written, what's been presented at Microsoft Build (more on that in a second), and word-of-mouth, so I'm going to fill in the gaps here. I'm going to take a break from the C# <-> Rust series (which I'll resume soon!) to talk about Hyperlight in more detail. I'll cover what it is, why we're building it, and how it works. Background</h2> At the 2023 Microsoft Build conference, Mark Russinovich</a> showed a demo of Hyperlight</a> in his Azure Innovations keynote. In that same keynote, he described Hyperlight as a solution for improving the management and security of Web Assembly (Wasm)</a> workloads on Azure, and explained the motivation of the project as follows: “Wasm lacks the kind of isolation that we require for running a public cloud” </blockquote> In this post, I’m going to detail what he means by that and describe Hyperlight in greater detail. This is the first of two posts, and in the second, I’ll dive into much greater detail on how it works. Built on the shoulders of giants</h2> Hyperlight borrows from, and builds on, substantial prior art, and it’s important to not only credit the technologies on top of which we’ve built this project, but also to establish a common foundation before we delve deeper into the project itself. Hypervisors</h3> One of the most important of these technologies is the hypervisor. Simply put, a hypervisor is a system, often built right into your operating system and hardware stack (but not always! Read more about the various kinds of hypervisors here</a>), that can create virtual “machines” (Hypervisors are also often called Virtual Machines or VMs), or computing devices, on top of which you run some other piece of software. These systems have a long history</a> of</a> research</a> and are one of the critical foundational technologies behind cloud computing. These days, VMs are flexible enough to run software ranging from the most demanding industrial workloads in the cloud to full-blown operating systems on a laptop for a consumer. In fact, I’m writing this article on Windows running in a virtual machine on my Mac as we speak! Importantly for our purposes, hypervisors provide virtualized hardware devices -- including CPUs, hard disks, networking devices, and memory -- on which our “guest” software (the software that runs Wasm workloads; more on these mechanics below) runs. This hardware virtualization layer provides an important security isolation boundary we can use to guarantee a specific level of security to our users’ running applications. VMs often ship with a lot of associated technologies, many of which we don’t use directly, but are important to understand for context. We’ll talk more about them when we dive deeper in part 2. Rust</h3> Another important technology we use is the Rust programming language</a>, or just “Rust”. I’ll talk less about Rust in this initial post in the series than in post 2, but I do want to cover one of Rust’s most important features for our use case: memory safety. Traditionally, a limited set of programming languages were available to do systems programming – writing programs to interact directly with the computer’s hardware. The most popular were the C and C++ languages, and both required developers to manually manage the memory they wanted to use. Situations where a developer manages memory incorrectly – like accidentally trying to use it after they’ve already freed it – have led to many</a> security</a> vulnerabilities</a> across the entire computer software industry, including in Microsoft software. Problems like these have become so well-known and serious that our very own (and aforementioned) Mark Russinovich has tweeted</a> his support for deprecating use of C/C++ in new projects in favor of Rust. He’s not alone; many large companies and well-known projects are investing in Rust</a> because it’s simultaneously a powerful systems programming language and a memory safe one. The way it achieves this memory safety is somewhat novel and very interesting but I don’t have space today to detail it. Anyway, with this powerful combination of features, Rust fills a niche not filled by nearly any other languages, including the mighty C and C++. There’s much more to say about Rust and Hypervisors. We’ll discuss both at length in part 2 and beyond, but we have enough to set a foundation for part 1. Onward! WebAssembly is executed by a virtual machine</h3> When we describe Hyperlight, we often need to differentiate between two different kinds of virtual machines (VMs). While we’ve described hypervisors above, we also need to consider the virtual machines that execute Wasm (called “Wasm VMs” hereafter)! Fortunately, there’s a very clear line between the two, and it’s all about how hardware is represented and utilized. As mentioned in the previous section, hypervisor-based VMs provide a set of virtual hardware devices, but current Wasm VMs do not. Importantly, the latter provide something that looks like a virtual “CPU”, but it only executes the web assembly instruction format and provides no support for any hardware devices. There are APIs built atop Wasm, like WASI, that abstract over hardware devices (like doing network calls and file system accesses), but it is up to the user to implement these APIs in terms of any hardware they see fit. In other words, Wasm VMs don’t provide hardware virtualization, while hypervisor-based VMs do. The latter generally provides additional flexibility and a different kind of security isolation boundary. These differences also imply important differences in how these platforms execute code. Many (but not all) programs can be compiled to Wasm, and any system with a Wasm VM can then execute it, regardless of the underlying hardware architecture. If you’re familiar with .Net or the Java Virtual Machine (or JVM), you can think of Wasm similarly to how you think of CLR or JVM bytecode. Alternatively, I can compile any program directly to native code and run it directly on the hardware (or virtualized hardware!) A VM inside a VM?</h2> I just spent a whole section describing the difference between the Wasm VM and a Hypervisor-based VM, and you may be surprised to learn that Hyperlight runs a Wasm VM inside a hypervisor-based VM. Let’s discuss here why we use both. If you recall Mark’s key comment in his Azure Innovations keynote, he said that while, Wasm provides an isolation scheme, it lacks the isolation we need to run a secure public cloud. This statement has several important points in it, and I want to elaborate on them. The kind of isolation Wasm does have</h3> We already discussed how Wasm and hypervisor-level isolation differ, so let’s discuss how those differences apply to the needs of a public cloud like Azure. You may not know that Microsoft’s online infrastructure is one of the most attacked in the world. This kind of malicious activity isn’t new to us, so we’ve been investing heavily in the security of our software for a long time. One of our most powerful tools to keep our systems secure, and maintain our customers’ trust, is defense-in-depth</a>. If you haven’t heard of this defense-in-depth approach, it’s simply a way to ensure sensitive data and systems sit behind as many security boundaries as possible, while still ensuring suitable performance, efficiency, and more. We’re big believers in this approach at Microsoft, and we employ it wherever possible to achieve at least the following two major objectives: To present an attack surface for any would-be attackers that is ever more difficult to breach</li> To limit the scope (also called the “blast radius”) of any potential breach if one were to happen</li> </ol> While Wasm does provide a provably safe virtualization technology</a>, it’s only “safe” if the engine we use to execute Wasm is always bug-free (it’s not). Since we usually can’t guarantee any software is bug-free (this is why we work to improve our software, software construction methods, and more!), we don’t want to rely only on a Wasm VM to provide the security boundaries we need, so we add another layer of a different type. This new layer is a hypervisor-based VM. Now, we have two VMs, each possibly with bugs in them, with one running inside of the other. This arrangement ensures that security bugs in one VM have a lower impact because of the additional security boundary provided by the other VM layer. In other words, the impact of a security bug in one VM is reduced because of the additional security boundary provided by the other VM layer. How do we do all this?</h3> Well, as I mentioned at the beginning of this post, I’m going to save most of the “how” for part 2, but I want to wrap this post up with a preview. Running software inside of a Wasm VM is relatively easy these days – there are several high quality Wasm VMs available (Microsoft uses both Wasmtime</a> and WAMR</a>, and Visual Studio Code uses Node</a>, backed by the V8 engine</a>), and most have very good APIs we can use to load and run a Wasm module. Our challenge is twofold, as follows: Bundle a Wasm VM and associated executable customer code (called a module), into a format that can run inside a hypervisor-based VM we build, and...</li> Build the aforementioned hypervisor-based VM that can load the Wasm VM + customer module, run a specified function, get the return value, and tear everything down, all with latency measured in microseconds (not milliseconds) </li> </ol> As you’ll see in part 2, the first task is much simpler than the second from the above list, and we’ve spent a lot of time ensuring both low latencies and high “density”. One strategy we use to achieve both things, and reduce the attack surface even further, is to eliminate the operating system kernel entirely from the native code running inside the VM. If you’re familiar with Unikernels</a>, you’ll notice some similarities. I’m looking forward to continuing this discussion. More to come! Calling Rust from C#, and back again: Part 2 Unknown — Thu, 16 Nov 2023 21:58:32 +0000 Back in part 1</a>, I talked about Hyperlight, its history, and some of the challenges involved with the project, including calling Rust</a> from C#</a>. At the very end of that post, I promised another talking about the other direction -- calling C# from Rust. So, here we go! The Challenge</h2> Fundamantally, we want to have C# call into Rust, which we covered in part 1, but in some cases, we want to pass a C# function to Rust, and have the Rust code call back into C#. To do that second part -- the callback -- we have to dive into a topic the C#/.Net world calls unmanaged. For our purposes, "unmanaged" means C APIs/ABIs implemented with Rust. The process to make C# code call unmanaged code is well documented, and the C# compiler and .Net runtime give us explicit support for doing so. It's not much of a stretch to go one step further and implement those C APIs with Rust. We essentially use C as an abstraction layer between C# and Rust, and it works remarkably well. On the other hand, it's far less clear how to get that callback functionality working. We have to put together several different C# and .Net features and wire those things up with Rust to make this work, and I'll show herein how to do it. Pieces of the puzzle</h2> The process to get this callback system working is three-fold, as follows: Convert a C# function (usually called a "delegate") into a C function pointer</li> Consume that C function pointer in Rust and call it</li> Finally, we need to clean up the function pointer's memory when we're done with it</li> </ol> I'll describe these steps below, each in more detail. Converting a C# delegate to a C function pointer</h2> In this first step, the overall goal is to convert a C# delegate</code></a> into an unmanaged function pointer. "Unmanaged function pointers" in C# roughly correspond to C function pointers. Given a C# Action</code></a> -- a function closure with no return value -- the code for converting it to a function pointer isn't obvious, but relatively straightforward, as shown below. // this is the action we want to convert to a C-compatible function pointer. Action<bool> myAction = (myBool) => { Console.WriteLine(myBool); }; // establish a delegate type, and convert our action to it private delegate void MyDelegate(bool param); MyDelegate myActionWrapper = myAction; // then tell the .Net runtime, including the garbage collector, to "forget" // about this function pointer. we need to do this so we can pass the // function pointer to Rust, and be sure that C# won't delete it at some // point before Rust needs it. GCHandle myActionHdl = GCHandle.Alloc(myActionWrapper); // finally, convert our delegate to a function pointer. IntPtr myActionPtr = Marshal.GetFunctionPointerForDelegate<MyDelegate>( myActionWrapper ); // we can now pass myActionPtr to Rust via a C API </code></pre> This code could also work with C# Func</code></a>s, which are function closures with return values. In either case, you have to ensure your parameter types and return types (where applicable) are compatible with C. </blockquote> Passing the C#-generated function pointer to Rust</h2> Now, in Rust-land, we have to accept the IntPtr myActionPtr</code> we just created in C#-land. Rust has very good, nearly zero-cost 1</a> support for both calling C functions and defining C-callable functions, but this topic is nevertheless very nuanced. Collectively these features are called "FFI" (foreign-function interface), but we're interested in only the latter feature. Defining a function callable by C code is as simple as the following: /// accept a boolean and print it out #[no_mangle] pub extern "C" fn my_func(val: bool) { println!("hello! the given value is {}", val); } </code></pre> Take note of a few important highlights of this function prototype. #[no_mangle]</code> - by default, rustc</code> (the Rust compiler) reserves the right to "mangle" the names of functions, methods, and other types. The reasons for mangling are beyond the scope of this article, but we want to turn this feature off for this function so we can call it from C code</li> extern "C"</code> - this nomenclature tells Rust to generate code for this function that adheres to the C calling convention</a> You can read more about calling conventions in the Rust nomicon</a>. </li> </ul> </li> Argument and return types - the Rust compiler will make sure all types in the public function prototype -- parameters and return values -- are FFI-compatible. This feature is invaluable for writing FFI functions in Rust because you can be sure your FFI function is at least technically callable 2</a> from C if it compiles.</li> Header file and linkage - The above function prototype tells the compiler what to output for that function, but doesn't guarantee C code can compile or link against it. I can't go into more detail here without doubling the length of this post, but here's the very high-level way to get that working: Use cbindgen</a> to generate C header files from your extern "C"</code> functions</li> Set your crate type to cdylib</code></a> so your C programs can link against your code</li> </ul> </li> </ul> Functions as parameters</h4> Armed with this ability to define C functions in Rust, we can now create higher-order functions as follows: pub extern "C" fn outer_func(inner_func: extern "C" fn(bool)) { // we likely want to save our inner_func value for later usage, but // that's outside the scope. // // we can effectively call into C# by simply calling inner_func: inner_func(true); inner_func(false); } </code></pre> Cleaning everything up</h2> Cleaning up Rust-owned function pointers generated from C# can be a complex topic that requires some care. Rust vs. C# ownership</h4> First, Rust doesn't truly "own" the function pointer, since it can't reliably and correctly clean up its backing memory (only C# can do that). This fact implies we need to make sure Rust is "done" with it before cleaning it up in C#. There are a lot of different possible ways to do that, varying from just "eyeballing it" to using reference-counting or other more formalized strategies. Inside Hyperlight, for example, we use a home-grown system vaguely similar to reference counting</a>. C# cleanup code</h4> Next, after we're sure Rust is "done" with the function pointer, we can clean it up in C#. Recall in the first step ("Converting a C# delegate to a C function pointer", above) we called GCHandle.Alloc</code> to get the garbage collector to forget a delegate</code>, which means we need to manually clean it up. The following code does the cleanup work: // First, if the type to which myActionHdl points is an IDisposable, clean // that up by calling Dispose() on it var tgt = myActionHdl.Target as IDisposable; if (null != tgt) { tgt.Dispose(); } // Finally, free the actual memory for the delegate to which myActionHdl points myActionHdl.Free(); </code></pre> Recall that myActionHdl</code> is a GCHandle</code> returned from GCHandle.Alloc</code> </blockquote> Caveat: closures</h4> And finally, a big caveat. The C# delegate</code> from which we created myActionHdl</code> is a closure, which means it can close over and/or reference memory outside of its definition. In either case, the garbage collector may still clean up that memory. If that happens, we'll run into use-after-free issues that will likely be very difficult to debug, since the function is invoked from Rust. There is no perfect way to solve this issue, since we're fundamentally bridging two completely different environments (C#/.Net and Rust) over a low-fidelity abstraction (C), but once again, reference counting can help here. Conclusion</h2> Putting the concepts from part 1</a> and this post together, we now have the tools to make C# code call Rust and vice-versa. If you build out a joint C# and Rust codebase, you'll likely encounter several practical issues, the biggest of which will likely be memory management. Throughout this post, I showed how to manually reconcile the .Net memory management system -- the garbage collector -- with the Rust one -- ownership, borrowing and deterministically freeing memory without any runtime overhead. Doing this reconciliation usually requires a lot of error-prone work, so in many cases it's helpful to build a memory management system to unify the two environments. As I implied several times herein, Hyperlight uses a system that uses something similar to reference-counting to do so. Stay tuned for part 3, where I'll detail that system. 1</a> Rust's foreign-function-interface (FFI; basically, its ability to call C code) is officially a zero-cost abstraction</a> (also see this document</a>), but for many practical applications of extern "C"</code> functions, you will need to do additional memory allocations, which incur a cost. 2</a> There is unfortunately a big distance between theory and practice in this case but regardless, knowing your Rust code can theoretically be called from C is a big step forward. Calling Rust from C#, and back again: Part 1 Unknown — Thu, 09 Nov 2023 21:58:32 +0000 I'm working on a project at Microsoft called Hyperlight. The project is all about making very small-footprint virtual machines that start up specific kinds of applications very quickly. At a very high level, we're using modern virtualization technology and some concepts from Unikernels to make these things happen. Mark Russinovich highlighted the project at his keynote at Microsoft Build 2023</a>, so if you want more background on the project, go watch that keynote</a>. The project was prototyped initially with C#, but it turns out C# isn't the best choice for a variety of reasons. Two of the biggest of those reasons are as follows: It takes care and effort to call native code, like ioctl</code> system calls, from C# (or really, all .Net) applictions</li> It also takes care and effort to deal with memory not controlled by .Net -- often called unmanaged memory</li> </ul> Background: Rewriting C# in Rust</h2> In response to these challenges (and more), we decided to move the project to Rust</a> because it's a much better fit for the needs of the project. Rust is a fast, safe, and modern systems programming language, and a great fit for this project. We chose to move the C# codebase incrementally to Rust, and wanted to end up with both a C# and Rust SDK, with the former being a thin wrapper around the latter. Ironically, this work forced us to write a lot of temporary C# code that did precisely the kinds of things we were trying to avoid, like calling into native code and dealing with unmanaged memory. As the rewrite progressed, we were able to remove some of that code, but two things became clear as we began to understand the mechanics of the final joint C#/Rust codebase: We indeed would always need to call "system calls" (sometimes called P-Invoke</a> in C#) from C#, since that's how the Rust APIs are exposed</li> In some cases, we'd need to go the other way and call C# functions from Rust</li> </ol> In our experience so far, (1) is relatively well defined, but (2) is not. We figured out how to do both in a somewhat robust and reliable way. In this post, In this post, I'll describe (1), and save (2) for a follow-up. Calling Rust from C#</h2> C# was designed from the start to be able to call "native" APIs, mainly so developers could still integrate with the many C/C++ APIs already present in Windows. This feature is really useful for us, because we can implement some Rust code, expose it as a C API, then call it from C#. Assuming we design this system carefully, the C# code won't know it's calling Rust code, and the Rust code won't know it's being called from C#. Decoupling is a beautiful thing! In other words, C# can call C APIs [1] and Rust can produce C-compatible APIs, so on either side of the "fence", we just need to figure out how to deal with C-compatible APIs. Creating C APIs in Rust</h3> Rust has strong support for creating C-compatible APIs [2]. While there are a few gotchas, if you can compile Rust code like the following, it will be callable by C code: #[no_mangle] pub extern "C" fn callable_from_c(i: i64) -> i64 { i+1 } </code></pre> There's not a lot of code here, but there are some important features to note, as follows: The #[no_mangle]</code> is important, because it ensures the Rust compiler doesn't modify ("mangle") the name, so C code can call it without understanding the internals of Rust name mangling</a></li> The extern "C"</code> is also important, because it tells the Rust compiler to produce a C-compatible ABI for the function, so C code can call it without understanding the internals of Rust ABIs</a></li> Finally, the types of parameters and return types must all be #[repr(C)]</code> #[repr(C)]</code> is code you put atop a Rust type that tells the Rust compiler the type must be binary-compatible with C code. It will not work on all types, but if you do use it and your code compiles, it's guaranteed to work</li> </ul> </li> Rust pointers (not references!) are compatible with C pointers, so if you have a type that is not #[repr(C)]</code>, you can use a pointer to it in your C-compatible API Warning: most of the Rust code you'll write that deals with pointers in any meaningful way, will be unsafe</code>!</li> </ul> </li> </ul> Once you have a bunch of these C-compatible functions, you need to put them in a library crate that only has type cdylib</code>, like the following: [lib] name = "my_c_lib" crate-type = ["cdylib"] </code></pre> Note: if you have a cdylib</code> crate, you can't also compile it as a "standard" Rust crate. If you want one of those too (hint: you probably will, eventually!), split your C-compatible code into a separate crate </blockquote> Calling C APIs from C#</h3> The other side of this is calling C APIs from C#, which as I mentioned above is a well-known feature of C#. There are a lot of resources out there (reminder: search for "P-Invoke"), but I'll just give a quick example here: [DllImport("my_c_lib", SetLastError = false, ExactSpelling = true)] // You may need this, depending on how your code searches for shared libraries // (i.e. DLLs on Windows or shared objects on Linux). // If you do use it, the C# compiler will throw warnings [DefaultDllImportSearchPaths(DllImportSearchPath.AssemblyDirectory)] [return: MarshalAs(UnmanagedType.U1)] // An i64 in Rust corresponds to a long in C# static extern long callable_from_c(long input); </code></pre> Calling C# from Rust</h2> I've written a lot here, but believe it or not, we only covered the first half of this process. Going the other way -- calling C# from Rust -- involves more details, including some .Net runtime internals. I will cover those details in my next post. See you then! [1] Technically, the C# compiler knows how to call C APIs, and the runtime knows how to call C ABI</a>s [2] Similar to what the C# compiler does for calling C API/ABIs, the Rust compiler knows how to expose ABI-compatible C APIs Synchronizing the KEDA HTTP Addon Request Routing Table Across Hundreds of Interceptor Pods Unknown — Fri, 29 Oct 2021 21:58:32 +0000 The KEDA HTTP Addon project</a> contains three major components: the operator</a>, scaler</a> and interceptor</a>. Of these, the interceptor is the only component that sits in the critical path of all incoming HTTP requests. We also run them in a fleet that is horizontally scaled by software. We're going to focus on how we ensure that any interceptor replica can route an incoming request to the correct backing application at any time. Implications of Multi-Tenancy</h2> The interceptor component is designed to run in a Deployment</code> that KEDA automatically scales. This high-level design has a few implications: There must be a centralized, durable copy of a lookup table -- called a routing table -- that maps any incoming request to the correct backing Service</code> and port.</li> All interceptor pods must reliably stay up to date with the central routing table</li> All interceptor pods must be able to handle any valid incoming request, regardless of application it was intended for</li> All interceptor pods must be able to quickly execute lookups to the routing table</li> </ol> Keeping up to Date with the Routing Table</h2> Since the interceptor needs to do a lookup to the routing table before forwarding any request (or returning an error code), lookups need to be as fast as possible. That means storing the routing table in memory and keeping each interceptor's in-memory copy up to date with the central copy. We do this wth a relatively simple event loop, outlined in the below (Go</a>-like) pseudocode: table = fetch_table_from_kubernetes() report_alive() ticker = start_ticker(every_1_second) for { select { case new_table <- kubernetes_events_chan: table = new_table case <-ticker: table = fetch_table_from_kubernetes() } } </code></pre> A few important points to note about this event loop: On startup, we fetch a complete copy of the routing table from Kubernetes before we report to the Kubernetes liveness probe</a>. This means that Kubernetes doesn't consider any interceptor pod "ready" until it has a complete initial copy of the routing table.</li> The kubernetes_events_chan</code> receives notifications about changes to the central routing table in near-real time. When we get a change notification, we immediately update the in-memory copy.</li> The ticker</code> fires a signal every second, at which time we do a full refresh of the routing table. This mechanism ensures that all interceptor replicas receive changes to the routing table within 1 second of the change being made to the central copy.</li> </ol> The Event Loop Pattern in Action</h2> The KEDA HTTP Addon runs a fleet of interceptors and enlists KEDA to actively scale the fleet. This means that as we send more HTTP traffic to the cluster, we expect the interceptors to automatically scale up. This behavior is one of the most important features of the HTTP Addon. We've built this event loop into the interceptors to ensure that there can be thousands [1] of them running at once, and they all stay up to date with the central routing table -- data that they need to do their job. [1] As we see in the pseudocode above, each interceptor issues requests to the Kubernetes API, so as we scale them out, we generate more consistent traffic to the cluster API. As you scale further than the low thousands of replicas, you would need to add an intermediate layer of caching between the interceptors and the API to ensure that you don't crash the cluster control plane. Fan-in and fan-out with Go Unknown — Mon, 11 Oct 2021 19:46:49 +0000 Hacking on the KEDA HTTP Addon</a>, I found myself having to do something familiar: Split some work into N pieces, do them all concurrently, wait for them all to be done, and then merge all the results together. </blockquote> I've done this a bunch of times before, but this time I forgot how to do it. I took a few minutes away from the computer to gather my thoughts and came back to it. So I don't have to forget how to do it again, I want to write the algorithm down here! What are we doing here?</h2> First thing's first - we need a problem we can break down into a bunch of pieces. Sometimes it's called an "embarassingly parallel" problem. Note that concurrency and parallelism aren't equivalent, but I'm going to be using the word "parallel" hereafter because I'm hoping the machine you run this algorithm on will be able to execute the work on different cores simultaneously. </blockquote> The primary goal here is to run each different piece of the work in a goroutine. That's pretty easy in Go - just put go</code> before the function call that does the work. The tougher part is to get the results of the work, check for errors, and wait for them all to be done -- not necessarrily in that order 🤣. Even though the concept is simple, there is a big-ish gotcha when the rubber hits the road. Below is some code that does "fake" work, annotated with comments to explain it: workToDo := []int{"do", "some", "work"} for idx, work := range workToDo { // make sure you pass the index and work into the // function that runs in the goroutine. // this mechanism makes sure that the goroutine // gets a (stack-allocated) _copy_ of the data. // if you don't do this, idx and work will change out // from under it as the loop progresses. go func(idx int, work string) { fmt.Println(idx, work) }(idx, work) } </code></pre> The biggest gotcha is in that comment inside the for</code> loop. Waiting for the work to be done</h2> Now that we've got goroutines running with the right parameters, let's add a sync.WaitGroup</code></a> to the mix. This mechanism will let us wait for all these goroutines to finish. var wg sync.WaitGroup workToDo := []int{"do", "some", "work"} for idx, work := range workToDo { // add 1 to the waitgroup _before_ you start the goroutine. // you want to do this in the same goroutine as where // you call wg.Wait() so that you're sure that, even if // none of the goroutines started yet, you have the // right number of pending work. wg.Add(1) // make sure you pass the index and work into the // function that runs in the goroutine. // this mechanism makes sure that the goroutine // gets a (stack-allocated) _copy_ of the data. // if you don't do this, idx and work will change out // from under it as the loop progresses. go func(idx int, work string) { // wg.Done() tells the WaitGroup that we're done in // this goroutine. In other words, it decrements // the internal WaitGroup counter, whereas wg.Add(1) // above increments it. // Most commonly, we just do this in a defer statement. defer wg.Done() // this is the "work". in the next section, we'll be // changing this to return a value, because we'll // need to send that value somewhere fmt.Println(idx, work) }(idx, work) } // wait for all the goroutines to finish. this call // blocks until the WaitGroup's internal count goes // to zero wg.Wait() </code></pre> Getting the results</h2> So, now we know when all the work is done, we need to get the results. There are two kinds of results we need to get - the actual values of the work we're doing -- we'll call this the "success value" -- and the errors that it might have returned. Let's focus on the success values first. We're going to use one group of channels, one "final" channel, and a clever way of shuttling data between the former and the latter: Read this code on the Go Playground</a>, if you prefer // this is the channel that will hold the results of the work resultCh := make(chan string) var wg sync.WaitGroup workToDo := []string{"do", "some", "work"} for idx, work := range workToDo { // add 1 to the waitgroup _before_ you start the goroutine. // you want to do this in the same goroutine as where // you call wg.Wait() so that you're sure that, even if // none of the goroutines started yet, you have the // right number of pending work. wg.Add(1) // this is the loop-local channel that our first goroutine // will send its results to. we'll start up a second // goroutine to forward its results to the final channel. ch := make(chan string) // make sure you pass the index and work into the // function that runs in the goroutine. // this mechanism makes sure that the goroutine // gets a (stack-allocated) _copy_ of the data. // if you don't do this, idx and work will change out // from under it as the loop progresses. go func(idx int, work string) { // this is the "work". right now, it just returns an // int. in the next section, it will return both an int // and an error res := doSomeWork(idx, work) ch <- res }(idx, work) // start up another goroutine to forward the results from // ch to resultCh go func() { // we want to indicate that we're done after we forward // the result to the final channel, _not_ just when we're // done with the actual computation. this arrangement // will be useful below, in our final goroutine that // runs after the for loop is done defer wg.Done() res := <-ch resultCh <- res }() } // start up a final goroutine that is going to watch for // the moment when all of the loop goroutines are both // // 1. done with their work // 2. done sending their results to the final channel // // after that, we can close the resultCh. this closure is // important for the following for loop, since ranging over // a channel will only stop after that channel is closed go func() { wg.Wait() close(resultCh) }() // now that we have that final goroutine running, we can // be sure that this for loop will end after: // // 1. all goroutines are done with their work // 2. all goroutines are done sending their work to resultCh // 3. we have processed each result // (in this case, we just print it out) for result := range resultCh { fmt.Println("result:", result) } </code></pre> The code is extensively commented, but notice a few more "meta" things about it: We're enlisting another goroutine for each loop iteration, so now we're using 2N</code> goroutines rather than N</code> (where N</code> is the number of work items to do). If you're worried about the extra goroutines, remember that a Go program can run hundreds of thousands of them comfortably on a relatively modern laptop. Plan accordingly with that in mind.</li> </ul> </li> We use one extra final goroutine to determine when the final goroutine should be closed</li> We no longer use wg.Wait()</code> in the main goroutine. Instead, we range over resultCh</code> to both get the results and determine when all the work items are done.</li> </ul> A final wrinkle: error handling</h2> Now that you (hopefully) have a decent grasp over the code in the previous section, consider that, for most workloads, you'll also have to deal with error handling. It doesn't take a lot of additional code to do it, but it does add a bit more complexity. Let's see how it works: Read this code on the Go Playground</a>, if you prefer Note that you can reduce complexity by using the errgroup</code></a> package. The code herein implements functionality similar to that of errgroup</code>. </blockquote> // this is the channel that will hold the results of the work resultCh := make(chan string) // this channel receives all the errors that occur. // for each work item, either resultCh or errCh will receive // precisely once. both channels will be closed immediately // after all receives happen errCh := make(chan error) var wg sync.WaitGroup workToDo := []string{"do", "some", "work"} for idx, work := range workToDo { // add 1 to the waitgroup _before_ you start the goroutine. // you want to do this in the same goroutine as where // you call wg.Wait() so that you're sure that, even if // none of the goroutines started yet, you have the // right number of pending work. wg.Add(1) // this is the loop-local channel that our first goroutine // will send its results to. we'll start up a second // goroutine to forward its results to the final channel. ch := make(chan string) // this is the loop-local channel that our first goroutine // will send errors on. for each loop iteration, exactly // one of ch or errCh will receive eCh := make(chan error) // make sure you pass the index and work into the // function that runs in the goroutine. // this mechanism makes sure that the goroutine // gets a (stack-allocated) _copy_ of the data. // if you don't do this, idx and work will change out // from under it as the loop progresses. go func(idx int, work string) { // this is the "work". right now, it just returns an // int. in the next section, it will return both an int // and an error res, err := doSomeWork(idx, work) if err != nil { eCh <- err } else { ch <- res } }(idx, work) // start up another goroutine to forward the results from: // // - ch to resultCh // - eCh to errCh go func() { // we want to indicate that we're done after we do the // forward operation, similar to the code in the // previous section defer wg.Done() // only one forward operation will happen per loop // iteration, so we use a select to choose exactly // one of the channels - either the success or error // one. select { case res := <-ch: resultCh <- res case err := <-eCh: errCh <- err } }() } // start up a final goroutine that is going to watch for // the moment when all of the loop goroutines are both // // 1. done with their work // 2. done sending their results to the appropriate channel // // after that, we can close both resultCh and errCh. go func() { wg.Wait() close(resultCh) close(errCh) }() // we're now at a point where we have two "final" channels: // // - one for the successful results // - one for the errors // // we have a few choices on how to handle them, and it's // largely up to your use case how you handle errors or success // results. In our case, we'll loop through both channels, // print out the result either way, and then exit when all // receives happen. // these two booleans are going to keep track of when // each channel is closed and done receiving resultsDone := false errsDone := false // we're going to use an infinite loop and break out of it // when both channels are done receiving for { if resultsDone && errsDone { break } select { case res, valid := <-resultCh: if !valid { resultsDone = true } else { fmt.Println("result:", res) } case err, valid := <-errCh: if !valid { errsDone = true } else { fmt.Println("error:", err) } } } </code></pre> A few more things to note here: We've handled errors and success values with equal importance. In many cases, you might want to immediately exit if you see an error. In that case, make sure that you find a way to receive the rest of the errors and success values on errCh</code> and resultCh</code> (respectively), or tell the remaining goroutines to exit. If you intend to do the latter, I highly recommend using context</code></a></li> </ul> </li> There is a lot of code here! And for that reason, it's worth repeating that you can reduce complexity by using the errgroup</code></a> package.</li> The primary source of complexity is the parallelism (that's the reason this blog post exists!) If you're thinking of using this pattern, I encourage you to measure the serial (non-parallel) version of the algorithm first to determine whether you really need to take on this complexity</li> </ul> GopherCon 2021 CFP OMG! Unknown — Wed, 07 Apr 2021 11:06:38 -0700 Hi Gophers, it's your friendly co-chair of the GopherCon program committee! For the past two</a> years</a>, I've written about the GopherCon call for proposals (CFP)</a> when it opened. Much like 2020, this year's conference will be a bit unique because of the lingering effects of the Coronavirus pandemic, but the CFP process is the same as always. We're looking for great proposals that will become even better talks; and this applies despite your physical presence in San Diego! Whether virtually or in-person, we intend to deliver an engaging, educational, and exceptional conference. As a reminder, we are looking especially for first-time speakers. This means that all else held equal, we’ll choose a proposal from a first-time speaker over that from a veteran. This isn’t meant to discourage experienced speakers; we just want to assure potential submitters that we continuously look for new and fresh ideas from community members we haven’t heard from before. What We Do</h2> Before I go into details on what you need to include in your proposal, I want to explain how we review your proposal. We think it’s important to be transparent about this, and if you’re already aware of the criteria, a reminder never hurts. Here's how it works: You submit your proposal(s) to PaperCall</a>.</li> We have a group of reviewers ranging from repeat GopherCon attendees to experienced speakers who conduct initial double-blind</a> reviews. We do each review based on the criteria below. Every proposal submitted before the CFP closes is reviewed the same way during this step.</li> After the CFP closes, we shortlist proposals with higher than average scores and choose proposals for the final program from this shortlist. At this point, we de-anonymize proposals so we can choose the best lineup of talks for our audience and can contact you if your proposal is accepted.</li> </ul> </li> If your proposal is accepted, our mentoring team will work with you from the time you are notified until the day the conference begins to ensure your success on stage. Our mentoring team will communicate frequently, review your slides, preview dress rehearsals of your talk, give you tips, and be a support onsite. Of course all of this is provided based on your comfort level. We won’t micromanage your preparation, except to ensure your talk is ready to go a few weeks prior to the conference.</li> </ul> </li> </ul> How (and What) You Should Submit</h2> So, given how we work, what should you keep in mind while submitting? Relevance</h3> GopherCon is a Go conference and people pay to attend (Though, you can watch the programming via a livestream for free if you choose not to travel this year!) because they want to learn more about Go. That means your proposal needs to be about Go and contain relevant information. Make sure you tell us, in your proposal, how your talk relates to Go if that’s not obvious. If you're unsure how to do that, you can be explicit, i.e. "This talk relates to Go because ....". Also, keep in mind that Main Theater talks need to be relevant to everyone at the conference. If you're planning to deep-dive into a topic, propose a 45-minute tutorial. Clarity</h3> This one might sound obvious, but it's easy to forget that the review committee and chairs only know what you want to talk about by reading your proposal. Remember that during most of the review process, reviewers don't even know who you are. Make sure that everything you're thinking is clearly explained in your proposal. If you have to write more about what you plan to talk about to achieve clarity, do it. Proposals are commonly rejected because they're unclear, but I've never seen a proposal rejected specifically because it was too long. Correctness</h3> We only accept proposals if the speaker clearly has knowledge of the topic they are proposing. We don't need you to be an expert, by any means, but we want you to at least be speaking from a place of experience. Remember that experience can have many meanings, including: What it was like for you to learn something</li> How you overcame a challenge at work by using $TECHNOLOGY</code></li> What it was like fixing a bug in $MODULE</code></li> </ul> The only way we know whether you can deliver a talk from your experience or expertise is from your proposal. While we're evaluating your proposal, we don't know who you are or anything about you, so validating your knowledge on the subject you are proposing is key. Achievability</h3> The duration of Main Theater talks is 25 minutes and tutorial sessions are 45 minutes. In either case, those timeframes are a lot shorter than they sound. You need to show us that you've thought about how long your talk will take and you know how you'll fit your content into the timeframe you've chosen. The best way to do this is to include an outline of your talk with timing estimates. An outline shows us that you've thought about timing and it also greatly improves the clarity of your proposal. We've rejected talks we liked, but weren't sure if the author knew how they would fit their talk into the time allotted. </blockquote> Impact</h3> This is the piece that's missing in the vast majority of rejected proposals. We look for talks that will teach the audience something new, so we need you to tell us what you plan to teach. Don't assume that it's obvious, it likely isn't. If you're not sure how to do that in your proposal, write this: "The audience will walk away with $THING_YOU_PLAN_TO_TEACH</code>." More Reading</h2> There's a lot of literature out there on how to submit a great proposal. We've added links at the bottom of the CFP page</a>, but this year, one of my co-chairs, Kris Brandow, is writing a detailed blog series on this topic. Please check out post 1</a>, post 2</a>, and post 3</a>, with a final installment publishing on Monday, April 19th. Conclusion</h2> We receive anywhere from 180-225 proposals during our short 3-week CFP process and multiple people double-blind review each and every one. With that said, the following are some important tips to keep in mind: Don't assume that a topic is implicit, especially if it's detailed or intricate. It may be to a coworker, but it's likely not to a reviewer who is volunteering time to read your proposal and doesn’t know the source.</li> We like details! When in doubt, elaborate on an important point in your proposal. Be explicit!</li> Proposals that don't state their impact on the audience have a low chance of success.</li> You don't need to be an expert on your topic, but you do need to demonstrate knowledge of it. Make sure to clearly tell us what your experience is.</li> </ul> And with that, I wish you good luck during the CFP process and I hope to see you on stage soon! Edits for clarification were made to this post on April 16, 2021 Next Steps for KEDA HTTP Unknown — Wed, 09 Dec 2020 11:58:14 -0800 I recently wrote</a> about a new project I'm starting in the Kubernetes world. Its overarching goal is to build an open source, app developer friendly system built on top of Kubernetes. Since then, there has been a lot of interest and a few developments that I'm excited about. Technical Reference and Proposal Document</h2> Along with the GitHub issue</a> that started the project, there is now a proposal and technical overview</a> of the system. Like most proposals, this document will serve as the proposal submitted to the KEDA maintainers and also as a reference document that the project will build against. Included in this document is an overview of all the components in the system, an architectural diagram of how they're assembled, and a discussion of the extensibility of the system [1]. Acceptance to the KEDA Organization</h2> The proposal has been accepted and a new repository</a> under the KEDA organization</a> to house these components. This development is important because the people and the idea now have the support [2] of the KEDA organization and a wider discussion has been started around the direction and featureset. We also have a wider team of developers that have joined to contribute. What's Next</h2> I'm pleased with the direction of this project and looking forward to the next few months. Things are moving very quickly in the new repository. I did a pairing session yesterday with coworkers in which we completed the entire serving and metrics infrastructure [3], and the next component on deck is the operator</a> [4]. If you're interested in seeing this component progress -- along with the project at-large -- I live code it on Twitch</a> 2-3 times per week. I encourage you to tune in and ask questions/leave comments in the chat. </blockquote> Contributing</h2> We're not yet ready for contributions from the community at-large. We have two major tasks to complete before we'll be open for contributions from the at-large community. First, we need to finish the minimal infrastructure. The components and supporting artifacts (Helm charts, CI scripts, etc...) are being built in PR #2</a>, and once we have them completed, we will merge it [5]. Second, we need to establish a roadmap. We're beginning to outline it now and will finish it shortly after merging. We believe that when these two tasks are complete, the project will have an established foundation and be well situated to accept contributions and support a larger community. I along with others will broadcast when we're ready to welcome the wider community. [1] In addition to the discussion in there, I've had more concrete discussions elsewhere about extensibility. Most of them are around alternative levels of abstraction that this system could support. The given architecture is highly extensible, and after this particular abstraction is built, we'll explore others. [2] The word "support" has different implications in different open source projects and communities. In this case, I mean that there is a repository created and the general idea to support HTTP based autoscaling is on the roadmap of the larger KEDA project. [3] We based much of our code on the prototype codebase</a> [4] This component is notable because it's being implemented in Rust</a>. The decision to do so was inspired by my former colleagues in the Deis Labs</a> group at Microsoft. They are innovating in significant ways in the Cloud Native community. [5] Depending on how we progress, we may create and merge multiple PRs. Deploying Apps With KEDA-HTTP Unknown — Mon, 23 Nov 2020 14:26:41 -0800 When you search for "what is Kubernetes?"</a>, content abounds with technical details of the system. There's less about what Kubernetes is at a high level. Here's my attempt at that in two sentences: Kubernetes is an operating system for your cluster of computers. Like any OS, you get a syscall API for CPU, memory, networking, storage and more, and build abstractions from there. </blockquote> If you're new to Kubernetes, let those two sentences sink in before you move on. And if you're an app developer trying to deploy to Kubernetes, remember the things you're not implementing: an HTTP server/router, a database driver, a JSON parser, and so on. You're using abstractions. Kubernetes should be no different. App Developers on the Web</h2> I'm using the term "app developer" in a very imprecise way; that's on purpose. I'm not out to define what this group developers does or is, I'm out to assert the tasks they shouldn't have to do while building their app. Developers should not have to understand details of virtual machines, the Kubelet, Pod</code>s/Deployment</code>s/Service</code>s/etc..., or kubectl</code> [0]. They also shouldn't need to deal with Helm</a> charts or watch Kubernetes dashboards after a deploy [1]. In other words, they shouldn't have to know a distributed syscall API to accomplish everyday deployment and management tasks. Kubernetes is a robust platform on which to run web applications, not an adequate experience for web developers to deploy them. A Better Experience</h2> In 2015, Rob Pike gave a talk at dotGo entitled "Simplicity is Complicated"</a>. Referring to the Go language, he explained that while Go is simple for developers, the code that provides that simplicity is very complex. Indeed, the Go standard library provides a very simple and flexible API to access underlying system resources. Kubernetes is the underlying system. The task we have now is (a) defining what a simple and flexible interface needs to look like, and (b) implementing it. Many cloud providers have relevant products we can draw inspiration from [2]. While these platforms are diverse, most or all of them share roughly the following features. You focus on running your code, not underlying resources like VMs, containers, VNets and so forth</li> Your app can automatically scale up and down based on demand</li> A CLI is available to deploy your code to production with a command</li> Your app can built in the cloud, instead of your computer</li> It's easy to connect your code to other databases and other cloud services</li> You can opt in to advanced features like gradual rollouts and more</li> </ul> Kubernetes has the foundational technologies to build these features and more. Our challenge is to choose the right ones, manage the complexity, and expose a simple and flexible abstraction purpose-built for Kubernetes. KEDA-HTTP</h2> That brings us to KEDA-HTTP</a>, a new project I'm starting. I'm building KEDA-HTTP to bring Kubernetes closer to app developers plain and simple. Knowing approximately what a good platform looks like, the technical details matter. Under the Hood</h3> This project primarily assembles existing building blocks to build the right developer experience [3]: Standard Kubernetes Deployment</code>s, Service</code>s and Ingress</code>es and Ingress Controllers for running pods ("compute") and routing HTTP traffic to them</li> KEDA</a> for scaling Deployment</code>s up and down</li> The operator pattern</a> and Custom Resource Definitions (CRDs)</a> for defining a comprehensive abstraction and API over KEDA and the standard Kubernetes resources</li> </ul> The Future</h3> Kubernetes has democratized distributed systems. Any developer can access lots of distributed primitives over a standard API, but doing so requires understanding large concepts and keeping lots of context in one's head. That requirement alone is a barrier to entry for a lot of people. Further, a large subset of applications don't need some of these primitives [4]. I believe that we know what a simple and flexible interface needs to look like. As I see this system evolve, I'm optimistic that this next level of abstraction will greatly increase the number of developers who can use the technology. Work is ongoing at github.com/osscda/kedahttp</a> and there is a proposal</a> open for adding this technology to the KEDA project. A small team of my Cloud Advocate peers at Microsoft, the great people in the KEDA project and me are working on this project. We're not quite ready for help, but watch the proposal issue</a> for progress. We'll be slowly modularizing the codebase and contributing functionality piece by piece into KEDA. [0] As is the case with every abstraction, developers should somewhat understand inner workings. [1] In many cases, developers also function in SRE or DevOps roles as well. In those situations, they almost certainly would need to use these dashboards to assess the health of their code in production. There is a clear separation of roles or "personas" implied here - the developer writes application code and deploys it to production, while the SRE/DevOps engineer operates it in production. [2] In no order, with no preference and non-exhaustively, these include App Platform</a>, App Engine</a>, Elastic Beanstalk</a>, Heroku</a> and App Service</a> [3] Personally, I'm excited about the fact that the project is partially being implemented in Rust</a>. While the language has a reputation for being hard to learn, I believe that the language provides several features that make it well suited for writing Kubernetes components [4] Such as leader election and distributed leases First Lines of Rust Unknown — Wed, 18 Nov 2020 16:10:08 -0800 After ~8 years of Go as the end-all-be-all of programming languages in my life, I started writing Rust code for a real project</a>, not just as a hobby. The project, briefly put, is a Kubernetes system to let people easily and quickly deploy HTTP-based containers to production with a CLI. There is more to the story than that, but I don't want to stray from what I'm excited about. I'm really learning a new language after all this time, from the ground up [1]. Unlearning Go is Hard</h2> Lots of people come to the Go community and start writing Goava or Gython (Java-like Go or Python-like Go). I usually spend time with many of them helping un-teach that language so that they can absorb Go and build up the cognitive "muscle memory" that they need to be productive in the language. Now, the tables have turned and I'm experiencing the same thing with Rust. One of the important first steps I've taken is accepting that I will still "know" Go even if I "unlearn" it so I really get productive with Rust. I'll always be a Gopher. Similarities to Go</h2> As I progress, I'm writing down some cognitive and structural similarities I see in the two languages. That's helping me make the transition from the Go way to the Rust way. Both are modern, systems-oriented programming languages so they view the world similarly and as such, they've landed on a few similar and important features. Memory Safe</h3> Go uses garbage collection and Rust uses the ownership concept, the borrow checker and (opt-in) reference counting, but both languages give you confidence at compile time that you won't access invalid memory. They also work hard to help you ensure that memory will be cleaned up as expected. Batteries Included</h3> I was going to entitle this section "Built in Tests", but more is built in than just tests. Both languages ship with almost all of the tools needed to build modern software. Go has the go</code> CLI and Rust has cargo</code>. In both cases, you run an installer to get a built in test framework, extensive standard library, dependency management (in Go 1.9 and up!), and more. I can't understate the effect these out-of-the-box tools have on the community. Anyone can come to the language, run an installer, and be ready to write production level code. Other language ecosystems make you learn -- and sometimes even choose from -- a wide variety of tools before you write your first line of code. That's a big barrier to entry for someone new to the language and can turn people off. Both ecosystems do have other tools you can learn, but you can be productive with what comes out of the box. Errors are Values</h3> One of Go's hallmarks is that "errors are values". In practical terms, that means that there is no exception system and the language makes you check any error that a function might return. Rust also lacks exceptions, but has a more expressive type system [2]. It usually represents errors using the Result<T, E></code> type [3]. Many people use match</code> statements to "pull" values out of the Result</code> (you can do it in other ways too), and Rust has an exhaustiveness checker that effectively forces you to check every error. Panic Only in Unrecoverable Situations</h3> Go has a panic()</code> function and Rust has a panic!()</code> macro. They look almost the same and they have approximately the same purpose; if there is no way to recover and your program can't continue under any circumstances, panic</code>-ing will exit immediately with a stack trace (optionally in Rust). Polymorphism</h3> Polymorphism is an umbrella term to encompass many features in a programming language and Rust and Go provide a similar subset of them. In Go, you can create an interface{}</code> that specifies methods that a concrete type can implement. Any concrete type that implements them automatically adheres to that interface, without specifying so. You can use any concrete type where the interface is expected. Rust has trait</code>s, which are very similar. You can define methods in them, and a default implementation for those methods as well (similar to an abstract class</code> in C++). One big difference in this area is that Rust requires that you explicitly state that a concrete type is implementing a trait. That small difference has a big impact in large codebases because with Go, you don't have to add a dependency to your app just to implement one of its interface</code>s [4]. More?</h2> I'm a month or so into building this app and becoming more productive by the day. I've been writing Go for a long time and I'll naturally draw more parallels to Rust as I progress. So far, I'm enjoying the process and looking forward to the future 🚀. [1] I've learned a few languages ad-hoc over the past years, but not from the ground up, and that's why this is exciting for me [2] Go has plans to add generics</a> to the language soon, but the proposed design is different from Rust's generics design and it lacks features that Rust has. [3] Since Result</code> is a regular enum in the standard library, it's only the de-facto standard for error checking; you don't have to use it as such [4] Go is sometimes credited with allowing "duck typing" for this reason, but the feature is closer to structural typing async/await: under the hood Unknown — Tue, 30 Jun 2020 15:00:33 -0700 I'm really interested in concurrency strategies in programming languages, and because there's a lot of written research out there on the topic, you can find lots of strategies out there. When you look at some of the more modern stuff, you'll find a lot</a> of</a> literature</a> on just about the same pattern: async</code>/await</code>. async</code>/await</code> is picking up steam in languages because it makes concurrency really easy to see and deal with. Let's look at how it works and why it helps, using Javascript to illustrate the concepts. I'm a Javascript dabbler at best, but it's a great language to illustrate these concepts with. Don't go too hard on my JS code below 😅 </blockquote> What It's About 🤔</h1> async</code>/await</code> is about writing concurrent code easily, but more importantly, it's about writing the code so it's easy to read. Solving Concurrency Three Ways 🕒</h2> This pattern relies on a feature called Promises in Javascript, so we're gonna build up from basics to Promises in JS, and cap it off with integrating async</code>/await</code> into Promises. Promises are called Futures in many other languages/frameworks. Some use both terms! It can be confusing, but the concept is the same. We'll go into details later in this post. </blockquote> Callbacks 😭</h3> You've probably heard about callbacks in Javascript. If you haven't, they're a programming pattern that lets you schedule work to be done in the future, after something else finishes. Callbacks are also the foundation of what we're talking about here. The core problem we're solving in this entire article is how to run code after some concurrent work is being done. </blockquote> The syntax of callbacks is basically passing a function into a function: function doStuff(callback) { // do something // now it's done, call the callback callback(someStuff) } doStuff(function(result) { // when doStuff is done doing its thing, it'll pass its result // to this function. // // we don't know when that'll be, just that this function will run. // // That means that the rest of our ENTIRE PROGRAM needs to go in here // (most of the time) // // Barf, amirite? console.log("done with doStuff"); }); // Wait, though... if you put something here ... it'll run right away. It won't wait for doStuff to finish </code></pre> That last comment in the code is the confusing part. In practice, most apps don't want to continue execution. They want to wait. Callbacks make that difficult to achieve, confusing, and exhausting to write</a> and read 😞. Promises 🙌</h3> I'll see your callbacks and raise you a Promise</code>! No really, Promises are dressed up callbacks that make things easier to deal with. But you still pass functions to functions and it's still a bit harder than it has to be. function returnAPromiseYall() { // do some stuff! return somePromise; } // let's call it and get our promise let myProm = returnAPromiseYall(); // now we have to do some stuff after the promise is ready myProm.then(function(result) { // the result is the variable in the promise that we're waiting for, // just like in callback world return anotherPromise; }).then(function(newResult) { // We can chain these "then" calls together to build a pipeline of // code. So it's a little easier to read, but still. // Passing functions to functions and remembering to write your code inside // these "then" calls is sorta tiring doMoreStuff(newResult); }); </code></pre> We got a few small wins: No more confusing nested callbacks</li> This then</code> function implies a pipeline of code. Syntactically and conceptually, that's easier to deal with</li> </ul> But we still have a few sticky problems: You have to remember to put the rest of your program into a then</code></li> You're still passing functions to functions. It still gets tiring to read and write that</li> </ul> async/await 🥇</h3> Alrighty, we're here folks! The Promise</code>d land 🎉🥳🍤. We can get rid of passing functions to functions, then</code>, and all that forgetting to put the rest of your program into the then</code>. All with this 🔥 pattern. Check it: async function doStuff() { // just like the last two examples, return a promise return myPromise; } // now, behold! we can call it with await let theResult = await doStuff(); // IN A WORLD, WHERE THERE ARE NO PROMISES ... // ONLY GUARANTEES // // In other words, the value is ready right here! console.log(`the result is ready: ${theResult}`); </code></pre> Thanks to the await</code> keyword, we can read the code from top to bottom. This gets translated to something or other under the hood, and what exactly it is depends on the language. In JS land, it's essentially Promise</code>s most of the time. The results to us programmers is always the same, though: Programmers can read/write code from top to bottom, the way we're used to doing it</li> No passing functions into functions means less })</code> syntax to forget</del> write</li> The await</code> keyword can be an indicator that doStuff</code> does something "expensive" (like call a REST API)</li> </ul> What about the async</code> keyword⁉</h4> In many languages including JS, you have to mark a function async</code> if it uses await</code> inside of it. There are language-specific reasons to do that, but here are some that you should care about: To tell the caller that there are Promise</code>s or await</code>s happening inside of it</li> To tell the runtime (or compiler in other languages) to do its magic behind the scenes to "make it work"™</li> </ul> 🏁</h2> And that's it. I left a lot of implementation details out, but it's really important to remember that this pattern exists more for human reasons rather than technical. You can do all of this stuff with callbacks, but in almost all cases, async</code>/await</code> is going to make your life easier. Enjoy! 👋 How to Run Windows With a Mac Keyboard Unknown — Tue, 16 Jun 2020 11:53:00 -0700 I wrote a bit about PowerToys previously</a>, but today we're gonna talk about my second-favorite feature: the keyboard manager. My first-favorite feature is Powertoys Run - the spotlight-like feature for Windows. I talked about this one last post. </blockquote> Make Windows Feel Like Mac 🍏</h1> I switched to Windows from Mac</a> a few months back, and I'm used to just about everything except the keyboard. Muscle memory keeps kicking in and my thumbs reach for the cmd</code> key instead of ctrl</code>. There's no cmd</code> on Windows - it just opens the start menu. I've opened the start menu a lot. The Keyboard Manager ⌨</h1> The key feature (see what I did there?) of PowerToys that lets my keyboard be all apple-ey is the keyboard manager. It looks like this: There's a lot going on in that window. Let's break it down... Remap Keyboard 🗺</h2> This one is simple in theory. Like the heading says, you get to map a single key. Take the first one in there - I mapped Page Up to the home key (that one wasn't Mac specific - I just kept accidentally hitting it 😉). Here are the keys I remapped to Mac-ify: Caps Lock</code> => Win (Left)</code></li> Win (Left)</code> => Ctrl (Left)</code></li> </ul> This is basically making the Windows key to the left of the spacebar into the Ctrl key, which does similar things as the Cmd key on a Mac. Since I was overriding the Windows key, I moved that up to Caps Lock, since I never use it anyway. You could also do this with the Windows key on the right, if you have one. To do these mappings, you click "Remap a Key', then the Plus icon at the bottom. Click on each box and type the keys you want to map. The "before" key goes on the left, and the "after" key on the right. ❗ Do the mapping from Win</code> -> Ctrl</code> last </blockquote> Remap Shortcuts 🍕</h2> This one piggybacks on key remapping. As you know, Windows has lots of key combinations that "do stuff"™. For example, alt+tab cycles through Windows, alt+tab switches Windows, win+s opens the search menu, win+d shows the desktop, win+e the explorer, and so on. Here are the shortcuts I set for mac-ifying: Ctrl (Left) + Tab</code> => Alt (Left) + Tab</code> cmd+tab on Mac lets you cycle through Windows. This makes it feel the same. This still has a bug</a>, but it's usable enough for now</li> </ul> </li> Ctrl (Left) + Shift (Left) + [</code> => Ctrl (Left) + Shift (Left) + Tab</code> This lets me cycle through tabs quickly, just like on a Mac. This particular one cycles left.</li> </ul> </li> Win (Right) + Shift (Right) + [</code> => Ctrl (Left) + Shift + Tab</code> Cycle to the left, from the right side of the keyboard</li> </ul> </li> Ctrl (Left) + Shift (Left) + ]</code> => Ctrl (Left) + Tab</code> Cycle to the right, from the left side of the keyboard</li> </ul> </li> Win (Right) + Shift (Right) + ]</code> => Ctrl (Left) + Shift + Tab</code> Cycle to the right, from the right side of the keyboard</li> </ul> </li> </ul> </li> </ul> I hope those help! It's by far not everything to make Windows feel like a mac - some Windows things just don't apply to Mac and vice versa. But I think you'll find this a good setup. Enjoy! Storing Data with Dapr (Dapr Series #3) Unknown — Thu, 21 May 2020 10:55:19 -0700 Dapr! No, not dapper</a>, dapr</a>. If you haven't read my past articles</a> about dapr, that's ok. Let me try to boil this big project down to a sentence: 👋 Dapr is a process you run that does all the hard stuff for your app you'd rather not do </blockquote> If you want to read more about it, see the intro blog post</a>. And with that, let's talk about (and demo!) probably the hardest thing that most (any?) apps have to deal with: storing data. 🙋‍♀️ Dapr does other things</a> besides storage too </blockquote> Why It Matters</h1> Dealing with data stores is hard. You have to get the right SDK, mock out your data store for unit tests, write integration tests for round-tripping data, make sure you deal with migrations, failover, and more... Dapr helps with all that, but it also comes with some limitations too. Before we go demo the thing, some pros and cons for storage: ➕ You read/write data with a pretty simple REST API (or gRPC!)</li> ➕ You get a lot of useful features (retries anyone?)</li> ➕ You can swap out implementations (Dapr comes with a bunch of different implementations</a> of the data API across clouds and open/closed source.)</li> ➖ You have to fit your data model into key/value. Not all apps fit into this model</li> ➕ and ➖ You have to do a bit of reading to take advantage of some useful storage features: Strong consistency</li> Optimistic Concurrency</li> Retry policies</li> </ul> </li> </ul> ℹ In Dapr land, the data storage API isn't called a database on purpose because it gives you a key/value storage API. It's not SQL or any other kind of rich query language, so keep that in mind when deciding what to use. </blockquote> If you want, you can check out more details on storage here</a>. 🏄‍♀️ Surf's Up, Let's Demo This Thing</h1> Dapr comes with a bunch of different implementations</a> of the data API across clouds and open/closed source. We'll use the Redis implementation here. ℹ You'll need to have Docker</a> installed and running for this demo to work </blockquote> 1️⃣ - Installing 🛠</h2> I followed the installing</a> instructions, using the Linux script (since I'm using WSL2</a>): $ wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash </code></pre> I had to provide my password because the script sudo</code>s things, so if you're not into letting a random script from GitHub get superuser on your computer, you can download binaries</a> instead. </blockquote> It was installed and I could test things out: $ dapr --help __ ____/ /___ _____ _____ / __ / __ '/ __ \/ ___/ / /_/ / /_/ / /_/ / / \__,_/\__,_/ .___/_/ /_/ ====================================================== A serverless runtime for hyperscale, distributed systems Usage: dapr [command] Available Commands: components List all Dapr components configurations List all Dapr configurations help Help about any command init Setup dapr in Kubernetes or Standalone modes invoke Invokes a Dapr app with an optional payload (deprecated, use invokePost) invokeGet Issue HTTP GET to Dapr app invokePost Issue HTTP POST to Dapr app with an optional payload list List all Dapr instances logs Gets Dapr sidecar logs for an app in Kubernetes mtls Check if mTLS is enabled in a Kubernetes cluster publish Publish an event to multiple consumers run Launches Dapr and (optionally) your app side by side status Shows the Dapr system services (control plane) health status. stop Stops a running Dapr instance and its associated app uninstall Removes a Dapr installation Flags: -h, --help help for dapr --version version for dapr Use "dapr [command] --help" for more information about a command. </code></pre> It even has ascii art 🤡! 2️⃣ Hello, World 👋</h2> Let's get something running. The hello world tutorial</a> actually goes much farther than I expected it would. You get an app up and talking to the storage API (backed by Redis), but then you get a second app up and running and have it call an API on the first one. It's actually showing off service invocation</a> too 🎉 I first got the hello world sample code: $ git clone https://github.com/dapr/samples.git && cd samples/1.hello-world </code></pre> The sample code has a Python app and a Node app. Both has a REST API in them. The Node App 🗼</h3> First, the Node app. The demo README</a> first goes through the Node code a bit. The really interesting bit I saw was that we're doing a POST</code> request with the fetch</code> API to the local Dapr API to store some data. What was cool was you can also just return some JSON and it'll be automatically stored. { "state": [{ "key": "nomnomnom", "value": "pizza" }] } </code></pre> I really like that part because you don't have to manually do anything to get data into the database. It just happens. On to running the thing. You do the standard thing to install the JS dependencies: $ npm install npm WARN node_server@1.0.0 No repository field. added 55 packages from 41 contributors and audited 55 packages in 0.626s found 0 vulnerabilities ╭────────────────────────────────────────────────────────────────╮ │ │ │ New patch version of npm available! 6.14.4 → 6.14.5 │ │ Changelog: https://github.com/npm/cli/releases/tag/v6.14.5 │ │ Run npm install -g npm to update! │ │ │ ╰────────────────────────────────────────────────────────────────╯ </code></pre> But you then run it with the dapr</code> CLI, not npm</code> or node</code>. This spits out a lot of log lines, and that's normal. Here's approximately what it looks like: $ dapr run --app-id nodeapp --app-port 3000 --port 3500 node app.js ℹ️ Starting Dapr with id nodeapp. HTTP Port: 3500. gRPC Port: 42491 == DAPR == time="2020-05-18T16:26:52.702694-07:00" level=info msg="starting Dapr Runtime -- version 0.6.0 -- commit e99f712-dirty" app_id=nodeapp instance=DESKTOP-DQP07VM scope=dapr.runtime type=log ver=0.6.0 <snip> == APP == Node App listening on port 3000! == DAPR == time="2020-05-18T16:26:52.7568756-07:00" level=info msg="application discovered on port 3000" app_id=nodeapp instance=DESKTOP-DQP07VM scope=dapr.runtime type=log ver=0.6.0 <snip> ℹ️ Updating metadata for app command: node app.js ✅ You're up and running! Both Dapr and your app logs will appear here. </code></pre> Ok, now it's running. You use the dapr</code> CLI to also make calls to the web service. I like the simplicity of a single CLI. $ dapr invoke --app-id nodeapp --method neworder --payload '{"data": { "orderId": "41" } }' ✅ App invoked successfully </code></pre> You can also use curl</code> for this if you want </blockquote> And verifying that the request actually went through, and the data was stored, we have logs: == APP == Got a new order! Order ID: 41 == APP == Successfully persisted state. </code></pre> And we can also call the Node App's API to get the new data back out of the datastore $ curl http://localhost:3500/v1.0/invoke/nodeapp/method/order {"orderId":"41"} </code></pre> This and the above dapr invoke</code> call are using one of the features in Dapr's service discovery building block </blockquote> The Python App 🐍</h3> Now for the grande finale! The Python code calls the Node app in an infinite loop, so we should be able to see the Node app responding to requests and also see the datastore fill up at the same time. First, we set up the Python dependency: $ pip3 install requests <snip> Successfully installed certifi-2020.4.5.1 chardet-3.0.4 idna-2.9 requests-2.23.0 urllib3-1.25.9 </code></pre> And then I ran it with the same dapr run</code> command we used for the Node app, except in a new terminal tab. I cut out all the verbose log lines this time! $ dapr run --app-id pythonapp python3 app.py ℹ️ Starting Dapr with id pythonapp. HTTP Port: 43891. gRPC Port: 42163 ℹ️ Updating metadata for app command: python3 app.py ✅ You're up and running! Both Dapr and your app logs will appear here. </code></pre> Now, back in the Node app logs, storage is starting to fill up! == APP == Got a new order! Order ID: 1 == APP == Successfully persisted state. == APP == Got a new order! Order ID: 2 == APP == Successfully persisted state. <snip> </code></pre> That's It!</h3> From experience, getting storage and microservice-to-microservice calls working from scratch takes time, carefully crafted code, and effort. I wanted to compare this Node/Python codebase to compare my experience, and was pleasantly surprised. Let's break it down: 87 lines of Node code, including comments We could save a lot of lines if we returned the "data"</code> JSON dictionary instead of using fetch</code> calls</li> </ul> </li> 23 lines of Node code, including comments</li> </ul> From experience with Go, I can say that a battle tested storage layer for this kind of thing would be about 50 lines. That's an apples to oranges comparison, but I know for sure that all the battle-tested bits are now behind that slick storage API the Node app is using. And funny enough, that part is written in Go 💪. P.S. don't forget to clean up: dapr stop --app-id nodeapp && dapr stop --app-id pythonapp</code> </blockquote> Bonus: Behind the Scenes</h2> I mentioned above that you need to have Docker installed in order to get this working. That's all because Dapr needs two images running in the background for this demo to work. cb722412a5d3 redis "docker-entrypoint.s…" 4 weeks ago Up 6 hours 0.0.0.0:6379->6379/tcp dapr_redis </code></pre> The important one is ... Redis! By default, all the storage API calls the Node app is making get routed into Redis. Wrap Up</h2> So we've seen the storage API and a bit of the service invocation API. Honestly, those two pieces alone can get you a long way just on basic usage alone. As you get more advanced, you'll probably want to learn more about and use things like concurrency management</a> and retry policies</a>. Any way you go, the docs repository</a> and particularly the state management section</a> are great references. That's all for today. I hope you go forth and enjoy dapr-ing! 🤘🚀 Windows, Now With More Toys Unknown — Wed, 20 May 2020 11:39:16 -0700 If you were around way back in the Windows 95 days, you might remember the PowerToys from then</a>. Years and years later, it's back in a similar form, but for Windows 10. I just got started with it and it's another tool in my belt to help me feel comfortable using Windows every day after many</a> years</a> of using Macs. Let me take you through it. You might have read that I recently came from a Mac</a>. I have a follow-up post to this on how to use PowerToys Keyboard Shortcuts to make your keyboard more mac-like </blockquote> 🤯❔</h1> PowerToys is a few things in one package</a>, and different folks describe it differently. The official GitHub repository</a> says this: "...set of utilities for power users to tune and streamline their Windows experience for greater productivity." Well, I love productivity (if you've seen any of my screencasts, I hit the keyboard shortcuts hard), so I'm gonna go with that! 😎 By the way, PowerToys is 100% open source at github.com/microsoft.com/PowerToys</a> </blockquote> I just installed this thing a few hours ago and started messing around with it. It's at version v0.18.0, but the install was not as bad as I thought. Let's go over it! 🍟 Get a Package Manager</h2> There are a few ways to install it</a>, but I decided to use WinGet</a> because it's also new and I hadn't used it yet. My colleague has detailed instructions</a> for installing WinGet, but the gist of it is you need to be a Windows Insider</a> (free and open for anyone to join, despite how the name might sound 🎃) and then you can download it from the Windows Store</a>. That's the easiest way. WinGet is also open source here</a> </blockquote> 🥨 Use WinGet to Install It</h2> When you have it installed, getting PowerToys is a command: $ WinGet install powertoys </code></pre> Run this in PowerShell, not WSL2 </blockquote> 🍕 Get Used to the Features</h2> The very first thing to do is bask in the awesomeness of "PowerToys Run". Seriously. Just hit left-alt+space and a search bar pops up. You can type anything in there and quickly get to "things" on your computer. That generally means a metric ton of things that you didn't even know you had. But it also means ... YOU CAN GET TO YOUR APPS WITH NO CLICKS!!!!🎉🤘🏄‍♀️. I really missed that from my Mac days. Anyway, hit left-alt+space, type in "PowerToys", and hit enter. That will take you to something that looks like the standard Windows settings app, but it's actually the added features that PowerToys gives. Check out that left bar. 7 new features plus a "General" thing at the top. I'm gonna boil each complex feature down into a single sentence, Aaron style™. Here goes: 🍚 General</h4> Go here to do general stuff (😂), most importantly getting updates to PowerToys itself. 🥧 FancyZones</h4> Automatically resize each window in different patterns on the screen, like side by side or in rows. Yes, the name is amazing. The zones are indeed fancy 🍰! But the feature overall is pretty slick and powerful once you get the hang of it. You'll want to try out a few different setups to figure out the best one for you. </blockquote> 🍣 File Explorer preview</h4> See rendered previews of SVGs and rendered Markdown in the file explorer. Optional second sentence 💰: I imagine there will be more formats in the future. 🍤 Image resizer</h4> Resize any image with a right click from the file explorer. 🍎Keyboard Manager</h4> Remap any key or keyboard shortcut from a nice UI. 🍉 PowerRename</h4> Rename tons and tons of files by highlighting them all and right clicking. 🍍 PowerToys run</h4> The belle of the ball for me - basically Alfred or Spotlight for Mac, but for Windows. 🥦 Shortcut guide</h4> A full-screen pop-up that reminds you any time of the keyboard shortcuts you have set up. 🚢 Keep Calm and PowerToy On!</h2> Of course, I'm not doing any of these items justice in a single sentence. You can learn a bit more about each one here</a>, and follow the links in each section to go deeper. Look out for my next post on using PowerToys to make your Mac keyboard work smoother Windows. See you soon! 👋 Athens on Azure Kubernetes Service... Part 2 Unknown — Thu, 07 May 2020 16:18:19 -0700 Howdy all! It's been quite a while since I wrote about Athens on Azure Kubernetes Service</a>. If you don't remember much about it, don't worry. I didn't either... it's been over 6 months! I jumped back into the project, so let's recap and look to the future, and this time with plenty of emojis 👍🚀🎉🐿 The Deployment Right Now</h2> The Athens that you see at athens.azurefd.net</a> is running a modern Athens</a> version, and that's about it. It's supposed to be a global deployment, but it does some things that aren't great for something like that behind the scenes. Right now, it's a proxy, container and a database. We need to be doing more than that. Current Architecture Limitations</h2> I'm gonna pull an old PR trick here and hit you with the bad news first, and end on a good note. This section is all about what we can't do right now. In case you forgot, here's the current architecture of the deployment: you ---> Azure Front Door | | | ACI Container Running Athens | | | Azure Cosmos DB </code></pre> In other words, I'm running the Athens containers inside of ACI containers</a> right now. Each container uses some other cloud services to do the rest of their work. The whole system is a semi-reliable service but comes with some limitations, and when the containers crash, those limitations usually rear their ugly head. When I'm fixing a crash, I usually say something like "I could fix this if I could set up $USEFUL_NEW_THING</code> ... but ACI doesn't let you". ACI</a> is generally good for quickly launching one-off containers, specialized workloads, or using as "burst" computing capacity from a Kubernetes cluster (you can do that with Virtual Kubelet</a>). They're less good for long-running servers that need to scale </blockquote> The Sins of Athens</h2> Here are some of the worst things that Athens has to do right now: Use git</code> to ask GitHub for a list of tags all the time</li> Rely on a git</code> operation for potentially thousands of requests at one time</li> Try to git clone</code> a repo while someone is waiting for their dependency to come back</li> </ul> See a pattern? (Pro-tip: don't use git</code> as the backbone of your public server. It wasn't made for that) Gopher Break!</h2> This is a long section, so let's just take a pause and look at the Athens</a> gopher: Back to Your Regularly Scheduled Programming</h2> Most of those things above "don't happen that often" because a lot of the stuff that people need is behind a caching proxy, so Athens doesn't get the requests that trigger the git</code> operations. But when you run a service long enough, the "don't happen that often" things eventually happen, often. All those bad things that might happen in production will happen, just like water will eventually carve out the grand canyon. </blockquote> When problems crop up, the containers either run out of memory and crash, or just crash because they feel like it. And then I have to hope the cloud restarts them quickly. 🤞 Why Kubernetes? 🤔</h2> I asked myself that question. First off, Kubernetes (AKA "k8s") is complicated. If you get over that complexity, you can get some real power, though. I've been working in the Kubernetes space in some capacity since early 2016, so I tried to look at this decision as pragmatically as possible. 🍏 PRO: The current setup has outages that greatly concern me</li> 🍏 PRO: k8s gives me more architectural options than ACI</li> 🔴 CON: more options = more moving parts = more complexity = more things to break in production with k8s</li> 🔴 CON: There's always something changing or something new to learn in the ecosystem. It's confusing!</li> 🍏 PRO: I'm a bit of a grouchy old man and I'm confident I'll avoid most of the "bleeding edge" stuff that's almost certain to break/change. But I will nerd out on it.... 😏</li> 🔴 CON: There's a lot of YAML to learn (more on this later)</li> 🍏 PRO: I already know a lot of it, enough to not write any this time around (did I mention I'm grouchy??). More on this later</li> 🍏 PRO: I read a Basecamp article</a> recently describing their usage of Kubernetes for their new email service. I highly respect their pragmatic view on new technologies</li> </ul> That's 5 pros to 3 cons. Honestly, the Basecamp article tipped me over the edge though. I respect the technical decisions that team makes over almost any other group of people. Let's get to work ⚒. The Plan</h2> In case you forgot the high level architecture from last post, here ya go! you ---> Azure Front Door | | | Kubernetes ingress controller in AKS ---> Athens deployment (pods) | | CosmosDB </code></pre> Let's zoom in on the Athens deployment (pods)</code> part in that diagram. I need to run more services to add some reliability to the system, and that part is where it gets interesting. Hint: Kubernetes is really good at running, maintaining, and connecting services together. </blockquote> What New Services Do We Need?</h2> The most important service I need is an external storage server</a> for Athens. You don't need to know specifically what that is, just that it's an unreleased feature and that I'm a cutting edge genius for knowing about it. Or, I worked with the other Athens maintainers on it. I actually didn't do that much of the work. I'm gonna go with genius though 😛. There's another service, too, called the "crawler for Athens", also known as crathens</a>. Same thing here, you don't really need to know what that's all about for now. I'll go deeper into architecture in a future post. So now we have Athens, the storage service, and crathens. Dare I say we're building microservices???? Progress!</h2> That storage server is gonna be easy to build. Most of the work is copypasta</a>-ing code from Athens and making it my own. That's how all open source projects succeed, right? Crathens will need a little work, but it's pretty much good enough™ for now. So at this point, I'm working on deploying this stuff. If you haven't deployed an app to Kubernetes before, just know that the whole system is written in YAML. I'm pretty sure they invented their own CPU in YAML. If you're gonna wade into the Kubernetes waters for the first eim, make sure you update your LinkedIn profile before you do. Your old skills don't matter. It's all "YAML engineer" from here on out. </blockquote> Just kidding. You can also write Terraform</a> scripts if you want. I think that makes it a little easier. So naturally I chose to write YAML. Terraform + GitHub Actions 🍤</h2> Just kidding. I went with Terraform 😅. If you want to cut to the chase, here are all my Terraform scripts</a> </blockquote> Here's how they work: They have a bunch of input variables</a> in them so that I can control where I deploy to, all from GitHub Actions</a></li> GitHub Actions</a> runs on every push I do to that repo. That would be the way I deploy a new version of Athens, for example</li> The GitHub actions YAML files figure out if it's a pull request or the master branch. If it's a PR, it deploys to a staging AKS cluster. Otherwise, it deploys to a production one This part is in progress. Check out my Terraform file</a>, and shield your eyes from my sheer Terraform brilliance 😂</li> </ul> </li> ???</li> Profit</li> </ul> That's It?</h2> Yup! Terraform, AKS, and all the cloud stuff that's backing this make it pretty easy. My Terraform scripts and GitHub actions are just tying them all together. You already know some of what I'm using, but below is a complete list for you. I even incuded all the fancy acronyms so you can sound like a boss. Azure container registry (ACR)</a> - for storing all the Docker containers of my code. Not Athens, though. That's already on DockerHub</a></li> Azure Storage</a> - for storing the current state of my cluster. Terraform uses that to figure out what to update</li> Azure Kubernetes Service (AKS)</a> - for hosting the actual Athens and supporting services</li> Azure Front Door</a> - for doing all the fancy TLS/SSL stuff and caching module ZIP files so my poor little Athens instances don't have to serve them up over and over again</li> Azure Monitor</a> - for keeping track of all the metrics, tracing, and logging that Athens (and the other services) are going to be spitting out. I haven't integrated this in yet, though</li> </ul> That should do it! My work continues for now. I still have the storage server to do, and I need to finish up the Terraform scripts. Check back soon for an update. It's gonna be fun to see this thing take shape 😊 Til next time 👋 May 2020 Athens Update Unknown — Tue, 05 May 2020 16:29:07 -0700 You haven't heard from me in a while about Athens</a>. If you use Athens, you're probably all like 😒, but hopefully this post will help you be more like 😊. Plus, we're more than 4 months into the year and I want to share ✔. Before you read on, here's the Athens Gopher for you, because why not! </blockquote> If you just want a summary, we've already had 4 releases this year and I'm excited about some bigger features coming! </blockquote> Why are you writing this?</h2> Good question! Well, I've been writing code and some docs on the site</a> for a while. I haven't actually written about the project in a while 😨. Hopefully this post is helpful 😎. Releases</h2> We've had 4 releases in 2020, starting with v0.7.1</a>. That means mostly bug fixes. The "big" feature is in v0.8.0</a> where we added support for Redis Sentinel</a> as a backend for the distributed coordination system (AKA "single flight"). The biggest bugfix we made this year was to make it possible to set multiple values in the ATHENS_GO_BINARY_ENV_VARS</code> environment variable (#1404</a>) Coming Soon to and Athens Near You</h2> I'm happy with things so far but I'm really excited for what's coming. For a while now, we've been thinking about how to build two major new features: A completely-offline mode</li> Pluggable storages</li> </ul> The first one is still in progress</a> (click that link to go join the discussion). The second one is .... done</a>! 🥳🥳🥳🥳 We haven't released pluggable storages officially, so if you want to try it out, grab one of the recent Docker images from the athens-dev Docker repo</a> and go for it! When we do release it (soon), anyone will be able to write a simple HTTP server for their database of choice, and Athens will use it instead of its own built-in storage systems. The server is the standard download API</a> plus a few other APIs. Athens isn't ditching any of its built-in storage systems, but we'll likely be building future storage backends as external ones. Most importantly, this means that you can use any storage system you want - including internal, proprietary ones - without maintaining your own fork of Athens A Personal Note</h2> I'm really happy with how things are going. Honestly, by the end of 2019, I was burned out. I had a reflex to get involved with everything in the project. That wasn't sustainable for me or the project. Speaking of sustainability, I keep this proverb in my back pocket: If you want to go fast, go alone. If you want to go far, go together </blockquote> It helps me remember to listen more than I talk and ask for help. Both of those things are easy to do with everyone I've met in the Athens community. You are all smart (whether you think it or not), helpful, and good people. If I haven't met you, feel free to say hi to me. I'm @arschles</a> on Twitter and my DMs are open. Same username on GitHub and the Gophers Slack. Any of those channels work. And, of course I'd love to see you contribute to the project</a>! </blockquote> I'll do more of these updates from here on out, so until next time! Aaron💚</li> </ul> What is Dapr? (Dapr Series #2) Unknown — Mon, 04 May 2020 16:40:09 -0700 Welcome back to the dapr series! Last time</a>, we talked about the challenges of getting software right. If you need a quick refresher: The easiest thing about software is writing it</li> The next hardest thing is getting it right</li> The hardest thing is not crashing and burning in production</li> </ul> You might ask this very reasonable question: But writing code is pretty hard already. Does it really only get harder from there? </blockquote> And I'll hit you with one-worder: yes </blockquote> Production?</h3> Before we move on, let me take a second to explain what "production" is: It's a distributed system</a>. Wikipedia says it best: A distributed system is a system whose components are located on different networked computers</code></li> Your code getting relentlessly pounded by the forces of nature</li> </ul> (If you have an app with a DB, you're running a distributed system) In other words, "production" is a harsh, brutal hellscape that'll chew your code up and spit it out like it was nothing 😱. Uh, thanks a lot, sad man!</h2> Yea, that was a little bit of hyperbole but to be honest, I'm hitting you with all these scary things to ... scare you. But only a bit. Why? To get my point across, of course! And to recommend some tools to make your life easier now and down the line. Obviously I'm gonna talk about Dapr here, but here are other good tools you can and should consider. Here are some I can personally recommend from experience: PaaS systems</a> take your code, run it on some internal/proprietary system, and give you a few APIs you can call to do useful things. Azure App Services, Google App Engine, Heroku</li> </ul> </li> FaaS platforms</a> let you upload a function in the language of your choice to the system, and tell it when to run the function (e.g. an HTTP request to GET /dostuff</code>) Azure Functions, Google Cloud Functions</li> </ul> </li> Orchestrators</a> take your code and make sure it's always running exactly as you tell it to Kubernetes</a>, Nomad</a></li> </ul> </li> Service meshes</a> take care of making talking over any network more reliable. You get mTLS</a> for free too 🎉 LinkerD</a></li> </ul> </li> </ul> Dapr</h2> Dapr takes some of the above things and builds on some others. The idea is that you run this daprd</code> process (and maybe some others) "next to" your code (on localhost</code>). You talk to this thing via a standard API (HTTP or gRPC</a>) to get lots of things done: Service discovery</a> Getting the IP for the REST API that you need, when all you know is the name. Kind of like DNS for your app, but designed for apps in production instead of billions of connected devices on the internet. Design tradeoffs, amirite!?</li> </ul> </li> Databases</a> Storing your data somewhere, of course :)</li> Dapr calls these "State Stores"</li> </ul> </li> Publish/subscribe</a> One part of your app broadcasts "I did X" into the world. Another app hears it and does things</li> </ul> </li> Bindings</a> Automatically calling your app when something in the outside world happens</li> Or automatically calling something in the outside world when your app does something</li> These have some overlap with publish/subscribe </li> </ul> </li> Actors</a> Write pieces of your app like it's running on their own, and let dapr run each piece in the right place at the right time (or keep them running forever!)</li> These are great if you need to break your app up into tons of little pieces</li> </ul> </li> Observability</a> Collecting logs and various metrics so you can figure out how to fix your app when it spectacularly fails :)</li> and so you can figure out how it's doing before it fails!</li> </ul> </li> Secrets management</a> Safely storing your database password, API keys, and other stuff you'd rather not check into your GitHub repo </li> And, getting them back when your app starts up</li> </ul> </li> </ul> Each of these things is a standard API backed by pluggable implementations. Dapr takes care of the dirty details behind the scenes. Database implementations take care of getting the right SDK, hooking you to the right DB in the right place, helping you make sure you don't accidentally overwrite or corrupt your data at runtime, and more. As I write this, there are 15 implementations</a> you can use for Databases, from Aerospike to Zookeeper. Possibly most importantly, you don't have to pick up any SDKs to use any of this stuff. It's compiled into Dapr itself. Next up</h2> If you took nothing else away, just remember that you should pick up "distributed systems tools" to help you build your app. Even if you don't think that your app is a "hardcore distributed system". The best ones will make your life easier while you're building it, too. I think Dapr is one of those tools. Right when I start building, I can just use the simple(ish) REST API for my database instead of picking up some new SDK and reading all the DB docs up front. Same goes for some of the other features, they just come later. I recommend giving the README</a> a skim and seeing if it fits your app. So you want to build a computer Unknown — Mon, 27 Apr 2020 16:42:35 -0700 I once built a cheap tower with clearance parts from Micro Center. See if you can guess how long ago it was from the specs: CPU: Intel Pentium 3</li> RAM: 16GB</li> Disk: 128mb</li> </ul> I haven't done much DIY electronic stuff since then except for some IoT tinkering and playing around with drones. So when I realized I need a Windows machine for work</a>, I decided I was gonna get back into the DIY game and build one from scratch. Here's my story. Why build from scratch?</h2> If computers are your profession, it's nice to choose all your tools so you get exactly what you need. Just like if you look in a carpenter's toolbox you'll see tools from tons of different brands. They chose exactly what they need to do their best work. Also, you have to put time and work in up front but you come out with a completely custom computer for cheaper than what you can buy off the shelf. And you know all the ins and outs of the thing, so you can upgrade individual pieces in the future. How to started</h2> Luckily I knew the basics of a computer, so I sketched out the pieces I need. If you're unfamiliar, here they are with crude definitions: Motherboard - holds everything together (CPU, memory, hard drive, fans, peripherals)</li> CPU - Central Processing Unit - does all the fancy things that runs your OS, compiles code, etc...</li> Memory (AKA: RAM) - your processor and programs you build and/or run use this to store temporary data</li> Hard Disk - where your files live, forever</li> Case/Fans - The actual place you put all this stuff. Fans are mounted to the case (and the CPU) to cool all these electronics off while they're running</li> Video card - you plug this into the motherboard and the other side has hookups for your monitor. Some Motherboards have this built in.</li> Sound card - just like the video card but for sound. Just about all consumer-grade motherboards have this built in</li> PSU - Power Supply Unit. This is what you plug into the wall. The other side has wires with different connectors that attach to the video card, fans, motherboard, etc...</li> </ul> How to choose parts</h2> I started by thinking about the main reasons for this computer. I don't game and this was gonna be for work, so my needs were mostly compiling code and running lots of VMs. That means more CPU cores and memory and you can skimp on the video card (which can be a big cost). More CPU cores usually means a more expensive motherboard too, so I had to budget for that too. My big purchase was an AMD Ryzen Threadripper CPU. I had been saving up for a while so I could get one. Most of the work was figuring out what kind of motherboard I would need and how much power I would need from the PSU. Resources</h3> Once I had a more specific idea of what I needed, I went to NewEgg</a> to look at general prices for things. They also have some nifty tools like the power supply calculator</a> that can help you decide what to buy. I also went to /r/buildapc</code> on Reddit</a> to see reviews. You can also ask questions there if you want. Amazon had cheaper prices so I bought everything there. I would have gone to a physical store like to look at parts, but there wasn't a good option in my area. During the COVID pandemic, please research and buy everything online so you don't have to go out, even if you do have a parts store near you that's open </blockquote> Do the build!</h2> I made too many mistakes to list here, so I'll share the advice I learned from it all: Read through the entire step-by-step process for the motherboard, CPU and fan before doing anything</li> Don't force anything except for the memory. You'll probably need to apply pressure there until it clicks</li> Do everything on a clean table if you can. Don't kill your back doing it on the floor like I did 🤣</li> Attach the PSU before anything else</li> ... then attach the CPU and fan before you screw in the motherboard</li> ... then plug in all your components</li> ... then put in the hard disk, video card, etc...</li> When you're plugging things in, the wires will come from the PSU up to the motherboard and other components (often the hard disk and video card). Try to run everything below the motherboard Some cases come with zip-ties and loops so you can bunch wires together and attach them in place to the case. Use everything you can to organize the wires. You'll be happy you did later!</li> </ul> </li> </ul> Enjoy!</h2> I got all my components in January and I've had the thing running since February. As I said, I made a bunch of mistakes along the way. I still have things to improve (liquid-cooling and a bigger PSU for example!), but I'm actually happy I do. This is both a tool and a hobby! I'm really excited to get creative and keep my machine healthy and up to date as time passes. If you do a build, I hope you get the same amount of joy out of it as I am 😊 What Happens to Your Career After 10 Years? Unknown — Thu, 23 Apr 2020 16:41:19 -0700 I see a lot of great career posts on how to get into programming, and some about how to get into senior engineer roles. I'm coming up on 15 years as a software engineer now and I've been thinking lately about how I got here. I'm not an expert by any measure but I wanted to share some advice I've learned. New positions are scary, you should take them anyway</h2> I worked right out of college at Microsoft as the only junior engineer on a team of database engineers. I "knew" I wasn't good enough going in. I was terrified. I was going to get fired on day two. By the time I had left that job, I felt like I had mastered C++ (not even close, but I learned a lot and it felt really good 😂) and made some good friends. The next job I took was at a gaming startup that was growing fast. I had no idea what I would be doing and had no experience with startups. The interviewers assumed I had intimate knowledge a million technologies I had never touched. I guess they didn't care? On my first day, I realized how much I had to learn before I could get started. Over the years I worked there, I learned what I had to learn and ended up working on a lot of new things: Deploying code to millions of users</li> Testing in production</li> Leading a data ingestion system</li> A live gaming server</li> A Cocoa/Objective-C gaming engine</li> </ul> Lessons I still carry with me</h3> I keep having this experience where I run up against a job that would have me doing things I have no idea about and I still get scared. All these experiences taught me two lessons I still carry with me: The people who are "programming gods" are still human. If you ask them nicely for help or advice, they'll happily help you. People in our industry want to pass on their skills</li> All the code that's ever been written was built by a person. I (and you!) can figure out what they figured out, if I have the time and energy</li> </ol> You should write less code as you progress</h2> Non-tech skills are really important. In my first senior role, I felt like I had to work twice as hard and twice the hours. I actually did that for a while too! Obviously, 16 hour days burn you out pretty quickly, so I pulled back on that and had to figure something else out. Working smarter, not harder, is really easy in software roles </blockquote> I realized that about 5 years ago, but I think I'll be practicing it forever. The best ways to work smarter don't involve code, they involve people. Learn about new technologies</h3> New technologies can give you big advantages in your career. You might be able to accomplish big projects quickly by picking up that new web framework. Or you might get that new job or higher title because you were an expert in that new language before most other people. Learning new tech will always give you two more handy skills: Spotting vaporware quickly</li> Being practical about where (or if!) the new tech fits into your stack</li> </ul> Write and teach more</h3> Whatever you learn, teach it to someone else. Here are some ideas: Write internal documents for your company</li> Blog posts</li> Pair programming</li> Screencasting or live coding</li> Workshops</li> Talking about something you're working on at lunch</li> </ul> Any of these things will help other people work smarter too. Teaching actually helps you take your knowledge to the next level, too. Build communities</h3> This is kind of like teaching, but at a larger scale. Instead of passing your knowledge on, you'll teach other people how to pass their knowledge on. Think about an open source project. People start contributing because they're trying to get the same general thing done. If they want to solve something specific, they learn the best ways to do it, write up a GitHub issue, and teach other contributors what they learned. If you're the maintainer of the open source project, you wouldn't be telling anybody to do this, you'd just be making sure that they can do it. That's what building communities is all about, and it's not just about open source projects. These days we have to do things virtually, so you could set up a virtual brown bag lunch at your work, help out at a local meetup (the organizers always need help), or write an article series and encourage folks to contribute (here's fun example: testing on the toilet</a>). Don't focus too much on the future!</h2> Plans are guesses in disguise. In real life, if I knew what I would be doing for the last 15 years, I would have been too scared to even try. These days, I don't spend much time focusing on what I want to do more than about a year out. Make sure you still dream!</h2> Plans are different from dreams, though! Imagine you want to become the CTO of a startup. A plan would be to do project1, become VP of engineering at company2, then become an architect at startup3, then move up to CTO at startup4. In real life, most of those things won't happen in that order. A dream would be different. It would be more like "I want to become the CTO at a startup, but I bet I need to move a few levels up before I can get there. Maybe I'll even need to go to a few different companies" How to make your app harder to build (Dapr Series #1) Unknown — Tue, 21 Apr 2020 11:13:56 -0700 This is the first post in a series about Dapr</a> </blockquote> Software is very easy to write. For lots of us programmers, it's really exciting to create something new. Then it grows and changes over time. Like the foundation of a house, it looks indestructible at first, but the earth has forever to try and break it. It moves, causes cracks and eventually water leaks. The same thing happens to software. Software is easy to write but hard to get right </blockquote> Challenges of distributed systems</h2> Look at any non-trivial, modern app, and you might be able to find 25% of the bugs that it has in it. You'll have no idea about the rest of them, because time and pressure can only find them. Your app gets twice as hard to get right if it has to talk to other APIs, databases or microservices. Then you'll hit network blips, slow APIs, out-of-memory errors, and so on. In other words, the world will eventually find a way to break your app. You have to find a way to handle all of that. Frameworks</h2> That's where distributed frameworks come in. For modern distributed systems, these are sidecar processes/containers that give your app a simple API that helps you deal with these things. They help you: Save things to a database Even if there's a problem with the database</li> </ul> </li> Talk to other services Even when you have to try again</li> </ul> </li> Tell you what services your app is talking to Even when you can't tell from the code</li> </ul> </li> Responding to events like pub/sub Even when you don't expect that many events</li> </ul> </li> </ul> Dapr</a> a new framework that can help with these things. Stay tuned for my next post to learn more! Crystal Fridays Unknown — Fri, 17 Apr 2020 11:02:32 -0700 If you've been following my Twitch</a> you've probably seen that I'm doing "Crystal Fridays" live streams for the past few weeks. Catch me every Friday at 10am US Pacific time on Twitch</a> </blockquote> What is Crystal Fridays?</h2> The premise is simple. I live code writing apps in Crystal</a>. Currently I'm working on cprox</a> - a simple URL shortener that's implemented with a little persistent key-value store</a>. If you don't know about Crystal, it's a language that looks like Ruby but it's compiled. I'm really enjoying it and I encourage you to check it out if you have a spare moment. </blockquote> Join me!</h2> If you're interested in participating in Crystal Fridays, come watch me live code: When: 10am US Pacific Time</li> Where: twitch.tv/arschles</a> Tip: you can use time.is/compare</a> to find out what time that is in your time zone</li> </ul> </li> </ul> Recordings</h2> I record every Crystal Fridays stream and upload it to YouTube</a>, so you can catch up if you miss one. Videos should be up shortly after each stream. How to WSL 2 Unknown — Mon, 30 Mar 2020 17:21:48 +0000 Welcome, Linux-on-Windows-er! I recently wrote about my experience with WSL 2 and Windows</a>, coming from a Mac perspective</a>. This is an unboxing post for how I went from vanilla Windows 10 to a full working dev environment based on VS Code</a> and WSL 2</a>. If you're like me and were initially too lazy to read that link above, WSL 2 stands for "Windows Subsystem for Linux". The first version, WSL 1, was a syscall emulator that gave Linux programs most of the POSIX APIs they need. WSL 2 is a lightweight hypervisor that actually runs a full Linux kernel. In other words, WSL 2 is better than WSL 1 in a lot of ways. </blockquote> The pre-reqs</h2> At the moment, you have to jump through a hoop to get your Windows 10 setup to get WSL 2 working. Join Windows Insiders</h3> WSL 2 will be stable and generally available in the upcoming Windows 2004 release</a>. When that new release comes out, you won't need to be on Insiders and you'll be able to ignore this whole section. I'm excited for that! </blockquote> Being an Insider sounds exclusive but anyone can join. It just means that you'll run less stable versions of Windows. I haven't had any problems with it yet, though. There's a cool tradeoff here, though. You run a development build of the OS, so you get new features in development. Such as ... drumroll please ... WSL 2! It's pretty easy to get on Windows Insiders. Go here</a>, sign up, and follow the instructions. Get WSL 1</h3> You need WSL 1 for WSL 2 to work. That process is basically a PowerShell command and downloading/running a distro from the Microsoft app store</a>. Yes, the app store has Linux distros. I use Debian</a> and Ubuntu</a>. After you download it, make sure to fire up your distro at least once before you move on. Windows requires that, but also it'll let you set up your username and password on startup, which is handy. It'll look a little like this: Notice how many times I mistyped my password =P </blockquote> Check out the instructions</a> for all the details. WSL2-ify your distro</h3> Now it's time for your WSL 1 distro to grow up and get a real job! Convert it to WSL 2 with a PowerShell command. Make sure you run your PowerShell as an administrator. For Debian, which is my jam, it's wsl --set-version Debian 2</code>. You can swap out Debian</code> for your distro name and it'll work just the same. It should look like this in PowerShell: Check out that wsl -l</code> command in the above screenshot. You can always run that to see the distros you have installed </blockquote> If you ever need to delete a distro, you can always run wsl --unregister Debian</code> and it'll just disappear. That means everything in the VM is gone and you'll have to re-install your distro from the app store, so be careful with that one. Get Windows terminal</h3> I mentioned Windows terminal</a> in my previous post about WSL 2</a>. I really, super, highly recommend it for a bunch of reasons: It's a modern Windows app, so it has the most important thing: a cool icon</li> It's customizable</a></li> It's open source</a>!</li> </ul> Back in the day, you had to download the installer from GitHub and install it yourself. That wasn't terrible by any means, but I'm super lazy and it felt like a chore. I'm happy to report you can now get it from the app store</a> with a click. The app ties together the traditional command prompt (cmd</code>), PowerShell, Azure cloud shell</a>, and all your installed Linux distros. When you fire up the app, it'll have a tab that's probably open to PowerShell. Go click on the dropdown arrow next to the tab, choose your distro, and BAM, you're in a Linux terminal. Congratulations, you're WSL2-ing!</h2> Now that you're in a Linux terminal, you can start acting all Linux-ey and do everything on the command line with bash. Set your distro up</h3> I mentioned in my previous post</a> that I use Yadm</a> to configure everything. Yadm works on all Linux OSes, but I'm gonna be talking about Debian/Ubuntu distros. Here's how to bootstrap your install with Yadm so you can download all your dotfiles: $ sudo apt update $ sudo apt install yadm </code></pre> On almost all setups, you'll need to run these two commands with sudo</code> </blockquote> Here's what it looks like: SSH keys</h4> From here on out, you'll need to interact with GitHub repos a lot. So you'll need to set up your SSH keys and add them to GitHub. IF you don't know what to do, follow GitHub's guides in order: Generating a new SSH key</a>. Make sure to click on the Linux tab at the top of that page, not the Windows one.</li> Adding your SSH key to GitHub</a>. Again, make sure to click the Linux tab. In the first section, they say to use xclip</code> but that won't work on WSL 2. Instead, run cat ~/.ssh/id_rsa.pub</code>, highlight the whole public key that gets spit out, and press shift</code>-ctrl</code>-c</code>. That's how you copy something out of WSL 2 so you can paste it somewhere else (including Windows apps)</li> </ol> Here</a> are my super condensed instructions for how to do the SSH key dance, if you prefer. Get those dotfiles!</h3> Now we have yadm, let's do my favorite part: run one command and watch magic: $ yadm clone https://github.com/arschles/dotfiles.git </code></pre> You can swap out my dotfiles repo name for yours. If you want to use mine, that's all good too (PRs welcome!) </blockquote> Here's what it looks like: What's next</h2> You're in a Linux terminal using Windows Terminal, running on a hypervisor, on top of Windows 10. WAT. I know, right? I'll shamelessly plug my other post</a> again here. It explains how WSL 2 was the reason I decided to go over and give Windows a try. To complete your Linux-on-Windows coding experience, go get VS Code</a> for Windows. Install it, and go into a directory inside of your ... WSL 2 terminal. Type code .</code> and watch more magic happen. Your WSL 2 directory will open up in the VS Code Windows app. WAT. This magic works via a fast server that VS Code uses to communicate everything you do (saving, debugging, etc...) back to WSL 2. You're editing in Windows and running on Linux. You can take this a step further and store your Linux-ey things in the cloud with VS Online</a>. That's a whole other topic but worth checking out sometime. </blockquote> Finally, if you use Docker, that works for WSL 2 too. Check out these instructions</a> for how to set it up. Coming from a Mac to Windows & WSL 2 Unknown — Thu, 26 Mar 2020 10:00:41 -0700 I've spent most of my life as a professional programmer using a potpurri of Apple laptops and desktops. The first machine I used to do paid development work with was a MacBook in 2004. I've cycled through a few Mac Minis and MacBooks since then, and I'm typing this on yet another Mac Mini. I don't love Mac OS or Apple hardware per se, but I'm just so used to the Apple ecosystem. I'm talking about everything from knowing the ins and outs of an iPhone, all the way down to details like the many ways to use the cmd</code> key on the terminal. Basically, Apple products rule everything around me I recently even wrote about my Mac Dev Setup</a>. This is a story about why I'm using WSL 2 more and more for my daily work. I'm not going to be ditching the Mac any time soon though :) </blockquote> Windows...</h2> Until now! Apple has been taking a little bit of flack in the developer ecosystem. Lately it's been the keyboards and there have been some other minor hardware issues in the past. They've also made some software decisions that affect me in my day-to-day life. My list of grievances isn't huge, but it got me thinking that what I really want is a Linux environment to do coding in and macs aren't the only game in town. I'd go with Ubuntu, but there's something shiny out there that I wanted to try - WSL2! I haven't done anything on Windows since 2008, so I figured I would flail around, but it turned out to be fairly easy to get my day to day development etc... done. Doesn't hurt that I work at Microsoft, and unsurprisingly, Windows works really well on all of our internal work stuff. </blockquote> My setup</h2> First off, I built a new machine just to run Windows on. I'll let a picture do the talking. I love this thing. After I installed windows, the very first thing I noticed was muscle memory. I'm still catching myself reaching for the cmd</code> key and I'm retraining myself to hit ctrl</code> instead. C'est la vie! VS Code</h3> I was never an Emacs or Vim person. I like a nice looking GUI that I can write code in and I'm used to VS Code, so I picked it up and it feels just as comfortable as it did on Mac. Sometimes I feel homesick and I just open VSCode to feel better =P VS Code has an extension</a> that bridges the Windows world with the WSL 2 world (I'll get to that next), so you can use the Windows app to write code that runs on Linux... </blockquote> WSL 2</h3> This is the big one. My whole development environment aside from VS Code relies on a Linux terminal. WSL 2</a> is a Linux VM that runs on Windows. I'm not a VM expert by any means, but from my experience with VirtualBox</a> on Mac and cloud VMs, the WSL2 VMs aren't the same thing. Startup times are just about the same as regular apps, and resource utilization grows and shrinks just like a regular app. Windows Terminal</h3> WSL 2 VMs are distributed as apps for each Linux distro. For example, right now I have an "Ubuntu" app and a "Debian" app (I'm writing this from the Debian one). If you open one of the apps, you get a terminal in the VM of your choice. It works but you can't customize it and your distros are all scattered each in their own app. Windows terminal</a> is the open source terminal that gathers them all together and it supports the Windows console and Powershell too (hint: you need Powershell to set up WSL 2!). This + WSL 2 + VSCode feels just about the same as my Mac setup</a> and I can safely say that my muscle memory is the only thing killing my productivity at this point :) My dotfiles</h3> Last thing! I use yadm</a> to set up my new Mac and Linux environments. I store all of the scripts that yadm uses in my dotfiles repo</a>, so I went back and gave everything a shine, so it works really nicely on any Debian/Ubuntu VM on WSL 2. I tend to tear down my WSL 2 VMs a lot, so it's nice to set up a new one, set up SSH keys and just run yadm clone git@github.com:arschles/dotfiles.git</code> to have my dev setup all ready to go. If you're new to WSL 2 (or even Debian, for that matter), I encourage you to go check out -- and steal! -- the code in that repository. I've gone through about 5 WSL 2 VMs so far! I'll be posting details on how I set everything up next. </blockquote> Happy coding! Streaming tomorrow March 20, 2020 Unknown — Thu, 19 Mar 2020 16:20:19 -0700 Two days ago, I decided to start streaming</a>. I had a lot of fun just talking through my dev setup and showing how to make a small contribution to Athens</a>. It felt like doing a Go in 5 Minutes</a> screencast, but I could be a lot looser and more informal. I felt ok making mistakes and saying "I don't know". This wasn't me teaching a single topic, it felt like sharing. I'm going to stream again tomorrow. I'm always behind on triaging issues in the Athens project, so I'll do a bit of that. Details of the stream: I'll stream on Twitch at twitch.tv/arschles</a></li> It will be at 10 am US Pacific Daylight Time. Use time.is</a> to convert that to your time zone</li> </ul> Hope to see you there! New to working from home? Unknown — Thu, 19 Mar 2020 11:47:31 -0700 COVID-19 has really quickly turned our lives upside down, and we're truly living in unprecedented times. As tech folks, many of us are working from home now. From a WFH-er for over 5 years now, welcome to the WFH crew! I've had a lot of fellow colleagues and friends over the years who've had the same experience as me in the work from home world: When you're new to working at home. You're in work mode 24/7. You're getting so much done!</li> A few months go by. You burn out and you have to work less. You feel guilty because you think you're not working at all. Everyone says this is what happens when people work from home</li> You notice you're still doing good work despite your feelings. Time to ramp up the hours again!</li> Oh wait, you've gone down that road before. It didn't work out well</li> Then, you start learning how to be sustainable working from home</li> </ul> For some, this takes a few months. For me, it took a few years. Don't be like me. Here are some hard-earned rules that I've picked up over time to help keep me sustainable. I hope they can help you skip this whole process and go right into sustainable mode. Try them for yourself! I was talking to some of my fellow Gophers</a> on the Gophers Slack</a> about the new work setup a lot of folks have. In that discussion, I verbalized these rules for the very first time. What follows is a better-written version of what I wrote in that chat. </blockquote> Your "office" is fundamentally different</h2> You now work in the same place as you live. Your commute is now basically getting out of bed and there's no traffic. That's awesome, but it comes with some hidden gotchas! Having a separate workspace</h3> What you want to do is draw a line between "work mode" and the rest of your life. Don't try to be "on" every waking minute, even though it's tempting. If you can do so, work in a separate physical space. Maybe a spare bedroom or basement (basement office represent!). If you don't have that kind of extra space, separate your workspace out in your mind. One way to do that is to set an alarm on your phone to signal that work has started. It can help you put your brain into work mode in the morning. Then, set up another alarm to signal the end of the day. Make that end time non-negotiable. That means when the alarm sounds, the laptop gets shut, no slack, no IDE, no “just one more thing before I go to dinner”. Ending your day is hard to do. Sometimes you won't get it right. I got it wrong yesterday, in fact. That's ok. Don't beat yourself up and try again tomorrow. </blockquote> Finally, make your workspace yours. Work in a clean and comfortable desk area. That means ... wait for it ... you should clean it up. Having stacks of papers or crumbs around is annoying and distracting. Getting "hardware" for your office</h3> This doesn't mean you have to buy stuff. If you want to, and assuming you can get things delivered still (I have no idea), you can think about monitors, speakers, laptop stands, mice, and keyboards. But there's a lot you can do without spending money. If you're working on the dining room table, put your laptop on top of a book, cutting board, or anything else like that and plug in your headphones. If you have a sweater, roll it up and put it behind your lower back. Do the same thing and use that sweater to rest your hands on as you type. All of these things will help you concentrate and keep your body in shape. Any little improvement to make you more comfortable and you’re winning Don't cut yourself off from people</h2> If you were previously in an office, you were around people. Even if you hated it, don't let that environment go away overnight. Do video calls with people you like. This could be FaceTiming with family or doing a hangout with friends or colleagues. I do a "no expectations" Zoom chat with my work mates every few days. We just come to hang out and talk about whatever comes up. No need to talk just about work if you don't want. If you do, keep it to brainstorming or other low-pressure topics. Just make sure you're not always communicating by typing. Look at someone's face regularly. Talk to them with your voice. You know, human things! Take care of your body</h2> At the office, you have meetings, lunch, or your car at the end of the day. That means you walked to places. At home, there's none of that and no more natural prompts to walk around, so you have to replace them. Take breaks on purpose and get up, walk and stretch your body. There are some apps like Time Out</a> that can help you remember to get up, but I mostly roll with my phone's alarm. Walking outside and putting fresh air into your lungs is best, but that depends on whether it's allowed in your locale. If outside isn't an option, just walking around your domicile for five minutes works wonders. There are real physiological reasons to get up: Get more blood flowing to your legs. This is critical. Study after study shows that, tl;dr, your body is fundametally not designed to sit all day. You actually need to walk around to get proper bloodflow to your legs.</li> Let your shoulders and upper back relax. Especially if you're on a laptop that's below your natural eye level. Stand up straight, slowly pull your shoulders up to your ears, back, and back down. Go as far as you can and then a litle less. If it hurts, stop immediately, reset, and do it way slower and more gently.</li> </ul> Your eyes!</h3> This one is important - I even gave it a different header! Every 10 minutes or so, look at something far away. For at least 20 seconds. You'll exercise a lot of the muscles in your eyes. It's proven to help prevent various kinds of eye degeneration long term. It also helps prevent dry eyes. Sidenote: you won't break your flow doing this. Most of the time, it helps you keep your flow. Try to enjoy!</h2> Finally, try enjoy your time working from home! These are uncertain and possibly scary times, but the silver lining is that you can learn a new way to work and live. Be creative with your office, maybe take up a new hobby, read a non-tech book, teach someone how to code, help your community. Above all, be just a little bit safer than you think you should be out there. My Mac Dev Setup Unknown — Wed, 18 Mar 2020 17:04:19 -0700 Recently, I got into a really interesting discussion on Twitter</a> with @jongalloway</a>, and it was just about 100% constructive and civil. On Twitter! It's all about whether or not to use Homebrew</a> to set up and maintain a Mac for software development. I started programming on a Mac in late 2002, on Mac OS Jaguar, version 10.2. Since then I've used at least 10 different Macs, which means I've set up at least 10 different Macs. We developers are picky about our setups, so setting up a brand new vanilla computer takes time and energy. That got me thinking: I've used a Mac for almost 20 years as my main development machine. What do I do to set up and maintain my machines? Command Line Tools</h2> Like any good developer, I've tried a bunch to automate myself out of work, so a while back I tapped yadm</a> to install many of my tools and store, version, apply, and upgrade various configurations for them. I store all that stuff in github.com/arschles/dotfiles</a>, so let's start there! I use Hyper</a> for my terminal. I just switched to it from iTerm 2</a> primarily because it's configurable with a single json file. I also like the icon on the dock, not gonna lie</li> I use oh my zsh</a> (and the Z shell by extension) because it gives me plugins and a really uniform experience across different terminals. I switch to Linux a bunch and I can move my ~/.zshrc</code> file over and get the same plugins and everything else without any work</li> I use Homebrew</a> a ton. I was the one on Twitter advocating for it! Basically, I can give brew</code> my Brewfile</a> and it'll just go install everything I need with no fuss. It acts like apt</code> on Debian. In fact, it now works on Linux (that's pretty new) so I can move my Brewfile</code> over just like my ~/.zshrc</code></li> direnv</a> is a small little utility that lets me load/unload environment variables just in a single directory. It comes in handy all over the place. Ya know every time some getting started docs say "you need to have X, Y and Z variables in your environment"? That's when I reach for direnv and write up and .envrc</code> file inside my project.</li> hub</code></a> is a nice little CLI that lets me do GitHub "things". I do a lot of open source work (athens</a> a lot these days) so hub</code> can take me from committing a change alllll the way to submitting a PR to the upstream repo. I really only use it for quick PRs because I like to go into the browser to really think through my PR descriptions, but it's really nice for quick things</li> gh</code></a> is sort of a newer version of hub</code>. It's called the "GitHub's official command line too" so my money's on this thing implementing everything that hub</code> has. It's not quite there yet, but I'm starting to move away from hub</code> to this. The maintainers of the project are open to feedback</a> also, so I feel pretty good about the move </li> </ul> Desktop stuff</h2> I try to stay in the command line as much as I can when I'm programming and only break out for really good reasons. I mentioned that I go into the browser when I want to submit pull requests. That's so I pull my head out of the sand and think about the stuff I just built. Another good reason I pop out of the command line is for my editor. I never got into vim or emacs. I've always used IDEs. In recent-ish memory, I've used almost all of the JetBrains</a> IDEs, Visual Studio</a> (I've written a good amount of C#!), Sublime Text</a>, Atom</a> and now I'm currently on VS Code</a>. I plan to stick with VS Code for a while for my main editor. It's really flexible and seems to have the biggest ecosystem of all of the open source editors/IDEs. Plus there are some cool "add on" features like VS Online</a>. Actually, check that one out if you haven't. It's the VS Code GUI on your machine but you run/debug/whatever the code on a cloud VM And the last one! I use GitHub desktop</a> a ton. Technically I can do just about everything on the command line instead of using this app, but I have lots of good reasons why a lot of the time I go into this app instead: The colors! Not gonna lie, I like the green and red lines showing me in the fancy, shiny Mac color scheme what I'm about to commit</li> Partial commits. Being able to commit only some of a file is really easy in the app, and can be super useful. Especially when I implement 5 things at once, pick my head up, and realize I need to split stuff up into 5 separate commits</li> Everything in one place, in a UI. This app shows everything that's going on in your project at once. You can see what branch you're on, whether you need to pull code down from the server, how many commits you still need to push, history of commits, etc... all at once. On the command line you have to run a different command for almost all of that stuff. It's nice to see the whole picture sometimes</li> </ul> One more thing...</h2> Those are the big things. My development life is a hodgepodge of CLIs and GUI tools and it works great for me. I can make quick, focused progress on a PR or issue I'm working on, without many distractions, and that's all I can ask for! But! I'm about to start challenging myself, and change the most fundamental of all of my tools: the operating system. I'm starting to move to Windows as my main development OS. I'll be using WSL 2</a> to install and use many of the same CLI tools I'm already used to, but the rest will be interesting to say the least. I'm challenging myself to learn (and unlearn!) a lot to get comfortable with a brand new dev setup. Interesting times ahead - stay tuned :) Tips for submittiting to the GopherCon 2020 CFP Unknown — Tue, 17 Dec 2019 13:18:53 -0800 The GopherCon 2020 Call for Proposals</a> is open right now, and it closes on January 10, 2020. I am a co-chair of the program committee this year and I want to encourage you to submit a proposal. Now that the year is drawing to a close and many of us will take some time off, it's an especially great time to sit down with your ideas and write up a great proposal. In this post, I'm going to answer common questions and comments I hear and talk about some common mistakes that I see in proposals. Whether or not you've written proposals before, I'm highly encouraging you to take 20 minutes to read this article</a> by Dave Cheney. Dave is a very accomplished engineer and has years of experience reviewing GopherCon proposals. </blockquote> I've never done any public speaking before, can I submit?</h2> The Go community is a large and diverse group. People from every corner of the world write Go code and they all have different experience levels, backgrounds, races, gender identities, and so on. Every year, we aim to make the GopherCon speaker lineup reflect the community at large. We want to hear more of the ideas that our diverse community has to offer. If you're a new speaker and you have an experience you think is interesting, please consider submitting a proposal to us. Same goes if you're a new programmer. If we select your proposal, we'll give you all the help and support you need to give a great talk. Can I submit more than one proposal?</h2> Yes, absolutely! Please just make sure that all of your submissions are appropriate for GopherCon. See "Advice for Authors" on the CFP page</a> for a guideline on what makes a proposal appropriate. Papercall has a feature that lets you re-submit a talk from another conference to GopherCon, but we rarely select those talks. Instead of using that, please take a few extra minutes to tailor what you have to GopherCon, and then submit the new writing. </blockquote> What's the difference between a tutorial session and a keynote session?</h2> Tutorial sessions are 45 minutes long and happen in three tracks, which means that three of them will be happening at the same time at the conference. These sessions need to talk about or teach a Go-related topic in depth. If you're writing a proposal for a tutorial session, try to keep a specific audience in mind. "Go programmers interested in writing internet services" is too general. "Go programmers interested in building distributed databases" is better. Keynote sessions are 25 minutes long and happen one at a time, so everybody from the conference will watch them at the same time. If you submit a keynote proposal, make sure that the huge diverse group of Gophers at the conference will get something out of it. I submitted last year and my proposal didn't get selected, can I submit one this year?</h2> Absolutely. There are a lot of reasons why we don't or can't select proposals every year, so it might not be anything you missed last year. If you think your proposal is good, feel free to re-submit it, and if you have a new idea, you can submit that too. It's ok to submit 2 or more proposals. </blockquote> I don't like public speaking</h2> I hear this one so much that I need to convince you to submit a proposal anyway. I don't like public speaking either. I'm scared I'll mess something up and lose the respect of my peers. I still do it, though, because I want to share what I know with my colleagues in the community. Next time you talk yourself out of writing a proposal, try to focus instead of sharing your ideas, and go for it. If your proposal is selected, we'll do everything we can to help you feel comfortable and prepared for you talk. Last year myself and other reviewers sat with speakers and watched them rehearse their entire talk over video chats and at the conference. We're planning to do the same thing this year. I don't have any ideas for a proposal</h2> We all have unique experiences and ideas -- even if we're brand new to programming or Go -- but it's hard to get them out unless you can step back and think about your work and experiences at-large. Spend some quiet time to think about some new Go-related thing you've learned recently or what your experience has been like solving a technology problem at work or at home. This could be something as small as 10 minutes of quiet time at lunch or after work. Conclusion</h1> What I've written here is based on common questions and pitfalls I've seen in real life GopherCon proposals over the past 2 years. It's certainly not everything you need to know to write a great proposal. As I said at the beginning of this post, Dave Cheney has a much longer history of reviewing GopherCon proposals, and he's distilled that experience into a hugely valuable article</a>. I can nearly guarantee you'll find some way to improve your proposal after reading this post, so please take the time to do so. With that, good luck on your proposals, and happy new year, Gophers! Athens on Azure Kubernetes Service Unknown — Wed, 27 Nov 2019 17:09:38 -0800 I've been working on Athens for well over a year now, and I've mentioned before that I have been running the athens.azurefd.net</code> server on my own for almost all that time. The infrastructure that powered that server looked like this: you ---> Azure Front Door | | | ACI Running Athens | | | Azure Cosmos DB </code></pre> If you're not familiar with the acronyms, here are some short explanations for you: ACI</a> stands for "Azure Container Instances". It's a handy way to run a group of Docker containers together in Azure, and get a public IP & hostname for them</li> Front Door</a> is Azure's reverse proxy. It does a bunch of stuff, but the things that I care about are TLS termination and global caching. I've written and spoken about front door a lot in the past, and I'm a big believer in it. I think it's one of Azure's most handy services</li> Cosmos DB</a> is Azure's global "NoSQL" database. It supports the MongoDB API and does replication across Azure cloud regions very nicely, with no intervention from the operator (AKA me!)</li> </ul> It's important to note that Athens can run anywhere. I work for Microsoft so I chose to run it on Azure infrastructure and with CosmosDB as the database. I might consider moving to using Azure blob store later. </blockquote> Original requirements</h1> When I set up this public proxy, I wanted to give anyone interested an opportunity to try Athens without installing anything. I think it did exactly that, and it turns out the proxy also got Gophers to start thinking about modules and get their local environment set up. This deployment was the first of its kind, too! I had no intention of making it "production ready" though. My goal was to encourage teams and companies to roll out their own hosted servers, so the Athens team could focus more on empowering teams to run their own internal Athens servers. I needed a deployment that would mostly run itself. Those above technologies let me build an architecture that took care of mostly that. Most of the work I had to put into this deployment was managing the transition from Go 1.11 to 1.12 and then to 1.13. I generally spend no more than an hour a week taking care of this infrastructure, and I've talked to a nontrivial amount of folks using athens.azurefd.net</code> in their production CI/CD pipelines. That's a testament to how well this architecture worked over time. New requirements</h1> These days, there are three</a> good</a> proxies</a> that I know of which are public and mostly production ready proxy. In theory, my deployment isn't really necessary anymore. That being said, I firmly believe that Athens needs to live on the public internet as a testbed for our software. I still won't commit to it being production ready, so use it at your own risk (I'm sure some folks will!) But it's not going anywhere. Not only is it going to stick around for testing, I'm also going to use the infrastructure as an educational tool. Instead of running Athens on ACI, I'll be moving the hosting infrastructure to Kubernetes. This move will open up a lot of flexibility in how I run, debug and deploy the service. Here's a high level diagram for what I'm planning to do: you ---> Azure Front Door | | | Kubernetes ingress controller in AKS ---> Athens deployment (pods) | | CosmosDB </code></pre> And here are some things you don't see: This simple script</a> deploys to a (few) regions that Athens currently runs. I'll be using Terraform to deploy to one (and more in the future!) AKS</a> cluster</li> I'll likely use a service mesh system to control </li> I'm planning to use Athens' built-in distributed tracing support to help with debugging</li> I'm planning to use lathens</a> to cache the /list</code> and /latest</code> API calls for each module</li> I'm planning to use crathens</a> to crawl upstream VCS repos and proxies to keep modules up to date</li> I'm planning to centralize logs somewhere. Not sure where yet!</li> </ul> What's coming</h1> I've already gotten started. In fact, you can see a running prototype of an AKS-backed Athens at k8s.goreg.dev</a>. It's missing a lot of the features in that above list, but it's built approximately the same as the architecture in that diagram above. All the code to do all of this is open source</a>, and this is post #1 of a series that I'll use to chronicle my experience setting up and running Athens. See you soon! Building Full Stack Applications with Go, Vecty & WebAssembly Unknown — Thu, 21 Nov 2019 13:19:52 -0800 Building full stack web apps with Go, Vecty and WebAssembly</h1> Welcome folks! Today, we're going to talk about writing full stack web applications in a brand new way. I wrote this post a while ago and I'm reposting it on my blog now. It originally appeared here</a> </blockquote> Many of us have heard of -- and maybe written -- full stack web applications. We do them in a variety of different ways, but the common denominator is usually Javascript</a> & Node.js</a>. Today, we're going to break with that tradition and write a complete web application - front end and backend - without writing a line of Javascript. We’ll be comparing the developer experience to JavaScript along the way, but we’re going to be only writing Go</a> for this entire project. We'll learn how to build a single page link shortener application with just Go, and we'll end up with working code that shows it in action. Prerequisites</h1> Today, we're going to be focusing on Go so make sure you've installed</a> the tooling on your machine. I'm going to assume you have basic knowledge of Go, so check out the free Tour of Go</a> to brush up (or learn!) if you need to. All the shell commands that I'm going to be showing work on a Mac, but should also work on most Linux systems (including WSL</a>!). Finally, make sure to clone the repository</a> with the code for this article. And then you're good to go, so let's get started! Getting started</h1> First, we're going to get the application running locally. Coming from Webpack</a> and surrounding technologies -- which you'd use to build a web app with Javascript -- building and running this application is embarrassingly easy. There's a front-end and a backend part (more on that below), and you compile both of them with the go</code> tool, which requires no configuration. First, run the backend server: $ go run . </code></pre> Next, build the frontend in a new terminal window: $ cd frontend $ GOOS=js GOARCH=wasm go build -o ../public/frontend.wasm </code></pre> Finally, go to https://localhost:8081 in your browser to see the app in action. How this all works</h1> Like most web apps, our link shortener has a front end and backend piece. In our app, the backend is just a static server written in Go. All of the magic is in the front-end directory, so let's start there! If you're familiar with React</a> or the DOM</a>, you'll recognize lots of the concepts we'll cover. If not, this stuff should come naturally as we go through it. We're using a new Go framework called Vecty</a> to organize our application. Vecty makes you break down your app into components and arrange them into a tree. The whole scheme is really similar to HTML and the DOM or React. Here’s what our app’s high level components would look like if they were HTML: A h2</code> for the title of the page</li> A form</code> to enter the link to shorten</li> A div</code> to hold the shortened link This value is dynamically updated as the user types the link into the above</li> </ul> </li> An a</code> to save the short link</li> </ul> Vecty components are so similar to React that they look like the Go equivalent of JSX</a>, except that they have more parentheses =P. Let’s zoom in on one and see how it works. Here's the code for the form</code> component: elem.Form( elem.Input(vecty.Markup( event.Input(func(e *vecty.Event) { short := uuid.NewV4().String()[0:5] h.shortened = short vecty.Rerender(h) }), )), ) </code></pre> First, elem.Form</code> and elem.Input</code> on lines 1 and 2 are for the <form></code> and <input></code> tags, respectively. Those are both Go functions that take one or more arguments. Each argument is something that goes between the opening and closing HTML tags. For example, the stuff we pass to elem.Form</code> goes in between <form></code> and </form></code>. This is what the above Go code would look like in HTML: <form> <input></input> </form> </code></pre> Pretty simple, right? The last piece of code we didn't look at is that event.Input</code> function. This is an event handler just like in HTML/JavaScript. This function takes in another function, which in this case is roughly an [onchange](https://www.w3schools.com/jsref/event_onchange.asp)</code> handler. Just like you'd expect, that *vecty.Event</code> argument the handler takes in is roughly the same as the JavaScript event. The logic to actually shorten the link is all inside this handler, and it's fairly simple. Here is that code commented thoroughly, to explain what's going on: // First, make a new UUID and take the first 5 characters of it. // This will be our new shortcode short := uuid.NewV4().String()[0:5] // Next, write the shortcode to a variable. This variable is shared // with the <div>, so when we re-render this component, the <div> will // get updated h.shortened = short // Finally, re-render the component so that the <div> gets the new shortcode. // Unlike React, there's no automatic diff functionality. We tell Vecty // explicitly which components to re-render. vecty.Rerender(h) </code></pre> You Get Web Assembly for Free</h2> Vecty can scale to big applications because of this component structure, and we can scale our app as big as we want by adding more components as needed. For example, we can add a component above our current top level to dynamically route to different sub-components based on the URL. This would be similar to react-router</a> for the React community, for example. WASM == HTML?</h2> Nope! WASM is a full departure from the DOM and everything HTML. I compared all the components in the last section to HTML tags, but they aren't! That's where the big difference between Vecty / WASM and React comes in. We're compiling our Go code straight to WASM, which represents these components differently from the DOM. WASM == Javascript?</h2> The same thing goes for Javascript! That Go code we saw in the event handler doesn't ever turn into Javascript, and the browser doesn't run the Javascript interpreter to run it. Instead, the Go compiler translates it into a WebAssembly program</a>, and the browser runs that directly. You get some performance wins this way because your stuff doesn't need to go through the Javascript runtime. On the other hand, Go-based web assembly binaries can get pretty big (they're smaller with other langauges though), so there's a tradeoff there. Conclusion</h2> At the end of the day, you get some big benefits from using Go and Vecty to build apps: You get to think in terms of components and nesting, just like with React and the DOM</li> You can write as much dynamic logic as you want, right next to your components, all in pure Go</li> You can share code between the server and client, similar to writing a React client and a Node.js server</li> You get to take advantage of WASM Or you can compile your Vecty code to HTML too if you want! That's a whole other article ;)</li> </ol> </li> </ol> Go 1.13 for Private Repositories Unknown — Thu, 12 Sep 2019 13:37:57 -0700 Go 1.13 is a new beast. It's a patch release in semver terms, but there are some big changes going on here. Most of them are really good, but if you have private dependencies, your build is gonna break out of the box. I'm gonna explain here what's going on and how to fix it, using only two (I know, right??) shameless plugins for my favorite project -- which happens to solve these problems and I also am on the core team for -- Athens</a>. So, here's the big thing that changed and is gonna cause you headaches: This is the first time that the go</code> toolchain has out-of-the-box integration with a hosted service </blockquote> In more detail: Modules are now turned on everywhere by default</li> The default way you download new dependencies is from proxy.golang.org</a> - not hosted version control systems like GitHub</li> Go now consults the checksum database</a> every time you ask for a new module not in your go.sum</code> file</li> </ol> Let's go through these points one by one. Modules are now turned on everywhere by default</h1> Everything else being equal, this is amazing. There's been debate, accusations, fighting in the community over how to do Go dependency management. I stayed out of almost all of that because I don't have any more hairs on my head to lose. Not everyone is happy with modules, but just like every change, that will pass. More importantly, we have a good enough system, that we can make work, and that we can agree on. Standardization is good in software systems! Also, part of the formal modules standard is what makes Athens and all the other module proxies possible. Speaking of proxies .... The default way you download new dependencies is from proxy.golang.org</a></h1> This one can be good or bad, depending on the type of codebase you're working in. My explanation is long, so I'm not even gonna be that sad if you just scroll down to the bold text below. If you want details, read on! The module spec has a small section that details a module download API, which is fancy way to say "a few GET requests to download source code and go.mod</code> files." But the API means we can build servers for modules now. So as of 1.11, you didn't have to use git clone</code> for you package manager anymore. By the way, how did we last for almost 10 years using git clone</code> to install dependencies? Anyway, I started building one of the first module servers</a> and one of the most widely used open source ones. I also gave a talk about it at GopherCon US this year. My big message was that Athens is one of the tools we're going to use to keep the modules ecosystem decentralized - just like dependencies always have been in Go. I know I already bashed git clone</code> (see what I did there?) as a package manager, but it's a distributed versioning system, and we co-opted it to build a distributed package index. That takes me to the change in 1.13. Before 1.13, you had to set a GOPROXY</code> environment variable to tell Go to use this module API and to fetch modules from the server you specify. Otherwise, it would keep doing git clone</code>. Now things are different! GOPROXY</code> defaults to https://proxy.golang.org</code> as of 1.13. proxy.golang.org</code> is a module repository run by the Go team at Google. This change means that, out of the box, all of your go build</code>, go get</code>, go install</code>, etc... commands will be communicating with Google-run servers to fetch code. I don't mean to make that sound ominous, I'm not trying to throw any shade at Google, and Google didn't pull a fast one as you might think (or have read?) - this change was on the roadmap like all the others. So that's me softening the blow. If you're working in a private repository, this new thing sucks for you. By default, you'll be sending Google the names of your private modules, and your builds will break because the public Google proxy can't find your private modules. What's the solution? I'm glad you asked, friend! Here's how to fix it, in bold print for people who just skimmed the article, and with a shameless plug for Athens: If you have a private codebase, set your GOPROXY</code> environment variable to a private module proxy. Where do you get a private module proxy, you ask? Check out the biggest open source one if you're on the market: Athens</a> Ok, and there's one more thing you need to look at... Go now consults the checksum database</a> every time you ask for a new module</h1> This section's gonna be really familiar if you read everything in the last one. If you do a go get</code> (or similar) and the module isn't in your go.sum</code> file (yes, the checksum file), then the Go tool is gonna ask sum.golang.org</code> -- the checksum database -- what the checksum is supposed to be. That's to stop a proxy from giving you the wrong stuff the first time you ask for it. Since they're using their proxy by default, they're actually protecting you from ... themselves. The Go team is the hero Gotham deserves, not the hero it needs. But seriously, the idea behind the sum database is good - it helps you not get corrupted or tampered-with code, whichever proxy you decide to use. It's a really good tool to encourage a secure, diverse, and decentralized ecosystem. The problem is you pesky private repositories! The same send-your-data-to-Google bug is gonna bite you here too. There are two things you can do about it, and I'm not gonna bold them because I already let you lazy readers off the hook last section, dammit! Set GONOSUMDB</code> to have a comma-separated-list of module prefixes you don't want Go to send to the sum DB. For example, export GONOSUMDB="myvcs.internal/*,secretevilplans.com/modules/*"</code></li> Use Athens (shameless plug #2) to set a global GONOSUMDB</code> variable for your team. This will prevent anyone on the team who is misconfigured from spilling the beans to the Goog</li> </ol> So that's it for today. Make sure you go understand what's up with the Google proxy</a> and Google Checksum DB</a> and how they affect your code. If you have private repositories, really understand them. Your builds will break out of the box if you don't. Join Me For The Second Athens Office Hours Unknown — Thu, 27 Jun 2019 15:07:03 -0700 Athenians! First of all, thanks to everyone who came to the first office hours</a>. I think it was a success, and we're going to keep going. When</h1> Like last time, we're going to vote for the time that works best for the most people. Please vote for the time that works for you at this link: https://doodle.com/poll/tiypx72hs6ttmezq. </blockquote> What</h1> Last time, we went through a few things. First, we looked at the configuration system</a> internals. Then, we started to look at how the storage system</a> code uses the configuration to set up the appropriate storage backend. So, this time we're going to start where we left off! We started talking about how the MongoDB storage backend</a> works, so we're going to look at that part of the codebase in more detail. Other Logistics</h1> We'll be doing the meeting on the same Zoom</a> room as last time. Here's the link: https://arschles.com/zoom. Finally, as always, absolutely everybody is welcome. Anyone is free to join, listen, ask questions, or otherwise participate. I will always do my best to include anyone who wants to be involved with the Athens community in any way. Hope to see you there! Athens Office Hours! Unknown — Tue, 25 Jun 2019 13:37:55 -0700 Hey Gophers! If you don't know what Athens is, you should totally go check it out</a>. The quick overview is that it's a proxy server for Go modules, and an interesting open source project with a community full of supportive Go programmers that help, encourage, and welcome contributions from anyone interested (we make that a priority!). I'm starting an Athens office hours session. Absolutely anyone interested in Athens is welcome to join. The first session will be this Thursday or Friday (June 27 or 28). Please vote on the time(s) that work for you at this link</a>. </blockquote> Whether you're a seasoned Go programmer, a newcomer interested in getting involved with a Go open source project, or even someone brand new to open source -- we have contributors who fall into all these categories and more -- I want to make the project even more accessible for you. That's why I'm starting this Athens office hours session. This session is a place for learning and discussing different pieces of the codebase and architecture of Athens. And anyone who is part or the project or interested in the project can join. Details</h1> The office hours will be held on a Zoom</a> video chat at this link: https://arschles.com/zoom. Here's the format of the first office hours: I'll choose a section of the codebase to start looking at</li> I'll introduce the achitecture of this section at a high level</li> Anyone attending can ask questions, at any time</li> We'll navigate the codebase as the questions lead us (not limited to the area I chose!)</li> I am not the “expert”. I happen to know a little bit about a lot of the codebase, but hopefully others will attend who have a lot more in-depth knowledge about individual pieces. There’s power in numbers!</li> </ul> I intend to make these office hours recurring. The format might change in future office hours sessions if needed, but the only thing that won't change is that absolutely everybody is welcome. Time and Date</h1> If you're interested in coming to the first office hours session, please vote on the time(s) that works for you at the below link: https://doodle.com/poll/bwy8tiaxh3ehb5g6 Functional Programming in Go With dcode Unknown — Mon, 11 Mar 2019 14:06:10 -0700 Back in 2017, I gave a talk</a> at GopherCon about doing functional programming with Go. I went up on stage and talked about some FP basics like Functor</code>s, higher-order functions and so on. I heard some feedback that doing FP in Go isn't practical until Go has generics (fret not, folks, contracts</a> are on the table!). It's true that contracts will be a huge boon for functional Go code, but there's lots of functional programming we can do in Go without generics. You don't need generics to do FP in Go. dcode</h1> When I started writing that talk in 2017, I created the github.com/go-functional</a> GitHub organization to hold experimental projects for doing FP in Go. Beyond the code I showed on the slides, which is in core</a>, I've been prototyping some other functional libraries like quicktest</a> and pure functional SQL query builders</a>. The one area that I really latched onto was JSON, and that's how dcode</code> was born. Inspiration</h2> A long time ago, I built web services using Scala and lift-json</a>. That library provides a pure functional tree representation for a JSON object. It's also relatively fast for deep traversals. More recently, I found the Elm language</a>, and particularly its JSON parser</a>. This parser interface is a concise DSL that lets you say you you want to traverse a JSON object. It's incredibly self-documenting, even when dealing with large JSON objects. In fact, the DSL tends to look similar to JSONPath</a>. The Elm API makes the lift-json tree representation accessible to anyone, without requiring them to write their own tree traversal code. Usage</h2> You write one line of code to traverse any number of levels into a JSON object as necessary. For example, if you have this JSON (adapted from the JSONPath page): { "firstName": "John", "lastName" : "doe", "age" : 26, "address" : { "streetAddress": "naist street", "city" : "Nara", "postalCode" : "630-0192", "accessInstructions": { "gate": true, "message": "Please dial 0123 so I can buzz you into the gate" } } } </code></pre> And you want to get to $.address.accessInstructions.message</code> (that is, the value of the message</code> field inside the address access instructions object), you'd write this code to crate a decoder that, when it's called, will try to pull that value out of any JSON you give it. decoder := Field( "address", Field("accessInstructions"), Field("message", String()), ) </code></pre> There's also an accumulator interface so you don't have to nest all those calls to Field</code>: decoder := First("address").Then("accessInstructions").Then("message").Into(String()) </code></pre> Why?</h1> dcode</code> obviously doesn't fit every use case. If you're expecting to decode a big JSON object with lots of keys, and the incoming JSON value is the same every time, you'll most likely be more successful writing a struct</code> and using encoding/json</code></a> to decode into it (although dcode</code> does have functionality to decode into struct</code>s). This library does well, however, in a few cases: You need to traverse deep JSON objects to fetch values. This use case is common with programs that need to use a small part of a large JSON API</li> You don't know the "shape" of the object ahead of time. Often, you'll want to try several different decoding strategies at a time. This use case is common with web applications that deal with legacy API clients</li> </ul> Since dcode</code> is based on functional programming principles, it also comes with two helpful patterns: Composable</h4> Each Decoder</code> that you construct (with Field</code> or First()...Into()</code>) is a stateless function that is approximately: func (d *Decoder) Decode(b []byte, i interface{}) error </code></pre> Note: I've written this function here as an example. It doesn't exist in the library, but you can use the top-level Decode</code> function</a> instead to get the same result. </blockquote> Of course, this function signature looks a lot like (encoding/json).Unmarshal</code></a>, which is nice for familiarity. More importantly, these Decode</code> functions are built up by composing other Decode</code> functions together. The Field</code> or First()...Into()</code> interfaces both enable that composition. This API makes it obvious how to structure your JSON decoding logic, and trivial to refactor it. It also really encourages the same kind of composibility as Decoder</code>s "leak" into other parts of your codebase Bonus: dcode</code> itself takes heavy advantage of composition. Check out Map</code></a> and OneOf</code></a> for examples. </blockquote> Loosely Coupled</h4> Decoder</code>s are instructions for decoding a JSON object, and they can exist alone, without the values that the JSON should be decoded into. This lets you pass Decoder</code>s to any function so that it can decode any JSON into any value as it sees fit. In the future, we may consider adding functionality to serialize and deserialize Decoder</code>s themselves, so you can pass them over the network. To contrast, if you use encoding/json</code> to decode JSON, you usually create a struct</code> to define the "shape" of the JSON. This API requires that the values (struct fields) and the decoding instructions (the reflected field names, or struct tags) are tightly coupled. Get Started!</h1> Go check out the README</a> for more on how and why to use dcode</code>. Of course this is a work in progress, so if you find something missing, wrong, or just plain broken, please don't be shy and file an issue</a>! Keep on rockin', Gophers GopherCon 2019 CFP OMG! Unknown — Thu, 10 Jan 2019 14:23:56 -0800 Hey Gophers! I want you to submit a proposal</a> to talk at GopherCon 2019</a>. ... and I want you to start now because the Call for Proposals is only open until January 31st! Submit a proposal, you say? Let's talk a little bit about that. But I Could Never Speak At GopherCon!</h1> Yes, you could. All those "superstars" you see up on the stage? They submitted talks through the same form</a> as you do. You can submit, and you can get accepted. In fact, we love new speakers. All things being equal, we tend to prefer new-to-GopherCon speakers over speakers who have already spoken. But I Might Not Get Accepted :(</h1> We really want to see your proposal, but it's true, you could get rejected. We get many more than spaces available in the program, and we unfortunately have to turn down some awesome proposals. Not all is lost if you don't get accepted to this conference, though! A proposal isn't a one-and-done, and if it's not accepted to GopherCon 2019, it's far from wasted. I would encourage you to use it as inspiration for other conferences. There are tons of others just for Go</a> alone! Personally, I submit my rejected proposals - and there are a lot of them! - to other conferences, or use them as inspiration for new proposals. Ok, How Can I Make My Proposal Better?</h1> There are a lot of articles with tips and tricks for writing a great proposal. @carolynvs</a> - who is on this year's program committee with me - has some great tips tailored for this year's conference in her post</a>. There are also tons of other posts on how to write</a> great</a> proposals</a>. There's no single foolproof formula for writing the "perfect" proposal, but the selection criteria on the CFP page</a> lay out generally what we're looking for. I'll add a little bit of detail for each item. "Relevance"</h2> We look for talks that are related to Go. People come to GopherCon to learn about new stuff in the Go ecosystem, pick up new skills, and generally learn about Go and the community. If you're writing a proposal that you couldn't write about any other language, technology or community, you're on the right track. Other the other hand, if you're writing a proposal where you could swap out Go for another language, it probably won't be accepted. If you're writing a proposal that's not related to programming or technology, it almost certainly won't be accepted. "Clarity"</h2> Show us you've thought through what you want to say in your talk. We want to see it in clear, detailed writing. I want to really stress this part. Write down everything you plan to talk about. As long as it's clear writing, we want to read it. I personally love it when folks write a rough outline of their talk in the proposal. You can do this in the "notes" section of the CFP. As it says in the CFP page</a>, no proposal has ever been rejected because it was too long. "Correctness"</h2> In order for us to accept your proposal, we have to know that you know what you're talking about. So tell us that you do! It's easy to talk yourself out of submitting because you don't feel like you're an expert. I do it all the time. Here's what helps me get over that feeling every time: We're not looking for experts, we're looking for experience. Experience can mean a lot of things. Here are two experiences that are equally valid and valuable for GopherCon: "Writing a scalable system to handle 1,000,000 requests per second at a large company"</li> "How I learned Go as a Javascript developer"</li> </ul> So tell us clearly what your experience is and how it applies to your talk. Sometimes your experience will be easy to write about in your talk description, but if it doesn't fit there, make sure to tell us about it in the notes and/or bio section of the CFP. "Achievability"</h2> Sometimes I want to talk about everything in my talk. I get so excited about some technology that I want to go into 3 or 4 big things in one talk. But those proposals never get accepted. I said above that we're looking for clarity and detail, and that means your proposal needs to be focused on a specific topic. The elevator pitch section on the CFP requires that you tell us what you're going to talk about in 300 characters or less, so if you can't clearly summarize in that space, the talk is too long. If your proposal is too broad, it's great opportunity to split it up into multiple different talks! "Impact"</h2> This is the absolute most important part for me. Tell us what the audience will get out of your talk. A good proposal will usually have something like the following at the end of the description: "the audience will walk away knowing ..." Also, remember that you're not just telling the reviewers what the audience will get, you're telling the audience why they should come to your talk. You're selling both us and the audience in your talk description. That's It!</h1> If you feel like you have something to talk about, I really hope you submit a proposal</a>. I can't personally help folks with their proposals this year, but if you'd like help, please ask folks in the Gophers Slack</a> for help. Good luck!! Athens and Microsoft Unknown — Thu, 20 Dec 2018 17:45:40 -0800 As you might have read, Google recently announced</a> some of their plans for Go modules in 2019. And because I'm obsessed with module proxies/registries/repositories (more on the naming of these things in a future post), I paid special attention to the announcements about the services they're launching in 2019. I'm happy that Google is stepping up in a big way to help dependency management in the Go ecosystem. I'll be writing more in future posts about how Athens works with this new stuff, but I'm focusing here on the Athens community. Obviously none of the opinions in here necessarily reflect the policies or opinions of Microsoft (my employer). Athens is a Community Project</h1> The Go team mentioned the Athens project very briefly in their post. I was obviously disappointed that we weren't mentioned a little more, but it's their announcement and their blog, so I respect their choice. I was, however, very disappointed in how they referred to the project as "Microsoft's Athens project." Since that post came out, I've seen and personally gotten lots of questions on what they meant by that. I regret that I have to write this article, but the Athens community means more than anything else to me. Because the Go team's post didn't acknowledge that the vast majority of amazing Athenians are not Microsoft engineers, I'd like to clear that up here. Quick note: I talk to lots of folks on the Go team regularly and I like and respect them. I'm not trying to call any one of them out here, instead I want to set the record straight on Athens and to acknowledge the amazing community we have. </blockquote> Athens is a Community Project, Not Microsoft's</h1> I wrote the first prototype of Athens in my free time. After I moved it out of my personal organization into the gomods</code> org, I focused on growing a diverse open source community outside of Microsoft to work on Athens. The gomods</code> org is administered by everyone on the Athens maintainers team</a>. Microsoft pays me now in part to work on Athens (I have other unrelated responsibilities as well), and I'm empowered to keep working on Athens and help grow the community as I have been. There's nothing proprietary in Athens and it's MIT licensed. Both me and lots of other folks involved use Athens (or have plans to) and contribute bugfixes and features upstream to gomods/athens</code>. Everything with the project happens on Github, in the open. This is the part that I absolutely want you to read. If you want more details, read on. More Details</h1> At the moment, me, @bketelsen</a> and @carolynvs</a> are the 3 folks from Microsoft on the core maintainers team</a> (there are 8 total), and the MS maintainers don't have special voting rights or influence over the project. We also have clear guidelines</a> on how anyone can advance from community member to contributor to maintainer, and there's no language in that document that indicates that you have to work at a specific company or organization to advance. We have a track record on both the contributors and maintainers teams to prove that as well. One of our contributors, chriscoffee</a>, submitted an issue</a> two days ago to golang/go</code> asking that the "Microsoft" be removed from "Microsoft's Athens Project" on the blog. There's been a little bit of movement on it since then. I hope that until Google removes that language from their post, this post will be enough to show that Athens is not Microsoft's project. It is the Go community's project, and Microsoft engineers contribute (some of them get paid to contribute) to the project. More to Come</h1> I'll be writing more about the technical aspects of Athens and specifically how it fits into the awesome new module-ey stuff in the community, particularly JFrog's GoCenter and of course the three new Google announced technologies. Give Away Your OSS Power Unknown — Wed, 31 Oct 2018 11:25:58 -0700 Pretend you're an OSS maintainer - how do you figure out if your project is doing well? Some ideas: Number of contributors</li> Forks</li> Stars</li> PRs closed</li> Average issues opened per day</li> Twitter followers to your project account</li> Views to your project website</li> All of the above?</li> </ul> And the list goes on... The Bus Factor</h1> There's one more big one to add to the list. How many people have to drop out before the project is totally screwed and how hard would those people be to replace? In other words, what is your project's bus factor</a>? That should be the first measure of any project's success and health, because everything else is a vanity number if if can all go away in the blink of an eye. I learned this lesson with Athens</a> sorta recently when I had to drop out of the project for a week. I had to go to the hospital and get my appendix removed on Monday, and didn't start doing work again til Friday. I wrote on the THAT Conference blog</a> how I learned to step away from a project, and how valuable that can be for everyone. But I also learned how crucial it is to not hold all the power on the project. Early on with Athens, I wanted to set a precedent that I would not be the one with all the power. That meant I wasn't the only one with the login to a thing, and also I wasn't the only decision maker. I was lucky that the precedent stuck early, and we haven't lost it. When we get a new core maintainer, they immediately get the keys to everything: On the "maintainers" Github team in the gomods</code> org</li> Administrative privilege for the gomods</code> org</li> Admin on our Dockerhub org</li> Admin on our Netlify account (for our docs site)</li> </ul> There's another big thing, too. All the other maintainers work really hard to make sure they actually start making decisions in public. It's really important that they and everyone else feel comfortable with that. If we didn't have that process, I would have probably come back from the hospital to a bunch of stagnant PRs and smart folks just stuck waiting to make progress. Athens Running on Zeit Unknown — Wed, 24 Oct 2018 12:31:07 -0700 I've been a long time admirer-er of Zeit</a>. Besides the technical part (which is really good), their platform gives people really easy, understandable and (most importantly) trustworthy tools to get a real app into production. I've wanted something like this to exist for Kubernetes for a while (I even wrote</a> about that, but I digress.) The platform was for just Node applications for a while, but containers are a thing y'all, and Zeit lets you deploy them to its platform now (they announced it mid August</a>.) It does all the same stuff as it does for Node, but with Docker: Write a Dockerfile</code></li> Run now</code></li> Profit</li> </ol> There's a ton going on there in step 2, but the basics: Push the Dockerfile</code> and context up to Zeit</li> Zeit does the docker build</code> on their infrastructure</li> Zeit pushes the image to their registry (I assume they run an internal registry but no idea really)</li> Zeit pulls the image into their serving infrastructure, configures it, does routing, checks if you paid and gives you all the amazing hardware if you have and otherwise runs you on a RaspberryPi</del> and runs your image on their servers</li> </ul> I mean, I've blogged about ACR Builds</a> before, but this is next level. One command gives you everything. Athens-ing it Up</h1> There's this rad project</a> I know about. Something about Go and dependencies IDK but it has a Dockerfile</code> and you can do cool stuff with it. So why not see if Zeit can handle this beast?? I had to tweak the Dockerfile a little bit because the CLI (now</code>) doesn't let you specify custom Dockerfile</code> locations (I don't think???) but that was pretty much it (check out the PR</a>.) Now I can do this: ➜ athens git:(zeit-now) now --docker > Deploying ~/github/athens under arschles > Your deployment's code and logs will be publicly accessible because you are subscribed to the OSS plan. > NOTE: You can use `now --public` or upgrade your plan (https://zeit.co/account/plan) to skip this prompt > https://athens-ukiespgiyz.now.sh [in clipboard] (sfo1) [5s] > Building… > Sending build context to Docker daemon 37.83MB > Step 1/16 : FROM golang:1.11-alpine AS builder > ---> 95ec94706ff6 > Step 2/16 : RUN mkdir /proj > ---> Using cache > ---> 7690d95ac3b4 > Step 3/16 : WORKDIR /proj > ---> Using cache > ---> 89a1e2d291ec > Step 4/16 : COPY . . > ---> 6fdf77ac18ec > Step 5/16 : ENV GO111MODULE=on > ---> Running in 61601842b526 > Removing intermediate container 61601842b526 > ---> 75be7b20b225 > Step 6/16 : ENV GOPROXY=https://microsoftgoproxy.azurewebsites.net > ---> Running in 8e92f2e31291 > Removing intermediate container 8e92f2e31291 > ---> 8f7e12c7fef4 > Step 7/16 : RUN GO111MODULE=on CGO_ENABLED=0 go build -o /bin/athens-proxy ./cmd/proxy > ---> Running in b25589823d39 > go: finding github.com/fatih/color v1.7.0 > go: finding github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b > go: finding github.com/minio/minio-go v6.0.5+incompatible > go: finding github.com/gobuffalo/mw-csrf v0.0.0-20180802151833-446ff26e108b > go: finding github.com/tinylib/msgp v1.0.2 [SNIP] > go: downloading github.com/markbates/goth v1.46.0 > go: downloading github.com/rs/cors v1.5.0 > go: downloading github.com/gobuffalo/buffalo v0.13.1 > go: downloading github.com/unrolled/secure v0.0.0-20181005190816-ff9db2ff917f > go: downloading github.com/mitchellh/go-homedir v1.0.0 [SNIP] > Removing intermediate container b25589823d39 > ---> d61630e38cd9 > Step 8/16 : FROM alpine > ---> 196d12cf6ab1 > Step 9/16 : ENV GO111MODULE=on > ---> Using cache > ---> cd147951c9a7 > Step 10/16 : COPY --from=builder /bin/athens-proxy /bin/athens-proxy > ---> Using cache > ---> a462fc7bd78b > Step 11/16 : COPY --from=builder /proj/config.dev.toml /config/config.toml > ---> 3c99d68507fe > Step 12/16 : COPY --from=builder /usr/local/go/bin/go /bin/go > ---> bcac2f09281a > Step 13/16 : RUN apk update && apk add --no-cache bzr git mercurial openssh-client subversion procps fossil && mkdir -p /usr/local/go > ---> Running in 49fd4fa48969 > fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz > fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz > v3.8.1-38-g898a0bb28a [http://dl-cdn.alpinelinux.org/alpine/v3.8/main] > v3.8.1-35-ga062ffc9e8 [http://dl-cdn.alpinelinux.org/alpine/v3.8/community] > OK: 9539 distinct packages available > fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz > fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz > (1/33) Installing libbz2 (1.0.6-r6) > (2/33) Installing expat (2.2.5-r0) > (3/33) Installing libffi (3.2.1-r4) > (4/33) Installing gdbm (1.13-r1) > (5/33) Installing ncurses-terminfo-base (6.1_p20180818-r1) [SNIP] > Executing subversion-1.10.0-r0.pre-install > Executing busybox-1.28.4-r1.trigger > Executing ca-certificates-20171114-r3.trigger > OK: 123 MiB in 46 packages > Removing intermediate container 49fd4fa48969 > ---> e8a15877e355 > Step 14/16 : ENV GO_ENV=production > ---> Running in ac91ac08096a > Removing intermediate container ac91ac08096a > ---> 096f14de684e > Step 15/16 : EXPOSE 3000 > ---> Running in eb60ce775db6 > Removing intermediate container eb60ce775db6 > ---> 0075a1cdb3c5 > Step 16/16 : CMD ["athens-proxy", "-config_file=/config/config.toml"] > ---> Running in 33e543286967 > Removing intermediate container 33e543286967 > ---> 211b19e9d4a4 > Successfully built 211b19e9d4a4 > Successfully tagged build:RWio8EYjdNbuK5HXd31rKHAp_1540411816 > ▲ Assembling image > ▲ Storing image (64.0M) > Build completed > Verifying instantiation in sfo1 > [0] buffalo: Unless you set SESSION_SECRET env variable, your session storage is not protected! > [0] time="2018-10-24T20:12:29Z" level=info msg="Exporter not specified. Traces won't be exported" > [0] buffalo: Starting application at :3000 > ✔ Scaled 1 instance in sfo1 [14s] > Success! Deployment ready </code></pre> That's using a multi stage build</a> to build Athens inside a Dockerfile</code>, on Zeit's infrastructure. And I inception-ed it because I used a hosted Athens module proxy</a> to build Athens itself inside the build, on Zeit's infrastructure. Here's what it looks like after it's deployed: ➜ athens git:(zeit-now) now list > 5 total deployments found under arschles [314ms] > To list more deployments for an app run `now ls [app]` app url inst # type state age athens athens-ukiespgiyz.now.sh - DOCKER READY 4m </code></pre> There's a UI for all this on the zeit.co site too. They even stream build logs to the site too #AMAZE Trying it Out</h1> So basically I ran a few commands and got a ton of output on the CLI. Machines are doing my bidding; success! Now that it's running, I wanted to see how it performed. I started small with bld</a> (blog post</a> for background on that tool if you're interested): ➜ bld git:(master) export GOPROXY=https://athens-ukiespgiyz.now.sh ➜ bld git:(master) sudo rm -r $GOPATH/pkg/mod ➜ bld git:(master) time go build go: finding github.com/magefile/mage v1.4.0 go: finding github.com/tdewolff/parse v2.3.3+incompatible go: finding github.com/spf13/cobra v0.0.3 go: finding github.com/mitchellh/go-homedir v1.0.0 go: finding github.com/mattn/go-isatty v0.0.3 go: finding github.com/spf13/viper v1.2.1 go: finding github.com/dlclark/regexp2 v1.1.6 go: finding github.com/tdewolff/minify v2.3.5+incompatible go: finding github.com/spf13/pflag v1.0.3 go: finding github.com/magiconair/properties v1.8.0 go: finding gopkg.in/yaml.v2 v2.2.1 go: finding github.com/fsnotify/fsnotify v1.4.7 go: finding golang.org/x/text v0.3.0 go: finding golang.org/x/sys v0.0.0-20180906133057-8cf3aee42992 go: finding github.com/pelletier/go-toml v1.2.0 go: finding github.com/spf13/jwalterweatherman v1.0.0 go: finding github.com/mitchellh/mapstructure v1.0.0 go: finding github.com/spf13/pflag v1.0.2 go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 go: finding github.com/spf13/afero v1.1.2 go: finding github.com/spf13/cast v1.2.0 go: finding github.com/hashicorp/hcl v1.0.0 go: finding github.com/davecgh/go-spew v1.1.1 go: downloading github.com/spf13/viper v1.2.1 go: downloading github.com/spf13/cobra v0.0.3 go: downloading github.com/mitchellh/go-homedir v1.0.0 go: downloading github.com/magefile/mage v1.4.0 go: downloading github.com/mitchellh/mapstructure v1.0.0 go: downloading github.com/hashicorp/hcl v1.0.0 go: downloading github.com/pelletier/go-toml v1.2.0 go: downloading gopkg.in/yaml.v2 v2.2.1 go: downloading github.com/spf13/jwalterweatherman v1.0.0 go: downloading github.com/magiconair/properties v1.8.0 go: downloading github.com/fsnotify/fsnotify v1.4.7 go: downloading github.com/spf13/pflag v1.0.3 go: downloading github.com/spf13/cast v1.2.0 go: downloading github.com/spf13/afero v1.1.2 go: downloading golang.org/x/sys v0.0.0-20180906133057-8cf3aee42992 go: downloading golang.org/x/text v0.3.0 go build 4.97s user 2.06s system 7% cpu 1:32.16 total </code></pre> So, that kinda took a while. Building Athens itself took forever as well (about 6:30), but there are a few reasons for that, and ways to get big ole speedups from experience hosting Athens: Run this with more than one instance</li> Hook this up to external storage You'd need to do this for #1 to work anyway</li> This is using local disk and I have no idea what or how fast that is on Zeit</li> </ul> </li> Pre-seed the Athens cache with things Everything I showed was from a cold local ($GOPATH/pkg/mod</code>) and hosted (on the Zeit Athens server) cache</li> </ul> </li> </ol> All in all, this is a pretty rad platform. I give it pretty rad. Not mega rad - that's reserved for clippy</a> level things, but wayyy higher than semi-rad. Zeit: not quite clippy level, but pretty darn close!</div> Keep on rockin' everybody! Absolutely Everybody is Welcome Unknown — Tue, 23 Oct 2018 20:53:21 +0000 I started the Go Study Group</a> and Athens</a> both about 6 months ago. I did each for super different reasons, but I wanted both to have strong communities. The people are what keeps these communities alive. Not me, not the technology, nothing else. It's the people. And that's why I'm so insistent on strong communities. I've been part of lots of OSS communities - as a leader and as a member - and each one has either lifted me up or ground me down. I'm at a point in my career now where I can tell pretty quickly which type of community it's gonna be for me. Like after reading a few issues or doing a PR or two. So with that experience, here's what a strong community is to me: We're all nice to each other</li> We welcome everybody who wants to join us</li> Nobody dictates</li> We respect each others' time and ideas</li> We make it easy to get started and easy to keep going</li> </ul> I went into detail on a lot of this in the Athens project with a "philosophy document"</a>. Inclusion</h1> There's no way I can sum up all those things in that list above, but I want to be able to say something to newcomers to whatever thing to basically say: "you're welcome here, and we'll do everything we can to make it nice here." So I say this: "absolutely everybody is welcome" I don't have some grand story of how I came up with this catch phrase or anything, I just feel like it sums up that list above (by the way I know it doesn't technically come close :P). I use this all the time to say: "Hey folks! You're welcome to join us. We're inclusive, but we're more too. Someone is gonna be there to greet you and help you. You're not gonna be judged. You're not gonna have to run some huge gauntlet just to get started or prove yourself. We're happy you joined and you're not different than us. We have work to do and we want you to do it with us." If you want to join either/both of these communities, here's how: Go Study Group: we're a weekly group of Gophers who get together and chat about Go. Meeting invite is here: https://gophersource.com/study-group</a></li> Athens: A module proxy for Go's dependency management. More information on getting involved is here: https://docs.gomods.io/contributing/community/participating/</a></li> </ul> az acr build or docker build? Unknown — Mon, 08 Oct 2018 16:19:36 -0700 I had heard about acr build</code></a> when it came out and didn't pay much attention to it. I was like "I don't wanna send my whole build context over the internet that's crazy." That was kinda a facepalm in retrospect because it's kinda sorta my job to keep up on Azure things that are container related. Anyway, now that I actually looked at this thing, I have some feels and codes! How I Un-Facepalmed Myself</h1> I was looking at Brian</a>'s acrtasks repo</a> the other day and really learned what Azure Container Registry tasks are about. You run acr build</code> and the CLI zips up your local build context, sends it to the server, and that does the docker build</code> for you. Not that exciting, but as Brian said, it'll automatically re-build when base images change. Rad! Here's some more rad things that I like even better than that: I don't need a local Docker daemon!</li> I don't have to push image layers up to a registry</li> I don't have to do registry auth - just auth to Azure</li> I don't need a local Docker daemon</li> I don't need a local Docker daemon :P</li> </ol> Basically, I can just write my Dockerfile</code> and run acr build</code> without setting up the daemon. I'd want to on my dev machine, but I don't have to figure it out in other places: CI/CD systems like Buildkite</a> or Travis</a>. Travis has Docker support but you have to spin up a whole VM. You can run Buildkite agents on Kubernetes</a>, but talking to the Docker daemon from inside a pod is crazytown (I'll die on that hill, btw :P) </li> I repeat: don't let pods talk to the external Docker daemon. DinD</a> isn't much better on Kubernetes</li> Underpowered systems - Ever try building Buffalo</a> on a Raspberry Pi? Not fast, kinda painful</li> </ul> Basically, I just feel like not having to run a Docker daemon sometimes is one less thing to worry about. And it's a big thing. Enter bld</code></a></h1> Getting real again, I basically wanted to do docker build</code>s on my dev machines and az acr build</code>s playing with Buildkite and stuff. I pretty quickly got tired of writing the same bash script to decide which one to use. So I wrote this little, super basic tool that just does the if</code> for me. Basically you do this: bld -t my/image . </code></pre> It does docker build</code> if the Docker CLI is on the system, otherwise does az acr build</code>. It's suuuper basic and that's kinda the point (at least right now). You decide whether you want docker build</code> or az acr build</code> based on whether you have docker</code> (the CLI) installed. Other Stuff Already Exists</h2> Yup, and honestly lots of it is better and I might steal it :P The coolest thing I know of is part of draft</code></a>. The rad Matt</a> saw this and DM'd me with this code</a> in the project. The basic difference between bld</code> and draft</code> is that the former is just for building Docker images. The latter is for everything from coding to deploying to Kubernetes. By the way, Draft is rad and you should check it out :) I want to keep this thing simple and I'm not sure whether to shamelessly steal</del> use that code or just keep relying on az</code>. To be determined</a> I guess! Keep on rockin' Emerging Languages Unknown — Fri, 05 Oct 2018 12:01:16 -0700 My history with programming languages and frameworks is approximately this: I started my career writing PHP 3</li> Then I wrote C/C++ (Boost</a> forever!)</li> Then I discovered Java, Scala, Objective-C, Perl, Python, Clojure and Haskell all at once</li> Then I ditched it all for Go</li> </ul> Oh, and throw some javascript in there. I've been pretty focused on Go for a while and was missing learning new languages. Brian Ketelsen</a> introduced me to Nim</a> and then I wanted more. So between my curiousity and his endless list of cool new tech, I've been checking out a few languages. Here they are, along with my summaries of them: Elixir</a> - Ruby with less monkey patching but more macros</li> Rust</a> - C++ with more bullets to shoot yourself in the foot with, but less foot</li> Crystal</a> - Compile Ruby to machine code. And now you can use all the cores!</li> Ballerina</a> - Kinda like Metaparticle</a>, except it's a language</li> Swift</a> - Objective C and Scala had a baby, and you can compile it on the LLVM</li> </ul> Ok, I know it seems like I'm pretty sarcastic about all these languages, but I'm legitimately excited about all of these! I'm gonna play around with them for a while until I find something real to build.

Aaron Schlesinger's Personal Site

What is Hyperlight (part II)?

What is Hyperlight?

Built on the shoulders of giants</h2> Hyperlight borrows from, and builds on, substantial prior art, and it’s important to not only credit the technologies on top of which we’ve built this project, but also to establish a common foundation before we delve deeper into the project itself.</p>

Calling Rust from C#, and back again: Part 2

Pieces of the puzzle</h2> The process to get this callback system working is three-fold, as follows:</p> Convert a C# function (usually called a "delegate") into a C function pointer</li> Consume that C function pointer in Rust and call it</li>

Functions as parameters</h4> Armed with this ability to define C functions in Rust, we can now create higher-order functions as follows:</p>

Cleaning everything up</h2> Cleaning up Rust-owned function pointers generated from C# can be a complex topic that requires some care.</p>

Calling Rust from C#, and back again: Part 1

Creating C APIs in Rust</h3> Rust has strong support for creating C-compatible APIs [2]. While there are a few gotchas, if you can compile Rust code like the following, it will be callable by C code:</p>

Synchronizing the KEDA HTTP Addon Request Routing Table Across Hundreds of Interceptor Pods

Fan-in and fan-out with Go

What are we doing here?</h2> First thing's first - we need a problem we can break down into a bunch of pieces. Sometimes it's called an "embarassingly parallel" problem. </p>

GopherCon 2021 CFP OMG!

How (and What) You Should Submit</h2> So, given how we work, what should you keep in mind while submitting?</p>

Correctness</h3> We only accept proposals if the speaker clearly has knowledge of the topic they are proposing. We don't need you to be an expert, by any means, but we want you to at least be speaking from a place of experience. Remember that experience can have many meanings, including:</p>

Next Steps for KEDA HTTP

Deploying Apps With KEDA-HTTP

First Lines of Rust

async/await: under the hood

What It's About 🤔</h1> async</code>/await</code> is about writing concurrent code easily, but more importantly, it's about writing the code so it's easy to read</strong>.</p>

Callbacks 😭</h3> You've probably heard about callbacks in Javascript. If you haven't, they're a programming pattern that lets you schedule work to be done in the future, after something else finishes. Callbacks are also the foundation of what we're talking about here.</p>

How to Run Windows With a Mac Keyboard

Remap Keyboard 🗺</h2> This one is simple in theory. Like the heading says, you get to map a single key. Take the first one in there - I mapped Page Up to the home key (that one wasn't Mac specific - I just kept accidentally hitting it 😉).</p> Here are the keys I remapped to Mac-ify:</p>

Storing Data with Dapr (Dapr Series #3)

Windows, Now With More Toys

🍚 General</h4> Go here to do general stuff (😂), most importantly getting updates to PowerToys itself.</p> 🥧 FancyZones</h4> Automatically resize each window in different patterns on the screen, like side by side or in rows.</p>

🥧 FancyZones</h4> Automatically resize each window in different patterns on the screen, like side by side or in rows.</p>

🍣 File Explorer preview</h4> See rendered previews of SVGs and rendered Markdown in the file explorer. Optional second sentence 💰: I imagine there will be more formats in the future.</p>

🍤 Image resizer</h4> Resize any image with a right click from the file explorer.</p>

🍎Keyboard Manager</h4> Remap any key or keyboard shortcut from a nice UI.</p>

🍉 PowerRename</h4> Rename tons and tons of files by highlighting them all and right clicking.</p>

🍍 PowerToys run</h4> The belle of the ball for me - basically Alfred or Spotlight for Mac, but for Windows.</p>

🥦 Shortcut guide</h4> A full-screen pop-up that reminds you any time of the keyboard shortcuts you have set up.</p>

Athens on Azure Kubernetes Service... Part 2

The Plan</h2> In case you forgot the high level architecture from last post, here ya go!</p>

May 2020 Athens Update

Coming Soon to and Athens Near You</h2> I'm happy with things so far but I'm really excited for what's coming. For a while now, we've been thinking about how to build two major new features:</p> A completely-offline mode</li>

What is Dapr? (Dapr Series #2)

Production?</h3> Before we move on, let me take a second to explain what "production" is:</p>

So you want to build a computer

How to started</h2> Luckily I knew the basics of a computer, so I sketched out the pieces I need. If you're unfamiliar, here they are with crude definitions:</p>

What Happens to Your Career After 10 Years?

Lessons I still carry with me</h3> I keep having this experience where I run up against a job that would have me doing things I have no idea about and I still get scared.</p> All these experiences taught me two lessons I still carry with me:</p>

Built on the shoulders of giants</h2>
Hyperlight borrows from, and builds on, substantial prior art, and it’s important to not only credit the technologies on top of which we’ve built this project, but also to establish a common foundation before we delve deeper into the project itself.</p>

Cleaning everything up</h2>
Cleaning up Rust-owned function pointers generated from C# can be a complex topic that requires some care.</p>

How (and What) You Should Submit</h2>
So, given how we work, what should you keep in mind while submitting?</p>

What It's About 🤔</h1>
`async</code>/await</code> is about writing concurrent code easily, but more importantly, it's about writing the code so it's easy to read</strong>.</p>`

🍣 File Explorer preview</h4>
See rendered previews of SVGs and rendered Markdown in the file explorer. Optional second sentence 💰: I imagine there will be more formats in the future.</p>

🍤 Image resizer</h4>
Resize any image with a right click from the file explorer.</p>

🍎Keyboard Manager</h4>
Remap any key or keyboard shortcut from a nice UI.</p>

🍉 PowerRename</h4>
Rename tons and tons of files by highlighting them all and right clicking.</p>

🍍 PowerToys run</h4>
The belle of the ball for me - basically Alfred or Spotlight for Mac, but for Windows.</p>

🥦 Shortcut guide</h4>
A full-screen pop-up that reminds you any time of the keyboard shortcuts you have set up.</p>