Lambda: using AWS SAM Local to test Lambda functions locally on Windows

AWS SAM Local is a “CLI tool for local development and testing of Serverless applications.” It uses Docker to simulate a Lambda-like experience. The docs explain well how to get started, and the GitHub repo has lots of samples as well. As of this writing, it supports python, java, .net, and nodejs.

This is a quick post to show how to use it in Windows Subsystem for Linux (WSL) and Docker For Windows.

Installing on Windows

The instructions recommend installing with npm. That didn’t work for me, giving me errors about file not found. I’m not sure if this is a problem with npm inside of WSL, if it’s a problem with the current installer, or what.

I did get it installed by using the next option, which is go get github.com/awslabs/aws-sam-local. I then aliased that to sam: alias sam='aws-sam-local'

Running on Windows

Out of the box, using Docker for Windows as the Docker daemon, the sam command itself worked fine, but actually invoking a function did not work for me when using the docker client on Ubuntu within WSL. With a simple python hello-world example, I’d get this:

marc@ulysses:sam-local-play$ sam local invoke "HelloWorldFunction" -e event.json
2018/01/20 08:59:18 Successfully parsed template.yml
2018/01/20 08:59:18 Connected to Docker 1.35
...
Unable to import module 'main': No module named 'main'
END RequestId: c9edd13a-000e-49bd-a4a7-a8a23258a03b
REPORT RequestId: c9edd13a-000e-49bd-a4a7-a8a23258a03b Duration: 1 ms Billed Duration: 0 ms Memory Size: 0 MB Max Memory Used: 19 MB
{"errorMessage": "Unable to import module 'main'"}

In the instructions below, I’ll tie together several GitHub issues and a gist from three separate GH users.

First, Kivol, in this  GH issue comment from GH user, shows a 2-fold solution:

1. bind-mount /c to /mnt/c and then within Ubuntu make sure you’re on /c (from this comment on another GH issue from aseering)

2. pass --docker-volume-basedir in the sam invoke command

Here’s how to do all that (again, mostly copying from a few GH issues and adding some color commentary):

# bind-mount /c. this mount lasts as long as your current terminal session; instructions below for how to make this persistent if this works for you
$ sudo mkdir /c
$ sudo mount --bind /mnt/c /c

# now you can use /c instead of /mnt/c
$ cd /c/path/to/project $ sam local invoke --docker-volume-basedir $(pwd -P) --event event.json "HelloWorldFunction"

And voila, it worked. I get this beautiful output:

2018/01/20 12:16:08 Successfully parsed template.yml
2018/01/20 12:16:08 Connected to Docker 1.35
2018/01/20 12:16:08 Fetching lambci/lambda:python3.6 image for python3.6 runtime...
python3.6: Pulling from lambci/lambda
Digest: sha256:0682e157b34e18cf182b2aaffb501971c7a0c08c785f337629122b7de34e3945
Status: Image is up to date for lambci/lambda:python3.6
2018/01/20 12:16:09 Invoking main.handler (python3.6)
2018/01/20 12:16:09 Mounting /c/dev/projects/lambda-projects/sam-local-play as /var/task:ro inside runtime container
START RequestId: d7a9dcb0-751e-445d-9566-a025f4e804b0 Version: $LATEST
Loading function
value1 = value1
value2 = value2
value3 = value3
END RequestId: d7a9dcb0-751e-445d-9566-a025f4e804b0
REPORT RequestId: d7a9dcb0-751e-445d-9566-a025f4e804b0 Duration: 4 ms Billed Duration: 0 ms Memory Size: 0 MB Max Memory Used: 19 MB

Note: that cd /c/... is really important! If you do all the above but stay in your shell at /mnt/c — like I did 🙂 — it’s still not going to work.

Big thanks to GitHub users Kivol and aseering for putting this together!

Persistently mounting /c

If the above worked for you, then you’ll probably also want to persistently mount /c so that you don’t have to redo it every time you want to use sam from within Linux / WSL.

If you’ve got a Linux background, you’re thinking: just mount it in /etc/fstab. It seems that as of now, anyway, WSL isn’t loading /etc/fstab entries when you open a new shell (i.e. it seems as if you have to run mount -a every time), at least according to comments in this MSDN post and this WSL issue.

Fortunately, linked in those comments, sgtoj has a gist that sets all this up nicely. I saved this locally, ran it once, and now /c is mounted for all new Linux sessions.

Really?

Coming from doing all personal development on Linux for the past 3 or so years, these kinds of hacks are disappointing. So far in this experiment with going back to Windows, these hacks have been few, and so far for me have all been related to wanting to use a docker client inside of Linux / WSL. More on that in a future post.

Suffice to say: yeah, it’s hacky, but it’s not that bad. Annoying, sure, but certainly not enough to sully the overall experience so far in moving back to Windows. I can live with this one.

A note on aws-sam-local and Powershell

After initially failing to get aws-sam-local running successfully within WSL, I figured I’d try it out all on the Windows side of the house. I ran into problems there, too. First, using go-get to install it, I got path-too-long errors. WTF. That led me to choco install nodejs and use the npm install route. That did work, and sam invoke worked fine.

So far, this is the first time that working in WSL was kind of a shit-show. Thankfully smart people figured out how to get it working correctly. I am always grateful when I find answers in GitHub comments along the lines of “maybe sharing this will help.”

I really did not want to have to use Powershell for this. Not because I don’t like Powershell, but mostly because it kind of pierces the vale of the single development experience I’m trying to achieve, I guess. It’s a bit of a context switch to be doing most of the work in WSL, and then for this one thing, need to pop over to Powershell, which also means maintaining duplicate installs of software in Windows land and WSL land. A small thing, for sure, but I’d like to avoid it if possible.

 

Windows IPv6 slow-or-broken: resolved

Chocolatey slowness?

When I originally posted about package management with Chocolatey, I mentioned two problems I had on a brand new laptop: 1) inability to download large packages; 2) general slowness when downloading.

Turns out, these are not Chocolatey’s problems at all.

Wait… Powershell too?

I noticed when working in Powershell that curl, which just wraps Invoke-WebRequest, was taking a really long time. Simple commands were returning results like this:

PS C:\WINDOWS\system32> Measure-Command { curl https://microsoft.com }

Days : 0
Hours : 0
Minutes : 0
Seconds : 43
Milliseconds : 316
Ticks : 433164486
TotalDays : 0.000501347784722222
TotalHours : 0.0120323468333333
TotalMinutes : 0.72194081
TotalSeconds : 43.3164486
TotalMilliseconds : 43316.4486

It would consistently take about 43 seconds.

Resolution: my IPv6 settings

I narrowed it down to a problem on my network — either the main PC, router, or modem — by trying out the curl command above while connected to my phone’s WiFi hotspot instead of our home router.

I’ll spare the sleuthery for later and cut to the chase:  I resolved the problem two separate ways:

  1. by setting the IPv6 DNS settings in my wireless router (A D-Link DIR-880L); I did not stick with this
  2. ultimately, by finding the broken DNS setting in my primary PC’s IPv6 config. This is the one I went with, but the first might be instructive or useful, so:

In my router’s config, I simply set the Primary and Secondary DNS server settings to Google’s DNS

Once I did that, the IPv6 connection in my router went from Not connected to Connected. My IPv6 readiness score went from 0/10 to 10/10, Powershell curl commands returned to reasonable amounts of time, and Chocolatey was both fast and able to download large packages (300+MB with no problem).

However, I couldn’t let go of something… why would I need to do this anyway? It occurred to me that a very, very long time ago — maybe as far back as 2010, when I first bought our current home PC, I had monkeyed with a DNS setting for some reason or another related to my job at the time.

So I went into the IPv6 settings (here’s how), and sure enough in the DNS tab I had overridden the default with some mumbo-jumbo I’ve long forgotten about. I set it back to the default, undid the changes I had made in the router, and voila, same expected behavior with Powershell, Chocolatey, and IPv6 readiness.

The End. Stop here if you don’t care about how I figured this out.

Sleuthery for those who like stories about troubleshooting

Some people like stories about troubleshooting — I know I love reading them — so here’s the story of troubleshooting this problem.

It is admittedly crazy on first take: that a buried-kinda-deep TCP/IP setting on one computer would not affect that computer, but would affect another computer on the network.

When I noticed the problem with Chocolatey originally, I saw that it was timing out in Invoke-WebRequest. That’s what eventually led me to even be curious about the behavior of curl / Invoke-WebRequest in Powershell itself, outside of Chocolatey. It just took me a few weeks (and a day off of work) to make time to investigate.

I noticed that initial requests were taking about 43 seconds, and a second request would take the same time as well. Then, subsequent requests would complete quickly. After a few seconds of waiting, they’d return to 43 seconds. A very helpful person on the StackOverflow post I made provided a quickie Powershell script to gather some data, which confirmed the above behavior.

I tried curl in Ubuntu via Windows Subsystem for Linux, and it looked fine.

I disabled, and then uninstalled, Antivirus. No effect.

I then connected to my phone’s WiFi hotspot instead of our home wireless, and voila, everything worked fine.

Now that’s interesting. Sounds like a network problem. Let’s try to isolate the various components in the home network and pin one down as the culprit.

I went to our primary PC and tried an Invoke-WebRequest in Powershell, and it was fine, too.

So here’s where I was: 1 computer on the network worked fine, and 1 didn’t. What’s the difference?

Turns out, sometime within the past year, on the home PC, I needed to disable IPv6 for something related to Comcast email and Outlook, and I remembered that.

So on the laptop, I tried disabling IPv6 just as I had on the home PC, and that solved the problem on the laptop.

Fantastic.

Except… why? Why did that work? It made no damn sense. I hated not understanding that.

At first, I tried investigating the problem via the wireless router. I went into the configs and everything was at the default, but I also noticed that it was telling me that IPv6 was Not Connected. Weird. I tried monkeying with a few settings whose purpose I didn’t know, and that made no difference. I then returned it to the default setting and on a whim looked up Google’s DNS IPv6 servers and plugged them in.

After restarting the router, the router showed me Connected for IPv6, and back on the laptop I re-enabled IPv6 and everything seemed to work fine.

To be clear: changing DNS here wasn’t a momentary stroke of brilliance. Often, my troubleshooting strategy boils down to “I wonder what happens if I twist this knob,” and that’s what this was. In my line of work, I’ve seen DNS cause all manner of network goofiness, so this was kind of like “hmmm…. here’s an empty text field related to DNS, and since DNS can act-a-fool, I wonder what happens if I put something legit in it.”

Fantastic-er. Can I be done now?

Why? I mean, sure, it worked, but why would I need to override DNS in the router?  That, also, just didn’t sit right. And I could not let it go.

The only thing I could think of at this point was that there was something wonky about the home PC’s IPv6 settings, and overriding the DNS settings in the router was working around that wonkiness.

I then went into the home PC’s IPv6 settings, and everything looked default.

Clicked “Advanced”, and then “DNS”, and that’s when I saw it: There was a radio button checked and then it had a text box with DNS entries that I have no recollection of setting, and one of the FQDN’s was the internal domain for one of my old jobs. WTF?

So then I returned that setting back to the default, undid the router DNS change, and success!

Though… there’s something else: I’ve had Mac, Linux, Android, iOS, and Windows devices using this network for years and none have hit this problem. Just Powershell. Curious, don’t you think?

Finally, if you’re looking for more of these troubleshooting stories, here’s one from a while back on The Curious Case of the Slow Jenkins Job.

Goin’ back to Windows: Windows Subsystem for Linux

This is part of a series on moving from desktop Linux back to Windows.

The first post is here. The previous post is here.

In short: I would not have moved back to Windows had it not been for the Windows Subsystem for Linux (WSL).

It has made the move from Linux back to Windows so much easier because I can still do the things that I’ve been doing for years, and do every day, quick and easy and familiar: installing and updating development environments (Python, Go, nodejs), databases (PostgreSQL); interacting with source control (git); running apps and tests; running docker clients; etc. So many of the technologies I work with day to date are “Linux/Mac first”, or maybe more accurately “Windows… last”. That was a big motivator for me moving off of Windows in the first place.

WSL is, for me, a game changer when it comes to the local development experience on Windows.

What a world we live in.

What is WSL?

WSL, in short, lets you install Linux environments on Windows without using a virtual machine. Jessie Frazelle has a great write-up on how it works under the hood. For installing and using WSL, the docs are great, too.

Ubuntu on Windows
Ubuntu on Windows

 

What can you do in WSL?

You can run Linux/Mac-first programming environments

Python in WSL

You can install and run services, such as PostgreSQL.

PostgreSQL in WSL

You can call Windows binaries from Linux, and vice versa. Here’s a ridiculous example that you’d never do in real life because the win key or launchy both make running windows binaries so easy, but hey, what the heck:

Running a Windows binary in WSL

 

 

Result of previous command

Note: you need Windows 10 PRO to run Docker for Windows!

You can run a docker client in Linux, talking to Docker containers running via Docker for Windows

Docker client in WSL

You can use a terminal manager like ConEmu — for me, kinda just like iTerm on Mac or Terminal on Ubuntu — and create tabs to your heart’s content (and you can even mix-n-match with linux, cmd, and powershell terminals; and, yes, you can do it all with keyboard shortcuts)

Multiple terminals via ConEmu

You can, of course, tmux

tmux in WSL

You can edit your code in your favorite editor (VS Code is a Windows favorite) and run it on Linux.

VSCode on the left, running Go on the right

Wrapping up

There are, of course, shortcomings and bugs (but how cool is it that WSL itself is on GitHub?!).

I haven’t used it enough in anger yet, and admittedly have not used it heavily enough to have hit any absolute blockers (I’m sure they exist). Though I have brought several of my personal programming projects from Ubuntu onto Windows — all using Go, PostgreSQL, and gulp — and it was all straightforward. Except gulp (natch), b/c of some npm weirdness that took me as long to resolve as it did to bring over the Go apps and databases.

Thus far, though, I am very happy with the local development experience on Windows, thanks in large part to WSL.

Goin’ back to Windows: small, fuzzy apps and compatability settings

This is part of a series on moving from desktop Linux back to Windows.

The first post is here. The previous post is here.

In the first post, I mentioned that one of the motivators for moving back to Windows was that I really liked the hardware of the Lenovo Yoga 920.

I bought one of the “4k Ultra HD Touch Screen” models, not because I cared about the 4k but because it was overall better spec’d. I’m not even sure 4k is realistically perceptible, or useful, on a 14″ laptop.

In fact, on Windows 10, it does have a drawback: lots of apps I’ve installed by default show up really small and fuzzy. And some apps, while “right-sized”, had fairly fuzzy text.

One of the first apps I installed (via Chocolatey, see previous post) was Launchy. And after installing it and activating it with alt-space, I got this:

See how small that is in the pic? Crazytown. It should look like this:

I got the same thing for other apps such as Gimp and VS Code.

Fortunately, the fix is straightforward, if annoying since it seems you have to do it on a per-app basis.

The trick is to change the “Compatibility — High DPI Scaling” setting to either “Application” or “System” (a bit more on that later), for each app that suffers from this problem. Here’s how:

  1. Use the win key to open up the Windows search bar.
  2. Type the name of the app in question
  3. When the app pops up, right click and select “Open file location”

  1. Then, select the app, right click, and click “Compatibility”
  2. Check the checkbox beside “Override high DPI…”
  3. Select either “Application” or “System”
  4. Click “OK” and re-launch the app and see if it helps

My experience so far is that:

  • “System” works best most of the time for apps whose UI was really tiny
  • “Application” works for apps whose UI was right-sized — like Firefox — but whose text was fuzzy

Props to this post about fuzzy fonts for leading me to this solution, which fixed more than just fuzzy fonts for me.

Goin’ back to Windows: package management with Chocolatey

This is part of a series on moving from desktop Linux back to Windows.

The first post is here. The previous post is here.

Windows software management: the current default state

Three years ago, I wrote about moving from Windows to Linux for personal computing and development. Part of my frustration with Windows was around package management — installing, keeping up to date, and removing software. And part of my motivation for experimenting with moving back to Windows is the existence of what appears to be a serviceable solution for package management on Windows: Chocolatey. More on that soon.

But first, let’s talk the current state of installing and keeping software up to date on the main modern operating systems for consumer use.

Linux effectively makes package management a first-class citizen; Mac doesn’t, but homebrew has become de facto package management for Mac, at least for a developer machine.

For the devices we use for most of our personal computing — the little super-computers in our pockets — we’ve come to expect that software is installed from one place, automatically updated, and simple to remove. Even in the wild and woolly world of Android, you get all your software from Google Play; it’s rare to be downloading and installing .apk files from the Internet.

Yet we continue to suffer this craziness on Windows. If you’ve used Windows for a long time — and especially if you’ve used Windows exclusively — this is probably your software install/update experience:

  1. download an executable (.exe, .msi) from the Internet
  2. click it to install
  3. maybe change its install location but probably not
  4. to update… do the same things
  5. over, and over, and over

If you’ve done this long enough, it might even seem sane, reasonable, no big deal.

But I’ll tell you this: after working on Linux and Mac for enough years, that process above drives me f**king crazy and I want to do it as little as humanly possible.

Installing & Updating Software on Linux and Mac

Let’s say you want to install a recent version of PostgreSQL; or Python; or Redis; or Intellij IDEA; or Docker; or &tc… In Windows, you’d do the thing you do above.

But on Mac, you’d probably use homebrew. You’d install homebrew once; and then from a terminal you’d brew install redis. Or brew install docker. You get the picture. To update, you’d use brew to update: brew upgrade redis. Or brew upgrade docker. Or to update all just brew update; brew upgrade. Easy peasy. No “go out to the Internet and download/click things” nonsense.

On Linux, it’s about the same. Depending on the variant, it might be apt install redis or yum install redis. To update: apt update redis or just apt update to update all.

This is the typical experience. Sure, on both, you sometimes need to go out to the internet and download a thing. But that’s the exceptional case. And, yes, keeping that software up to date is often just as much of a hassle as it is on Windows, which is why you try to avoid that as much as possible

Aside from the huge (to me) benefit of being able to easily update all software with a few commands, there’s also the benefit of being able to see all the software you have installed on your system in a single place.

Regrettably, perhaps until you actually see that stuff in action, you might not realize just how incredible it is. My experience with several years on Ubuntu is that it is so brain-dead easy to keep software up to date that it’s practically more difficult not to. If you believe that up-to-date software generally results in a more secure, reliable system, then Ubuntu makes the right thing to do the easy thing to do.

Enter Chocolatey.

Chocolatey

On Windows, there has existed for several years a similar method for installing, listing, and updating software: Chocolatey. If you checked it out a few years ago and thought, as I did, “it’s just not there yet”… it’s there now. Or, at least, it’s come a loooong way.

As of this writing, there are over 5000 packages available via Chocolatey.

If you read my last post giving props to Jessie Frazelle, and if you checked out her boxstarter file, you see lines like this:

choco install Microsoft-Hyper-V-All -source windowsFeatures
choco install Microsoft-Windows-Subsystem-Linux -source windowsfeatures
choco install sysinternals -y
choco install googlechrome

That’s Chocolatey.

Once installing Chocolatey, from powershell or cmd, you can search for packages with choco search [string]; install with choco install [package]; uninstall with choco uninstall [package]; upgrade with choco upgrade [package]; upgrade all with choco upgrade all. And much much more.

You can see all your Chocolatey-installed packages with choco list -l and you get nice output like this:

PS C:\Users\marc> choco list -l
Chocolatey v0.10.8
7zip.install 16.4.0.20170506
avastfreeantivirus 17.9.3761.0
baretail 3.50.0.20120226
chocolatey 0.10.8
chocolatey-core.extension 1.3.3
ConEmu 17.12.26.0
DotNet4.5.2 4.5.2.20140902
Firefox 57.0.2
gimp 2.8.22.20171021
git 2.15.1.2
git.install 2.15.1.2
golang 1.9.2
GoogleChrome 63.0.3239.84
notepadplusplus.install 7.5.3
spotify 1.0.66.478
sysinternals 2017.12.13
visualstudiocode 1.18.1
windirstat 1.1.2.20161210
18 packages installed.

All Chocolatey-installed software is registered as usual with Windows and so they show up in “Apps and Features” as well.

And, as demonstrated in Jessie’s boxstarter file, you can also install Windows features such as Windows Subsystem for Linux 😍. It’s a darn comprehensive tool.

In short: I like Chocolatey and will use it whenever possible.

Shortcomings

Note: This section has been edited, after figuring out the problem. I’m keeping these problems in here, though, in the very unlikely case anyone else stumbles across this behavior.

I had initially observed 2 problems with Chocolatey:

  1. it hadn’t worked, for me, for installing “big” software (eg Intellij IDEA, Docker for Windows)
  2. it was pretty slow, at least compared with homebrew or apt/yum, and downloading packages even of small size seemed to take a long time

Turns out, the cause of both of these problems was IPv6 settings in my wireless router. There was nothing wrong with Chocolatey. I wrote about how I resolved Chocolatey / Powershell slowness via fixing my router’s IPv6 settings.

Conclusion

Chocolatey seems to be the only game in town for sane package management on Windows. I’m grateful that it exists, I like it, and I’m happy to use it. If you’re on Windows, I encourage you to give it a shot.

Next post: Small, fuzzy apps and compatibility settings