Weeknote 20200119 - Python, pip, dependencies

I started working with a new team focussed on using data science to make GOV.UK better for the end users. My previous team focussed on long term maintenance of the platform. Data science is a whole new world for me. I'm excited to get stuck in and learn.

The code is largely in Python and is the first time I have worked with a significant piece of production Python code. I immediately bumped into problems trying to install the requirements for the project.

In the project, two of the top level dependencies rely on Pygments. Pip installed an up to date version of Pygments when it resolved the first dependency, but the second dependency required an earlier version of Pygments. I think this was the result of merging Dependabot PRs without running pip install and ensuring a clean install.

Pip, to my surprise, does not resolve dependency versioning issues and I am reminded of DLL Hell.

My current understanding of workflow on a Python project is developers occasionally use pip freeze to write requirements to a file. Subsequent efforts can install from the file. Apart from the versioning issues, it is also easy to commit unwanted packages to a project if developers are not careful.

I spent some time investigating how other projects manage Python dependencies. A colleague pointed me towards PEP-0518 which looks interesting, but in my limited view of repositories around GDS, I did not see a .toml file being used so I don't know whether people are doing this in the wild or not.

I found pipdeptree which outputs a tree view of dependencies similar to Bundler's Gemfile.lock.

I discussed the problem with one of the other developers on my new team and went digging around some other Python projects. We found a pattern of two requirements files. One for base project dependencies that is hand crafted, and one autogenerated with all dependencies and their sub-dependencies. The advantage is developers are more aware of the dependencies they are adding and this should address the unwanted or unneeded packages sneaking their way into a project.

Going forward, I want to add some automated process around the projects to catch these errors earlier. The first thing to do is spin up a new virtual environment and run pip install -r requirements.txt on each branch push. After that, I want to put some linting in place. The code is still small enough for this not to be too daunting.

How to run a scheme script from the command line

Here's a small bit of scheme code:

(define (adding a b)
(+ a b))

(display (adding 7 4))

To run this directly from the command line, you can use the following:

mit-scheme --quiet < script.scm

How to force line breaks in markdown

I always forget this.
I don't know why, because my editor shows big nasty red blotches whenever I have trailing spaces.
To force a line break in markdown, end your line with two spaces.

A limerick

A couple of dev with nice Macs
Thought up some quite clever hacks
They opened PRs
Approved them too fast
Now production is being rolled back

How to output the current RSpec example

Here's a snippet you can add to your rspec config to print out the name of the method being run:

config.before do |example|
puts "----- state ----"
puts example.metadata[:full_description]
puts "inline: #{Sidekiq::Testing.inline?}"
puts "fake: #{Sidekiq::Testing.fake?}"
puts "----------------"

Why on earth would you wan to do this? Sometimes you are working with something like Sidekiq where state can bleed between tests (for example changing from inline to fake); this little snippet helps you figure out where it's happening without having to investigate all your test files.

Generating mongoid config in rails hangs

If setting up a new Rails project with Mongoid and trying to run rails g mongoid:config hangs, try

spring stop
spring start

Not In queries in Rails with MongoDB and Mongoid

Sometimes you want a quick "not in" or "not equal" query in Rails, but you're running Mongo. ActiveRecord allows us to write where.not against relational databases, but this doesn't work when using Mongoid. Instead, you can use nin or it's alias not_in

Book.not_in(title: "Of mice and men")

The method is also chainable, so this is valid:

Book.where(author: "Steinbeck").not_in(title: "Of mice and men")

If you prefer not to chain your methods, you can also write the query by sending nin to a symbolised version of your model's property and then using the old hashrocket syntax to specify the value:

Book.where(author: "Steinbeck", :title.nin => ["Of mice and men"])

Module not found: Error: Can't resolve 'fs' using webpack and Electron

Module not found: Error: Can't resolve 'fs'

If you've set up webpack with an Electron application and you hit this strange error when trying to bundle your project, you've probably forgotten to include the target in your webpack config. This is easily fixed by editing webpack.config.js to include the target, something like this:

const path = require('path');

module.exports = {
entry: './app.js',
output: {
filename: 'bundle.js'
target: 'electron'

Fixing a broken npm tha errors with "cannot find module 'process-nextick-args’"

Following along with Free Code Camp's Node course, I was playing with how-to-npm from Node School, I followed the first exercise and ran how-no-npm verify. My npm version was out of date and following the prompt, I ran npm install npm -g. That seemed to complet successfully, however, running how-to-npm verify subsequently gave me a lovely string of errors, the top one being

Error: Cannot find module 'process-nextick-args’

Running any npm command seemed to result in something similar.

To fix this, I did the following - first upgrade Node itself with nvm:

Step 1 - upgrade node itself with nvm:

nvm install v8.9.3

Step 2 - install how-to-npm again:

npm install how-to-npm -g`

Step 3 - upgrade npm

npm i -g npm

Now I can run how-to-npm verify again and everything works

Add non blog pages to a middleman blog

Having set middleman up to be a static blog, you might find yourself wanting to add some static pages outside of the blog structure. For example, if you install any analytics on your site, you often need to include a static file with an authentication code to your analytics provider. These pages are often barebones, with no common header or footer and no styling and you almost certainly do not need them to be processed from markdown. In other words, you want to create a page that does not use the layouts that the rest of your site uses.

To set these up, create the html page and then edit your config.rb to include something like the following:

page "my_barebones_page.html", directory_index: false, layout:false

Using layout: false is reasonably self-explanatory; you are instructing middleman to forego any layout for this page. The inclusion of directory_index: false makes sure that the page is accessible with the .html suffix. This is especially useful if you are using pretty urls for your blog.

Next Page