How to integrate New Relic with Django, Apache and mod_wsgi

I just finished setting up New Relic application monitoring on Recommendify. It was a little bit of a painful process and it wasn’t properly described in their docs, so I’m going to note the steps I went through here. Hopefully it will help somebody else in the same situation.

Install the New Relic agent

First, install the newrelic package using pip.

Generate the New Relic configuration

Then copy the command to generate the configuration file from your New Relic dashboard. It will look something like this:

LICENSE-KEY will be the specific license key for your account.

This command will create a file called newrelic.ini. Copy it somewhere safe on your server and make sure it has permissions to be read by the user Apache runs under.

Edit wsgi.py

The next step took some time to figure out. New Relic provides a wrapper script for Python applications, but it doesn’t work for setups using embedded interpreters, such as Apache with mod_wsgi. As this is how Recommendify is configured, I had to find another way.

I edited the wsgi.py file from my Django project to wrap the WSGI application object.

This script imports the socket module and used the socket.gethostname() function to get the hostname of the current machine. I did this so that New Relic would only log data in production and not during development.

At the bottom of the script, I check if the application is running on the production environment. If it is, I import newrelic.agent, initialize it by passing it the path to the newrelic.ini created during the previous step, then wrap the WSGI application object using the WSGIApplicationWrapper class.

You will need to use your own Django settings module, hostname string, and path to newrelic.ini, of course.

Once you deploy the modified wsgi.py, you should start seeing monitoring information in New Relic after about five minutes. Here’s what it looks like for me.

new_relic

What to do after Codecademy

Today I want to offer my opinion on a question that lots of people have asked me since I started this blog. “What should I do after I have finished an introductory course on Codecademy?”

Codecademy is basic stuff

You should be proud to have finished your first course and dipped your toes into programming. It’s a great achievement, but you also have to be realistic: this is just the beginning. You have made your way to base camp, but you haven’t yet attempted to reach the summit.

The fact is that Codecademy, while providing a solid grounding in the basics, doesn’t go much farther than that. On its own, it only teaches you a fraction of the things that you need to know to program professionally (if that’s your goal).

So be proud, but be humble. There is so much to learn that you will never know everything. And that’s a good thing, as long as you like learning.

Identify your goals

Maybe it’s obvious, but the path you take from now should be dictated by what you want to be able to do. If your goal is to write web applications, you should get to grips with frameworks such as Django, Flask and Bottle. If your goal is to use Python for data science, you should be learning things like Numpy, Scipy and Pandas (and hitting the math books hard). If you want to create desktop software, you should learn PyQt and Tkinter.

Once you have to make the decision about what area (or areas) to focus on, then go and find some beginner materials for them. I’m not going to list any here, because there are just so many, but rest assured that they are easy to find.

(That reminds me of another point: Google is your friend. Programmers like to joke that the job is 80% Googling, and there is some truth to that, depending on what you’re building. So you’d better improve your Google-fu and learn how to ask good questions.)

Projects are key

Working through courses and tutorials, and reading technical books, are important ways of improving your knowledge, but there is really no substitute for trying to build something and picking up what you need to know as you go along. The skills of identifying bugs quickly, deciphering arcane error messages and knowing when to stop fiddling with your code and move on are as important to programming as knowledge of advanced language features or algorithms.

With that in mind, I suggest you set yourself a project. Maybe you already have something in mind, in which case, awesome! Or maybe you don’t. In that case, take a look at this list of projects and pick one that seems fun and achievable.

http://www.dreamincode.net/forums/topic/78802-martyr2s-mega-project-ideas-list/

Learn some real computer science

Assuming you’re like a lot of the people picking up coding these days and your goal is to make web applications, then you can go a long way without even thinking about the theoretical underpinnings of programming.

Don’t be that guy. Apart from the often mooted practical considerations of writing efficient code or gaining access to higher paying jobs, there is a world of elegance and beauty in programming that you may not have expected. I am going to suggest a few resources for this because people have remarked that they don’t know where to start.

Code: The Hidden Language of Computer Hardware and Software

This is the amuse-bouche. Just read it one straight through for fun and try to become inspired!

NAND2Tetris

I can’t say enough good things about this course. It took me from a vague understanding of how computers executed my code to being able to conceptualize from the ground up how it all works. If you follow the course the whole way through you will build a (simulated) computer from first principles, create an assembler for it, a compiler for a Java-like language, and a basic operating system. Do it!

Coursera – Algorithms Part 1

This course, taught by Robert Sedgewick, will teach you a basic algorithmic toolbox. The algorithms is deals with are sort of like a “Greatest Hits” collection. Knowledge of these algorithms and design techniques behind forms a kind of lingua franca among programmers. For that reason, they are also really popular interview questions.

Phew! If you get to grips with all that stuff, you’ll be doing well!

Learn source control

I often see people advising beginners to learn Git before they do anything else. Git is a great program, and version control is one of the key skills needed to develop software at a professional level, but I can’t agree with the sentiment. The fact is that Git is extremely complicated. Even experienced programmers end up having to trawl the documentation and StackOverflow to figure out how to do certain things. So I suggest you learn Mercurial instead. It has many of the advantages of Git while being much more user-friendly. As a bonus, it’s written in Python!

They key thing you need to acquire at this stage of your education is a habit of using version control to track the development of your personal projects. You need to go through the process of changing code, breaking your whole project and being able to revert to a working state so that you can really understand why people use version control. For that purpose, the steep learning curve of Git is just going to put you off.

I’m not saying you shouldn’t learn Git later. You should, especially if you want to participate in open source projects. But at that point knowing Mercurial is only going to help, because you will already understand the concepts.

Learn an IDE and/or a text editor

Now is also the time to learn about IDEs and text editors. If you’ve been hanging around on programming forums (you should be), you will no doubt have got an inkling of the holy war between Vi and Emacs users, and the strong opinions of people who think IDEs are unnecessary bloatware. I don’t want to come down on either side of these arguments. I just want to suggest that you are now at the point where you should be investigating them for yourself.

But for my money, PyCharm is an amazing piece of software. 😀

The end

I hope I’ve given you something to think about with this article. If it all seems like a lot of work, well, it is! I’ll let Peter Norvig provide some perspective.

As always, I’m happy to answer your emails.

Comparing files in Python using difflib

Everybody knows about the diff command in Linux, but not everybody knows that the Python standard library contains a module that implements the same algorithm.

A basic diff utility

First, let’s see what a minimal diff implementation using difflib might look like:

The context_diff function takes two sequences of strings – here provided by readlines – and optional fromfile and tofile keyword arguments, and returns a generator that yields strings in the “context diff” format, which is a way of showing changes plus a few neighbouring lines for context.

The library also supports other diff formats, such as ndiff.

Let’s use the utility to compare two versions of F. Scott Fitzgerald’s famous conclusion to The Great Gatsby.

The exclamation marks (!) denote the lines with changes on them. file1.txt is of course the version we know and love.

Fuzzy matches

That’s not all difflib can do. It also lets you check for “close enough” matches between text sequences.

When I saw this first, I immediately thought “Levenshtein Distance”, but it actually uses a different algorithm. Here’s what the documentation says about it:

The basic algorithm predates, and is a little fancier than, an algorithm published in the late 1980’s by Ratcliff and Obershelp under the hyperbolic name “gestalt pattern matching”. The basic idea is to find the longest contiguous matching subsequence that contains no “junk” elements (R-O doesn’t address junk). The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that “look right” to people.

HTML diffs

The module includes a class called HtmlDiff that can be used to generate diff tables for files. This would be useful, for instance, for building a front end to a code review tool. This is the coolest thing in the module, in my opinion.

The class also has a method called make_file that outputs an entire HTML file, not just the table.

Here is what the rendered table looks like:

difflib_html

Go forth and diff!

There are a few other subtleties, but I have covered the main functionality in this post. Check out the official documentation for difflib here.

The bool function in Python

Python’s built-in bool  function comes in pretty handy for checking the truth and falsity of different types of values.

First, let’s take a look at how True and False are represented in the language.

True and False are numeric values

In Python internals, True is stored as a 1 and False is stored as a 0. They can be used in numeric expressions, like so:

They can even be compared to their internal representation successfully.

However, this is just a numeric comparison, not a check of truthiness, so the following comparison returns False:

bool to the rescue

The number 5 would normally be considered to be a truthy value. To get at its inherent truthiness, we can run it through the bool function.

The following are always considered false:

  • None
  • False
  • Any numeric zero: 0, 0.0, 0j
  • Empty sequences: "", (), []
  • Empty dictionaries: {}
  • Classes with __bool__() or __len__() functions that return False or 0.

Everything else is considered true.