MOCKS

MOCKS

1. The basics

   • Call mechanics
     • Action / assertion
       • 1. Inject your mock 2. Stage environment for your function to run in 3. Run the function 4. Check the outcomes
   • Mock objects
     • Auto create attributes
     • Callable by default
       • mock_add = Mock(return_value=10)
       • Non-callable is an option
         • I ran into this once when mocking where I shouldn't have been, deep in the Django ORM, something was testing for callability.
     • Magic Mocks
       • Also mock magic methods
       • patch() returns these
   • assert_called_*
     • Be exact with args
       • or use call_args_list
     • Less fussy about order and number of calls
     • Be careful what you call!
       • assert_called() does not exist
       • assert_called_once() does not exist
       • assert_not_called() does exist in recent versions!
   • side_effect
     • can be a list of...
       • exception
       • function
       • return value

2. Gotchas

   • The "name" property
     • You have to use configure_mock Also, spec - but that's less likely to be used externally.
   • Injecting in the wrong place
     • "is not a package" error - the next thing down might not be there
   • Misspellings
     • Specs can help
     • Proper red/green testing can also help, but nobody's perfect
   • Property mocks are tricky
     • Don't be tempted to modify the class
   • Mocks can be cached like any other object
     • beware when testing memoization / memory cache

3. py2 vs py3

   • Rolling backport
   • Is is better to use external lib with 3.3 for example?
     • e.g. assert_not_called available in backport but not in builtin version of py3.3
   • Import differently

4. Pros / cons

   • Can be ugly.
   • Can hide inter-layer errors
   • Drive better design
   • Give a more honest idea of explicit coverage
     • I think it can be good to mark up the integrated tests that might give you these problems. These tests penetrate through all layers, but don't explicitly test all their behaviours - so can give false coverage reports. They should still be run, but also have the option to be excluded from the run so you can see accurate coverage reports.
   • Test doubles can fall out of sync
     • Integration contact tests compare real service to stub service every so often. Mocks rely on and contact either side of the contracts between objects. It's useful (see testing goat) to call down through those contracts - so we know that it's still valid all the way down. That's useful to reassure ourselves about the relationship between our own components, and also their relationship to the outside world.

5. Types of mocking

   • Action / assertion
     • mock
   • Record / replay
     • mox: I think this died
       • Unmaintained, never really moved off google code, not ported to py3, last updated 2010 :-) I think it's dead! I think it's harder to read.
     • Any living specimens?

6. Libraries

   • betamax (requests)
     • it's not record/replay. Well, it's a different type of record/replay!
   • httmock (requests)
     • Could influence choice of library - e.g. github3.py over pygithub
   • django-mock (querysets)
   • flexmock
   • py.test monkeypatch - this seems to be less powerful
     • mock can be used with yield fixtures

7. Style

   • Is naming a good thing?
     • I say this because I look at Mock(name='something')  and think the name property is being mocked. I think mock has let the side down here. It gets the name right quite a lot of the time. I don't bother naming.
   • Specs are your friend
   • Where to mock?
     • Find injection point, then use assignments
       • "backwards" assignments
         • e.g. taking a reference to return_value of mock class
     • How does mock do lookups?
     • Make sure the code under test still runs
       • e.g. if you are mocking another method on the same object, make sure you just mock that.
   • Decorators
     • Personal preference here
       • you get more line space
     • Can mess with introspective stuff, e.g. py.test cleverness
       • Can wrapper helpers help here?
     • Inside out order can be confusing
     • Further away from the action
       • maybe your test shouldn't be that big
       • you can still keep data setup nice and close
     • Take it up to class level if it's used a lot
   • Context manager
     • Makes longer lines
       • This is annoying if you love 80 character line limits!
     • Multiple mocks a bit of a pain
   • Deep mocks **{}
   • Start/stop
     • Quite clunky
   • Return value pass check - no need for a value

8. Uses

   • Django ORM?
     • It gets messy quickly... don't go deep!
     • If you find yourself doing this, think more about your design
     • Sometimes you need to, e.g. simulated error
     • Fat models - test them with the db, mock them out to test their clients
     • It's faster not to hit the ORM too much
     • What about the service layer approach?
   • Time sensitivity
     • User's birthday
     • Special leap year day
     • Time zone rollover
     • Publish dates
   • Simulated failure
     • The disk is full
     • The database did something bad
     • A network connection failed
     • The API endpoint hit its rate limit
   • Timeouts
     • e.g. testing a retry loop. You don't wanna wait
   • Network calls
     • You want a known response from a remote webservice or database
       • Unless you control that service, chances are you're not gonna get one.
     • They'll also slow your tests down.
   • Randomness
   • Filesystem mocks
     • Good idea or not?
     • Probably faster
     • Saves cluttering the filesystem
     • You may not have access to that file
     • You may not actually want to affect that file
     • Better for concurrent tests
     • You may want to check what happens when you *don't* have access to it

9. What?

   • A way for your test to focus on the right things
     • The wrong things are...
       • Other people's code (externals)
       • Other bits of your code
         • Why?
           • You want bad code to fail the right tests... not all of them
             • See Gary Bernhardt, "Fast Tests, Slow Tests" for why you want this. Unit tests should not have heaps of dependencies, or invoke too much of the system - this has implications for (a) understanding test failures (b) test performance
           • Speed
           • Saves excessive setup
           • Unit tests should have few dependencies
   • A way to control the environment in which your test runs
     • Fix the inputs, check the outputs

10. Why?

    • Tests should be deterministic
      • We all hate tests that sometimes fail
      • Tests should always fail for a good reason that's related to your code!
    • Tests should run quickly
    • Tests should run offline
    • You should have control over the inputs
      • We shouldn't have to cause actual failure to test failure

11. About the library

    • Author: Michael Foord
    • Standalone lib
    • Standard in py3.3+

12. What about functional tests?

    • You could hit a real service if results predictable
    • These are allowed to be slower
    • They don't have to hit every edge case
    • They're probably allowed to use the internet too
    • Should still be deterministic, within reasonable constraints
    • Should go all the way to the edge of the system
    • Consider surface area (number of calls)
      • Ideally, you should take your functional tests all the way to the edge of the system - but that can be quite a large surface area (e.g. example where a couple of simple github api calls are actually 4-5 requests, and those requests can return quite rich / large documents to populate API calls. That's fine - you can save off the documents and httmock them or otherwise replace the real calls. (betamax, pretenders... various options here). But they can be quite large. It is messy to keep them alongside the code because they are so big. So you keep them on the filesystem. And then it starts to smell a bit like json fixtures. Is that bad? Json fixtures are kind of bad for other reasons (e.g. shcema changes) I don't know if having it separate is actually inherently bad, other than mental leaps between files. Tradeoff: haveone set of tests that pushes all the way through to the edges. Mock the rest. e.g. fetching somebody's github repos can be a few calls, involving rich info. 

13. How it works

    • Lookups
    • Chaining and its limits