<-- home

Property-Based Testing with Clojure and Datomic - Part 2

This is the second part of a series that explores clojure.spec, property-based testing, and how those tools can be used to test Datomic. Check out part 1 for an introduction to clojure.spec and property-based testing. All of the code from this post is in the example repo

As we discussed in the first post, property-based testing works well for pure functions but that excludes a good chunk of code we would like to thoroughly test such as code that makes updates to a database. I also hinted that Datomic is uniquely suited to be tested by property-based tests

Datomic transaction speculation

A helpful way to think of Datomic is as an immutable fact database. When we have a new fact that we want Datomic to learn we pass it to Datomic who stores the fact along with the time the it learned that fact. If we want to retract that fact later on, we tell Datomic to retract it and Datomic makes a note of when that "fact" is no longer true. This architecture allows us to get a consistent, unchanging view of the database at any moment in time, as in "Tell me everything Datomic knew about the world on Wednesday at noon."

For our examples, let's create a Datomic database schema:

    [{:db/id #db/id[:db.part/db]
      :db/ident :user/email
      :db/valueType :db.type/string
      :db/cardinality :db.cardinality/one
      :db/unique :db.unique/value
      :db.install/_attribute :db.part/db}

     {:db/id #db/id[:db.part/db]
      :db/ident :user/friend
      :db/valueType :db.type/ref
      :db/cardinality :db.cardinality/many
      :db.install/_attribute :db.part/db}]

Our schema includes two kinds of facts for users. The first is an email address, which is listed as :db.cardinality/one, meaning that for our program, users can only have one email address at a time. The second kind of fact is for friend relationships, which has the type :db.type/ref which means that it will be a pointer to other users in the system, and has :db.cardinality/many which allows a user to have multiple friends.

For testing purposes Datomic allows for easily creating "alternative universes" of facts. The datomic.api namespace has a function called with that adds the new data to a snapshot of the database and returns a new snapshot of the database. This is done in a way that doesn't mutate the original snapshot, so anyone with a reference to that snapshot sees the same database value that they always have.

For example we can create a virtual database with some new users this way.

    (defn conn []
      ;; ...
      ;; function that returns Datomic connection. Implement this however
      ;; works for you
      (d/with (d/db (conn))
        [{:db/id (d/tempid :db.part/user)
          :user/email "example@email.com"}
          {:db/id (d/tempid :db.part/user)
          :user/email "another@email.com"}])
      ;; => {:db-before datomic.db.Db@e3d26a38,
             :db-after datomic.db.Db@e36ac546,
             [#datom[13194139534313 50 #inst "2016-06-22T18:47:24.005-00:00" 13194139534313 true] #datom[17592186045418 63 "example@email.com" 13194139534313 true] #datom[17592186045419 63 "another@email.com" 13194139534313 true]],
             :tempids {-9223350046623220749 17592186045418, -9223350046623220750 17592186045419}}

The map that we get back from datomic.api/with includes the new database value under the key :db-after. Often that is the only relevant thing we need while creating a virtual database so we can make some helper functions for our tests.

    (defn speculate [db txn]
      (:db-after (d/with db txn)))

    (defn speculate-many [db txns]
      (reduce speculate db txns))

So now that we know how virtual databases work with Datomic our next task is to create some transaction data for new users. Using our email spec from the last post, we can create a spec for new user transactions.

    (require '[clojure.spec :as s]
             '[clojure.spec.gen :as gen])

    (s/def :user/email
      (s/with-gen (s/and string? email-address?) (constantly email-generator)))

    (s/def :user/new-user (s/keys :req [:user/email]))

    (gen/generate (s/gen :user/new-user))
    ;; => {:user/email "hi6NYI7QSBxC7be4@yQl7.CPgB3w9w1WY"}

A good way to write code for Datomic that makes your code testable is write functions that return Datomic transaction data instead of calling transact themselves. For instance, we want to write a code for creating new users, it's better to write a function like this:

    (defn create-user-txn [props]
      [(assoc props
         :db/id (d/tempid :db.part/user))])

rather than:

    (defn create-user! [conn props]
      (d/transact conn [(assoc props
                          :db/id (d/tempid :db.part/user))]))

The first version above allows our application to compound all the transaction data it needs for the given request, and can transact it all at once from the web request handler. It also allows us to create a virtual database with the transaction data in our tests. Let's use the spec we created earlier to generate new user transaction data.

    (defn new-user-txd [num-users]
      (into []
          (map create-user-txn)
        (gen/sample (s/gen :user/new-user) num-users)))

    (new-user-txd 5)
    ;; => [{:user/email "0@K.EM", :db/id {:part :db.part/user, :idx -1000467}}
           {:user/email "A@3.wP", :db/id {:part :db.part/user, :idx -1000468}}
           {:user/email "tUV@6T.ymA", :db/id {:part :db.part/user, :idx -1000469}}
           {:user/email "tu7e@JZA.IC", :db/id {:part :db.part/user, :idx -1000470}}
           {:user/email "C5@q86.8s", :db/id {:part :db.part/user, :idx -1000471}}]

Creating a new virtual database populated with some users is easy.

    (defn db-with-users []
      (let [result (-> (conn)
                     (d/with (new-user-txd 5)))]
        {:db (:db-after result)
         :user-ids (-> result :tempids vals)}))

The database is returned alongside the new user ids, which are resolved from the Datomic tempids in the transaction data.

Property-testing Datomic

Now that we've got a database with users, we want code that allows users to friend each other.

    (defn add-friend-txn [props]

    ;; Friend request
    (s/def :db/id int?)
    (s/def :user/friend int?)
    (s/def :user/friend-request (s/keys :req [:db/id

    (s/fdef add-friend-txn
      :args (s/cat :friend-request :user/friend-request))

The spec hints that we just need to pass a map with keys to :db/id and :user/friend and Datomic entity ids for the values. In our test code we can create a generator for friend transaction data:

    (defn gen-tx-add-friend [user-ids]
        (gen/fmap (fn [[a b]] (add-friend-txn
                               {:db/id a
                                :user/friend b}))
            (gen/elements user-ids)
            (gen/elements user-ids)))))

    (gen/sample (gen-tx-add-friend [1 2 3]))
    ;; => ([]
           [[{:db/id 1, :user/friend 3}]]
           [[{:db/id 3, :user/friend 3}] [{:db/id 2, :user/friend 2}]]
           [[{:db/id 1, :user/friend 1}]]
           [[{:db/id 2, :user/friend 3}]
            [{:db/id 3, :user/friend 1}]
            [{:db/id 3, :user/friend 3}]
            [{:db/id 2, :user/friend 2}]
            [{:db/id 2, :user/friend 2}]]

Our generator has to be provided with a user-ids parameter so that it can create transactions that are associated with real users in the database we want to test. We can't have our generator just produce random integers, because it's extremely unlikely that it would ever produce a valid friend relationship between two present friends in our database.

So as part of our application logic we want to ensure that users can't add themselves as friends. First let's write a datomic query to see if their are any self-friended relationships in a given Datomic database.

    (require '[datomic.api :as d])

    (defn self-friended?
      "Returns true if database has a self-friend relationship present,
      false otherwise."
            '[:find ?e
              [?e :user/friend ?e]]

So now we want to test that for any friend transaction data we can generate that the resulting database will not have a self-friended relationship. Here's the test.check spec that can check that property:

    (require '[clojure.test.check.clojure-test :refer [defspec]
             '[clojure.test.check.properties :as prop]])

    (defspec prop-no-self-friending
      (let [{:keys [db user-ids]} (db-with-users)]
        (prop/for-all [db (gen/return db)
                       txs (gen-tx-add-friend user-ids)]
          (let [db-after (speculate-many db txs)]
            (not (self-friended? db-after))))))

So the first thing we do is create a database with some users, then in our properties we use gen/return which returns the same database every time the generator is run. Each time the test runs it generates a vector of friend transaction data, creates a virtual database by applying those transactions to a blank database and then checks that no self-friend relationships exist in the resulting database. When we run this test we get a failure.

    {:result false,
     :seed 1466691790420,
     :failing-size 3,
     :num-tests 4,
     :fail [datomic.db.Db@c3c1d5f4 [[{:db/id 17592186045420, :user/friend 17592186045422}]
                                    [{:db/id 17592186045422, :user/friend 17592186045422}]
                                    [{:db/id 17592186045421, :user/friend 17592186045420}]]],
     :shrunk {:total-nodes-visited 10,
              :depth 2,
              :result false,
              [datomic.db.Db@c3c1d5f4 [[{:db/id 17592186045422, :user/friend 17592186045422}]]]},
     :test-var "prop-no-self-friending"}

The first failure that the test encountered was when passed the values under the :fail key. Test.check has a feature called shrinking where it keeps trying smaller inputs to see if it kind a more minimal failing case, which is what is returned under the :smallest key in :shrunk. We can easily see the problem which is that we end up with a self-friending relationship whenever we allow a transaction that has them same entity value for the user and the friend reference. I'll admit here that the self-friend example is a little contrived and easy to spot, but let's fix the problem anyway.

    (defn add-friend-txn [props]
      (if (not= (:db/id props) (:user/friend props))
        ;; No self friending

If our transaction creating function determines that the data it is creating is a self-friend relationship, it instead returns a data for a no-op transaction. It's worth noting that this isn't a database constraint, and therefore self-friend relationships can still get into the database if we write code somewhere else that transacts a self-friend relationship directly, bypassing the add-friend-txn. It's up to us to ensure that the only way friendships are added to database is via the add-friend-txn function.

From here

So that was a basic example of how property-based testing could work with Datomic. I haven't seen any prior art in this specific area and I'm interested in your opinions of this approach to testing Datomic.

I have a dream about using this in a real application that has a lot of different kinds of transactions for different application entities, and then test.check can create a sampling of different kinds of transactions, essentially creating a virtual database in a random state using your application code. From there you can assert that the any number of properties hold true about that virtual database. The problems I foresee relate to how to generate transactions that contain a reference to a valid, present Datomic entity id in a generic way. And if we can generate a transaction data list with dependent references, how will it work with test.check's shrinking.

So I'm still not sure if that kind of generic property-based testing with Datomic is possible or practical, but I will continue to experiment in this arena when I have the time. Please share your thoughts!

- Adam Frey, June 2016

<-- home