John Collins

John Collins

Developer. Thinker

© 2021

Conj Talk (Draft)

Table of Contents

  1. Intro
  2. The Project
  3. The Macro
  4. Why the hell Would you use Clojure?
  5. Limitations of ROS
  6. Limitations of Python
    1. Clojure vs. Python: libraries
    2. Clojure vs. Python: performance.
    3. Clojure vs. Python: interactivity
    4. Clojure vs. Python: reach
    5. Clojure vs. Python: program analysis
  7. Coming Into Rhythm
  8. The Formats Problem
  9. Perception
  10. Planning
    1. logic-based planners?? Maybe?
    2. Rules-based planning policies?
    3. Odometry
  11. Databases > dashboards
  12. Where is The Cloud?
  13. Thinking Big
  14. Global Costmaps
  15. Some Libs
  16. Growth is Good

Intro

  • I’m John Collins. I want to say thanks for having me, it’s great to be here. I’ve been following and tip-toeing in the Clojure world for a little while now so it’s it’s exciting to get to talk about my first experiences trying to do actually do something with the language.

The Project

  • I’ve been working on a robotic fleet analysis application written in Clojure and Clojurescript.
  • In some ways it’s your typical Clojure app: dump a bunch of crap into a database and query it. It’s a way to start to think about how to model and represent robot data at scale.
  • It’s an application that brings together telemetry and event data to aid in assist analysis and fleet monitoring.
  • One of the important things we deal with a lot is this concept of an assist, or any time the robot needs help from a human. Driving down assists is one of our keystone variables.
  • It’s Clojure+Ring+Postgres on the backend, plus Docker for deploying internally (for now). I’ve also been very much enjoying using jdbc+honeysql for db interaction, and spec for request validation and destructuring of event streams (which I’ll talk about a bit later).
  • On the frontend, I’m now using ClJS+Reagent+Recom+Datascript+Three.js+Geom. The backend DB is roughly replicated in the cljs frontend but with of course datalog style queries.

The Macro

  • As we transition to a 5G world, enterprise data is likely to oustrip consumer data over the next decade, which means that the biggest companies of the future may not actually be consumer facing companies (like Apple, Google, Amazon, Facebook, etc.).
  • We’re moving to a world where humans are not the primary consumers of data.
  • This is not really a world of accounts and users and departments and so on, nor is it really similar to current high-bandwidth applications like video distirbution (more about this later), so we may need to be rethinking our assumptions about how we model and organize information.
  • Perhaps this is something Clojurists should be thinking about, given that it’s in the information game.

Why the hell Would you use Clojure?

  • Clojure is certainly not the first language that comes to mind when you think of robotics. Certainly, given it’s GC’d, has a large runtime, it’s not realtime (unless you’re will to fork out big bucks for a specialized GCs), etc..
  • That said, it’s certainly not more crazy than using Python, which is quite common in the industry.
  • There will be robots which are not safety critical. For example, there will be cute fuzzy little novelty bots, and that’s certainly not an area where you need a realtime OS. And there’s certainly nothing “realtime” about what’s going on between your ears.
  • Even for safety critical systems, there are other approaches. For example, we’ve taken a strong isolation approach where we run safety critcal firmware independent hardware without an OS, while the high-level navigation stack is not strong realtime.

Limitations of ROS

  • First of all, ROS is awesome. Tons and tons of useful libraries for getting started building a robot. Localization, mapping, planning, it’s all there.
  • That said, there are many issues that come into play as you scale up.
  • It’s an actor model, i.e. highly non-derminstic. No deterministic replays. I think we’re well versed with these problems in the Clojure world.
  • Very bad fit for building a deterministic navigation system.
  • Very hard to test.
  • Built on some kind of “hive mind” idea with tons of networked robots. In reality, you want your stack to look something much more like a Pixar-style rendering pipeline than a hive of asyncronously iteracting agents. It just is not practical. That said, there will be a hive mind going on at some level, but it’s a big mistake to carry that mindset all the way down.
  • ROS is built on the realtime clock as core syncronization primitive: NTP timeshifts can wreck havoc on your system. You really wanna be using the monotonic clock.
  • Does not have a model for runtime schema migration.
  • ROS mesage is not a particularly compact message format. It’s not an ideal format to be storing huge amounts of data in. Also, lots of cycles end up going to message passing over TCP.
  • It’s fundamentaly insecure. I’m not an expert here, but given that it’s built on the networking stack, there’s potentially a lot of attack surface here, which is quite scary if you consider how many autonomous cars are using it.

Limitations of Python

Python is the defacto standard high-level language to use for robotics, certainly for research work. We use c++ for any performance sensitive stuff and pybind to it, which works quite well. Nevertheless, you still run into a lot of issues using Python, which many of you are probably aware of.

  • The GIL really does suck. Python cannot own main.
  • Absolute unmitigated disaster of python 3 (exaggerating).
  • Package management is pretty bad.
  • Broken generators implementation.
  • Highly imperative with poor support for FP (partly because calling functions is just so slow).
  • No JIT.
  • Exceedingly slow.
  • That said, people love their Python.

Clojure vs. Python: libraries

  • Python has the libraries. ROS, Pytorch, tensorflow, numpy, etc. etc. (but capital excesses carries its own set of challenges)
  • However, Neanderthal + bayadera + DeepDiamond is very exciting!!
  • Something needs to be done about core.matrix (more on this later).

Clojure vs. Python: performance.

  • Python is a pig.
  • Must rely on C linkage+pybind for all performance sensitive code.
  • Clojure is much, much more ammenable to perf optimization within clojure. Shoutout to clojure-goes-fast.com!
  • Memory is a different story. I’ve already run into cases where the memory usage is obsurd relative to the entropy inherent in what’s going on, especially when using jdbc. I’m definitely excited to migrate to jdbc.next.
    • just look at the AWS pricing: memory isn’t cheap.
    • you just need to be more careful in this area.
  • JNI ins’t so bad honestly, but java’s frozen primitive set is a problem for robotics, even when it comes to peripheral applications. I know there are some “good” arguments for a reduced primitive set, but honestly I don’t buy it, especially in robotics-related apps where you are often dealing with unsigned data. A 10% compression of Depth data, for exmaple, can save you millions of dollars on your LTE bill. There encoding details really matter.

Clojure vs. Python: interactivity

  • In general, Clojure’s repl experience is much better (especially with CLI tools).
  • However, I actually miss some of the REPL features available in Python and Javascript.
  • `let` is actually slightly problematic, as it cannot be evaluated piece-wise
  • These repl features are totally available nowadays in Python, JS, etc., so we should make sure we follow them
  • Maybe there’s a way to do this via. Clojure REBL.
  • `let` requires additional scope. I often find myself wanting to explode a let binding into a series of defs. Is there something we can do here? You can have some kind of “other” thing, but that’s contrary to the LISP way of staying within the language. Maybe REBL could help here (haven’t beeen able to get it to work on Linux yet).
  • I love the built-in cider debugger.

Clojure vs. Python: reach

  • Not much to say here.
  • Clojure targets the browser, node, and the JVM.
  • Python also doesn’t abstact the OS as much.

Clojure vs. Python: program analysis

  • Java obviously easily takes the cake here. This is actually super important. We deal with system performance related matters all the time.

Coming Into Rhythm

  • Getting up to speed was harder than expected.
    • Clojure+Java+Clojurescript+Javascript+emacs+cider etc. etc. is a lot to take on.
  • Takes a while for clojure.core to really snap, however I find clojure code to be quite grippy once it does.
  • Getting a sane Clojurescript workflow was espcially hard. `cider-jack-in` ended up being a truly unmitigated disaster for me. I honestly believe we should remove it or strongely recommend connecting to an external process.
  • Still don’t have good exceptions in clojurescript repl, which leaves me occcasionally stuck bisecting errors.
  • Clojure CLI tools was totally critical.
    • Single shortest path to invoking Clojure code. Wonderful.
    • I never felt like I could get a good grip on the mapping of lein configuration to what the heck was ever going on.
    • I no longer fear the death of a repl session because it starts in under a second.
    • Overall super bullish on the CLI tools. I think they’re super critical and we should keep moving (destructively) in that direction. Tooling should grow via. libraries with a direct mapping to invoked clojure code.
      • The proliferation of build tools certainly problematic, however.
  • core.matrix has some immediate let downs for someone coming from numpy. I’m super excited about what’s going on with the uncomplicate project, and I’ve followed some of the arguments surrounding the whole debate of whether it should be a core.matrix backend, and I’m sympathetic to both sides. But I do think we need a very solid, high-level numpy/matlab/whatever style API. I’m not totally sure what the right path foward is, but I hope we can come to an agreement on it. That’s sort of a general thing. I wish we could spend more time collectively at a conceptual, strategic, sort of white paper level.
  • Speaking of which, Clojurescript front-end libraries. I still feel like we have yet to fully crescendo here. In particular, I’m excited about datascript but I’m not sure how it fits in to the “ecosystem” (hate that word). I reeeally want to be able to express views as reactive datalog queries–which is apparently exactly what a rules engine provides–but I don’t yet see a mature offering in this area. I’m scraping by with Reagent+reposh, which tries to extend reagent to use a datascript db with reactive query, but it doesn’t work that well and isn’t maintained.
  • Then there’s the fact that the UI as essentially a DOM valued function of your application state basically sucks. Doesn’t it? Why is layout management the problem of like every frontend developer world?? What happened to WYSIWYG? I just wanna say here’s my set of information, please make it pretty! I feel like if we could truly decouple this layout stuff, like there’s our killer app. Anyway, I don’t know anything about frontend (except that I suck at it).
  • To be fair, recom goes a long ways. It’s fantastic, highly recommended.
  • I love spec, but it didn’t end up satisfying the use case of destructuring event streams very well.
    • There’s no equivalent of re-findall for spec. Instead, you’re spec has to match the entire sequence. I’d like to be able to dos something like this:
      (s/def ::auto-assist (s/cat ::auto-assist assist-require?
                                  ::auto-clear assist-clear?))
              
      (s/def ::auto-pause (s/cat ::auto-pause auto-pause?
                                 ::auto-resume auto-resume?))
              
      (s/def ::auto-session (s/cat ::auto-start auto-start?
                                   ::auto-stop auto-stop?))
              
      (s/def ::autonomy-assists (s/cat ::auto-start auto-start?
                                       ::auto-assists (s/* ::auto-assist)
                                       ::auto-stop auto-stop?))
              
      (s/def ::autonomy-pauses (s/cat ::auto-start auto-start?
                                      ::auto-pauses (s/* ::auto-pause)
                                      ::auto-stop auto-stop?))
              
      (s/def ::autonomy-session (s/cat ::auto-start auto-start?
                                       ::auto-events (s/* (s/alt ::auto-pause  ::auto-pause
                                                                 ::auto-assist ::auto-assist))
                                       ::auto-stop auto-stop?))
      

      And then do some enrichment very cleanly with something like this:

      (defn derive-assists
        "Derive assist records associated with a match."
        [{:keys [::specs/auto-start
                 ::specs/auto-assists
                 ::specs/auto-stop]
          :as match}]
        (for [{:keys [::specs/auto-assist
                      ::specs/auto-clear]} auto-assists
              :let [auto-start-data (:events/data auto-start)
                    auto-assist-data (:events/data auto-assist)
                    id (derive-ulid (list* auto-start auto-stop auto-assist auto-clear))]]
          #:derived-assists
           {:id id
            :autoStartEvent (:events/id auto-start)
            :autoStopEvent (:events/id auto-stop)
            :deviceId (:events/deviceId auto-assist)
            :eventId (:events/id auto-assist)
            :telemetryId (:telemetry_id auto-assist-data)
            :startTime (:events/eventTime auto-assist)
            :endTime (:events/eventTime auto-clear)
            :routeHash (:route_hash auto-start-data)
            :pose (-> auto-clear :events/data :pose)
            :code (:code auto-assist-data)
            :reason (:reason auto-assist-data)
            :activityId (:activity_id auto-assist-data)})) 
      

      But, in order to make this work I have to implement a very gross “find-all” function:

       (defn find-all [spec sample-size events]
         (let [spec (s/cat ::match spec ::rest (s/* any?))]
           (loop [events events matches []]
             (if (empty? events)
               matches
               (let [[sample more] (split-at sample-size events)
                     match (s/conform spec sample)]
                 (if (s/invalid? match)
                   (recur (next events) matches)
                   (recur (concat (::rest match) more)
                          (conj matches (into {} (::match match))))))))))
      
      • It ends up not really being worthwhile, so I’m back to doing some parsing with a hedious state machine.
      • Geom et. al. is amazing. It’s unfortunate the clojure world never seemed to notice Karsten Schmidt aka. Postspectacular. The guy is an absolute fiend. He pretty much printed geom and a dozen other libraries with like no tests and no bugs. He has moved on to other things, but his talk was one of the things that got me interested in Clojure and he is still is very much an inspiration to me. I like to think that I’m continuing his footsteps to try to push Clojure into new and exciting domains.
        • That said, I think the project could really use better documentation, specification, and tutorials.
        • I’m working on moving from Three.js to Geom (CLJC), but I’ve realized it’s going to take a bit more time to get that effort to a good place.
        • In the meantime, I’ve been working on migrating all the examples to the latest Clojure(Script), and I’ve also started specing it, which I think could help a lot.
          • However, geom makes extensive use of direct calls to protocol methods, which I’m not aware of a way to spec. For example, I’d love to be able to talk about what’s going on here: ```clojure g/IFaceAccess (faces ([] (get _ :faces)) ([ opts] (if opts (map #(g/raw % _) (get _ :faces)) (get _ :faces)))) … (remove-face [{:keys [vertices edges faces fnormals vnormals] :as _} f] (if (get faces f) (let [fv (g/vertices f _)] (loop [verts vertices vnorms vnormals edges edges fedges (partition 2 1 (conj fv (first fv)))] (if fedges (let [[a b] (first fedges) e #{a b} efaces (disj (get edges e) f) edges (if (seq efaces) (assoc edges e efaces) (dissoc edges e)) ve (filter #(not= (get % :f) f) (get verts a))] (if (seq ve) (recur (assoc verts a (into #{} ve)) vnorms edges (next fedges)) (recur (dissoc verts a) (dissoc vnorms a) edges (next fedges)))) (assoc _ :vertices verts :vnormals vnorms :edges edges :faces (disj faces f) :fnormals (dissoc fnormals f))))) _))

        g/INormalAccess (face-normals [_ force?] (if (seq (get _ :fnormals)) (get _ :fnormals) (if force? (get (g/compute-face-normals _) :fnormals)))) ```

    But I’m not aware of a way to do this. I could easily be missing something.

The Formats Problem

  • Part of the goal of the project is to develop this notion of a “brain scan”, or a snapshot of the robot in time. This is essential for deterministic replay.

  • A Brain scan could look something like this, for example (pseudo code):

    struct LaserScan {
      id             @0 :UInt8;
      data           @1 :Tensor;
      transformIndex @3 :UInt16;
    }
        
    struct FilterMask {
      id @0 :UInt8;
      # Filter Mask Data.
      data  @0 :Tensor;
      # Index of scan or costmap the filter was applied to.
      index @1 :UInt16;
    }
        
    struct Costmap {
      type           @0 :Text;
      data           @1 :Tensor;
      transformIndex @3 :UInt16;
    }
        
    struct Percept {
      costmaps    @0 List(:Costmap2D);
      laserScans  @1 List(:LaserScan);
      depthImages @3 List(:DepthImage);
      filterMasks @4 List(:FilterMask);
    }
        
    struct Plan {
      source        @0 :Source;
      valid         @1 :Bool;
      acceptedPath  @2 :Path;
      rejectedPaths @3 List(:Path);
    }
        
    struct TricycleDriveCommand {
      velocity   @0 :UInt32;
      wheelAngle @1 :Float32; 
    }
        
    struct BrainScan {
      const version :UInt32 = 0;
      monotonicTime @0 :Float32;
        
      percepts @1 List(:Percept);
      plan     @2 :Plan;
      command  @3 :TricycleDriveCommand;
    }
    

Perception

  • We call our basic perception primitive the “Costmap”. Not a new term. Essentially it’s just a 2D byte array of data.
  • We use tons and tons of costmap layers. You have of course obstacle maps, cliff layers, specialized people detection layers, semantic layers.
  • Dynamic costmap compositing.

Planning

  • Plan serialization.

  • We operate with a variety of planners, from local DWA style planners, to sort of elastic band style planners, to long-term AStar-style planners, deep reinforcement planners, the list goes on. It’s one of those areas where things will definitely be quite different in 5 years.

  • The real dificulty here not the question of finding an “optimal” path, which is what a log of the research focuses on, but defining the cost function itself. Defining the “least risky” path is hard.

Logic-based planners?? Maybe?

(defne travel [a b visted path]
  ([a b visited [b . visited]]
     (edge a b))
  ([a b visited path]
     (fresh [c new-vis]
            (edge a c)
            (!= c b)
            (conso c visited new-vis)
            (travel c b new-vis path))))

(defne reverseo
  "w is reverse of l"
  [l z w]
  ([() x x])
  ([[x . y] z w]
     (fresh [nz]
            (conso x z nz)
            (reverseo y nz w))))

(defn path
  "p is all paths between a and b"
  [a b p]
  (fresh [z]
         (travel a b [a] z)
         (reverseo z [] p)))

Rules-based planning policies?

  • As you can imagine, a lot of control logic goes into managing these planners. Boy do you run into a hairy mess trying to specify and reason about planning policies.
  • It’s very important for us to serialize a lot of information about planner state and continue to drive toward better abstractions of planning.
  • The “Automatic Rule Explanations” could be very useful.
  • You could also try to do learning on top of the rule activations to figure out which chains of actions are working. Could ultimately feed back into planning, or more practically could be rich data for analysis.

Odometry

  • What is the robot’s actual state, in terms of velocities, wheel angles, etc.? Not to mention all the actual floor scrubber states and functions… Did we actually clean the place?

Databases > dashboards

  • There’s a good number of people doing dashboards and realtime telemetry/monitoring stuff, but that’s not where most the value is.
  • I love the “liberate the database” sort of mission at Cognitect, and I think that definitely applies here.
  • You want query and indexing. You want leverage.
  • Can you… Show me all the tight turns? Narrow passageways? Darkest environments? Where is the robot getting stuck the most? You really want this rich query capability.
  • You also want raw data. You want big, dense datasets and sparse datasets (semantics).
  • You want integration with testing, research, etc., etc.
  • You want tools.

Where is The Cloud?

  • Is a robot fleet a cloud? Should we think of it as a cloud?
  • This gets into this question of where does the code live and should I even have to care.
  • The equation changes when you’re talking about an order of magnitude difference in the amount of data being transfered.
  • When do we push, when do we pull?
  • If you think about it, robot utiliztion is quite low and there’s a quite a lot of excess capacity.
  • It’s something to think about application-wise: what can we be doing while these machines are sleeping?
  • The value is also highly localizalized: this is not blasting Game Of Thrones to four hundred million viewers.
    • We’re talking about parsing the semantics of a particular environment.
  • How will control be managed? This is an important problem and it’s something we need many people thinking about and staking themselves in.
  • Then there’s the fact that these machines are server-like, in the sense that utilization and capacity are key concerns (more on this later). What percentage of the time is a floor scrubber actually scrubbing floors? Not much.
  • This question of where does the code live and where does the data live and should I even care is wide open in the context of robotics. Of course, compliance is a huge issue around this.
  • It’s one thing to say “Oh well cearly it’s going to be visual slam in the cloud with RBG data”, but then you do a little napkin math to work out that the LTE bill only to realize it’s completely unafordable. It’ll be interesting to see this play out.
  • There’s many “situtated” programs that I think will live in this 5G world. Clojure is very good at information aggregation. It might be fruitful to think about what these situated programs will be.

Thinking Big

  • One interesting game to play is, “what would robot datastucture X look like at scale”?
  • Think massive KD Trees in the cloud. Could it make sense to model in datomic? idk.
  • Or think massive Pose Graphs.
  • Or massive transform trees. etc.

Global Costmaps

  • It’s also interesting to think about the novelty problem generally in the context of Clojure and robotics, which I see as the core problem Clojure addresses itsef to. That is, what if you imagine every robot scan as an update to some big persistent immutable datascruture, which is relected lazily to the robot as it navigates. So now you have a robot that that’s continually diffing it’s state with the world state–the result being some kind of sparce thing–and sending the delta.
  • What will these probabilistic calculations look like at scale?
  • Remember, when you’re doing semantics, you’re sparcifying the data big time.
  • Of course, we must keep in mind that basically in robotics localizcation isn’t a thing. A local planner must be sufficient in all cases.
  • Novelty driven push? The value of data is proportional to the delta in information gain.
  • Ultimately the entire question of Push vs. Pull should not exist.

Some Libs

  • I wrote a few basically trivial libraries to hopefuly help with getting going programming robots.
    • clj-rosbag YAW (yet another wrapper) for reading and writing ROS data. BTW, there’s tons of free ROS data available out there. Take advantage.
    • clj-tf a very simple transform library for clojure. It’s based a similar
  • I’ll be releasing more libraries over time.

Growth is Good

  • “The biggest risk to the future is bad ideas.” – Dr. Zubrin …
  • kinda another way of stating the “conception problem” Rich has talked about, but more broadly.
  • Robots are scary. Ai is scary, big data is scary.
  • But we need growth.
  • Fact is, we’re not in the fourth industrial revolution at all. Not yet.
  • We’re in a prolonged period of economic stagnation.
  • We need to stay rooted in reality and work really hard on achieving real economic gains.
  • That means making smart investments with our time.
  • Aaand, that’s it.