-
-
Notifications
You must be signed in to change notification settings - Fork 73
Expand file tree
/
Copy pathembedded.html
More file actions
115 lines (113 loc) · 12 KB
/
embedded.html
File metadata and controls
115 lines (113 loc) · 12 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
<!DOCTYPE html PUBLIC ""
"">
<html><head><meta charset="UTF-8" /><title>Embedding Clojure In Python</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="highlight/solarized-light.css" /><script type="text/javascript" src="highlight/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a> with <a href="https://github.com/xsc/codox-theme-rdash">RDash UI</a> theme</h2><h1><a href="index.html"><span class="project-title"><span class="project-name"></span> <span class="project-version">2.006</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="Usage.html"><div class="inner"><span>Usage</span></div></a></li><li class="depth-1 "><a href="design.html"><div class="inner"><span>LibPython-CLJ Design Notes</span></div></a></li><li class="depth-1 current"><a href="embedded.html"><div class="inner"><span>Embedding Clojure In Python</span></div></a></li><li class="depth-1 "><a href="environments.html"><div class="inner"><span>Python Environments</span></div></a></li><li class="depth-1 "><a href="new-to-clojure.html"><div class="inner"><span>So Many Parenthesis!</span></div></a></li><li class="depth-1 "><a href="scopes-and-gc.html"><div class="inner"><span>Scopes And Garbage Collection</span></div></a></li><li class="depth-1 "><a href="slicing.html"><div class="inner"><span>Slicing And Slices</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>libpython-clj2</span></div></div></li><li class="depth-2 branch"><a href="libpython-clj2.codegen.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>codegen</span></div></a></li><li class="depth-2 branch"><a href="libpython-clj2.embedded.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>embedded</span></div></a></li><li class="depth-2 branch"><a href="libpython-clj2.java-api.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>java-api</span></div></a></li><li class="depth-2"><a href="libpython-clj2.python.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>python</span></div></a></li><li class="depth-3"><a href="libpython-clj2.python.np-array.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>np-array</span></div></a></li><li class="depth-2"><a href="libpython-clj2.require.html"><div class="inner"><span class="tree" style="top: -52px;"><span class="top" style="height: 61px;"></span><span class="bottom"></span></span><span>require</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#embedding-clojure-in-python" name="embedding-clojure-in-python"></a>Embedding Clojure In Python</h1>
<p>The initial development push for <code>libpython-clj</code> was simply to embed Python in Clojure allowing Clojure developers to use Python modules simply transparently. This approach relied on <code>libpython-clj</code> being able to find the Python shared library and having some capability to setup various Python system variables before loading any modules. While this works great most of the time there are several reasons for which this approach is less than ideal:</p>
<ol>
<li>In some cases a mainly Python team wants to use some Clojure for a small part of their work. Telling them to host Python from Clojure is for them a potentially very disruptive change.</li>
<li>The python ecosystem is moving away from the shared library and towards compiling a statically linked executable. This can never work with libpython-clj’s default pathway.</li>
<li>Embedded Python cannot use all of Python functionality available due to the fact that the host process isn’t <code>python</code>. Specifically the multithreading module relies on forking the host process and thus produces a hang if the JVM is the main underlying process.</li>
<li>Replicating the exact python environment is error prone especially when Python environment managers such as <code>pyenv</code> and <code>Conda</code> are taken into account.</li>
</ol>
<p>Due to the above reasons there is a solid argument for, if possible, embedding Clojure into Python allowing the Python executable to be the host process.</p>
<h2><a href="#enter-cljbridge" name="enter-cljbridge"></a>Enter: cljbridge</h2>
<p>Python already had a nascent system for embedding Java in Python - the <a href="https://pypi.org/project/javabridge/">javabridge module</a>. </p>
<p>We went a step further and provide <code>cljbridge</code> python module.</p>
<p>In order to compile <code>javabridge</code> a JDK is required and <strong>not just the JRE</strong>. <a href="https://github.com/tristanstraub/">tristanstraub</a> had found a way to use this in order to work with <a href="https://github.com/tristanstraub/blender-clj/">Blender</a>. We took a bit more time and worked out ways to smooth out these interactions and make sure they were supported throughout the system.</p>
<h2><a href="#from-the-python-repl" name="from-the-python-repl"></a>From the Python REPL</h2>
<p>The next step involves starting a python repl.</p>
<p>This requires a python library <code>cljbridge</code>, which can be installed via</p>
<pre><code>export JAVA_HOME=<--YOUR JAVA HOME-->
python3 -m pip install cljbridge
</code></pre>
<p>This will install and eventually compile <code>javabridge</code> as well.</p>
<p>If the installation cannot find ‘jni.h’ then most likely you have the Java runtime (JRE) installed as opposed to the Java development kit (JDK).</p>
<p>So we start by importing that script:</p>
<pre><code class="python">Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from clojurebridge import cljbridge
>>> test_var=10
>>> cljbridge.init_jvm(start_repl=True)
Mar 11, 2021 9:08:47 AM clojure.tools.logging$eval3186$fn__3189 invoke
INFO: nREPL server started on port 40241 on host localhost - nrepl://localhost:40241
</code></pre>
<p>At this point we do not get control back; we have released the GIL and java is blocking this thread to allow the Clojure REPL systems access to the GIL. We have two important libraries for clojure loaded, nrepl and cider which allow a rich, interactive development experience so let’s now connect to that port with our favorite Clojure editor - emacs of course ;-).</p>
<h2><a href="#from-the-clojure-repl" name="from-the-clojure-repl"></a>From the Clojure REPL</h2>
<p>From emacs, I run the command ‘cider-connect’ which allows me to specify a host and port to connect to. Once connected, I get a minimal repl environment:</p>
<pre><code class="clojure">;; M-x cider-connect ...localhost...40241
;; I am not sure why but to initialize the user namespace I have to eval ns user
user> (eval '(ns user))
nil
user> (require '[libpython-clj2.python :as py])
nil
;; Python has been initialized and libpython-clj can detect this
user> (py/initialize!)
:already-initialized
user> ;;We can share data via the main module
user> (def main-mod (py/add-module "__main__"))
#'user/main-mod
user> (def mod-dict (py/module-dict main-mod))
#'user/mod-dict
user> (keys mod-dict)
("__name__"
"__doc__"
"__package__"
"__loader__"
"__spec__"
"__annotations__"
"__builtins__"
"cljbridge"
"test_var")
user> (get mod-dict "test_var")
10
user> (.put mod-dict "clj_fn" (fn [& args] (println "Printing from Clojure: " (vec args))))
nil
user> ;;Now if we stop the repl server we can access our python environment again
user> (require '[libpython-clj2.embedded :as embedded])
nil
user> (embedded/stop-repl!)
</code></pre>
<h2><a href="#and-back-to-python-" name="and-back-to-python-"></a>And Back to Python!!</h2>
<p>Shutting down the repl always gives us an exception; something perhaps to work on. But the important thing is that we can access variables and data that we set in the main module -</p>
<pre><code class="python">>>> Exception in thread "nREPL-session-d684061e-f21c-4265-a9a2-828b99dcaf42" java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at nrepl.transport$bencode$fn__7714.invoke(transport.clj:121)
at nrepl.transport.FnTransport.send(transport.clj:28)
at nrepl.middleware.print$send_streamed.invokeStatic(print.clj:136)
at nrepl.middleware.print$send_streamed.invoke(print.clj:122)
at nrepl.middleware.print$printing_transport$reify__8149.send(print.clj:173)
at cider.nrepl.middleware.track_state$make_transport$reify__17923.send(track_state.clj:228)
at nrepl.middleware.caught$caught_transport$reify__8184.send(caught.clj:58)
at nrepl.middleware.interruptible_eval$evaluate$fn__8250.invoke(interruptible_eval.clj:132)
at clojure.main$repl$fn__9121.invoke(main.clj:460)
at clojure.main$repl.invokeStatic(main.clj:458)
at clojure.main$repl.doInvoke(main.clj:368)
at clojure.lang.RestFn.invoke(RestFn.java:1523)
at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:84)
at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:56)
at nrepl.middleware.interruptible_eval$interruptible_eval$fn__8258$fn__8262.invoke(interruptible_eval.clj:152)
at clojure.lang.AFn.run(AFn.java:22)
at nrepl.middleware.session$session_exec$main_loop__8326$fn__8330.invoke(session.clj:202)
at nrepl.middleware.session$session_exec$main_loop__8326.invoke(session.clj:201)
at clojure.lang.AFn.run(AFn.java:22)
at java.lang.Thread.run(Thread.java:748)
>>> # So let's call our new clojure fn
>>> clj_fn(1, 2, 3, 4, "Embedded Clojure FTW!!")
Printing from Clojure: [1 2 3 4 Embedded Clojure FTW!!]
>>>
</code></pre>
<h2><a href="#loading-and-running-a-clojure-file-in-embedded-mode" name="loading-and-running-a-clojure-file-in-embedded-mode"></a>Loading and running a Clojure file in embedded mode</h2>
<p>We can runs as well a .clj file in embedded mode. The following does this without an interactive pytho shell, it just runs the provided clj file with <code>clojure.core/load-file</code></p>
<pre><code class="bash">python3 -c 'import cljbridge;cljbridge.load_clojure_file(clj_file="my-file.clj")'
</code></pre>
<h2><a href="#are-you-not-entertained-" name="are-you-not-entertained-"></a>Are You Not Entertained???</h2>
<p>So there you have it, embedding a Clojure repl in a Python process and passing data in between these two systems. This sidesteps a <em>ton</em> of issues with embedding Python and provides another interesting set of possibilities, essentially extending existing Python systems with some of the greatest tech the JVM has to offer :-).</p>
<ul>
<li><a href="https://github.com/LeeKamentsky/python-javabridge">javabridge github</a></li>
<li><a href="https://github.com/clj-python/libpython-clj/blob/94c72ca0ac94b210a9b126805cd4112024ad0b96/cljbridge.py">libpython-clj cljbridge.py</a></li>
<li><a href="https://clj-python.github.io/libpython-clj/libpython-clj2.embedded.html">libpython-clj embedded docs</a></li>
<li><a href="https://github.com/tristanstraub/blender-clj/">blender-clj - the inspiration</a></li>
</ul></div></div></div></body></html>