"ClojureScript journey: from little script, to CLI program, to AWS Lambda function" Alexander Khokhlov

Alexander Khokhlov
@nots_ioNots.io
ClojureScript journey
From little script,
to CLI program,
to AWS Lambda function

README.md
Code Comments
JavaDoc
GDocs
Confluence / DokuWiki / Wiki system

Nots.io
Add docs for block of code,
function, module, ﬁle,
commit or branch
01

Notes, Tied
to Code
Easy to Discover
Easy to Explore
Easy to Get Scope
Easy to Ask and Discuss
01 • Nots.io

We Track
Relevance
You always know what’s fresh
and what’s not.
Promotes keeping docs
up-to-date.
Rewarding when everything is ✅
01 • Nots.io

Discuss with
your Team
You won’t loose a dispute
that is written down.
It’s tied and has context
01 • Nots.io

And many
more
Integration with GitHub
IDE/Editors plugins
Markdown formatting
@mentions
GitHub PR as a Note
Attachments
One-on-one conversations
…
01 • Nots.io

02
https://medium.com/basecs/reading-code-right-with-some-help-from-the-lexer-63d0be3d21d
The task:
Detect comment scope

02 • The task
Analyze!
AST is the best
But tokens are good enough

Tokenizers
ANTLR 
Pygments 
vscode-textmate
02 • The task

vscode-textmate
02 • The task
var vsctm = require('vscode-textmate');
var registry = new vsctm.Registry({
loadGrammar: function (scopeName) {
var path = ‘./javascript.tmbundle/Syntaxes/JavaScript.plist';
if (path) {
return new Promise((c, e) => {
fs.readFile(path, (error, content) => {
if (error) { e(error); } else {
var rawGrammar = vsctm.parseRawGrammar(
content.toString(),
path);
c(rawGrammar);
}});});}
return null;
}});
// Load the JavaScript grammar and any other grammars included by it async.
registry.loadGrammar('source.js').then(grammar => {
// at this point `grammar` is available...
var lineTokens = grammar.tokenizeLine(
'function add(a,b) { return a+b; }');
for (var i = 0; i < lineTokens.tokens.length; i++) {
var token = lineTokens.tokens[i];
console.log('Token from ' + token.startIndex +
‘ to ' + token.endIndex);
}
});
Sample

vscode-textmate
02 • The task
{ tokens:
[ { startIndex: 0,
endIndex: 1,
scopes:
[ 'source.js',
'string.quoted.single.js',
'punctuation.definition.string.begin.js' ],
text: "'",
line: 0 },
{ startIndex: 1,
endIndex: 11,
scopes: [ 'source.js', 'string.quoted.single.js' ],
text: 'use strict',
line: 0 },
{ startIndex: 11,
endIndex: 12,
scopes:
[ 'source.js',
'string.quoted.single.js',
'punctuation.definition.string.end.js' ],
text: "'",
line: 0 },
{ startIndex: 12,
endIndex: 13,
scopes: [ 'source.js', 'punctuation.terminator.statement.js' ],
text: ';',
line: 0 },
Output

Tool which works
great with sequences
02 • The task
🤔

03 • CLJS
Dialect of LISP
Dynamic
Immutable
Persistent
Compiled to JS
Homoiconic
Data-Driven

03 • CLJS
98 functions to work
with collections🔥
Isn’t that enough?

03 • CLJS
"It is better to have 100 functions
operate on one data structure than 10
functions on 10 data structures."  
—Alan Perlis

03 • CLJS
Interop
;; Globals
;; alert("Hello!")
(js/alert "Hello!")
;; Function Call
;; "string".toUpperCase()
(.toUpperCase "string")
;; Properties
;; "string".length
(.-length "string")
Interoperability
with JS

03 • CLJS
Interop
;; Chain calls
;; "string".toUpperCase().charCodeAt(1).toString()
(.toString (.charCodeAt (.toUpperCase "string") 1))
(.. "string" (toUpperCase) (charCodeAt 1) (toString))
(.. "string" toUpperCase (charCodeAt 1) toString)
(-> "string" .toUpperCase (.charCodeAt 1) .toString)
;; Chain properties
;; document.body.lastChild.innerHTML.length
(.. js/document -body -lastChild -innerHTML -length)
(-> js/document .-body .-lastChild .-innerHTML .-length)
Interoperability
with JS

03 • CLJS
Interop
(ns myapp)
(defn ^:export func [a]
(str "Hey, " a))
;; in JS:
;; myapp.func("NodeUA!");

Tokenize
04 • How we did it

vscode-textmate
(require '[cljs.build.api :as b])
(b/build "src"
{:main 'notsapp.citation.core
:optimizations :simple
:target :nodejs
:npm-deps {:vscode-textmate “4.1.1”}
:install-deps true
:output-to "notsapp_citation.js"})

Get tokens
(ns notsapp.citation.registry
(:require [vscode-textmate :as vstm]
[cljs-node-io.core :as io]
[cljs-node-io.fs :as fs]))
(def reg (new vstm/Registry))
(defn load-all-grammars []
(->> (fs/readdir “grammars")
(filter #(re-find #".json$" %))
(map
#(let [grammar-path (str "grammars/" %)
grammar (io/slurp grammar-path)]
(->> (vstm/parseRawGrammar
grammar grammar-path)
(.addGrammar reg))))))
(defn tokenize-file [file source scope-name]
(when-let [grammar-promise
(.grammarForScopeName reg scope-name)]
(.then grammar-promise
#(.tokenizeLine % source)))))

(defn common-block-scopes [input-tokens]
(let [last-idx (-> input-tokens count dec)]
(->>
input-tokens
(keep-indexed
(fn [idx token]
(when (or
(re-find
#"((r)?n){2,}"
(-> token
:text
(str/replace #"[ t]+" "")))
(= idx last-idx))
idx)))
(cons 0)
distinct
(partition 2 1)
(map (fn [[start end]]
(subvec input-tokens (inc start) end))))))
Transform
stream of tokens

05 • Transform to CLI
Transform to CLI
deps.edn
{:deps {org.clojure/clojure {:mvn/version "1.9.0"}
org.clojure/core.async {:mvn/version "0.4.490"}
org.clojure/clojurescript {:mvn/version "1.10.439"}
cljs-node-io {:mvn/version "1.1.2"}
org.clojure/tools.cli {:mvn/version "0.4.1"}
}}

Transform to CLI
clojure.tools.cli
(ns notsapp.citation.core
(:require [cljs.nodejs :as nodejs]
[clojure.tools.cli :as cli]))
(def cli-options
[["-l" "--lang LANG" "Language of a source code file passed via stdin"]
["-s" "--scope LINENUMBER" "Get the scope by given line number"
:parse-fn #(js/parseInt %)]
["-c" "--comments" "Show comments scopes"]
["-h" "--help"]])
(defn -main [& args]
(let [opts (cli/parse-opts args cli-options)
file (-> opts :arguments first)
lang (-> opts :options :lang)
scope-line-number (-> opts :options :scope)
show-comments? (-> opts :options :comments)]
(.exit nodejs/process 0))
(set! *main-cli-fn* -main)

Transform to CLI
stdin
(defn read []
(.on
stdin
"readable"
(fn on-readable []
(let [string
(loop [buf (.alloc js/Buffer 0)]
(if-let [data (.read stdin)]
(recur (.concat js/Buffer #js [buf data]))
;else
(.toString buf "utf8")))]
(.removeListener stdin "readable" on-readable)
string))))

Transform to CLI
stdout
(ns notsapp.citation.stdout
(:require [clojure.string :as str]
[cljs.nodejs :as nodejs]))
(def stdout (.-stdout nodejs/process))
(defn write [data]
(let [buf (.from js/Buffer data)
data-len (.-length buf)
len-buf (.alloc js/Buffer 4)]
(.writeUInt32BE len-buf data-len 0)
(->> #js [len-buf buf]
(.concat js/Buffer)
(.write stdout))))

Compile & exec
> clj build.clj
> node notsapp_citation.js
(b/build
"src"
:target :nodejs
:install-deps true
:output-to "notsapp_citation.js"})build.clj

Surprisingly
simple to
transform
; WAS
;(set! *main-cli-fn* -main)
;NOW
(set! (.-exports js/module) #js {:scopelambda scopelambda})
(defn scopelambda [event ctx cb]
(if-let [body (.parse js/JSON (.-body event))]
(cb
nil
#js {:statusCode 200
:headers #js {"Content-Type" "text/plain"}
:body "Hey There!"})
;or else return BAD REQUEST response
(cb
nil
#js {:statusCode 500
:headers #js {"Content-Type" "text/plain"}
:body "Cannot parse request body"})))
06 • AWS Lambda

Compile
build.clj
(b/build
"src"
:target :nodejs
:install-deps true
:output-to "notsapp_citation.js"})
$ clj build.clj
06 • AWS Lambda

Deploy & exec
$ serverless deploy
serverless.yml
package:
include:
- notsapp_citation.js
- node_modules/**
- grammars/**
exclude:
- src/**
- .git/**
- out/**
functions:
citation:
handler: notsapp_citation.scopelambda
$ serverless invoke -f citation -l
06 • AWS Lambda

07 • Testing
Testing
Entry point
(ns notsapp.citation.core-test
(:require [cljs.test :as t]
[cljs.nodejs :as nodejs]
[notsapp.citation.js-test]))
(nodejs/enable-util-print!)
(defn -main [& args]
(t/run-tests
'notsapp.citation.js-test))
(set! *main-cli-fn* -main)

Testing
The test
(ns notsapp.citation.js-test
(:require [cljs.test
:refer-macros [deftest is]
:as t]))
(t/use-fixtures
:once
{:before load-all-grammars})
(deftest js-scopes-arrow
(is (= (js-scope-arrow-funciton)
[[ 2, 2 ], [ 4, 5 ], [ 7, 8 ]])))
07 • Testing

Testing
Async tests
(deftest js-scopes-arrow
(t/async
done
(->
(tokenize-and-prepare-file
"samples/js/arrow-functions.js")
(.then
(fn [tokens]
(let [scope (js-scope-arrow tokens)]
(is (= scope [[2 2] [4 5] [7 8]])))
(done))))))
07 • Testing

Testing
Build & Execute
> clj build.clj
> node notsapp_tests.js
(b/build
(b/inputs "test" "src")
{:main 'notsapp.citation.core-test
:optimizations :none
:target :nodejs
:output-to "notsapp_tests.js"
:npm-deps {:vscode-textmate "3.3.3" }
:install-deps true
:output-dir "out_tests"})
07 • Testing

08 • Final thoughts
Pros
Simple, Elegant & Readable
Easy to reason about
Small compassable libraries, not frameworks
Rich collection manipulation functions
FP, immutability, purity, ﬁrst-class functions
High level data manipulation
CLJ/CLJS code reuse
Macros
Seamless interop with JS/Node.js

Cons
Learning curve, but it’s just an initial hump
Need to study FP
Good to deep dive into CLJ philosophy
Additional JS code added by CLJS itself
Still tangled error messages
Compiled code is hard to read
Need time to fully master the language
Know what you do

Best Fit
If the task is dedicated
If the team is skilled enough
If you see limit of your current stack
If you’re curious enough
If you want to broaden yours horizons
If you want to be 10x productive
In our opinion

Alexander Khokhlov
point@nots.io
Nots.io
nots.io
blog.nots.io
@nots_io
facebook.com/nots.io

"ClojureScript journey: from little script, to CLI program, to AWS Lambda function" Alexander Khokhlov

More Related Content

What's hot

Similar to "ClojureScript journey: from little script, to CLI program, to AWS Lambda function" Alexander Khokhlov

More from Julia Cherniak

Recently uploaded

"ClojureScript journey: from little script, to CLI program, to AWS Lambda function" Alexander Khokhlov