The general idea behind the fix is that quoted literals need to be treated as data. Arc now has two new functions for this purpose: quoted and unquoted.
The fact that (quote {a 1}) now becomes a hash table is a little strange. I’m not entirely sure that’s correct behavior. It depends whether (car '({a 1})) should yield a hash table. It seems like it should, which is reified in the code now.
EDIT: Ok, I've force-pushed the fixed commit. (Sorry for the force-push.)
If you `git reset --hard HEAD~1 && git pull` it should work now.
Personally, I found I prefer racket's pretty-printing with the horrible hash tables compared to something like {pipe "water" a 1 b 2 c ...} because if you try to evaluate `items` or `profs` you won't have a clue what the data is without pretty-printing.
And it turns out I suck at writing pretty-printers. Someone else do it!
You may have noticed one of elisp's interesting features: closures are printed out.
> (define (adder n)
(lambda (x)
(+ x n)))
(closure (t) (n) (function (lambda (x) (+ x n))))
> (adder 42)
(closure ((n . 42) t) (x) (+ x n))
That means you can theoretically write and read closures to/from disk.
Anyone remember the "Unknown or expired link" errors in the old days of Hacker News? Arc generates closures on the fly, then stores them on the server keyed by a random ID. It sends this random ID down to the browser as links: https://www.laarc.io/x?fnid=ls1wNQuImEEYyvWtGKs1Sj. When the user visits the link, arc looks up the closure and calls it. This effectively allows Arc code to pause computation and then resume later. It's kind of like traditional node-style fs.readFile callbacks, but way more interesting because Arc uses macros to make it feel very natural to write server-side code in this style. You don't feel like you're writing nested callbacks. I've never caught myself wishing for async/await, for example.
Now, the drawback of this technique is that it starts to consume a lot of memory. The closures don't need to be stored forever, but they do need to be stored for a reasonable amount of time. Arc solves this by "harvesting" the closures, meaning it walks through the global closures table and casts out the old ones. Like a startup. (Hmm.)
The other drawback is that a server reboot will wipe all the closures. That's why you'd sometimes get "Unknown or expired link" in the middle of the day on HN. The memory usage got pretty extreme, and a periodic reboot was a quick fix (that no doubt made pg wince each time he ^C'd the server).
So, emacs lisp is very interesting here, because it illustrates that it ought to be possible to store Arc's dynamic closures on disk rather than in memory. That would solve the problem completely: you can generate as many functions as you want, and you don't need to worry about a thing till you blow through >10GB of disk space.
The best part is that the server should be able to reboot without losing the closures.
All the functions in news.arc operate either on objects in memory, like users, or on ephemeral objects, like requests. (The arc server creates a new `request` instance for each incoming connection, and it stores things like the user's cookies and the query arguments.) Both of these can already be serialized straight to disk. And the dynamic closures are almost always generated in the middle of building an HTML page -- i.e. you want to describe "build a form; here's what to do when the user submits it". That latter half is a function, which becomes a dynamic closure keyed by fnid. The form is sent straight down to the user's browser. When the form is submitted, the server picks up right where it left off. That means you can interweave your "view" code and your "model" code, in the MVC sense. And there's no need for a controller; the controller is the closure, which knows what to do thanks to lexical context.
HN eventually solved the fnid problem by getting rid of them. You'll rarely see any dynamic closures on the site. My hypothesis is (a) it's faster to write features using the fnid technique, and (b) the fns can be serialized to disk.
(a) seems true, and (b) is worth exploring. As https://www.laarc.io/ gains momentum, we're hoping to preserve this technique. If it works out, users shouldn't notice any "Unknown or expired link" messages – the closure will be on disk, and they'll last a long time, to put it mildly.
"The best part is that the server should be able to reboot without losing the closures."
Might want to remember to wipe them out if you make certain changes to the code, so that you don't have to think about the effects of running old code and new code in the same system.
"There's one thing you can't do with functions that you can do with data types like symbols and strings: you can't print them out in a way that could be read back in. The reason is that the function could be a closure; displaying closures is a tricky problem."
I've sometimes wondered just what the connotation of 'tricky' is here. Is it hard in some theoretic sense, or just "Arc is a prototype so we don't have this yet"?
(Edit: Oops, I replied out of order and didn't read shawn's comment with the elisp examples before writing this.)
I suspect what pg means by it is primarily that it's tricky to do in Racket (though I'm not sure if it'd be because there are too few options or too many).
Essentially, I think it's easy to display a closure by displaying its source code and all the captured values of its source code's free variables. (Note that this might be cyclic since functions are often recursive.)
But there is something tricky about it, which is: What language is the source code in? In my opinion, a language design is at its most customizable when even its built-in syntaxes are indistinguishable from user-defined ones. So the ideal displayed format of a function would ideally involve some particular set of user-definable syntaxes. Since a language designer can't anticipate what user-defined syntaxes will exist, clearly this decision should ultimately be up to a user. But what mechanism does a user use to express this decision?
As a baseline, there's at least one straightforward choice the user can make: The one that expresses the code in terms of Arc special forms (`fn`, `assign`, `if`, `quote`, etc.) and Arc functions that aren't implemented in Arc (`car`, `apply`, etc.). In a language that isn't aiming for flawless customizability, this is enough.
Now suppose we try to generalize this so a user can choose any set of syntaxes to express things in terms of -- say, "I want the usual language, but with `=` treated as an additional built-in." If the code contains `(= x (rfn ...))`, then the macroexpander at some point needs to expand the `rfn` without expanding the `=`. That's not really viable in terms of Arc's macroexpander since we don't even know `(rfn ...)` is an expression in this context until we process `=`. So this isn't quite the right generalization; the right generalization is something trickier.
I suppose we can have every function printed in terms of its pre-macroexpansion source code along with printing all the macros and other macroexpansion-time state it happened to rely on as it macroexpanded the first time. That would narrow down the problem to how to print the built-in functions. And we could solve that problem in the language design by making it so nothing ever captures a built-in function as a first-class value, only as a late-bound reference to the global variable it's accessible from.
Or we could have the user specify their own macroexpander and make it so that whenever a function is printed, if the current macroexpander hasn't expanded that function yet, it does so now (just to determine how the function is printed, not how it behaves). This would let the user specify, for instance, that `assign` expands into `=` and `=` expands into itself, rather than the other way around.
These ideas are incomplete, and I think making them complete would be pretty tricky.
In Cene, I have a different take on this: If a function is printable (and not all are), then it's printable because it's a callable struct with a tag the printer knows about. It would be printed as a struct. The function implementation wouldn't be printed. (The user could look up the source code information based on the struct tag, but that's usually not printable.) There may be some exceptions at the REPL where information is printed that usually isn't available, because the REPL is essentially a debugging context and the debugger sees all. (Racket's struct inspectors express a similar debugger-sees-all principle, but I haven't seen the REPL take advantage of it.)
You're hitting on a problem I've been thinking about for years. There are a few reasons this is tricky, notably related to detecting whether something is a variable reference or a variable declaration.
(%language arc
(let in (instring " foo")
(%language scm
(let-values (((a b c) (port-next-location in)))
(%language el
(with-current-buffer (generate-new-buffer "bar")
(insert (prin1-to-string c))
(current-buffer)))))))
To handle this example, you'll need to know whether each form is a function call, a variable definition, a list of definitions (let-values), a function call, and which target the function is being called for.
For example, an arc function call needs to expand into `(ar-apply foo ...)`
And due to syntax, you can't just handle all the cases by writing some hypothetical very-smart `ar-apply` function. If your arc compiler targets elisp, it's tempting to try something like this:
(ar-apply let (ar-apply (ar-apply a (list 1))) ...
which can collapse nicely back down to
(let ((a 1)) ...)
in other words, it's tempting to try to defer the "syntax concern" until after you've walked the code and expanded all the macros. Then you'd collapse the resulting expressions back down to the language-specific syntax.
But it quickly becomes apparent that this is a bad idea.
Another alternative is to have a "standard language" which all the nested languages must transpile tO:
(%let in (%call instring " foo")
(%let (a b c) (%call port-next-location in)
(|with-current-buffer| (%call generate-new-buffer "bar")
(%call insert (prin1-to-string c)
(%call current-buffer)))))
Now, that seems much better! You can take those expressions and easily spit out code for scheme, elisp, arc, or any other target. And from there it's just a matter of adding shims on each runtime.
The tricky case to note in the above example is with-current-buffer. It's an elisp macro, meaning it has to end up in functional position like (with-current-buffer ...) rather than something like (funcall #'with-current-buffer ...)
There are two ways to deal with this case. One is to hook into elisp's macroexpand function and expand the macros as you go. Emacs calls this eager macroexpansion, and there are some cases related to autoloading (I think?) that make this not necessarily a good idea.
The other way is to punt, and have the user indicate "this is an elisp form; do not mess with it."
The idea is that if the symbol in functional position is surrounded by pipe chars, then the compiler should leave its position alone but compile the arguments. So
Then you'll be in for a nasty surprise: not only does it look visually awful and annoying to write, but it won't work at all, because it'll compile to something like this:
(let (ar-funcall2 (a 1) (b 2))
(ar-funcall2 _+ a b))
I am not sure it's possible to escape the "syntax concern". Emacs itself had to deal with it for user macros. And the solution is unfortunately to specify the syntax of every form explicitly:
I can't speak to elisp, but the way macro systems work in Arc and Racket, the code inside a macro call could mean something completely different depending on the macro. Some macros could quote it, compile it in their own way, etc. So any code occurring in a macro call generally can't be transformed without changing the meaning of the program. Trying to detect and process other macro calls inside there is unreliable.
I have ideas in mind for how macro systems can express "Okay, this macro call is over; everything beyond this point in the s-expression is an expression." But that doesn't help with Arc or Racket, whose macro systems aren't designed for that.
So something like your situation, where you need to walk the code before knowing which macroexpander to subject each part of it to, can't reliably treat the code as code. It's better to treat the code as a meaningless soup of symbols and parentheses (or even as a string). You can walk through the data and find things like `(%language ...)` and treat those as escape sequences.
(What elisp is doing there looks like custom escape sequences, which I think is ultimately a more concise way of doing things if new macro definitions are rare. It gets into a middle ground between having s-expression soup and having a macro system that's designed for letting code be walked like this.)
Processing the scope of variables is a little difficult, so my escape sequences would be a bit more verbose than your example. It's not like we can't take a Racket expression and infer its free variables, but we can only do that if we're ready to call the Racket macroexpander, which isn't part of the approach I'm describing.
(I heard elisp is lexically scoped these days. Is that right?)
This is how I'd modify the escape sequences. This way it's clear what variables are passing between languages:
(%language arc ()
(let in (instring " foo")
(%language scm ((in in))
(let-values (((a b c) (port-next-location in)))
(%language el ((c c))
(with-current-buffer (generate-new-buffer "bar")
(insert (prin1-to-string c))
(current-buffer)))))))
Actually, instead of just (in in), I might also specify a named strategy for how to convert that value from an Arc value to a Racket value.
Anyhow, once we walk over this and process the expressions, we can wind up with generated code like this:
We also collect enough metadata in the process that we can write harnesses to call these blocks at the right times with the right values.
This is a general-purpose technique that should help with any combination of languages. It doesn't matter if they run in the same address space or anything; that kind of detail only changes what options you have for value marshalling strategies.
I think there's a somewhat more convenient approach that might be possible between Arc and Racket, since their macroexpanders both run in the same process and can trade off with each other: We can have an Arc macro that expands its body as Racket code (essentially Anarki's `$`) and a Racket macro that expands its body as Arc code. But there are some difficulties designing the latter, particularly in terms of Racket's approach to hygiene and its local macros, two things the Arc macroexpander has zero concept of. When we switch from Racket to Arc and back to Racket, the hygiene information and local macro scopes will probably be obliterated.
In your arcmacs project, I guess you might also be able to have an Arc macro that expands its body as elisp code, an elisp macro that expands its body as Racket code, etc. :-p So maybe that's the approach you really want to take with `%language` and I'm off on a tangent with this "escape sequence" interpretation.
No. I feel slightly bad for being so blunt, but Racket's language facilities have absorbed more of my mental time and energy than any other aspect of Racket.
Let's put it this way. Racket is an excellent choice for implementing languages. It has every feature you could want. And yet it is very difficult to do even the simplest things. For example, when an error is raised in Anarki, you'll see a stack trace that points to ac.rkt rather than the actual location within the arc file that caused the error. This is because, frankly, ... Well, let's just say I've redacted some expletives here. But those expletives were aimed at the fact that it's really quite difficult to translate the vision in your mind into an implementation using Racket.
Now, that being said, if you sit down with Racket and take the time to learn it, you will find that it's one of the most robust, flexible, and performant language runtimes in existence. The thread-local variable support is a killer feature. The custodian support is rock solid, which makes sandboxing trivial and super reliable. And it's the premiere implementation of Scheme, so it might continue to have a thriving community for decades to come. But it took me years to become very effective with Racket. (This says more about my own shortcomings than about Racket, though!)
It's a Lisp that runs in both JS and Lua. It's self-hosted, meaning all .js and .lua code is generated by Lumen itself. Diff this technique with the fact that ac.scm is written in Scheme rather than Arc. Very mysterious! I remember how electrifying it felt that first day I stumbled across it and realized the significance of what I was looking at.
It's one of the most incredible projects I've ever seen. It's so small and clear, yet does so much – a perfect example of why simple systems run circles around competitors.
There is a branch that adds Python support at https://github.com/shawwn/lumen/tree/features/python which speaks to Lumen's flexibility and power. (I wonder if there is another Lisp that you can use natively from three different languages without any FFI. And if there is, I doubt it's <3000 lines.)
At this point I use Lumen to explore new languages. E.g. I learned R by implementing a Lumen to R compilation target: https://github.com/sctb/lumen/pull/193
You can also do some things with Lumen that I'm not sure you can do with any other Lisp. I hesitate to claim that, but... Well. Wanna see a magic trick?
https://news.ycombinator.com/item?id=17958650
"[Racket] has every feature you could want. And yet it is very difficult to do even the simplest things. For example, when an error is raised in Anarki, you'll see a stack trace that points to ac.rkt rather than the actual location within the arc file that caused the error."
Is that really "the simplest things"? :-p It seems to me Arc goes out of its way to avoid interleaving source location information on s-expressions the way Racket does. Putting it back, without substantially changing the way Arc macros are written, seems to me like it would be pretty complicated, and that complication would exist whether Arc was implemented in Racket or not. (I think aw's approach is promising here.)
---
It's fun to see Lumen is designed for compiling to other languages even when it's self-hosted. That's always how I figured Arc or pretty much any of my languages would have been written once they were self-hosted.
I've been trying to write Racket libraries that (among other benefits) let me implement Cene in Racket the way I'd like to implement Cene in Cene, which should make it an easier process to port it into a self-hosting Cene implementation. But I certainly don't have it working yet, the way Lumen clearly is. :)
Is that really "the simplest things"? :-p It seems to me Arc goes out of its way to avoid interleaving source location information on s-expressions the way Racket does. Putting it back, without substantially changing the way Arc macros are written, seems to me like it would be pretty complicated, and that complication would exist whether Arc was implemented in Racket or not.
Surprisingly it's possible to get pretty close. The trick is to read-syntax rather than read, and then have ac map over syntax objects properly. At that point it's a matter of using eval-syntax rather than eval.
The takeaway is that the error messages are much, much nicer. I'm talking "the error is at line 213 of app.arc in function init-userinfo" nicer.
It's not perfect. And to get to perfect, you'd have to do as you say and rework how macros behave. But it's maybe 90% of the benefit with little work.
That's really cool, thanks for making the case for Lumen. I now have it on my todo list to determine how much the standard library of Lumen matches the names Arc uses.
On a slight tangent, ycombinator.lol is not affiliated with Y Combinator, and https://github.com/lumen-language is not affiliated with Scott Bell the creator of Lumen. I clarify this because it took me a while to figure out. Am I characterizing it right, Shawn?
> I now have it on my todo list to determine how much the standard library of Lumen matches the names Arc uses.
Lumen looks awesome. I was looking for the docs to determine how much of arc actually exists within lumen, but I couldn't find anything. So if you do this, please let me know.
Also, If I can get some time down the road, I'd like to implement some basic dom manipulation functions. Personally I see Lumen as the best means to do mobile app development in Arc (which is probably one of the best things that can happen for Arc IMHO). Arc on the server side, Lumen on the client side and a code base that's useable by both would be really nice.
I think so. It definitely had the same style as https://news.ycombinator.com/login in the sense that it was a plain html page consisting solely of textarea inputs. And YC launched in 2005, which was long after pg began work on Arc in 2001. http://paulgraham.com/arcll1.html
I once replicated the YC application by writing it in Arc (similarly to how we extended Hacker News at http://laarc.io/), and the resulting app looked mostly identical in style without much tweaking.
Wow. Freaky fast. Thanks!
I was thinking of going even further and finding out if arc could output tables and read in tables in that structure?
{todo:({id 1 name "get eggs" done nil} {id 8 name "fix computer" done t})}
looks so much better than the #hash() equivalent and this gets extreme with nested tables. It's also much easier to think through a table structure writing it out.
It would be important to ensure that circular structures don't cause an infinite loop, and I'd be nervous about straying too far from Racket's `write` facility. For better or worse, it's a limitation of racket that you can't `read` a table you've written. But it could be worth doing.
It looks like that's thanks to a March 8, 2012 commit by akkartik (https://github.com/arclanguage/anarki/commit/547d8966de76320...)... which, lol... Everything I was saying in a couple of recent threads about replacing the Arc reader to read mutable tables... I guess that's already in place. :)
I think you should have to quote them. Like how you have to quote lists:
'(foo bar)
is just a list, but
(foo bar)
will either call the function foo or error if it doesn't exist.
So in a similar way
{foo "bar"}
should give a syntax error, but maybe it can have some kind of semantic meaning later. I've been considering that square brackets could be used for assignment/local binding, to cut down on the need for LET (not necessarily in arc just in lisp in general).
I don't agree. quoting a list is a way to protect the expression from evaluation, in this case because round brackets normally indicate an expression that needs to be called. A table literal {...} doesn't need protection from evaluation as a callable expression as it's just data and like any other data it should evaluate to itself. And, frankly, it would really suck having to protect that data everywhere in my code because someone wants a really nuanced use case to work.
Really what should happen is that [] should be implemented such that we don't need to protect lists of data.
>For better or worse, it's a limitation of racket that you can't `read` a table you've written. But it could be worth doing.
I've been trying to get the tables generated by the personal data link in news to export as JSON for a while now[0]. Part of the problem seems to be related to this - Racket doesn't seem to know what to do with #tagged tem or #hash (much less nils.)
When I implemented Arc in Lumen (more info on Lumen: http://arclanguage.org/item?id=20935), this technique was ported over naturally. You can play with it here:
git clone https://github.com/lumen-language/lumen
git checkout 2018-10-29/arc
npm i
rlwrap bin/lumen-node # rlwrap is optional
> (load "arc.l")
function
> (mac awhen (cond . body)
`(let it ,cond
(when it ,@body)))
> (awhen 'hi (print it))
hi
> (awhen nil (print 'hi))
> (awhen 1 (print it))
1
> (awhen 1 (print it)
(let-macro ((awhen (cond . body)
`(let it ,cond
(unless it ,@body))))
(awhen false (print it)))
(awhen 42 (print it)))
1
false
42
>
It's not quite a complete port, but it's pretty close. It also runs on node, so you can use any npm library in Arc.
> I just wanted to throw together a quick demo of the Arc tooling I've been working on.
To what end, might I ask? i.e. are you planning to contribute this to the arc community? Is this something the community can leverage with their own tooling?
I could have other questions, but if it's not something I can use or look at then they become somewhat pointless (the questions, that is).
