The bouncing between parens and square brackets is interesting ^_^ I weakly feel it's not worth optimizing for what's easy to type because things can change (or be changed, with keybindings, etc.) so easily. Better to optimize for how things look. But even there, parens vs brackets is in the eye of the beholder.
Your train of thought is very similar to a factor in several of my designs over time: Jisp[1], Blade[2], Penknife[3], Chops[4], and now the syntax I'm planning to use with Era[5].
I've always used use bracket nesting to determine where the syntax's body begins and ends. This way, most code requires no escape sequences, and the few times escape sequences are necessary, at least they stand a chance of being consistent when cutting-and-pasting across staging levels.
(re /\)*/) ; Broken.
(re /(?#()\)*/) ; Fixed with a regex comment. (Does this work?)
(re /\>*/) ; Fixed by redesigning the escape sequence.
Reader macros are a different approach, where the entire rest of the file/stream has unknown syntax until the reader reaches that point. That's simple in its own way, but I prefer not to make my code that lopsided in its semantics I guess. :)
(EDIT: Half an hour after I posted this, I realized I had a big mess of previous drafts tacked onto the end. I've deleted those now.)
---
[1] Jisp was one of the first toy languages I made, and it was before I programmed in Arc (or any other lisp). When it encounters (foo a b c), it resolves "foo" to an operator and sends "a b c" to that operator.
[2] Blade didn't get far off the ground, but it was meant to have a similar parser, with the explicit goal of making it easy to combine several languages into a single compiled program, with an extra emphasis on having no accidental code-order-dependent semantics at the top level. I switched to square brackets-- [foo a b c] --since these didn't require the shift key.
[3] Penknife was meant to be a REPL companion to Blade, and I developed the syntax into a more complicated, Arc-like combination of infix and prefix syntaxes (http://www.arclanguage.org/item?id=13071). Penknife was complete enough that I used it as a static site generator for a while. However, at this point I realized all the custom syntax processing I was doing was really killing the compile-time performance.
[4] Chops is a JavaScript library that achieves Blade-like parsing without any goals for infix treatment or general-purpose programming. I use it as a markup language and a JavaScript preprocessor now that my static site generator runs in the browser.
$rc.rcPage( "/", $cg.parseIn( [str RocketN[i I]A.com] ),
"19-Nov-2012", $cg.parseIn( [str 2005[en]2010, 2012] ),
{ "title": "RocketNIA.com, Virtual Index of Ross Angle",
"breadcrumbs": $cg.parseIn(
[str RocketN[i I]A.com: Virtual Index of Ross Angle] ) },
$cg.parse( [str
((This is the open source version of my site.
[out http://www.rocketnia.com/ The online version] has a bit more
content.))
...
] ) )
[5] Era is a module system I'm making, and I intend to compile to those modules from a lisp-like language. I've switched back to parentheses-- (foo a b c) --because smartphone keyboards tend to omit square brackets. The code (foo a b c) parses as a four-element list of symbols in the hope of more efficient processing, but the code foo( a b c) parses as a single symbol named "foo( a b c)".
That code is pretty rudimentary, but allows low level access to git commands from arc, plus storage and retrieval of arc objects. After my previous comment though, I'll probably change it so that each object gets a separate branch, with 'meta branches' listing which branches to load if necessary.
Wart's tokenizer already knows about backslashes inside strings, so "\"" becomes one token. It seems plausible to try tokenizing in a backslash-aware way everywhere and not just inside strings. Other than that you would have to treat slashes as a delimiter like double-quotes.
It might be an ugly design, but I think it would work, and it would be worth trying out.
I ran into a similar issue with regex syntax when attempting to incorporate it into Arcueid's reader. There seems to be no easy way to parse a regular expression using Perl-like /.../ syntax, not if you also want symbols that use /'s for other things, e.g. the division function. Arcueid thus uses for now, r/.../ for regular expressions, and that syntax could be more easily distinguished from other legitimate uses of symbols with a minimum of fuss.
I'm actually fairly convinced that my original idea as stated above wouldn't work, because without specific syntactical support already built in for whatever new reader you're trying to add, the original reader would have no good way of knowing when your macro ended, i.e.:
(re /\)*/)
How does the original reader know to pass all of "/\)*/" in to the 're macro? Maybe there's some clever way to tell the difference between "escaped" special characters and normal ones, but it would limit the possibilities on the temp reader.
Maybe one option would be to have more flexibility when defining macros in the first place by taking advantage of the fact that macros can be defined and evaluated at the reader stage, so they can specify their own read semantics for their evaluation if they choose. I.e. make it so that macro definitions can specify a more generic level of control than just unevaluated, pre-parsed s-exps. That would make them more of scoped 'reading macros' that shadow the existing reader, rather than 'reader macros' that just hook into it.
I think that some form of scalability would be valuable, but that could easily be achieved with some sort of single threaded worker for each db 'server', and then have multiple instances running to provide the scalability. In order to make the single threaded semantics work well even in a multi-threaded application, I already have a short library for erlang-style pattern matched message passing.
Given the data volumes I've been planning on working with, I mostly want to use the permanent storage for history and fault tolerance, as opposed to live access. That could probably be handled in-memory for the most part. So maybe some form of flat file system would work without causing too many problems.
I originally started using git to effectively achieve that design without having to manage the trees, history, and diff calculation myself, but I've discovered that storing thousands of tiny objects in the git index may not be very efficient. I still think something similar is a good idea, but I would want to separate 'local' version states for each object from the 'global' version, so that it doesn't take forever to save the state of a single object. Maybe storing each object in a git 'branch' with the guid of the object as the branch name would work, since only one object would be in each index. The overhead for saving each object would be slightly higher, but it should be constant, rather than linear with the total number of objects.
Any obvious flaws with that idea that I'm missing? Have any better ideas or foundations to build off of?
I've been working at big newspapers/news companies here in Brazil for many years, but I'm not that happy with these companies strategies, so I decided to try some side project...
I don't know much about programming, so I decided to try something that was really simple. I'm a reader of Hacker News, discovered it's open source, and started looking for how to set up a clone. São Paulo is a really large city (11,32 million people live here), but we lack hubs to connect all these people (I think our news companies aren't doing that very well). So, I came up with this idea of applying the HN concept to a local news project.
To be honest, my main goal here is to learn more about programming and other technical stuff, so, it's been an awesome experience. The site runs on Amazon Linux AMI, and the majority of users are still my friends and the friends of my friends. (I finished the translation work yesterday). :P
It will be a pleasure to share more data with you in the following weeks! And thanks for all the help and support I found here.
PS.: "Espaço" can be translated as "Space", or a generic place. It's a common word in portuguese.
In less than a month marcvs has setup an AWS server, installed racket and arc, tweaked the arc codebase and the HN app, configured email, and translated all the copy in the HN codebase to spanish. All this without having much programming experience at all. Kudos!
So I guess that makes a consistent foreign function interface something very important for Arcueid to start having then. I think I've built up a foreign function API (sorta) and am now working out the details for dynamic loading of shared libraries so you can do something like (load "mysql.so") and have it dynamically link into Arcueid, as well as a way to compile C sources into such an extension shared object.
Hmm, do you care about threading and consistency at all? If not, you could probably do everything with just arc macros over the existing flat file approach..
If arc had support for something like DBA or SQLAlchemy built in, I might just have used it with either postgres or sqlite. However, neither of those databases really fit the arc data model very well, imo, because arc is very hash table and list oriented. Objects have very little in the way of a set schema, and hash tables map pretty well to... hash tables.
Anyway, I mostly want to leave all the objects in memory and use direct references between them; my data relations aren't that complicated, and explicit relations where necessary are actually fairly efficient. In fact, that's what most orm's a la SQLAlchemy seem to do; whenever an object is loaded, you can specify desired relations that also get loaded in memory, so you don't have to explicitly query the database each time.
Memory is cheap these days, and I was hoping for something that allowed versioning and perhaps graph-db features.
I would like to point out that although it's a Pratt parser, it's been specifically modified to work better with Lisps, meaning it operates on lists of symbols rather than on a single expression. I have not seen another parser like it.
This makes it much more powerful while also being much easier to use. Using the Nulan parser, adding in new syntax is as easy as writing a macro!
For instance, in Nulan, the "->" syntax is used for functions:
foo 1 2 3 -> a b c
a + b + c
The above is equivalent to this Arc code:
(foo 1 2 3 (fn (a b c)
(+ a b c)))
And here's how you can implement the "->" syntax in Nulan:
$syntax-rule "->" [
priority 10
order "right"
parse -> l s {@args body}
',@l (s args body)
]
As you can see, it's very short, though it might seem cryptic if you don't understand Nulan. Translating it into Arc, it might look like this:
(syntax-rule "->"
priority 10
order "right"
parse (fn (l s r)
(with (args (cut r 0 -1)
body (last r))
`(,@l (,s ,args ,body)))))
The way that it works is, the parser starts with a flat list of symbols. It then traverses the list looking for symbols that have a "parse" function.
It then calls the "parse" function with three arguments: everything to the left of the symbol, the symbol, and everything to the right of the symbol. It then continues parsing with the list that the function returns.
So basically the parser is all about manipulating a list of symbols, which is uh... pretty much exactly what macros do.
Going back to the first example, the "parse" function for the "->" syntax would be called with these three arguments:
1 (foo 1 2 3)
2 ->
3 (a b c (a + b + c))
It then destructures the 3rd argument so that everything but the last element is in the variable "args", and the last element is in the variable "body":
args (a b c)
body (a + b + c)
It then generates the list using "quote", which is then returned.
Basically, it transforms this:
foo 1 2 3 -> a b c (a + b + c)
Into this:
foo 1 2 3 (-> (a b c) (a + b + c))
As another example, this implements Arc's ":" ssyntax, but at the parser level:
$syntax-rule ":" [
priority 100
order "right"
delimiter %t
parse -> l _ r
',@l r
]
So now this code here:
foo:bar:qux 1 2 3
Will get transformed into this code here:
foo (bar (qux 1 2 3))
I've never seen another syntax system that's as easy and as powerful as this.
Oh yeah, and there's also two convenience macros:
$syntax-unary "foo" 20
$syntax-infix "bar" 10
The above defines "foo" to be a unary operator with priority 20, and "bar" to be an infix operator with priority 10.
Nulan's parser is powerful enough that almost all of Nulan's syntax can be written in Nulan itself. The only thing that can't be is significant whitespace.
Even the string syntax (using "), whitespace ( ), and the various braces ({[]}) can be changed from within Nulan.
Because it's the simplest solution for Arc. Arc does not have any kind of SQL library, and it doesn't even have an officially supported way to drop down to Racket libraries, nor does it have any kind of FFI whatsoever. So flat files are indeed the fastest way to get up and running.
Perhaps in the long run it might be better to have some sort of way to connect to an SQL DB, but keep in mind that Arc is still a work in progress, and unfortunately has not been updated in a long time. For a while, Arc didn't even have Unicode support, because pg was working on more important things.
If you think of Arc as being a prototype, then it all makes sense. pg probably intended to eventually flesh it out into a full language, but I hear he hasn't had the time.
Interesting, although I don't know if it can be done with efficiency in the worst case. Backtracking regexes can take in their worst-case exponential time for matching, as illustrated in the regex in my original post.
I wonder why Arc seems to have stayed away from the traditional SQL databases that have long been used for web apps. There are few problems for which relational databases are inappropriate, and not using one results in reinvention of the wheel which I think is hardly a good thing to be doing. A DBI work-alike by which one can connect to a MySQL, Postgres, or Sqlite database would be a lot simpler to my mind.
Pauan has a Pratt parser baked into his language (https://github.com/Pauan/nulan) while my attitude for wart has been "if you want to change the syntax, hack the parser". Not to be flippant; my goal explicitly is to make the code dead simple for anybody* to hack on. Not there yet, but I'd love for you to take a stab at your regex idea with it :) Perhaps we could pair on it sometime?
* who knows C. Other terms and conditions may apply.
I've thought about this as well for a while. Arc's approach seems to be flat files so far, which means any move to multiple servers is terra incognita on some level[1]. I mean to build a nosql system at some point, but I want to do something besides yet another[2] project. I want to understand[3] why we must couple technology stack choices with where we want to be in CAP space[4]. Why can't we have a single project that lets us tweak one knob for RAM vs persistent store, strong vs eventual consistency, and so on? Laying out the design space in one place may give us some chance at least accumulating lessons as we reinvent the wheel.
One thing I've been thinking about recently, and might be handy for implementing regex or other features would be to allow macros access to the string contents of their original invocation. I've been in favor of meta-code features like that in order to enable help functionality and exploratory programming features, like 'src on anarki, but it might help for allowing users to write their own "reader macros" because, within the scope of the call, any macro can become a reader macro. Then, as long as you have decent string manipulation and parser library, it would be easy to just add a regex library that allows you to call (re /a?/), and add custom regex syntax that way. That or just make a standard way to add special characters to the read macro list like the #* set in racket.
How are you all currently supporting read macros, and what is your opinion on the best way to do so?
I've not yet come across a Lisp dialect that incorporates regexes the way Perl and Ruby do, so I wonder what's a clean way of using them that respects the idioms of Arc. Then there's the matter of taint mode... How would a taint mode in Arc be like?
I take it you're trying to compile Arcueid from sources cloning the repository on Github. There is no configure script in the source repository because that is automatically generated by autoconf. It is a general rule to never include anything in the repository that is automatically generated. Basically, you need to install the autotools (autoconf, automake, and libtool), the dependency headers (most importantly gmp and readline), and then run autoreconf -i in the main source directory. This creates the configure script and everything else it needs.
As I mention in more detail in Arcueid's README:
If you are trying to build this by cloning the Git
repository (git://github.com/dido/arcueid.git), you
need the following prerequisites installed:
- The autotools: autoconf, automake, and libtool.
- GNU MP and any development headers (libgmp and
libgmp-dev on Ubuntu/Debian)
- GNU Check (http://check.sourceforge.net, available
under the package name 'check' on Ubuntu)
- GNU Readline and any development headers
(libreadline-dev)
- pkg-config (http://www.freedesktop.org/wiki/Software/pkg-config,
pkg-config on Ubuntu/Debian)
You need all of this since the process will create a
complete configure script and all the development
dependency headers are required for it to generate the
rules for all the checks configure has to perform. Once
all these prerequisites are installed run 'autoreconf
-i' in the main directory. This should generate a
configure script and Makefile.in's, and from there it
should be possible to do ./configure ; make ; make
install.
Well, there's some interesting stuff in the present git head, such as nested quasiquotes and some stability fixes (stomping out memory leakage and such), although be warned that it isn't always 100% stable.
It would be great if one could easily write in Arc as opposed to Javascript. Nulan and RainbowJS are in a position to infiltrate web-based programming through the browser.
I too wish Arc could be embedded. The discussion on the size of a language core is suspiciously parallel to the discussion on the size of an operating system core.
One more footnote. This article catalyzed the focus on memory leaks for me when I read it back in 2011: http://www.lshift.net/blog/2008/11/14/tracing-python-memory-.... It turned out that the memory leak was caused in a library which defined a specific method (del) that (was documented to) disabled garbage collection in some cases. You look here and the blame goes there, you look there and the blame goes here. Something is very wrong with software; who's in charge here?
