We've just had a long exchange about you creating boxes at assignment time and me using compiling-a-reference time instead. Here's a recap:
---
You: Here's how you do it. The definition of "=" is the same: if the variable exists, mutate it, otherwise create a new variable. But now you add in a new primitive called "var"[...]
Me: The behavior I'd use is that any compile-time variable access (even under a lambda) creates a new, uninitialized variable binding if a binding doesn't already exist.
You: Yeah I'd do that too, if I wanted to graft dynamic variables onto a hyper-static system. But since Arc uses dynamic variables, I proposed to graft hyper-static onto it instead.
Me: How do you make the even/odd code work? Under the approach you described, the first line refers to an undefined variable (odd), and I interpret that as an error. I was recommending a fix.
You: Easy: I have a macro called "defs" that handles mutual recursion
Me: While I appreciate 'defs, it's a non-answer. The even/odd example [...] should work without modification.
You: It will work in my system as well, because undefined symbols automatically create new boxes.
How it works is, anytime the compiler sees an undefined symbol, it creates a new box for it like as if it had been created with "var".
Another way to think about it is... the compiler would replace this:
(= foo (fn () ... bar ...))
(= bar (fn () ... foo ...))
With this:
(var bar)
(var foo)
(= foo (fn () ... bar ...))
(= bar (fn () ... foo ...))
What happened is, when it encountered the undefined variable "bar", it created a new box for it. Then it encountered the undefined variable "foo", so it created a new box for it. Then it did the assignments like normal.
Given how you said "compiling-a-reference-time", I think we're talking about the same thing. Why did you mention assignment time?
I mostly followed along, but I don't understand "It will work in my system as well, because undefined symbols automatically create new boxes." You were talking about having them create new boxes at assignment time, and I was recommending compiling-a-reference-time instead so that we don't get an unbound variable error in the first definition.
"While I appreciate 'defs, it's a non-answer. The even/odd example I posted and the evenp/oddp example dido posted are idiomatic Arc code. While you and I don't care much about Arc compatibility, it's something dido wants for Arcueid, so these examples should work without modification."
For this example, let's suppose there was a file "foo.arc" that contained idiomatic Arc code that implements evenp/oddp. This code works in Arc 3.1. It will work in my system as well, because undefined symbols automatically create new boxes. Basically, it'll work, but name collisions are possible, just like in Arc 3.1.
If you then write a new file "bar.arc" that uses hyper-static idioms (var, defs, etc.), it can import "foo.arc" and everything will work fine. "foo.arc" will clobber any existing evenp/oddp definitions, but "bar.arc" will not clobber "foo.arc". And of course "bar.arc" can use "w/include" and "w/exclude" to prevent "foo.arc" from clobbering things.
If you wanted to make it so that "foo.arc" behaves correctly without needing to use "w/include" and "w/exclude", you would indeed need to rewrite it to use "defs". But it's still usable even without a rewrite. So it's a perfectly graceful degradation.
My system is designed so that it can correctly use all existing Arc 3.1 code, while new code is written with the hyper-static idioms. Then, slowly, old code can be migrated to use hyper-static scope, until eventually you could make Arc purely hyper-static.
There's three issues I see with my proposal:
1) If you're writing Arc code in a hyper-static fashion, you really want "arc.arc" to be changed to be hyper-static. But old Arc code will need the non-hyper-static "arc.arc". I think the simplest solution to this is to have two versions of "arc.arc", one that uses hyper-static scope, and one that doesn't. Then you would need to make sure to load the non-hyper-static version before loading Arc 3.1 code. This could be automated a tiny bit by using a macro, something like "w/arc3".
2) "load" occurs at run-time, which is why my definition of "w/include" needed to use "eval". Nulan doesn't have this problem because file importing occurs at compile-time. Perhaps the best way to solve this is to keep "load" as-is, and add in a new "import" macro that does all its work at compile-time.
3) If you think (eventually) making Arc purely hyper-static is a bad thing, you won't like my proposal.
---
"Am I getting this right? This sounds very workable. :) And whaddayaknow, Nulan works. ^_-"
Yes, that's more or less correct. The one detail that's different is... Nulan doesn't have a "get-variable-box" function. The reason is because "quote" internally uses (the equivalent of) "get-variable-box". So in Nulan, rather than using "get-variable-box", you'd just use "quote". And if you want to break hygiene, you'd explicitly use the "sym" function.
Now to build a high-level import system on top of this, the kind dido is looking for. I'll assume we're restricted to interned symbols.
Just before loading arc.arc, the initial symbol replacement rule is the identity function, and the initial environment contains only the builtins. Each builtin value is located in its own compilation box (no sharing!) and filed under its familiar name as an interned symbol. For instance, the + procedure is filed under the "+" symbol, with no prefixing.
Just before we go to the REPL or the main program file, we take a snapshot of the current global environment and the current symbol replacement rule.
Suppose we want (import foo) to do an unqualified import of everything in foo.arc, including the things it depends on. This will be very much like (load "foo.arc"), but foo.arc will see only the core utilities, not our additional definitions. Here are the steps:
1) Create a new global environment based on the core snapshot.
2) Generate a unique symbol prefix like "ns/foo/1-".
3) Create a new symbol replacement rule which checks whether or not the symbol exists in the core global environment. If it does, it's returned as-is. Otherwise, the prefix is attached.
4) Process each command in foo.arc as described above, using the created environment and replacement rule as context. Then come back to our original context.
5) Replace our current global environment with a function that tries the final foo.arc environment first and our preexisting environment second. (This lookup won't affect run time efficiency, since we use compilation boxes.)
6) Replace our current symbol replacement rule with a function that checks whether or not the symbol exists in the final foo.arc environment. If it does, the function defers to the final foo.arc replacement rule. Otherwise, it defers to our preexisting replacement rule. Now, if we write "bar" and foo.arc defines bar, it'll rewrite to "ns/foo/1-bar", which is part of our new environment.
If we want to do a qualified import (import-as foo Foo) instead, then step 6 is different:
6) Replace our current global environment again so that "Foo" maps to the final foo.arc symbol replacement rule. Now, if we write "Foo::bar" and foo.arc defines bar, it'll rewrite to "ns/foo/1-bar", which is part of our new environment.
---
Whew! I know this is too much to expect anyone to read and understand it all at once, let alone to implement all at once. ^_^
Here are some things to consider, dido, if you generally like this synthesis of ideas but want to spin it up or simplify it:
In the (import ...) mechanisms I described, we filter the symbol replacement rule by querying the environment, so this mechanism actually requires both notions of first-class namespace. If we provide some other way to filter, such letting each file build its own list of export symbols, then replacement rules could exist as the sole namespace system.
On the other hand, if we use first-class environments, we can simply bind "bar" in our namespace to a compilation box we get from foo.arc, and we don't need to fiddle with replacements like "ns/foo/1-bar" and the ssyntax issues. I think there's much more conceptual simplicity to this approach. Unfortunately, it demands the use of hygienic macros, which undermines Arc compatibility.
After Pauan's comment at http://arclanguage.org/item?id=17449, I think I've figured out a convoluted but surprisingly comprehensive approach Arcueid could take. This would support (most) existing Arc code, including a programming style that still uses Arc's unhygienic macros, while also supporting first-class namespaces and hygienic macros using Pauan's get-variable-box recommendation.
A programmer would not observe it to have a hyper-static global environment, because neither does Arc! Nevertheless, a hyper-static environment could be supported as a compiler option, in the sense of having alternative versions of 'eval, 'load, and the REPL. I think the important part of the hyper-static discussion was the use of first-class namespaces.
What dido's been talking about is a system like Common Lisp's or Clojure's. I haven't used either system firsthand, but it seems they both transform all unqualified symbols in a file by associating them with an implicit prefix, and both languages use unhygienic macros and support large community projects.
In response to this, Pauan was saying something about implementing first-class namespaces by using symbol replacement tables. What I'm about to describe is a spin on that: We use first-class namespaces with Pauan's notion of boxes, as I currently understand it, and we also use symbol replacement tables to stand in for CL-like symbol qualification.
I'm actually going to call these concepts "first-class (global) environments," "compilation boxes," and "(symbol) replacement rules," because otherwise I'd confuse myself. We essentially have two notions of first-class namespace at the same time, a compilation box won't be just any generic kind of box, and I'll actually represent symbol replacement rules as functions, not tables.
---
Okay, here's a comprhehensive overview of the command loop:
1) Read a command as though by using 'read. The result can contain some symbols of the form "Moral::sin", and that's not a problem.
2) Code-walk over the command s-expression. For each symbol:
2a) If the symbol does not contain "::", walk over its non-ssyntax segments and replace them according to the current replacement rule. For instance, the symbol "sin.x" could become "ns/math/1-sin.ns/example/1-x". (The "1" is here in case we want to load the same file multiple times.)
2b) If the symbol begins with a non-ssyntax string followed by "::", look up that string using the current replacement rule and the current global environment, and use that value as the current replacement rule for the rest of the symbol. For instance, the symbol "Math::sin.pi" could become "ns/math/1-sin.ns/math/1-pi", and "Foo::Bar::abc:def" could first look up "ns/foo/1-Bar" and then become "ns/bar/1-abc:ns/bar/1-def".
2c) If the symbol contains "::" but has ssyntax before that, report an error. The precedence is unclear, and it's pretty much unimportant, since the programmer can almost always write out their code without using ssyntax at all.
3) Macroexpand the resulting s-expression using the usual Arc semantics. If programmers inspect the s-expressions they're manipulating here, they'll see things of the form "ns/bar/1-abc:ns/bar/1-def", but that's fine.
4) Compile the results of macroexpansion--possibly as the macroexpander goes along, as Arc 3.1 does internally.
4a) When compiling a (get-variable-box ...) form or global variable reference, look up that name in the current global environment. If it doesn't exist, create a new compilation box with a unique ID, and entirely replace the current global environment with another environment that has this binding. The compilation box's ID isn't necessarily related to symbols; it could be a numeric index or a JavaScript identifier, or whatever else the compiler will need.
4b) When compiling a literal compilation box or a global variable reference, use the compilation box's ID as part of the compilation result.
5) Execute the compiled code.
---
As usual, the programmer gets to run code during macroexpansion and execution. At this point, the programmer needs access to several low-level builtins in order to take full advantage of the system. Here are some reasonable builtins we could provide:
- Create and manipulate first-class environments. An environment can just be (a function that takes (a symbol) and returns (a zero- or one-element list where the element is (a compilation box)). This representation offers no way to view the set of bound variables, but neither does Arc 3.1.
- Get, set, or dynamically bind the global environment.
- Create and manipulate a symbol replacement rule. A symbol replacement rule can be just be (a function that takes (a non-ssyntax symbol) and returns (a non-ssyntax symbol)).
- Get, set, or dynamically bind the current symbol replacement rule.
- Dynamically evaluate an s-expression using a particular global environment and a particular symbol replacement rule, and return three things: The final global environment, the final symbol replacement rule, and the evaluated result.
---
Sometimes (in parts 2a and 2b of the command loop), replacement symbols will be inserted in the midst of a single ssyntax symbol. When the replacements are interned symbols, we can just use string manipulation for this. But what if they're uninterned? We have a few different options for dealing with this:
1) It just won't happen, because all symbols in the language are interned. Even the gensyms returned by (uniq) are interned. This is true in Arc 3.1, Jarc, and Rainbow, but Anarki "fixes" it.
2) Uninterned symbols exist, but symbol replacement rules must always return interned symbols anyway, just so that they can take part in ssyntax.
3) Replacement symbols can be uninterned, but there's a dynamic error if these replacements would end up taking part in ssyntax.
4) Interned and uninterned symbols exist, but there's also a third category of symbols with some rigid nested structure. Given an interspersed list of symbols and ssyntax operators, we can construct a symbol that will execute that ssyntax properly, even if some of the original symbols are gensyms.
5) Interned and uninterned symbols exist, and every uninterned symbol is associated with an arbitrary ssexpansion result (usually itself). (In Racket versions of Arc, a weak table would suffice to implement this.) If we try to insert uninterned symbols into ssyntax, instead we make a fictional ssexpansion and create a new uninterned symbol that will ssexpand to that.
My favorite options are 3 and 5. It would be nifty to see 5 in other versions of Arc, even if it somehow breaks assumptions I made in my old code.
---
I have to continue this in a separate post. This is the first time I've seen "That comment is too long." XD
"Easy: I have a macro called "defs" that handles mutual recursion"
While I appreciate 'defs, it's a non-answer. The even/odd example I posted and the evenp/oddp example dido posted are idiomatic Arc code. While you and I don't care much about Arc compatibility, it's something dido wants for Arcueid, so these examples should work without modification.
---
I'm about to disagree with myself, but first I want to reiterate and clarify what I was saying at "caught me on a technicality":
For this discussion I don't see a much of a reason to distinguish between macros which insert mutable boxes and macros which insert functions. Either system can pretty much support the other as a special case: We can translate spliced boxes into spliced getter/setter functions, and we can translate spliced functions into spliced functions-in-the-box. Because of that equivalence, these systems share the disadvantage of being challenging to serialize.
If dido considers compilation to be important (do you, dido?), then this hygiene approach might be unsuitable, and thus the use of first-class namespaces might be unsuitable. (As I explained at "match the variables the macro author expected," first-class namespaces make hygiene more important.)
---
"What I'm talking about happens entirely at compile-time using boxes."
Ah. I think you have a point!
For compiling Nulan to JavaScript, I guess the boxes you're using aren't arbitrary getter/setter functions, and they aren't merely some mutable container either; they're globally associated with a JavaScript variable name. When you compile the macroexpansion result and it contains a (get-variable-box ...) form, you decide on its JavaScript variable name at that time. If the macroexpansion result contains a box, you use the attached variable name to compile it to JavaScript. Am I getting this right? This sounds very workable. :) And whaddayaknow, Nulan works. ^_-
I seem to remember understanding this before, when you and I talked about Nulan compilation in depth, but I guess I had to retrace the steps just now.
Anyhow, get-variable-box is fantastic IMO, but first-class namespaces still might not be ideal for Arcueid due to Arc's unhygienic macros.
dido, are you comfortable with breaking existing Arc macro idioms in favor of hygiene?
---
I have a convoluted but surprisingly comprehensive idea of how to integrate get-variable-box into a system that's compatible with unhygienic Arc macros, but I've put it in a separate simultaneous post: http://arclanguage.org/item?id=17464
Actually, it's two separate posts, because it's otherwise too long for the forum. If this becomes a tl;dr scenario, I won't be surprised. ^_^
Whoops, I guess I never posted it here. XD I made it right after we had a thread about yet another one of these languages, and then I chatted about it with Pauan and never returned to it.
Maybe I was waiting until I could document what I meant with the columns. Google Pages disappoints me, because although it'll support a sortable table like this, there's no option to add an explanatory caption. I was thinking of posting an explanation in a comment, but that's not a community-editable part of the wiki, right?
Guess I'll describe my intentions right here. Feel free to migrate these explanations to a better home, and of course, feel free to redesign the table to make it better.
---
Language: The name of the language.
Whitespace: In what way does whitespace matter to the parsing of a program text?
Infix arithmetic: Some users care about expressing their arithmetic-heavy code using traditional infix notation. Does the language syntax tailor itself for this purpose by special-casing particular arithmetic operators? If so, the answer is "Hardcoded." Otherwise, does it provide generic mechanisms so this user can be satisfied? If it doesn't, the answer is "No." If it doesn't provide hardcoded mechanisms but does provide generic ones, what are they?
Affects semantics?: To make it onto this list, the language must have a syntax that transforms into a uniform yet non-semantic nested structure like s-expressions. It must also have some sugar that's explained as an abbreviation of particular kinds of structure. By the time the programmer has a structured representation of the code to work with, are there still visible remnants of these abbreviations? If so, how so?
---
Maybe the infix arithmetic column would be better off as two: A "generic infix" column and a "domain-specific notation" column. Just because a language provides good generic mechanisms doesn't mean it can't hardcode some traditional notations too.
Maybe if the "Affects semantics?" answer is "No", we should add another shade of meaning to indicate that the project actually has no semantics to affect. I think Readable Lisp S-expressions pretty much falls into this category, and so does "Gel: A Generic Extensible Language."
On the other hand, Gel actually saves a lot of detail about the user's input (e.g. which parentheses they used), and any language that uses Gel would probably have to normalize away the unwanted detail in a second stage, possibly in a way that the programmer can notice.
To digress further... Racket allows macros to discover which parentheses were used, but then it treats all parentheses the same after macroexpansion. Do alternate parentheses count as an s-expression abbreviation? If so, I think Racket should make this list, and it should be listed as having an abbreviation layer that affects semantics. (Consider that Racket also has an infix "abbreviation": (a . -> . b) means (-> a b). Not to mention that (a b c) abbreviates (a . (b . (c . ()))) in a wide variety of lisps.)
"But I don't follow the 'affects semantics?' column."
I think it means whether the parsing happens before macros are run or not. In Arc, ssyntax happens after macros, so macros can affect the meaning/behavior of ssyntax. In most Lisps, syntax is read before macros, so macros can't change the meaning.
---
Ah yes, Nulan has changed since that page was added, I'll go in and update it...
Sorry to say, but I don't believe any of this actually lets you use macros before they're defined. Calling the macro's rep will just succeed at expanding the macro, not executing the resulting code (and not suppressing expansion and execution of the arguments).
Ordinary Arc wouldn't need such a macro, since the system I describe isn't purely hyper-static.
---
"The way I understand it, the compiler would create a box for oddp the way an actual binding using var would have, but this box would be empty as it were."
Yes, exactly.
---
"One problem though is what happens when you try to redefine evenp in terms of a new oddp. Without having a way to remove the old bindings of evenp and oddp, this seems to be impossible. One of them will continue to use the old definition of the other no matter what order you define them."
You don't change the bindings. You mutate the box using "=". The box itself stays the same, it just has a different value at runtime.
So if you want to change oddp in such a way that evenp notices the changes, you would say this:
(= oddp ...)
And if you want to change oddp in such a way that evenp doesn't notice the changes, you would say this:
(var oddp ...)
In fact, that's how Nulan defines self-recursive and mutually-recursive functions. Using Arc syntax, this:
(def foo () ...)
Would get macroexpanded into this:
(var foo)
(= foo (fn () ...))
Notice that it first creates the box, and then assigns to it.
"Er, is it the other way around? It looks like at least this post of mine came after the email discussion"
Yes, but the discussion started with the Arc topic and then moved to e-mail and then moved back to the Arc topic.
---
"I admit your approach handles the practical cases."
Well then! I'll consider that "all", since I only care about the practical cases.
---
"Your approach has developers hardcoding filenames within their source code. If a developer wants to use two files of the same name, they must find a way to segregate the files into subfolders, or they must rename a file and invade some library source code to rewrite the filename occurrences. Please let me know if I'm wrong about this. :)"
Yes. It is tied to the filesystem, or website URLs, or Git commits, or whatever. But, the filesystem already enforces a "no two files with the same name in the same folder" rule, so no biggie.
If you wanted to create something that manages dependencies at a more abstract level, that's fine, and you can build it on top of my system, but I personally don't see much use for that (yet).
If your worry isn't about filenames at all, and is simply about putting filenames into the source code, I think the answer is really easy: just do something like RubyGems, where you have a standard file called "dependencies" that imports all the dependencies in the right order. And then when you want to load the library, you'd just load the "dependencies" file. This doesn't require any changes to my system, since it's purely user-convention.
This still allows for putting the dependency information straight into the source code, which is useful for quickie scripts and such. But big projects and libraries would use the "dependencies" convention. And as Ruby showed, this kind of user-convention can be applied after the language is already in use. So it doesn't need to be baked in ahead of time, though there might be some minor transition pain. But I'll worry about that once libraries and projects become big enough that a "dependencies" convention becomes useful.
"In the latter case its little more than taste, and I see that Pauan is doing the same thing."
The reason I numbered them was just to make it easier to navigate the code. As a user, if you see "01 nu.rkt" you know it's loaded first, and that "02 arc.arc" is loaded second. And each one builds on the stuff defined previously, so you can read it in a linear order.
I only did that for the stuff that's automatically loaded when you start up Arc. You'll notice that the "lib" and "app" stuff is un-numbered. And I don't expect user-created code or libraries to use numbers! So I definitely don't take it as far as wart does.
"Well, one of the reasons I was interested in compiling at least the standard arc library was for improving startup speed, say for use as a simple cgi script on a shared host, or a basic command line utility."
Oh! That's what you're talking about! Sure, I've experimented with that too. I think it's possible, but it would require mucking about with Racket's code loader... still, that's not a bad idea.
---
"I suppose its possible that trying to add fexprs to arc would just make things worse, but I was always attracted to the idea of making environments and scope easier to manipulate, and macros fit in better. As it currently is, macros are very separate from the rest of the language and stick out a little bit. I've rarely had a really good reason to need it, but it just bothers me sometimes that they aren't first class citizens."
Absolutely. I too like the idea of fexprs. I'm just saying that from a purely practical perspective, Arc was not designed around fexprs, so tacking them on after the fact won't work so well. It shouldn't be hard to make a variant of Arc that is built around fexprs. In fact, wart is very Arc-like but has fexprs (or something that's like fexprs anyways).
"The middle ground doesn't seem worthwhile to me."
Retaining Arc compatibility in general doesn't seem worthwhile to me, but a lot of people want it, so I gave a system that retains Arc compatibility while tacking on some new shininess. Nulan doesn't have to worry about Arc compatibility, so it has pure hyper-static scope and hygienic macros by default.
---
"How do you make the even/odd code work? Under the approach you described, the first line refers to an undefined variable (odd), and I interpret that as an error. I was recommending a fix."
Easy: I have a macro called "defs" that handles mutual recursion:
(defs
even (x)
(if (is x 0)
t
(odd (- x 1)))
odd (x)
(if (is x 0)
nil
(even (- x 1))))
The above macroexpands into this:
(var even)
(var odd)
(= even (fn (x)
(if (is x 0)
t
(odd (- x 1)))))
(= odd (fn (x)
(if (is x 0)
nil
(even (- x 1)))))
Basically, it first creates new boxes, and then it assigns the functions to the boxes. This is one of a few reasons why I prefer mutable boxes over immutable boxes. Though, you could probably have "defs" expand to a Y-combinator instead, if you really wanted immutability...
---
"I think you caught me on a technicality. :)"
Maybe it was just a simple miscommunication. What you were talking about sounded exactly like the technique of splicing function values using macros:
What I'm talking about happens entirely at compile-time using boxes. The effect is very similar, but the implementation is very different.
---
"Anyway, I'm a fan of that approach when it works, but it doesn't work so well when compilation is involved: The macroexpanded code contains unserializable values--namely, the procedures or boxes we're talking about"
Sure, if I cared about serialization, I'd have to make it more complicated. Thankfully, the only serialization I care about right now is compiling to JavaScript code, which is easy enough to do with variable renaming.
Also, what's the point in serializing boxes since functions still can't be serialized? If you found a way to serialize functions, then it'd be much more useful to be able to serialize boxes.
Huh. Somehow that rep trick never occurred to me either.
One of my big motivations for first-class macros was to be able to organize my code in any order. If there's no external constraints it increases the odds that a given snapshot will have some reasonable organization that's easy for newcomers to understand. Any constraints just increase the odds of stale organization. I've ranted before about this: http://arclanguage.org/item?id=15587 (footnote 1); https://github.com/akkartik/wart/blob/master/001organization. If macros can be written in any order, that reduces a lot of my need for first-class macros.
"If it's the architects or BAs writing the tests, then they might actually have _more_ information about how the system should be working than the programmers,"
Oh, absolutely. I didn't mean to sound anti-non-programmer.
I tend to distrust labels like 'architect' and 'programmer'. Really there's only two kinds of people who can work on something: those who are paid to work on it, and those who have some other (usually richer and more nuanced) motivation to do so. When I said, "Organizations which separate those two functions tend to gradually suck", I was implicitly assuming both sides were paid employees.
If non-employees (I'll call them, oh I don't know, owners) contribute towards building a program the result is always superior. Regardless of how they contribute. They're just more engaged, more aware of details, faster to react to changes in requirements (which always change). Your idea sounds awesome because it helps them be more engaged.
But when it's all paid employees and you separate them into testers and devs, then a peculiar phenomenon occurs. The engineers throw half-baked crap over to testers because, well, it's not their job to test. And test engineers throw releases back at the first sign of problems because, well, it's not their job to actually do anything constructive. A lot of shuffling back and forth happens, and both velocity and safety suffer, because nobody cares about the big picture of the product anymore.
(Agh, this is not very clear. I spend a lot of time thinking about large organizations. Another analogous example involves the patent office: http://akkartik.name/blog/2010-12-19-18-19-59-soc. Perhaps that'll help triangulate on where I'm coming from.)
(BTW, I've always wondered: what's that cryptic string in your profile?)
Organizations which separate those two functions tend to gradually suck
Hmm... Well, that's certainly a valid opinion, and it may even be true in a lot of cases, however I think the issue is largely due to two other related issues: 1) The requirements aren't specified clearly enough, and are dissociated from the tests as well, and 2) they just don't have good enough tools.
Tests can serve many purposes. The most basic, given the fundamental concept of a test, is to tell you when something works, or when it doesn't. TDD focuses on the former, unit testing and regression testing focus more on the later. Tests can be used at different points in the development cycle, and if you use the wrong kind at the wrong point, it's not really all that helpful.
My concern is that its too difficult to write the correct kind of test, so most developers use the wrong kind or don't use any at all. There's nothing really wrong with that, I think it's just an unfortunate inefficiency, like trying to debug without stacktraces. >.> Hmm. Something to look forward to going back to arc I suppose. Anyway, my goal is to make testing easy enough to do, either for developers who just want a quick way to check if they broke something after making a 'minor' change, or for larger companies that want to know that all their requirements have actually been met.
So, to solve the first problem I'm hoping to utilize a lot of reflection and code inspection so that at least the outline of the test cases can be generated automatically, if not more. Then it should be really easy for the programmer to just add the missing details, either as specific test vectors or by using a more general definition of requirements using something like QuickCheck's generators.
In the long run the plan is for the tool to be able to support working the other direction, from requirements to tests. Hopefully with better tool support, and more intelligent interaction with the system under test, it should be possible for the architects to specify the requirements, and the tool should be able to verify that the code works.
Yes, divorcing tests from code could mean that different people do them. Doesn't have to be the case, but it becomes a possibility. And that means that they could will have a different perspective on the operation of the system, but not necessarily a worse one. If it's the architects or BAs writing the tests, then they might actually have more information about how the system should be working than the programmers, especially in the case that the programmers are outsourced. At which point allowing someone else to write the tests is an improvement. When developers write the tests, it doesn't help if they use the same incorrect understanding of the requirements for both the tests and the code.
Hopefully by making an easy enough tool that supports rapidly filling in tests based on code analysis (which would help anyone that doesn't know much about the actual code base match it up with the requirements they have) reducing boiler plate and barriers to testing, making it a much easier to use tool for developing. Maybe if it gets easy enough, developers would find that testing actually saves enough time testing to be worth the few seconds specifying test vectors for each method. And if it can do a good enough job at turning requirements into tests in a way that is clear enough to double as documentation, it should save the architects and BAs enough time, as well as make implementation easier for developers, that I might actually be able to sell licenses :P
Pauan, although I disagree with you on precisely how the hyper-static scope primitives should be used to create an actual module system, the idea seems to be general enough that it might actually be worth implementing in Arcueid in more or less the manner which you describe. The only problem I have is what happens with forward references, e.g. such as arise with mutual recursion. To go with the classic example of mutual recursion:
(def evenp (x) (if (is x 0) t (oddp (- x 1)))
(def oddp (x) (if (is x 0) nil (evenp (-x 1)))
Without a special operator, it seems impossible to define something like this in pure hyper-static scope. With the sort of hybrid hyper-static scope you envision, what would compiler do if it got this sort of definition?
The way I understand it, the compiler would create a box for oddp the way an actual binding using var would have, but this box would be empty as it were. If no subsequent definition of oddp followed, that would result in an unbound free variable error at runtime when evenp was used, exactly the way the reference Arc implementation would. If oddp were defined, however, it would then fill in the empty box that the reference to it in evenp created, and so the call to oddp would make use of the first definition of evenp that followed it, just as a non-hyper-static system would. Subsequent definitions of oddp using var would have no effect on evenp though, unless evenp were redefined.
One problem though is what happens when you try to redefine evenp in terms of a new oddp. Without having a way to remove the old bindings of evenp and oddp, this seems to be impossible. One of them will continue to use the old definition of the other no matter what order you define them.
Ah, I was unaware of fitnesse! Thanks for the pointer, that's a really neat idea. Tests are a huge part of wart's goal of 'empowering outsiders'[1].
Sucks that fitnesse is stuck in the java eco-system. Just porting it to lisp/arc/wart would be awesome for starters..
Arguably much of the benefit of testing comes from the same person doing both programming and testing. Organizations which separate those two functions tend to gradually suck. If you accept that, inserting a web interface between oneself and one's tests seems like a bad idea. Perhaps fitnesse-like flows would be best utilized to involve the non-programmer in integration testing, testing the entire site as a whole rather than individual functions. Perhaps script interactions with a (non-ajax for starters!) app so that the CEO/QA engineer doesn't have to know about REST and PUT/GET? Hmm, that would be cool..
Don't mind me, I'm just thinking aloud :)
[1] https://github.com/akkartik/wart/blob/531622db6a/000preface. I spend a lot of time thinking about this, and at half an opening will fill your ears about all the ways in which tests, whilst awesome, aren't sufficient. Along with ideas for complementary mechanisms.
Don't feel like you have to be fair :) The world isn't fair, and I understand about differences in taste. My goal with wart and some other stuff has been to figure out how to empower outsiders to bend a codebase to their will and taste with a minimum of effort. For example, one experiment I'd love to perform on you is to measure how long it takes you to fork wart to toss out infix and get back your beloved hyphens :) But no pressure.
For whatever it's worth, I still technically disagree with the "all" here. I admit your approach handles the practical cases.
Your approach has developers hardcoding filenames within their source code. If a developer wants to use two files of the same name, they must find a way to segregate the files into subfolders, or they must rename a file and invade some library source code to rewrite the filename occurrences. Please let me know if I'm wrong about this. :)
The LtU post also goes over at least one use case where a library user wants to update the dependencies of the library without also updating the library itself. Fortunately, this time I think we can agree that this isn't the problem we're discussing. :) It's something I care about in a module system, but it's not directly related to name collision.
