WIP: Allow newlines before where in a function definition

[tkluck]

This allows nicely aligning the wheres, for example

julia> function f(x::I)
                where I <: Integer
           return x
       end
f (generic function with 1 method)

julia> f(3)
3

This first naive attempt is not backwards-compatible, as evidenced by the necessity to modify the name of a variable called where in loading.jl. We should fix that before merging this.

Related Discourse thread: https://discourse.julialang.org/t/some-nice-to-have-syntax/24331

[tkluck]

Bold thought: we could deprecate where as an identifier in 1.2, so we can merge this for 1.3?

[yurivish]

Being able to put the where clause on a new line will be a great improvement. I remember wishing this was possible in the past. Thanks for working on it!

Re: deprecating where as an identifier, though, it's a good policy in general not to steal identifiers if it can be avoided, and I think it was decided not to do so in this case (see one of my favorite issues, #20337).

It seems where T ... is not valid syntax, so I don't think it would be ambiguous with identifiers named where if an appropriate parsing strategy can be found.

[JeffBezanson]

If we allow this, I think we should permit at most one line break before the wheres (and between them). We will also have to look ahead one token and only continue parsing wheres in certain cases. There are some ambiguities unfortunately. For example where ( could be part of a type expression where (T) or it could be an attempt to call a function called where with a forbidden space before the paren.

[tkluck]

For reference, let me paste a few examples / corner cases here:

using Test

function f(x::I) where I <: Integer
    return x
end

@test f(3) == 3

function g(x::I)
         where I <: Integer
    return x
end

@test g(3) == 3

function h()
    where where where # an expression equal to Any
end

@test h() === Any

function j(x::I) where(Union{} <: I <: Any)
    return x
end

@test j(3) == 3

function k(x::I)
        where (Union() <: I <: Any)
    return x
end

@test k(3) == 3

function where(x)
    return x
end

function l0(x::I) where(I)
end

@test l0(3) == nothing

function l(x::I)
    where(x)
end

@test l(3) == 3

function m(x::I) where I
    where(Union{} <: I <: Any) # a function call: where(true)
end

@test m(3) == true

function n(x::I) where J
    where (Union{} <: I <: Any) # a continuation of the where clause
end

@test n(3) === nothing

function o(where::Integer)
    where
end

@test o(3) == 3

function p(integer::where) where where
    where
end

@test p(3) == Int

function q(x::I) where I <:
                       Integer
   return x
end

@test q(3) == 3

function r(x::I) where
                 I <: Integer
   return x
end

@test r(3) == 3

[tkluck]

If we allow this, I think we should permit at most one line break

@JeffBezanson what's your opinion on lines that are empty except for comments?

function f(x::I, y::J)
         # I represents the type of x
         where I <: Integer
         # J represents the type of y
         where J <: Real
    return x * y
end

[JeffBezanson]

Hmm, yes, comments look useful there. Maybe allowing unlimited whitespace and comments is ok.

[tkluck]

Just pushed a working version(!).

I got stuck on this for a while trying to implement lookahead in the token stream in lisp. That was quite fragile as even token parsing is dependent on some global variables related to the parsing of white space. I gave up after a while.

This week, I realized that there's already some buffering going on at the C level, waiting to be (ab)used for this. At the cost of some breakage of separation-of-concerns, I pushed the where lookaheads down to that level. That allowed a successful implementation.

I wouldn't be very surprised if it's still possible to create adversarial examples that make this not-quite backwards compatible, but I included some corner cases as tests with this commit and they are all fine.

In my opinion, this might be good to go (modulo the todo's below). I'd also recommend making where a keyword in julia 2.0 just to remove this annoying parsing situation from that point forward.

Opinions?

Still to do:

• wrap the lookahead functions as methods on the token stream
• enumerate all possible white space (currently only ' ', should also add \t, maybe more?)

[tkluck]

PS thanks to Stefan Karpinski for this comment. Has been helpful to me in many different circumstances!

[StefanKarpinski]

Suggested change

	while(1) {
	while (1) {

Glad it's working out! It's kind of a relief when you realize you don't have to figure out the while loop condition right at the start of the loop 🙂

[tkluck]

@JeffBezanson looking forward to any comments you may have. Without pressuring (as I'm sure there a lot on your plate), do you think you can give an ETA for this? That would help me set my expectations.

[fingolfin]

This surprisingly has no conflicts :-) But of course these days it would also require changes in JuliaSyntax (CC: @c42f)

But most importantly I think no final verdict was made on whether this is wanted ...

[c42f]

it would also require changes in JuliaSyntax

I'm unconcerned - JuliaSyntax is capable of peeking at an arbitrary number of future tokens which seems to be the main issue implementing this. (Unlike the flisp parser, which only does a single peek ahead, but also restructures expressions after they've already been parsed to compensate for the fact that a single peek ahead is not really adequate 😆)

In general I'm for this syntax change. However the complicated cases of allowing where to be both an identifier and (contextual) keyword mixed up are concerning - I do wonder if there's a way to make those a parse error.