Adding modifiers to NCI

[Geoffrey Broadwell]

(I've been out of this thread for a few days due to being quite sick;
thankfully, chromatic has been doing a better job than I would have of
arguing a viewpoint I share.  Thanks, chromatic.)

On Fri, 2008-12-26 at 21:17 -0800, chromatic wrote:
> "Intuitively obvious" has never been my goal nor complaint.  I just want 
> something easier to use than XS typemaps.
> 
> I want people writing NCI bindings to C functions to say "Hey, these signature 
> strings remind me of C function signatures so well that it's very difficult to 
> get them wrong!"
> [...]
> What could be more consistent with 40 years of C function signatures than the 
> same C function signature syntax that C function signatures have had for 40 
> years?
> [...]
> It doesn't, because every NCI signature expresses C calling conventions.  
> That's what they're for.
> [...]
> The details behind the proposed abstraction will always leak through the 
> abstraction because the details are the point of the abstraction.  (Also, if 
> we're still using C calling conventions within Parrot for flow of control in 
> two years, we've probably done something wrong.)

I'm completely in agreement that NCI signatures must express the full
generality of C signatures -- including (as Reini pointed out) such
things as variant calling conventions.  Anything else just means at the
very least hard-to-debug problems, and quite likely entire APIs that we
can't reach without hand-written C wrappers to impedence match -- which
seems like a pointless exercise for an NCI system.

As a data point: Under the current NCI signature system, my OpenGL
wrappers completely lie about data types in more than a few places,
pretty much praying that "most of the time" things will work.  But even
then there are quite a few signatures -- and entire APIs -- that can't
be expressed at all.  I ignored this under the assumption that NCI
signature syntax would get a massive overhaul at some point.  I'm glad
this appears to be on people's minds.  :-)

However, given the comment that you (chromatic) want something easier to
use than XS typemaps, I'm not entirely clear if you're suggesting that
we stop using our own NCI signature syntax (and just directly parse C
signatures), or if you're proposing a cleaner NCI signature syntax that
expresses the same information without the 40 years of horrid syntactic
cruft.

To anyone that wants to make a new NCI signature syntax that expresses
the semantic meaning of C signatures with a "better" syntax:  What
exactly do you propose?  There are an amazing number of edge cases to
consider.

Near and dear to my heart are: callbacks that don't include an opaque
user-provided pointer, *both* magic and exact data sizes (some APIs want
to deal with a "long", others with an "int32"), byte ordering and
signedness, data types larger or smaller than native CPU registers,
union types, and pointers that mean all of:

1. Out param: any simple value (*)
2. Out param: any single pointer on this list (**)
3. Out param: any double pointer on this list (***)
4. Pointer to struct (*)
5. Pointer to struct of pointers (*)
6. Pointer to list (*)
   a. counted by a different parameter in the signature
   b. counted by the first element
   c. terminated by a special value (usually some number of 0 bits)
   d. implicitely counted (API has multiple entry points for "same"
      function, each with different list lengths -- common in OpenGL)
7. Pointer to list of pointers (**)
   -  (any combination of variants seen under item 3)
8. Pointer to list of structs (* or **)
9. I'm just getting started ....

Obviously, what C signatures see as only 3 variations (*, **, ***), or
perhaps 6 if you explicitely include 'struct *' et al., in reality
unfold into a great many different calling conventions (in the generic
sense, not stuff like __stdcall).  How do you propose to support this
huge mess in a way that is any cleaner than C signatures and manual
poking of the bits?

Last I looked, (Un)ManagedStruct combined with current NCI was way
insufficient to express all this; it doesn't handle edge cases well, or
at all.  Even when it can be made to work, slinging (Un)ManagedStruct
around can be a major pain in the kiester for something that seems like
it should be much easier.


-'f