whiteknight/io_cleanup1 branch, ready to merge

[Andrew Whitworth]

io_cleanup1 is merged. I'll be focusing 100% of my attention on
threads until that's ready to merge. Let's wait at least a few days to
make sure io_cleanup1 doesn't cause any problems before we push ahead
with the next thing.

--Andrew Whitworth


On Mon, Aug 27, 2012 at 5:11 PM, Reini Urban <rurban at x-ray.at> wrote:
> io_cleanup1 is good to merge for me.
> All tests passed on linux x86_64, solaris and cygwin.
> moritz and kid51 also gave their okay.
>
> I believe threads is also mergable in it's current state.
> sleep is a bit problematic, but it's really hard to reproduce,
> and all important platforms have no problems with threads.
> It would be a big win if rakudo can use async blocks in
> multiple cores and do hyper and race in paralell.
>
> I talked to nine and leont at length at the YAPC, what could be the
> cause of our sleep deadlocks.
> 2 theories:
> 1. alarms (unhandled signals) disturb the Timer Task.
> 2. overly optimistic task-switch granularity.
>
> On Mon, Aug 27, 2012 at 10:02 PM, Andrew Whitworth
> <wknight8111 at gmail.com> wrote:
>> I've heard back from ruban today, and done some more testing myself.
>> Everything is looking good and I am planning to merge the branch
>> tonight unless I hear otherwise.
>>
>> On Sun, Aug 26, 2012 at 9:11 PM, James E Keenan <jkeen at verizon.net> wrote:
>>> How does this branch perform on Windows?
>>
>> I've tested this today on win64, the only one I can reliably do any
>> testing on, and all tests pass. A win32 smoke report would be nice,
>> but I can't generate one of those reliably and I don't think it will
>> be much different from the win64 case.
>>
>>> Note:  Since this branch has been under development for a long time, once it
>>> is merged it would be good to have a succinct -- no more than 3 paragraphs
>>> -- post on list as to
>>>
>>> * the problems the branch was meant to address
>>>
>>> * what problems it did address and how it did address them
>>
>> This branch is a complete ground-up rewrite of the IO subsystem.
>> Originally I hadn't intended the scope to be so large, but as I
>> started cleaning and finding more messes, I just kept going. Nearly
>> 100% of the code in src/io/*.c has been rewritten. Relatively small
>> changes were made to other parts of the system, such as the IO-related
>> PMCs (FileHandle, Socket, StringHandle) and the low-level platform
>> interface (src/platform/*/file.c, socket.c, io.c, etc). It's easier to
>> list changes made than to write them all up as prose:
>>
>> 1) Pipe logic has been separated from FileHandle logic internally, in
>> preparation for a new, separate Pipe PMC type
>>
>> 2) Buffering logic has been completely unified for all types.
>> Previously, FileHandle had a completely separate buffering mechanism
>> from Socket. Now all IO types share the same exact buffering logic and
>> the same utility routines. We're able to cut down on overall code
>> volume because we now have one implementation of buffers instead of
>> two, one implementation of readline instead of 3, etc.
>>
>> 2a) Buffers were previously set up as fields on the individual handle
>> type. Now Buffers are a new struct type with it's own API (and
>> eventually it's own PMC wrapper).
>>
>> 3) Likewise various bits of functionality have been standardized,
>> because each individual type had it's own interface and semantics for
>> some operations: getting/setting encoding information, getting/setting
>> buffer details, readline semantics, open/close/is_open, etc.
>>
>> 4) type-specific operations are now broken up into a new IO_VTABLE
>> structure, which encapsulates most of the details.
>> src/io/filehandle.c, socket.c, pipe.c and stringhandle.c implement
>> these new vtables. Previously, several API functions contained large
>> and clumsy switch/case blocks for different types, with capabilities
>> and semantics being different between types.
>>
>> For one example of the kinds of semantic differences between different
>> PMC types, read
>> http://whiteknight.github.com/2012/06/13/io_readline.html. There are
>> at least a dozen other examples of varying degrees.
>>
>> 5) Several bugs have been fixed, including several that had not
>> previously been reported (but which were found in detailed testing and
>> comparison). This is especially true of Sockets and StringHandles,
>> both of which played very loose with string buffers and encodings in
>> some situations.
>>
>> 6) The C-level IO API has been completely rewritten, with old-style
>> functions kept around as thin compatibility wrappers (which can be
>> removed after some delay). Functions at the C level are all named and
>> implemented using a much cleaner, standard scheme. Also, the source
>> code is commented much more than it ever has been, so other people can
>> see what i've done and fix any mistakes I've made much more easily in
>> the future.
>>
>>> * what problems it failed to address
>>
>> This is only the first stage of what is, I hope, an ongoing effort to
>> make the IO system better. I cut off at the first reasonable stopping
>> point, where almost all features that had existed in the old system
>> had been recreated. This includes some semantics that I think are
>> lousy, but are needed to keep the tests happy. Some new features,
>> especially dealing with pipes and buffers, have not yet been
>> implemented. Creating and managing buffers, for instance, is still
>> extremely clumsy to do from PIR or above. Readline in particular has
>> had a major overhaul and there may be opportunities to improve
>> performance there, but performance of most of the rest of the system
>> should show no major regressions.
>>
>> This is the first, and largest of what I hope are many waves of
>> improvements to this subsystem, and it brings a very welcome measure
>> of sanity to a system that hasn't gotten much love in a long time.
>>
>> --Andrew Whitworth
>> _______________________________________________
>> http://lists.parrot.org/mailman/listinfo/parrot-dev
>
>
>
> --
> Reini Urban
> http://cpanel.net/   http://www.perl-compiler.org/