Custom renderer PagerAdapter not rendering children

I've been banging my head against the wall and the decompiler for hours on this. Maybe someone smarter than me could help a guy out
I put together a Carousel renderer for Android that uses a ViewPager. My PagerAdapter creates items like so:

public override Java.Lang.Object InstantiateItem(ViewGroup container, int position)
{
        var view = Carousel.Children[position];
        var renderer = RendererFactory.GetRenderer(view);
        renderer.ViewGroup.Tag = view.Id.ToString();
        container.AddView(renderer.ViewGroup);
        return renderer.ViewGroup;
}

This works fine when the carousel's children are images but now I have the need for the children to be complex views. When the renderer that's returned from RendererFactory.GetRenderer is a RendererFactory_DefaultRenderer then nothing is rendered. I'm having a hard time tracking down specifically what the issue is though. I've also tried passing back a custom ViewGroup like the CarouselPageAdapter does, same result.

Anyone run across something like this? Maybe someone from the Xamarin team could shed some light and at least let me know if this would ever work or not?