Populating an AnyExistentialContainer

[NSExceptional]

It would be nice if there were an inverse function of AnyExistentialContainer.projectValue() that accepts a pointer and does all the proper checks to read the correct number of bytes and store the value!

I see there are two initializers that take type metadata, but without this helper function we are left to reading how you implemented projectValue() to figure out how to store data in the container ourselves

Edit: after thinking about it more, maybe this should be a discussion, since it is not obvious to me how memory management of Any works for heap objects when you create it yourself… O_o Like, when is a HeapObject allocated for an Any container when the value isn't stored inline?

Can you convert this to a discussion for me @Azoy?

[Azoy]

A HeapObject is allocated as soon as one needs Any for a value that can't be stored inline.

struct Big {
    let w: Int
    let x: Int
    let y: Int
    let z: Int
}

func thing() {
    // swift_allocObject
    let bigThing: Any = Big(w: 0, x: 1, y: 2, z: 3)
    // swift_release
}

[NSExceptional]

I'm referring specifically to your less-magical version of Any 😅

In my case, I'm using pointers to allocate buffers for types as I construct them. The language won't help me here; there's no way for me to put something into Any without a concrete type first, which I do not have. This is what I was starting to do, but then I realized this is only going to work for inline values:

extension AnyExistentialContainer {
    mutating func store(value newValuePtr: UnsafeMutableRawPointer) {
        let valuePointer = UnsafeMutableRawPointer(self.projectValue())
        valuePointer.copyMemory(from: newValuePtr, byteCount: self.metadata.vwt.size)
    }
}

...

let value: Any = ...

// Decode the value into a buffer, copy the buffer into
// a new AnyExistentialContainer and return it as Any
let newValue = try self.decode(type: type, from: value)
var box = AnyExistentialContainer(metadata: type)
box.store(value: newValue)
props[key] = box.toAny // inverse of container(for:)

[Azoy]

Ah I think what you're looking for is allocateBoxForExistential. There was some code I wrote a while back that reimplemented Swift's Mirror entirely in Swift that I think could help you here: https://github.com/Azoy/Echo/blob/f3ba95c4f9eeb72a63c84dc6f944fe1852990df9/Sources/EchoMirror/ReflectionMirror.swift . I eventually got rid of it because I wanted this library to be the building blocks and fun things like that should be elsewhere.

[NSExceptional]

That appears to be what I need, but… where does the actual value buffer get stored in that function?

public func allocateBoxForExistential(in container: ...) -> UnsafeRawPointer {
    guard !vwt.flags.isValueInline else {
        return container.raw
    }

    let box = swift_allocBox(for: self)
    // Store a reference to the heap object in the container...
    container.pointee.data.0 = Int(bitPattern: box.heapObj)
    // but not a reference to this, the buffer?
    return box.buffer
}

The definition for HeapObject appears to have no reference to the buffer itself:

public struct HeapObject {
    public let type: Any.Type
    let _refCount: UInt64
}

But if the buffer reference is not stored in Any's data tuple, how does the runtime access it later? You return it to me, yes, but isn't it also supposed to be stored in the Any itself?

Edit: looking at your projectValue() method, am I right in assuming that the runtime expects the buffer to directly follow the HeapObject in memory, like below?

  BoxPair                                   
┌──────────────┐            ┌──────────────┐
│ HeapObject * │───────────▶│ type         │
├──────────────┤            ├──────────────┤
│ Buffer *     │─────┐      │ _refCount    │
└──────────────┘     │      ├──────────────┤
                     └─────▶│ Buffer       │
                            │              │
                            │              │
                            │              │
                            │ ...          │
                            └──────────────┘

Seems like a waste of space not to just copy the entire BoxPair into the first two words of the data buffer

[Azoy]

So the way allocating HeapObjects works is that given:

struct Big {
  let data: (Int, Int, Int, Int)
}

when we want to store a value of type Big in an Any, we have to allocate a heap object for it right. The information that we give when allocating are the size and alignment, so in this case the size of the heap object is sizeof(HeapObject) + sizeof(Big) which on a 64b computer is 48 bytes. The layout of a heap object actually looks something like this:

struct HeapObject<T> {
  let type: Any.Type
  let refCount: Int
  let value: T
}

so if T is equal to Big, then we get

struct HeapObject {
  let type: Any.Type
  let refCount: Int
  let data: (Int, Int, Int, Int)
}

BoxPair in this case returns a pointer to the start of the heap object and the buffer pointer which is immediately after the ref count of said heap object to begin storing values in the space allocated.

While writing this it seems you've figured out most of it. Swift runtime functions operate on HeapObject * so storing the buffer pointer doesn't really mean anything because we already have a pointer to the entire heap object. But yes, that's how when given the buffer pointer we are actually storing values within the Any because it has that pointer to the whole heap object.

[NSExceptional]

Gotcha, it's a trailing object. I need to start a PR adding some comments to help address #19.