[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: D
Labels:
None
Environment:

DMD 2.060

Description

import msgpack;

class Foo
{
    int x;
    Bar obj;
}

class Bar
{
    int x;
    Foo obj;
}

void main()
{
    auto foo = new Foo();
    auto bar = new Bar();
    foo.obj = bar;
    bar.obj = foo;
    ubyte[] data = msgpack.pack(foo);
}

This creates a stack overflow.

Trackbacks

just click the following webpage (just click the following webpage)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

last minute vakanties zon (last minute vakanties zon)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

peachy payday loans (peachy payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

ez payday loans (ez payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

speedy payday loans (speedy payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

criminal background check free (criminal background check free)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

on line Payday loans (on line Payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

instant approval payday loans (instant approval payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

payday loans vancouver (payday loans vancouver)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

pay day loans today (pay day loans today)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

quick and easy payday loans (quick and easy payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

online payday lender (online payday lender)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

payday loans (payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

united payday loans (united payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

payday cash loan (payday cash loan)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

Online Payday Loans Canada (Online Payday Loans Canada)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

castle Payday loans (castle Payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

payday loan relief (payday loan relief)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

loan modifications (loan modifications)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

Payday loans same day (Payday loans same day)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

payday loan (payday loan)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

cheap bank loans (cheap bank loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

Direct Payday Loans (Direct Payday Loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

american payday loans (american payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

peachy payday loans (peachy payday loans)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

online payday loans no fax (online payday loans no fax)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

pay day advance (pay day advance)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

payday loans over the phone (payday loans over the phone)

[#MSGPACK-81] Cannot serialize class with circular references - MessagePack

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Andrej Mitrovic added a comment - 11/Nov/12 9:49 PM

Ok I can see a comment in msgpack-d that it's not supported. I have a branch that implements a workaround, however it can't be compatible with the MessagePack protocol because it introduces a new header (essentially a tag that marks a field as being a reference to an object). And I'm not sure if it would work with streaming.

Show

Andrej Mitrovic added a comment - 11/Nov/12 9:49 PM Ok I can see a comment in msgpack-d that it's not supported. I have a branch that implements a workaround, however it can't be compatible with the MessagePack protocol because it introduces a new header (essentially a tag that marks a field as being a reference to an object). And I'm not sure if it would work with streaming.

Hide

Permalink

Masahiro Nakagawa added a comment - 13/Nov/12 2:12 AM

Hi Andrej,

> Ok I can see a comment in msgpack-d that it's not supported
Yes. Some serialization protocols don't support circular reference, e.g. MessagePack, JSON, Thrift and etc. This is important for fast / compact format and easy implementation. No circular reference almost never seems to be an issue in many production environments.
On the other hand, language specified serialization often supports its feature, e.g. Ruby, Java, C#'s DataContractSerializer and D's Orange. But those protocols can't communicate other languages and generally slow...
If you want circular reference support, you should choose trade-off. Of course, extending msgpack internally is another resolution

In addition, MessagePack doesn't consider circular reference support now.

Show

Masahiro Nakagawa added a comment - 13/Nov/12 2:12 AM Hi Andrej, > Ok I can see a comment in msgpack-d that it's not supported Yes. Some serialization protocols don't support circular reference, e.g. MessagePack, JSON, Thrift and etc. This is important for fast / compact format and easy implementation. No circular reference almost never seems to be an issue in many production environments. On the other hand, language specified serialization often supports its feature, e.g. Ruby, Java, C#'s DataContractSerializer and D's Orange. But those protocols can't communicate other languages and generally slow... If you want circular reference support, you should choose trade-off. Of course, extending msgpack internally is another resolution

In addition, MessagePack doesn't consider circular reference support now.

Hide

Permalink

Andrej Mitrovic added a comment - 13/Nov/12 3:03 AM

Yeah, compatibility would be broken. But it seems fairly easy to implement: https://github.com/AndrejMitrovic/msgpack-d/tree/ClassRefs

It will come in handy for me for offline use.

All that's needed is a counter that creates sequential IDs (based on the order in which the objects are serialized) and a hashmap that holds key:address->value:ID of objects which are serialized. When an object reference is found which is in this hashmap it means it has already been serialized, so the packer retrieves the ID and stores a header and the ID instead of storing the object again.

So for example in this code:

import msgpack;

class A { A a; B b; C c; }
class B { A a; B b; C c; }
class C { A a; B b; C c; }

void main()
{
    A a = new A;
    B b = new B;
    C c = new C;
    a.a = a; a.b = b; a.c = c;
    b.a = a; b.b = b; b.c = c;
    c.a = a; c.b = b; c.c = c;
    
    ubyte[] data = msgpack.pack(a);
    writeln(data);
}

In the current implementation the byte array would never be completed:

[93 93 93 93...] -> stack overflow

In the new implementation I use a new header "0xFA", followed by either 4 or 8 bytes (32bit/64bit) denoting the ID.

Some samples:
https://github.com/AndrejMitrovic/msgpack-d/blob/ClassRefs/example/circular_class.d
https://github.com/AndrejMitrovic/msgpack-d/blob/ClassRefs/example/circular_class_2.d

I haven't implemented it for pointers yet, just class references.

Show

Andrej Mitrovic added a comment - 13/Nov/12 3:03 AM Yeah, compatibility would be broken. But it seems fairly easy to implement: https://github.com/AndrejMitrovic/msgpack-d/tree/ClassRefs It will come in handy for me for offline use. All that's needed is a counter that creates sequential IDs (based on the order in which the objects are serialized) and a hashmap that holds key:address->value:ID of objects which are serialized. When an object reference is found which is in this hashmap it means it has already been serialized, so the packer retrieves the ID and stores a header and the ID instead of storing the object again. So for example in this code:

import msgpack;

class A { A a; B b; C c; }
class B { A a; B b; C c; }
class C { A a; B b; C c; }

void main()
{
    A a = new A;
    B b = new B;
    C c = new C;
    a.a = a; a.b = b; a.c = c;
    b.a = a; b.b = b; b.c = c;
    c.a = a; c.b = b; c.c = c;
    
    ubyte[] data = msgpack.pack(a);
    writeln(data);
}

In the current implementation the byte array would never be completed: [93 93 93 93...] -> stack overflow In the new implementation I use a new header "0xFA", followed by either 4 or 8 bytes (32bit/64bit) denoting the ID. Some samples: https://github.com/AndrejMitrovic/msgpack-d/blob/ClassRefs/example/circular_class.d https://github.com/AndrejMitrovic/msgpack-d/blob/ClassRefs/example/circular_class_2.d I haven't implemented it for pointers yet, just class references.

People

Assignee:

Masahiro Nakagawa

Reporter:

Andrej Mitrovic

Vote (0)

Watch (1)

Dates

Created:

10/Nov/12 9:21 PM

Updated:

14/Nov/12 2:26 AM