Hey there fellow programmers!

I need some help with a program of mine. I need to read chunks of data from a TCP socket, and I can't figure out a smart way to do so.

It is a client/server model, using TCP sockets. The client is supposed to send ingormations about files, and the server has to process them. The data for each file is:

* filename (arbitrary size, up to PATH_MAX)
* sha512 digest for this file (128 bytes)
* mtime for this file (arbitrary size, up to 19 bytes)

I'm now wondering what could be the best way to send this data from the client to the server, and for the server to process the data.
Keeping in mind that I want to be able to send data about multilple files over the same connection (I might have to uses threads here though).

So far, the best solution I could think of is to create a structure for each file, and send the full struct over the network. This doesn't seem perfect as it would always send the maximum size for each field in the struct.

What do you guys think?

PS: Link to the project, if you want to give it a go: http://git.z3bra.org/synk/file/synk.c.html (for now, only the timestamp is send through the socket, as I couldn't find a correct way to handle it)

You could use the structure you have at the moment adding an extra field at the beginning for the number of fields.
It could be just a byte so it's small enough and the server can always unpack the first byte and know that's the number of fields to decode next.

Other than that there are other stuffs like ASN.1 and some TLV. However, those are more generic way of sending structure and thus they might be heavier than predefined static fields.

(23-08-2016, 12:34 AM)venam Wrote: adding an extra field at the beginning for the number of fields.

By "field", you mean the number of full structures that will be sent? Because the number of field INSIDE the structure will be fixed.
The ASN.1 and TLV methods are indeed too complex for my need. I wasn't aware of these though, so thanks for those links!

(23-08-2016, 04:21 AM)z3bra Wrote: By "field", you mean the number of full structures that will be sent?

Yes, that's what I mean.
You unpack the first Byte of the stream received and you can assume it's the number of structures in the rest of the stream.

I could also just read in chunks of "sizeof(struct data)" until the client closes the connection, right?

It seems your issue is with the arbitrary sizes.
So maybe you could divide it into:

That would optimize it for big values but has an overhead for small values.
However, with that approach you don't have to set a static size for the whole struct.

The structure would look like this:


By using a structure, I don't need to pass everything as a string. The conversion to "strings" will then be done on the server side.

The above would result in a packet size of 255 + 64 + 8 = 327 bytes. That's totally acceptable in the end.
As my "biggest" values are still low, I don't think it's worth passing the length along. The size might be arbitrary, but in this case I'll be passing the maximum size for each field. For example, even if my filename is only "/etc/profile", I'll pass it as a 255, right padded with 0. This will make it easier to read from the socket as I know in advance that I need to read "sizeof(struct metadata)" bytes.

I ended up using the static structure approach, which seems to work pretty well! (At least, it's easy to work with). You can see it through this commit: http://git.z3bra.org/synk/commit/8c80e23...060a7.html

By the way, it doesn't mean I'm mind-closed, so if someone as a more efficient idea, please go ahead!

Off topic/on topic? Is this going to be a simpler alternative to syncthing? I'm quite intrigued.

This is something similar yeah. But more like a one shot tool that you can fire up either via cronjobs or inotify events.

Right on. Willy, can I call you Willy?, you may be the first person I follow on the interwebs. I know, I know. Such an honor. But who am I anyway??

Haha that's the first time I see my first name on a forum :)

hey whats up willy hows it goin